Preventing Kinect Flickering

Lately, I’ve been playing around with real-time matching using 3D point clouds taken with the Kinect sensor (read my previous post for details). When visualizing the stream of data, I detected some kind of flickering, which expresses itself in drastic change of the number of valid points between two frames. Browsing the web I found multiple references to folks reporting the same kind of behavior. Up to this date it remains unclear to me of what causes the flickering, but I suppose it has todo with any of the following points

  • Lighting conditions
  • USB port troubles
  • Failure of the device driver thread to collect data fast enough.
  • Internal algorithm details

Data flickering becomes a real world problem, when doing real-time object tracking, where one is interested in stable point cloud sizes between subsequent frames.

We tackle the flickering with a simple statistical approach that classifies a given frame as flicker or not-flicker. It works like this: accumulate the samples (a sample corresponds to the number of valid points in a frame) using a rolling mean. If the sample deviates significantly from the mean, classify it as flicker. Otherwise, classify it as non-flicker. Update the accumulator with the query sample to accommodate for changing environments. We’ve found that a window size of twenty samples works well in practice.

Here’s the code that does the flicker detection (requires Boost).

namespace acc = boost::accumulators;

/** Detect Kinect frame flickering. 
  *
  * \note flicker_detection uses a rolling mean over the last 
  *       twenty samples, where each sample is the number of 
  *       valid points per frame. If current number of points
  *       deviates significantly from the mean, the sample is
  *       classified as flicker.
  *       
  *       No matter how the sample is classified, the 
  *       accumulator is updated with the last sample to 
  *       accommodate for changing environments.
  *       
  *
  * \author Christoph Heindl 
  *         christoph.heindl@gmail.com
  *       
  */
class flicker_detection {
public:
  /** Construct a new instance of the flicker detection class */
  flicker_detection() 
    : _acc(acc::tag::rolling_window::window_size = 20)
  {}

  /** Test for flickering and update accumulator*/
  bool operator()(unsigned npoints) {
    unsigned m = acc::rolling_mean(_acc);
    _acc(npoints);

    return npoints < unsigned(m * 0.8);
  }

private:
  /** Defines the accumulator */
  typedef acc::accumulator_set<
    unsigned, 
    acc::stats< acc::tag::rolling_mean > 
  >  rolling_mean_acc;

  rolling_mean_acc _acc;
};

I think it is pretty self-explanatory, but just in case, here’s how to use it

flicker_detection fd;

while (/*some condition */) {
  points = kinect.frame();
  if (fd(points.size())) {
    // frame suffers from flickering
  } else {
    // frame does probably not suffer from flickering
  }
}

You may use the above source code at your own risk, provided that full and clear credit is given to Christoph Heindl and cheind.wordpress.com.

About these ads

5 thoughts on “Preventing Kinect Flickering

  1. Pingback: Kinect Support on Windows through OpenCV « Christoph Heindl

  2. Hello Christoph, I am doing a mildly similar thing with Kinect data and the Boost Accumulator libraries – running averages along the axis to help with gesture recognition, ran into a snag and ran across this blog entry. I was wondering if you found the Accumulator libraries to actually release the out of date sample data by using the rolling window? My test implementation seems to not release the old samples, resulting in an egregious memory issue. Cool stuff in this blog entry, btw!

  3. Hi Steve,

    by the time of writing i was using boost 1.46 and outdated samples were dropped by the accumulator. Did you specify the window size correctly? I still have the implementation around and will ensure my claims by today or tomorrow.

    Best,
    Christoph

    • Hi Christoph,
      No worries, don’t intend for you to be Boost technical support! :)
      I was running a barebones test case but the memory leak was caused by not properly dereferencing the return value. I was monitoring the Accumulator itself by adding the Count stat to the Accumulator template, and count(acc) does indeed count the number of samples pushed in, not the total samples stored – constant in a circular buffer, which threw me off.
      I truly hope you didn’t spend much or any time on this!
      Cheers,
      Steve

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s