Kinect Support on Windows through OpenCV

Update 2011/08/03: Updated the post to work with the latest trunk revision (6281 at the time of writing) of OpenCV. The old post was moved into a subsection of this post.
Update 2011/06/11: I was informed that the patch provided in this blog post was applied against the the OpenCV code-base in revision 5415.
Update 2011/04/17: You may want to read my follow-up post on limiting the effect of data flickering.

OpenCV 2.3.1 rev 6281

Since revision 4640, OpenCV, OpenNI and PrimeSense KinectMod drivers have received major updates. Here’s a short write-up on how to get it going with the latest OpenCV developer revision. The following how-to uses 32bit drivers and OpenCV builds on a 64bit Windows 7 system.

1. Make sure the Kinect is unplugged.

   I’ve seen the Kinect failing to work when this step was skipped, so make sure you unplug the Kinect.

2. Uninstall any previous OpenNI and PrimeSense Sensor KinectMod drivers.
3. Install OpenNI (Version 1.3.2.3 32bit)

   Download from http://www.openni.org/downloadfiles/opennimodules/openni-binaries/20-latest-unstable

4. Install PrimeSense Sensor KinectMod (Version 5.0.3.4 32bit)

   https://github.com/avin2/SensorKinect/tree/unstable/Bin

5. PlugIn Kinect now.

   Wait for the device drivers to be installed. In the device manager locate

   • PrimeSense/Kinect Audio
   • PrimeSense/Kinect Camera
   • PrimeSense/Kinect Motor

   and make sure that the devices are ready to use.

6. Checkout OpenCV rev. 6281

   https://code.ros.org/svn/opencv/trunk/opencv/

7. Patch OpenCVFindOpenNI.cmake

   This step is necessary if you own a 64bit OS Windows. In this case OpenNI and PrimeSense Sensor KinectMod drivers get installed to c:/program files (x86), which is not searched by OpenCVFindOpenNI.cmake. The following patch might not be necessary if you use the 64bit versions of OpenNI and PrimeSense Sensor KinectMod.

   Index: OpenCVFindOpenNI.cmake
   ===================================================================
   --- OpenCVFindOpenNI.cmake	(revision 6281)
   +++ OpenCVFindOpenNI.cmake	(working copy)
   @@ -20,7 +20,7 @@
    	
    if(NOT OPENNI_LIB_DIR OR OPENNI_LIB_DIR STREQUAL "")
        if(WIN32)
   -        set(OPENNI_LIB_DIR "c:/Program Files/OpenNI/Lib" CACHE PATH ${OPENNI_LIB_DESCR})
   +        set(OPENNI_LIB_DIR "c:/Program Files (x86)/OpenNI/Lib" CACHE PATH ${OPENNI_LIB_DESCR})
        elseif(UNIX OR APPLE)
            set(OPENNI_LIB_DIR "/usr/lib" CACHE PATH ${OPENNI_LIB_DESCR})
        endif()
   @@ -28,7 +28,7 @@
    	
    if(NOT OPENNI_INCLUDE_DIR OR OPENNI_INCLUDE_DIR STREQUAL "")
        if(WIN32)
   -        set(OPENNI_INCLUDE_DIR "c:/Program Files/OpenNI/Include" CACHE PATH ${OPENNI_INCLUDE_DESCR})
   +        set(OPENNI_INCLUDE_DIR "c:/Program Files (x86)/OpenNI/Include" CACHE PATH ${OPENNI_INCLUDE_DESCR})
        elseif(UNIX OR APPLE)
            set(OPENNI_INCLUDE_DIR "/usr/include/ni" CACHE PATH ${OPENNI_INCLUDE_DESCR})
        endif()
   @@ -36,7 +36,7 @@
    	
    if(NOT OPENNI_PRIME_SENSOR_MODULE_BIN_DIR OR OPENNI_PRIME_SENSOR_MODULE_BIN_DIR STREQUAL "")
        if(WIN32)
   -        set(OPENNI_PRIME_SENSOR_MODULE_BIN_DIR "c:/Program Files/Prime Sense/Sensor/Bin" CACHE PATH ${OPENNI_PRIME_SENSOR_MODULE_BIN_DESCR})
   +        set(OPENNI_PRIME_SENSOR_MODULE_BIN_DIR "c:/Program Files (x86)/PrimeSense/SensorKinect/Bin" CACHE PATH ${OPENNI_PRIME_SENSOR_MODULE_BIN_DESCR})
        elseif(UNIX OR APPLE)
            set(OPENNI_PRIME_SENSOR_MODULE_BIN_DIR "/usr/lib" CACHE PATH ${OPENNI_PRIME_SENSOR_MODULE_BIN_DESCR})
        endif()
   
   

8. Start CMake with OpenCV directory

   Select your compiler of choice and set the address model is 32bit.

   Make sure to enable WITH_OPENNI
   Make sure to enable BUILD_EXAMPLES
   Click Configure and then Generate. You might need to click Configure several times.

9. Open Solution OpenCV.sln

   Build: Release > Build all > INSTALL
   Build: Debug > Build All > INSTALL

10. Verify

    Run the kinect_maps example to test Kinect support and get a feeling for how to use the API.

OpenCV 2.2.0 rev 4640

This weekend I’ve been busy integrating the Kinect sensor into our vision framework at work, which requires a few steps to get it up and running. I decided to put them online, so anyone who cares can benefit from it.

First, I decided to go with the offical PrimseSense drivers that integrate into the OpenNI framwork. Second, the current OpenCV trunk shows efforts to integrate the drivers into its video capture framework. Since I don’t want to reinvent the wheel, I will stick with OpenCV for the remainder of this post.

Install Prerequisites

You need to install the OpenNI Framework and PrimeSense drivers. At the time of writing the current versions are

• OpenNI v1.0.0.25 (unstable)
• PrimeSensor 5.0.0 (unstable)

Install both in the given order. Note, if you are on Win64, make sure both get installed to c:\program files instead of c:\program files (x86), or you will need to patch the OpenCV CMake files.

Checkout OpenCV

Checkout the HEAD revision, or stick with the revision I’ve used in my tests 4640, of the trunk using Subversion.

Patch OpenCV

Note: Starting with revision 5415 this patch has been incorporated into the OpenCV code-base. So in case you are sticking with revision 5415 or later this patch is no longer necessary.

I needed tweak OpenCV slightly to fit my needs:

• Fix temporaray file generation on Windows.
• Align 3D point coordinates and color information.

Both tweaks are summed up in the following patch

--- cap_openni.cpp	(revision 4640)
+++ cap_openni.cpp	(working copy)
@@ -183,8 +183,7 @@
         // Write configuration to the temporary file.
         // This is a hack, because there is a bug in RunXmlScript().
         // TODO: remove hack when bug in RunXmlScript() will be fixed.
-        char xmlFilename[100];
-        tmpnam( xmlFilename );
+        char xmlFilename[] = "kinect_xml_tmp.xml";
         std::ofstream outfile( xmlFilename );
         outfile.write( XMLConfig.c_str(), XMLConfig.length() );
         outfile.close();
@@ -213,6 +212,8 @@
         CV_Assert( depthGenerator.SetMapOutputMode( depthOutputMode ) == XN_STATUS_OK );
         CV_Assert( imageGenerator.SetMapOutputMode( imageOutputMode ) == XN_STATUS_OK );
 
+        depthGenerator.GetAlternativeViewPointCap().SetViewPoint(imageGenerator); // align xyz with rgb.
+
         //  Start generating data.
         status = context.StartGeneratingAll();
         if( status != XN_STATUS_OK )

Compile and Run

Setting the correct definitions in CMake is crucial. Make sure you enable

• BUILD_EXAMPLES
• WITH_OPENNI

Build OpenCV and the examples that come along with OpenCV. Run the kinect_maps example to test Kinect support and get a feeling for how to use the API.

Note, the following screenshot is showing parts of our vision system at work, not the OpenCV example.