Facoltà di Ingegneria. Kinect calibration. Ilya Afanasyev Trento, /01/2012 1/20

Size: px

Start display at page:

Download "Facoltà di Ingegneria. Kinect calibration. Ilya Afanasyev Trento, /01/2012 1/20"

Janel Brooks
6 years ago
Views:

1 Facoltà di Ingegneria Kinect calibration Ilya Afanasyev Trento, /01/2012 1/20

2 Content 1. The description. 2. About the existing software. 3. Tests of Nicolas Burrus software binaries. 4. Open Kinect drivers installation. 5. Compilation of Nicolas Burrus source code. 6. Results. 7. Conclusions 8. References. 25/01/2012 2/20

The description. 16/01/2012 The software for Kinect Calibration should be used for: 1. Saving the RGB images and IR-Depth data from Kinect sensor under OS Windows. 2.

- distortion coefficients (kc_ir, kc_rgb). - the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb) and extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)).

3 The description. 16/01/2012 The software for Kinect Calibration should be used for: 1. Saving the RGB images and IR-Depth data from Kinect sensor under OS Windows. 2. Calibration the Kinect sensor with obtaining [1]: - the intrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb)). - the extrinsic parameters (i.e. extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)). - distortion coefficients (kc_ir, kc_rgb). - the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic parameters for IR camera (K_ir) & RGB camera (K_rgb) and extrinsic mapping between IR (depth) and RGB Kinect camera (R,T)). 3. Developing the interface of different algorithm with Kinect for different real-time object recognition applications. 3/20

4 About the existing software. The existing analogues of the software: 1. - The author doesn t know analogues under Windows7/64 bit. It can be created, basing on Microsoft Kinect SDK for developers [6] under the platform.net Framework ver There is the software developed by Nicolas Burrus [2] under OS Linux / Ubuntu / 32 bit, which is claimed to be compiled also with Windows-32 bit. Can it be used under Windows7/64 bit?? -> It seems that NO!! 25/01/2012 4/20

5 Microsoft Kinect SDK for developers [6] 1. SDK (Software Development Kids) includes drivers, APIs for raw sensor streams and human motion tracking, installation documents, and resource materials. It provides capabilities offered by the Microsoft Kinect device connected to computers under Windows 7 / Windows 8 to developers who build applications with C++, C#, or Visual Basic by using Microsoft Visual Studio /01/2012 5/20

-> supports OpenNI/Nite backends [3] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit -> there is no the

6 Nicolas Burrus software [2] Nicolas Burrus software [2]: There are different versions of the Burrus software: 1. Test programs running from binaries -> Windows binaries -> Kinect RGBDemo v > supports OpenNI/Nite backends [3] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit -> there is no the extrinsic mapping between IR and RGB camera at Kinect calibration. 2. Compiling from source -> 2.1 Compilation on Windows -> with OpenNI/Nite backends doesn t give the extrinsic mapping between IR and RGB camera at Kinect calibration. with Libfreenect backend -> I wasn t succeed to compile under Windows 7/64 bit?! 2.2 Compilation on Linux (Ubuntu bit) -> supports Libfreenect backend -> the only workable variant with getting all calibration parameters (intrinsic, extrinsic and distortion) was realized on Alberto Fornaser s PC (UniTN) 16/01/2012 6/20

1 st attempt The using of Nicolas Burrus software [2] with OpenNI backend under Windows 1. OpenNI backend: 1. Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports OpenNI/Nite backends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit.

raw and depth.raw files. -> the software doesn t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion parameters)!! 4.

7 1 st attempt The using of Nicolas Burrus software [2] with OpenNI backend under Windows 1. OpenNI backend: 1. Nicolas Burrus software [1]: Kinect RGBDemo v > supports OpenNI/Nite backends [2] and has the experimental infrared support with OpenNI (still buggy) -> I tried to check this work for OpenNI backend under Windows 64bit. 2. Install Sensor Kinect drivers under Windows 32 bit [4] and OpenNI/Nite modules (Win32) [3] according to the instruction [5] and verified that they work properly. 3. Use RGB-D Capture [1] to grab Kinect RGB and IR images with intensity.raw and depth.raw files. -> the software doesn t grab IR images!! -> only RGB image, intensity.raw, depth.raw and calibration file (calibration.yml without distortion parameters)!! 4. calibrate-openni-intrinsics --pattern-size grab1 calibration.yml -> gives openni_calibration.yml with Intrinsic matrix and distortion coefficients for Kinect RGB camera!! 5. calibrate-openni-depth.exe --pattern-size grab1 gives partly processed figures with a message about errors. 6. The software doesn t produce a file with xyz- coordinates and RGB-colors info of every point!!! 16/01/2012 7/20

The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu 10.

8 The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu bit) - OpenNI backend (under OS Windows 32bit) Libfreenect backend OpenNI backend 16/01/2012 The same figures of Intrinsic Matrixes for Kinect RGB and Depth Cameras The Depth Camera Calibration for OpenNI backend is NOT available!!! -

9 The comparison of Kinect calibration results for - Libfreenect backend (under OS Linux / Ubuntu bit) - OpenNI backend (under OS Windows 32bit) Libfreenect backend OpenNI backend 23/11/2011 There is NO extrinsic mapping between Kinect Depth and RGB cameras for OpenNI backend The reprojection of 3D points from Kinect Depth Camera to RGB image for OpenNI is NOT possible!! -

- Kinect Sensor drivers for Windows 7 (OpenKinect-libfreenect oftheo/ofxkinect) [9].

10 Install OpenKinect for Windows 7 [7] It is necessary to have: - Visual Studio 2010 (as a compiler on a 32 bit Windows 7 installation). - Kinect Sensor drivers for Windows 7 (OpenKinect-libfreenect oftheo/ofxkinect) [9]. - freenect windows dependencies [8]: libusb-win (Currently there is a breaking change in libusb , which I used!!); pthreads-win32 Glut 17/01/ /20

11 Kinect Sensor drivers installation [7] - Download & extract the latest GitHub repository [9]. - Make sure your Kinect sensor is plugged into a USB port on your computer. - Open Device Manager (Start Control Panel System and Security System Device Manager) - Find Human Interface Device (Xbox NUI Motor) Update Driver Software uninstall the previous Kinect drivers (if there are) set path like C:\Special\OpenKinect_libfreenect_v0_1_1_56\pl atform\windows\inf\xbox nui camera\ the drivers will correspond to the processor type. 18/01/ /20

$dll (x64) in C:\Windows\System32\ and libusb0.$

12 Kinect Sensor drivers installation [7] 19/01/2012 Check the files: libusb0.dll (x64) in C:\Windows\System32\ and libusb0.sys in C:\Windows\System32\drivers\ and libusb0.dll (x86) in C:\Windows\SysWOW64\ 12/20

13 Cmake configuration & Generation of a Visual Studio 10 solution [7] 1. Open Cmake GUI and set the Browse Source path to /libfreenect/openkinect. 2. Set the browse build path to /libfreenect/openkinect/build 19/01/2012 The software should be configured when Kinect turned off!! 13/20

14 Cmake configuration & Generation of a Visual Studio 10 solution [7] You will now need to point certain build paths to the freenect dependencies (pthread). 19/01/2012 The software should be configured when Kinect turned off!! 14/20

$19/01/2012 If take "msvc" instead of "msvc_x64" -> 1 fatal error of compilation less!!! It doesn't matter if choose THREADS\LIB\ pthreadvc2.lib or pthreadvse2.$

15 Cmake configuration & Generation of a Visual Studio 10 solution [7] You will now need to point certain build paths to the freenect dependencies (libusb). 19/01/2012 If take "msvc" instead of "msvc_x64" -> 1 fatal error of compilation less!!! It doesn't matter if choose THREADS\LIB\ pthreadvc2.lib or pthreadvse2.lib 15/20

Cmake configuration & Generation of a Visual Studio 10 solution [7] Run Configure again. It should successfully complete the process this time but we still need to add the Glut references.

16 Cmake configuration & Generation of a Visual Studio 10 solution [7] Run Configure again. It should successfully complete the process this time but we still need to add the Glut references. 19/01/2012 Run "Configure" and it should successfully complete. Click Generate afterwards and then open the libfreenect.sln file from the build folder the output was placed in - /openkinect/build/ 16/20

Visual Studio 10 solution [7] Disable the Antivirus if it disturbs the creation of the VS10 solution! Choose Visual C++ 2010 and then select Build Solution from the Debug Menu or press F7.

17 Visual Studio 10 solution [7] Disable the Antivirus if it disturbs the creation of the VS10 solution! Choose Visual C and then select Build Solution from the Debug Menu or press F7. 19/01/2012 RESULT: Rebuild All: 13 succeeded, 2 failed, 2 skipped. You will receive a few errors when building the solution but it should compile correctly regardless. The binaries will be placed in build/bin/debug. 17/20

18 The result of building the c for Windows 7 [7] The final step is to place a few.dll files in the appropriate folder and then we can run some of the samples provided: Copy freenect.dll and freenect_sync.dll from \build\lib\debug Copy from dependencies folders: pthreadvc2.dll, glut32.dll, libusb0.dll (choose the correct version x86/x64) Copy tiltdemo.exe, regtest.exe, glview.exe, glpclview.exe, glpclview.exe, regview.exe, hiview.exe from \build\bin\debug It can be helpful to copy freenect_sync.lib, freenect.lib, freenect.exp from \build\lib\debug to \lib\ Successful start of demo-samples Kinect works! 19/01/ /20

19 Demo-samples of OpenKinect for Windows 7 tiltdemo.exe drives the Kinect motor. regview.exe shows the combined IR&RGB video (IR up to 1 m) regtest.exe grabs bad-quality pictures: -- registration_test_depth_registered.pgm, -- registration_test_rgb.ppm, -- registration_test_depth_raw.pgm, -- registration_test_depth_mm.pgm glview.exe gives IR&RGB video in separate windows (IR up to 3.5 m) 19/01/ /20

5 m, RGB has hi-quality & hi resolution) glpclview.

20 Demo-samples of OpenKinect for Windows 7 hiview.exe gives IR&RGB video in separate windows (IR up to 3.5 m, RGB has hi-quality & hi resolution) glpclview.exe gives color 3D video with big shadows behind the objects. cppview.exe produces a continuous list with messages under the phrase: thread execution started 19/01/ /20

2 nd attempt 19/01/2012 Compiling RGBDemo-0.6.1 under Min GW from source with OpenKinect on Windows 7 [2] The step-by-step procedure for Min GW, in case using libfreenect: 1.

21 2 nd attempt 19/01/2012 Compiling RGBDemo under Min GW from source with OpenKinect on Windows 7 [2] The step-by-step procedure for Min GW, in case using libfreenect: 1. Install QT opensource for Windows. This will also install Min GW. 2. Add C:\Special\QtSDK\mingw\bin\ to the Path environment variable: Start Computer (properties) Advanced System Settings Environmental Variables (button) Variable: PATH Edit Add. 3. Install and run cmake on rgbdemo Disable the NESTK_USE_OPENNI cmake variable Could not find module FindOpenCV.cmake Configuring incomplete, errors occurred! 4. Ungrouped Entries OpenCV_DIR (NOT FOUND) C:/Special/opencv. QT_QMAKE_EXECUTABLE (NOT FOUND) C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\qmake.exe it is better to use qmake.exe from QtSDK\mingw?? 21/20

22 Compiling RGBDemo under Min GW from source with OpenKinect on Windows 7 [2] The step-by-step procedure for Min GW, in case using libfreenect: 20/01/2012 Open the CMakeLists.txt in Qt Creator (from C:\Special\RGBDemo Source\RGBDemo Source\CMakeLists.txt open with C:\Special\QtSDK\QtCreator\bin\qtcreator.exe). It creates the folder: qtcreator-build Add QT-qmake & OpenCV path to the Path environment variable: C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\ C:\Special\opencv\ to the Path environment variable: Start Computer (properties) Advanced System Settings Environmental Variables (button) Variable: PATH Edit Add. 22/20

23 Compiling RGBDemo under Min GW / MSVS 2010 from source with OpenKinect on Windows 7 [2] The step-by-step procedure for Min GW / MSVS2010, in case using libfreenect: Cmake Ungrouped Entries PKG_CONFIG_EXECUTABLE C:\Special\opencv\OpenCVFindPkgConfig.cmake Results: Cmake can t compile the QT & OpenCV_DIR OpenCVConfig.cmake We should use Qt!! Qt Run Cmake CMAKE_MODULE_PATH C:\Special\qtcreator-build\ Run Cmake CUDA_TOOLKIT_ROOT_DIR not found or specified [10] if set path to C:\Special\PCL_1_4_0\ which contains CUDA libraries, then we have the messages CMake Error: The source directory "C:/Special/PCL_1_4_0/3rdParty/flann/bin" does not appear to contain CMakeLists.txt. Specify --help for usage, or press the help button on the CMake GUI. 20/01/2012 Burrus software RGBDemo is not installed on Window7-64bit!! 23/20

24 Conclusions 1. Nicolas Burrus software with OpenNI binaries [2] works under Windows 7-64 bit, but the software doesn t give the extrinsic mapping between Kinect Depth and RGB cameras and doesn t produce a file with xyz- coordinates and RGBcolors info of every point!!! 2. Although the drivers of OpenKinect has been successfully installed under Windows 7-64 bit, the Nicolas Burrus source code RGBDemo (with LGPL License) for Kinect Calibration [2] is not compiled properly under Windows 7-64 bit. It should be used another software or another Kinect drivers!! 3. Both Microsoft Kinect SDK for developers [6] and OpenKinect drivers [9] can be applicable to save RGB images and IR-Depth data to calibrate Kinect sensor and use it under Windows 7-64 bit, but no suitable software has not been produced yet. 25/01/ /2 0

25 5. References 1. Camera Calibration and 3d Reconstruction. // OpenCV (Open Source Computer Vision) v2.1 documentation: bration_and_3d_reconstruction.html 2. Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize Kinect output. // OpenNI Modules. // 4. SensorKinect drivers. // github social coding How-to: Successfully install Kinect on Windows (OpenNI and NITE). // Vangos Pterneas blog: 6. Microsoft Kinect SDK for developers. 16/01/ /20

5. References 7. Install OpenKinect for Windows 7 and XP. // http://kinect.dashhacks.com/kinect-guides/2011/01/09/installopenkinect-windows-7-and-xp 8. Getting Started using OpenKinect.

26 5. References 7. Install OpenKinect for Windows 7 and XP. // 8. Getting Started using OpenKinect. // OpenCV: CUDA_TOOLKIT_ROOT_DIR not found or specified. 16/01/ /20

Index C, D, E, F I, J

Index C, D, E, F I, J Index A Ambient light, 12 B Blurring algorithm, 68 Brightness thresholding algorithm float testapp::blur, 70 kinect.update(), 69 void testapp::draw(), 70 void testapp::exit(), 70 void testapp::setup(),