L3D Cube visualizations Part 4: depth and color projection with the Kinect


The Kinect is a traditional camera doubled with an Infra Red Camera, enabling it to perceive depth in addition to the color information.

Originally sold for Xbox, it is now available for PCs under the brand Kinect for Windows. Don’t let the name fool you, it will work just as well on OSX or Linux.

Similarly to what we did for the webcam, we connect to the video stream of the Kinect, analyze each frame and downsize them so that they can be displayed on the cube’s 8*8 resolution.

But this time, we will also extract the depth information that the Kinect returns along with the color information of every pixel. In the same way that we averaged the RGB values to recompose a smaller output image, we will compute the average depth of each new pixel. This depth will be used to position the voxels on the z axis of the cube.

This demonstration was made using a Kinect V2.


If you have not already done so, I encourage you to read the last part of this series of tutorials as it explains how the algorithm used to downsize an image works.

Initial set-up

The following steps concern the Windows installation. For any other OS, please refer to this documentation.

Kinect SDK and drivers

First download the official SDK from Microsoft at this address.

Then you need to download this utility to install the alternative drivers used to communicate with processing: Zadig.

Finally follow these instructions in order to install the proper drivers.


The code

Link to the repository.

Server: Processing sketch

Link to the sketch.

Initialize the Kinect

Import the library and create a global variable for the Kinect.

Initialize the kinect in setup(), making sure to initialize the modules needed as well.

Color image extraction

We use the same method employed with the webcam, except that we allow the user to provide a fraction of the original image as an input to be downsized.

This is done so that the area mapped to the cube can be made smaller than the default one, as such a wide area can map poorly when downsized especially when depth is involved.

We use the registered image and not the one straightly from the video because the first has been adjusted so that its orientation match the image from the IR camera, which is not the case with the latter.

First define the global variables that will hold the options.

Define a new getColorSubset() function. It is analog to the pixelateImage() we saw in the last part, except that it implements the new resizing functionality that we evoked.

Depth image extraction

The algorithm applied is the same, but this time on the greyscale image representing depth value returned by the IR camera. The original value lies between [0 ; 4500]. This image is composed of the correspondingly mapped greyscale value (range: [0 ; 255]).

We average this value over the delimited areas of the image and store it in an output image as the red value of the pixel.

Some values are discarded as they correspond to an unreachable depth and act as noise in the result.

UI Events

Same thing then with the webcam, but with a twist: we want to be able to toggle between the depth and color image view. This will be triggered by pressing any key on the keyboard.

First the state global variables.

And the event listeners.

Rendering functions

Let’s start by setting up everything for the cube.

Then continue with the function that will render the 2D image. Remember that it gives the possibility to display the depth or the color image, and that the depth will be displayed as a level of red.

Finally add the function that will render the kinect’s output on to the cube. We generate two 8*8 output images from a 256 centered extract of the original images.

One of the image holds the colors and is used to set the color of the voxels. The other stores depth information as a level of red. This value is mapped on a [0 ; 7] range and used to position each voxel on the z axis of the cube.

Putting it all together

Populate the draw() loop with the rendering functions and the display options.

Client: Photon firmware

Upload the following code to your device.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.