Programming

L3D Cube visualizations Part 3: webcam stream projection

Overview

The video stream’s frames are divided into 8×8 squares of equal surface. The average RGB values of every pixel in the square is extracted and used to recompose a smaller image. The image is then projected on to the cube.

With enable3d option set to true, the past 7 frames are stored and displayed on the back frames of the cube with a delay set by the variable updateFrameRate.

The webcam stream could easily be substituted for any video stream if need be.

Walkthrough

Libraries

  • L3D Library: this one we know of already.
  • Processing Video library: it comes with Processing so no need to install it. It is used to communicate with the computer’s webcam.

The reduction algorithm

Since the cube is an 8*8 display, we need to downsize the original image from the webcam so that it fits on 64 pixels.

It is important to understand how the algorithm used to reduce the number of pixels of an image works.

You will quickly see that, if broken down into small pieces, it is not a difficult concept to grasp at all.

Let’s take the example of an image of width and height of 32 pixels. It is composed of 1024 pixels total. Say we want to downsize it to a 4*4 image.

First we divide the initial image in 64 squares of equal surface. We then iterate through each pixel of each of these surfaces, extracting the average R, G and B value from every pixel lying on the same surface.

This average result is used to re-compose a downsized image.

Here is a poorly made gif that might help you understand the idea, as well as an example of the result obtained with this method.
I don’t know why but the thumbnail of the gif eats up part of the graphics so you might want to click on the link to see the full version.

The code

Link to the repository.

Server: Processing sketch

Link to the sketch.

Capture webcam stream

Import the library used to stream the webcam feed and declare a Capture global variable in order to store the frames.

import processing.video.*;

Capture video; // webcam stream object

Start the webcam feed in setup().

void setup() {
  // start webcam stream 
  video = new Capture(this, 640/2, 480/2);
  video.start();
}

You also need to register a new event listener that will read the new frames incoming from the webcam.

void captureEvent(Capture c) {
  c.read();
}

Downsize frames images

We create a new pixelateImage() function that reads a frame from the webcam, resizes it to a square so that it fits the cube, creates a new output image that will be composed following the algorithm aforementioned.

PImage pixelateImage(int sideSize) {
  PImage p;
  p = video.copy(); // copy current webcam frame in PImage object
  p.resize(inSideSize,inSideSize); // resize to get a square (since we want an 8x8 output)
  // Create empty image that will store the current frame pixels values
  PImage pOut = createImage(outSideSize, outSideSize, RGB);
  pOut.loadPixels(); // init pixels
  
  int pxSize = inSideSize/sideSize;
  
  // Apply algorithm to extract average RGB value of each area of 
  // the input image and recompose the output image with the results
  for (int x=0; x

We also create a function that will allow us to display the resulting image in the rendered view for test purposes.

void show2D(int size) {
  PImage pOut = pixelateImage(size); // generate pixelated image
  scale(inSideSize/outSideSize); // scale image object to fill the rendering screen
  image(pOut, 0, 0); // display output image
}

Project to cube

We use the functions just created to project the output image pixel's values to the cube's first frame's voxels.

As mentioned, if enable3d is set to true, the past webcam frames are sent to the back of the cube with a delay.

We begin by importing the L3D library and setting the global variables.

...
import L3D.*;

// Instanciate cube;
L3D cube; 	

...

int[][] frames = new int [8][64]; // Array used to store the frames pixels values
int nextFrame = 0; 

boolean enable3d = false; // if set to false, only 1 frame is used.
int updateFrameRate = 5; // if enable3d set to true, frequency at which the past frames are updated

int inSideSize = 512; // Size of the origin image once resized. Must be a multiple of 8. 
int exp = 9; // used to recompute outSideSize on keyboard action
int outSideSize = 512; // number of pixels/side for the output image
...

Then we start the cube in setup() and prepare the view.

void setup() {
  size(512, 512, P3D); // start simulation with 3d mode enabled
  
  ...

  cube=new L3D(this); // init cube
  cube.enableMulticastStreaming(2000); // start streaming current animation on port 2000
}

Then we define the function used to render the cube.

void showCube() {
  cube.background(0); // clear cube background
  
  PImage pOut = pixelateImage(8);
  frames[7] = pOut.pixels; // save current frame pixels (7 = front of the cube)
  // loop through the 8 frames and light up pixels row by row 
  for (int j=7; j>=0; j--) {
    int i = 0;
    for (int y=7; y>=0; y--) {
      for (int x=0; x<8; x++) {
      cube.setVoxel(x, y, j, frames[j][i]);
      i++;
      }
    }
  }   
  
  // update frames displayed by pushing old frames values to the back
  // the frequency is set in global variable 'updateFrameRate'
  // 'enable3d' mode must be enabled by setting the global variable to 'true'
  if (enable3d&&nextFrame

UI events

Weather it be for curiosity or testing purposes, it can be useful to try different output resolutions and to toggle between the 2D and 3D view.

In order to do that, we register a new global variable and two event listeners which will toggle the views between the cube and the output image on a right click. Additionally 0 and 1 can be used to decrease or increase the output image's resolution.

int mode = 0; // used to switch between pixelated image and cube view

...

void mousePressed() {
  if (mouseButton == RIGHT) {
    mode = (mode == 1 ? 0 : 1);
  }
}

void keyPressed() {
  if (keyCode ==49) {
      exp += 1;
      outSideSize = (int)pow(2, exp);
  } else if (keyCode == 48) {
    exp -= 1;
    outSideSize = (int)pow(2, exp);
  }
}

Putting it all together

Populate the draw() loop with the two rendering functions we created.

void draw() {
  cube.background(0); // init all voxels to black
  background(0); // clear simulation background

  // toggle between 2D and 3D view
  if (mode == 0) {
    show2D(outSideSize);
  } else {
    showCube();
  }
  
}

Client: Photon firmware

Upload the following code to your device.

Sources

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.