The Camera as a Sensor *

Transcription

1 OpenStax-CNX module: m The Camera as a Sensor * Davide Rocchesso This work is produced by OpenStax-CNX and licensed under the Creative Commons Attribution License 3.0 Abstract Simple yet signicant computer vision tasks are made accessible in the Processing language and environment. Pixel-by-pixel processing is possible through the GSVideo library. The JavacvPro library provides access to a large set of functions of OpenCV. In Computer Science, computer vision studies the computational aspects of vision and develops camerabased systems as input devices. Simple techniques of pixel-by-pixel manipulation can be used to implement simple vision mechanisms. The examples that follow are taken from Learning Processing by Daniel Shiman 1 and adapted to use the GSVideo 2 library. A nice introductory article on Computer Vision for Artists and Designers: Pedagogic Tools and Techniques for Novice Programmers has been written by Golan Levin 3. 1 Color Tracking In the code p. 1, the target color is searched pixel-by-pixel in the image buer video, of size video.width * video.height. For example, the pixel of coordinates (2,3) (third row, fourth column) is accessed in position 3 + 2*video.width of the unrolled array containing all the pixels. The target color is selected via mouse clicking. // Daniel Shiffman // // Example 16-11: Simple color tracking // Modified by Davide Rocchesso // Variable for capture device GSCapture video; // A variable for the color we are searching for. color trackcolor; * Version 1.2: May 7, :15 am

2 OpenStax-CNX module: m void setup() { size(320,240); video = new GSCapture(this,width,height,15); video.start(); // Start off tracking for red trackcolor = color(255,0,0); smooth(); void draw() { // Capture and display the video if (video.available()) { video.read(); video.loadpixels(); image(video,0,0); // Before we begin searching, the "world record" for closest color is set to a high number that is eas float worldrecord = 500; // XY coordinate of closest color int closestx = 0; int closesty = 0; // Begin loop to walk through every pixel for (int x = 0; x < video.width; x ++ ) { for (int y = 0; y < video.height; y ++ ) { int loc = x + y*video.width; // What is current color color currentcolor = video.pixels[loc]; float r1 = red(currentcolor); float g1 = green(currentcolor); float b1 = blue(currentcolor); float r2 = red(trackcolor); float g2 = green(trackcolor); float b2 = blue(trackcolor); // Using euclidean distance to compare colors float d = dist(r1,g1,b1,r2,g2,b2); // We are using the dist( ) function to compare the current col // If current color is more similar to tracked color than // closest color, save current location and current difference if (d < worldrecord) { worldrecord = d; closestx = x; closesty = y; // We only consider the color found if its color distance is less than 10.

3 OpenStax-CNX module: m // This threshold of 10 is arbitrary and you can adjust this number depending on how accurate you requ if (worldrecord < 10) { // Draw a circle at the tracked pixel fill(trackcolor); strokeweight(4.0); stroke(0); ellipse(closestx,closesty,16,16); void mousepressed() { // Save color where the mouse is clicked in trackcolor variable int loc = mousex + mousey*video.width; trackcolor = video.pixels[loc]; 2 Background Removal By explicitely storing a background in a buer, it is relatively easy to remove it from the images grabbed with a video camera, provided that the illumination condition remain constant and that the camera is not automatically adjusting its exposure. In the code p. 3, the background is stored upon a mouse click. At each frame, the color of each pixel is compared with the corresponding pixel in the background image, and if such dierence is large enough it is considered to stay in the foreground. // Daniel Shiffman // // Example 16-12: Simple background removal // Modified by Davide Rocchesso // Click the mouse to memorize a current background image // Variable for capture device GSCapture video; // Saved background PImage backgroundimage; // How different must a pixel be to be a foreground pixel float threshold = 20; void setup() { size(320,240); video = new GSCapture(this, width, height, 30); video.start(); // Create an empty image the same size as the video backgroundimage = createimage(video.width,video.height,rgb);

4 OpenStax-CNX module: m void draw() { // Capture video if (video.available()) { video.read(); // We are looking at the video's pixels, the memorized backgroundimage's pixels, as well as accessing // So we must loadpixels() for all! loadpixels(); video.loadpixels(); backgroundimage.loadpixels(); // Begin loop to walk through every pixel for (int x = 0; x < video.width; x ++ ) { for (int y = 0; y < video.height; y ++ ) { int loc = x + y*video.width; // Step 1, what is the 1D pixel location color fgcolor = video.pixels[loc]; // Step 2, what is the foreground color // Step 3, what is the background color color bgcolor = backgroundimage.pixels[loc]; // Step 4, compare the foreground and background color float r1 = red(fgcolor); float g1 = green(fgcolor); float b1 = blue(fgcolor); float r2 = red(bgcolor); float g2 = green(bgcolor); float b2 = blue(bgcolor); float diff = dist(r1,g1,b1,r2,g2,b2); // Step 5, Is the foreground color different from the background color if (diff > threshold) { // If so, display the foreground color pixels[loc] = fgcolor; else { // If not, display green pixels[loc] = color(0,255,0); // We could choose to replace the background pixels with something updatepixels(); void mousepressed() { // Copying the current frame of video into the backgroundimage object // Note copy takes 5 arguments: // The source image // x,y,width, and height of region to be copied from the source // x,y,width, and height of copy destination backgroundimage.copy(video,0,0,video.width,video.height,0,0,video.width,video.height); backgroundimage.updatepixels();

5 OpenStax-CNX module: m Motion Detection To highlight motion, it is sucient to compare, pixel-by-pixel, the current frame with the previous frame. In the code Section 3 (Motion Detection ), if the color dierence is larger than a threshold than the pixel is colored black. // Daniel Shiffman // // Example 16-13: Simple motion detection // Modified by Davide Rocchesso // Variable for capture device GSCapture video; // Previous Frame PImage prevframe; // How different must a pixel be to be a "motion" pixel float threshold = 50; void setup() { size(320,240); video = new GSCapture(this, width, height, 30); video.start(); // Create an empty image the same size as the video prevframe = createimage(video.width,video.height,rgb); void draw() { // Capture video if (video.available()) { // Save previous frame for motion detection!! prevframe.copy(video,0,0,video.width,video.height,0,0,video.width,video.height); // Before we read t prevframe.updatepixels(); video.read(); loadpixels(); video.loadpixels(); prevframe.loadpixels(); // Begin loop to walk through every pixel for (int x = 0; x < video.width; x ++ ) { for (int y = 0; y < video.height; y ++ ) { int loc = x + y*video.width; // Step 1, what is the 1D pixel location

6 OpenStax-CNX module: m color current = video.pixels[loc]; // Step 2, what is the current color color previous = prevframe.pixels[loc]; // Step 3, what is the previous color // Step 4, compare colors (previous vs. current) float r1 = red(current); float g1 = green(current); float b1 = blue(current); float r2 = red(previous); float g2 = green(previous); float b2 = blue(previous); float diff = dist(r1,g1,b1,r2,g2,b2); // Step 5, How different are the colors? // If the color at that pixel has changed, then there is motion at that pixel. if (diff > threshold) { // If motion, display black pixels[loc] = color(0); else { // If not, display white pixels[loc] = color(255); updatepixels(); 4 OpenCV In 1999 Intel Research decided to develop a software library to encourage the development of CPU-intensive applications. Image processing and computer vision tasks certainly make intensive use of the CPU. Objectives of OpenCV to advance research by providing a set of optimized base functions; to encourage the production of readable and reusable software; to encourage the production of commercial software that uses an open and free infrastructure (through proper licensing). The library is written in C language, for maximal eciency and portability, but there are wrappers for C++ (openframeworks), C#, Pyton, Java (JavaCV), and Processing (OpenCV and JavacvPro). It is available under several operating systems, including Linux, Windows, Mac OS, Android, and ios. After several beta releases the rst 1.0 version was released in 2006, and the 2.0 was released in october 2009, with a new C++ interface. At the moment of writing OpenCV 2.4 has been released and is available through the developer web site 4. Within the Processing language and environment, a small set of OpenCV functions has been available for years through the OpenCV "hypermedia" library. The available functions were real-time capture, video le import, basic image treatment, face and body identication, and blob detection. A quick introduction to the use of this library can be found in Andy Best's tutorial 5 or in Programming Interactivity by Joshua Noble 6. This library, no longer updated, is now superseeded by the JavacvPro library 7, which has a much larger set of functions. The JavacvPro library makes its operations on a buer, and it uses two auxiliary buers where images can be stored and recalled with remember() and restore() operations. This allows, for example, to compute the dierence between two neighboring frames

7 OpenStax-CNX module: m Manipulation of buers in OpencvPro Figure 1 Example 1: Background subtraction import monclubelec.javacvpro.*; OpenCV opencv; GSCapture cam; //declare an OpenCV object //declare a camera object void setup(){ size(320,240); opencv = new OpenCV(this); //objectcv instance opencv.allocate(320,240); cam = new GSCapture(this,320,240); //prepare for reading a video stream cam.start();

8 OpenStax-CNX module: m void draw(){ if (cam.available()){ cam.read(); //frame grabbing opencv.copy(cam.get()); opencv.absdiff(); //difference of frames: result in Memory2 image(opencv.getmemory2(),0,0); //display the difference image void keypressed() { opencv.remember(); //store the frame in the memory buffer println("remember"); // stop processing public void stop(){ cam.delete(); super.stop(); By addition of opencv.remember() as the last line of the internal block of the draw() function, the dierence is computed between two contiguous frames, thus obtaining a visualization of motion. If, instead of the image(opencv.getmemory2(),0,0), we put the instructions opencv.restore2(); opencv.gray(); opencv.threshold(0.5); image(opencv.getbuffer(),0,0); the dierence is reduced to gray levels and then thresholded at 0.5 level. Example 2: Blob Tracking Within a frame, a blob is a contiguous area represented in a data structure. A blob has a contour that represent a contrasting edge with neighboring regions. Tracking an object means to follow in time its corresponding blob that is moving across the screen. Previous-frame blobs are compared with the current-frame blobs to identify those blobs that are likely to keep representing the same object. The following code has been derived from Programming Interactivity by Joshua Noble 8, adapted to use the JavacvPro library. import monclubelec.javacvpro.*; int IMG_WIDTH = 320; int IMG_HEIGHT = 240; int BLURAMOUNT = 7; float THRESHOLD = 0.8; OpenCV opencv; 8

9 OpenStax-CNX module: m GSCapture cam; void setup(){ size(img_width*2, IMG_HEIGHT*2); cam = new GSCapture(this, IMG_WIDTH, IMG_HEIGHT); opencv = new OpenCV(this); opencv.allocate(img_width, IMG_HEIGHT); cam.start(); void draw(){ if (cam.available() == true) { cam.read(); opencv.copy(cam.get()); opencv.remember(); opencv.blur(bluramount); image(opencv.getbuffer(),img_width,0); opencv.restore(); opencv.invert(); image(opencv.getbuffer(),0,img_height); opencv.restore(); opencv.gray(); opencv.threshold(threshold); image(opencv.getbuffer(),0,0); // Args: minareain, maxareain, maxblobin, findholesin, maxverticesin, debug Blob[] blobs = opencv.blobs(opencv.area()/256, opencv.area()/2, 20, false, 100, false); for (int i=0; i < blobs.length; i++){ nofill(); rect(blobs[i].rectangle.x, blobs[i].rectangle.y, blobs[i].rectangle.width, blobs[i].rectangle.height stroke(0,0,255); line(blobs[i].centroid.x-5, blobs[i].centroid.y, blobs[i].centroid.x+5, blobs[i].centroid.y); line(blobs[i].centroid.x, blobs[i].centroid.y-5, blobs[i].centroid.x, blobs[i].centroid.y+5); fill(255,0,255,64); stroke(0,255,0); if (blobs[i].points.length > 0){ beginshape(); for (int j=0; j < blobs[i].points.length; j++){ vertex(blobs[i].points[j].x, blobs[i].points[j].y); endshape(close); Example 3: Face Detection With OpenCV, it is possible to load the conguration of a classier that has been trained to recognize a specic shape, such as a human face or body.

10 OpenStax-CNX module: m import monclubelec.javacvpro.*; import java.awt.*; // for classes Point, Rectangle,... OpenCV opencv; GSCapture cam; //declare an OpenCV object //declare a camera object void setup(){ size(320,240); opencv = new OpenCV(this); //objectcv instance opencv.allocate(320,240); cam = new GSCapture(this,320,240); //prepare for reading a video stream cam.start(); opencv.cascade("/usr/local/share/opencv/haarcascades/","haarcascade_frontalface_alt.xml"); // load th void draw(){ if (cam.available()){ cam.read(); //frame grabbing opencv.copy(cam.get()); image(opencv.getbuffer(),0,0); Rectangle[] faces = opencv.detect(); //look for faces nofill(); stroke(255,0,0); for (int i=0; i < faces.length; i++) rect(faces[i].x, faces[i].y, faces[i].width, faces[i].height);