Week 11: Motion Tracking

Size: px

Start display at page:

Download "Week 11: Motion Tracking"

Gwenda Cannon
6 years ago
Views:

is another approach for interactive application: when real-live object (e.g.

used to drive the 2D or 3D elements in the virtual scene. TouchDesigner is not mainly designed for motion tracking.

The first step in motion tracking is background subtraction: remove the background pixel from the input web-cam image.

result: white background image pixel is foreground object, transparent pixel is background We can implement this idea using the

1 Week 11: Motion Tracking Motion Tracking is another approach for interactive application: when real-live object (e.g. human) is moving in front of a web-cam, its movement (including its center, size, etc) is tracked, and this information can be used to drive the 2D or 3D elements in the virtual scene. TouchDesigner is not mainly designed for motion tracking. However, we can still use it to perform simple motion tracking tasks. The first step in motion tracking is background subtraction: remove the background pixel from the input web-cam image. A simple method is to subtract the input image from a stored background image: (A) a stored (B) web-cam image abs(a-b) Threshold result: white background image pixel is foreground object, transparent pixel is background We can implement this idea using the Cache TOP and the Difference TOP: Monochrome TOP Usually we only want to work on monochrome image Cache TOP Cache Size: 1 Store a background image Video In TOP Get web-cam image Difference TOP Compute abs(a-b) When the On parameter of the Cache TOP is 1, it will store the input image (and also output it). When it is zero, it will stop getting the input image, but output the stored image. Therefore, we can use a pulse to control this parameter, use either a button or a Keyboard CHOP: 1

Use a pulse to control the On parameter of the Cache TOP.

Usually the subtracted image will be threshold (using a Threshold TOP), so that if the color difference is

e. treated as background ): Threshold TOP Threshold: 0.

will not move, and (iii) the lighting environment is stable.

approach. Whenever the environment changed, you have to snapshot another updated background image.

2 Use a pulse to control the On parameter of the Cache TOP. When there is a pulse, the Cache TOP snapshot an image from the input. Usually the subtracted image will be threshold (using a Threshold TOP), so that if the color difference is larger than a threshold value (say, 0.1), the pixel value will be set to 1 (i.e. treated as foreground ); otherwise, it will be set to zero (i.e. treated as background ): Threshold TOP Threshold: 0.1 This method is simple, and is suitable when: (i) the background contains flat color only; (ii) the web-cam will not move, and (iii) the lighting environment is stable. However, try to move your web-cam a little bit, or turn on one more light, then you will see the problem of this approach. Whenever the environment changed, you have to snapshot another updated background image. The second approach is called frame differencing : to compute the difference between two consecutive frames coming from the web-cam: (A) current frame (B) previous frame (C) final image = (A)-(B) We can also implement this idea using the same network, with different parameter settings for the Cache TOP: 2

Keep getting the web-cam image Store 2 frames but output the previous frame This method can extract the foreground

However, the disadvantage of this method is that only the silhouette of the moving object can be found out (not the

The last approach is quite similar to the previous one, but gives you a little bit better performance.

(A) The result from frame differencing (B) The final result in previous frame, but reduce the brightness by 10%

3 Keep getting the web-cam image Store 2 frames but output the previous frame This method can extract the foreground object image even if your web-cam shifted a little bit, or the lighting condition changed. However, the disadvantage of this method is that only the silhouette of the moving object can be found out (not the interior region of the object); and if the object is not moving, it will be treated as background. The last approach is quite similar to the previous one, but gives you a little bit better performance. It starts from a simple frame differencing method, but the result is merged with a trail of the previous result: (A) The result from frame differencing (B) The final result in previous frame, but reduce the brightness by 10% Final result = the maximum of (A) and (B) The final result in previous frame Reduce brightness by 10% The result from frame differencing Maximum of the 2 inputs The final result 3

Since the above idea involves a loop, we need to implement this idea using the Feedback TOP: Feedback TOP Target TOP: null2 Level TOP Opacity: 0.9 Brightness 2: 0.

some funny image processing effects. One example is to use it for displacing (i.e. distorting) an image.

4 Since the above idea involves a loop, we need to implement this idea using the Feedback TOP: Feedback TOP Target TOP: null2 Level TOP Opacity: 0.9 Brightness 2: 0.9 (under Post tab) The result from frame differencing Composite TOP Operand: Max Class exercise: image distortion by motion tracking The background-subtracted image can already be used for creating some funny image processing effects. One example is to use it for displacing (i.e. distorting) an image. An image can be distorted by the Displace TOP: Displace TOP Red color in the 2 nd input distorts the image horizontally. Blue color distorts it vertically. An input value 0.5 means no distortion. Constant TOP Color: Use the background-subtracted image to modify the Constant TOP in the above network, so that you can use the motion tracking result to distort the image. 4

Finding bounds Assume we have found out a background-subtracted image. How can we make use of it?

3D elements: The bounding box of the foreground The center point of the foreground The area of the foreground Once you

So we will focus on how to find the bounding box of the foreground.

to find this pixel location We can start from a TOP To CHOP, which convert an image into channels of data: We are only

5 Finding bounds Assume we have found out a background-subtracted image. How can we make use of it? A typical approach is to find out the following information, and then use this information to drive your virtual 2D and 3D elements: The bounding box of the foreground The center point of the foreground The area of the foreground Once you find the bounding box of the foreground, the center and the area can be obtained from simple calculation. So we will focus on how to find the bounding box of the foreground. Assume that our web-cam image has size 320x240, and we want to find the leftmost edge of the foreground object: We want to find this pixel location We can start from a TOP To CHOP, which convert an image into channels of data: We are only interested in the Alpha channel, and we want to rename the channel to xmin. This CHOP contains 240 channels: one channel per row. Each channel contains 320 index values (the values are the pixel alpha of the thresh1). 5

Then we connect it to a Math CHOP, to combine all channels together (using the maximum operation).

Each value represents the maximum pixel alpha value of the corresponding column. 320 values Now we have one channel with 320 index values.

One way to do that is to swap the channel and index (using Shuffle CHOP), so that we have 320 channels, and each channel has one single value.

6 Then we connect it to a Math CHOP, to combine all channels together (using the maximum operation). Therefore, we are finding out the maximum pixel alpha value of each column: Math CHOP Combine Channels: Maximum This CHOP contains one channel with 320 values. Each value represents the maximum pixel alpha value of the corresponding column. 320 values Now we have one channel with 320 index values. We need to find the first index that has a non-zero value (the arrow position in the above image). One way to do that is to swap the channel and index (using Shuffle CHOP), so that we have 320 channels, and each channel has one single value. Then we can use the Fan CHOP to find out which is the first channel that has a non-zero value. This will be the pixel location we want to find out: Shuffle CHOP Method: Swap Channels and Samples Fan CHOP Operation: Fan In Lastly, it is even better to normalize the resulting pixel position to the range [0, 1], using another Math CHOP after the Fan CHOP: Of course, we can use 320 here. However, a better approach is to use the expression chopn() to get the number of channels from the Shuffle CHOP. 6

Here is the summary of the CHOP network: How about the other edges?

to find this pixel location A simple idea is: you can flip the image horizontally,

subtracting 320 from the calculated value: We can find this pixel location, and

So the CHOP network is quite similar to the previous one, with some minor differences

7 Here is the summary of the CHOP network: How about the other edges? For example, I want to find out the rightmost edge of the foreground image: We want to find this pixel location A simple idea is: you can flip the image horizontally, apply the same method as mentioned above, and then adjust the final answer by subtracting 320 from the calculated value: We can find this pixel location, and subtract the result by 320. So the CHOP network is quite similar to the previous one, with some minor differences in the parameter settings: Flip TOP (with Flip X) Math CHOP Multiply: -1 Post-Add: chopn( shffle2 ) TOP To CHOP Alpha: xmax Fan CHOP Math CHOP From Range: 0 to chopn( shuffle2 ) 7

8 Class exercise: finding the bounds Continue on the previous example, try to find out: 1. The top and button edges of the bounding box of the foreground (hint: you can use a Flip TOP and set the Flop option to rotate the image 90 degrees); and 2. The center and the area of the foreground. To test your answer, you can import and use the component W:\SM4125\Wk11\draw_bounds.tox. You can plug your original web-cam image, xmin, ymin, xmax and ymax values into the component; and the component will draw out the bounding box. An optional improvement You may notice that when there is no foreground object found, the bounds we created cover the whole image. This will introduce a sudden change in the bounds when the foreground object leaves the web-cam image. To eliminate the sudden change, we can test the area of the foreground : if the area is reasonable, we use the computed bounds; otherwise, we keep the previous bounds without changing its value. We can use an Expression CHOP and several Hold CHOPs to implement this idea. I only show the xmin value here; the other values can be treated in the same way: The computed xmin value The computed area value Out CHOP This is the final output of xmin value. Expression CHOP Expression 1: ic(0,0,0) > 0.2 && ic(0,0,0) < 0.8 If the area is within 0.2 and 0.8, we treat it as a reasonable area. Hold CHOP Sample: While On If the expression CHOP output 1, this CHOP output the input value (i.e. the computed xmin). Otherwise, this CHOP output the previous xmin value. 8

$Class exercise: motion tracking camera You can start from the file W:\SM4125\w11_exercise_tracking_start.toe. In this file, I have created a simple 3D scene, and imported a turn around camera.$ You can ignore A small view to show the web-cam the zoom-in zoom-out operation in this exercise. image Hint: go inside the camera component.

You can ignore A small view to show the web-cam the zoom-in zoom-out operation in this exercise. image Hint: go inside the camera component.

9 Class exercise: motion tracking camera You can start from the file W:\SM4125\w11_exercise_tracking_start.toe. In this file, I have created a simple 3D scene, and imported a turn around camera. Use the motion tracking technique we covered in this class to drive the turn around camera. Use motion tracking to control the turn around camera. You can ignore A small view to show the web-cam the zoom-in zoom-out operation in this exercise. image Hint: go inside the camera component. In the previous class we used the Panel CHOP s channel u, v, and rselect to drive the camera movement. You can replace these channels by the motion tracking results: Hint: go inside the camera component. Instead of using Panel CHOP, try to use the motion tracking results to replace these channels. ** Week 11 END ** 9

Graphical Solutions (How to solve equations graphically; how to find intersection of two lines)

Graphical Solutions (How to solve equations graphically; how to find intersection of two lines) Dr. Gisela Acosta-Carr. (8-page document) Let us review: Solve the equation 2x + 1 = 7 algebraically. First,