Technical Reference Document Team Vision 2020 version 1.0

Transcription

1 Technical Reference Document Team Vision 2020 version 1.0

2 Index I. Tools & Definitions 1. CCV 2. FLOSC 3. Unity 4. Flash 5. Smartfox 6. UDPflashlc-bridge II. The Touchscreen 1. Technology III. Software and Setup 1. CCV 2. CCV Setup 3. CCV Calibration 4. TUIO Flash Libraries and Documentation: 5. Unity - Flash Integration 6. Loading Software IV. Appendix 1. Appendix A: Hardware

3 Software Tools & Definitions 1. CCV (Community Core Vision) Community Core Vision, CCV for short (aka tbeta), is a open source/cross-platform solution for computer vision and machine sensing. It takes an video input stream and outputs tracking data (e.g. coordinates and blob size) and events (e.g. finger down, moved and released) that are used in building multi-touch applications. CCV can interface with various web cameras and video devices as well as connect to various TUIO/OSC/XML enabled applications and supports many multi-touch lighting techniques including: FTIR, DI, DSI, and LLP with expansion planned for the future vision applications (custom modules/filters). CCV is built on the open source TUIO Platform and is developed primarily in C++, though some other languages are supported FLOSC (Flash Open Sound Control) FLOSC is a Java server that can communicate with anything that uses the Open Sound Control protocol. This includes software audio synthesizers like Max/MSP, Reaktor, and SuperCollider, and hardware like Lemur and SmartController. It can also communicate with Macromedia Flash (versions 5 and later) using Flash s XMLSocket feature. FLOSC acts as a gateway between OSC and Flash, allowing messages to go in both directions. This enables many combinations of powerful, programmable platforms for high-quality animation, audio, video, and interactive production.

4 flosc can accept simultaneous input from multiple OSC connections and Flash clients, and can broadcast messages to multiple Flash clients, but it does not do any routing. Think of it as a simple gateway or bridge between the generally UDP world of OSC and the TCP world of Flash XMLSockets. IN SHORT: Flosc takes the UDP (User Datagram Protocol) stream from CCV and converts it to an XML data stream. Since Flash cannot use UDP data, the FLOSC server is required for Flash to handle the data Unity Unity is a 3D game development tool. The primary programming language used in Unity is JavaScript. Unity tutorial for Flash Developers (explains Unity in terms of Flash) can be found here: 4. Flash In terms of this project, Flash is our interface handler. It processes the XML data from CCV (via FLOSC) and acts as the control center for tansmitting data to Unity to control the model.

5 5. Smartfox SmartFoxServer is a comprehensive platform for rapidly developing massive multi-user applications and games with Adobe Flash/Flex/Air, Java, Android,.Net, Unity3D, Silverlight, Apple iphone etc... It was developed with simplicity in mind, allowing developers to quickly create any type of multiplayer interactions, from basic chat applications to complex real-time games and MMOs. SmartFoxServer comes with a rich set of features, one of the best sets of documentations with many source examples, powerful administration tools and an active online support forum UDPflashlc-bridge UDOflashlc-bridge is an alternative to FLOSC with just as much support. LikeFLOSC, UDPflashlc-bridge is loaded from the command lineadditinal information and the source code can be found at:

6 The Touchscreen 1. Technology: Primer: There are multiple versions of touchscreen technology that can be used for this project. A complete list which in depth explainations can be found here: For this project, we will be using FTIR (Frusterated Total Internal Reflection). Operation: Infrared light is shined into the side of an acrylic panel (most often by shinning IR LEDs on the sides of the acrylic). The light is trapped inside the acrylic by internal reflection. When a finger touches the acrylic surface this light is frustrated causing the light to scatter downwards where it is picked up by an infrared camera. A silicone rubber layer is often used as a compliant surface to help improve dragging and sensitivity of the device. When touching bare acrylic, one must press hard or have oily fingers in order to set off the FTIR effect. With a complaint surface (like silicone rubber) the sensitivity is greatly improved. Note: For part requirements, please refer to the parts list in the Appendix. Note: For Build instructions, refer to : or multitouch_surface_computer

7 Image 1. FTIR in action.

8 Software and Setup 1. CCV: CCV will be handling the motion monitoring and processing. It has a relatively easy to use interface designed to control blob detection and assignment. CCV (also known as tbeta) is a front end developed by the NUI group. It is really just an implementation of TUIO* protocol. The CCV software will allow us to instantly set up blob detection, adjust the slide controls to refine the detection, and instantly set up a local data stream for the touch surface. It will work with both IR and visible light detection. On the next screen you will find a screen shot of the CCV interface. To use CCV, simply download the software here: nuigroup.com/ Then open the package and run the application. Please refer to the Setup section for detailed setup information. *TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states. This protocol encodes control data from a tracker application (e.g. based on computer vision) and sends it to any client application that is capable of decoding the protocol. There exists a growing number of TUIO enabled tracker applications and TUIO client libraries for various programming environments, as well as applications that support the protocol. This combination of TUIO trackers, protocol and client implementations allows the rapid development of table based tangible multitouch interfaces. TUIO has been mainly designed as an abstraction for interactive surfaces, but also has been used in many other related application areas. Technically TUIO is based on Open Sound Control - an emerging standard for interactive environments not only limited to musical instrument control - and can be therefore easily implemented on any platform that supports OSC. For more information, please refer to:

9

10 2. CCV Setup: The following information can also be found on the NUI Group with_tbeta Note: In order to track fingers or objects, CCV first needs to be configured. The goal is to have a final tracked image that has white blobs coming from fingers or objects placed on the touch surface (with no background noise or false blobs visible or tracked). In order to reach this goal, various image filter sliders are adjusted. By following the steps below, a final tracked image with clear and distinct blobs should be achievable (like the one you see on the picture above). The settings that are used in the picture above are good ones to start with. I advise you to play a little bit with each slider to understand how it works a get a nice tracked image. 1. Download and extract the latest build of tbeta. Open the tbeta folder. 2. Launch the tbeta application (.app,.exe, etc).

11 3. The application will launch and a camera or video image should be displayed under Source Image. 4. If the wrong camera is displayed. Press Next Camera to switch to the next attached camera. Press Previous Camera to return to a previous camera. 5. While no fingers are present, press the background remove button or b to capture the background. If in an environment where lighting changes often, turn on dynamic subtract.

12 6a. (FTIR and LLP users only) * Turn off the highpass and amplify filters. * Turn on the smooth filter. * Adjust the smooth filter to remove noise. * If blobs are weak, turn on the amplify filter to brighten them. 6b. (DI and DSI users only) * Turn off the smooth and amplify filters. * Turn on the highpass filter. * Adjust the highpass blur and noise sliders ntil fingers are clear and distinct. * If blobs are weak, turn on the amplify filter to brighten them.

13 7. Lastly, adjust the threshold slider in the Tracked Image until only fingers are tracked and no false blobs are present. 8. Press Save Settings to save the current settings. note: settings will also be auto-saved on exit. 9. Continue to the Calibration guide below to perform calibration.

14 3. CCV Calibration: In order to calibrate CCV for your camera and projector, you ll need to run the calibration process. Calibrating allows touch points to line up with elements on screen. This way, when touching something displayed on screen, the touch is registered in the correct place. In order to do this, CCV has to translate camera space into screen space; this is done by touching individual calibration points. Following the directions below will help explain how to setup and perform calibration. For those displaying an image on the touch surface (projector or LCD), you ll need to set up your computer so that the main monitor is the video projector so that CCV is displayed on the touch surface. Now that everything is configured nicely and blobs are tracked well in the Tracked Image box, let s calibrate. 1. Press the enter calibration button or c to enter the calibration screen. A grid of green green cross points will be displayed. These crosses are calibration points you touch once you begin calibration. (see screenshot on next page). There is a white bounding box that surrounds the calibration points. If the sides of the bounding box are visible and aligned with the touch surface or a visual image is not being displayed on the touch surface, skip to Step 3. Otherwise, continue.

15 2. If the white bounding box is not fully visible or not aligned with the projected touch surface, follow the directions under Aligning Bounding Box to Projection Screen displayed on the CCV screen to align the bounding box and calibration points so they fit the touch surface. To Align the Bounding Box: * Press and hold w to move the top side, a to move left side, s to move bottom side, or d to move right side. * While holding the above key, use the arrow keys to move the side in the arrowed direction.

16 Basically all you have to do is this : Hold up then left on your keyboard to get the upper corner at the top left corner of your screen, then hold s + down then d + right to get the downer right corner in position. Up, down, right, left will make the box move, and a combination of up, down, right, left and z,q,s,d or equivalent on qwerty keyboard will make the edges move 3. If using a wide angle camera lens or want higher touch accuracy, more calibration points can be added by following the Changing Grid Size directions on screen. note: adding additional calibration points will not affect performance.

17 To Change Grid Size: * Press + to add points or - to remove points along the x-axis. Hold shift with the above to add or remove points along the y-axis. * If this not works, you may want to try _, and +/- from the numerical pad if you hx-axis. 4. Now, you can begin calibration by pressing c. 5. A red circle will highlight over the current calibration touch point. Follow the directions on screen and press each point until all targets are pressed. If not projecting an image on the touch surface, you can guess or draw the touch points directly on the touch surface so you know where to press. If a mistake is made, press r to return to the last touch point. If there are false blobs, press b to recapture the background. 6. After completion the calibration screen will return and accuracy can be tested by pressing on the touch area. If calibration is inaccurate, calibrate again (Step 4) or return to the main configuration window ( x ) to adjust the filters better.

18

19 4. Flash Libraries and Documentation: Since CCV is based on the TUIO framework, all TUIO Flash framework should work with CCV through the FLOSC gateway. Flash must have a Local Network Connection open. 1. TUIO Flash Information: 2. TUIO AS3 Framework files/tuio_as3_v_0_5.zip 3. TUIO AS3 Documentation: tuio-as3-library/ OR Download the complete documentation here:

20 5. Flash with Unity: I do not have a lot of research and documentation done on this area yet. However, there is a lengty video on the Unity site that discusses this topic: integrating-flash-and-unity-content.html

21 6. Loading Software: 1. Add the Flash program directory to your flash global security settings: 2. Launch the FLOSC Gateway script in the terminal. On a Mac, open terminal, navigate to the FLOSC directory. Enter: bash nameofyourfloscscript.sh On a PC, open the executable FLOSC Script This will load the FLOSC gateway. 3. Open CCV (tbeta) and click on send TUIO. 4. Open your smartfox connection 5. Launch your Flash Application

22 Appendix A: Hardware Parts List based on 24 x 24 Surface Area Electronics: 5 Meter 850nM LED strip Playstation Eye Camera VR Glasses 12Volt power supply X2 to power the LED Arrays Cabeling 2X Computers Construction Materials Two 4 x 8 sheets of MDF (Medium Density Fiberboard) 3/8 in thick Acrylic Sheet Shower channel to hold LED Arrays Silicone gel sealant - 2 tubes Vellum or other screen material Small paint roller for rolling on the Silicone Masking tape