Interactive Edutainment App Tutorial

Transcription

1 Version 0.90 Samsung Smart TV 1

2 1. OVERVIEW INTRODUCTION INTERACTIVE APPLICATION INTERACTIVE DEVICES DEVELOPMENT ENVIRONMENT OCI CUSTOM PROFILE DEVICE EMULATOR OVERALL STRUCTURE RUNNING THE CUSTOM APP CREATING THE INTERACTIVE EDUTAINMENT APPLICATION PRIMITIVE MOTION API INTERACTIVE DEVICE EMULATOR USAGE Device Emulator on PC Mobile Device Emulator

3 Preface Purpose of Document. This document is a tutorial that represents an overview of the application/device and a programming guide for interactive edutainment applications. With this tutorial, programmers can understand how to incorporate interactions using the interactive device (Motion Control Input Device) into App. This programming guide for interactive edutainment applications is based on the OCI Custom Profile; therefore it is recommended that you read OCI App API Guide and OCI App Tutorial Custom first. Target Readers This document is aimed at programmers who have used web development languages such as HTML, CSS and JavaScript, and will be even more helpful for those who have web development experience. This document has been written on the assumption that readers have already read the OCI App Tutorial Custom. 3

4 1. Overview This tutorial will demonstrate the development of Apps which are controlled by a motion-detecting interactive device. The software modules that receive the motion sensing data from the device and detect the gestures communicate the recognized gesture events through the OCI Custom profile interface. Therefore it is strongly recommended to read the OCI App Tutorial Custom before developing the interactive applications. 2. Introduction This document describes the interactive application development method for Smart TV. Interactive applications are designed to be played with interactive devices connected to the TV. Users can download the interactive applications through the Samsung TV AppStore in Smart TV. Interactive applications are expected to provide new experiences to TV users. In particular, edutainment applications for children are the intended type of interactive applications. Figure 1. Interactive Edutainment Application on Smart TV 2.1. Interactive Application Interactive applications are supposed to be played with interactive devices connected to TV. Interactive applications can communicate the users interactions or gestures to the contents on the TV screen and give the users feedbacks such as sound and vibration within interactive devices. To implement such applications, an interaction API supporting various gestures is provided. Programmers can develop a variety of interactive applications with the interaction API. The detailed examples for developing interactive applications are shown in sections 3 to 6. 4

5 2.2. Interactive Devices An interactive device is an input device that can control the interactive application running on the TV. Interactive devices detect the users actions and send data to the TV allowing the interactive framework on the TV to recognize the users gestures. Figure 2. Bar Type Interactive Device Figure 3. Using the Interactive Device The Ice Cream is the name of an example interactive device which is designed for this interactive system. The device is a bar, shaped similarly to Maracas, microphone, or an ice-cream cone. It is designed so that users, especially young children, can handle this device with one hand. An interactive device, such as the Ice Cream, has motion sensors including an accelerometer and a gyroscope. The motion sensing data from these sensors are transferred to the TV through a wireless connection. The TV processes this data to the recognized gesture events. For power on, touch sensors in the Ice Cream s handle detect when the user grabs the device. The device also has a couple of feedback modules. The device can play sounds using embedded media stream files, and can also vibrate. An interactive application can trigger the sound or vibration within the device. The interactive device connects to the TV through a wireless communication channel (Wi-Fi). When a user grabs the device, the device detects the user s touch, powers on, and tries to connect to the TV. 5

6 3. Development Environment The SDK is provided for developers to create interactive applications. The SDK is composed of a App emulator and a device emulator. Developers can make interactive applications using the Primitive Motion API based on the interaction data from the device. The App emulator provides an integrated development environment (IDE) OCI Custom Profile OCI, the Open Convergence Interface provides the interface between the interactive device and the Smart TV. Specially, the interactive device requires the OCI Custom Profile which is used for the non-hid device. Please refer to the OCI App Tutorial Custom for the detailed information of OCI Custom Profile Device Emulator An interactive application is supposed to work with interactive devices. Therefore, developers need an interactive device to make interactive applications. In case they don t have the interactive devices, the SDK provides the device emulator, which functions like an interactive device. The device emulator triggers the pre-defined gesture events using a GUI application window. Detailed information about the device emulator is described in chapter Overall Structure Please refer to the Overall Structure chapter of the OCI App Tutorial Custom document. 5. Running the Custom App Please refer to the Running the Custom App chapter of the OCI App tutorial Custom document. 6

7 6. Creating the Interactive Edutainment Application This chapter shows how to create the application based on the interactive device, which can detect the user s motion. As mentioned above, the gesture event data from the interactive device can be received through the OCI custom profile interface. Please refer to the OCI App Tutorial Custom and OCI App API Guide for detailed description. This chapter will focus on receiving the gesture events and making application using those data. This chapter will show the Primitive Motion API, the sample App with its source code and the device emulator Primitive Motion API The Primitive Motion APIs consist of the 17 interactions. The physical interactions are defined for use in the application based on the children s movement. Each interaction can be mapped to the application as an input command. Please refer to the Figure 6 to 8 for examples. Figure 4. Primitive Motion API Directions Figure 5. Primitive Motion API State Figure 6. Example of Primitive Motion in Application (UP for Fishing) 7

8 Figure 7. Example of Primitive Motion API in Application (Twirl for Moving around) Figure 8. Example of Primitive Motion API in Application (Down for Slicing, Left for Drawing) 6.2. Interactive Device Emulator Usage The App emulator includes an interactive device emulator to allow programmers to test interactive applications that they develop with the interaction API. With the interactive device emulator, programmers can test an interactive application using various device interactions even if they do not have actual interactive devices. There are two types of device emulator; the device emulator that runs on PC and runs on mobile device (Galaxy S2) Device Emulator on PC This interactive device emulator includes a GUI button for each direction and state buttons. By using them, programmers can send 8 direction, tilt, blow, push, pull, shake, and bump of interactions to application. Figure 9 shows the GUI for the device emulators. Execute the Device Emulator Program after running the application on SDK App emulator, and then you can control the application by clicking the button on emulator. Also, the motions that are mapped to each input commanded are visualized in the black window within the device emulator. 8

9 Figure 9. Device Emulator (on PC and Mobile) Mobile Device Emulator In case that the developer wants to control the application with motion, the mobile device emulator provides the same functions of the interactive device. To use the mobile device emulator, it is required to install the emulator program (AndroidDeviceEmualtor.apk) on Galaxy S2. After launching the installed emulator as connecting to the same Wireless AP (Access Point) with the SDK, set the host IP address (TV IP address) on the emulator program. Pressing the start button will begin streaming sensor data to the TV. The top of the mobile device should be pointed toward to the TV for the proper gesture recognition. Figure 10. Device Emulator (on PC and Mobile) 9

10 Sample Application Figure 10 is the main UI for the Sample Application. Figure 11. Edutainment Interactive Application Sample After connecting the device emulator, it shows the result through the text and the object movement for each input command. Figure 12. Application Shows the Gesture Recognized Results 10

11 Tutorial_onLoad() function in InteractiveEdutainmentApp.js shows how to initialize the OCI custom profile and call the input handling function. InteractiveEdutainmentApp.js function Tutorial_onLoad() var sadevinfo = null; var ndevnum = 0; var i = 0; gwidgetapi = new Common.API.Widget(); //Create Common module gtvkey = new Common.API.TVKeyValue(); gwidgetapi.sendreadyevent(); //Send ready message to Application Manager gsdevuid = ""; gsdevname = ""; if (OCI_initialize()!= OCI_NO_ERR) start(); return; ndevnum = OCI_getNumberOfConnectedDevices(OCI_PROFILE_CUSTOM, OCI_DEVICE_BENELUDI); alert("[custom Tutorial]: ndevnum(oci_device_joystick) = " + ndevnum); if (ndevnum > 0) for (i = 1; i <= ndevnum; i++) sadevinfo = OCI_getConnectedDeviceInfo(i, OCI_PROFILE_CUSTOM, OCI_DEVICE_BENELUDI); if (sadevinfo!= null) gsdevuid = sadevinfo[0]; gsdevname = sadevinfo[1]; alert("[custom Tutorial]: gsdevuid = " + gsdevuid); alert("[custom Tutorial]: gsdevname = " + gsdevname); if (gsdevuid!= "") if( OCI_checkDeviceConnection(gsDevUID) == OCI_OK ) gncustomdeviceid = OCI_createCustomDevice( gsdevuid ); if( gncustomdeviceid < 0 ) alert("[custom Tutorial]: OCI_createCustomProfile() fail! :" + gsdevuid); document.getelementbyid('txtcs').innerhtml = " Connected Custom: None"; gncustomdeviceid = -1; else alert("[custom Tutorial]: OCI_createCustomProfile() success! :" + gsdevuid); document.getelementbyid('txtcs').innerhtml = "Connected Custom: " + gsdevname; handlecustominputevent(); 11

12 function handlecustominputevent() if( gncustomdeviceid >= 0 ) var inputdata = OCI_CUST_receiveCPMessage( gncustomdeviceid ); if( inputdata == null) settimeout( "handlecustominputevent()", 100 ); /* 0.1 seconds */ return; var msgcmd = Number(inputData[0]); var msgmessage = inputdata[1]; var custommessage = msgmessage.split(","); var msgtime = Number(customMessage[0]); var msgcode = Number(customMessage[1]); var msgdata = String(customMessage[2]); var event_type = Number(customMessage[3]); var device_type = Number(customMessage[4]); var axis_x = Number(customMessage[5]); var axis_y = Number(customMessage[6]); var axis_z = Number(customMessage[7]); var magnitude = Number(customMessage[8]); var duration = Number(customMessage[9]); var frequency = Number(customMessage[10]); var cur_x = Number(customMessage[11]); var cur_y = Number(customMessage[12]); showbeneludievent(event_type, device_type, axis_x, axis_y, axis_z, magnitude, duration, frequency, cur_x, cur_y); settimeout( "handlecustominputevent()", 100 ); /* 0.1 seconds */ else document.getelementbyid('txtc').innerhtml = ""; return; In handlecustominputevent(), the messages are parsed to clarify their meaning. The event_type variable contains the gesture detection result. The following function handles each event_type input to show in App. function showbeneludievent(event_type, device_type, axis_x, axis_y, axis_z, magnitude, duration, frequency, cur_x, cur_y) var eventtypestr = ""; var devicetypestr = ""; switch(event_type) case Enums.CLInteractionEvent.IE_B_UP: eventtypestr = "IE_B_UP"; directup(); 12

13 case Enums.CLInteractionEvent.IE_B_DOWN: eventtypestr = "IE_B_DOWN"; directdown(); case Enums.CLInteractionEvent.IE_B_PULL: eventtypestr = "IE_B_PULL"; directpull(); case Enums.CLInteractionEvent.IE_B_PUSH: eventtypestr = "IE_B_PUSH"; directpush(); case Enums.CLInteractionEvent.IE_B_SHAKE: eventtypestr = "IE_B_SHAKE"; stateshake(); case Enums.CLInteractionEvent.IE_B_BUMP: eventtypestr = "IE_B_BUMP"; statebump(); case Enums.CLInteractionEvent.IE_B_TWIRL_CLOCKWISE: eventtypestr = "IE_B_TWIRL_CLOCKWISE"; statetwirlc(); case Enums.CLInteractionEvent.IE_B_TWIRL_COUNTERCLOCKWISE: eventtypestr = "IE_B_TWIRL_COUNTERCLOCKWISE"; statetwirlcc(); default: return; switch(device_type) case Enums.DeviceType.ANGELUS_DEVICE_BARS: devicetypestr = "ANGELUS_DEVICE_BARS"; case Enums.DeviceType.ANGELUS_DEVICE_DRUM: devicetypestr = "ANGELUS_DEVICE_DRUM"; case Enums.DeviceType.ANGELUS_DEVICE_MOBILE1: devicetypestr = "ANGELUS_DEVICE_MOBILE1"; default: devicetypestr = "No Device Type"; 13