Augmented Feedback for Childhood Apraxia of Speech Design Specification Document

Transcription

1 Augmented Feedback for Childhood Apraxia of Speech Design Specification Document Document Version: 1.0 Status: Under revision Department of Computer Science & Engineering York University October 2013

2 Version History Version Name Date Description 1.0 D. Raymond 10/27/13 First release

3 Table of Contents 1. INTRODUCTION Scope and Purpose Project Design Constraints DEVELOPMENT ENVIRONMENT Windows Unity3D Assets Scripts OpenCv cvblob OpenCVSharp DESIGN Visuals Main Screen Configure Screen Play Screen IMPLEMENTATION Overview Details BlobScript SphereTrack Hit FUTURE WORK APPENDIX: SOURCE CODE Main Screen Configure Screen Play Screen... 19

4 1. INTRODUCTION 1.1 Scope and Purpose The purpose of this design specification document is to outline the procedures and implementation practices used when developing the game titled Rock Climber. Rock Climber is the implemented by-product of the document titled Augmented Feedback for Childhood Apraxia of Speech Requirements Analysis Document. Though the model digresses from its initial intended use as presented in the aforementioned document, it serves as a foundation upon which our final product is intended to be developed. Rock Climber is an interactive game designed for children with speech disorders, namely Apraxia of Speech. Though still a prototype, the designed game is outlined to provide jaw exercises for children through a series of incentive-based actions. The input of the game will consist of frame by frame images rendered from the webcam, which is then processed by tracking algorithms. These frames are then mapped to the vertical movements of a game object, emulating the movement of the mandibular along the sagittal plane. This document will outline the user interface design and implementation details of the game. It will list the platform specified controls, features and libraries used to implement the game. Furthermore, it also serves as a reference for future development of the prototype. 1.2 Project Design Constraints Rock Climber provides for an initial prototype upon which our intended model is to be built. Our final envisioned product as discussed in the Requirements Analysis Document is to be developed on a portable device, preferably an ipad or similar tablet. Using a tablet will provide for a more user friendly platform for its patients the children. Because Rock Climber is developed on a desktop, the user will be restricted to the location(s) in which he or she is playing the game. Rocker Climber is presented on a desktop computer, therefore the only form of input comes from the machine s built in webcam. The webcam only has a limited frame per second rate, so details such as preciseness and accuracy of tracking must be overlooked when collecting feedback data. Our current methods of development have led to procedures that do not have the accuracy of the game as a focal point. All in all, Rock Climber is presented as a basic working model for future development.

5 2. DEVELOPMENT ENVIRONMENT 2.1 Windows Rock Climber is targeted for the Windows desktop platform. Users of the game must therefore be present in front of a monitor on which output can be seen. A webcam, either built-in or external, is additionally needed to provide input in real time. These input images will be processed to provide for the main controller of the game. As noted earlier, since the webcam is being used, accuracy is not a main feature that has been taken into consideration when developing this prototype. 2.2 Unity3D Unity is a game engine designed to provide an easy interface for 3D interactive game development. It acts as a cross-platform game engine to develop games for operating systems such as Windows, ios and Android. Unity was chosen as the game engine for this project, due to its easy interface for creating 3D games. Not only that, but its popularity has grown, giving its massive community of developers easy access to tutorials and forums. Unity has a drag-and-drop style interface that allows easy development of games. With a powerful 3D rendering and physics engine, Unity has the necessary features that are needed for our prototype. Before discussing the implementation, it is very important to understand the structure of objects and terminology that Unity uses in development. These concepts will be essential to understanding the overall picture and linkage of all code and classes. We will not give an elaborate overview of every feature used in Unity. Rather, we will give a general introduction to all conceptual features needed in our game, sufficient enough to understand the development of it. More curious readers can seek additional references on Unity s main web page Assets Every object in Unity can be thought of under the general umbrella of objects, namely Assets. An asset, with regards to Unity, can be anything both physically and conceptually. It can represent a tree, a character, a camera and even the lighting features. In the development of our game, every object interacted with (and not) is an asset. This includes the rock face, the terrain, the buttons, the tracking object, a movement and the position of the camera (from which the user is viewing from). Assets represent objects in which a developer or user can interact with. As a user, one can move a character an asset. As a developer, one can assign certain actions to characters both assets.

6 2.1.2 Scripts Scripts are the code that provides functionality to our objects. They represent, in our context, C# or JavaScript classes that when instantiated create some dynamic change on screen. We note however that scripts, in Unity s definition, is not defined in the traditional sense. When a developer thinks of the word script, some key words come to mind JavaScript, Perl or Python. In Unity s definition, any piece of code whether traditional script or defined class, is a script. 2.3 OpenCv Unity was used to create a static version of our game, Rock Climber. To implement the tracking algorithms that were proposed, a library known as OpenCV was used. Open Source Computer Vision Library, abbreviated as OpenCV, is a library of programming functions implemented in C++ aimed towards the field of computer vision. It acts as a library that supports built in calls to real time image processing and manipulation techniques. The majority of the code used to process the frames rendered from the webcam, will be calls to the OpenCV library. One important factor that may cause confusion is the tracking rate within Unity. The OpenCV library functions manipulate images on a frame by frame basis. Therefore, we would need some way to incorporate that within Unity. When implementing code, two proprietary functions must be always introduced: void Start() and void Update(). The Start() method acts as a point of initialization for all code within the module. The Update() function is called every frame. This is where the majority of our OpenCV library functions will be called, since manipulation of every frame, in essence, is a moving picture cvblob The input of our game will come from the rendered frames of the webcam. Additionally, a specific tracking object will need to be used to act as a controller on screen. This tracking object needs to be chosen as an object with a colour not prevalent in the background of the input frame. In other words, the tracking object needs to be chosen so that it stands out on screen. Ideally, the tracking object will be placed on the users chin and will be used to control the object in our game. A separate open source library extending OpenCV s features is used to detect this tracking object. This library is known as cvblob. CvBlob provides for basic blob tracking through image processing requiring the definition of contours and binary labeling OpenCVSharp The main difficulty regarding our project was integrating the OpenCV library with Unity. Because Unity is developed in C#, using the IDE MonoDevelop, a wrapper library was needed to

7 call OpenCV functions. The wrapper library used in our case was OpenCVSharp. OpenCVSharp provides for the majority of OpenCV functions for.net using C# syntax and semantics. An example of the correspondence between OpenCV library function call (1) and OpenCVSharp library call (2) is as follows. IplImage *frame = cvcreateimage(); //1 IplImage frame = Cv.CreateImage(); //2 Note the non-usage of pointers in 2 due to the fact that we are using.net

8 3. DESIGN 3.1 Visuals Below is a description of the games features, discussing its static visual appearance. The implementation details will be left to the remainder of the document Main Screen The opening screen in Rock Climber is the Main Screen. The background of the Main Screen is set to look like a rock face, to stay consistent with the intention of our game. This rock face has three buttons - Play, Configure and Exit. The Play button will lead us to the game screen, referred to as Play Screen and the configure button will lead us to the configuration screen, known as the Configure Screen. The exit button, upon click, will exit the game. Everything else on this screen is static it was created to have a 3D effect with the only intention of a being visually appealing Configure Screen Figure1: The Main Screen The Configure Screen is where the user configures the game to track a specific object. When the user enters this screen they will be prompted by three separate windows. Two of them are visual augmentations of the webcam being captured in real time. The third screen will consist of six sliders.

9 The sliders are used to set the minimum and maximum hue, saturation, and value (HSV) values that denote the threshold that will be used to identify the color of the tracking object. By adjusting these values, through trial and error, one will have a clearly processed color that is alone being tracked on the third window, which is an augmentation of a live webcam capture. The HSV values selected in the slider are filtered from the live webcam capture and converted to a binary image, the second window. The pixels represented by the colour white are the ones that fall within our defined HSV threshold, and the pixels with the colour black are everything outside of this threshold. This binary image frame is then processed using the cvblob library to display the tracking object on the live webcam capture. The tracking object will be outlined in pink on the third screen to indicate that it is being located. An example user configuration would be as follows. 1. The user will hold up an object of color, not prevalent in skin tones (something that stands out) 2. Using the sliders, the user will begin adjusting the HSV values until the binary image on the second window shows the object in a color of white (and everything else as black) 3. When the user has done this to the smallest error, they can verify that the image is being tracked on the third window (it should be outlined in pink) Play Screen Figure 2: The Configure Screen The Play Screen is the game. The tracking object will be mapped to an tracker asset on screen. The user will be prompted to adjust the location of the tracker, and hence the tracking object, to a specified location. When this initial configuration has been done, the user is instructed to pronounce the word on screen. The word is displayed directly under the initial location of the tracker. The objective is to pronounce the word with minimal jaw sliding. The result will be the tracker colliding with the word, and the word exploding (as a prompt showing that the motion was correct). Jaw sliding will lead to the object not correctly colliding with the word, and therefore the user will be prompted to pronounce the word again.

10 Figure 3: The Play Screen

11 4. IMPLEMENTATION 4.1 Overview The implementation details discussed in this section will only relate to the Play Screen, since this is where the majority of the dynamic and interactive code comes into play. As mentioned in the previous section, within the Play Screen will be an object that is tracked according the movement received via webcam input. Below is a table created of the assets and their corresponding scripts being used in the Play Screen: Asset Script Description Tracking BlobScript Used to render frame input and locate tracking object. Tracker SphereTrack, Hit Used to map tracking object to co-ordinates on screen (thereby moving tracker asset). Used to detect collisions on Word asset. Word Used as a target for collision. The Tracking asset is positioned in a location that is theoretically positioned where user is viewing the screen. Tracking is not a physical asset, but rather a conceptual one as discussed earlier. It represents the object that is being controlled by the input and allows for an attachment of a script. The script being attached to Tracking is BlobScript. BlobScript detects the tracking object that is being used to control the game and stores its x and y coordinates respective to the screen. The Tracker asset is the object that is being moved according to the controller. Visually, it is represented by a red sphere. Attached to the Tracker are two scripts SphereTrack and Hit. SphereTrack retrieves the Tracking asset s x and y coordinates and synchronously moves Tracker to the appropriate position on screen. The Hit script is used to instantiate an explosion when the Tracker and another object (the Word asset) collide. The Word asset is randomized and represents a target object that is aimed for. When the Tracker asset collides with the Word asset (thereby calling the script Hit), the Word asset disappears and an explosion appears on-screen.

12 4.2 Details BlobScript The BlobScript script is the most important of our created classes. It has the function of tracking a specified "object" on screen and retrieving its coordinates relative to our screen size. In its core, BlobScript retrieves the majority of its functions from two external libraries - OpenCV and cvblob (both represented by the wrapper class OpenCVSharp). Both libraries allow for image processing techniques and functions within their respective fields. OpenCV provides for the majority of color manipulation and image displaying functions, while cvblob provides for the "blob tracking". We will not explain the entirety of the code, but we will explain the main calls that enable the tracking of the said object. Cv.CvtColor(frame, hsvframe, OpenCvSharp.ColorConversion.BgrToHsv); Cv.InRangeS(hsvframe, new OpenCvSharp.CvScalar(bMin, gmin, rmin), new OpenCvSharp.CvScalar(bMax, gmax, rmax), threshy); Cv.Smooth(threshy, threshy, OpenCvSharp.SmoothType.Median); uint result = CvBlobLib.Label(threshy, labelimg, blobs); CvBlobLib.RenderBlobs(labelImg, blobs, frame, frame); CvBlobLib.FilterByArea(blobs, 60, 500); The first three method calls are from the OpenCV library. They provide standard image processing techniques that are then combined with functions from the cvblob library. Firstly, the input must be retrieved from the webcam on a frame by frame basis. Once the frame is retrieved, the first operation that occurs is converting the BGR or normal color space (of the frame) to an HSV color space. This operation is performed using the function CvtColor() and its purpose is later explained. The new HSV color frame is then filtered according to the HSV threshold values that are specified. This is done with a call to the method InRangeS() and the threshold minimum and maximum values are given as input. The resulting frame is then smoothed to reduce noise with a call to Smooth().The result of these linear operations is a frame that is filtered according to a minimum and maximum HSV value that we provide as input. Theoretically, if a tracking object is used with a color not prevalent in its background, a blob will be displayed in the frame. The next three calls are from the cvblob library. The Label() method is used to retrieve the said blob from the frame. We note that there can be more than one (depending on the threshold) however for our project we limited ourselves to only one the tracking object. The RenderBlobs() method renders the blobs in this frame. Finally, the FilterByArea() method limits the size of the blob according to specified dimensions.

13 4.2.2 SphereTrack The SphereTrack script has a general function of mapping the tracking object to the Tracker asset shown on screen. It maps this process using an instantiated script of type BlobScript (attached to the Tracker asset). This function is called within the Start() function of the script, to denote an instantiation: public GameObject blobobject; public BlobScript blobscript; void Start () { blobscript = blobobject.getcomponent<blobscript>(); After instantiating the BlobScript, it calls the tracked coordinates synchronously and stores the coordinates in a vector object of type Vector: Vector3 pos = new Vector3(); Finally, these coordinates are mapped and scaled to an appropriate location on the screen, thus creating a boundary for where our Tracker asset can move. This happens by storing the coordinates in the Vector object created earlier: float halfw = (float)(screen.width/2f); pos.x = blobscript.xcoordinate - (halfw * 0.55f); temp1 = pos.x; float halfh = (float)(screen.height/2f); pos.y = (halfh * 1.5f) - blobscript.ycoordinate; temp2 = pos.y; transform.position = pos; Hit The script titled Hit has only one function. When the asset tagged Target "collides" with the asset upon which this script is attached, some code is executed. In our case, the Word asset is tagged as Target. if (collision.gameobject.tag=="target") { Instantiate(prefab, transform.position, Quaternion.identity); Destroy(collision.gameObject);

14 Inside the if statement, we have only 2 function calls. The first function instantiates an "explosion" (already initialized) and targets its position to the exact location on which the collision occurred. The second function, Destroy, eliminates the collided object onscreen, thereby creating an explosive effect.

15 5. FUTURE WORK Although Rock Climber does basic functions, it does not encompass the rich functionality that we had defined in the Requirements Analysis Document. The first step towards improvement would be to develop the game on a tablet platform, specifically the ipad. As discussed, a tablet will be more effective with regards to mobility when in use by children. The libraries that we have used to provide tracking are also available for ios which allows for a smooth transition and minimal changes in implementation. Aside from the change in platform for which the application is to be developed on, several changes must be made to the actual algorithms that are used to track and provide control. In this prototype, we use an external object (the tracking object) that is to be placed on the patients jaw in order to provide control. Although this method results in controlling the game, we need a more user-friendly way of providing input. One method to achieve this result, is to specifically track facial feature points, specifically the jaw region. By tracking only the jaw, the configuration steps needed will be reduced and resulting will be an increased adaptability in game play. The Active Shape Model (ASM) and Active Appearance Model (AAM) provide for these enhanced techniques. Both AAM and ASM are computer vision algorithms that provide statistical models of mean shape and variations, specifically targeted to human facial features. In this sense, by tracking points specific to the jaw region, there will no longer be a need for a tracking object.

16 APPENDIX: SOURCE CODE Main Screen MenuObject.cs: Used to detect on-click events using UnityEngine; using System.Collections; public class MenuObject : MonoBehaviour { public bool isquit = false; public bool isconfigure = false; public bool isback = false; void OnMouseEnter() { renderer.material.color = Color.red; void OnMouseExit() { renderer.material.color = Color.white; void OnMouseDown() { if (isquit) Application.Quit(); else if (isconfigure) Application.LoadLevel(2); else if (isback) Application.LoadLevel(0); else Application.LoadLevel(1); Configure Screen opencvtest.cs: Opens 3 windows, 2 of which are augmented webcam captures and the other being a sliders window using UnityEngine; using System.Collections; using OpenCvSharp; using OpenCvSharp.MachineLearning;

17 using OpenCvSharp.Blob; using System.Runtime.InteropServices; public class opencvtest : MonoBehaviour { int H_MIN = 0; int H_MAX = 256; int S_MIN = 0; int S_MAX = 256; int V_MIN = 0; int V_MAX = 256; const int CAPTURE_WIDTH = 640; const int CAPTURE_HEIGHT = 400; CvCapture capture; CvBlobs blobs = new CvBlobs(); private IplImage frame; private IplImage hsvframe; private IplImage labelimg; private IplImage threshy; private IplImage binary; void track_function_h_min(int x) { H_MIN = x; BlobScript.hMin = x; void track_function_h_max(int x) { H_MAX = x; BlobScript.hMax = x; void track_function_s_min(int x) { S_MIN = x; BlobScript.sMin = x; void track_function_s_max(int x) { S_MAX = x; BlobScript.sMax = x; void track_function_v_min(int x) { V_MIN = x; BlobScript.vMin = x;

18 void track_function_v_max(int x) { V_MAX = x; BlobScript.vMax = x; void createtrackbars() { Cv.NamedWindow("Trackbars"); Cv.CreateTrackbar("H_MIN", "Trackbars", H_MIN, H_MAX, track_function_h_min); Cv.CreateTrackbar("H_MAX", "Trackbars", H_MAX, H_MAX, track_function_h_max); Cv.CreateTrackbar("S_MIN", "Trackbars", S_MIN, S_MAX, track_function_s_min); Cv.CreateTrackbar("S_MAX", "Trackbars", S_MAX, S_MAX, track_function_s_max); Cv.CreateTrackbar("V_MIN", "Trackbars", V_MIN, V_MAX, track_function_v_min); Cv.CreateTrackbar("V_MAX", "Trackbars", V_MAX, V_MAX, track_function_v_max); void Start() { Cv.NamedWindow( "Calibrate"); capture = Cv.CreateCameraCapture(0); frame = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,3); //Original Image hsvframe = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,3);//Image in HSV color space labelimg = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),CvBlobLib.DepthLabel,1);//Image Variable for blobs threshy = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,1); //Threshold image of yellow color binary = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,1); createtrackbars(); void Update() { IplImage fram = Cv.QueryFrame( capture ); if( fram == null ) Destroy(); Cv.Resize(fram, frame, OpenCvSharp.Interpolation.Linear); Cv.Flip(fram, frame, OpenCvSharp.FlipMode.Y); Cv.CvtColor(frame, hsvframe, OpenCvSharp.ColorConversion.BgrToHsv); Cv.InRangeS(hsvframe, new OpenCvSharp.CvScalar(H_MIN, S_MIN, V_MIN), new OpenCvSharp.CvScalar(H_MAX, S_MAX, V_MAX), threshy); Cv.Copy(threshy, binary);

19 Cv.Smooth(threshy, threshy, OpenCvSharp.SmoothType.Median); uint result = CvBlobLib.Label(threshy, labelimg, blobs); CvBlobLib.RenderBlobs(labelImg, blobs, frame, frame); CvBlobLib.FilterByArea(blobs, 60, 500); Cv.ShowImage("Binary", binary); Cv.ShowImage( "Calibrate", frame ); char c = (char)cv.waitkey(33); if( c == 27 ) Destroy(); void Destroy() { Cv.ReleaseCapture(capture); Cv.DestroyAllWindows(); Play Screen BlobScript.cs: Used to detect blobs on screen using UnityEngine; using System.Collections; using OpenCvSharp; using OpenCvSharp.MachineLearning; using OpenCvSharp.Blob; using System.Runtime.InteropServices; public class BlobScript : MonoBehaviour { //cvscalar BGR Min values static public int hmin = 140; static public int smin = 25; static public int vmin = 140; //cvscalar BGR Max values static public int hmax = 186; static public int smax = 256; static public int vmax = 256; public int xcoordinate = 0; public int ycoordinate = 0; const int test = 22;

20 const int CAPTURE_WIDTH = 640; const int CAPTURE_HEIGHT = 400; CvCapture capture; CvBlobs blobs = new CvBlobs(); private IplImage frame; private IplImage hsvframe; private IplImage labelimg; private IplImage threshy; int screenx; int screeny; void Start() { capture = Cv.CreateCameraCapture(0); Cv.NamedWindow("Live"); frame = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,3); //Original Image hsvframe = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,3);//Image in HSV color space labelimg = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),CvBlobLib.DepthLabel,1);//Image Variable for blobs threshy = Cv.CreateImage(Cv.Size(CAPTURE_WIDTH,CAPTURE_HEIGHT),OpenCvSharp.BitDepth.U8,1); //Threshold image of yellow color screenx = 1600; screeny = 900; void Update() { IplImage fram = Cv.QueryFrame( capture ); if( fram == null ) Destroy(); Cv.Resize(fram, frame, OpenCvSharp.Interpolation.Linear); Cv.Flip(fram, frame, OpenCvSharp.FlipMode.Y); Cv.CvtColor(frame, hsvframe, OpenCvSharp.ColorConversion.BgrToHsv); Cv.InRangeS(hsvframe, new OpenCvSharp.CvScalar(hMin, smin, vmin), new OpenCvSharp.CvScalar(hMax, smax, vmax), threshy); Cv.Smooth(threshy, threshy, OpenCvSharp.SmoothType.Median); uint result = CvBlobLib.Label(threshy, labelimg, blobs);

21 CvBlobLib.RenderBlobs(labelImg, blobs, frame, frame); CvBlobLib.FilterByArea(blobs, 60, 500); foreach (CvBlob b in blobs.values) { double moment10 = b.m10; double moment01 = b.m01; double area = b.area; int x1; int y1; x1 = (int)(moment10/area); y1 = (int)(moment01/area); xcoordinate = (int)(x1*screenx/capture_width); ycoordinate = (int)(y1*screeny/capture_height); Cv.ShowImage( "Live", frame ); char c = (char)cv.waitkey(33); if( c == 27 ) Destroy(); void Destroy() { Cv.ReleaseCapture(capture); Cv.DestroyAllWindows(); Hit.js: Used to detect collisions var prefab:gameobject; function OnCollisionEnter(collision: Collision) { //If Tracker collides with Target... if (collision.gameobject.tag=="target"){ //Instantiate "Explode" prefab and destroy Target Instantiate(prefab, transform.position, Quaternion.identity); Destroy(collision.gameObject);

22 SphereTrack.cs: Used to map tracking object to screen coordinates using UnityEngine; using System.Collections; public class SphereTrack : MonoBehaviour { public GameObject blobobject; public BlobScript blobscript; // Use this for initialization void Start () { blobscript = blobobject.getcomponent<blobscript>(); // Update is called once per frame void Update () { Vector3 pos = new Vector3(); float temp1, temp2; float halfw = (float)(screen.width/2f); pos.x = blobscript.xcoordinate - (halfw * 0.55f); temp1 = pos.x; float halfh = (float)(screen.height/2f); pos.y = (halfh * 1.5f) - blobscript.ycoordinate; temp2 = pos.y; transform.position = pos;