GuZhenDio: Using a String Instrument Interface to Control Live-Action Sports Video Streams

Transcription

1 GuZhenDio: Using a String Instrument Interface to Control Live-Action Sports Video Streams ABSTRACT Calvin Ching Department of Electrical and Computer Engineering University of British Columbia 2329 West Mall Vancouver, BC V6T 1Z4 calvin.k.w.ching@gmail.com This paper introduced the GuZhenDio, a prototype built to explore the effectiveness of using a string instrument interface to control live-action sports video streams. We explored whether the timing, precision, coordination, and high proprioception skills humans can develop on a string instrument interface could be transferrable from making music to controlling another form of spatio-temporal data. User studies were performed and the initial findings showed the GuZhen- Dio as a promising interface. CCS Concepts Information systems Information systems applications; General and reference Experimentation; Keywords spatio-temporal data; bimanual asymmetry; string instrument interface; 1. INTRODUCTION Live-action sporting events are often multi-billion dollar businesses for broadcasting companies. As a result, broadcasting companies have embraced the latest technologies and invested in different resources to gain a competitive advantage. As an example, Turner Network Television partnered with Intel to provide a 360 degree experience at the 2016 NBA All-Star game [8]. These live-action camera angles and technologies provide viewers with immersive experience that take them right at court or field level. With so many different cameras and functions such as zoom in/out, an entire television and production crew is needed to control these components. Live-action sporting events pose additional challenges since the action is often fast paced and unpredictable due to the nature of sports. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists requires specific permission and/or a fee. HIT2016, Vancouver, BC, Canada. c UBC Yingying Wang Department of Electrical and Computer Engineering University of British Columbia 2329 West Mall Vancouver, BC V6T 1Z4 dora93wang@gmail.com Controlling so many cameras and features in the context of sports becomes a spatio-temporal challenge. That is, it takes coordination between camera systems, precision and speed in selecting the correct camera to broadcast, and high level of spatial selection and temporal coordination to effectively control the many variable components. As such, these requirements also tend to describe string musicians who need to precisely select and pluck specific strings at a specific time. If musicians are capable of developing the coordination, quickness, timing, precision, and high proprioception that are needed to perform musical scores on a string interface, like a guitar or a guzhen, then perhaps these skills can be translated to use the string interface to control live-action camera sports video streams since both tasks involve mapping strategies for control of spatio-temporal data. This paper explores this idea to determine the effectiveness of using a string instrument interface to control live-action camera sports video streams. In the following sections, the paper will discuss some relevant work done by others that is related to our research context. Then it will present a description of the implementation of our research prototype, named GuZhenDio. The paper will then detail the user testing strategy to evaluate our prototype and the results. Lastly, it will summarize our research with a conclusion and relevant future work. 2. BACKGROUND AND RELATED WORK A number of research and related work have been proposed over the years for sports broadcasting. The Focus + Context interface scheme proposed by Sarkar et al. [10] displays the focus within the context in a single continuous view like the fisheye lens, which eliminates the spatial and temporal separation. And Foote et al. [1] conducted a touch screen interface based on Focus + Context scheme and the interface was able to produce live multi-camera sports broadcasts. Kuzuoka et al.[3] proposed a touch interface for remote camera controlling. When the finger of a user touches the touch display, corresponding object would come to the center of the view and the direction and velocity could also be determined with fingers. In 2002, Liu et al. [5] proposed a general real-time camera control interface which combined a pan-tilt-zoom camera with a context camera. Users could draw a box on the area they would like to see in the context camera view and the pan-tilt-zoom camera would automatically zoom in that area. The drawback for this is that the user needs to continuously draw and switch between the two

2 Table 1: String Sensing Interface Serial Output Number Position Value Range Details 1-1, 0, 1 Output from Potentiometer 1 2-1, 0, 1 Output from Potentiometer 2 3-1, 0, 1 Output from Potentiometer 3 4 0, 1 Output from Capacitive Sensor 1 0: user is not touching string 1 1: the last pressed string is string 1 5 0, 1 Output from Capacitive Sensor 2 0: user is not touching string 2 1: the last pressed string is string 2 6 0, 1 Output from Capacitive Sensor 3 0: user is not touching string 3 1: the last pressed string is string 3 cameras. There has also been work done to explore giving the power of broadcast control to viewers. Matthews [6] conducted a system letting the viewer to choose camera views which eliminate the role of director in sports broadcasting. And the Free View Television (FTV) proposed by Masayuki [11] generated a new image unifying the visual information with multiple images caught by spatially distributed cameras for better human understanding. The drawback for FTV is that the overhead of establishing the camera networks leads to a heavy budget. The usage of a string instrument interface is also based on research done by Yves Guiard. In Guiard s theoretical framework for the study of asymmetry in the context of human bimanual action, it is suggested that the human s two hands are used differently [2]. A human s non-preferred hand tends to work on a coarse level, providing the frame of reference and context for the preferred hand. The preferred hand typically controls the finer details of a particular task and follows in the frame of reference that was set up by the non-preferred hand. Playing a string instrument is an example of the asymmetric tasks assigned to each hand. For example for a guitar, the non-preferred hand leads, setting the frame of reference by pressing the correct strings on the correct frets. The preferred hand works within the frame of reference and provides the finer details in guitar playing by plucking the correct strings with a particular force. Taylor et. al introduced a new interface to apply a mapping strategy for controlling a different type of spatio-temporal data. Instead of sports broadcasting, Taylor created VPlay, a multi-touch tabletop application that allows users to support and augment the practice of video jockeying [12]. The goal was to explore whether a more interactive interface with direct manipulation, bimanual controls, and increased physical access can be a more effective means for video jockeying. Their research concluded that VPlay showed promise as an effective interface. 3. IMPLEMENTATION The following section details the implementation of our research prototype. The goal was to create a simplified version of a string-like instrument that acts as an interface to a video controlling software. The prototype in whole is shown in Figure 1. Figure 1: GuZhenDio Prototype The implementation of the prototype was divided into two main components, the string-like instrument interface and the video controlling software. 3.1 String-like Instrument Interface The main tasks of the string-like instrument interface are to be able to perform finger sensing and output the sensed data to the video controlling software. The hardware platform consists of three strings that need to detect when a user plucks a particular string with his or her dominant hand and

3 when a user presses onto a particular string with his or her non-dominant hand. Two types of sensors were used to provide the finger sensing requirements. Detecting finger plucking is done using the capacitive sensing capabilities of the Arduino Duemilanove development board. Three metal guitar strings are connected to three digital I/O pins of the Arduino board. Using the Capacitive Sensing Library provided by Arduino Playground, it is able to sense the electrical capacitance of human body [9]. Contact with a metal string results in a different capacitance value sensed on the corresponding digital I/O pin. Three linear membrane potentiometers (one for each string) were attached on the fingerboard. When there is contact between a string and a linear potentiometer, the resistance within the potentiometer changes depending on where the contact is made. The three potentiometers are connected to the analog pins on the Arduino. Each pin contains an analog to digital converter which can then be processed by the ATMEGA328P microcontroller on the Arduino. The video controlling software is coded using Quartz Composer, which is a OS X software by Apple for processing and rendering graphical data using a graphical programming language [4]. Quartz Composer is able to interface to Arduino serially by using the Kineme Serial I/O plugin [7]. The plugin when pointed to the correct serial device at the correct serial I/O configurations (e.g. baud rate, data bits, etc.), is able to receive the serial data from an Arduino and use it as a String within the Quartz Composer program. Figure 3 provides a logic overview of the video controlling software design. Figure 3: Logic Overview of the Video Controlling Software Interface Figure 2: Picture of the String Interface The raw sensor data received on the ATMEGA328P are further processed to be output via the serial interface on the Arduino board. The data are represented as six numbers separated by a space and ends in a \n as shown below: The final graphical user interface (GUI) shows the three videos on the left side of the program while the main output video is shown on a larger rendering surface on the right side. Figure 4 shows a screenshot of the GUI. # # # # # # Table1 describes what each of the numbers represent and the possible values of each. All the components are mounted onto a wooden board for ease of transport. Figure 2 shows a picture of the hardware interface. 3.2 Video Controlling Software The video controlling software provides a graphical representation of the video streams. It takes the serial output from the Arduino board and uses that as input to control the three video streams and the final output video. The mapping strategies to control the videos are summarized in Table 2. The mappings are chosen to take advantage of Guiard?s work on the asymmetry in the context of human bimanual action as described in Section 2.0. Table 2: Video Control Mapping Control Control Level Interface Mapping Camera Selection Fine String Plucking Camera Zoom Coarse String Pressing Figure 4: Video Controlling Software GUI 4. TESTING AND RESULTS The following section details the testing we performed using the GuZhenDio prototype. Afterwards, the results are presented and discussed within the context of the research scope and topic. 4.1 Test Setup The GuZhenDio was tested by seven users in our user testing. The seven users have a wide range of experience

4 Table 3: User Response Summary Question Item Average Response/Score (1 = strongly agree; 7 = strongly disagree) Easy to follow instructions 1.71 In general, task 1 is easy 1.57 In general, task 2 is easy 1.71 In general, task 3 is easy 3.43 Task 3 is easier to do with non-dominant hand 4.29 The second time doing a task is easier than the first 1.43 The more experience using the interface, the less I have to look at the interface to perform a task 1.57 GuZhenDio is easy to use 2.14 GuZhenDio is fun to use 1.86 (from a lot of experience to no experience) with different string instruments, including guitar, violin, and cello. The breakdown of the user experience is shown in Figure 5. Each user was given a brief one to two minute introduction to give each a sense of what the research topic is about. Instructions on how the GuZhenDio string interface controls the video controlling software were also explained to each user. The test consisted of asking users to control three video streams of pre-recorded videos of two people playing a table tennis match. Three cameras were used to record the table tennis match: one camera behind player A, one camera behind player B, and one camera from the side capturing both players. Each user was given three tasks to complete twice. The tasks get progressively more complicated and so the user can get to learn how the GuZhenDio works. The first task consists of walking through the user with selecting different cameras and zooming in/out of the videos. The content of the videos are ignored for the first task. For the second task, the user is asked to select and zoom in/out the camera closest to the player performing the serve. And as soon as the player has served, the user needs to select the side camera video while the rally is ongoing. The user performs this task for at least three rallies. The third task requires the user to choose the camera that is closest to the player hitting the table tennis ball. Ultimately, the user needs to quickly select different camera views in a fast pace manner. Figure 5: User Testing User Experience Breakdown 4.2 Results The users were given a questionnaire to fill in immediately after they finish with the user study test to get an idea of their experience using the GuZhenDio. The results are summarized in Table 3. As summarized in Table 3, Aall users find the first task to be simple to do, rating the easiness of the task with an average of 1.57 on a scale of 1 to 7 (with 1 being easy and 7 being difficult). The second task was more difficult, with the users rating the task to be 1.71 in the difficulty scale. Users find the third task to be most difficult, giving it an average rating of These difficulty ratings were expected as each task got progressively harder. The second and the third tasks required users to use the string interface while concentrating on the content of the videos. Furthermore, the third task required users to exhibit timing, precision, and coordination in order to follow the table tennis ball during a rally. The questionnaire was also designed to attempt to answer different parts of our research question. Based on the user?s responses, the mapping strategy (Table 2) is effective as their responses tend to suggest that camera selection is best performed using their dominant hand versus their non-dominant hand. The users also agree that the string interface can be easily controlled without having to look at the actual interface. This shows that the users are able to control the interface with high proprioception, a skill that is highly helpful in controlling spatio-temporal data. Lastly, all users strongly agree that multiple repetition of using the interface improves their ability to effectively use and control the GuZhenDio. 5. CONCLUSION AND FUTURE WORK In this study, GuZhenDio is a string instrument interface created to explore whether it can be an effective means to control live-action camera sports video streams. Using sensors, the interface is able to detect user plucking and pressing pressing three strings. These actions are mapped to two different camera controlling functions: camera selection and camera zoom. Due to the scope and limited resource for this research, future work can be done to expand on these mappings to other string affordances. For example, the pitch can be mapped to camera panning and string vibration levels can be mapped to camera tilt. With pan and tilt in addition to camera selection and zoom, the string-like instrument interface can have more refined control of the video streams. Based on our user research, we find that using a string instrument interface to control live-action camera sports video streams shows promise. The seven users were put through different tasks and given a questionnaire in order to help us validate different parts of our research topic. The responses showed that the GuZhenDio prototype can be easily learned

5 to adopt the necessary timing, precision, coordination, and high proprioception to control spatio-temporal data such as live-action video streams. The responses regarding the mapping strategy confirms the works of Guiard relating to the asymmetry in the context of human bimanual action. [12] S. Taylor, S. Izadi, D. Kirk, R. Harper, and A. Garcia-Mendoza. Turning the tables: an interactive surface for vjing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages ACM, ACKNOWLEDGMENTS We sincerely appreciate the great help Sidney and Victor offered during the project. Plenty of valuable suggestions enabled us to decide the project and further discussion and timely reply helped us get rid of obstacles and move forward to realize our idea. Also, thanks for taking us to the workshop, looking for sensors and offering various abundant resources. Thank you very much. 7. REFERENCES [1] E. Foote, P. Carr, P. Lucey, Y. Sheikh, and I. Matthews. One-man-band: A touch screen interface for producing live multi-camera sports broadcasts. In Proceedings of the 21st ACM international conference on Multimedia, pages ACM, [2] Y. Guiard. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior, 19(4): , [3] H. Kuzuoka, T. Kosuge, and M. Tanaka. Gesturecam: A video communication system for sympathetic remote collaboration. In Proceedings of the 1994 ACM conference on Computer supported cooperative work, pages ACM, [4] M. D. Library. Introduction to quartz composer user guide. documentation/graphicsimaging/conceptual/ QuartzComposerUserGuide/qc intro/qc intro.html, [5] Q. Liu, D. Kimber, J. Foote, L. Wilcox, and J. Boreczky. Flyspec: A multi-user video camera system with hybrid human and automatic control. In Proceedings of the tenth ACM international conference on Multimedia, pages ACM, [6] J. H. Matthews III. Interactive television system and method for viewer control of multiple camera viewpoints in broadcast programming, Feb US Patent 5,600,368. [7] K. I. Media. Serial io. QuartzComposerPatches/SerialIO/category, [8] I. Newsroom. Intel redefines the fan experience for nba all-star weekend 2016 [press release]. https: //newsroom.intel.com/news-releases/intel-redefinesthe-fan-experience-for-nba-all-star-weekend-2016/, [9] A. Playground. Capacitive sensing library. CapacitiveSensor?from=Main.CapSense, [10] M. Sarkar and M. H. Brown. Graphical fisheye views of graphs. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages ACM, [11] M. Tanimoto. Free viewpoint television-ftv. IEICE Communications Society GLOBAL NEWSLETTER Vol. 2, page 2, 2007.