MOBILE VIDEO COMMUNICATION ON ANDROID FOR DEAF USERS. Michael B Motlhabi, Mr. William D Tucker, Dr. Baccalaureous Scientiae (Honours)

Transcription

1 MOBILE VIDEO COMMUNICATION ON ANDROID FOR DEAF USERS By Michael B Motlhabi, Mr. William D Tucker, Dr. A thesis submitted in partial fulfillment of the requirements for the degree of Baccalaureous Scientiae (Honours) University of the Western Cape Date: November 2, 2011

2 3

3 DECLARATION OF AUTHORSHIP I, Michael B Motlhabi, certify that the work presented here is, to the best of my knowledge and belief, original and the result of my own investigations, except as acknowledged, and has not been submitted, either in part or whole, for a degree at this or any other University.

4 3

5 University of the Western Cape Abstract MOBILE VIDEO COMMUNICATION ON ANDROID FOR DEAF USERS By Michael B Motlhabi Supervisory Committee: Supervisor: W.D Tucker Department of Computer Science The main aim of this project is to investigate and identify the best way Deaf users can use their Android devices to communicate cheaply with video. Deaf people have access to the same mobile phones that the rest of the general public are using, but Deaf users are only limited to text messaging. This forces them to communicate in written language, which in most cases is not their preferred mode of communication. Deaf users in South Africa (S.A) really want to use their phones to communicate with each other in South African Sign Language (SASL), or any sign language using video, and that's what this project is all about. The use of Android phones plays a critical role in making this project a reality. There are many other communication tools that have been used by Deaf people (most of which are PC-based) and very few are designed to work on mobile phones. Those that are mobile-based are very expensive and work only on one platform like MobileASL which works only on Windows Mobile OS. Some of these communication tools were not designed with the needs of Deaf users in mind and those that are, are extremely expensive. With today s technology it is therefore fitting to make a tool suited solely for the communication needs of Deaf users. 4

6 5

7 TABLE OF CONTENTS Declaration of Authorship... 2 Table of Contents... 6 List of tables... 3 Acknowledgments... 5 Glossary... 3 Chapter introduction... 5 Deaf with a capital D... 5 Introduction... 6 Motivation... 6 Chapter user requirements document... 9 Introduction... 9 Users view of the problem... 9 Description of the problem What the software solution is capable of doing What the software solution is not capable of doing Chapter requirements analysis document Introduction Designers interpretation of the problem Breakdown of problem into parts Existing solutions Chapter user interface specification Introduction Description of the complete user interface What the user interface looks like to the user How the user interface behaves How the user interacts with the system Summary Chapter high level design (Object oriented analysis) Introduction User interaction design (use case diagram) class diagram showing the name, attributes, and methods of each class

8 Summary Chapter low level design (Object oriented design) Introduction The activity lifecycle Implementation of the lifecycle (pseudo-code) Chapter implementation Introduction Software s deployed Hardware s deployed Challenges Code documentation deafplayer.java class mediarecorderrecipe.java class newmail.java class player.java class Summary Chapter testing Introduction Testing plan and Criteria People and User Interface Device People and Cost Video quality results (MobileASL) Video quality testing (lab) Summary (Analysis) Chapter user guide Introduction Application Installation on Android Using the application (Getting started) APPENDICES APENDIX A First informal Interview APENDIX B The deafplayer.java class APENDIX C The MediaRecorderRecipe.java class APENDIX D The newmail.java class APENDIX E The player.java class

9 APENDIX F Project Plan and Timeline Usability Questionnaire Reliability Questionnaire Install and Uninstall Questionnaire bibliography

10 List of figures Number FIGURE 1: THIS FIGURE SHOWS A MOBILEASL VIDEO CONVERSATION... 5 FIGURE 2:THIS FIGURE SHOWS A MOBILEASL VIDEO CONVERSATION FIGURE 3: THE VIDEO IS RETRIEVED BY THE OTHER USER IT MUST FIRST BE STORED IN THE MMSC FIGURE 4: THIS FIGURE SHOWS HOW THE USER REQUIREMENTS ARE ORGANIZED FIGURE 5 : ALLAN EC MULTIFUNCTION-TERMINAL FOR SIGN LANGUAGE, TEXT AND VOICE FIGURE 6:MOBILEASL SIGN LANGUAGE VIDEO OVER MOBILE PHONES FIGURE 7:THE GENERIC HOME SCREEN FOR ANDROID FIGURE 8: THE MAIN MENU SCREEN FOR ANDROID SHOWING THE APPLICATION ICON FIGURE 9: THE MAIN INTERFACE OF THE SOLUTION FIGURE 10: THE PICTURE SHOWS THE RECORDING OF A VIDEO FIGURE 11: USE CASE DIAGRAM OF ONE USER INTERACTING WITH ANOTHER FIGURE 12: USE CASE DIAGRAM OF THE SOLUTION FIGURE 13: SHOWS THE CLASSES THAT HAVE BEEN IMPLEMENTED FIGURE 14: SHOWING THE STATE DIAGRAM OF MEDIARECORDER FIGURE 15: THE GRAPH SHOWING THE DEGREE OF COMPLEXITY USERS ENCOUNTERED FIGURE 16: SHOWING THE RECOVERABILITY OF THE APPLICATION AFTER RECORD FIGURE 17: GRAPH OF APPLICATION PERFORMANCE USING TIME FIGURE 18: ON THE LEFT SCREEN SHOT AT 1 FPS AND ON THE RIGHT SCREEN SHOT AT 10 FPS FIGURE 19: THE INSTALL INTERFACE FIGURE 20: ANDROID HOME MENU SCREEN FIGURE 21: APPLICATION HOME MENU SCREEN FIGURE 22: SHOWING VIDEOS ON DEVICE MEMORY FIGURE 23: SHOWING THE RECORDING OF A NEW VIDEO WHEN THE RECORD BUTTON IS PRESSED FIGURE 24: SHOWING THE CONTACT LIST AFTER THE SEND BUTTON WAS PRESSED FIGURE 25: TIMELINE (GANT CHART) Page 9

11

12 LIST OF TABLES Number TABLE 1: THE ACTIVITY TABLE TABLE 2: SHOWING WHICH DEVICES THE APPLICATION FUNCTIONS ON WITHOUT ERRORS TABLE 3: NETWORK PROVIDES COST FOR MMS TABLE 4: SUBSCRIPTIONS OFFERED TO CUSTOMERS TABLE 5: THIS TABLE SHOWS HOW WE PLAN TO CARRY OUT THE TESTING PHASE OF THE APPLICATION TABLE 6: SHOWS THE THEORETICAL VALUES ACHIEVED BY RELATED APPLICATIONS TABLE 7: SHOWS HOW THE VIDEO PROPERTIES CHANGE AS THE ENCODER IS CHANGED TABLE 8 SHOWS CONSTANTS DURING LAB TESTING Page 3

13 4

14 ACKNOWLEDGMENTS I thank my classmates for their motivation and staying behind with me on all the late nights in the cold Honours lab. Thanking my friends Ntsako, Maggie, Diego, Ibraheem, Dane and Warren for not shutting their ears when I was complaining and giving me their shoulders to cry on. Much appreciation goes to a special person in my life Athandile for always believing in me when I didn t believe in myself and helping me realize my place in Computer Science. I would also like to thank my supervisor Dr. W.D Tucker for his unwavering support during hard times (present & future) and always making me sound smart in our weekly meetings. Lastly and most importantly I thank my mother Gertrude Motlhabi, father Apollo Motlhabi and my sister Lerato Motlhabi who have been the vessel and source of my strength since the day I was born and who continue to inspire to be great. 5

15

16 GLOSSARY 3G API BANG Bastion Codec CR CT DCCT DeafSA fps GPRS HD IM JPEG MMS MMSC MobileASL NGO RoI SASL SIMBA Third Generation (of cellular data networks) Application Programming Interface Bridging Applications and Networks Group The building used by the DCCT NGO in Cape Town Compression/Decompression or Encoder/Decoder Compression Ratio Compression Time Deaf Community of Cape Town, a local NGO Deaf Federation of South Africa, a national NGO frame per second General Packet Radio Service High Definition Instant Messaging Joint Photographic Experts Group Multimedia Messaging Service Multimedia Messaging Service Center Mobile American Sign Language Non-Government Organization Region of Interest South African Sign Language Softbridge for Instant Messaging Bridging Application 3

17 4

18 C h a p t e r 1 INTRODUCTION Figure 1: This figure shows a MobileASL video conversation: the main aim of this project is to develop a system where assynchronous video communication between two sign language users can be made possible and affordable. (mobileasl.cs.washington.edu) Deaf with a capital D In deaf culture there is a principal difference between big D for Deaf and small d for deaf. Deaf with a capital D denotes people who use sign language as a preferred language. This means the distinction between "Deaf" and "deaf" is based on an individual s preferred language (spoken or sign) rather than on the actual degree of hearing loss [1]. 5

19 Introduction The need for Deaf users to communicate is very important because they have a limited way to do so. Most Deaf people are not literate due to late introduction to formal education and a background of poverty. This means few can read and write well, either English or Afrikaans [4]. In a nutshell this project seeks to introduce a mobile communication tool that can be used by Deaf users to do asynchronous video with full screen or with split screen on Android. The fact that Deaf people cannot use spoken language makes things tricky for them when they have to use mainstream communication devices like cell phones. Deaf people possess an equal right to communicate in a way that makes them comfortable and understandable. When Deaf people buy a handset, they have to use the applications they find on the phone or download some other random application not targeted at fulfilling their needs to communicate, like the default video recorder which wastes resources by including sound (which Deaf people don't need) or text messaging which they might not prefer. This presents problems for Deaf users because they have to pay for most of these functionalities they never get to use. However with the introduction of smart cell phones, more importantly Android phones, efforts can be made to correct this technological injustice. Motivation Today most mobile phones (smart or not) have a very good back-facing camera. And all Android phones have a camera, but these are generally for taking pictures and recording fun home videos. Some also have a low resolution front-facing camera used for video conferencing. The plan is to extend the basic use of these cameras to offer a better use for Deaf users at a cheaper cost with no compromise user experience. 6

20 Deaf users already use cell phones for sign language communication in countries like America, Japan and Sweden [5]. It s about time we create a solution for Deaf users in S.A who wish to follow that line of communication. They will start-up their Android device and record a video, send this video to a friend who will later use their own Android mobile device with software installed to watch the video and maybe respond later. Communication will not be synchronous but asynchronous. The advantage of this is better video quality. We will explore this later. 7

21 8

22 C h a p t e r 2 USER REQUIREMENTS DOCUMENT Introduction This section describes the problem from the user s point of view. Here we consider take the user s needs and specifications regarding the product and the problems that the users are facing. We interviewed a user/developer and extracted from her what she and Deaf users need from a product of this nature. Given the users experience in using this sort of technology I expect to receive clear and precise feedback onto which I will start producing a usable solution for the problem at hand. Users view of the problem The need for a better mobile video communication tool in the Deaf community has always been eminent, and over the years Deaf users in Deaf Community in Cape Town (DCCT) have had different models to use and test. And from this experience we plan to draw information from both the users and the previous developers of video communication tools. Deaf users simply want to have a communication tool that is easy to use and does not cost them a lot of money. Deaf users, like any other mobile cell phone user want to communicate with friends and family as often as possible using the most reliable methods and tools. The most important thing that users seem to care about is the ability to be able to read their friend s sign language on the phone. This is not possible with conventional interfaces like the one that appears in 9

23 figure 2. The figure below shows a conventional interface that is found on most synchronous video communication tool. Here you can clearly see that the other user (owner of the phone) is appearing on a smaller part of the screen, this is the interface that the MobileASL have chosen to go with, whereas South African users split screen. Figure 2:This figure shows a MobileASL video conversation: where synchronous video communication between two sign language users. (mobileasl.cs.washington.edu) Description of the problem In South Africa, there is a very huge gap between the haves and the have not s. An even bigger gap exists between Deaf and hearing users. Cell phone and network companies (service providers) in South Africa mostly develop mobile solutions for hearing users. In addition they offer very expensive rates for video calling (perhaps the most expensive in the world). Deaf users are restricted to text 10

24 based communication, forcing them to use English which they do not prefer, and forcing them to use instant messaging applications like Mxit. Deaf users are also forced to pay for voice called when they purchase air-time (recharge voucher). This is unfair because they don t ever make voice calls, but they end up paying this extra fee because some service providers don t have SMS (short-massaging-serves) bundles. Some of these service providers offer MMS (multi-media-service) at ridiculously high prices [2]. Research suggests that Deaf users do not want a synchronous communication system because they do not favor the video quality because it s poor and the additional high cost is thought of as not favorable, since most Deaf users are unemployed because of lack of education and skills. The same complains were raised by Deaf users when they were confronted with PC based applications [3]. MTN contract customers are expected to see an average increase of 5.48%, while the only prepaid service that will have its rates adjusted is the Pay As You Go Call per Second, which will be an overall increase of 1.53% on the prepaid plan. Vodacom's increases in terms of monthly bills range from 0.26% to 7.89%, and the percentages are based on an average customer's spend profile, meaning they will vary from customer to customer, dependent on which service is used. Cell C's overall average increase - dependent on which package is being used - ranges from 2% to 5% [2]. All this means that Deaf users are forced to pay more for most of the services that they don t need. Video calling is very expensive for Deaf users because it has some extra data (sound) that Deaf users don t need. What the software solution is capable of doing The software solution will be designed from the beginning as an alternative communication tool to Deaf users, and because of this fact it should allow users 11

25 to use their Android phones from version 2.1 up-wards to make video calls. This software should be able to minimize the cost of transmission independent of the subscriber (MTN/CellC or VodaCom) being used, thus allowing more time to Deaf users to use the tool. Most importantly the solution should be easy to use, and users must spontaneously understand how to customize it to suit their needs. The product will also be able to offer users an option of moving away from the text-based tools to a more natural way of communication. In accordance with the user requirements the system/software should be able to perform all these tasks that the users have put forward. See YY s interview below. There are eight user requirements that we started with. And they are as follows. 1) Application must have an easy natural interface 2) Application must not take a long time to set up 3) The application must be able to play pre-recorded video 4) The application must hide the technical details from the user 5) The user must never think about saving or compressing videos. 6) System must be adaptable to other functions if need be in the future 7) The system must be reliable and do what the user expects to be done 8) The application must have a respectable video quality compared to other video applications like the iphone s facetime. 12

26 What the software solution is not capable of doing The product is simply a tool to aid Deaf users to make video calls and stay in touch with their friends and family, it cannot teach Deaf or hearing users how to communicate in South African sign language and it cannot be used as most other communication tools like Mxit (Mxit is platform independent), this software is extremely platform sensitive and may only work on specific Android phones. The product will minimize the cost of video transmission for the user but it will not let a user without money make or receive video calls. 13

27 14

28 C h a p t e r 3 REQUIREMENTS ANALYSIS DOCUMENT Introduction Here the user requirements are transformed to a solution that will solve the user s problem. The system and software requirements will be highlighted in this section. The actual details of the problem/solutions will also be highlighted here. In this section some software engineering principles will be deployed in order to systematically analyze the user requirements and produce a suitable solution. Designers interpretation of the problem The system will be designed look just like popular video applications like Facetime on iphone 4 except it will not be synchronous but rather asynchronous. The software application will be implemented on an Android mobile phone. The acceptable frame rate for recording, storing and sending good quality videos is about 13fps to 16fps and most of these devices can easily reach this threshold. We want Deaf users to send video from their Android cell phones. This means the application should allow them to record, compress and transmit the video without the user having to deal with the underlying complexity. The interface of the application will need to be simple enough to allow users this comfort without compromising the functionality of the system. Deaf users will also need a notification system that does not include sound, because they cannot hear sound. The application should be able to cause the phone to vibrate when a new video message comes in from a friend. The user can 15

29 either set this option on their phone as a default or the application will do so for him/her. The fact that the application is video intensive and Deaf users depend on what they see with their eyes to communicate means that a test will have to be made to check which video codec produces the best video quality after the processing i.e. after sound has been stripped and video has been heavily compressed. The user will simply start-up the application and record video with on Android phone or select a previously recorded video from a memory cards and use the application to send the compressed and striped-down video. The video will then be sent over a GPRS/2G/3G cellular networks and/or WIFI with ADSL. Figure 3: This figure shows how video will be sent from one Android phone to another, but before the video is retrieved by the other user it must first be stored in the MMSC of the service provider. Breakdown of problem into parts The crux of the issue is that Deaf people want to be able to make and receive video calls as cheap as possible. They are faced with several problems: is the issue 16

30 of cost of transmission. As has been mentioned before in chapter 2, cell phone rates in South Africa are some of most expensive in the world. The second is the issue of applications that are not developed with the needs of Deaf users in mind resulting in on interface that might be too small and complicated to use. This is a setback for Deaf people because their interface needs are different, for instance Deaf people need to see themselves sign as much as watching others sign too. The third and the most critical issue is that the camera resolution might be too low (especially if front cam is used). Deaf people need a high resolution camera like the one at the back because they not only communicate with their hands, but also with their facial expressions. And these facial expressions usually give meaning to what they are trying to convey. The Figure 4 below depicts combine use and designer needs of the system. This picture comes directly from the user requirements as discussed in the previous chapter. The diagram also shows how the problem can be solved with some technical issues. It also highlights the hardware needed to accomplish a project of this nature. From this figure a clear distinction is made between user s needs and preferences and designer s contributions: showing how user requirements reduce to an analytical solution that depicts exactly what the user expects from the designer 17

31 Figure 4: This figure shows how the user requirements are organized and analyzed to give the user an impression of the solution. Existing solutions There are many existing applications on the web which promise to deliver good usability but most of these are not open-source, which means developers cannot improve and customize them for the specific needs of Deaf users. More-over these applications only work for some specific and expensive mobile phones, which means Deaf users who are not well-off cannot afford to own these devices. Apple recently released a new version of iphone which has one of the best camera resolutions in the market today. In this new iphone 4 they have created an application called Facetime and targeted for video communication and to some small extent to Deaf users. The problem with iphone 4 s Facetime is that it consumes a lot of bandwidth resulting in users having to pay more to buy it. For 18

32 South African users, Deaf or not, high-end phones like the iphone 4 is not a solution, but rather a white elephant. Here are a few communication tools which Deaf users can use, both cell phone-based and PC based: From Omnitor AB. Allan ec: a communication platform that gives you the possibility to communicate in a way that suits you best. This application is only useful if you have internet and a personal computer, most South Africans living in townships don t have internet connection or a computer. Figure 5 shows the PC interface of Allan ec which uses both voice and video. During the call, there is a possibility to have a real time text conversation. It can be used for example for avoid misunderstandings, or for making addresses and phone numbers easier to perceive correctly. This opportunity can also be used to carry the whole conversation in text, for example between deaf people and hearing people who do not know sign language. Figure 5 : Allan ec Multifunction-terminal for sign language, text and voice. 19

33 Allan ec can be used also by speech-impaired people. Voice can be used in one direction and written text as reply. Hearing-impaired people get the support by lip reading the counterpart in the video window. MobileASL Figure 6:MobileASL sign language video over mobile phones. Sign language is a visual language, mobile video phones have the potential to give Deaf users access to real-time mobile communication in their preferred sign language. MobileASL project aims to expand accessibility for Deaf people by efficiently compressing sign language video to enable mobile phone communication as shown in Figure 6. The project envisions users capturing and receiving video on a typical mobile phone. The users wear no special clothing or equipment, since this would make the technology less accessible. This application works on windows mobile phones, which are known to be very expensive, more-over the research to develop the application was done on a completely different group of Dear users, much different from South African Deaf users both socially and economically. Figure 5 shows the interface for the MobileASL interface. 20

34 Skype A program that enables voice phone calls over the internet, has a video chat component. Free software is widely available, and video service is built into services such as Google chat and Windows Live messenger. Video phones also enable Deaf-hearing communication, through video relay service, in which the Deaf user signs over the video phone to an interpreter, who in turn voices the communication over a regular phone to a hearing user. 21

35 22

36 C h a p t e r 4 USER INTERFACE SPECIFICATION Introduction The previous two chapters discussed the user requirements and how those will be analyzed to produce the best possible solution for a Deaf. Also possible methods of implementation were discussed. This chapter introduces the User Interface Specification (UIS). The description of the UIS includes the snap shots of the interfaces (screen) that the user interacts with. This chapter describes exactly what the user interface is going to do, what it looks like, and how the user interacts with the program. Description of the complete user interface The UIS consists of one main graphical user interface (GUI), which consists of different operations enlisted in the options. The figures shown are snap shots of the Android 2.1 emulator. The main window shows all the functionality of the application. The options on the menu are: (see Figure 6) Phone number textbox user will enter phone number they wish to communicate with here Record button used to record the video Finish recording finish recording or playing (e.g. record, play) Send - sends the video file to another user 23

37 Save saves the video file in the emulator s memory Loads - loads the video file from the emulator Exit - closes the application What the user interface looks like to the user Figures 4 to 7 show the UIS. The main screen shows how the user will follow a series of steps to finally end up with the applications allowing him/her to carry out a require task. This enables also allows the user to see and follow the steps that are taking place during the use of the application. Figure 7:the generic home screen for Android 24

38 Figure 8: The main menu screen for Android showing the application icon Figure 9: The main interface of the solution 25

39 Figure 10: The picture shows the recording of a video How the user interface behaves The user interface enables the user to carry out operations that are listed above. The system can be ported to an Android supported cell phone. The UI enables the user to record, play, send, stop and save the video file. The send option first connects to the MMS server and uploads the file. If a user has been sent a video file by a friend, then the MMSC will coordinate the communication between sender and receiver. 26

40 How the user interacts with the system The user must first record a video through the Record option to be able to use the other functions. The back camera is used to record the video. Figure 10 shows the record function. After recording the video can be sent to another user. The other operations are applicable only the emulator. The most important object is to make the system as easy for the user as possible to understand in order to avoid large amounts of ICT training. Furthermore, the system must hide all the relevant technical information, such as IP addresses, transport protocols, and control management of compressed files. Summary In this chapter User Interface were analyzed and observed. The various screens involved on the cell phone where analyzed. The various options or a menu that enables the user to interact with the application were analyzed. The system was designed to be simple and easy to use for all kinds of users, whether it s their first time using a touch screen phone or not, the main idea was to make the interface as simple as possible. In the next chapter, the High Level Design is analyzed. The object oriented analysis of the solution is discussed. 27

41 28

42 C h a p t e r 5 HIGH LEVEL DESIGN (OBJECT ORIENTED ANALYSIS) Introduction This chapter concentrates on the Object Oriented Analysis (OOA) or High Level Design of the problem. A detailed breakdown of the technical solution is discussed. At this point no algorithms will be discussed. However, a set of class diagrams will be given with detailed interaction between subsystems, including interface subsystems. User interaction design (use case diagram) This is the high level design of the solution from the point of view of the user. The diagram below shows a simple and straight forward story of how the user will interact with the application to carry out the required results. Figure 11 demonstrates the idea of one user interacting with another during video communication. In Figure 12, we see use case with two actors, the user and the Android cell phone. The user will use the interface to interact with the applications which is installed on the phone, without this relationship the user will not be in a position to carry out the specific task of communicating with friends and loved ones. The basic high level functions are entering phone number, recording a video, 29

43 sending videos, receiving videos preview the video and exiting the application. Figure 11: Use case diagram of one user interacting with another. Figure 12: Use case diagram of the solution. 30

44 class diagram showing the name, attributes, and methods of each class Figure 13: shows the classes that have been used and some functions which are called by the application. The main aim of Figure 13 is to show how the control from one module is handed over to another to give the application is functionality. Figure 12 shows the detailed description of the class diagrams and how the application will go about achieving its purpose; the main area of interest here is the media recorder which is the one responsible for almost the whole system function. 31

45 Summary In this chapter an analysis of the High Level Design was obtained. The various classes involved and how they interact with each other were described as well as the relationship between them. The problem solution was analyzed from an object oriented view. In the next chapter, the classes discussed in this chapter are analyzed further; pseudo codes of the solution are extracted from the classes. 32

46 C h a p t e r 6 LOW LEVEL DESIGN (OBJECT ORIENTED DESIGN) Introduction The chapter above described the object orientated design which is also known as high level design in great detail. This chapter takes the classes further by creating pseudo codes. During the rest of this chapter the high level design will be broken down into modules and programs, logic design will be done for every program and documented as program specifications. Figure 14: Showing the state diagram of MediaRecorder, responsible for playing and recording video. 33

47 The activity lifecycle This is how the applications will follow a predictable number of steps to accomplish the task that has been described in the analysis section. The application follows these set steps to ensure users friendliness and application stability, this is how the application will be controlled to perform the tasks that have been set forward by the user in the user requirements section. Table 1 describes in detail how these activities will take place, and what event or activity takes effect before another can be executed. The lifecycle callback methods that are listed in table 1, which describes each of the callback methods in more detail and locates each one within the activity's overall lifecycle, including whether the system can kill the activity after the callback method completes. Method Description Killable after next oncreate() Called when the activity is first created. This method is passed a Bundle object containing the activity's previous state, if that state was captured (see Saving Activity State, later). Always followed by onstart(). No onstart() onrestart() Called after the activity has been stopped, just prior to it being started again. Always followed by onstart() No onstart() onstart() Called just before the activity becomes visible to the user. Followed by onresume() if the activity comes to the foreground, or onstop() if it becomes hidden. No onresume() or onstop() onresume() Called just before the activity starts interacting with the user. At this point the activity is at the top of the activity stack, with user input going to it. Always followed by onpause(). No onpause() 34

48 onpause() Called when the system is about to start resuming another activity. This method is typically used to commit unsaved changes to persistent data, stop animations and other things that may be consuming CPU, and so on. Followed either by onresume() if the activity returns back to the front, or by onstop() if it becomes invisible to the user. Yes onresume() or onstop() onstop() Called when the activity is no longer visible to the user. This may happen because it is being destroyed, or because another activity (either an existing one or a new one) has been resumed and is covering it. Followed either by onrestart() if the activity is coming back to interact with the user, or by ondestroy() if this activity is going away. Yes onrestart() or ondestroy() ondestroy() Called before the activity is destroyed. This is the final call that the activity will Yes Nothing receive. It could be called either because the activity is finishing (someone called finish() on it), or because the system is temporarily destroying this instance of the activity to save space. A distinction between these two scenarios can be seen with the isfinishing() method. Table 1: The activity table 35

49 When taken together, these above methods define the entire lifecycle of the application activity. By implementing these methods, we then can monitor three nested loops in the activity lifecycle: The entire lifetime of an application happens between the call to oncreate() and the call to ondestroy() or onexit(). The application can then perform setup of "global" state such as defining layout in oncreate() were the interface is called to the front, and release all remaining resources in ondestroy(). In some cases the application activity has a thread running in the background to upload data to the network, it will create that thread in oncreate() and then stop the thread in ondestroy(). The visible lifetime of an activity happens between the call to onstart() and the call to onstop(). During this time, the user can see the activity on-screen and interact with it just as they would interact with any other application on the mobile device. Let us say for example, onstop() is called when a new activity (application) starts and this one is no longer visible. Between these two methods, resources can be maintained that are needed to show the activity to the user. For example, the user can register a BroadcastReceiver in onstart() to monitor changes that impact the UI, and unregister it in onstop() when the user can no longer see what you are displaying. The system might call onstart() and onstop() multiple times during the entire lifetime of the activity, as the activity alternates between being visible and hidden to the user. The foreground lifetime of an activity happens between the call to onresume() and the call to onpause(). During this time, the activity is in front of all other activities on screen and has user input focus. The video communication tool we are using will frequently be in transition in and out of the foreground for example, onpause() is called when the device goes to sleep or when a dialog appears. Because this state can transition often, the code in these two methods 36

50 should be fairly lightweight in order to avoid slow transitions that make the user wait. Implementation of the lifecycle (pseudo-code) This section explains how specific parts of the code will look like when they are coded and implemented, the logic of the code can be found and explained here, the program specifications are documented here. Below I will focus on specific tasks that need to be done in order for the application to yield usable results or outcomes. public class ExampleActivity extends Activity public void oncreate(bundle savedinstancestate) { super.oncreate(savedinstancestate); // The activity is being protected void onstart() { super.onstart(); // The activity is about to become protected void onresume() { super.onresume(); // The activity has become visible (it is now protected void onpause() { super.onpause(); // Another activity is taking focus (this activity is about to be protected void onstop() { super.onstop(); // The activity is no longer visible (it is now protected void onselect() { super.onselect(); // The activity is now selectable (user can now chose the video the protected void onpreview { super.onpreview(); 37

51 // The activity is visible for user to see videos (it is now protected void onsend() { super.onsend(); // The activity is visible (user can now send protected void ondestroy() { super.ondestroy(); // The activity is about to be destroyed. 38

52 C h a p t e r 7 IMPLEMENTATION Introduction The previous chapter gave an overview of how the actual implementation of how the project looks like and how it will be made to function on the Android mobile phone. Various classes together with their pseudo codes were discussed and explained further for simplification purposes. The previous chapter holds the introductory keys to this chapter, which are now detailed. In this chapter a method of implementation is discussed which is the cell phone implementation of the application. We will also discuss how the video from one user will be sent to another user with an Android device through the service provides hardware and software as shown in Figure 2. Software s deployed Android SDK This is a software development kit for Android applications. The SDK enables developers to build applications using the Java programming language. It includes an Android virtual device (AVD) which is used for testing of applications before being imported to the real mobile phone device. The use of integrated development environment (IDE) which is based on the Eclipse IDE. It enables developers to quickly and efficiently create code, test and deploy software for the Android operating system. Android ScreenCast suite It is an application programming interface (API) that creates an interface of connectivity between an Android phone and a PC. 39

53 Multimedia Messaging (MMS) The application will use the MMS system from available service providers in South Africa. The MMS will use (Wireless Application Protocol) WAP Protocol Stack (WAP 1.2 or WAP 2.0) for message delivery. When a MMS Message is sent to a (Global System for Mobile Communications) GSM or (Code division multiple access) CDMA recipient, the GSM/CDMA device will receive an SMS message in WAP Push format. This WAP Push message includes a notification that a new MMS message exists. Then, the device will open a data connection using a specific APN and use the WAP 1.2 or WAP 2.0 protocol to download the actual MMS Message from the MMSC (MMS Center). When sending MMS messages the GSM/CDMA device also uses WAP 1.2 or WAP 2.0 and WAP Gateway/Proxy to deliver the message to the carriers MMSC. This is how the application will allow users to send and receive videos, as shown in Figure 2. Hardware s deployed Phone Camera (back) used for capture the video that will be sent to other users. Android Device used to house the application that will allow the user to perform tasks which have been discussed in earlier chapters. MMSC this is the service provider s cell phone tower. This is where the video messages get redirected to reach their destinations. Regardless of which service provider the video was sent from. Challenges Every programming and software engineering project has hardships and this one was no exception. There were a lot of technical problems encounter throughout the development of the project. We had technical challenges when it came to 40

54 version control, where some API s existed in one version but were dropped in the next version so we had to edit and/or develop for the API which was most suitable for our project. During the development and implementation of this application we used the emulator and since sending and receiving MMS video is a real time activity and emulators do not support real time video recording this presented a challenge. We were forced to use the because it is not easy to debug on the mobile phone, if the application cannot be tested on the emulator before ported on the mobile phone always raises doubting views as far as efficiency and time is concerned. Nonetheless the project will be continued on the mobile phone and developed from both the Android Virtual Device (AVD) and the mobile phone. The most testing challenge was forcing the application not to record sound and also to use full screen video. To achieve this was especially difficult because Android 2.1 does not support a version of the MediaRecorder that allows a developer to modify multimedia properties. To fix the problem we have to move the application one API version up so we can get to use the newer version of the MediaRecorder. Once we upgraded the application we could then call the Camcorder class which gave us more flexibility. The other challenge we faced during the development of the application was that we had used a new Android device from vodafone that can only send/receive MMS from only the same model. This caveat exposed us to some small setbacks like fixing the screen dimensions to fit exactly the screen size of the vodafone

55 Code documentation The source code is given in the Appendices shows the basic functionalities of the project which implement video communication on Android. The following sections discuss four classes which allow the application to function the way it does and satisfies user s requirements. deafplayer.java class This class can be found in Appendix B. This class is responsible for opening up the video that has been received and notifying the receiver that there exists a video that is ready for retrieval and it has not yet been opened. This class further carries the details of the new video and keeps a record of those details. The details are: the sender s name, the size of the video, the duration, the date sent and received, and were the video can be found on the phone memory card. This class is also responsible for creating a new file path were the video can be found if it s the first video to be received and it also creates a new file name if the videos current name already exists in this directory. If the video happens to be irretrievable then the class will return an error to the user s screen (this can happen when there is not enough space on the memory card of the phone). This class is also responsible for communicating with the notification manager class which is an inbuilt Android application programming interface (API) which is responsible for making the users mobile device vibrate or give some form of signal or interrupt. mediarecorderrecipe.java class This class can be found in appendix C. This class is called by the MediaRecorderRecipe.java class. The MediaRecorderRecipe is responsible for the video recording capabilities of the whole application. This class sets all the desired properties of the video that the application will eventually output. It functions by 42

56 calling the MediaRecorder.java class which is the earlier and much simpler implementation of the multimedia class used in Android today. This class uses the MediaRecorder.java for setting the surface view and the surface holder. The next class that is used by the MediaRecorderRecipe.java class is the latest Applications Programming Interface (API) which is called the CamcoderProfile, which is responsible in setting the bitrates to the required value of the programmer. This class also creates a directory where the newly recorded video will reside until the use deletes them or uses them. MediaRecorderRecipe.java allows the user to have time in positioning their mobile device as it lets the application go to sleep just for 6 to 12 seconds depending on the user s needs. newmail.java class This class can be found in appendix D. This class does a few important things for the rest of the application. It is the class that is responsible for the allowing the user to send a video to a friend. It controls the methods of sending and it communicates directly with the applications processing interface for calling the phonebook so the user can pick a list of contacts they want to use in order to send the video. This class also communicates with the Home Menu application. This is to allow the user to enter and exit the application. The applications programming interface is directly linked to this class since the user will be calling this class when they first start the application. Here the user can first start up the application and immediately record a new video is selected one from the memory card. This class lets the user PLAY, SELECT, RECORD or EXIT the application. This is the center of the application; it takes input as touch gestures that the user gives when interacting with the application. 43

57 player.java class This class can be found in appendix E. This class is the one that deals with the PLAY button when the user selects and decides to preview a video. This class communicates with the Android MediaController class and the mediaplayer class since when the user selects PLAY the controllers for the next, previous, pause and rewind needs to be available. This class makes it possible for us to not recode the existing classes and use the API's that has been provided by Android. This class is also used to verify that the video file exists and the path name that has been specified is correct for all newly recorded video. Depending on the buttons the use presses this class will cause the whole application to behave in a certain unique way. Summary In this chapter an approach for implementing the project was discussed in detailed together with the source code. The project was implemented on the mobile phone emulator using and the actual Android device; however the emulator does not support video streaming. The mobile development will be continued using real mobile phone devices. The application turns off the phones microscope in order to reduce the resultant file-size. The code documentation given shows two basic functionalities about video communication, send, record and SD card select button functionalities will be found on the application CD. 44

58 C h a p t e r 8 TESTING Introduction The previous chapter focused on the implementation of the application. It gave a detailed documentation of the code used and explained how each part works to make the application usable, functional and how each component contributes to the project. The testing chapter will discuss the usability, functionality and the performance of the system and then evaluate the results. The process of testing is documented for both the user interface and the system, to ensure that the solution meets the system requirements and that it is robust. Testing plan and Criteria In chapter we will conduct seven kinds of testing in six weeks which are as follows: People and User Interface Usability testing (week 1): We did basic usability testing with computer science and some information systems students, the application is relatively straight forward because it does record, send or select and send. So participants said it was easy and after seeing the demo they did not need spend much time learning it. We gave the app to some friends at the university residence with Android phones and once they knew the purpose of the app they understood it quickly, but they could not send any MMS since their phones were incompatible. But they did evaluate the interface. 45

59 Figure 15: The graph showing the degree of complexity users encountered while using the application From the results obtained here we found that the application is very easy to understand and easy to learn as shown in figure 1, this increased user involvement and encouraged users to use the application more. From the MobileASL literature we found that Deaf users are not prepared to spend time learning a complicated application, instead they are likely to use the applications they were using other than the one they were given. The purpose of Table 1 is to find out what users like and/or dislike about the application, to check if there is anything that can be improved to make a better user experience for them. From Table 1 we find that users generally liked using the application and would recommend it to their friends. Multi-User/Load testing (week 4): In this section of the test we will let the application be used by a lot of users, sending and receiving MMS constantly, this 46

60 is important to check the amount of stress the application can take before it cannot function anymore. The main object of this testing exercise is to break that application, so we can identify the maximum operating capacity of the application. Here we were sending and receiving MMS between 858's. We were also checking how quickly the application can open and close when new mail comes in. The thing we discovered is that the Android API which handles new mail (SMS, or MMS) is very efficient in handling incoming and outgoing traffic, so we did not have to do any coding that managed the queuing of outgoing MMS and the presentation of the incoming. We found because the API manages traffic, Android does not allow my application to intercept incoming MMS so we could not do it with my application. This is not a bad thing because the Android API is already so good at doing this. Device Performance testing (week 2): In this part of test we will ensure that the system provides acceptable response times (which should not exceed the maximum allowable time for MMS). And set benchmarks on the maximum/minimum time the user must expect to receive a video message from the sender. This part was a bit tricky to test because we had to do two types of performance testing. a) The performance of the application itself, recording, blocking sound and resizing the video on these small CPU phones is very resource expensive, especially because we doing all this at the same time. So we could only test this part by comparing how long my application recovers after recording on different Android phones. b) The sending and receiving time of MMS, but this we did not get into detail with because there is nothing programming wise that we could do to improve the time an MMS is received. Question Sony Ericss Moto rola HTC Wildf Vodaf one Vodaf one Samsu ng 47

61 on MB52 ire Galaxy X10 5 S Does the app work on this device? N N N N Y N Can you send MMS to this device N N N N Y N Can you receive an MMS from this device? N N N N Y N Y Y Y N Y Y Can the app be installed on this device Table 2: Showing which devices the application functions on without errors We also tested the performance of the application. This was done to check how quickly the application responds to user input. From the results we see that the applications performance is reasonable. Any response time that was less than 3 seconds on a mobile application is considered reasonable [11]. We also discovered that the applications quick load time depends on the mobile phone being used. Figure 16: Showing the recoverability of the application after record Since the only device that will work full with the application is the Vodafone 858 we had do test how fast it can recover after recording in order to perform the next user request. From Figure 2 above we see that the application takes mostly 1 48

62 second to recover. This is important to know because the Vodafone 858 is a lowend Android device and it has very little computing power. The users did not find waiting for 1 second to be such a drawback. In some cases they did not even notice the wait. So there will be no need to improve the applications algorithm at present. Figure 17: Graph of application performance using time. Figure 3 simply tells the same story that Figure 2 telling; out of 10 users 5 of them say that the performance of the application in general is acceptable. None of the users who were involved in the testing found the application to be misusing resources and thus taking time to perform a required action. (User) Acceptance test (week 3 for 3 days): In this test we plan to ensure that the application will operate in the manner that is expected, we will test to see if all the functions are behaving the way they were intended to behave. We will look at all the devices and the source code in more detail to make sure that everything operates the way the user has specified. This is when we will mimic the conditions the application will be used under, once the application is installed on Android devices and is in user s hands. 49

63 Install/uninstall testing (week 5): Testing for full, partial, or upgrade install/uninstall processes on different Android devices and different Android versions. Here we were installing the application on different Android devices. Our main aim was to see if the application installs correctly without errors on Android devices running 2.2 and also to check if the uninstall process is as painless as it should be. Devices running Android 2.1 and lower do not support this application, so they application could not be installed on them, but Android devices running 2.2 and higher do support the app, and the install and uninstall was pain free and successful. Multi-platform testing (week 5): Here we are investigating how the application works on different Android 2.2 devices. We recently found that the application works very well on 2.2 devices except for the Vodafone 858, so some reason users can not send MMS to other 2.2 Android devices which are not 858. Further investigation shows that this issue lies with the service provider (both MTN and Vodacom). Vodacom has come out saying they have not yet released MMS settings for the 858 device which will allow it to send MMS to other phones which are not 858's. So far in the testing phase, sending MMS to other device models has been temporarily put on hold. The conclusion here is that the application is functioning as it should; the holdup is with the service provider People and Cost The application uses MMS which users are already familiar with. MMS is therefore accessible to all of our target users. This is the primary reason we have decided to use MMS. Service provider lets their customers use the MMS service at a cost. In this section we will investigate how cheap or expensive the MMS service is to our target group. Provider SIZE(KB) COST(ZAR) 50

64 MTN to OTHER MTN to MTN MTN 300 R0.95 R0.75 VODACOM 300 R0.80 R0.80 Table 3: Network provides cost for MMS From table 8 we can see that the user has to pay considerably more when they are sending to another user who is on a different network. The file size of the video or MMS can not exceed 300KB. So we would like to calculate how much the user will have to pay if they buy MMS bundles. Since users can also chose to purchase MMS bundles which are charged at R1.00 per Mb as shown in Table 9. So we achieve this by performing a calculation as follows The real calculation would go something like this: From the equation above we see that the user pays much more when they use MMS bundles. Product Monthly Subscription MMS cost MMS cost prepaid Bulk MMS Free subscription R0.99 to R0.91 R1.28 MMS lite Fee subscription R0.75 to 0.72 R0.96 Table 4: subscriptions offered to customers 51

65 Table 5: This table shows how we plan to carry out the testing phase of the application in six weeks. Video quality results (MobileASL) The Mobile ASL group has studies four different frame rate combinations for videos containing periods of signing and periods of not signing. Previous studies indicate that 10 frames per second (fps) is adequate for sign language intelligibility [6], so they chose 10 fps as the frame rate for the signing portion of each video. For the non-signing portion, they studied 10, 5, 1, and 0 fps. The 0 fps meant that one frame was shown for the entire duration of the nonsigning segment regardless of how many seconds it lasted (a freeze-frame effect) [10]. For this project we set the frame rate to 13fps on the Android mobile phone. Because Android phones are different and they run different API s it is not possible to control the frame rate. It looks like the frame rates only change as a result of the environment lighting conditions. This is because MediaRecorder.setVideoFrameRate(some integer) does not affect the average frame rate as it can be seen on Table 3, which is always close to 17fps to 31fps, regardless of the asked rate from 4 to 30 fps [11]. 52

66 Figure 18: On the left Screen shot at 1 fps and on the right Screen shot at 10 fps. The fact that Android auto tunes the frame rate according to the lighting condition helps because it always brings up a video that is not too low (left of Figure 15) and not too high (more than Figure 18ftp). It sets it as the Mobile ASL team have researched users want it. Video quality testing (lab) The following table shows default output formats from the MMS application on the home device and some video attributes from MobileASL. Table 3 serves as a benchmark for the application we are developing. MMS MobileASL file size (KB) 286 unknown video format video/mp4v-es X246 from H246 resolution fram rate (fps) to 15 bit rate (Kbps) to 80 Table 6: shows the theoretical values achieved by related applications 53

67 Table 3 shows the same data as in table 1 the application we are developing. Here we start with all MediaRecorder video attributes as DEFAULT and change the video encoder and the output format. This table is meant to reveal what values change as we change the video encode from DEFAULT to MPEG_4, H264 and H263. file size EXPERIMENTAL (KB) video format video/h video/h video/mp4-es X-H246- VIDEO video/h output format Default MPEG_4 MPEG_4 MPEG_4 MPEG_4 video encoder Default Default MPEG_4_SP H264 H263 video source Default Camera Camera Camera Camera resolution frame rate (fps) 21 to to to to 31 bit rate (Kbps) Table 7: shows how the video properties change as the encoder is changed Table 4 show that each time the video encoder is changed from DEFAULT the video format also changes. This result is expected since the encoder is directly tied to the video format. The interesting parameter to look at is the frame rate which seems to stay relatively the same throughout the whole experiment. It looks like the frame rates only change as a result of the environment lighting conditions. This is because MediaRecorder.setVideoFrameRate(some int) does not affect the average frame rate, which is always close to 17fps to 31fps, regardless of the asked rate from 4 to 30 fps. 54

68 SET CONSTANTS IN THE CODE set frame rate (ftp) 13 resolution max file size (kb) max record time (ms) Table 8 shows constants during lab testing Table 5 shows the values that the application was running with during the testing phase. These were treated as constants and not expected to change. But in some cases they did change. Like in the case of frame rates which kept on changing because of lighting conditions. Some values were consistent like the resolution and the max file size. Summary (Analysis) In this we discussed different testing strategies that were used during testing. We also discussed test results of various components of the application and gave graphical and tabular representation of some. The testing was successful as all the testing strategies conducted were completed successfully. We also took a look into the results and analysis that was conducted by the Washington s University group known as the MobileASL. The most important discovery made during this testing phase is the finding that Deaf users need higher video quality for communication than non-deaf users. The MobileASL found that although users need higher quality video they cannot tell the difference when the frame rate is between 10ftp to 15ftp on cellphones [10]. 55

69 We also performed usability testing, this form of testing was critical simply because most applications Deaf users interface with are not meant for them. From the results obtained here we found that the application is very easy to understand and easy to learn as shown in figure 16, this increased user involvement and encouraged users to use the application more. From the MobileASL literature we found that Deaf users are not prepared to spend time learning a complicated application, instead they are likely to use the applications they were using other than the one they were given [7, 8, 10]. We also tested the performance of the application. This was done to check how quickly the application responds to user input. From the results we see that the applications performance is reasonable. Any response time that was less than 3 seconds on a mobile application is considered reasonable [11]. We also discovered that the applications quick load time depends on the mobile phone being used. Thus we can conclude that the user s requirements gathered in chapter 1 were met. However since the application is still under development, we can say that the first prototype met the desired results. Thus following incremental model of software engineering life cycle, there is a need for improvements in other aspects of the application. 56

70 57

71 C h a p t e r 9 USER GUIDE Introduction In the previous two chapters we discussed the implementation and the testing results of the project. This chapter gives our instructions and guidelines on how to use the application. In this chapter we discuss further the guidelines or methods which the user will deploy in using the application to its fullest potential. We will be discussing how the user will first install the application on their mobile device and how they should continue to use it on a daily bases. Application Installation on Android In order for the user to have the application on their mobile device, the user has to download the application from with their Android device. The file which the user must download is an apk Android file; all Android application files have this extension. Once the user has the file on their device they can navigate to wherever they have save it and tap on it to run it. When the user has tapped the file the install interface will pop-up just like the one in Figure 18. When the install screen shows up the user will be shown a number of permissions that the application needs in order to perform all its tasks. At this point the user should select install and wait for about 30 seconds depending what Android device the user is using. After the user presses the install button the application will install and save itself on the device physical memory and the application icon will be found on the phones menu screen. 58

72 Figure 19: The install interface After the application has been installed then the application icon appears as shown on Figure 19. Here the user can start to run the application by just tapping on the icon. Using the application (Getting started) The user can start the application by going to the device menu screen where the application icon/shortcut as shown in Figure 17: lives and from here the user can then tap the icon to start the application where the user can then begin to interact with the internals of the application. 59

73 Figure 20: Android home menu screen Figure 19 shows how the user can start using the application. When the MobileVideo icon is pressed then the application menu screen which is shown by Figure 20. Figure 21: Application home menu screen 60

74 In the application menu screen there are five buttons which are PLAY, SELECT, RECORD, SEND and EXIT. The user will interface with these buttons to use application. SELECT this button allows the user to select a video from the Android memory storage. Figure 22: Showing videos on device memory RECORD this button lets the user record a new video with the back-facing camera as shown in Figure

75 Figure 23: showing the recording of a new video when the RECORD button is pressed PLAY this will be inactive until the user selects a video they want to replay. SEND this button allows the user to send a video to their friends, this button calls the Android Messenger and opens the MMS API which lets the user send the video using the MMS functionality. Figure 24: Showing the contact list after the SEND button was pressed 62