A Hands Free Browser Using EEG and Voice Inputs

Transcription

1 A Hands Free Browser Using EEG and Voice Inputs Joy Bose, Amit Singhai, Ankur Trisal, Vinod Keshav, Utkarsh Dubey Web Solutions Samsung R&D Institute India- Bangalore Bangalore, India {joy.bose, a.singhai, ankur.trisal, v.keshav, Abstract Mobile web browsers are increasing in usage due to the widespread use of various mobile devices including smartphones and wearables. It is useful to enable such browsers to have hands free functionality to enable people to use them when their hands are occupied when driving or performing other tasks. A hands free browser would be also useful for disabled or paralyzed people. In this paper we describe the design of a hands free browser. It uses EEG inputs using a NeuroSky EEG sensor, as well as voice inputs. Various functions in the browser including add to favorites, scroll up or down, next and previous tab, open an URL that is part of the favorites, are mapped to varying attention and blink levels recognized by the EEG sensor. Maintaining of the concentration level higher than a threshold enables the user to cycle through a list of items, which could be menu items or history items or most visited sites in the browser. A deliberate double blink enables the user to open the currently selected URL or menu item. Other functionalities are performed using voice in a way that is more intuitive and user friendly. We describe the browser architecture and various implementation issues, as well as some screenshots from the implementation. Keywords EEG; voice based browsing; machine learning; mobile web browser; accessibility I. INTRODUCTION Wearable devices fitted with EEG sensors are becoming more popular for a wide variety of users, with a number of commercial kits developed mainly for the gaming market by a number of manufacturers such as NeuroSky. Nowadays the EEG kits available for mass consumer electronics are cheaper, faster and easy to use. They have dry electrodes, are portable and lighter and seem almost like an audio headset, making it possible to use with ordinary applications in daily life. Mobile web browsers are becoming more common as a means for many users to browse the web, due to the ubiquity of mobile phones and the portability of such browsers in devices such as smartwatches, smart TVs and cars and even in appliances such as fridges and washing machines. The importance of hands free functionality for mobile browsers can be described in the following ways: Hands free usability: When the user is driving a car or the hands are occupied for various tasks and not available for controlling the web browser, it might be useful to have a browser that does not require use of the hands at least for control functions. Many people use the web browser on their mobile phones when travelling in public transport or waiting in queues. A hands free browser can present a more intuitive and better user experience by taking advantage of multimodal inputs to complement or replace the touch functionality while using the browser. With the advent of a wide variety of wearable devices including smart watches and health bands and headsets, such a browser can become a reality. For paralyzed or disabled people who may not be able to use their hands for browsing, a hands free browser can be a great help. In this paper we present the design and implementation of a hands free version of the mobile browser by taking input from EEG sensors for navigation, along with voice inputs for opening URLs. The rest of the paper is organized as follows: in section 2 we look at related work in the area. Section 3 introduces our proposed solution combining EEG and voice based browsing. Section 4 gives the browser architecture, introducing the elements we needed to support EEG based navigation. Section 5 gives details of the Android implementation. Section 6 discusses some issues with the system. Section 7 proposes some user interfaces for various browser functions based on EEG input, and Section 8 suggests some avenues for future work, and concludes the paper. II. RELATED WORK EEG sensors have been in vogue for many decades, mostly for medical purposes. However, consumer oriented cheaper and mass produced EEG sensors that can easily interface with phones and other devices via publicly available APIs have only recently become popular. NeuroSky s MindWave [1], Emotiv s EPOC [2], Muse [3] and Melon [4] among others have kick started a revolution in EEG driven applications and gadgets, especially in the arena of games and virtual reality. We found one work related to web browsing using EEG sensors. Bensch, Karim et al developed Nessi [5], an EEG based browser where the user could move the mouse pointer by training and modulating their thoughts. However this work did not focus more on the widespread usability of the browser. Besides it used a full research grade EEG kit, which is difficult and cumbersome to set up and has no interfaces with devices

2 operating systems such as Android, instead of a widely available commercial one. Apple Siri [6] and the voice interface in Google Now [7] are more common solutions for Voice based browsing, although they are more geared towards web searches based on voice commands. the same. Tan et al [8] used a low cost NeuroSky EEG sensor (similar to the one used in this paper) to detect mental states. Liu et al [9] similarly used the NeuroSky Brainwave EEG sensor to detect attention levels. Vourvopoulos [10] used the NeuroSky kit to control robot navigation by manipulating the meditation and concentration levels. But till now we have not come across any implementation of a web browser or such core applications using any cheap commercially available EEG sensor. In this paper we propose such a system. Fig. 1. Illustration of the high level architecture of the hands free browser. Fig. 2. Flowchart illustrating various preprocessing steps for the EEG input on the web browser, where the recorded EEG output is compared to a pre-existing standard in order to determine the user s emotion or stress level. Fig. 3. Architecture of the web browser showing various EEG related components Using the NeuroSky sensor kit, some research has been done in detection of mental states and functionality based on III. PROPOSED SOLUTION: EEG WITH VOICE BASED BROWSING The steps to achieve the desired functionality of controlling the web browser via EEG and voice are as follows: The application on the mobile device maps the user s concentration level, meditation or relaxation level, blink to commands in web browser. The SDKs provided with the NeuroSky EEG sensor are used for this purpose. This is supplemented by voice functions. Mapping a pre-trained thought (a predefined fixed URL for example) with browsing action (such as browse to this URL). Yes-No thought pairs that are trained and recognized using a suitable machine learning algorithm over EEG raw data can be used for activating and deactivating certain functions in the browser. Only a fixed number of thoughts can be mapped in this way and requires training and a machine learning algorithm such as SVM for training and classification. This is far more difficult to achieve since the EEG signals may not be distinguishable, given the amount of noise and the fact that there is only one sensor in the NeuroSky headset. In the next few subsections we describe in detail the steps taken to process the EEG and voice inputs. A. Processing of EEG inputs Here we consider the general steps for processing the EEG inputs on the web browser. Data Capture: EEG Sensor output is captured through a wearable EEG kit, such as NeuroSky s Mindwave and sent to a mobile computing device such as a smartphone, which is paired using Bluetooth. Preprocessing and feature detection: The output is then preprocessed on an application running on the mobile computing device. Feature detection is done on the EEG output in the mobile device to detect and estimate the emotion, stress level, concentration and other EEG related parameters. Additional data from other sensors and sources such as the front camera, wearable heart rate monitor, pulse rate monitor etc. can also be combined with the EEG sensor output to produce a more accurate reading of the emotion. We have not needed this step in case of the NeuroSky kit since we have used their existing APIs. In case of the API available with the NeuroSky EEG kit, the concentration and meditation level of the user are detected and provided, as well as blink and heartbeats.

3 UI Changes in the web browser: The User Interface component of the web browser running on the smartphone is then updated in response to the EEG inputs. The flowchart in Fig. 2 illustrates these steps. B. Browser components related to voice based browsing In our implementation, as mentioned, EEG is used for taking user inputs. However in certain cases such as opening a specific URL, EEG inputs can be too clumsy and inconvenient for the user. For such cases we have taken voice as an additional input. In this subsection we consider browser components related to voice browsing. Voice browsing involves the following steps: Using the existing W3C MediaStream specification [11], the user s voice is captured by the browser. The captured audio stream is then sent to the server for Speech to Text translation. We used the Google Speech API [12-14] available with Google s Chrome browser and sent the voice to the Google server for conversion to text. The converted text is then received from the server by the browser in the form of a JSON response. This is then parsed by the web browser. If a command keyword or phrase (such as add favorite, add bookmark, open new tab, search, open ) is detected, then the browser translates the command to an executable action. For example, if Open New Tab command is provided as input, the browser will convert the text to a new tab command which is responsible for opening a new tab on the Browser. Input data can also be specified along with the command. For Example: When Open bbc command is detected, Open is taken as the command and bbc as the input to the command. The browser, will now look-up all actions (pre-configured and stored in the browser) that can be performed using that command. In this case, bbc.com webpage is loaded automatically. For our browser, we use the voice browsing component for opening URLs and the EEG component for performing navigation related functions. IV. PROPOSED BROWSER ARCHITECTURE In this section we describe the architecture of the mobile web browser that supports EEG and voice inputs. An activity on the Android device listens for inputs from the EEG sensor. Once the EEG input is detected, it classifies the input as belonging to one of the determined categories (concentration, blink, meditation, etc) and finally invokes the specific browser function as required. The SDK available with the EEG kit encapsulates much of the mentioned functionality. One thing to note is that the range of concentration levels varies from user to user and so it is important for us to establish the baseline concentration level for each browser function for each user. For this purpose we record and plot the concentration level and blink level of the user that is captured by the EEG sensor when the subject performs some common browser tasks including selection, and navigation. Then we determine the optimal concentration level to use as a threshold, in such a way that the user feels comfortable when using the browser and the automation of functions happens in an intuitive way. An additional factor is visual feedback (by seeing how the navigation and other controls) or biofeedback, which helps the user to better control their concentration levels when using the EEG device so as to get a more smooth operation when navigating the menus and other actions. Our focus is to optimize the usability of the hands free operation on the web browser. The components of the system for implementing the EEG interface on the web browser are listed below. It is to be noted that the provided SDK contains some of the functionality of the components. A. EEG Driver This is part of the OS kernel of the device. This is the low end driver that communicates with the EEG sensor (such as NeuroSky) and receives the user s raw EEG data stream in real time at the moment when they are using the browser application. B. EEGRec This is part of the hardware abstraction layer, the role of this component is to record, store and preprocess (filter out noisy components etc) the raw EEG data received from the EEG driver component and feed the filtered EEG data, corresponding to the user s concentration, meditation or emotion to the next layer. Since we are using the APIs provided by the manufacturer of the EEG kit, we did not perform any extra preprocessing. C. EEGLib This is a software library that takes as input the EEGRec output from the EEG recorder in the hardware abstraction layer, processes the EEG data to extract the concentration or meditation level, and outputs it to the browser component of the application layer. As mentioned, the SDK provided with the NeuroSky kit contained the required functionality of the library. D. Browser interface This is a service that runs in the background of the browser on the mobile device as part of the application layer and listens and responds to the EEG signals relayed via Bluetooth. It also implements the user interfaces to control various browser functions via EEG inputs. In our system we used an Android Web View to implement the browser interface. Details of the Web View are provided in the following section.

4 V. ANDROID IMPLEMENTATION In this section we describe the implementation details on the system on a mobile device running Android and communicating with the NeuroSky EEG kit via Bluetooth. First of all, we introduce the NeuroSky SDK and later we describe the web browser components to implement our system. A. NeuroSky SDK The NeuroSky SDK (Developer Tools 3, downloadable from the NeuroSky store) [15] includes the necessary APIs. We start a new activity in the Android system that listens for the EEG input from the NeuroSky sensor. The attention level, meditation level, blink and raw EEG data are obtained in the Android device from the output of the EEG sensor, using the NeuroSky EEG SDK [15]. B. WebView in the web browser We use a Web View in the Android operating system to implement the browser interface in a mobile device. A similar interface to the web view can be used in case of other mobile OS s such as ios (FSWebView) Mozilla Firefox OS, Tizen OS etc. For our purposes, we wrote a new web view that lists the favorites, bookmarks etc. In the Android application we create two Activities: the Main Activity and the Browser Activity. Fig. 5. Screenshot of a mobile web browser on an Android system, with category items to navigate. The currently selected item is shown in Orange. Using the EEG input data obtained in real time from the EEG sensor, the user navigates to items in the device home screen and selects a particular item to see the contents. For visualization for items under navigation a user friendly interface is provided. One such simple interface is shown in Fig. 5, where the currently selected item is highlighted. C. Main activity The Main Activity is visible to the user when the browser application is launched. It contains categories of websites which the user wants to view, such as Latest News, Weather etc. This is illustrated in Fig. 4. Fig. 6. Screenshot of a mobile web browser showing the browser activity with selected web page loaded Fig. 4. Screenshot of a mobile web browser on an Android system showing the Main Activity, with category items to navigate.

5 D. Browser Activity Once the item is selected by user, we launch the Browser Activity and the URL associated with the item is loaded into the embedded web view. This is illustrated in Fig. 6. From the browser activity in order to return back to the home screen or main activity, the user has to do a deliberate double blink. on this, we kept the difference of 20 as a threshold level to distinguish deliberate blinks for this user. In our web browser interface, we used deliberate double blinks to open the currently selected item or link in a new browser tab. Fig. 7. Plot of the blink level of the user as detected by the NeuroSky EEG sensor Vs time in seconds, distinguishing the deliberate blinks (red circles) with the casual blinks and other eye movements (peaks in the blue lines other than the red circles) for one user over a series of trials. E. Threshold level for deliberate double blink In our browser implementation, we use deliberate blinks detected by the EEG sensor to open a currently selected item in the web browser. It is to be noted that user blinks can also be detected using a front facing camera on a mobile device. However using the EEG sensor to detect the blinks makes it more accurate. The NeuroSky sensor fires a Blink event and the SDK gives a numerical indicator of blink strength (value between 0 and 100) when it recognizes a blink. We have seen that the blink event is fired sometimes for sudden eye movements as well. One issue while dealing with blinks is distinguishing a casual blink or eye movement from a deliberate blink, since we intend to use deliberate blinks by the user to select the menu items in the web browser. In order to determine this, we recorded the blink values for a series of trials on one user, with a mixture of deliberate double blinks and casual single blinks. In our experiment, we found that it is the difference between the current and previous values, rather than the absolute blink value, that determines if a blink is casual or deliberate. The blink levels obtained for a series of trials with the NeuroSky kit are plotted in Fig. 7. For plotting, we used the OpenWave visualizer tool code [18] and GraphView library [19] available from GitHub. From the graph in Fig. 7, we can see that the difference of 20 or more between the previously recorded blink level and current blink level correctly indicated a double blink. So it seems that the device can differentiate between a normal (or casual) eye blink and a deliberate or forceful eye blink. Based Fig. 8. Plot of the attention level of a user, recorded over three iterations (plotted in green, blue and red), while reading a Google news article on a mobile web browser. Scilab is used for plotting. 1 Hz is the recording frequency. The attention level is a value between on the Y axis, Vs time in seconds on the X axis. F. Threshold level for attention or concentration value We also recorded and used the concentration level recorded by the NeuroSky kit to navigate across a list of items (such as menu items or history items) in the web browser. Fig. 9. Plot of the meditation level of a user, recorded over the same three iterations as the attention level (plotted in green, blue and red), while reading a Google news article on a mobile web browser. Scilab is used for plotting. 1 Hz is the recording frequency. The meditation level is a value between on the Y axis, Vs time in seconds on the X axis.

6 The concentration or attention level recorded by the EEG kit is another value between 0 and 100. In order to determine the threshold for the attention or concentration level using the Neurosky kit, we made a series of recordings of the concentration level when the user was reading a google news article using the web browser. The results are plotted in Fig. 8 over three iterations, plotted using Scilab. We can see that the concentration level remains above 30 consistently (100 being the maximum) with comparatively less fluctuations, when the user is paying attention to the article being read. Fig. 10. Plot of the attention (in green) and meditation (in red) level of a user recorded simultaneously, while reading a Google news article on a mobile web browser. The meditation level is much more variable than the attention level G. Navigating across list items based on concentration level While recording the attention or concentration level of the user, we noticed there was a delay between the user s changing concentration and the time it registered an event on the system. Another issue was that the concentration level fluctuated quite widely. In light of the observed issues, we decided to keep the concentration threshold as a parameter in the system, which the user would set after playing with the system for a while. This is because the concentration level varies from user to user and depends on a number of parameters such as the user s mood or time of day even for the same user. In case of the user with whom the readings in the previous subsection were taken, we kept it as 30. While navigating a list in the web browser, if the concentration level stayed over the threshold for 1 second, the system would navigate to the next item in the list. If the level dropped below the threshold, the system would not navigate further but stay at the currently selected item. If the user makes a deliberate double blink, the system would open the selected link in a new browser tab. H. Recording and using the meditation level We also recorded and plotted the meditation level of the same user simultaneously along with the attention level. The results are plotted and shown over three iterations in Fig. 9. In Fig. 10, we plotted the simultaneous attention and meditation levels of the user when performing the same browsing task. It can be seen that the recorded meditation level is much more variable than the attention level and cannot be controlled easily without some prior training. Therefore, in the current implementaation, we did not use the meditation level for browsing and navigation. VI. ISSUES WITH THE SYSTEM There are a few issues with using the NeuroSky headset to manipulate the EEG values. This is partly because of limitations associated with the EEG technology and partly with the specific sensor kit we are using, manufactured by NeuroSky. In this section we discuss some of them. Limitations associated with EEG include the following: the EEG sensor readings can fluctuate sometimes, it has to be customized for individual users and can only give the attention and concentration level. EEG cannot read thoughts, also it is not possible with most available EEG kits to locate spatial coordinates such as left, right, up, down associated with the thought. Some more advanced commercial kits such as Emotiv Insight and EPOC [2] have a higher number of sensors attached to a number of areas on the head (NeuroSky has just one sensor fitted on the forehead) and can recognize motions such as left or right eye wink, head shaking, teeth clenching and so on, but the current version of the NeuroSky kit cannot. So we have to manage our application as per the functionality available with the current kit. As mentioned previously, the variables that the user is able to modify with the NeuroSky EEG headset include the attention level, concentration level and blink. A certain amount of training is needed to make the user comfortable with manipulating their own attention level as desired. One suggested method on the manufacturer s website is to mentally count back from 100 to 1, which will increase the concentration or attention level. Another issue is overloading of browser functions, for example when the user varies their concentration level and more than one browser function is possible such as move to next item and scroll. To prevent such cases, the event handler has to be modified to have a priority list of events where the EEG input can be applied. Manipulating the meditation level is trickier and the user might need some practice of meditating and playing with the EEG kit, so we did not use the recorded meditation level at this point of time. Also, the threshold of concentration or attention is different for different users and so the system has to be customized for users individually, it is better to keep concentration as a parameter.

7 B. Most visited As mentioned in menu navigation, a similar method is used. When the most visited tab is currently open, the user cycles through a list of most visited sites stored in the web browser by maintaining the attention level above a certain threshold value. Upon a double blink action, the currently selected most visited site is opened in the web browser in a new tab. Fig. 11. Bookmarks section of a web browser in a mobile device. The EEG input at the attention level over a certain threshold enables the user to switch between each of the bookmarks. Upon the double blink by the user, the currently selected bookmark is selected to open. A similar technique is used for the most visited and history sections of the web page. Considering the fact the blink and concentration thresholding varies from user to user and also for different moods of the same user, it might be useful to incorporate machine learning so that the device learns what level of thresholding would be appropriate for a given context and for a given user. Besides, the noise level of the EEG sensor can also make the thresholding seem ad-hoc. Learning would also be helpful to set a threshold that is more robust to noise. VII. BROWSER USER INTERFACES In this section we present the various user interfaces on the web browser side to respond to the EEG inputs. As mentioned in the previous section, only once the user is comfortable with using the kit and the thresholds for attention and blink are customized for the particular user, will these interfaces work as expected. There is a familiarization process required where the user has to play with the browser and learn to modulate their own concentration level with visual feedback, before they can be comfortable using it. The user s EEG concentration level is maintained at a specific value range higher than a threshold in order to move between items. A deliberate double blink action (two blinks within a short time interval) is used for opening the current most visited or favorite page or menu item. This is the general method and as per the context (browser menu, bookmarks etc.) of usage of the web browser, different functionalities are programmed. A. Menu navigation If the user s concentration or attention level is above a certain threshold, the next item in the menu will be navigated. If the user makes a double blink, then the current menu item is selected. This method can be used also for navigating in a list of application icons in a mobile device. C. Favorites or Bookmarks When the favorites tab of the web browser is open, the user can move to the next favorite website by maintaining the concentration or attention level above a certain threshold. Blink is used to select the URL and open the currently selected URL in a browser tab. Fig. 11 illustrates the favorites tab in a mobile web browser. D. Add to favorites When the concentration level is above a certain threshold, the system automatically adds the currently browsed website URL to the favorites. E. History Here, by maintaining the concentration level, the user moves to the next history item. Blink is used to select the URL. F. Scroll Up/Down In this case the user scrolls up or down by keeping their concentration level above a threshold. G. Next/Previous tab Here, if the attention level is more than a first threshold, the user navigates to the next tab in the currently opened tabs. If it is less than a second (lower) threshold, the user can move to the previous tab. H. Menu navigation If the user s concentration or attention level is above a certain threshold, the next item in the menu will be navigated. If the user makes a double blink, then the current menu item is selected. For our implementation, currently we only use the attention and blink levels for navigation in the browser. However, for a better differentiation of user inputs and functions, the meditation level (also detected by the NeuroSky headset as a value between 0 and 100) can additionally be used along with the concentration. This is only possible once the user has trained in meditation and is able to better control the variability in the meditation level. Also, two different thresholds for attention or concentration can be maintained, at a concentration value greater than the first threshold a particular functionality can be invoked on the web browser, while at the concentration value less than a second threshold, a different functionality can similarly be invoked.

8 Fig. 12. Flowchart illustrating the logic on the mobile device in response to inputs received from the EEG sensor By training the user to control their concentration levels and increase and decrease them in a steady manner at will, the user can be trained to control desired functions on the web browser. The flowchart in Fig. 12 provides the flow for invoking functionality in the web browser in response to detected meditation, attention and blink events. VIII. CONCLUSION AND FUTURE WORK In this paper we have presented the design of an EEG based browser. We focused on more simple use cases in the browser using just the concentration and blink levels. In future, we plan to increase the functionalities that the browser can support, and use the meditation level as well as learning to cover an increased number of browser functionalities. Also, we plan to perform the usability study on disabled or paralyzed people, by giving each user a number of common browser tasks (such as navigating to the favorites page or opening a URL from a list of Alexa top 100 websites) and asking them to complete a questionnaire at the end. This would enable us to get a better idea of the usability of such a browser. ACKNOWLEDGEMENT The authors would like to thank Hari Manassery Koduvely for helpful comments during internal reviews of the paper. REFERENCES [1] NeuroSky store: Mindwave [Online]. Available: store.neurosky.com/products/mindwave-1 [2] Emotiv EPOC headset [Online]. Available: emotiv.com/epoc.php [3] MUSE: The Brain Sensing Headband. [Online]. Available: choosemuse.com [4] Melon. [Online]. Available: thinkmelon.com [5] Michael Bensch, Ahmed A. Karim, Jürgen Mellinger, et al., Nessi: An EEG-Controlled Web Browser for Severely Paralyzed Patients, Computational Intelligence and Neuroscience, vol. 2007, Article ID 71863, 5 pages, doi: /2007/71863 [6] Apple Siri [online]. Available: apple.com/ios/siri/ [7] Jason Cross. A list of all the Google Now voice commands. [Online]. Jan 12, Available: greenbot.com/article/ /a-list-of-all-theok-google-voice-commands.html [8] Bao Hong Tan. Using a Low-cost EEG Sensor to Detect Mental States. PhD thesis, Carnegie Mellon University. August URL: andrew.cmu.edu/user/btan/thesis.pdf [9] Ning-Han Liu, Cheng-Yu Chiang and Hsuan-Chin Chu. Recognizing the Degree of Human Attention Using EEG Signals from Mobile Sensors. Sensors 2013, 13, [10] Vourvopoulos, Athanasios, and Fotis Liarokapis. "Brain-controlled NXT Robot: Tele-operating a robot through brain electrical activity." Games and Virtual Worlds for Serious Applications (VS-GAMES), 2011 Third International Conference on. IEEE, [11] W3C media capture and Streams. 5 December [Online]. Available: w3c.github.io/mediacapture-main/getusermedia.html [12] Google Speech API. 3 Feb [Online]. Available: gist.github.com/alotaiba/ [13] How to use Google Speech API. 16 Sep [Online]. Available: x2q.net/2013/09/16/how-to-use-google-speech-api/ [14] Sebastian Germesin. Google Speech API. [Online]. Available: sebastian.germes.in/blog/2011/09/googles-speech-api/ [15] NeuroSky Store. Developer Tools 3: Android. [Online]. Available: store.neurosky.com/products/developer-tools-3-android [16] Android API guide, Building Web Apps in WebView. [Online]. Available: developer.android.com/guide/webapps/webview.html [17] Android Developer Documentation. WebView. [Online]. Available: developer.android.com/reference/android/webkit/webview.html [18] Alex Evanczuk. Openwave Visualizer Code. Dec 31, [Online]. Available: github.com/ /OpenWave/blob/master/src/com/example/OpenWave/Visualizer.java [19] GitHub. GraphView: Chart and Graph library for Android. [Online]. Available: github.com/jjoe64/graphview