Brain-Machine Interface Based on EEG Mapping to Control an Assistive Robotic Arm

Transcription

1 The Fourth IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics Roma, Italy. June 24-27, 2012 Brain-Machine Interface Based on EEG Mapping to Control an Assistive Robotic Arm Andrés Úbeda, José M. Azorín, Nicolás García, José M. Sabater, Carlos Pérez Abstract In this paper, a non-invasive spontaneous brainmachine interface (BMI) based on the correlation of EEG maps has been used to control 2D movements of an assistive pneumatic planar robot. The main goal of the system is to assist disabled people in pick and place tasks. The BMI has been tested in order to check the accuracy of the system. To that end, several planar movements between different positions have been performed. The control of the 2D movement is performed by using a hierarchical control where the user has to first choose the axis and then decide the movement direction. All the commands are generated using the spontaneous BMI. The results obtained show a very high reliability on the positioning and indicate that this control can be very useful in future assistive applications for disabled users. Further research will be centered in performing pick and place operations with daily objects using a pneumatic gripper attached at the end effector of the planar robot. I. INTRODUCTION A Brain-Machine Interface (BMI) registers the bioelectrical activity of the brain in order to control external devices, such as a computer or a robot [1]. BMIs are an alternative to classical methods of human-machine communication like a keyboard or a mouse. In this sense, this kind of interfaces, alone or in combination with others, e.g. electrooculography [2], [3] or videooculography [4], are very useful for people with a severe motor disability [5]-[7]. In BMIs, brain signals are registered through invasive and non-invasive techniques. In invasive BCIs, the activity of a neuron or small groups of them is registered using microelectrodes implanted directly in the brain. Invasive techniques have been used in animals [8], [9] and humans [10]. Nevertheless, the use of non-invasive techniques is more suitable when working with humans because of ethical implications and to avoid medical risks. In this case, several electrodes are placed over the scalp with no harm for the user obtaining the electroencephalographic (EEG) signals [11]. In this paper, a non-invasive motor imagery spontaneous BMI based on the correlation of EEG maps is going to be used to control a pneumatic planar robot arm in an assistive application. EEG mapping has been mainly used related to clinical diagnosis of mental diseases whose origin is located in EEG alterations such as epilepsy [12] or schizophrenia [13]. This technique consists of obtaining a visual plotting of the activity of the brain (usually in terms of frequency), This work has been supported by the Ministerio de Economa y Competitividad of the Spanish Government through project DPI C Andrés Úbeda, José M. Azorín, Nicolás García, José M. Sabater and Carlos Pérez are with Virtual Reality and Robotics Lab, University Miguel Hernández, Elche, Spain aubeda@umh.es, jm.azorin@umh.es, nicolas.garcia@umh.es jm.sabater@umh.es and carlos.perez@umh.es which is a more representative method for determining EEG alterations. However, the use of EEG mapping in real-time applications has been little explored. In this sense, it has been previously proved that the use of a normalized crosscorrelation between EEG maps is a quite accurate way to classify mental tasks [14]. In this paper, this classifier is going to be used to differentiate between two mental tasks related to motor imagery. The application designed consists of moving the end effector of the robot in a plane in order to reach an objective. The movement of the robot is based on a hierarchical control. The user sits in front of a screen located in the robot environment. A visual interface shows the different movement options. Using the BMI, the user has to first choose the axis and then he/she must decide the movement direction. The planar robot moves a predefined distance to the corresponding direction. This control loop continues until the goal is achieved. The results obtained show that the user reaches the goal with a quite high reliability. This means that the system is ready to perform more complex tasks, such as pick and place operations of daily objects. To that end, a pneumatic gripper has been attached at the end effector for future experiments. These experiments will be aimed at showing the usefulness of this BCI approach in assistive robotics and daily environments. The remainder of the paper is organized as follows. Section II, Material and Methods, explains the system architecture and the control protocol used to perform bidimensional movements with the pneumatic planar robot. It also describes the classifier based on EEG mapping correlation. In section III, the results obtained with the application designed to control the planar robot are shown and discussed. Finally, Section IV contains the conclusions. II. MATERIAL AND METHODS A. EEG signals register and processing To register the EEG signals, the gusbamp device of g.tec has been used. This device has 16 channels. The signal is registered with a sample frequency of 1200 Hz and two filters are applied: a bandpass filter between 0.1 and 100 Hz, and a Notch filter of 50 Hz, to eliminate the network noise. These filters are internally included in the device. The software used for registering the EEG signals has been developed in Matlab using the API (Application Programming Interface) provided by the device (gusbamp MATLAB API). Two different mental tasks related to motor imagery are going to be classified: /12/$ IEEE 1311

2 1) Imagination of low circular movements of the left arm ( left mental task). 2) Imagination of low circular movements of the right arm ( right mental task). The selection of the electrodes over the scalp is based on an extension of the International System 10/20. As previous studies indicate, the imagination of a movement generates the same mental process as the performance of the movement itself [15]. Based on this fact, the measured electrodes be mainly located on the motor cortex, which is the area of motor activation (Figure 1). Seven electrodes have been chosen to register the EEG signals: FC1, FC2, C3, Cz, C4, CP1 and CP2. This is an improvement of previous configurations, like the one selected in [16], where 16 electrodes were used. The main goal of this reduction is to obtain a more specific area of the motor cortex activity to improve the classification. Fig. 1. Electrodes location on the motor cortex. Fig. 2. Example of EEG map. The scale is normalized between 0 and 1 as it can be seen on the scale bar. Each electrode is placed in its particular position and the value generates the map. After registering the EEG signals, a Laplacian smoothing filter is applied as it is shown in [17]. It consists of subtracting, on each electrode, the contribution of the surrounding electrodes by taking into account the distance to the main electrode: where: V LAP i = V ER i j S i g ij V ER j (1) g ij = (1/d ij )/( j S i 1/d ij ) (2) According to this formulation, S i is the set of electrodes that surround the main electrode (in our case, all the surrounding electrodes in the selected set), while d ij is the distance between the main electrode i to the surrounding electrodes j. The EEG signals are processed with a particular overlap (20 times per second) in windows of 1 second each. Afterwards, the extraction of the features of the EEG signals is done using the Fast Fourier Transform (FFT), which decomposes the input signal into different frequencies. The considered range for EEG signals is between 8 and 30 Hz with a resolution of 2 Hz. This means that for each electrode, 12 frequency features will be obtained centered in the selected frequencies. B. EEG Mapping Classifier The classifier is based on the comparison of EEG maps. The EEG maps used as models for each task are obtained and adjusted from register sessions and then used in realtime for classification. The EEG maps of the EEG signals registered have been plotted using Matlab. To that end, only one particular frequency can be represented. With the 7 frequency features (one for each electrode), a geometrical grid of 99x99 pixels is created by interpolating from the real frequency value of the electrodes and using the real position of each electrode. As a result, an example of a EEG map obtained can be seen in Figure 2. The axes show the position of the electrodes in a bidimensional space. The color map represents the value in frequency scaled between 0 and 1 to normalize the difference between each electrode. To obtain the models for classification, the registered data will be used obtaining an average EEG map for each task and frequency. These models will be tested in classification to obtain the most suitable one. It is expected that the best results will be obtained in the so-called Mu band (8-12 Hz), where motor activity is mainly produced. The classifier used in the BMI is based on a normalized cross-correlation [18] between the models obtained for each task (left and right) and the data tested. This method has been proved to be very accurate in previous works [14]. It is an improved technique that solves the problems of shape mismatching of the classical cross-correlation. In this case, the position and size of the shapes obtained in each EEG map can change, so it is necessary to perform a correlation that ignores these changes. When a particular session of data is tested, each trial is compared with the models using the normalized cross-correlation algorithm. After this comparison, an index of correlation for each task is obtained. The maximum value of the index is selected obtaining the 1312

3 corresponding task. Afterwards, two uncertainty conditions are evaluated: The first uncertainty condition is applied by introducing a fixed threshold to both indexes of correlation (left and right). If none of the models fulfil this condition, the trial is rejected as uncertainty. This uncertainty condition prevents wrong classifications due to noise or bad performance of the amplifier. If the index of correlation does not exceed a 90% the trial is rejected. The second uncertainty condition is applied by introducing a threshold between both indexes of correlation (left and right). If both are too similar, the trial is rejected as uncertainty. This uncertainty threshold is selected when the uncertainty rate is the same as the error rate after performing a cross-validation between sessions. This method seeks the same proportion between error and uncertainty. C. Planar Robot The planar robot used during the experiments is the PuParm, a force-controlled planar robot designed and developed by the nbio research group at the Miguel Hernández University of Elche. A pneumatic swivel module with angular displacement encoder (DSMI A-B of Festo) has been used as actuator for each two joints. This kind of actuators can exert enough driving power despite being lightweight and having a small size because the ratio of its output power to its weight is large. The semi-rotative drives are controlled by two proportional pressure valves (MPPE manufactured by Festo) to achieve a maximum torque of 5 Nm at 6 bar and a maximum swivel angle of 270. The valve MPPE is designed so that pressure output is proportional to voltage input through a proportional electromagnet. With this configuration (two proportional valves and a pneumatic actuator), the pressure of the two chambers of the pneumatic drive can be regulated to get a desired output force. The core of the control system is a motion controller board (DMC-40) manufactured by Galil. It operates stand- alone or interfaces to a PC with Ethernet 10/100Base-T or RS232. The controller includes optically isolated I/O, high-power outputs capable of driving brakes or relays, and analog inputs for interfacing to analog sensors. Four analog outputs from the DMC-40 board are used to control each pneumatic actuator through two proportional pressure valves. An electronic board, called distributor, has been designed to convert each joints control signal in two voltage inputs for its respective proportional pressure valves (it is assumed that the valves behavior is identical). D. Control Protocol and Application The experimental environment can be seen in Figure 3. The user sits in front of the planar robot and the screen. The PuParm can be moved over the table and it is controlled in position. The application designed consists of positioning the end effector of the planar robot over a particular goal. To that end, the user is able to move to any of the four possible directions (up, down, left or right) a predefined distance of 10 cm. The decisions are sent via UDP to the computer that controls the planar robot and the robot is controlled via USB. The control application is a Simulink scheme which receives the high-level position commands and translates them into cinematics commands to move the planar robot. Each decision is taken using the BMI following a hierarchical control protocol (Figure 4). Two decision menus are shown to the user on the screen. With the first one, the axis of movement is chosen. Afterwards, another menu asks for the direction of the movement depending on the axis previously selected. The visual interface shows a cursor that is moved with the BMI (left or right) to select one of the options. The minimum time taken to select one of the options is around 5.4 seconds, so the minimum time taken to perform a movement is around 10.8 seconds. This has been done to prevent errors on the selection as the success rate of the BMI is quite high but it is still not perfectly accurate. Fig. 3. Experimental environment where the user performs the tests. Fig. 4. Hierarchical Control Protocol. The user takes two decisions to perform a single movement of the planar robot. As it can be seen in Figure 3, several objectives are marked on a table. These positions can be reached using the planar robot controlled by the BMI. The end effector of the PuParm is placed over the start position (the one shown in the image) and then the user takes control of the robot and moves it to the desired position. This is an initial approach to more 1313

4 complex tasks taken with this planar robot. In the future, a pneumatic gripper will be used at the end effector to perform pick and place tasks of daily objects in a more realistic environment (Figure 5). This additional degree of freedom can be solved by including a third hierarchical decision or by implementing alternative control protocols or introducing a multimodal approach, e.g., including the blink of the user. As it has been mentioned, the user sits in front of the planar robot and the visual interface (Figure 3). Three objectives tagged with a number are placed on the table covering the robot workspace. The user must reach these objectives by controlling the end effector movement with the hierarchical BMI. The whole experiment is repeated 3 times. In Figure 6, a view of three perfect trajectories is shown. The minimum movement decisions taken to reach the objectives 1, 2 and 3 are 3, 4 and 6 respectively. In Table I, the results obtained are shown. Time taken to reach each objective is measured, as well as decision errors made on each trajectory along with the total movement decisions made to reach the objective. The average time taken in each decision (movement) is also shown. Fig. 5. Planar robot with gripper for assistive applications Fig. 6. Example of trajectories (Planar view). III. RESULTS One able-bodied volunteer took part in the experiments. The volunteer was a man, right-handed, who had previous experience using BMIs. Before performing the final application, the protocol to adjust the classifier followed several steps: 1) Five sessions of offline registers are done and combined to obtain the EEG maps models for each mental task (left and right) and frequency. 2) A set of three online sessions (with visual feedback) is registered to test the models. Models with the frequency that obtains a higher success rate are selected and updated. 3) Afterwards, sets of three online sessions are repeated using the models obtained with the frequency selected to improve even more the classification. For each set, the models are updated again. 4) When the success rate stops improving, the final models are saved to be used in the real-time application. The results showed an average success rate of 81.60% before introducing uncertainty. With uncertainty, the success rate was 76.25% and error rate was only 13.12%. These results suggested the classifier would behave accurately in the planar robot application, so the models obtained from these sessions were selected. TABLE I EXPERIMENTAL RESULTS: ERRORS AND TIME (SECONDS) TAKEN ON EACH DECISION AND FINAL TIME (SECONDS) OF EACH TRAJECTORY. Target Test Errors/Decisions Decision time Test time 1 0/3 11,7 76, /3 16,4 98,7 3 1/5 9,2 93,2 12,4 ± 3,7 89,5 ± 11,4 1 1/6 9,1 110, /6 9,6 115,7 3 2/8 8,8 142,2 9,2 ± 0,4 122,7 ± 17,1 1 1/8 6,4 103, /6 6,8 86,2 3 0/6 10,7 129,7 8 ± 2,4 106,4 ± 21,9 The results obtained in terms of decision success are significantly high (88.2 %), as the user generally takes no more than one wrong movement decision in a single trajectory (except from one case). Any wrong decision needs to be solved in the following movement affecting the final time taken to reach the objective. However, the introduction of the hierarchical control solves the lower reliability of a pure BMI and allows the user to achieve every objective in a quite reasonable time. The average time taken to reach the furthest goal (objective 3) is seconds. Contrary to expectations, the second goal takes more time than the third (122.7 seconds). This is due to the hierarchical control 1314

5 protocol, where the decisions taken to achieve this objective imply a greater number of mental task changes. This has been proved to be more difficult for the user. It is also interesting to note that in all nine cases of these experimental tests, the user is capable of reaching the objectives and the average time to take a decision (9.9 seconds) is not too far from the minimum possible time (5.4 seconds) considering a perfect classification, which makes this system very reliable. IV. CONCLUSION In this paper, a non-invasive motor imagery spontaneous BMI based on the correlation of EEG maps has been used to control a pneumatic planar robot to perform 2D movements. To that end, a hierarchical control protocol where the user decides the axis and direction of the movement has been proposed. Three different objectives where placed over a table and the planar robot was moved to reach each position. The results show a high success rate when taking each movement decision and a reasonable time to reach each objective. These findings confirm that the EEG mapping correlation classifier implemented can be successfully used to perform 2D trajectories with a robot, which could be very useful in more complex assistive applications. In future works, different control protocols will be tested and compared with the present hierarchical approach and the application will be used by a greater number of volunteers. Moreover, a pneumatic gripper has been attached to the end effector of the planar robot. In this sense, more complex applications to pick and place objects in a realistic environment will be designed to show the usefulness of the whole architecture as an assistive technology. [9] J. K. Chapin, K. A. Moxon, R. S. Markowitz and M. A. L. Nicolelis, Real-Time Control of a Robot Arm using Simultaneously Recorded Neurons in the Motor Cortex, Nature Neuroscience, vol. 2, pp , [10] M. D. Serruya, N. G. Harsopoulos, L. Paninski, M. R. Fellows and K. Donoghue, Instant Neural Control of a Movement Signal, Nature, vol. 416, pp , [11] J. R. Millán, P. W. Ferrez and A. Buttfield, Non Invasive Brain- Machine Interfaces - Final Report, IDIAP Research Institute - ESA, [12] M. V. Sebastián, M.A. Navascués and J.R. Valdizán, Surface Laplacian and Fractal Brain Mapping, Journal of Computational and Applied Mathematics, vol. 189, pp , [13] M. T. H. Wong and F. Lieh-Mak, Topographic Brain Mapping of EEG and Evoked Potentials in Chinese Normal and Psychiatric Patients - Preliminary Findings, J.H.K.C. Psych., vol. 1, pp. 6-11, [14] A. Úbeda, E. Iáñez and J.M. Azorín, Mental Tasks Classification for BCI Using Image Correlation, International IEEE EMBS Conference, pp , [15] J. Decety and M. Lindgren, Sensation of Effort and Duration of Mentaly Executed Actions, Scandinavian Journal of Psychology, vol. 32, pp , [16] E. Iáñez, J. M. Azorín, A. Úbeda, J. M. Ferrández and E. Fernández, Mental Tasks-Based Brain Robot Interface, Robotics and Autonomous Systems, vol. 58(12), pp , [17] McFarland, D.J. et al., Spatial filter selection for EEG-based communication, Electroencephalography and clinical neurophysiology, vol. 103(3), pp , [18] J. P. Lewis, Fast Normalized Cross-Correlation, Industrial Light and Magic, REFERENCES [1] M. A. L. Nicolelis, Actions from Thoughts, Nature, vol. 409, pp , [2] Y. Punsawad, Y. Wongsawat and M. Parnichkun, Hybrid EEG-EOG Brain-Computer Interface System for Practical Machine Control, International Conference of Engineering in Medicine and Biology Society (EMBC), pp , [3] A. Úbeda, E. Iañez and J. M Azorín, Wireless and Portable EOG- Based Interface for Assisting Disabled People, IEEE/ASME Transactions on Mechatronics, vol. 16(5), pp , [4] E. C. Lee, J. C. Woob, J. H. Kim, M. Whang and K. R. Park, A BrainComputer Interface Method Combined with Eye Tracking for 3D Interaction, Journal of Neuroscience Methods, vol. 190, pp , [5] X. Gao, X. Dignfeng, M. Cheng and S. Gao, A BCI-based Environmental Controller for the Motion-Disabled, IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 11, pp , [6] W.-K. Tam, K.-Y. Tong, F. Meng and S. Gao, A Minimal Set of Electrodes for Motor Imagery BCI to Control an Assistive Device in Chronic Stroke Subjects: A Multi-Session Study, IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 19(6), pp , [7] F. Galán, M. Nuttin, E. Lew, P.W. Ferrez, G. Vanacker, J. Philips and J. del R. Millán, A Brain-Actuated Wheelchair: Asynchronous and Non-Invasive Brain-Computer Interfaces for Continuous Control of Robots, Clinical Neurophysiology, vol. 119, pp , [8] J. M. Carmena et al., Learning to Control a Brain-Machine Interface for Reaching and Grasping by Primates, PloS Biol., vol. 1, pp ,