EEG/ECG data fusion using Self-Organising Maps

Transcription

1 EEG/ECG data fusion using Self-Organising Maps Nuno Bandeira 1, Victor Sousa Lobo 2,1, Fernando Moura-Pires 1, 1 Computer Science Department, Faculty of Science and Technology / New University of Lisbon, Portugal 2 Portuguese Naval Academy, Lisbon, Portugal Abstract Empirical results are presented concerning data fusion performed over several combinations of EEG/ECG channel readings of sport shooting athletes. Our purpose in applying different data fusion approaches was that of finding a satisfactory set of features, such that would allow us to build adequate classifiers on the data. The resulting data sets were used for building SOMs (Self-Organising Maps), used for visual inspection of coherence between clusters found and shooting accuracy. Keywords: Self-Organising Maps, EEG, data fusion 1. Introduction According to sport shooting experts, the shooter s ability to concentrate on the shooting task is crucial in improving one s performance, once high physical technique levels have been achieved (steady body position, respiration, muscular and eye-movement control). Since concentration is mainly a cerebral activity,we conducted an experiment where EEG and ECG signals were read and digitised in real-time during the shooting activity. Previous work (see [9]) suggested that these could be good indicators of concentration. Once we had all the data (around 80Mb -120 Mb per shooting session), we had to devise adequate pre-processing techniques in order to handle the high volume of data. Many techniques are known for transforming EEG data into feature vectors suitable for clustering and classification [2][10][11][3]. We opted for the use of Fast Fourier Transforms (FFT) as described in the next section. But the best that the FFT could give us were different types of channel spectra, therefore resulting in 20 spectra per shot, one per EEG channel. Since we wanted to apply SOMs to visually inspect potential hidden relations in our data, we also had to find ways of merging all channels into single feature vectors. Different approaches were tried and are described in sections 4 and EEG/ECG signal acquisition and pre-processing The subjects from which we recorded our data are shooters from the sport shooting team of the Portuguese Navy. So far we have recorded data from 7 such shooters, but because of difficulties with the recording software, in this paper we only present the results of shooters numbers 3 to 7. Each shooter spent one morning at a shooting range, firing up to 12 rounds of 5 shots. For each shot, besides the EEG and ECG, we kept the target, and classified the shot according to the score obtained (10 is right on the centre, 0 is outside the target). We then considered that shots with a score of 9 or 10 were good, 7 or 8 were average, and up to 6 were bad. The electrodes were placed according to the standard system [7][8]. The electrode leads are connected to a Braintronics ISO1032 preamplifier, that sends the signal to a Braintronics CONTROL 1032 amplifier. There the signals are amplified, filtered by a 50Hz notch filter and a 4 th order 70Hz low-pass filter. A DATA TRANS- LATION DT2821 ADC board is then used to digitise the resulting signals. The recorded data consists of 22 signals, recorded with 12 bit resolution and a 512Hz sampling rate. Channel 22 is the ECG, from

2 where the heart beat rate is extracted using a simple spectra based algorithm. Channel 18 is the signal of the right ear, that is used as reference for the differential amplifiers, and thus contains no information. The remaining 20 channels are all subject to the same initial preprocessing. First, the last 5 seconds before the shot are selected (2.5K points). This signal is then broken up into 9 blocks of 512 points, with 50 % overlap between them (so as to later obtain a Walsh periodogram). Each of these blocks is then multiplied by a Hamming window to reduce frequency leakage, and it s spectrum is calculated with a 512 point FFT. Thus each channel produces 9 spectra with 256 bins of real frequencies and a width of 1Hz. Since we used a 70Hz low-pass filter and standard EEG bands range from 1-30Hz we opted for using only the lower 30 bins. Thus, when all information is used, we have 20 EEG channels with 9 spectra of 30 bins, totaling 5400 EEG features, plus one heart beat rate feature. All subsequent pre-processing is done on this EEG data. 3. SOM - Self Organising Maps Self-Organising Maps, also known as Kohonen Maps, in honour of its creator, are thoroughly described in [5] and have been widely used in many applications, including as a tool for data fusion [4] (an excellent bibliography can be found in The SOM concept is based on the human brain s cortex interactions, simplified in a model in which different prototypes (neurones) try to represent the input data by competing with each and every other neurone in every iteration, for a better mapping of the input data. The basic SOM algorithmic procedure is as follows: 1. For a given training pattern x: 1.1 Calculate the distance of each neurone to the training pattern x (Calculation phase) 1.2 Find the neurone with smaller distance, and call it the winner W (Voting phase) 1.3 Change the network neurones with a function G, which depends on the learning rate α, the distance d to W (in the output plane), and the neighbourhood function F. Due to the nature of the neighbourhood function, only the neurones closer to W (in the output space) will be changed. (Update phase) 2. Update the learning rate α and the neighbourhood function F according to some rule 3. Repeat steps 1 and 2 for the next training pattern, until some stopping criteria is reached. In all our analysis we ran our algorithms 6 times with different initial values, to make sure that the process always converged to the same final map. Whenever this did not happen, we simply increased the number of iterations and the initial learning radius until a stable solution was found. A distributed SOM implementation was also used in building the maps for the largest dataset (5401 features). A detailed description of this algorithm and its empirical evaluation can be found in [1]. To visualise the results of the clustering performed by SOM, we frequently used U- matrices [12]. The U-Matrix of a SOM is obtained by calculating the distance, in the input space, between neighbouring neurones. These distances are then represented on a map in grayscale (black being the greatest distance, and white the smallest). Clusters can easily be identified as clear areas (nearby neurones) separated by dark ridges (large distances to other clusters). 4. Data fusion Since our main objective was finding an adequate set of features that would provide high visual correlation between the EEG signal and shooting performance, most of our work at this stage was concentrated on data fusion. Due to this fact, data fusion was performed using different types of feature aggregation, motivated by several different reasons. Each choice of feature aggregation led to a different training set, upon which the clustering procedure was applied. The training sets used were: I) All the features (1 set of 5401 features). In this training set, all features mentioned in Section 2 are used. It seems reasonable that this approach would capture the dynamics of the signal prior to the shot. In this set, the heart beat rate is also used. II) All average spectra. (1 set of 601 features).

3 In this training set, we averaged the spectra of each channel, and by doing so assumed that the signals are stationary during the 5 seconds before the shot. Each resulting spectrum is the average of 9 spectra, and thus the signal to noise ratio is improved considerably. In this set, the heart beat rate is also used. III) Average spectra separated by hemisphere (2 sets of 330 features). In these training sets, we separated the data in right and left hemisphere. Each hemisphere consists of 8 EEG channels unique to that hemisphere, plus the three central channels (Fz, Cz, Pz [7][8]). This choice of features is motivated by the fact that the left and right sides of the brain are reasonably distinct and all but one of the shooters were right handed and used the right eye for aiming. We performed clustering on each side separately, and later merged the results. IV) Average spectra by channel (19 sets of 30 features). In these training sets, we used the spectra of each channel as a separate training set. By analysing the ability to cluster the data sensibly, based on each channel independently, we tried to determine if there were any channels more relevant to the task at hand. V) Characteristic frequency bands (4 sets of 120 features for the alpha band, 320 for beta, and 80 for delta and theta). In these training sets, power spectra within each band (alpha, beta, delta, and theta [6]) were selected for all channels. Since each band has a different width, the number of features selected varies. According to classic literature in the area [6] these frequency bands correspond to well established activity patterns within the brain, and thus are the natural choice for discriminating between the shots. 5. Decision fusion Decision fusion was used for merging the results obtained in III and IV. The generated datasets were used in building the corresponding SOMs, which were then labelled using the same datasets. Labelling a SOM consists in finding the winning neurone in a SOM for each data vector in the dataset and appending the data vector s class label to the neurone s label. This labelling (usually called calibration in SOM terms) allowed us to use the SOMs as classifiers, simply by having each neurone belong to the class that, in its label, is most frequent. Two different strategies were applied in fusing the SOMs classifications: - Majority (III, IV). This is the simplest decision fusion, where the final class is simply the most ocurring class in the lower level classifiers. It is used for evaluation of variation in SOMs classifications. - Use of another SOM layer (IV). In this case, the results of the classification by the original SOMs are fed as features to a fusing SOM. It is then used for visual inspection of dispersion of the first level SOMs classifications. Higher levels of agreement on the first level SOMs should lead to a smooth fusion SOM. If the decision fusion SOM is messy or has outliers, then there is disagreement on the first level SOMs. In such cases, simply by glancing at the outliers neighbours it s easy to spot which class most of the first level classifiers chose for it. 6. Results With all datasets, with the exceptions of VI that will be discussed later, and certain channels of IV, the data was clustered by shooter. We present these results for the first dataset in Figure 2, and the others are very similar. Since the data is clustered by shooter, it cannot be clustered by score. So as to cluster by score, it is necessary to join all good shots in one cluster and bad ones in another, thus mixing in those clusters the different shooters (which as in this case does not happen). Thus, to classify the shots by score we have to analyse the data of each shooter individually. None of the datasets tested provides good clustering by score for all shooters. However, 2 of the individual channels (F7 and T3) provided reasonable clustering by scores, even when all shooters are considered simultaneously. Furthermore, some of the shooters have their shots clustered by score with some of the datasets. Shooter 3 has his shots clustered by score in dataset IV (channel Cz), shooter 4 in dataset V, shooter 5 in datasets II, IV and V, and shooter 7 in dataset IV. With shooter 6 no

4 dataset was capable of clustering his shots by score. To visualise the maps produced, we shall represent the mapped shots as crosses if they correspond to good shots, triangles if they correspond to average shots, and circles otherwise, as shown in Figure 1. are probably outliers that correspond to lucky shots that are good despite the bad conditions. Figure 1 - Legend for the maps The results for each of the training sets presented in section 4 are as follows: I) Training set with all 5401 features. Different shooters are clearly identified for, as we can see in Figure 2, shooters 4 and 5 have very distinct clusters (separated from the others by dark lines in the U-Matrix), and shooters 6, 7, and 3, while in the same cluster, are mapped to different areas. To obtain these maps we used the distributed version of SOM mentioned in section 3. This allowed us to reduce the total training time of each map from 2h21m to 1h16m when using 2 machines, and even more when more machines where available Figure 3 - Map obtained with training set II for shooter Figure 4 - U-Matrix obtained with training set II for shooter Figure 2 - U-Matrix obtained after applying a SOM to training set I. If we train SOM maps for each individual shooter, the results are generally bad. II) Training set with Average Spectra (601 features) This training set is only useful for shooter 5. In Figure 3 we can see that all good shots are on the upper right corner, while the average shots are on the bottom left. Furthermore, in the U-Matrix presented in Figure 4 we can see that there is a clear distinction between these two areas. It could be argued that there are some good shots in the bad area, but these III) Training set with average spectra separated by hemisphere (2 sets of 330 features). We were unable to obtain good clusters of the data by scores for any shooter with this dataset. However, as with all others, we could cluster rather easily by shooter. When we fused the results obtained by each hemisphere, we managed to obtain the results presented in Table All Best Hemisphere Fusion Gain Table 1 - Percentage of correct classification for each shooter, with and without decision fusion IV) Training set with average spectra by channel (19 sets of 30 features).

5 IV.i) Individual Channels Channels F7 (left frontal), and T3 (left temporal) proved to be quite good at clustering by score. Figure 5 shows the results for all shooters, and we can see an area of bad shots on the left center, and an area of confusion on the lower right, with good shots on most other areas. Figure 7 - Map obtained with training set IV, channel Fz, for shooter 7 Figure 5 - Map obtained with training set IV, channel T3, for all shooters. When considering individual shooters, shooter 5 again has his shots clustered by score, but now we can also do the same for shooter 7 and 3, as can be seen in Figure 6 and Figure 7. IV.ii) Fusion by majority As can be seen in Table 2, the biggest improvements were achieved with shooter 4 and all shooters together, with an increase of 18% and 17% of correct answers. Average improvement was 11% ALL Best channel Fusion Gain Table 2 - Percentage of correct classifications for each shooter, with and without decision fusion IV.iii) Fusion by another SOM layer As expected from shooter 4 s low error rate in fusion by majority, his map is pretty clean, having only one average shot amidst the bad shots, as can be seen in Figure 8. Figure 6 - Map obtained with training set IV, channel Cz, for shooter 3. Figure 8 Results of fusion of the individual channels by a SOM, for shooter 4.

6 shooter 5 s shots by score, as can be seen in Figure 11. Figure 9 - Results of fusion with a SOM layer for shooter 6. Figure 10 - Results of fusion with a SOM layer for shooter 3. On the other hand, the maps of Figure 9 and Figure 10 show that shooters 3 and 6 are in very distinct situations, although their results in fusion by majority are quite similar. In shooter 6 s map there are many neurones that have both good and average class labels, indicating that this set of features is not good enough. In shooter 3 s map, we can observe that this choice of features led to an embedding of the average shots amidst the good ones (in the SOM s 2D projection). This leads to the conclusion that, for the data vectors in the bottom left corner, most of the single-channel SOMs classified these good shots as below average. So in this case, what we have is not a mix-up, but a set of data vectors that should be put aside and carefully analysed. A possible solution to this kind of error could be the addition of another classifier, prior to decision fusion, that would mainly handle this cluster. V) Training sets with characteristic frequency bands (alpha beta, delta and theta). The alpha band, separated the shooters better then most others, with the advantage that it uses only 6 features. It was also useful in separating Figure 11 - Map obtained with training set V, Alpha band, for shooter 5. The beta band provided the best separation of all amongst shooters. It was however useless in separating by score, as was the theta band. The delta band provided a reasonable clustering of shooter 4 s scores, as can be seen in Figure 12. Figure 12 - Map obtained with training set V, delta band, for shooter Conclusion Our main objective in this first phase of our work was to gain further insight over our data, which was accomplished. Although most of the data fusion possibilities had a strong imprint of each individual s personal EEG traces, we were able to find some clues as to what were the important frequency ranges in each case. These have already led us to try and find new criteria that combine different frequency bands, on which we are currently working. The data from the ECG did not influence in any way our results, since it was almost

7 constant for each shooter, and even amongst shooters the differences were not significant. Our use of SOMs as a clustering tool for visual inspection of our data fusion options was, as we expected, very useful. Also, decision fusion wise, we found SOMs to be very useful as a visual inspection tool of a set of classifiers. High coherence means cleaner, smoother maps, messy maps mean lots of variance within the classifier set and maps with disjoint clusters could be good indicators that an extra classifier is needed to handle specific data partitions. 7. Bibliography [1] Bandeira N, Lobo V, Moura-Pires F: Training a Self-Organizing Map Distributed on a PVM Network, Proceedings of IEEE Joint Conference on Neural Networks, 1998 [2] John ER, Prichep LS : Principles of Neurometric analysis of EEG and Evoked Potentials, Electroencephalography, 4th ed (Niedermeyer E, ed), Williams & Wilkins, 1993, [9] Schober F, Schellenberg R, Dimpfel W : Reflection of mental exercise in the dynamic quantitative topographical EEG, Neuropsychobiology 1995;31; [10] Silva FL : EEG analysis: Theory and Practice, Electroencephalography, 4th ed (Niedermeyer E, ed), Williams & Wilkins, 1993, [11] Silva FL : Computer-assisted EEG diagnosis: Pattern recognition and brain mapping, in Electroencephalography, 4th ed (Niedermeyer E, ed), Williams & Wilkins, 1993, [12] Ultsch A: Siemon, H.P.; Exploratory Data Analysis Using Kohonen Networks on Transputers, Dep.of Comp.Science, Dortmund FRG, December [3] Kalayci T, Özdamar Ö : Wavelet preprocessing for automated neural network detection of EEG spikes, IEEE Engineering in Medicine and Biology, March/April 1995, [4] Lobo VS, Bandeira N, Moura-Pires F: Distributed Kohonen Networks for Passive Sonar Based Classification, Proceedings of the International Conference on Multisource- Multisensor Information Fusion FUSION 98 [5] Kohonen T : Self-Organizing Maps, Springer-Verlag, 1995 [6] Niedermeyer E: The Normal EEG of the Waking Adult, in Electroencephalography, 4th ed (Niedermeyer E, ed), Williams & Wilkins, 1993, [7] Nuwer MR, et al.: IFCN guidelines for topographic and frequency analysis of EEGs and EPs. Report of an IFCM committee. [8] Reilly E: EEG Recording and Operation of the Apparatus, in Electroencephalography, 4th ed (Niedermeyer E, ed), Williams & Wilkins, 1993,