Common Spatio-Temporal Patterns in Single-Trial Detection of Event-Related Potential for Rapid Image Triage

Transcription

1 Common Spatio-Temporal Patterns in Single-Trial Detection of Event-Related Potential for Rapid Image Triage Ke Yu 1, Kaiquan Shen 1, Shiyun Shao 1, Kenneth Kwok 2 and Xiaoping Li 1,* 1 National University of Singapore, Singapore Temasek Laboratories, National University of Singapore, Singapore * mpelixp@nus.edu.sg ABSTRACT Searching for targets in large volume imagery is a challenging problem. A potentially promising solution, rapid image triage (RAIT), is essentially a cortically-coupled computer vision technique based on single-trial detection of event-related potentials (ERP). This paper presents a novel feature selection method, termed common spatio-temporal patterns (CSTP), which is critical for robust single-trial detection of ERP. Unlike past methods such as conventional common spatial pattern (CSP) method whereby only spatial patterns of ERP are considered, the proposed method exploits both spatial and temporal patterns of ERP, providing complementary spatial and temporal features for high accurate single-trial ERP detection. Numerical study using data collected from 20 subjects demonstrates that the proposed method offers significant performance improvement over conventional CSP method (p-value < 0.01) for rapid image triage. Keywords: single-trial, common spatial-temporal pattern, rapid image triage. INTRODUCTION Like in many other process optimization problems, triage techniques can be extremely effective to improve the efficiency of target searching in large volume imagery. Recently, some pilot work has been done to explore the feasibility of neurophysiologically-driven rapid image triage methods that leverage split-second human perceptual judgement capability [1, 2, 3]. In these RAIT methods, the user, an image analyst or trained personnel, is presented, using rapid serial visual presentation (RSVP) paradigm (which is essentially a visual oddball paradigm [1, 2, 3]), a sequence of image chips, some of which contain targets to be identified. Then, an unique brain signal recorded non-invasively from the scalp known as event-related potential, represented by P300, a large positive voltage deflection (5 µv or greater) occurring at approximately 300 ms after the onset of a target image chip, is used to determine whether or not the user sees a target image chip within the high speed sequences of image chips via RSVP. It is worth noting that RAIT is a fast, preliminarily step to identify suspicious images of high target likelihood based on single-trial ERP detection; the resulting suspicious images are then subject to a follow-up detailed analysis for detection of true target images. Typically, image chips are presented to the user at a speed of 6 15 images per second, which represents about a ten-fold speed up in target searching [1, 2, 3]. In those pilot studies, a trainable classifier was used to learn the ERP pattern in relation to brain responses to target images so as to classify automatically whether the user sees a target or a non-target image. Such a cortically-coupled computer vision technique offers a promising solution for performance augmentation in image analysis. Unfortunately, the challenging problem of the single-trial ERP detection has not been carefully addressed and therefore very little evidence exists on the efficacy of incorporating the single-trial ERP detection technique into an effective RAIT system. In paticular, though surprisingly, no feature extraction was done and only the raw electroencephalogram(eeg) data, congregated from all channels, were to form the feature vector subjected to the classifier. This resulted in an extremely sparse problem: there were thousands of features but only very limited training samples. For such a sparse problem, it is well

2 known in the domain of machine learning that, if no dimensionality reduction is provided, a standard classifier will certainly perform poorly due to the limitation called curse of dimensionality [4]. the problem of single-trial ERP detection (target v.s. non-target) in the context of RAIT is typically a highly unbalanced problem due to the fact that, in a large volume imagery, only very few images may contain targets to be identified. For such an unbalanced problem, a standard classifier used in the past work would likely fail to perform satisfactorily. This paper proposes a novel feature extraction method for robust single-trial ERP detection. By searching for both spatial and temporal projections of multi-channel ERP signals, very small number of tempo-spatial features that provide the best separation between ERPs induced by target and non-target images can be extracted to aid the classification. Though the proposed method is accomplished through simultaneous digitalization of two covariance matrices, the same technique used in conventional CSP method [5, 6], it defers significantly from the CSP because the latter only considers the spatial patterns while ignoring the temporal patterns of ERP. In fact, in the case of rapid image triage, the temporal patterns of ERP are the most prominent features because the topography of ERPs induced by target images are dominated by P300: its slow cortical potential component that can only be defined in the sense of temporal patterns. For comparison, the performances of proposed method and the conventional CSP method are both reported. EXPERIMENTAL METHOD RAIT experiments were done by using a self-developed rapid image triage system as shown in Fig. 1. Twenty right-handed subjects (21-30 years old, 15 males and 5 females) were recruited from local tertiary institutions who fulfilled the inclusion criteria of not being on any medication, having no history of neurological or psychiatric problems, with normal or corrected-tonormal sight. They participated in RAIT experiments on real satellite images provided by Singapore Armed Forces. Recruitment of human subjects for this study was reviewed and approved by the National University of Singapore Institutional Review Board (NUS-IRB). Each subject underwent a 2-h RAIT experiment in a temperature-controlled laboratory in an ambient of silence. Prior to each experiment, a detailed orientation was given to the subject by the operator. Each experiment consisted of one training session and one test session and there was a 10-min break in between sessions. Following a standard RSVP paradigm [7], in both training session and test session, subjects were pre- Figure 1: The self-developed rapid image triage system which comprised of a 64-channel ANT R amplifier (ANT B.V., Enschede, Netherlands) and a laptop running Presentation R stimulus delivery and experimental control program (Neurobehavioral Systems Inc., Albany, USA) and a customized labview program with real-time capability for single-trial ERP detection. sented, using Presentation R (Neurobehavioral Systems Inc., Albany, USA), bursts of images via an LCD screen, each having 50 images in a sequence with each image showing on the screen for about 100 milliseconds. Every burst was separated by a fixation screen (a black screen with a fixation cross in the center) for a subject-controlled duration (up to 10 seconds) to break the monotony and to minimize possible eye strain. This alternation of image bursts and fixation screens lasted for about 20 mins. Both training session and test session consisted of two of such alternating series, separated by a 5-min break in between. The images shown consisted of both non-target images and small amount of target images, according to a pre-defined target concept. For this experiment, images shown in RSVP were grayscale image chips ( pixels) chopped from broad-area satellite imagery following the raster scan order, some of which contained pre-defined targets, e.g. oil tanks in the case of present study (see Fig.2). Subjects were required to voluntarily respond to target images by pressing a button. In both training session and test session, the subjects EEG signals were recorded referentially using a 64-channel ANT amplifier (ANT B.B., Enschede, Netherlands) at 250 Hz, with the reference set to the linked ears and the grounding electrode on the forehead. As the time stamp of the onset of each image displayed to the subjects, digital event triggers sent over a parallel port by Presentation R, the EEG signals were then segmented into single-trial ERP epochs, each epoch corresponding to the EEG segment falling into the event-locked window of [-200ms 500ms], i.e. from 200 ms before to 500 ms after the on-

3 Burst of 50 images Common spatio-temporal pattern method target The motivation of the proposed CSTP method is to find spatial or temporal filters that maximize the variance of filtered signals of one condition and at the same time minimize the variance of filtered signals of another condition. The variances of the filtered signals can then be subjected to the classifier and used as discriminating features for optimal separation of two conditions. Technically, the CSTP filters v can be obtained by extremizing the following objective function: time Figure 2: The standard RSVP paradigm, fifty images were presented in fast bursts of 5 seconds, with each image lasting 100 ms set of each image. For each epoch, the baseline mean was calculated by using the data within the window [-200ms 0ms] and subsequently subtracted from each channel. As a result, the problem of rapid image triage boiled down to the classification problem of single-trial ERPs elicited by non-target images versus single-trial ERPs elicited by target images (non-target ERP vs. target ERP for short). SINGLE-TRIAL ERP DETECTION For the ease of presentation, let s begin with the general notations used. Let X1 and X2 be the (channel time) data matrices of the EEG signals (concatenated trials) under two conditions (target ERP v.s. non-target ERP) and Σ1 and Σ2 be the corresponding estimates of the normalized spatial covariance matrices, i.e. Σi = Xi Xi /trace(xi Xi ) where denotes the matrix transpose operator. Then, two matrices Σd and Σc are defined as follows: Σd = Σ1 Σ2 :spatial discriminative matrix; Σc = Σ1 + Σ2 :spatial composite matrix. (1) Similarly, let Σ 1 and Σ 2 be the estimates of the normalized temporal covariance matrices, i.e. Σ i = Xi Xi /trace(xi Xi ) and Σ d = Σ 1 Σ 2 :temporal discriminative matrix; Σ c = Σ 1 + Σ 2 :temporal composite matrix. (2) {max, min}v RM v Σd v v Σc v, {max, min}v RT v Σ d v, v Σ c v or (3) (4) where M is the number of channels and T is the length of each single-trial epoch. It is worth noting that Σd and Σc (or Σ d and Σ c ) in the above equation can be seen as the discriminative covariance matrix (the signal) and the composite covariance matrix (the pooled noise), respectively. The objective function of CSTP is actually looking for optimal filters that offer the highest signal-to-noise ratio for separation of two conditions. This is very similar to the objective function used for the well-known Fisher s linear discriminant analysis (see e.g. [4]) where a similar generalized Rayleigh quotient is used. It is apparent from Eq.(3) and Eq.(4) that spatial filters and temporal filters that provide best separation between two conditions can be obtained independently. Since their respective objective function shares the same form as the one used for conventional common spatial pattern method [5, 6], they can be solved by the same technique called simultaneous diagonalization of two covariance matrices Σ1 and Σ2 (or Σ 1 and Σ 2 ). Specifically, let s assume P be the whitening transformation of Σc which satisfies P Σc P = P Σ1 P + P Σ2 P = I, (5) where I refers to the identity matrix. Let S1 and S2 denote P Σ1 P and P Σ2 P, respectively. Since S1 + S2 = I from Eq.(5), it follows from spectral theorem for matrices that S1 and S2 share common eigenvectors, i.e., if then S1 = B Λ1 B, S 2 = B Λ2 B and Λ1 + Λ2 = I, (6) where Λ1 and Λ2 are the diagonal matrices of corresponding eigenvalues for S1 and S2, respectively. It is not difficult to see from Eq.(5) and Eq.(6) that projection matrix V = (BP ) simultaneously diagonalizes Σ1 and Σ2 and at the same time ensures the sum

4 of two corresponding eigenvalues to be always one. Noting that whitening of Σ c is retained by V, i.e. V Σ c V = I, only very few vectors of V associated with the largest eigenvalues in Λ 1 and Λ 2 extremize the objective function Eq.(3) and thus guarantee the optimal discrimination of the two conditions. They are the desired spatial filters. Practically, they can be easily identified as the first and last columns of V, if the eigenvalues forming Λ 1 in Eq.(6) are assumed to be sorted in descending order. Similarly, the desired temporal filters for optimal separation of two conditions as defined by Eq.(4) can be found by simultaneous diagonalization of Σ 1 and Σ 2. Typically, among all the spatial filters and temporal filters obtained, only a few (here 4 to 8) associated with the largest eigenvalues for either of two conditions are used. The features used in the classification are the corresponding log-variance of the filtered signals. The selection of the filters can be done by visual inspection of separability of each feature based on training data, or a more refined feature selection approach (e.g. [8]). It is worth noting that the length of each epoch is usually larger than the number of channels (T > M) unless a high-density EEG is used. This may lead to singular Σ 1 and Σ 2 and eventually makes the solution given by simultaneous diagonalization of Σ 1 and Σ 2 not meaningful. However, this can be easily avoided by desampling of EEG signals when calculating temporal filters. Evaluation Evaluation in this paper was simply to test whether the temporal features provided by the proposed CSTP method can offer augmented information for better separation of target ERP and non-target ERP, compared with the case when only spatial features are used. Therefore, two scenarios of classification of target ERP and non-target ERP were considered: 1) only spatial features were fed to classifier (conventional CSP scenario); 2) both spatial features and temporal features were fed to classifier (the proposed CSTP scenario). For both scenarios, a weighted support vector machine(svm) [9]wasusedastheclassifier. Itisamodifiedversionofthe standardsvmthat iswellsuitedfor unbalanced classification problems (For each subject in this study, there were about 4,800 non-target ERPs versus about 70 target ERPs for each of the training and test data: a case very similar to real rapid image triage applications). For the same reason, balanced accuracy (see [10]) was used as the performance measure. Training was done separately for each subject by using the training data collected in the training session and evaluation was done on the test data collected in the test session. For fair comparison, model selection was carefully done separately for two scenarios through a grid search together with a cross-validation procedure(for details, please refer to our previous work [10]). RESULTS AND DISCUSSION For comparison, the performances of the proposed CSTP and the conventional CSP are both shown in the Table 1. From both training accuracy and test accuracy, the proposed CSTP method outperformed the conventional CSP for most of the subjects (17 out 20). The better performance of the proposed CSTP method was also statistically significant (p-value < 0.01 for paired t-test). Comparing to the conventional CSP which only allows to exploit the spatial patterns, the better performance of the proposed CSTP implies that the temporal features provided by the proposed method do capture additional information for better separation of target ERP and non-target ERP. The conventional CSP is essentially a time-invariant method and it captures the spatial patterns by making use of the temporal redundancy, which means that it does not make any difference if the EEG data for each condition are temporally and randomly shuffled as the temporally shuffling does not affect the spatial covariance matrices, the input of the conventional CSP. In other words, the conventional CSP is not able to capture temporal patterns of the signals. In contrast, the proposed CSTP allows to capture both the spatial patterns and temporal patterns by identifying optimal spatial and temporal filters separately. Technically, the difference between the conventional CSPandthe proposedcstpisthat thestartingpoint of the former is the spatial covariance matrices Σ i, i, while the startingpoint oflatter isboth the spatialcovariance matrices Σ i, i and the temporal covariance matrices Σ i, i. The resulting spatial and temporal filters have rather different physically meanings which can be visualized but however is not shown here due to the limited space. The proposed CSTP may be generally useful for detection of single-trial slow event-related potential. It is known that the conventional CSP is well suited for capturing the spatial patterns of fast event-related desynchronisation(erd) and synchronization (ERS), such as ERD/ERS of Hz brain rhythms associated with self-paced movement imagination (see e.g. [6]). However, it may not be ideal for detection of slow event-related potentials, such the event-related potentials elicited by a target image which are characterized by slow potential deviation like P300 compo-

5 Training Session Test Session Subject CSP(%) CSTP(%) CSP(%) CSTP(%) Table 1: The performance of the proposed CSTP method and the conventional CSP method. The better performance are highlighted in bold. nent. The P300 component, the large positive voltage deflection occurring at approximately 300 ms after the subject seeing a target image is the major signature exploited by rapid image triage. Apparently, temporal patterns, in particular the latency of the P300, is critical for accurate detection of the P300, which is however not captured by the conventional CSP. This coupled with the fact that, for the case of rapid image triage, images were shown at the speed of 10 images per second. That means a target ERP epoch associated with a target image (200 ms before to 500 ms after the onset of the image) had 600ms or over 85% overlapping with its adjacent non-target ERP epoch for a non-target image shown immediately after the target image. This makes it extremely challenging for the conventional CSP to differentiate these two adjacent epoches because both epoches contains the P300 component and they differ only in the sense of temporal patterns. Based on the results of this study, the proposed CSTP appears promising for addressing such pitfalls associated with the conventional CSP, while it retains the simplicity of the algorithm and the desirable high dimension-reduction capability for feature extraction. REFERENCES [1] S. Mathan, D. Erdogmus, Y. Huang, M. Pavel, P. Ververs, J. Carciofini, M. Dorneich, and S. Whitlow, Rapid image analysis using neural signals, in In Proceedings of Conference on Human Factors in Computing Systems, New York, NY , United States, 2008, pp [2] A. D. Gerson, L. C. Parra, and P. Sajda, Cortically coupled computer vision for rapid image search, Neural Systems and Rehabilitation Engineering, IEEE Transactions on, vol. 14, no. 2, pp , [3] Y. Huang, D. Erdogmus, S. Mathan, and M. Pavel, Large-scale image database triage via eeg evoked responses, in Acoustics, Speech and Signal Processing, ICASSP IEEE International Conference on, 2008, pp [4] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern classification, 2nd ed. New York: Wiley, [5] K. Fukumizu, Active learning in multilayer perceptrons, ser. Advances in Neural Information Processing Systems 8. Proceedings of the 1995 Conference. Cambridge, MA, USA: MIT Press, 1996, pp , active learning multilayer perceptrons hidden-unit reduction Fisherinformation-based methods information matrix singularity condition. [6] H. Ramoser, J. Muller-Gerking, and G. Pfurtscheller, Optimal spatial filtering of single trial EEG during imagined hand movement, vol. 8, no. 4, pp , [7] S. Thorpe, D. Fize, and C. Marlot, Speed of processing in the human visual system, Nature, vol. 381, no. 6582, pp , [8] K.-Q. Shen, C.-J. Ong, X.-P. Li, and E. P. V. Wilder-Smith, Feature selection via sensitivity analysis of SVM probabilistic outputs, Machine Learning, vol. 70, pp. 1 20, [9] E. E. Osuna, R. Freund, and F. Girosi, Support vector machines: Training and applications, MIT, Technical Report AI Memo 1602, [10] S.-Y. Shao, K.-Q. Shen, C.-J. Ong, E. P. V. Wilder-Smith, and X.-P. Li, Automatic EEG artifact removal: a weighted support-vectormachine approach with error correction, IEEE Trans Biomed Eng, 2008, in Press.