Modeling Actuations in BCI-O: A Context-based Integration of SOSA and IoT-O

Transcription

1 Modeling Actuations in BCI-O: A Context-based Integration of SOSA and IoT-O Semantic Models for BCI Data Analytics Sergio José Rodríguez Méndez Pervasive Embedded Technology (PET) Lab, Computer Science Department, National Chiao Tung University (NCTU), Hsinchu, Taiwan, R.O.C

2 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks Brain-Computer Interface (BCI) EEG: Basic Operations BCI System: Basic Components EEG (Electroencephalogram) EEG Signals Metadata in BCI Just a quick overview BCI: BRAIN-COMPUTER INTERFACE 2

3 The Human Body A biological machine composed of different systems (mechanical, chemical, etc.) Components: organs. Lungs. Liver. Heart. Brain. 3

4 Brain-Computer Interface (BCI) A computer-based technology that allows the brain to communicate with external devices in order to restore, assist, or augment cognitive, sensory, and/or motor functions. BCI research began in the 1970s under a grant from the NSF followed by a contract from DARPA. BCI research was initially conducted on animals. The first BCI prosthetic device was implanted in humans in the mid-1990s. EEG-based BCI is the most studied and perhaps the most clinically promising BCI technology non-invasive, superior temporal resolution, ease of use, portability, and low set-up cost 4

5 EEG: Basic Operations Analog Signal Digital Signal 5

6 BCI System: Basic Components 6

7 BCI Evolution 7

8 EEG (Electroencephalogram) It is a recording of the electrical activity of the brain from the scalp. 2 general methods to measure Invasive: Requires operation to remove scalp and place electrode directly on the surface of brain. Very clean EEG signals. Non-invasive: Electrodes are placed on scalp. Noisy EEG signals (artifacts) 8

9 EEG (Non-Invasive): Sensors Evolution Wet Electrodes Not portable Expensive and complicated setup Dry Electrodes Portable Moderate cost and easy to setup 9

10 EEG Channels (10-20 EEG System) Green electrodes denotes the EEG channels used for experiment * Parietal (P), Occipital (O) are the channels that shows strongest EEG response when presented with a visual stimuli 10

11 EEG Signals Resting State E.g. open eyes, relaxed Alpha State E.g. close eyes 11

12 EEG Signals: Artifacts Similar EEG signals on all channels Artifact 1: Eyeball muscle (blinking) Artifact 2: Poor electrode contact Artifact 3: Swallowing Artifact 4: Poor electrode contact (Reference channel and skin) 12

13 Metadata in BCI Raw bio-medical signals: different human activity measurement levels over time Heterogeneous / multimodality Metadata Identify relevant characteristics Descriptive set of annotations/tags/attributes. Purpose: classification (raw data and models about them) 13

14 Metadata in BCI 14

15 Metadata in BCI 15

16 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks The Research Project "Augmented" Brain-Computer Interaction (A-BCI) Issues Research Description A mechanism to query and discover similarities among the BCI Metadata? BCI ONTOLOGY: ORIGINS & MOTIVATION 16

17 The Research Project Research area: Advanced Computational Approaches. Program: the Cognition and Neuroergonomics Collaborative Technology Alliance (CaN-CTA). Sponsorship: the U.S. Army Research Laboratory (ARL). Participants: Pervasive Embedding Technologies (PET) Lab, National Chiao Tung University (NCTU), Taiwan. Swartz Center for Computational Neuroscience (SCCN), University of California in San Diego (UCSD). Goal: to develop a semantic model that can aid the search for correlated neuro-physiological features for characterizing individual s cognitive states including fatigue, vigilance and enlightenment. the gathering of useful data sets for conducting interpersonal Transfer Learning. 17

18 Augmented Brain-Computer Interaction (A-BCI) Operations: natural and intuitive, mainstream Human-Computer Interaction in real-world activities + [Data] collect vast amounts of real-world multimodal (EEG) and activity data + [Metadata] annotation with metadata specifying relevant situations and events + [Search] for similar data sets = [To aid] machine learning in producing accurate brain state classification models 18

19 Issues Costly labor-intensive and time-consuming, to collect a significant amount of raw data from each human subject (individual) to generate BCI informative training data. Hinders the applications of BCIs in real-world settings. Substantial inter-subject variability of human raw data could deteriorate more than improve the BCI performance. A reliable mechanism to identify and select (query) similar raw data sets is needed based on the available metadata. Metadata formats. Stored in non-structured and semistructured proprietary file formats. Not interoperable. No formal structure for the metadata. BCI metadata lacks a formal and organized structure to capture the data features (concepts and relationships). No easy way to perform metadata retrieval tasks: No queries and look for hidden patterns (discovery of relationships). 19

20 Proposal Deliverable Goal Main Contribution Exploration Purpose Vision Research Description A semantic-based approach for BCI metadata allows identifying similar raw data sets through queries and discovery constructs among the metadata structured on an OWL 2 ontology An OWL 2 ontology to organize the BCI metadata Based on Design Patterns; aligned to upper/standard ontologies for sensors/actuators. Integration of models: reconcile concepts and alignments for interoperability. Context Model: a novel model to capture the architectural features of any environment. Use a semantic overlay as a means to look for relevant sets (similarity and inferencing) among BCI metadata An axiomatization to capture relevant descriptive and predictive features of the multimodal data Semantic search and inference rules to discover implicit metadata patterns To aid/ease the tasks that perform brain state predictions (classification of brain states analysis) special interest in feature-based Transfer Learning Assist A-BCI to build profiles and trends of brain dynamics in diverse real-life activities/circumstances 20

21 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks Subject (BCI interaction) Context Skeleton: Modeling Components BCI-O s Descriptive Features Context Model: Unity's Gaming Modeling Architecture A human-environment interaction model for any BCI activity BCI ONTOLOGY: 21

22 Subject (BCI interaction) Context Integrates Sense Model ( SOSA/SSN) Actuation Model ( SOSA & SAN IoT-O) Context Model ( Unity ) for BCI data capture 22

23 Skeleton: Modeling Components Wearing a set of sensors (device) and/or through actuators (actuator), human beings (subject) interact with an environment (context) while performing (session) real-world activities (activity), where stimuli (stimulus) triggered by contextual events, are observed, recorded (record) and marked (marker) in the sensed multimodal (modality) BCI data Conceptual abstractions for subject (person), context (environment), session, multimodality, and event annotation tags. 23

24 Session: The integration of the Sense and Actuation Models for BCI-O's Descriptive Features 24

25 Context Model: Unity's Gaming Modeling Architecture 25

26 BCI-O Contributions An axiomatization to capture relevant descriptive and predictive features of the multimodal data Core BCI Interaction Model Sense Model aligned to SOSA/SSN based on SSO ODP Actuation Model integrated alignment to SOSA/SSN and SAN (IoT-O) based on AAE ODP Context Model a novel model for describing any kind of real/virtual environments based on Unity Impact: A semantic-based approach to organize, query and find patterns (inferencing) among the BCI Metadata 26

27 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks Actuation Model: SOSA ontology & AAE Design Pattern (SAN/IoT-O) Alignment to SOSA/SSN. Following the AAE ODP and alignment to SAN (IoT-O). Based on design patterns from SOSA/SSN and SAN/IoT-O. BCI ONTOLOGY: ACTUATION 27

28 Actuation Model: SOSA ontology & AAE Design Pattern (SAN/IoT-O) AAE Design Pattern SOSA/SSN Actuations 28

29 Actuation Model: Alignment to SOSA/SSN 29

30 Actuation Model: following the AAE ODP and alignment to SAN (IoT-O) 30

31 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks Use Case: Modeling Actuations in BCI-O An example for modeling actuators in BCI-O. BCI ONTOLOGY: USE CASE 31

32 USE CASE: MODELING ACTUATIONS Actuation: Automated / General Examples / Actuation 32

33 Important: BCI-O's Actuation Model Provides a mechanism to correlate the observed/analyzed raw data, with the contextual components that the subject interacts with, through actuators (IoT devices), identifying how the actuations affect the context. This is useful in BCI context-aware applications to model how subjects use actuators and interact with the environment, for intelligent subject-context personalization. 33

34 BCI: Brain-Computer Interface BCI Ontology: Origins & Motivation BCI Ontology: Overview BCI Ontology: Actuation Model BCI Ontology: Use Case Modeling Final Remarks Conclusions Q&A Wrapping up FINAL 34

35 Final Remarks BCI-O makes BCI metadata "FAIR data" < : Findable Accessible Interoperable Reusable. A domain ontology for BCI sensors and actuators with a special interest in real-time IoT M2M environments with Big Data sets. A novel Context Model that makes BCI systems to be semantically context-aware for real/virtual-world situations. Thus, it gives a semantic foundation for Augmented BCI applications, assisting ambient intelligence's settings in sensor systems for any kind of BCI. BCI-O s Actuation Model integrates carefully both standard axiomatization models for actuations, developed by W3C/OGC and IoT communities, based on its Context Model. Its structure follows closely the AAE ODP, while aligning to SOSA/SSN and SAN (IoT-O) concepts, based on contextual notions. The SOSA-SAN integrated Actuation Model of BCI-O represents a major contribution to the IoT and BCI communities, especially because its structure includes contextual notions that enables its usage in contextaware BCI-IoT integrated applications. 35

36 36

37 <The-End /> THANK YOU FOR YOUR ATTENTION! 37

38 Transfer Learning (TL) & Adaptive BCI TL from a BCI perspective: From metadata used by an application A (source), use similar metadata sets that may apply to B to aid training models used in B (target). Adaptive BCI: adapt a BCI system for personal use through model refinement (calibration/adjustments of prediction and classification models). Use feature-based TL in online EEG classification: source: archived data collected from other users; similarity: yielded similar responses under similar situations; (plus) target: a small amount from the new user. 38

39 Marker & Model: Key Abstractions for BCI-O's Predictive Features

40 Sense Model: SOSA/SSN Ontology & SSO Design Pattern SSO Design Pattern SOSA/SSN Observations

41 Modeling Observation & Context 25Hz Flicker 30Hz Flicker VR Scene Before flickering objects appears Event Marker (Flickering object in view) Flickering objects appears VR Scene 28Hz Flicker Visual Stimuli Video: 41

42 INFERENCING Semantic Matching Using Inference Rules Discover new knowledge to enrich BCI metadata with new individuals and relationships Implicit or hidden patterns An SPARQL construct rule is applied by an analyzer to relate available models with activities, as a relevant relationship for further BCI analysis and processing 42

43 CerebraWeb.net < 43