Analyzing Students Attention in Class Using Wearable Devices

Transcription

1 Aalyzig Studets Attetio i Class Usig Wearable Devices Xi Zhag 1, Cheg-Wei Wu 1, Philippe Fourier-Viger 2, La-Da Va 1, Yu-Chee Tseg 1,* 1 Departmet of Computer Sciece, Natioal Chiao Tug Uiversity, Hsichu, Taiwa 2 School of Natural Scieces ad Humaities, Harbi Istitute of Techology, Shezhe, Chia zhag@cs.ctu.edu.tw, cww3@ctu.edu.tw, {ldva, yctseg * }@cs.ctu.edu.tw, philfv@hit.edu.c Abstract Detectig studets attetio i class provides key iformatio to teachers to capture ad retai studets attetio. Traditioally, such iformatio is collected maually by huma observers. Wearable devices, which have received a lot of attetio recetly, are rarely discussed i this field. I view of this, we propose a multimodal system which itegrates a headmotio module, a pe-motio module, ad a visual-focus module to accurately aalyze studets attetio levels i class. These modules collect iformatio via cameras, accelerometers, ad gyroscopes itegrated i wearable devices to recogize studets behaviors. From these behaviors, attetio levels are iferred for various time periods usig a rule-based approach ad a datadrive approach. The former ifers a studet s attetio states usig user-defied rules, while the latter relies o hidde relatioships i the data. Extesive experimetal results show that the proposed system has excellet performace ad high accuracy. To the best of our kowledge, this is the first study o attetio level iferece i class usig wearable devices. The outcome of this research has the potetial of greatly icreasig teachig ad learig efficiecy i class. Keywords Activity Recogitio; Attetio Sesig; Body-Area Network; Machie Learig; Wearable Computig I. INTRODUCTION The Iteret of Thigs (IoT) [2] ad wearable techologies [16] have rapidly become key research areas i computer scieces, as they have multiple real-life applicatios i a wide rage of domais. A wearable device is a microelectroic computig ad sesig system that ca be comfortably wor by its users. Its embedded software ad sesors allow measurig users vital sigs ad ambiet coditios. I cotrast to traditioal computers, wearable devices are lightweight, small, geerally iexpesive, ad close to the wearer s bodies. They are thus highly promisig for collectig ad aalyzig huma behavioral data. I the techology idustry, may of the largest compaies such as Google, Apple, ad Microsoft have recetly desiged wearable devices. Numerous traditioal accessories have bee trasformed ito wearable devices [16], icludig glasses, watches, clothes ad eve diapers. As poited out i a report by the Iteratioal Data Corporatio [1], the worldwide shipmet of wearable devices is expected to reach millio uits by 19. Numerous research studies have bee devoted to wearable applicatios, such as detectig users activities [5], social actios [8], falls [24], ad users browsig behaviors i retails [21]. Although wearable techology has bee applied i several domais, it is rarely cosidered for measurig studets attetio levels i class. Attetio ca be defied as the behavioral ad cogitive process of selectively cocetratig o certai pieces of iformatio [1]. I a previous study [23], it was foud that selective ad sustaied attetio has a sigificat impact o learig. Learers attetio is closely related to their learig efficiecy ad learig outcomes. Kowig the attetio levels of studets i class would thus greatly beefit both studets ad teachers. This iformatio ca help studets better uderstad their learig processes ad adapt their learig strategy, while teachers ca use this feedback to gauge studets iterest ad adjust teachig strategies to capture ad maitai studets attetio. Traditioal ways of measurig studets attetio levels iclude fillig questioaires [22], performig experimetal tests [13], ad direct observatio [14]. Although these methods are suitable i some scearios, they are time-cosumig, require huma itervetio, ad most of them caot be applied i real-time, or to a large group of studets. Some studies [3] have used surveillace cameras ad face recogitio models to automatically measure studets attetio levels. However, the accuracy of these approaches is substatially iflueced by factors such as lightig, camera positios, ad backgroud iterferece. Some studies [3] used eye trackers to detect visual attetio. However, these devices are quite expesive ad prologed use may cause eye ijury [19]. To address these drawbacks of curret approaches, this study desigs a approach for measurig studets attetio levels i class usig low-cost wearable devices. Achievig this goal is however challegig sice attetio is a iteral cogitive process that ca oly be idirectly observed through its effects o exteral actios, such as motio chages, visual focus, ad physical behaviors. To tackle this challege, this paper proposes a ovel system that captures head-motio, pemotio, ad visual-focus data for attetio iferece. The major challeges of this research are listed below. 1. Recogizig users differet activities requires the use of multiple sesors, which geerate heterogeeous data. As more types of sesors are used, it becomes icreasigly difficult to joitly aalyze the heterogeeous data that they geerate to build a iferece model 2. Idetifyig the features to be used to build a iferece model is also challegig. Huma activities are usually complex ad ca be described usig umerous features. Whe both physical ad psychological data are collected, feature selectio becomes eve more difficult.

2 3. Collectig accurate ad reliable data is ofte difficult i real-world applicatios. Data ca be oisy. Moreover, i may situatios, data are streamig at various speeds ad are ubouded. Also, real-time applicatios require fast processig time. 4. Aother challege is that persoal differeces may ifluece the success of attetio detectio, as differet persos may behave differetly i the same situatio. Moreover, some participats i the experimetal evaluatio may lie by pretedig to have bee focused whe they were distracted. To address the above issues, we propose a ew system that itegrates a Head-Motio module, a Pe-Motio module, a Visual-Focus module, ad a Active-App module for attetio iferece. These modules ru o iexpesive hardware, ad rely o machie learig techiques to select appropriate features to recogize attetio-related activities, such as head motios, had motios, ad visual behaviors. These modules are implemeted o the Raspberry Pi [] platform. I particular, the J48 decisio tree learig algorithm [27][28] is used to trai the motio classificatio models usig discrimiative features of sesor data. The behaviors recogized by these models are the fed to a attetio iferece egie, which cosists of two differet attetio iferece algorithms. Oe applies a rule-based approach, ad the other employs a data-drive approach. Fially, visual reports are geerated ad preseted to the users. Extesive experimetal results show that the proposed system is efficiet ad accurate. The outcome of this research has the potetial of greatly icreasig teachig ad learig efficiecy i class. The remaider of this paper is orgaized as follows. Related work is reviewed i Sectio II. Sectio III presets the proposed system. Sectio IV evaluates the performace of the proposed system. Fially, Sectio V draws the coclusio ad discusses future work. II. RELATED WORK This sectio first reviews the mai studies o attetio detectio. The, it takes a broader perspective to review other models of huma activity recogitio. Traditioal methods for measurig attetio levels ca be geerally categorized ito three types. The first type cosists of askig learers to fill questioaires [22]. The secod type is physiological experimets [13], where experts observe the reactios of learers whe performig tasks, to ifer their attetio levels. The third type is direct observatio [14], where experts evaluate the thought process of learers based o data recorded durig a set period of time usig devices like video ad voice recorders. Although traditioal methods are commoly used, a major drawback is that they require huma itervetio. Various devices have bee used for attetio iferece: 1. Electroecephalography devices [15][17] record the electrical activity of huma brais. Although they ca measure brai waves of a perso, they igore other aspects such as body movemets ad visual focus. 2. Eye trackers [29][3] detect eye movemet trajectories. However, they ca be quite expesive ad log term use may cause eye ijuries [19]. 3. Video cameras [3] have bee used to detect facial expressios ad body movemets. However, their use raises privacy issues, requires more computig power, ad their accuracy is iflueced by factors such as lightig coditios, camera positios, ad backgroud iterferece. I terms of huma activity recogitio, solutios ca be categorized ito sesor-based ad visio-based approaches. Sesor-based approaches rely o body-wor iertial sesors to ifer physical activities ad lifestyles [11]. Daily routies of wearers have bee detected usig topic models [9]. I aother study, sigle body-wor accelerometers were utilized to recogize social actios, icludig speakig, laughig, gesturig, drikig, ad steppig [8]. Body-wor accelerometers have also bee cosidered to recogize household activities for cotext-aware computig [5]. A lot of visio-based solutios have bee proposed. A camera-based surveillace system was desiged for detectig huma movemets [18]. Other researchers [3] have explored the relatio betwee head rotatio, eye gaze directio ad facial features to ifer huma attetio. To reduce the computatioal cost for image processig, a collaborative model usig a depth camera ad a iertial measuremet sesor was proposed [6]. The model uses esemble classifiers at both the feature levels ad decisio levels. III. ATTENTION INFERENCE SYSTEM A. Classroom Sceario This study cosiders the followig sceario (Fig. 1): (1) A teacher shows slides for teachig usig a projector. Each slide cotais a special mark, such as a school logo, used by our system to check if each studet is payig attetio to the slides. (2) The teacher has a computer i frot of him/her, which acts as a server. It collects sesor data ad ifers studets attetio levels. (3) Each participatig studet sits at a desk ad wears smart glasses or a smart cap. The wearable device is i either case equipped with a camera, a accelerometer ad a gyroscope. (4) Each studet uses a smart pe to take otes as usual. A SesorTag [26] embedded with a accelerometer ad a gyroscope is attached to each pe. (5) Alteratively, a studet may also take otes o his/her otebook. It is the assumed that otes are iserted i a PowerPoit file or usig a PDF editor. (1) (2) (1) Projectio image (with logo) (2) Server & iferece egie (3) Smart glasses/cap (4) Smart pe with SesorTag (5) tebook (or Tablet) (5) (4) (3) Fig. 1. The classroom sceario cosidered i this work. (1) projectio image ( (2) server & iferece (3) smart glasses/cap (4) smart pe (5) otebook (PPT or

3 Server Side Cliet Side Attetio Level Iput Process Output Logo of slides, provided by the teacher i advace Image data captured by the smart cap s camera ACC ad GYRO data geerated by the smart cap s MPU 5 ACC ad GYRO data geerated by the smart pe s SesorTag App logs from otebook ad tablet Head-Motio Module Visual-Focus Module Pe-Motio Module Active-App Module Attetio Iferece Egie Attetio Level Report Time Activity Report Head motio = {Still} Head motio = {Dow} Head motio = {Up} Time Fig. 2. Hardware compoets of our prototype system. B. Prototype Desig This sectio presets the desig of our prototype system, based o commodity hardware ad a cliet-server architecture. O the cliet side, a wearable device implemeted o a Raspberry Pi Model B+ [] has bee desiged for sesig motio ad the visual focus of users. It itegrates a camera ad a sesor board MPU 5 cotaiig a 3-axis accelerometer ad a 3-axis gyroscope. The accelerometer ad gyroscope of MPU 5 are used to recogize head motios (e.g. raisig/lowerig the head), while the camera is used to capture the focused field of the wearer for visual-focus detectio. These compoets ca be easily itegrated ito a cap or glasses (see Fig. 2). A SesorTag is attached to each smart pe, ad cotais a 3-axis accelerometer ad a 3-axis gyroscope. The SesorTag o the smart pe is used to detect had motios (e.g. writig/still) of the user. The commuicatio betwee SesorTag ad Raspberry Pi relies o Bluetooth Low Eergy (BLE) protocol. O the server side, Wi-Fi protocols ad Apache Http Server Versio 2.4 [25] are used to collect data set from cliet side. The, the pe-motio, head-motio ad visual-focus modules are ru o server to recogize behaviors of users. The recogized behaviors are the fed to a attetio iferece egie to calculate users attetio levels. Fially, visualized reports are geerated by the egie to idicate each user s attetio levels for differet time periods. C. Overview of the System Architecture Fig. 3 shows the Iput-Processig-Output (IPO) model of the proposed system for a sigle user. It hadles streamig data usig a batch model. Durig each iteratio, it receives a batch of data, which icludes: (1) The image of the special logo o slides. (2) The image data captured by the user s smart glasses/cap. (3) 3-axis accelerometer (abbr. ACC) ad 3-axis gyroscope (abbr. GYRO) data geerated from MPU 5 of smart cap. (4) 3-axis ACC ad 3-axis GYRO data geerated from SesorTag of smart pe. (5) The logs of Apps used by the users. The system outputs a attetio level report ad a activity report, which idicate the attetio level of the user ad his/her recogized behavior durig each time period, respectively. The processig part of the system cosists of five modules, amely the Head-Motio, Pe-Motio, Visual- Focus, Active-App ad Attetio Iferece Egie modules. These modules commuicate based o a cliet-server architecture, as depicted i Fig. 4. These modules are explaied i the followig subsectios. Fig. 3. The IPO model of the proposed system for a sigle user. Pe-Motio Module Pe-motio Data Preprocessig Accelerometer Data Data Segmetatio Gyroscope Data Pe-motio Type Recogitio Pe-motio Feature Extractio Pe-motio Classifier Writig Still Other Recogized Behaviors Fig. 4. The architecture of the proposed attetio iferece system. D. Head-Motio Module Head motio is a exteral behavior that may serve as a importat idicator of studet attetio. Durig the data preprocessig stage, raw sesor data are sampled ad segmeted. There are six sesor sigal sources (received from a 3-axis ACC ad a 3-axis GYRO). The samplig rate is 5Hz. Data are partitioed ito uiform o-overlappig three-secod segmets. Segmeted data are set to the server. Let S = s 1, s 2,..., s be oe of the sesor sigal sources, where s i (1 i ) is the i-th sample i S. The proposed system extracts the followig six time-domai features related to eergy or magitude from S. Mea is a measure of cetral tedecy, which is defied as Mea S = s = i=1 s i. (1) Variace represets the expected squared deviatio from the mea, which is defied as Var(S) = Head-Motio Module Head-motio Data Preprocessig Accelerometer Data Data Segmetatio Head-motio Type Recogitio i=1 (s i s ) 2 Gyroscope Data Head-motio Feature Extractio Pe-motio Classifier Up Still Dow Attetio Iferece Egie Attetio Iferece Algorithms Attetio Scorig Fuctios Active-App Module Visual-Focus Module Attetio Level Report ad Activity Report. (2)

4 Root Mea Square (RMS) is the square root of the arithmetic mea of the squares of the values i S, which is defied as RMS(S) = i=1 s i 2. (3) Average Absolute Differece (AAD) measures the statistical dispersio of cosecutive samples i S, which is defied as AAD(S) = i=2 s i s i 1 1. (4) Zero Crossig Rate (ZCR) measures the umber of sig chages i S, which is defied as ZCR(S) = i=2 sig s i sig (s i 1 ) 2, (5) where sig(x) is a fuctio which returs -1/+1 if the sig of x is egative/positive ( is regarded as positive). Mea Crossig Rate (MCR) measures the umber of sig chages usig the mea as baselie, which is defied as MCR(S) = i=2 sig s i s sig (s i 1 s ) 2. (6) Sice there are two types of sesors (i.e., ACC ad GYRO) ad each sesor has three axes, calculatig the above features results i (2 3 6)=36 features. Moreover, for each type of sesor, the system also measures the covariace ad correlatio betwee each pair of its two sesor sigal sources. Let A = a 1, a 2,..., a ad B = b 1, b 2,..., b be two sesor sigal sources i a segmet. The covariace ad correlatio betwee A ad B are measured usig the followig two formalizatios. Covariace measures how much A ad B chage together, which is defied as Cov(A, B) = i=1 a i a (b i b). (7) Correlatio measures the stregth ad the directio of the relatioship betwee A ad B, which is defied as Corr(A, B) = cov (A,B) σ A σ B, (8) where σ A ad σ B are the stadard deviatios of A ad B, respectively. Sice there are two types of sesors ad each sesor has three axes, measurig the covariace ad correlatio betwee each pair of axes results i 12 features. Therefore, the system totally extracts (36+12) = 48 features. The, the 48 defied features are take as iput by the head-motio module to trai a classifier offlie, which is the used for olie recogitio of head motios. Three head motios are cosidered: {Still, Up (raisig the head), Dow (lowerig the head)}. The J48 decisio tree learig algorithm offered i Weka [27] is used for traiig phase. The traiig phase is performed i three steps. Step 1: Head-motio traiig data are collected ad each data is labeled with its class. Let C be the set of classes {Still, Up, Dow}. The result is a traiig set TraiDS = {(d 1, g 1 ), (d 2, g 2 ),..., (d m, g m )}, where d i (1 i m) is the i-th data istace ad g i C (1 i m) is the label of d i. Step 2: Each traiig istace d i is trasformed ito a feature vector f i described by the 48 defied features. This results i a trasformed dataset FDS = {(f 1, g 1 ), (f 2, g 2 ),..., (f m, g m )}. Step 3: This dataset FDS is fed to the J48 decisio tree learig algorithm to trai the head-motio classifier, called HM-Classifier(.). The traied classifier is the used for olie recogitio. As the server cotiuously receives segmeted head-motio data from cliets, it applies the HM-Classifier(.) to recogize the head-motio type of each segmet. This is performed i two steps. Step 1: For each received segmet, the 48 features are extracted, deoted as f SD. Step 2: HM-Classifier(.) is the applied to classify f SD. The output is its class label, deoted as HM-Classifier(f SD ). te that the umber of features ca be reduced by usig the gai ratio goodess fuctio [27][28] of the J48 classifier while preservig a high accuracy. E. Pe-Motio Module Pe motio is aother exteral behavior that ca help measure studet s attetio. The pe-motio module is desiged to detect if a studet is writig o a piece of paper. The SesorTag o the smart pe seds raw 3-axis ACC ad 3- axis GYRO data at a samplig rate of 5Hz to the smart cap. The smart cap collects ad segmets the received data ad the seds segmeted data to the server every three secods. The server the extracts the 48 features defied i Sectio III.D from the sesor data ad classifies the studet s pemotio data usig a classifier. This classifier is traied oce offlie. Three types of pe motios are cosidered: {Still, Writig, Other}. For the traiig phase, segmets of raw sesor data are maually labeled with these three classes. The traiig phase is doe usig the J48 classifier from Weka. The traied classifier is applied to olie recogitio every three secods to detect studet s pe motios. As previously explaied, the umber of classifier s features ca be reduced usig the gai ratio goodess fuctio [28]. By applyig this techique, the decisio tree learig algorithm selects oly the most discrimiative features for motio recogitio. I our implemetatio, this techique cosiderably reduces executio time for recogitio, while preservig a high accuracy. F. Visual-Focus Module The visual focus of a huma is also a importat idicator of the attetio, as humas ted to pay attetio to objects appearig i the ceter of their visual field. I our implemetatio, a camera is attached to the smart cap. The assumptio is that the camera captures what the studet sees, ad the images ca thus reveal what is curretly drawig the studet s attetio. I our desig, the camera is programmed to record image data oly whe the user s head is i motio.

5 Visual-Focus Module Head-motio Type Head-motio Type ={Up} Slide Detectio Had Detectio Head-motio Type = {Dow} Fig. 8. A example of ski color mask for had recogitio. Slide Foud Slide Had Foud Had Fig. 5. The system architecture of the visual-focus module. Fig. 6. Examples of slide detectio usig the SURF algorithm. Fig. 7. A example of feature poit detectio usig the SURF algorithm. Cosiderig the size of image data ad trasmissio rate, the Visual-Focus module is ru o the Raspberry Pi o the cliet side s smart cap. Fig. 5 shows the workflow of this module, which cosists of two sub-modules: Slide Detectio ad Had Detectio. (a) Slide Detectio If the head-motio type recogized by the Head-Motio module is {Up} (i.e., raisig the head), the Slide Detectio module will be triggered to idetify whether the user is payig attetio to the slides. Our system assumes that a school logo is show o each slide. If the logo is foud i a image captured by the camera, it is very likely that the slide is withi the studet s field of visio, ad that the studet is payig attetio to its cotet. To fid the logo from the captured image, we use the Speed-Up Robust Feature (SURF) algorithm [4] offered i the OpeCV library. Fig. 6 shows two examples of recogitio results usig SURF. As depicted i Fig. 7, SURF extracts feature poits from the logo image ad tries to match these poits with the captured image. If the umber of matched poits is greater tha a miimum matchig umber threshold, the system assumes that the slide is withi the studet s field of visio. Moreover, the system also verifies whether the slide is cetered i the studet s field of visio. If the slide is off-ceter by a distace greater tha a miimum off-ceter threshold, the system assumes that the studet is ot lookig at the slide. (b) Had Detectio If the head-motio type recogized by the Head-Motio module is {Dow} (i.e., the head is lowered), the Had Detectio module will be triggered to check if the user s had appears i the captured image. This iformatio will be further itegrated with the recogitio results of the pe-motio module to determie if the user is takig otes. The assumptio is that a studet s had should be visible i the images captured by the camera if he/she is takig otes, ad that it should ot be visible if the studet is lookig at the slides. To check if a had appears i a captured image, we apply a method based o ski color masks usig the YCbCr color space [7]. This method extracts all the potetially skicolored pixels from a captured image. If the Cb or Cr values of a pixel are i the [98, 142] or [133, 177] itervals, respectively, the pixel is cosidered as ski-colored [12]. For example, cosider the left part of Fig. 8, which shows a image captured by the camera. The right part of Fig. 8 illustrates the detected ski-colored pixels (represeted as white pixels). If the umber of ski-colored pixels is o less tha a miimum pixel threshold, the system assumes that the had of the user is visible i the captured image. G. Active-App Module Cosiderig that more ad more studets take otes o their otebooks ad tablets durig lectures, we foud that idetifyig active Apps that are curretly beig used by the studets is a efficiet way to ifer their attetio level. I this work, it is assumed that otes are take o PowerPoit, Word or PDF files. The Active-App module logs the Apps ad files curretly used o a studet s otebook or tablet, ad seds this iformatio to the server. The server uses this data to check if a studet is payig attetio to the course s slides. H. Attetio Iferece Egie Whe user s behaviors are detected (i.e., head motio, pemotio, visual focus, ad Apps used by user), the proposed attetio iferece egie will ifer the user s attetio level based o these behaviors. Two attetio iferece algorithms are proposed, called the rule-based approach ad the datadrive approach, respectively. (a) Rule-based Approach The mai characteristic of the rule-based approach is that it ifers studet s attetio levels based o user-defied decisio rules. These rules have the merit of beig ituitive, iterpretable ad to allow fast recogitio. Fig. 9 shows the workflow of the rule-based approach. The algorithm is a iterative method. I each iteratio, it takes the recogitio results of modules as iputs, ad outputs a attetio state. Two types of attetio state are cosidered: {Focused, Ufocused}.

6 Attetio State Focused Ufocused Start Is the user s head still? Still duratio > threshold? Recogized Behaviors High-level Feature Extractio Classifier Focused Ufocused Up What is the user s head motio type? Dow Fig. 1. Workflow of the data-drive approach. Does slide appear i the user s visual field? Does a had appear i the user s visual field? Head Motios Pe Motios Up Up Still Still Still Still Dow Dow Dow Dow Other Other Still Still Still Still Write Other Write Other Take otes by smart pe? Use class-related files o otebook or tablet? Foud Slide Foud Visual Focus Had Foud Had Slide Had Had Active Apps PDF PPT Fig. 11. A example of the iput of the data-drive approach. Focused Ufocused Focused Focused Ufocused Ufocused Fig. 9. Workflow of the rule-based approach. I a iteratio, the algorithm proceeds as follows. At the begiig, the algorithm checks whether the user s headmotio type is {Still} or ot. If it is true ad the still duratio is o less tha a threshold, the user is likely i a daze ad thus the algorithm outputs {Ufocused}. If the head-motio type is ot {Still}, the algorithm the checks whether the user s headmotio type is {Up} or {Dow}. If the head-motio type is {Up}, the algorithm uses the recogitio result of the Slide Detectio module to idetify whether the slide appears i the user s visual field. If the slide appears i the user s visual field, the algorithm outputs {Focused}. O the cotrary, it outputs {Ufocused} if the slide does ot appear i the user s visual field. If the head-motio type is {Dow}, the algorithm uses the recogitio result of had detectio to idetify whether the user s had appears i the user s visual field. If the had appears i the user s visual field, the algorithm uses the recogitio result of the pemotio module to idetify whether the user had writte or ot. If the recogitio result of the pe-motio module is {Writig}, the algorithm outputs {Focused}. If the recogitio result is ot {Writig} or the had does ot appear i the user s visual field, the algorithm uses the result of the Active-App module to check whether the user had used class-related Apps. If the result is true, the algorithm outputs {Focused}. Otherwise, {Ufocused} is outputted. (b) Data-drive Approach A o-empirical approach is to use a data-drive solutio. It uses machie learig methods to fid hidde iformatio i the collected data, ad utilizes this iformatio for buildig a decisio model for attetio iferece. If there are more useful traiig data beig used for model buildig, the iferece results will be more accurate. Fig. 1 shows the workflow of the data-drive approach. The algorithm is a iterative method. I each iteratio (also called widow), it takes -secod recogitio results of the Head-Motio, Pe-Motio, Visual- Focus, ad Active-App modules as iputs. I our implemetatio, is set to 3. The, the algorithm extracts high-level features from the iput data ad classifies the studet s attetio state (i.e., Focused or Ufocused) usig a classifier. The classifier is traied offlie. For the traiig phase, the class of each traiig istace is maually labeled. The traiig phase is doe usig the J48 classifier from Weka. The traied classifier is applied to olie recogitio every secods to detect studet s attetio state. Next, we itroduce the high-level features extracted by the data-drive approach. Let H 1, H 2, H 3, ad H 4 deote the - secod recogitio results of the Head-Motio, Pe-Motio, Visual-Focus, ad Active-App modules, respectively. Let H i = h i1, h i2,..., h i, 1 i 4, where h ij (1 j ) is the j-th data istace i H i. Data istaces i H i are also called activities. For example, i Fig. 11, the 3-secod recogitio results from the head-motio module is H 1 = Still, Still, Dow, Up, Dow, Up, Still, Still, Dow, Dow. Let R i = {r i1, r i2,..., r Ri } be the set of activities i H i. For example, H 1 has three types of activities {Still, Up, Dow}. The, for each iput H i (1 i 4), the system extracts the followig high-level features. The umber of two adjacet data istaces i H i that are differet activities. For example, i the aforemetioed H 1, that umber is 6. The umber of activities that are r ik (1 k R i ) i H i. For example, the umber of activities that are {Dow} i H 1 is 4. The maximum duratio of r ik (1 k R i ) i H i. For example, the maximum duratio of {Dow} i H 1 is max{1, 1, 2} = 2. The miimum duratio of r ik (1 k R i ) i H i. For example, the miimum duratio of {Dow} i H 1 is mi{1, 1, 2} = 1. The average duratio of r ik (1 k R i ) i H i. For example, the average duratio of {Dow} i H 1 is (1+1+2)/3 = (c) Attetio Scorig Fuctio The attetio scorig fuctio is used to idicate the stregth of the studet s attetio level i a visual maer. Recall that i our proposed attetio iferece algorithms, i each iteratio or widow, they will output a attetio state (i.e., {Focused} or {Ufocused}). Therefore, i the -th iteratio, attetio states are obtaied. Let ST = st 1, st 2,..., st be the ordered set of these attetio states. The attetio score of ST is defied as Score(ST) = 1 i=1 g(st i ), (9) where g(s i ) returs a value g 1 (g 1 ca be defied as a positive value) if st i is {Focused}, ad returs g 2 (g 2 ca be defied as a egative value or zero) if st i is {Ufocused}. I additio to the above attetio scorig fuctio, teachers ca desig other iterestig scorig fuctios depedig o the requiremets of the applicatios.

7 Accuracy (%) Accuracy (%) Accuracy (%) Accuracy (%) IV. EXPERIMENTS AND PERFORMANCE EVALUATION Experimets were coducted to assess the performace of the proposed Head-Motio module, Pe-Motio module, Slide Detectio module, ad the two attetio iferece algorithms. A. Performace of Head-Motio Module To assess the Head-Motio module s ability at recogizig head motios, a experimetal study with five participats was performed i a classroom eviromet. Three head motios were cosidered: {Still, Up (raisig the head), Dow (lowerig the head)}. To simulate a real classroom eviromet, participats were asked to sit at a desk ad wear the desiged head-mouted wearable device, which cotais a 3-axis accelerometer ad a 3-axis gyroscope. A video camera was used to record the sessio for groud truth labelig. Totally, 25 data istaces were collected. Amog those, 15 data istaces were used for the traiig phase ad the other for the testig phase. Characteristics of the traiig ad testig datasets are preseted i Table I. I Table I, the traiig ad testig datasets are deoted as HM_TraiDS ad HM_TestDS, respectively. The umber of data istaces that are labeled as {Still}, {Up}, ad {Dow} are deoted as #Still, #Up, ad #Dow, respectively. To recogize head motios, the J48 decisio tree learig algorithm offered i Weka [27] was used. The classifier was traied usig the 48 features of the 3-axis accelerometer ad 3- axis gyroscope data, preseted i Sectio III.D. Fig. 12 shows the recogitio results of the costructed classifier for each head-motio type o the HM_TestDS dataset. As show i Fig. 12, the classifier achieves a high recogitio rate, with a average precisio of 89.1%, a average recall of 88%, ad a average F-measure of 87.8%. B. Performace of Pe-Motio Module The performace of the Pe-Motio module was assessed i the same classroom eviromet. Three pe motios were cosidered: {Still, Write, Other}. The participats were asked to use the desiged smart pe, equipped with a 3-axis accelerometer ad a 3-axis gyroscope. A video camera was used to record the sessio for groud truth labelig. Totally, data istaces were collected, where data istaces were used for the traiig phase ad the other for the testig phase. Characteristics of the collected traiig ad testig datasets for pe-motio recogitio are show i Table II. I Table II, the traiig ad testig datasets are deoted as PM_TraiDS ad PM_TestDS, respectively. The umber of data istaces that are labeled as {Still}, {Write}, ad {Other} are deoted as #Still, #Write, ad #Other, respectively. A J48 classifier was traied usig the 48 features preseted i sectio III.D. The precisio, recall ad F-Measure were used to assess the recogitio rate of the pe-motio module. Fig. 13 shows the recogitio results of the costructed pe-motio classifier for each pe-motio type o the PM_TestDS dataset. As show i Fig. 13, the costructed classifier achieves remarkable recogitio rate with a average precisio of 97.27%, a average recall of 97.6%, ad a average F- measure of 97.6%. TABLE I. TABLE II. CHARATERISTICS OF TRANING AND TESTING DATASETS FOR THE HEAD-MOTION MODULE. Dataset #Istace #Sill #Up #Dow HM_TraiDS HM_TestDS CHARATERISTICS OF TRANING AND TESTING DATASETS FOR THE PEN-MOTION MODULE. Dataset #Istace #Sill #Write #Other PM_TraiDS PM_TestDS Precisio Recall F-Measure Still Up Dow Fig. 12. Effectiveess of the head-motio module o the HM_TestDS dataset. Precisio Recall F-Measure Still Write Other Fig. 13. Effectiveess of the pe-motio module o the PM_TestDS dataset M 2M_Oblique 2.5M Miimum Matchig Number Threshold Fig. 14. Accuracy of the slide detectio module for room EC5B whe the matchig umber threshold is varied M 4M_Oblique 5.5M Miimum Matchig Number Threshold Fig. 15. Accuracy of the slide detectio module for room EC543 whe the matchig umber threshold is varied.

8 Accuracy (%) Accuracy (%) C. Performace of Slide Detectio Module The performace of the Slide Detectio module was assessed i a experimet coducted i two differet locatios: a 1-perso room (EC5B) ad a -perso room (EC543) at Natioal Chiao Tug Uiversity. I this experimet, five participats were asked to atted a lecture ad press a butto whe they were lookig at the slides, to collect groud truth labels for the recorded data. The wearable device cameras captured images with a resolutio. The miimum offceter threshold was set to 3. We evaluated the accuracy of the Slide Detectio module uder the followig three factors: (1) the miimum matchig umber threshold, (2) the distace betwee the user ad projectio scree, ad (3) the viewig agle (i.e., oblique or frot facig) of the user. Fig. 14 ad Fig. 15 show the results. Based o Fig. 14 ad 15, we make the followig observatios. First, we observe that if the miimum matchig umber threshold is icreased, the accuracy icreases util it reaches a peak, ad the the accuracy decreases afterwards. This is reasoable sice a higher threshold meas a stricter requiremet for image recogitio. Whe the threshold is set too low, oise images may be cosidered as matchig with the slide logo. Secod, the accuracy decreases as the distace betwee the user ad the projectio scree icreases. This is because as the distace icreases, the captured image of the logo become smaller ad less clear. As a result, the recogitio rate of the system decreases. Third, the system is more accurate for frot facig participats that for those viewig the projectio scree from a oblique perspective. D. Performace of Attetio Iferece Egie The performace of the proposed attetio iferece egie, usig the rule-based approach or the data-drive approach, was also evaluated. The experimet was coducted with te participats. Each participat atteded a distict 5- miute lecture i a classroom, seated frot facig, at a 2m distace from the projectio scree. A video camera was used for groud truth labelig. The head-motio ad pe-motio classifiers, itroduced i Sectio IV.A ad Sectio IV.B, were used for head-motio ad pe-motio recogitios, respectively. For the Slide Detectio module, the miimum matchig umber threshold ad the miimum off-ceter threshold θ were set to 18 ad 3, respectively. For the Had Detectio module, the miimum pixel threshold was set to,. Fig. 16 shows results obtaied by the rule-based approach. It was able to correctly idetify the attetio state of beig focused ad ufocused with a average precisio of 78.1%, a average recall of 61%, ad a average F-measure of 69.2%. Fig. 17 shows the accuracy of the datadrive approach usig various machie learig algorithms, icludig J48 decisio tree, Radom Forest [28], ad Support Vector Machie (SVM) [28]. As show i this figure, differet classifiers have similar accuracy, but the J48 decisio tree achieves the highest accuracy. Fig. 18 shows the performace of the data-drive approach usig J48 i terms of precisio, recall, ad F-measure. The proposed data-drive approach achieves aroud % F-measure values for both the Focused ad Ufocused attetio states, which demostrates that it is effective at idetifyig the attetio states of studets i class. Precisio Recall F-measure Focused Ufocused Fig. 16. The performace of rule-based approach i terms of precisio, recall ad F-measure. J.48 RadomForest SVM J.48 RadomForest SVM Fig. 17. The performace comparisio of the data-drive approach usig differet classifiers. Focused Ufocused Fig. 18. The performace of data-drive approach i terms of precisio, recall ad F-measure. Rule-based Approach Precisio Recall F-measure Data-drive Approach Fig. 19. The performace compariso of two attetio iferece approaches. Fig. 19 compares the accuracy of the rule-based approach ad the data-drive approach. It is foud that the data-drive approach is the most accurate. This is because the data-drive approach uses machie learig methods to fid hidde relatioships betwee extracted high-level features ad attetio states for attetio iferece, while the rule-based approach uses ituitive decisio rules for iferece. However, the rule-based approach ca output the recogitio result every three secods, while the data-drive approach eeds to collect thirty secods of data to produce a result.

9 V. CONCLUSIONS AND FUTURE WORKS Assessig attetio levels of studets is highly desirable. It ca let studets uderstad their ow learig behavior so that they ca lear more efficietly. For teachers, this iformatio is also very importat as it idicates how studets react to their teachig. This iformatio is thus crucial for the desig of strategies for capturig ad maitaiig studet s attetio. I this work, we have demostrated that it is possible to accurately ifer the attetio levels of studets i a classroom based o their exteral behaviors, usig various types of wearable sesors. A ew wearable system was desiged, cosistig of four modules, amed the Head-Motio, Pe- Motio, Visual-Focus, ad Active-App modules. These modules recogize ad track differet activities of users, to provide behavioral data to the desiged attetio iferece egie, which calculates users attetio levels. This egie is equipped with two ovel attetio iferece algorithms, amed the rule-based approach ad the data-drive approach. The former uses ituitive decisio rules to ifer the attetio state of a studet, while the latter relies o machie learig methods. The egie ca geerate visual reports to idicate each studet s attetio levels ad class-related activities. Extesive experimets were coducted to evaluate the proposed system. Results have show that it is highly accurate i various real-life settigs. The proposed methodology thus has the potetial of greatly icreasig learig ad teachig efficiecy i the classroom. For future work, other types of activities ad sesors will be cosidered to further refie the attetio iferece methods preseted i this paper. Moreover, a larger scale evaluatio of the system is also plaed, where full classrooms of studets will be equipped with the desiged wearable modules. ACKNOWLEDGEMENT This work was partially supported by Miistry of Sciece ad Techology, Taiwa, R.O.C. uder grat E MY3. REFERENCES [1] J. R. Aderso, Cogitive Psychology ad Its Implicatios,, 6th ed. Worth Publishers, 4. [2] A. Al-Fuqaha, M. Guizai, M. Mohammadi, M. Aledhari, ad M. Ayyash, Iteret of Thigs: A Survey o Eablig Techologies, Protocols, ad Applicatios, IEEE Commuicatios Surveys & Tutorials, Vol. 17,. 4, pp , 15. [3] S. Asteriadis, K. Karpouzis, ad S. Kollias, The Importace of Eye Gaze ad Head Pose to Estimatig Levels of Attetio, i Proc. of IEEE Iteratioal Coferece o Games ad Virtual Worlds for Serious Applicatios, pp , 11. [4] H. Bay, A. Ess, T. Tuytelaarsm ad L. V. Gool, Speeded-Up Robust Features (SURF), Computer Visio ad Image Uderstadig, Vol. 11, Issue 3, pp , 8. [5] L. Bao ad S. S. Itille, Activity Recogitio from User-aotated Acceleratio Data, i Proc. of Iteratioal Coferece o Pervasive, pp. 1-17, 4. [6] C. Che, R. Jafari, ad N. Kehtaravaz, Improvig Huma Actio Recogitio usig Fusio of Depth Camera ad Iertial Sesors, IEEE Tras. o Huma-Machie Systems, Vol. 45,. 1, pp , 15. [7] DOCUMENTATION OpeCV, [8] H. Hug, G. Eglebiee, ad J. Kools, Classifyig Social Actios with a Sigle Sccelerometer, i Proc. of ACM Iteratioal Joit Coferece o Pervasive ad Ubiquitous Computig, pp. 7-21, 13. [9] T. Huyh, M. Fritz, ad B. Schiele, Discovery of Activity Patters usig Topic Models, i Proc. of ACM Iteratioal Coferece o Ubiquitous Computig, pp. 1-19, 8. [1] Iteratioal Data Corporatio (IDC), [11] A. M. Kha, Y.-K. Lee, S. Y. Lee, ad T.-S. Kim, A Triaxial Accelerometer-based Physical-activity Recogitio via Augmetedsigal Features ad a Hierarchical Recogizer, IEEE Tras. o Iformatio Techology i Biomedicie, Vol. 14,.5, pp , 1. [12] P. Kakumau, S. Makrogiais, ad N. Bourbakis, A Survey of Skicolor Modelig ad Detectio Methods, Patter Recogitio, Vol., Issue 3, pp , 7. [13] D. M. Laders, S. H. Boutcher, ad M. Q. Wag, A Psychophysiological Study of Archery Performace, Research Quarterly for Exercise ad Sport, Vol. 57, pp , [14] G. W. Lauth, B. G. Heubeck, ad K. Mackowiak, Observatio of Childre with Attetio-Deficit Hyperactivity (ADHD) problems i Three Natural Classroom Cotexts, British Joural of Educatioal Psychology, Vol. 76, pp , 6. [15] Y. Li, X. Li, M. Ratcliffe, L. Liu, Y. Qi, ad Q. Liu, A Real-time EEG- Based BCI System for Attetio Recogitio i Ubiquitous Eviromet, i Proc. of ACM Iteratioal Workshop o Ubiquitous Affective Awareess ad Itelliget Iteractio, pp. 33-, 11. [16] Oscar D. Lara ad Miguel A. Labrador, A Survey o Huma Activity Recogitio Usig Wearable Sesors, IEEE Commuicatios Surveys & Tutorials, Vol. 15,. 3, 13. [17] D. Mig, Y. Xi, M. Zhag, H. Qi, L. Cheg, B. Wa, ad L. Li, Electroecephalograph (EEG) Sigal Processig Method of Motor Imagiary Potetial for Attetio Level Classificatio, i Proc. of IEEE Iteratioal Coferece o Egieerig i Medicie ad Biology Society, pp , 9. [18] V. D. Nguye, M. T. Le, A. D. Do, H. H. Duog, T. D. Thai, ad D. H. Tra, A Efficiet Camera-based Surveillace for Fall Detectio of Elderly People, i Proc. of IEEE Coferece o Idustrial Electroics ad Applicatios, pp , 14. [19] W. R. Pruehser ad J. D. Ederie, Ifrared Radiat Itesity Exposure Safety Study for the Eye Tracker, Biomed Sci Istrum, Vol. 41, pp , 5. [] Raspberry Pi Foudatio, [21] S. Rallapalli, A. Gaesa, K. K. Chitalapudi, V. N. Padmaabha, ad L. Qiu, Eablig Physical Aalytics i Retail Stores usig Smart Glasses, i Proc. of ACM Iteratioal Coferece o Mobile Computig ad Networkig, pp , 14. [22] J. M. Schepers, The Costructio ad Evaluatio of a Attetio Questioaire, SA Joural of Idustrial Psychology, Vol. 33, pp , 7. [23] M. M. Sohlberg ad C. A. Mateer, Effectiveess of a Attetio Traiig Program, Joural of Cliical ad Experimetal Neuropsychology, Vol. 9, Issue 2, pp , [24] T. Shay, S. J. Redmod, M. R. Narayaa, ad N. H. Lovell, Sesorsbased Wearable Systems for Moitorig of Huma Movemet ad Falls, IEEE Sesors Joural, Vol. 12, pp , 12. [25] The Apache Software Foudatio, [26] The SesorTag Story-Texas Istrumets, SesorTag&HQS=sesortag. [27] Weka 3: Data Miig Software i Java, [28] I. Witte, E. Frak, ad M. A. Hall, Data Miig: Practical Machie Learig Tools ad Techiques, 3rd ed. Morga Kaufma, 5. [29] Y. Xu, N. Stojaovic, L. Stojaovic, ad T. Schuchert, Efficiet Huma Attetio Detectio based o Itelliget Complex Evet Processig, i Proc. of ACM Iteratioal Coferece o Distributed Evet-Based Systems, pp , 12. [3] Z. Ye, Y. Li, A. Fathi, Y. Ha, A. Rozga, G. D. Abowd ad, J. M. Rehg, Detectig Eye Cotact usig Wearable Eye-trackig Glasses, i Proc. of ACM Iteratioal Coferece o Ubiquitous Computig, pp , 12.