Practical Experience Recording and Indexing of Life Log Video

Transcription

1 Practical Experiece Recordig ad Idexig of Life Log Video Datchakor Tacharoe Dept. of Electroic Egieerig The Uiversity of Tokyo 5--5 Kashiwaoha, Kashiwa, Chiba, JAPAN Phoe: (+8) Toshihiko Yamasaki Dept. of Frotier Iformatics The Uiversity of Tokyo 5--5 Kashiwaoha, Kashiwa, Chiba, JAPAN Phoe: (+8) Kiyoharu Aizawa Dept. of Frotier iformatics The Uiversity of Tokyo 5--5 Kashiwaoha, Kashiwa, Chiba, JAPAN Phoe: (+8) ABSTRACT This paper presets a experiece recordig system ad proposes practical video retrieval techiques based o Life Log cotet ad cotext aalysis. We summarize our effective idexig methods icludig cotet based talkig scee detectio ad cotext based key frame extractio based o GPS data. The voice aotatio ad detectio is proposed for practical idexig method. Moreover, we apply a additioal body sesor to record our life style ad aalyze huma s physiological data for Life Log retrieval system. I the experimets, we demostrated various video idexig results which provided their sematic key frames ad Life Log iterfaces to retrieve ad idex our life experieces effectively. Categories ad Subject Descriptors H.3.3 [Iformatio Systems]: Iformatio Storage ad Retrieval, -Iformatio Search ad Retrieval Geeral Terms Huma Factors Keywords Life Log, video retrieval, cotet, cotext, wearable computig.. INTRODUCTION Nowadays, may people have their persoal digital cameras ad video camcorders to record their preferable experieces sice digital imagig devices are available ad portable. However, they may miss some to record some iterestig experieces i their life because the recordig device is ot ready all the time. For this reaso, we have developed the wearable video system which ca capture cotiuous video ad also various evirometal features sychroously. The wearable video system was applied to record both of audio/visual iformatio ad evirometal data icludig locatios, huma s movemets ad feeligs by usig Permissio to make digital or hard copies of all or part of this work for persoal or classroom use is grated without fee provided that copies are ot made or distributed for profit or commercial advatage ad that copies bear this otice ad the full citatio o the first page. To copy otherwise, or republish, to post o servers or to redistribute to lists, requires prior specific permissio ad/or a fee. CARPE 05, November, 2005, Sigapore. Copyright 2005 ACM /05/00 $5.00. GPS receiver, motio sesors, ad brai wave aalyzer []. These data were trasferred to a persoal otebook computer. We have cotiuously developed ot oly recordig but also retrieval system to memorize our experieces as we called the Life Log system. To log our life, the amout of captured data is very large. Therefore, efficiet retrieval techiques are eeded to avigate our experieces. I our previous studies, we applied cotext iformatio based o perso s brai waves as retrieval keys to extract huma s iterest []. Audio/visual iformatio was used as cotet to detect the coversatio scees ad GPS data was applied as cotext to extract spatiotemporal key frames from time ad distace samplig [2]. I [3], a ovel cocept to itegrate various features from cotet ad cotext was itroduced to retrieve the Life Log video. There are some related works o capture ad retrieval of life experieces icludig [4], i which the user s cotext such as locatio, ecouters with other people, ad some activities were stored ad used as retrieval keys. I [5], capture by a wearable camera ad PC was developed without cosideratio of retrieval. I [6], user s real-time physiological reactios were used as triggers for switchig a wearable camera o ad off. A perso s ski coductivity, heart rate, respiratio rate, muscle activity, were also used for key detectio. I [7], Life Log video scees were classified accordig to evets detected by aalyzig the data from a wearable camera ad a microphoe. Moreover, sese wear armbad was produced by body media [8] ad also provided some beefit data for aalyzig huma s life style icludig motio data, heat flux, galvaic ski respose (GSR) ad ski temperature [9]. Thus, it is beefit to use this device to record our experieces. I this study, we preset our Life Log system i terms of recordig ad retrieval. We have applied a compact touch scree computer istead of previous otebook computer ad also associated devices icludig mii-wearable camera, USB sesitive microphoe, ad body sesors to record our experieces. We ca use body sesors armbad to capture all possible recorded experieces because it is coveiet to wear almost all the time. This armbad ca be used to record physical activities, ad also physiological data. Life Log capturig system was applied to record the iterestig evets, i which we wat to retrieve ad view the experieces. Effective Life Log video retrieval techiques were summarized. Also, a practical idexig method based o voice aotatio was itroduced. Furthermore, prelimiary aalysis ad retrieval of Life Log data based o the features from body sesor was demostrated i this paper. 6

2 The developmet of Life Log system is preseted i Sectio 2. Effective video idexig methodology icludig cotet ad cotext based techiques are explaied i Sectio 3. I Sectio 4, body sesor armbad is itroduced with their useful features for retrieval system. The experimets of practical techique based o voice aotatio ad prelimiary results from body sesor features are demostrated i Sectio 5. The last sectio is coclusio. 2. LIFE LOG SYSTEM DEVELOPMENT Persoal Life Log system was created to record our life experieces i form of multimedia iformatio. The origial system was created usig optical wearable camera, lie-i small microphoe, gyro, acceleratio sesors, ad GPS receiver coected to a otebook PC as show i Figure. Our curret Life Log system was developed for comfortably wearable usage, which cotais various practical devices icludig a compact persoal touch scree computer with mii wearable camera ad USB microphoe, motio sesor, GPS receiver ad also body sesor armbad as demostrated i Figure. We used a compact touch scree computer which is easy to carry aywhere aytime to keep our experiece data. We applied practical devices icludig mii-wearable camera, ad USB sesitive microphoe to capture audio/visual iformatio. We used GPS receiver to capture the relative locatio at the same time of recordig video. Body sesors armbad was itroduced to capture all possible recorded experieces because it is coveiet to wear almost all the time. This armbad cotais acceleratio sesors to aalyze physical activities ad was used to replace the previous motio sesor which was attached o a cap. Moreover, it ca record physiological data such as ski coductivity, heat flux, ad ski s Figure 2. Block diagram of curret Life Log data stream. Life Log retrieval system. The block diagram of data stream Life Log system is show i Figure VIDEO INDEXING METHODOLOGY There are two fudametal video summarizatio methods by represetig movig pictures ad still images. There are some sigificat differeces betwee these two methods. Still image summarizatio ca be built faster ad displayed as a story board. However, movig image summarizatio ca make more sese to display the video. We have ivestigated both of these advatages to our Life Log summarizatio by usig key frame extractio i which each key frame ca represet the movig video at certai time. We ca display the desired video by selectig the extracted key frame ad see relative locatio o GPS map. Video retrieval dialog is show i Figure 3. We ca select a preferable idexig method usig this iterface. The retrieval system ca provide multiple retrieval results based o date ad time duratio, i which we select to view our experieces. The represetative key frames i retrieval results are based o selected idexig method. The followig techiques are applied to satisfy the video retrieval system. 3. Cotext based Extractio We aalyze cotexts from GPS data icludig latitude, logitude, speed, directio ad relative time. We ca extract the key frames by usig time samplig, distace samplig, speed detectio ad also directio chagig iformatio [2]. Figure. Our Life Log system Origial system New ivestigated system temperature. Various data are trasmitted though USB ad PCIMCA port to a persoal compact computer ad aalyzed for Figure 3. Video retrieval ad idexig iterfaces. 62

3 3.2 Cotet based Extractio Cotet iformatio ca be acquired from audio/visual data which are recorded from a microphoe ad a wearable camera. Talkig scee is a example which we could apply audio/visual cotet for video idexig. We applied voice detectio to detect talkig soud by cosiderig of huma talkig characteristics ad also huma face detectio based o ski color to detect existig faces [3]. Face talkig scee detectio was applied to icrease talkig detectio accuracy. The key frames from face talkig scee detectio are demostrated i Figure 4. We assume that face talkig scee should cotai more importat talkig topic. Thus, the detected scees eed to satisfy both of voice detectio ad face detectio. 3.3 Adaptive Key Frame Extractio Geerally speakig, cotet based processig is more computatioal expesive tha cotext. Therefore, cotexts are applied to extract key frames i geeral travelig scees. However, cotets are used to extract iterestig key frames i specific evet ad whe GPS sigal is uavailable. GPS sigal is detected by aalyzig GPS data. If GPS sigal is receivable, we Figure 4. Face Talkig Scee Detectio. apply cotext based extractio from GPS data otherwise we extract key frames based o the cotets of audio/visual iformatio. We ca apply voice detectio, view detectio, speed detectio, directio chagig, distace samplig ad time samplig. The extracted key frames are cocered with time iterval to maitai the etire iformatio. I Figure 5, we demostrate the extracted key frames based o adaptive samplig by usig voice detectio, speed detectio, directio chagig ad time samplig. The extracted results cotaied satisfyig key frames such as speed detectio ad directio chagig detectio which provide some sematic scees such as stoppig at crossig way or low speed to see some iterestig places ad also offer some hits by usig voice aotatio. 3.4 Voice Aotatio Aotatio is very useful i idexig ad retrieval process. We ca make aotatios i iterestig evets ad remark desired scees. I Life Log capturig, it is coveiet to remark iterestig experieces usig voice aotatios. The characteristic of voice aotatio is differet from backgroud oise that it provides domiat frequecy bad ad gives discotiuous sigal power. O the other had, backgroud oise has smooth power cotiuously durig a time period. Voice aotatio ca be made both of short ad log seteces. I Life Log capturig, we used a small microphoe attached to user s collar. Thus, the user s voice power is quite high compared to other souds. A importat poit is to cosider backgroud oise power. Thus, a adaptive threshold geeralized from backgroud oise was applied i this purpose. Based o this method, we ca detect voice key frames from Life Log video ad sychroize them with video data ad relative GPS sigal as demostrated i Figure 6. The proposed video idexig method by usig voice aotatio is explaied as follows. Audio sigal is processed separately to idex user s voice. Firstly, audio sigal was filtered accordig to samplig rate (f s ) ad cutoff percetage (pc) to maitai low frequecy voice bad. fc = ( f s / 2) * pc where, f c is the cut-off frequecy of the low pass filter. The the filtered sigal (f i ) is divided to equal frame with period T ad calculated their sigal power (P f ). ( T 2 Pf = f i ) T i= We applied adaptive threshold for voice detectio which ca cosider the backgroud oise power. The threshold (TH ) is calculated based o audio sigal durig previous D duratio. TH P = P avg avg * W, D * W, D Figure 5. Adaptive Key Frame Extractio. Pavg = P i D i= + D where, P avg is a average power durig previous time duratio D. 63

4 W is the weight value of threshold value compared to a average power ad also a adjustable parameter to determie the amout of desired key frames (large value will provide less key frames). S 0, P < TH =, P TH Voice sigal (S ) will be detected i the case of sigal power (P ) is more tha adaptive threshold value (TH ). The detected sigal is corrected by usig some heuristic rules to complete voice period ad remove some short utteraces. Voice aotatio is a practical techique usig small computatio based o voice power ad adaptive threshold to separate evirometal souds. I additio, this techique ca be applied with cotext iformatio such as GPS data to extract the sematic key frames for more efficiet video idexig. 4. INTRODUCTION OF BODY SENSOR I our ivestigated Life Log system, we itroduced a body sesor armbad which we could wear almost all the time to examie huma s daily life. We ca use the wearable video system whe we wat to capture the preferable evets. The various data from body sesor ad Life Log cotet ad cotext ca be sychroized based o relative time ad aalyzed i Life Log retrieval system. 4. Body Sesor Armbad The body sesor sese wear armbad as preseted i Figure 7 has bee desiged to collect ad aalyze a broad rage of data from the body ad its movemet allowig us to quatify physical activity ad eergy expediture. I additio, the sese wear armbad ca also record heat flux ad temperature sesors. It ca measure heat produced by the body as a result of basic metabolism ad also all forms of physical activity. This combiatio of multiple sesors eables the sese wear armbad is advatage ad overcome may limitatios of earlier devices. The sese wear armbad cotais data collectio chaels which ca collect physiological data at a rate up to 32 times per secod as show i Figure 7. The recorded data cotais motio forces measured by a accelerometer totally 6 chaels, heat flux which is the rate of heat exchaged from a perso s arm to the outside eviromet, Galvaic ski respose (GSR) which is a measure of the electrical coductivity betwee two poits o the ski as a ski s coductivity, ski temperature, ear body temperature, ad step couter. These features are useful to be aalyzed for huma s physiological activities. Library Library Figure 6. Video idexig based o voice aotatio Key frames Evets o GPS map. Figure 7. Body sesors ad their features Body sesor armbad ad the positio of various sesors Block diagram of data chaels i body sesor armbad. 64

5 5. EXPERIMENTAL RESULTS 5. Experimets o Voice Aotatio I this experimet, we examied Life Log video sequeces i daily life o travelig scee. O the way, we made some aotatios by usig voice at remarkable places ad iterestig objects. Three Life Log video with voice aotatio o the way from uiversity to home were examied. These video sequeces were recorded from 25 to 45 miutes log ad captured i ope eviromet icludig various souds. Aotatios were made duratio travelig by user s voice i each Life Log video at some iterestig evets. The ivestigated parameters were set up as follows. f c was 0.25x f s for low pass filterig. Audio sigal was divided ito frames (T) of 25 ms ad the power of each frame was calculated. We applied the adaptive threshold based o previous average sigal power by cosiderig previous 2 secods duratio (D). Heuristic rules were used to coect a aotatio withi 5 secods ad remove short utteraces less tha 2 secods period as oise. A video idexig result based o voice aotatio is show i Figure 6. Most of the key frames had sematic meaigs due to voice aotatio icludig aotatio o buildigs, itersectio, crossig ad specific locatios ad some iterestig evets. Some detected scees cotai soud from cars ad other loud oises. However, we ca igore or remove udesired scees by user iterface. We ca also display each key frame as a movig video ad see associated locatio o GPS map. As we ca see the key frame of voice aotatio about uiversity library ad relative locatio o GPS map i Figure 6. O the other had, we ca select the locatio o the map to display a associated video. Figure 8 shows the video idexig evaluatio based o voice aotatio. We ca determie the amout of desired key frames by adjustig threshold weight value (W ). Table demostrates that whe threshold weight value is higher, the umber of key frames ad recall rate would be smaller. Thus, we eed to cosider both of recall ad precisio rates. I Figure 8, precisio ad recall rates of variable weightig factors are preseted. If we eed the highest recall rate by determiig a low threshold, a precisio rate will be low. To maitai recall rate ad keep acceptable precisio, a suitable weight is determied based o empirical experimets. I Figure 8, three Life Log video sequeces were examied based o voice aotatio ad weightig factor was determied as W=4. Video idexig results gave the recall rate 0.9,.0 ad 0.86, respectively i three video sequeces ad preserved their precisio rate over Furthermore, we ca remove uacceptable key frames by usig our Life Log iterface to maitai oly the desired evets. Table. Retrieval results of variable weightig factor for video idexig based o voice aotatio. Weightig (W) Detect Precisio Recall W= W= W= W= W=2 W=3 W=4 W=5 Video Video2 Video3 Precisio Recall Precisio Recall Figure 8. Evaluatio of video idexig based o voice aotatio. Variable weightig factor evaluatio Video idexig precisio ad recall rate. 5.2 Prelimiary Experimets o Body Sesor Various data recorded from sese wear armbad was aalyzed to explore our experieces. Physical activities were detected based o motio forces from trasverse ad logitudial accelerometer. The accelerometer is a 2-axis micro-electro-mechaical sesor (MEMS) device that measures motio. The accelerometer mea absolute differece (MAD) ca measure movemet. We used physiological data from body sesor to classify the physical activities as active ad passive movemet ad sychroize them with Life Log video based o physical activity detectio algorithm [9]. Video idexig results were examied i Figure 9. Physical activities ca be detected by usig accelerometer i body sesor. Active scees are show while ridig a bicycle ad walkig. O the other had, passive scees are preseted while shoppig i a supermarket ad selectig some goods. I additio, we ivestigated physical activities from body sesor armbad by cosiderig accelerometer MAD icludig ruig, walkig, shoppig ad sittig. This experimet demostrated the differece of accelerometer MAD for each activity as show i Figure 0. Ruig period presets high accelerometer MAD, while walkig ad shoppig have the similar MAD level but differet sigal period. MAD gives low value while sittig or o move. By cosiderig these experimets, it looks possible to apply these features to classify the user s physical activities. Furthermore, the sese wear armbad was applied to record our experieces ot oly i physical activities but also durig sleepig period. We ca estimate the activities such as lyig dow, sleep, 65

6 ad deeply sleep duratio. These activities could be aalyzed usig average value from 2-axis: trasverse ad logitudial accelerometers. While lyig dow, trasverse accelerometer average was close to gravity ad logitudial accelerometer average was aroud zero. We ca also observe our sleepig activities as show i Figure 0. The top bar presets lyig dow duratio ad the lower bar shows sleepig period. This experimet shows that user could sleep well after lyig dow. Sese wear armbad is a practical device to record our experieces sice wake up util sleep. Armbad body sesor ca also provide other useful data such as heat flux, GSR, ski temperature, ad etc. These features are related to huma s subjective feelig ad physiological data. Thus, aalysis of these features for experiece retrieval is our challegig future research. 6. CONCLUSION A ivestigated Life Log capturig ad retrieval system was explaied. Various idexig methods for Life Log retrieval system were preseted to retrieve the remarkable evets i huma s experiece ad demostrate the importace of Life Log cotet ad cotext. The experimets demostrated that voice aotatio ad talkig scee detectio were applied to retrieve sematic key frames based o audio/visual cotet. O the other had, GPS data ad body sesor features were applied as cotexts to detect oticeable key evets. Therefore, the efficiet combiatio of cotet ad cotext from Life Log data would be advatageous for practical huma s experiece recordig ad retrieval system. 7. ACKNOWLEDGMENT The author would like to thak Japaese Govermet for supportig the Mobukagakusho scholarship ad also may colleagues for database ad their helpful discussios. 8. REFERENCES [] Aizawa, K., ad Ishijima, K., Summarizig Wearable Video. Proc. Itl. Cof. of ICIP 200, Vol.3, (Oct. 200), [2] Aizawa, K., Tacharoe, D., Kawasaki, S., ad Yamasaki, T., Efficiet Retrieval of Life Log Based o Cotext ad Cotet, ACM Workshop CARPE 2004, (Oct. 2004), [3] Tacharoe, D., ad Aizawa, K., Novel Cocept for Video Retrieval i Life Log Applicatio, Pacific-Rim Coferece o Multimedia (PCM), (Dec. 2004), [4] Lammig, M., ad Fly, M. Forget-me-ot Itimate Computig i Support of Huma Memory. Proceedigs of FRIEND2, 94 It. Symp. Next Geeratio Huma Iterface, (Feb. 994), [5] Ma, S. WearCam (the wearable camera): Persoal Imagig System for Log-Term Use i Wearable Computer Mediated Reality ad Persoal Photo/Video Graphic Memory Prosthesis. Proceedigs of ISWC 98, (Oct. 998), [6] Clarkso, B., Mase, K., ad Petlad, A. Recogizig User Cotext via Wearable Sesors. Proceedigs of ISWC 00, (Oct. 2000), [7] Healey, J., ad Picard, R.W. StartleCam: a cyberetic wearable camera. Proc. of ISWC 98, (Oct. 998), [8] Lide, C.B., Wolowicz, M. Beefits of the Sese Wear Armbad over Other Physical Activity ad Eergy Expediture Measuremet Techiques, Body Media White Paper, [9] Krause, A., Siewiorek, D.P., Usupervised Dyamic Idetificatio of Physiological ad Activity Cotext i Wearable Computig, Proceedigs of ISWC 2004, Sleep Duratio Acc. Avg. Log. Tra. Figure 9. Video Idexig based o physical activity. Figure 0. Activities based o body sesor features Physical activities Sleep duratio. 66