Audio-Visual Voice Command Recognition in Noisy Conditions

Transcription

1 Audio-Viual Voice Command Recognition in Noiy Condition Joef Chaloupka, Jan Nouza, Jindrich Zdanky Laboratory of Computer Speech Proceing, Intitute of Information Technology and Electronic, Technical Univerity of Liberec, Czech Republic Abtract Thi paper preent an experiment of audio-viual voice command recognition from mall vocabulary in imulated noiy condition. Electric and electronic device hould be controlled by thee voice command mainly in home of motor-handicapped people and viual part of peech could improve recognition rate if a noie i relatively trong. Therefore the main aim of thi experiment were to find out how viual part of peech can improve reulting recognition rate and if it i poible to ue uccefully two-tream Hidden Markov Model (HMM) for audio-viual voice command recognition. Index Term: audio-viual peech recognition in noiy condition, voice control of electric and electronic home device 1. Introduction The main reearch area of our Laboratory of computer peech proceing i automatic continuou peech recognition mainly for Czech language. Czech language belong to the inflectional language therefore recognizer lexicon and correponding language model i larger and different than e.g. for Englih. We have developed our own on-line peech recognizer which ue a lexicon of 30,000 mot frequent Czech word and it cover 99% of Czech independent text. Thi recognizer i ued for continuou peech dictation or for automatic broadcat programme trancription ytem [1]. Our next activitie are focued on the creation of tool for motor-handicapped people. In 005, we have developed oftware for voice control of a PC by voice command the ytem MyVoice []. More than 50 motor-handicapped people are uing thi ytem at preent. Handicapped people have different electric and electronic device (TV, radio, HI- FI, DVD, light, heating, air-condition, roller blind etc.) at home but they cannot control thee device therefore a ytem for voice control of different electric and electronic device ha been created [3] for them and thi ytem i connected with program MyVoice. But the problem i that ome of thee device produce noie o the recognition rate i lower in noiy condition. It would be poible to ue ome method for filtration of noie or to ue more microphone and method for blind eparation but we wanted to ue method for audio-viual recognition becaue motor handicapped people it in front of monitor without movement and becaue viual part of a peech can improve recognition rate in noiy condition. Proceing of a viual peech ignal i time demanding but PC are relatively fat at preent or it i poible to ue cluter of PC therefore ome ytem for large-vocabulary audioviual peech recognition were developed [4] till thi time (008). Word recognition in ytem MyVoice i baed on HMM of Czech phoneme. What hould be done next i to create the Czech audio-viual databae for training the HMM of vieme. Creation of uch a databae i very time conuming and expenive therefore we have decided that a mall audio-viual databae with mall-vocabulary will be created and whole-word HMM will be trained from thi databae and thee audio-viual HMM will be ued a pecial module unit to oftware MyVoice. Vocabulary wa created from voice command for controlling of mot common home device - light, electric heating, air condition and TV. Device i witched on by one voice command and it i witched off by the ame voice command if it i pronounced for econd time therefore vocabulary ha only 0 word including voice command for controlling of TV. The main idea of thi work wa to find out how viual part of peech can improve recognition rate in noiy condition or if it i poible to ue viual peech recognition only from mall vocabulary. The firt tak wa audio and viual feature extraction for audio-viual voice command recognition:. Audio and Viual Feature Extraction.1. Audio Feature Extraction Extraction of audio feature i relatively well-olved tak in automatic audio peech recognition reearch area at preent therefore tandard method were ued in thi work too. The audio ignal wa pre-proceed and it wa divided into 33.3 m frame without overlap m frame wa elected becaue viual tream wa ampled at 30 frame (video image) per econd (fp) and we wanted to have the ame time length of audio and viual frame becaue of next audioviual feature fuion. 13 Mel-Frequency Ceptral Coefficient (MFCC) were elected a audio feature and delta (dynamic) and delta-delta (acceleration) feature were calculated from thee MFCC feature between adjacent frame o 39 audio feature were ued.... Viual Feature Extraction It i poible to divide viual feature to three group [5]: Appearance-baed viual feature, hape-baed viual feature or combination of both. It i neceary to find hape of lip in video image for extraction of hape viual feature and it can be a difficult tak therefore appearance-baed viual feature are ued more often. Appearance-baed viual feature are computed from Region of Interet (ROI) with the help of ome linear tranform like Dicrete Coine Tranform (DCT), Principal Component Analyi (PCA), Dicrete Wavelet Tranform (DWT) etc. DCT i mot popular becaue of the exitence of a fat algorithm for computation of thi tranform. We ued hape and DCT viual feature in our previou work [6] and utilization of DCT feature gave to u the bet reult therefore DCT feature were ued in thi work too. Copyright 008 AVISA Accepted after peer review of full paper September 008, Moreton Iland, Autralia

2 Figure 1: Principle of audio-viual voice command recog. A method for extraction of thee viual feature i decribed below. It i neceary to find ROI for the extraction of DCT viual feature in the firt tep:..1. Region of Interet A region of interet for viual feature extraction i a quare or rectangular area of video image which contain peaker mouth area. It i a very difficult tak to find directly peaker mouth in a real video image therefore human face i detected firt and the mouth i located in the bottom part of a detected face. We have ued a method baed on haar-like filter [7] together with color image egmentation. Some probable region are located in image by haar-like filter which were trained from face or non-face image but ome non-face object which have imilar tructure to human face are ometime elected. Therefore color image egmentation i ued in the econd tep. All pixel which have imilar color to human kin are et to value 1 other pixel have value 0. Color image egmentation method baed on threholding wa ued for thi purpoe. Threhold were et from our human kin image databae. More than 000 image of a human face are in thi databae where all pixel, which are not from human kin, were manually removed from thee image. So region from haar-like filter method are elected a human face if at leat 40% of kin-like pixel were found in thi region. An object of mouth i located in the bottom part of face by color image egmentation but threhold for thi method ha been et automatically by algorithm baed on etimation of image hitogram [8]. Centre of gravity of mouth object and width of mouth i computed next. ROI of dimenion of 18 x 18 pixel i then elected around the centre of gravity o that the width of mouth would be 100 pixel therefore the original image i reized according to the mouth width. We have only one human peaker in our video recording therefore the tak of identification who i peaking wa not olved in our work.... DCT Viual Feature The extraction of DCT viual feature i performed by dicrete coine tranform which i calculated from image of ROI. A color ROI image i tranformed to gray-level and hitogram flattening method i applied to thi image at firt and DCT coefficient are calculated from ROI after that. Viual feature are elected from DCT coefficient according to the highet energy, variance, or ome other trategie are ued, e.g. viual feature election i baed on mutual Figure : Normalization of DCT viual feature. information [9]. Viual DCT feature election baed on the highet energy wa ued in thi work becaue it i imple, fat, and it give good reult [6]. DCT viual feature are normalized after election. Normalization of DCT viual feature i very important becaue the reulting recognition rate i relatively low without normalization or it i impoible to train HMM. Our normalization i implemented a follow: Logarithmic calculation of DCT viual feature i ued in the firt tep and Feature Mean Subtraction (FMS) i applied on viual feature vector in the econd tep, ee fig.. Delta and delta-delta feature are computed from tatic DCT viual feature by caual difference y f (i, j) = x f (i, j) x f (i 1, j), where y f i delta (or delta-delta) feature, x f i tatic (or delta) feature, i i index of frame and j i index of feature in feature vector of a ingle frame. 3. Audio-Viual Integration Several different method and algorithm have been propoed for audio-viual peech data fuion, like direct integration, eparate integration, motor recoding fuion, dominant recoding fuion, etc. [5]. We have ued HTK toolkit [10] for recognition of iolated word by HMM technique o we wanted to try the poibility of feature fuion where twotream HMM are trained from audio and viual feature. Audio feature vector wa ued for the firt tream and viual feature vector for the econd tream. The output HMM tate probability b of two-tream HMM i the following: b T M ctm t 1 m1 exp 0.5 x P 1 det tm T 1 x xtm tmx xtm t Where x i feature vector (audio or viual), P i number of feature, x i feature mean vector, i covariance matrix, M i number of mixture component in one tate, c m i the weight of mixture component, T i number of tream (T = in our cae) and t i a tream weight coefficient. 4. Signal to Noie Ratio Etimation Signal to Noie Ratio (SNR) wa neceary to etimate in our audio peech ignal before experiment. SNR i calculated (1) 6

3 from power of ignal P and from power of noie P n (which i added to ignal). N 1 i0 i P i0 SNR 10.log 10.log () N 1 Pn n i where N i the number of ample from ignal, [i] are ample from clear ignal (without noie), n[i] are ample from noie ignal which i included in original audio ignal x. Our hypothei wa that noie n i additive noie therefore x[i] = [i] + n[i]. P wa etimated from the power of audio ignal P x and from P n which wa computed from non-peech part of audio ignal - peech or non-peech detector wa ued for thi purpoe. N 1 N 1 N 1 i x i n i i0 i0 i0 P Px Pn (3) N N N A problem i that SNR () i computed from all peech ignal and dynamic change of peech ignal are not covered much [11] therefore egmental ignal to noie ratio SSNR give u a better reult: N 1 M 1 j i 1 i0 SSNR 10.log (4) N 1 F j0 n j i i0 where F i the number of frame from peech ignal and M i the length of one frame in ample. SSNR from audio ignal from our video databae wa about 1dB (SNR wa about 9dB), M wa et to 67 ample (33m for 8kHz ignal). 5. Noie Addition to Speech Signal Creation of a databae of audio recording with given SNR would be very difficult therefore noie i added artificially to audio ignal in tak of audio or audio-viual peech recognition in noie condition. An algorithm for noie addition to peech ignal wa developed for our next experiment. Additive noie ignal (babble, detroyer engine1, factory1, Volvo, pink and white noie) were ued from the NOISEX databae [1]. The algorithm wa the following: SSNR (SSNR e ) wa etimated from each audio ignal and noie ignal wa added according to given relative change of SSNR (SSNR). SSNR SSNR SSNR (5) w e where SNR w i new value of SSNR in audio ignal: SSNR P P x n w 10.log (6) Pn c. Pan P an i power of additive noie and c i gain coefficient: c P an P P x n SSNRw Pn P an Reulting audio ignal x n [i] i created from original ignal x[i] and from additive noie an[i]: x n (7) [ i] x[ i] an[ i]. c, 0 i N (8) 6. Experiment Our own audio-viual databae wa created for our experiment where 35 peaker narrated 0 word voice command for controlling of different home electric device. Human peaker were camera-canned from the front by imple web-camera. Parameter of video recording from thi databae are the following: Audio ample frequency i 8kHz (one ample 16bit.). Video frame rate i 30fp and ize of a ingle video image i 640 x 460 pixel. Audio and viual peech ignal from video recording wa proceed and parameterized and whole-word two-tream HMM were trained from parameterized ignal from 5 peaker. Video recording from additional 5 peaker were ued in reearch for finding of the weight for two-tream HMM, and video recording from thee lat 5 peaker were ued for voice command recognition in imulated noiy condition. 39 audio feature (13 MFCC + delta + delta-delta) and 15 viual feature (5 DCT + delta + delta-delta) were extracted from audio and viual ignal. 5 DCT viual feature were elected becaue the bet reult were obtained with thi number of feature in our previou experiment [6] of audio-viual iolated word recognition and the audioviual databae from thi work wa created in imilar condition to the previou one. Two-tream HMM had 14 tate and 1 mixture it would be neceary to have larger databae for more mixture. SSNR from audio ignal from our databae ha been counted about 1dB. The weight for audio and viual tream were found in the firt experiment. Additive noie wa added to audio ignal from the tet part of our databae, the weight for audio and viual tream in HMM were changed from 0 to 3 with tep 0.1 and the weight for given SSNR were choen according to the bet recognition rate. Figure 3: Recognition rate from audio-viual peech recognition for different audio and viual tream weight, where white noie wa added (SSNR 14 db). SSNR in audio ignal wa changed from 1dB to -4dB and the babble, detroyer engine1, factory1, Volvo (car), pink and white noie were ued a additive noie. The reult with detroyer engine1, factory1, and pink noie were relatively imilar to the reult with babble noie therefore only the reult with babble, Volvo and white noie are preented in table no. 1. and in fig Experiment with audio peech ignal recognition only (AASR) and video peech ignal recognition only (VASR) were made after that. One-tream HMM were trained from either audio or video feature and the recognition rate for VASR ha been 79% and 99% for AASR (SSNR 1dB). Trained two-tream HMM with the weight for given SSNR from the firt experiment were ued for voice command recognition in imulated noiy condition in the lat experiment where 0 word from the lat 5 peaker from our audio-viual databae were recognized, ee table no. 1. 7

4 7. Dicuion and Summary An addition of white noie to audio ignal had larget influence on rapid decline of recognition rate. On the other hand recognition rate from audio or audio-viual voice command recognition wa relatively high for low SSNR if car noie wa added to audio ignal. It i very intereting becaue the ame algorithm wa ued for addition of both noie type. Second intereting reult wa that recognition rate (67%) from audio-viual peech recognition experiment hould be lower than reult from viual peech recognition only (79%) if the weight of audio tream i 0. Reulting recognition rate wa expected 79%. Therefore next experiment were done where two-tream HMM were trained with audio weight = 0 and viual weight wa changed from 0.1 to 10 with tep of 0.1. The bet reult from thi experiment wa 69% for viual weight = 4.5 but it in t till 79%. Table 1. Reulting recognition rate [%] for audio only (AO) and audio-viual (AV) voice command recognition depending on different noie type, SSNR and audio (AW) and viual (VW) weight in AV. Figure 4: Recognition rate from audio, viual, audioviual (audio and viual weight = 1) and audio-viual (with different weight) peech recognition for different SSNR and for car noie. SNR [db] Noie Type White Noie Babble Noie Car Noie AO AV AW VW AO AV AW VW AO AV AW VW ,7, ,9, , ,6, ,9, , , ,6, , , ,8, ,5, , ,4, ,7, , ,, ,6 1, , ,, ,5, , ,1, , ,5, ,1, ,3, , ,1, ,, ,1, , ,3, , , ,4, ,5, , ,, ,8, , ,, ,8, , , , 1, , ,1, ,4, , ,1, ,, , , ,3, , , ,, , , ,1 1, , , ,1 1, , , ,, , , ,, , ,1 85 0,1 1,7 Figure 5: Recognition rate from audio, viual, audioviual (audio and viual weight = 1) and audio-viual (with different weight) peech recognition for different SSNR and for white noie. Maybe, ome pecial adjutment of HTK recognition or trained algorithm were created for fater calculation therefore it in t poible to get ame reult for viual and audio-viual (audio weight = 0) peech recognition. It i evident that audio weight (fig. 7) are different for different noie type for ame SSNR but the noie in the SmartRoom hould be very imilar a babble noie and reult from factory and detroyer engine were very imilar a reult with babble noie. Therefore HMM trained with weight from babble noie experiment were ued in our work where audioviual voice command recognition with mall vocabulary i ued a a pecial module in our ytem MyVoice []. Figure 6: Recognition rate from audio, viual, audioviual (audio and viual weight = 1) and audio-viual (with different weight) peech recognition for different SSNR and for babble noie. 8

5 Figure 7: Selected audio weight for AV ASR in different noiy condition. Human peaker i camera canned, SSNR i evaluated from audio ignal and applicable two-tream HMM are elected according to SSNR. Parameterization of one econd of audioviual peech ignal take about 1.5 econd (PC 3.4GHz) therefore it i poible to ue our audio-viual voice command recognizer in real condition. Utilization of two-tream HMM i poible only for mall vocabulary becaue we need everal HMM for ingle word (for different SSNR). Therefore it i better to ue ome another audio-viual fuion method for audio-viual peech recognition with large vocabulary. Experiment with car noie wa developed becaue we have wanted to know what influence ha a car noie on audio-viual peech recognition and the reult will be ued in our next work. 8. Acknowledgement The reearch wa upported partly by the Grant Agency of the Czech Academy of Science (grant no.1qs ) and partly by the internal grant provided by the Faculty of Mechatronic at the Technical Univerity of Liberec. 9. Reference [1] Nouza, J., Zdanky, J., Cerva, P. and Kolorenc J., Continual On-line Monitoring of Czech Spoken Broadcat Program, In: ICSLP 006, USA, pp , ISSN , 006 [] Nouza, J., Nouza, T. and Cerva, P., A Multi-Functional Voice- Control Aid for Diabled Peron, Proc. of Specom 005, Greece, pp , ISBN x, 005 [3] Nouza, J., Chaloupka, J., Zdanky, J., Silovky, J., Kroul, M. and Mader, Z., Voice Controlled Center for Home of Motor- Handicapped Peron, In: Speech and Computer International Conference, Ruia, pp , ISBN X, 007 [4] Potamiano, G., Neti, C., Iyengar, G. and Helmuth, E., Largevocabulary audio-viual peech recognition by machine and human, Proc. Europeech, Aalborg, 001. [5] Potamiano, G., Neti, C., Gravier,G., Garg, A. and Andrew, W., "Recent Advance in the Automatic Recognition of Audio- Viual Speech", Proc of the IEEE, Vol. 91, Iue 9, pp , ISSN , 003 [6] Chaloupka, J., Fat Method for Extraction of the Viual Speech Feature for Audio-Viual Speech Recognition, In: Specom 005, Greece, pp , ISBN x, 005 [7] Hong, K., Min, J., Lee, W. and Kim, J., Real Time Face Detection and Recognition Sytem Uing Haar-Like Feature/HMM in Ubiquitou Network Environment, In Lecture Note in Computer Science, Springer Berlin, Volume 3480, pp , ISBN , 005 [8] Chaloupka, J., Automatic Lip Reading for Audio-Viual Speech Proceing and Recognition, In: Proc. of ICSLP 004, Korea, pp , ISSN x, 004 [9] Scanlon, P., Potamiano, G., Libal, V. and CHU, S., M., Mutual Information Baed Viual Feature Selection for Lipreading In ICSLP 004, Korea, ISSN x, 004 [10] Steve, Y., Odel, J., OLLASON, D., Valtchev, V. and Woodland, P., The HTK Book, verion.1., In Cambr. Univ., UK, 1997 [11] Martin, R., An efficient algorithm to etimate the intantaneou SNR of peech ignal, In Europeech 1993, Germany, pp , 1993 [1] Varga, A., P., Steenekan, H., J., M., Tomlinon, M. and Jone, D., The noiex-9 tudy on the effect of additive noie on automatic peech recognition, Tech. Rep., DRA Speech Reearch Unit, 199 9

6 30