Local Binary Pattern (LBP) methods in motion and activity analysis

Transcription

1 Local Binary Pattern (LBP) methods in motion and activity analysis Matti Pietikäinen University of Oulu, Finland Texture is everywhere: from skin to scene images

2 Starting point 2-D surface texture is a two dimensional phenomenon characterized by: spatial structure (pattern) contrast ( amount of texture) Transformation Gray scale Rotation Zoom in/out Property Pattern Contrast no effect affects affects no effect affects? Thus, 1) contrast is of no interest in gray scale invariant analysis 2) often we need a gray scale and rotation invariant pattern measure Local Binary Pattern and Contrast operators Ojala T, Pietikäinen M & Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29: An example of computing LBP and C in a 3x3 neighborhood: example thresholded weights Pattern = LBP = = 241 C = ( )/5 - (5+2+1)/3 = 4.7 Important properties: LBP is invariant to any monotonic gray level change computational simplicity

3 Multiresolution LBP Ojala T, Pietikäinen M & Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7): arbitrary circular neighborhoods - uniform patterns - multiple resolutions - rotation invariance - gray scale variance as contrast measure Uniform patterns U=0 Uniform patterns (P=8) U=2 Examples of nonuniform patterns (P=8) U=4 U=6 U=8

4 Texture primitives detected by the LBP Spot Spot/flat Line end Edge Corner Estimation of empirical feature distributions Input image (region) is scanned with the chosen operator(s), pixel by pixel, and operator outputs are accumulated into a discrete histogram LBP P,R riu2 LBP P,R riu2 / VAR P,R LBP P,R riu2 P+1 VAR P,R Joint histogram of two operators VAR P,R B-1 VAR P,R LBP P,R riu2 LBP P,R riu2 / VAR P,R

5 LBP has been highly successful LBP has become widely used in various applications due to its high discriminative power, tolerance against illumination changes and computational simplicity. Among the applications are: Visual inspection Image and video retrieval Biomedical image analysis Aerial image analysis, remote sensing Facial image analysis Etc. For a bibliography of LBP-related research, see Examples of using LBP in other groups - Object detection: Zhang et al., On-line boosting: Grabner & Bishof, Object classification: Lisin et al., 2005; Autio Color-texture based indexing: Yao & Chen, 2003; Connah & Finlayson, Inspection of ceramic tiles: Lopes, 2005; Novak & Hocenski, Classification of underwater images: Marcos et al., 2005; Clement et al., 2005; Blaschko et al., Aerial image segmentation: Urdiales et al., Segmentation of multispectral remote sensing images: Lucieer et al., Intravascular tissue characterization: Pujol & Radeva, Mobile robot navigation: Hong et al., 2002; Davidson & Hutchinson, Steganalysis for stenography: Lafferty & Ahmed, Designing aesthetically interesting and informative displays: Fogarty et al., Ovehead view person recognition: Cohen et al., Face recognition: G. Zhang et al., 2004; W. Zhang et al. 2005; Li et al., 2006; Rodriguez & Marcel, Face detection: Jin et al., Facial expression recognition: Shan et al ; Liao et al., Gender classification: Sun et al., 2006; Lian & Lu, 2006

6 Face analysis using local binary patterns Face recognition is one of the major challenges in computer vision We proposed (ECCV 2004, PAMI 2006) a face descriptor based on LBP s Our method has already been adopted by many leading scientists - e.g. T.S. Huang, J. Kittler, S.Z. Li, W. Gao, H. Ai, B. Triggs, S. Gong, S. Marcel Outstanding results in face recognition and authentication, face detection, facial expression recognition, gender classification Our approach will have a significant role in a new EU project Mobile Biometry ( ) coordinated by IDIAP (Switzerland) Face description with LBP Ahonen T, Hadid A & Pietikäinen M (2006) Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12): (an early version published at ECCV 2004) A facial description for face recognition:

7 LBP in AuthenMetric F1 Institute of Automation, Chinese Academy of Sciences FP7 project: Mobile Biometry (MOBIO) ( The aim of is to investigate multiple aspects of biometric authentication based on the face and voice in the context of mobile devices To increase security and user acceptance - using standard sensors already available on mobile phones Coordinator: IDIAP Research Institute (CH) Partners: University of Manchester (UK), University of Surrey (UK), Universite d Avignon (FR), Brno University of Technology (CZ), University of Oulu (FI), IdeArk (CH), eyep Media (CH) A technology transfer tool referred to as MOBIO Community of Interest will be formed

8 Subtracting the background and detecting moving objects Heikkilä M & Pietikäinen M (2006) A texture-based method for modeling the background and detecting moving objects. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(4): (an early version published at BMVC 2004) Overview of the Approach We use an LBP histogram computed over a circular region around the pixel as the feature vector. The history of each pixel over time is modeled as a group of K weighted LBP histograms: {x 1,x 2,,x K }. The background model is updated with the information of each new video frame, which makes the algorithm adaptive. The update procedure is identical for each pixel.

9 Detection of moving objects A texture based method for modeling the background and detecting moving objects Dynamic texture descriptors for motion analysis Dynamic (or temporal) textures are textures in motion We proposed (PAMI 2007) simple spatiotemporal LBP descriptors for dynamic texture recognition outperforming the state-of-the-art This approach has been applied to facial expression regonition (PAMI 2007), face and gender recognition from video sequences (AMFG 2007, ICPR 2008), visual speech recognition (HCM2007), and recognition of actions (BMVC 2008) - with excellent results Our approach has potential for significant contributions in many applications and fundamental problems of motion and activity analysis

10 Dynamic texture recognition Zhao G & Pietikäinen M (2007) Dynamic texture recognition using local binary Determine the emotional state patterns with ofthean face application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6): (parts of this were earlier presented at ECCV 2006 Workshop on Dynamical Vision and ICPR 2006) Dynamic texture Motivation Dynamic textures or temporal textures are textures with motion. There are lots of DTs in real world, including sea-waves, smoke, foliage, fire, shower and whirlwind, etc. Click the figure

11 Volume Local Binary Patterns (VLBP) Sampling in volume Thresholding Multiply Pattern LBP from Three Orthogonal Planes (LBP-TOP)

12 DynTex database Our methods outperformed the state-of-the-art in experiments with DynTex and MIT dynamic texture databases

13 Facial expression recognition Zhao G & Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6): Determine the emotional state of the face Regardless of the identity of the face

14 (a) Non-overlapping blocks(9 x 8) (b) Overlapping blocks (4 x 3, overlap size = 10) (a) Block volumes (b) LBP features from three orthogonal planes (c) Concatenated features for one block volume with the appearance and motion Database Cohn-Kanade database : 97 subjects 374 sequences Age from 18 to 30 years Sixty-five percent were female, 15 percent were African-American, and three percent were Asian or Latino.

15 Happiness Angry Disgust Sadness Fear Surprise Comparison with different approaches [Shan,2005] People Num 96 Sequence Num 320 Clas s Num 7(6) Dynamic N Measure 10 fold Recognition Rate (%) 88.4(92.1) [Bartlett, 2003] N 10 fold 86.9 [Littlewort, 2004] N leave-onesubject-out 93.8 [Tian, 2004] N [Yeasin, 2004] Y five fold 90.9 [Cohen, 2003] Y Ours Y two fold Ours Y 10 fold 96.26

16 Demo For Facial Expression Recognition Low resolution No eye detection Translation, in-plane and out-ofplane rotation, scale Illumination change Robust with respect to errors in face alignment Visual Speech Recognition Zhao G, Pietikäinen M & Hadid A (2007) Local spatiotemporal descriptors for visual speech recognition. Proc. 2nd International Workshop on Human- Centered Multimedia (HCM2007), Augsburg, Germany, Visual speech information plays an important role in speech recognition under noisy conditions or for listeners with hearing impairment. A human listener can use visual cues, such as lip and tongue movements, to enhance the level of speech understanding. The process of using visual modality is often referred to as lipreading which is to make sense of what someone is saying by watching the movement of his lips. McGurk effect [McGurk and MacDonald 1976] demonstrates that inconsistency between audio and visual information can result in perceptual confusion.

17 Appearance Features-Local Spatiotemporal Descriptors For Visual Information Mouth region images (a) Volume of utterance sequence (b) Image in XY plane (147x81) (c) Image in XT plane (147x38) in y =40 (d) Image in TY plane (38x81) in x = 70 LBP-XY images LBP-XT images Overlapping blocks (1 x 3, overlap size = 10). LBP-YT images Features in each block volume. Mouth movement representation.

18 Experiments Two databases: 1) Our own visual speech database: 20 persons; each uttering ten everyday s greetings one to five times. Totally, 817 sequences from 20 speakers were used in the experiments. C1 C2 C3 C4 C5 Phrases included in the dataset. Excuse me C6 See you Good bye C7 I am sorry Hello C8 Thank you How are you C9 Have a good time Nice to meet you C10 You are welcome 2) Tulips1 audio-visual database 12 subjects, pronouncing the first four digits in English two times in repetition. Totally 96 sequences. Experimental Results-Own database Speaker-independent: Mouth regions from the dataset. Recognition results (%) x5x3 block volumes 1x5x3 block volumes (features just from XY plane) 1x5x1 block volumes 0 C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 Phrases index Results of speaker-independent experiments. Eye detection Blocks (1x5x3) Blocks (1x5x3+1x5x2) Manual % Automati c %

19 Experimental Results-Tulips1 audio-visual database Mouth images with translation, scaling and rotation from Tulips1 database. Comparison to other methods on Tulips1 audio-visual database (speaker independent). [Arsic 2006] [Arsic 2006] [Gurban 2005] Features MRPCA MI MRPCA Temporal Derivatives Features Normalization Y Y Y Results (%) Ours LBP TOP 8,8,8,1,1,1 Blocks: 3x6x2 N Demo for visual speech recognition

20 Recognition of actions 2D texture based approach w 1 w 2 w 3 w 4 V Kellokumpu, G Zhao & M Pietikäinen, "Texture Based Description of Movements for Activity Analysis". In Proc. VISAPP 2008 Demonstration

21 Dynamic texture based approach yt xt Dynamic texture based approach Feature histogram of a bounding volume V Kellokumpu, G Zhao & M Pietikäinen, Human Activity Recognition using a Dynamic Texture Based Approach". BMVC 2008.

22 Dynamic texture based approach KTH - dataset Box Clap Wave Jog Run Walk Box Clap Wave Jog.860, Run Walk SVM - 93,8% DT segmentation Chen J, Zhao G & Pietikäinen M (2008) Unsupervised dynamic texture segmentation using local spatiotemporal descriptors, ICPR 2008, in press.

23 A demo show Segmentation of a dynamic texture Input Output Experimental results Results on sequences ocean-fire-small (a) Frame 8 (b) Frame 21 (c) Frame 40 (d) Frame 60 (e) Frame 80 (f) Frame 100

24 Experimental results Results on a real challenging sequence (a) Frame 5 (b) Frame 10 Summary LBP and its spatiotemporal extensions are very effective methods for motion and activity analysis Our recent reseach has foced on applications in detection and tracking of moving objects, face, facial expression and gender recognition from videos, visual speech recognition, recognition of human actions, gait recognition The methods should be powerful in various industrial problems - computationally simple - robust to illumination variations - robust to localization errors