Video- to- Video Face Matching: Establishing a Baseline for Unconstrained Face Recogni:on

Transcription

1 Video- to- Video Face Matching: Establishing a Baseline for Unconstrained Face Recogni:on Lacey Best- Rowden, Brendan Klare, Joshua Klontz, and Anil K. Jain Biometrics: Theory, Applica:ons, and Systems Washington DC, USA September 30, 2013

2 Face Recogni:on in Video Abundance of video data Ubiquity of surveillance and mobile phone cameras Low- cost digital cameras Forensic and security applica:ons 2011 London riots 2011 Vancouver riots 2013 Boston bombings

3 Face Recogni:on Scenarios S"ll- to- S"ll S"ll- to- Video Video- to- Video

4 Commercial- off- the- Shelf (COTS) Performance on S:ll- to- S:ll FR %! FAR = 0.1% FRGC Database! 84%! MBGC Database! 54%! Constrained Constrained LFW Database! Unconstrained Unconstrained

5 Approaches to Video- based FR Sequence of face images with temporal ordering Explicitly leverage temporal dynamics between frames Simultaneous tracking and recogni:on [Zhou et al., TIP, 2004] Unordered set of face images Fuse informa:on prior to matching Output single representa:on or single face image 3- D modeling [Park and Jain, CVPR, 2007] Super- resolu:on [Arandjelovic and Cipolla, ICCV, 2007] Manifold- based methods [Wang et al., CVPR, 2008] Fuse informa:on a4er matching Combine match scores from sta:c face matchers Frame subset selec:on based on face quality and/or diversity [Thomas et al., IJCV, 2007]

6 Video Face Databases

7 Mo:va:on: Representa:ve Baselines Methods that can quickly be integrated into opera:onal environments are preferred over those that have been demonstrated as proof of concept Video matching algorithms are oben compared to sta:c frame- based matching We provide a baseline accuracy for unconstrained video- based face recogni:on by using state of the art commercial- off- the- shelf (COTS) face matchers

8 Face Track Extrac"on All Frame Pairs U = u1,u2,...,ua V = v1,v 2,...,v b Mul:- Frame Fusion Same Mul"- frame Score- level Fusion: t COTS Face Matcher... s( u1,v1 )... s(u,v ) Not Same <t mean median max min Similarity Matrix... s( ua,v b ) a b

9 Fusion of Mul:ple Matchers Mul"- Matcher Mul"- Frame (MMMF) Fusion Mul"- Frame Mul"- Matcher (MFMM) Fusion Mul"- Matcher Mul"- Frame Mul"- Frame Mul"- Matcher S A... S A... S AB s AB... s A s B s AB S B S B

10 Experimental Details YouTube Faces (YTF) Database [Wolf et al., CVPR 2011]: 1,447 subjects 3,226 videos 1 6 (average 2) videos per subject 48 2,157 (average 182) frames per video Faces detected with Viola- Jones detector 24 fps Aligned and cropped to pixels Experimental protocol 10- fold, cross- valida:on, pairwise tests 250 same, 250 not same face pairs per split

11 Experimental Details COTS Matchers: COTS- A, COTS- B, COTS- C All par:cipated in 2010 NIST MBE Previous Results on YTF DB: Matched Background Similarity (MBGS) [Wolf et al., CVPR, 2011] Adap:ve Probabilis:c Elas:c Matching (APEM) Fusion [Li et al., CVPR, 2013] Spa:al- Temporal Face Region Descriptor + Pairwise- constrained Mul:ple Metric Learning (STFRD+PMML) [Cui et al., CVPR, 2013] Rank Aggrega:on [Bham et al., ICIP, 2013]

12 Experimental Results Mul"- Frame Fusion

13 Experimental Results Quality- based Frame Subset Selec"on 1 COTS B True Accept Rate (TAR) all frames (mean) 30 frames (faceness) 30 frames (near frontal) frame (faceness) 1 frame (near frontal) False Accept Rate (FAR)

14 Experimental Results True Accept Rate (TAR) COTS A (mean) 0.3 COTS B (mean) COTS C (mean) 0.2 MMMF (tanh, sum, mean) MBGS APEM Fusion 0.1 STFRD+PMML Rank Aggregation False Accept Rate (FAR)

15 Experimental Results Genuine Examples High Score Low Score Impostor Examples

16 Experimental Results Mul:- matcher fusion overcomes the face enrollment problem No. of frames that fail to enroll (587,035 total frames) Face detec:on and landmark localiza:on are crucial to leverage all available frames in a face track

17 Conclusions All three COTS face matchers outperformed current published results on the YTF database Fusion of three COTS matchers improved performance Subsequent research on face matching should use COTS matchers as baselines Face tracks contain redundant facial informa:on Quality- based key- frame selec:on can be used to reduce the number of frames for matching

18 Thank you!