Quality Metrics for Pattern Evidence: Development and Evaluation

Transcription

1 Quality Metrics for Pattern Evidence: Development and Evaluation SAMSI Working Group #2: Karen Kafadar Department of Statistics University of Virginia Faithful participants: H. Eldridge, R. Falk, S. Huckerman, A. Luby, V. Mared, A.Rairden,H.Swofford Acknowledgements: H. Iyer, A.P. Peskin, E. Tabassi (NIST) 1

2 Goal of Working Group #2 Can we develop an objective metric of quality (QM) for pattern evidence, and correlated it with accuracy? Start with latent prints & ACE-V ( Universal Definition ) 1. A : Analysis Current definition: fundamental inventory of available informational components to be used in a comparison (qualitative) Goal: Developquantitative,repeatablequalitymetric Assessment of LP quality used in C Goal: Correlate QM with accuracy of call Subsequent slides: Some proposed QMs to be evaluated 2

3 2. C : Comparison Essentially, feature (minutiae) identification prediction of specific information, which has been identified and spatially located within one impression or object, then predicted to exist within the comparison exemplar Lights-out mode in AFIS: objective selection of features but proprietary to vendor Feature identification is not part of this project 3. E : Evaluation Identification, Exclusion, Indeterminate Alternative approaches: likelihood ratios (Neumann) Identify sources of variation that affect E 4. V : Verification (often not blind to 1st decision) 3

4 !"#$%!&'(!)*+,-.+/0'+1234' 5.'41-' '89' (488:0;'!"#$%!&' <38<87-='2' =-+,7,8.'+1234' South Dakota State University 4

5 2. Sources of Data: This study, and the AFIS LP accuracy study, need a set of ground truth pairs (Q and known S) Others (e.g., Jain) have used NIST SD27a NIST Special Database SD27a: Fingerprint Minutiae from Latent and Matching Tenprint Images (IR-6534), MD Garris & RM McCabe: The minutiae identified on each image in this database were validated by at least two professional latent examiners at the FBI. Therefore, the reported minutiae should be considered reliable and useful as ground truth for measuring performance (p7,l.4) Ideally, known source DFSC has 300 Q-S pairs (research only) We need reliable Q-S pairs, of varying quality levels. 5

6 3. Evaluation: NIST collaboration Peskin: Kafadar worked with Peskin on a quality metric for imaging biological cells (see below) NISTIR 8034, Fingerprint Vendor Technology Evaluation 4. Outreach: Involving LP laboratories: Pilot: QM algorithms in crime labs (DFSC, VADFS, HFSC) Refine, improve, combine QMs for maximum accuracy Forensic science journals & conferences (AAFS, IAI, etc.) 5. Performance Metrics was research useful for forensic science? Create data-based graph showing estimated accuracy Continually monitor accuracy over time (new Q-K pairs) Workflow metrics: Does the QM save LPE time? 6

7 Proposed Quality Metrics Underlying premise: High-quality minutiae More accurate calls Peskin (NIST) & Kafadar: gradient of contrast intensity around feature Swofford (DFSC): DFIQI score Hicklin, Buscaglia, Roberts (FSI, Mar 2013): Latent Quality Assessment Software (LQAS) Yoon, Liu, Jain (MSU): Proceedings of 5th Int l Workshop on Computational Forensics, Nov2012 7

8 Neumann: based on pixel quality (10%,..., 100%) of feature on image & of configuration around specific feature; also #, rarity (cf. DNA markers w/many alleles) Tabassi et al (2004): NFIQ 5-point quality scale for exemplars (contrast, feature clarity), tuned to matcher s performance (high [low] quality good [poor] match performance); NFIQ 2.0 for LPs (with Jain, Yoon, Cao, Liu 2016): nfiq 2.cfm Huckeman & Richter: Wavelet-basedmeasureofquality How useful in practice? LPE perspective: Eldridge, Rairden 8

9 Peskin et al. 2010: Measure cell image quality (NIST) Choice of segmentation based on edge quality Calculate numerical gradients from cell to background Next slide: gradient curves cells imaged under different conditions & edge qualities Plot gradient vs pixel intensity: gradient peaks at edges Sharper images sharper peaks (higher gradients) Quality metric = f( g max, g max ) =maximumgradient and derivative at that peak 9

10 Left: gradient magnitude vs pixel intensity for a cell imaged with 3 different exposure times Right: imaged with an optimized vs non-optimized filter 10

11 Apply concepts (gradient, contrast) to fingerprint images: Quality metric = Q i =functionof g max, g max, andcontrast (measured by range of pixel intensities) in small neighborhood of fingerprint feature i Repeat for manually-identified features (minutiae) Normalize Q i to [0,100], Q =averageq i How well does it work? Simulate increasingly degraded print via blurring; fingerprint quality score decreases accordingly How does quality score relate to accuracy of identification? (quality threshold) 11

12 12

13 13

14 14

15 15

16 Quality scores for 3 minutiae in 8 increasingly degraded images: Image# left center right (red) (blue) (green)

17 Details of quality metric: Underlying principle: LPE can distinguish features in blurry print by recognizing: 1. gradient of intensity between light & dark regions around a feature (minutia point) 2. overall contrast of intensity values around feature. Step 1: Locate the pixel in small neighborhood of feature that has the highest gradient Neighborhood: 10 pixels in each direction (considered other sizes) For each pixel (x, y) inthisneighborhoodsurroundingfeature f 0,sayN(f 0 ), compare pixel intensity i(x, y) toneighboring pixel intensity i(x + n, y + m) (n, m =-10,-9,...,9,10) 17

18 Divide by corresponding distance between pixels to get a measure of the gradient at pixel location (x, y): g(x, y; x,y )=[i(x, y) i(x,y )]/(n 2 + m 2 ) 1/2, n (x x ); m (y y ); n, m = 10,...,,10. Define the set of all 24 gradients in this neighborhood of (x, y) asg 10 (x, y) Define (x 0,y 0 )asthatpointintheneighborhoodthat maximizes G 10 (x, y); i.e., (x 0,y 0 ) argmax (x,y) G 10 (x, y) n (x x ); m (y y ); n, m = 10,...,,10. Max Gradient = G 10 (x 0,y 0 )(largestgradientinthesetg 10 of 24 gradients). Contrast: Largest contrast between (x 0,y 0 )anditsimmediate 3 3neighborhood: 18

19 contrast(x 0,y 0 )=max( i(x, y) i(x 0,y 0 ) /I M I M =maximumpixelintensity(e.g.255) (x, y) =(x 0 + n, y 0 + m) where(n, m) =-3,-2,...,2,3. contrast measurement highlights maximum change in the intensity among all 9 points surrounding the feature at (x 0,y 0 ) Quality metric = G 10 (x 0,y 0 ) contrast(x 0,y 0 ) (product of maximum gradient and contrast). Quality metric can exceed 100.0, but such quality metrics appear to be rare: of 15,008 minutiae from the 273 latent prints in NIST s SD27a database, quality metrics for only 54 (0.35%) exceed (truncate to 100 here). dbases.cfm ( good, bad, ugly ). 19

20 Swofford s DFIQI score: Combines 5 measures: Signal Percent Pixels Per Grid (S3PG): % of pixels in entire image with intensities above threshold (parameter: threshold) (Contrast) Bimodal Separation: 0.5( S B)/(σ S + σ B ) Accutance: Measures average sharpness in ROI (Choong et al 2003, Br J Ophthalmol): Average squared difference in pixel intensity between center & 8 neighbors: ( 8 n=1 (I C I n ) 2 )/(#) (parameters: 3 3 grid,#grids) Ridge width (large width distortion or background noise) Spatial frequency of peak magnitude DFIQI involves mean of these 5 (scaled) scores 20

21 H. Swofford, A Novel Approach for Quantifying the Weight of Fingerprint Evidence, NIJ Impression, Pattern, and Trace Evidence Symposium, San Antonio, Aug 2015: 1. Detect point patterns (PPs) in the two images and each feature s relationship to one another; 2. Orient PPs so Euclidean distances among features in the two images is minimized; 3. Map PPs into feature triplets (i.e. all possible triangles thatcanbe generated using the PP); 4. Calculate a measure of (dis)similarity of paired feature triplets; 5. Compare distributions of measured (dis)similarity of feature triplets in same [possibly distorted] source vs different source pairs; 6. DFIQI = midmean of (dis)similarities in same source prints. 21

22 Hicklin et al. (2014): LQAS Generates local clarity map (per ANSI/NIST-ITL standard) using gray level count/median/range, curvature, change in ridge orientation, magnitude of low frequency, maximum ridge magnitude, valid neighborhood, normalized magnitude, NIST MINDCT quality map LQMetric = estimate of probability that NGI image-only (LFIS) search would hit at rank 1 if mate NGI (ex: LQMetric = 80 latent is 80% likely to hit at rank 1 if the mate database LQMetric better at predicting high quality than low quality 22

23 Yoon, Lui, Jain (2012): On Latent Fingerprint Image Quality Define latent value determination Define set of features based on ridge clarity & minutiae Evaluate features for value in individualization Propose LFIQ = latent fingerprint image quality, to reject latents that cannot be identified with specific AFIS contrast enhancement (smoothing) Fourier analysis: Locations of largest two amplitudes Ridge continuity (RC) map (from Fourier analysis) Average minutiae quality metric Q M from RC map match score =weightedaverageofscoresfromlatent& reference: weight is empirically chosen to obtain the best rank-100 identification accuracy (p9) 23

24 Objective analysis steps: ACE-V (Evaluation) Subjective identification of corresponding minutiae Neumann et al. (2011): Calculate for each minutiae: radius, side length, angle, area, type (5-dimensional vector) Distribution of features in matching vs non-matching prints Likelihood ratio : LR = P { match dif<δ} P { match dif>δ} Objective comparison if minutiae selection were objective Note LR is not real life ; need posterior odds Posterior odds = Prior odds LR P{dif<δ match } P {dif>δ match } =Priorodds LR 24

25 AFIS LP Accuracy Study Identify population of examiners, latent prints, AFIS Randomly select examiners and prints for study Construct test sets (includes matches, non-matches) Include factors: print quality, digitization, #minutiae, AFIS system, lab, repetition,... Double-blind: Neitherexaminernoradministratorknows Collect concomitant information (experience, #points,...) Obtain information on non-responders Randomize to balance any factors not considered in design (e.g., presentation of sets to examiners) 25

26 Factors in experimental design: 1. Mode (2 levels): Lights-out vs not-lights-out 2. System (3 levels: NEC, Morpho, Cogent; possibly 7) 3. Size of database (3 levels: small, medium, large ) 4. Level of print difficulty (3 levels: easy, medium, hard ; levels to be determined by values of best quality metric) 26

27 Challenges: Multi-factor study Same AFIS system can connect to different databases (performance depends on size of database) K on Q-K pair must be inserted into database Conduct study with little overhead (cannot interfere with crime lab processes) Barriers: Culture, Confidentiality, Costs, Data Access 27

28 6. Summary of Needs Mathematical characterization of QMs? Known Q-S pairs (same, different) Algorithms for realistic distortions Feasibility of study of AFIS LP accuracy Identifying LPEs to participate Logistical organization of study $forconduct&implementation(vsdevelopment) Transition project to CSAFE support (NIST permitting) 28

29 References Box GEP; Hunter WF; Hunter JS (2005), Statistics for Experimenters Bradford T; Ulery R; Hicklin RA; Buscaglia J; Roberts M (2011), Accuracy and reliability of forensic latent fingerprint decisions, 108 PNAS, Kafadar K (2015), Statistical Issues in Assessing Forensic Evidence, International Statistical Review: Neumann C; Evett IW; Skerrett J (2011), Quantifying the weight of evidence from a forensic fingerprint comparison: a new paradigm, JRoyalStatSocA175(2), Peskin A; Kafadar K; Dima (2009), A Quality Pre-Processor for Biological Cell Images, Proc. 5th Int l Symposium on Visual Computing: Peskin AP; Kafadar K: A new measurement of quality for minutiae in latent fingerprints, preprint. 29

30 Spiegelman CS; Kafadar K (2006), Data Integrity and the Scientific Method: The Case of Bullet Lead Data as Forensic Evidence, Chance 19(2): Ulery BT; Hicklin RA; Buscaglia J; Roberts MA (2011), Accuracy and reliability of forensic latent fingerprint decisions, PNAS 108(19), Ulery BT; Hicklin RA; Buscaglia J; Roberts MA (2012), Repeatability and reproducibility of decisions by latent fingerprint examiners, PLoS ONE 7(3). Yoon S; Liu E; Jain AK (2012), On latent fingerprint image quality, Proc 5th Int l Workshop on Computational Forensics, Tsukuba, Japan, 11 Nov