Direct Matrix Factorization and Alignment Refinement: Application to Defect Detection Zhen Qin (University of California, Riverside) Peter van Beek & Xu Chen (SHARP Labs of America, Camas, WA) 2015/8/30 1
Background and Challenges Goal: Reliable and general automatic defect detection method Challenges: Multiple classes of defects, with high intra-class variance, arbitrary shape and size, weak intensity, object class specific Background may not be uniform or uniformly textured (may include object boundaries) Should have high detection performance, spatially accurate (pixel-wise) and scalable (computational cost) Limited training data 2015/8/30 2
Why existing methods do not work? Existing methods are largely ad-hoc, and make assumptions that are hard to generalize during algorithm design. Methods Assumptions Training based method (detection by classification) Anomaly detection (e.g. Phase-only Transform method, CVPR 2010) Normalized cross correlation (popular in industry) Golden Matching (direct differencing) (popular in industry) Large training set with labels, predictable defect features Uniform background or uniformly textured background, small defects Linearly correlated template, precise alignment, complex background, small defects Precise alignment, an identical template Require manual adjustment of parameters Require nontrivial pre/post-processing (more parameters ) Require manual adjustment of alignment or landmark points Design and use different/more complex systems for different applications and even new datasets 2015/8/30 3
Examples NCC NCC (Normalized Cross Correlation) does not work well it assumes complex background, and requires precise alignment with a template Input: NCC: Our result: 2015/8/30 4
Examples SSD ( Golden Matching ) Works well for defects with high intensity, but not weak ones. Also requires precise alignment with a template. Input: Golden Matching Our result 2015/8/30 5
Examples PHOT (CVPR 2010) Severe false alarms for non-uniform backgrounds, fails for large defects Input: PHOT Our result 2015/8/30 6
What we want A general algorithm as few and as realistic assumptions as possible Template differencing approach assumes a linearly correlated template image is available Relax the precise registration (alignment) algorithm assumption Dilemma: defect detection needs good alignment; however, defects complicate precise alignment Relax the need of an identical template image: robustness to existence of local distortions Relax the assumption of small defects: defects can be large 2015/8/30 7
Overview of Proposed Method 2015/8/30 8
Problem Formulation If we have well-aligned, defect-free single-channel input image I 1 0 and template images I 2 0,, I n 0 R w h, then A = vec I 1 0 vec I n 0 R m n should be low-rank (linearly correlated columns) But, in practice, we observe images I i = I 0 1 i + e i τ i containing defects (modeled by additive error components e i ) and mis-alignment (modeled by transformations τ i ) and containing noise [Formulation largely follows that of Peng et al., PAMI 2012, RASL: Robust alignment by sparse and low-rank decomposition. ] 2015/8/30 9
Problem Formulation cont. So the idea is to decompose the aligned observed image matrix D τ = vec I 1 τ 1 vec I n τ n R m n as D τ = A + E + ε where A is the defect-free aligned image matrix (low-rank part), E is the error (defect-related) image matrix, and ε is entry-wise noise We can formulate an optimization problem: min A,E,τ D τ E A F s. t. rank A K E 0 γ 2015/8/30 10
Optimization May be effectively and efficiently optimized (robust rank minimization, robust principal component analysis) Some algorithms (RASL) are relaxed versions (nuclear norm, L 1 norm) The most relevant work does not consider alignment We optimize directly in the primal form due to its decomposable structure We refer to our algorithm as DFAR: Direct Factorization and Alignment Refinement 2015/8/30 11
Optimization Algorithm Iteratively linearize and refine alignment common for alignment algorithms 2015/8/30 12
Optimization Algorithm 1) truncated Singular Value Decomposition (SVD) n approximation [1] to D τ + i=1 J i τ i ε T i E, two lines of Matlab code 2) Least square for each image pseudoinverse 3) Error detection problem with L 0 -norm constraints, can be solved by [2] with two lines of Matlab code Less than 10 lines of code in total [1] R. M. Larsen, Propack software for large and sparse SVD calculations, http://soi.stanford.edu/~rmunk/propack [2] Z. Lu and Y. Zhang, Penalty decomposition methods for l0-norm minimization, Simon Fraser University, Dept. of Math., Tech. Report, 2010. 2015/8/30 13
Trick 1 decomposing template-guided matrix Only two images (1 input and 1 template) errors may be encoded in the low-rank component, or evenly distributed Problem when errors are not sparse Idea: template guided low-rank decomposition 1 vs 1 -> 1 vs n (now n=3) We have E 0 γ, so defects can be big (in terms of the input), since it is sparse in terms of the whole matrix 2015/8/30 14
Trick 2 downsampling Guaranteed speed-up Better at catching large misalignment More robust to local distortions between input and template Original Downsampled 4x 2015/8/30 15
Evaluation Data-set: 152 images of LCD panel inspection data-set Each image contains circuit patterns, slowly varying background, and possibly defects. There are 5 types of defects. Pixel-based defect mask comparison with manually labeled Ground Truth Precision and Recall Methods DFAR (proposed) DFAR without alignment refinement RASL (PAMI 2012) PHOT (CVPR 2010) PHOT-R (refined) NCC (Normalized Cross Correlation) SSD ( Golden Matching ) 2015/8/30 16
Sample results 1 2015/8/30 17
Sample results 2 2015/8/30 18
Sample results 3 2015/8/30 19
Results: Type 0 Defects 2015/8/30 20
Results: Type 1 defects 2015/8/30 21
Results: Type 2 defects 2015/8/30 22
Results: Type 3 defects 2015/8/30 23
Results: Type 4 defects 2015/8/30 24
Observations and Analysis Robust rank minimization algorithms (DFAR, DF and RASL) consistently outperform other methods by a large magnitude, validating our idea of applying robust rank minimization to defect detection. DFAR and RASL generally perform better than DF (DFAR without alignment refinement), indicating the effectiveness of alignment refinement. The proposed algorithm (DFAR) always gets comparative or slightly better result than RASL, thus best performance overall, while possessing faster convergence than RASL. The performance gain comes from noise modeling and the more faithful optimization method. 2015/8/30 25
Observations and Analysis II NCC performs worst. NCC does not generalize directly to inspected objects with relatively large low-contrast regions. PHOT does not perform well as expected, as it is designed for uniformly textured images. We observe a lot of false alarms along landmark edges. Our modified PHOT-R gets good results when the defects are small and intense, but not when the defect is large. Standard SSD (Golden Matching) achieves reasonable results, especially for defects with strong intensities. 2015/8/30 26
Effectiveness of template-guided matrix All images 2015/8/30 27
Down-sampling Type 4 defects 2015/8/30 28
Sample result - Generalization 2015/8/30 29
Sample result - Generalization 2015/8/30 30
Conclusions Novel, effective, and efficient algorithm for defect detection. Algorithm is based on robust rank minimization methods. Relaxes need for precise alignment of input and template images. The algorithm is general, requires no application-specific postprocessing, and requires few parameters useful for practical application (by non-expert users). Demonstrated very good precision/recall performance. Can be scalable: Matlab code runs in ~1 second per image (original size 1024x768, applied 4x down-sampling). May be further improved via C implementation, GPU, parallelization). 2015/8/30 31