Segmentation of 3D Materials Image Data

Transcription

1 Enhanced Image Modeling for EM/MPM Segmentation of 3D Materials Image Data Dae Woo Kim, Mary L. Comer School of Electrical and Computer Engineering Purdue University

2 Presentation Outline - Motivation - SEM Imaging Modes - 2D and 3D Blurring Models - Expectation-Maximization/Maximization of the Posterior Marginals (EM/MPM) Segmentation - 2D and 3D Joint Deconvolution/Segmentation (JDS) - New prior model: Minimum Area Increment (MAI) - 3D EM/MPM - Results & Conclusions

3 Motivation Scanning electron microscope (SEM) images have blurring due in part to complex electron interactions during acquisition One particular problem that arises during segmentation is necking: the merging of particles thatt do not appear to touch in the original i image data We incorporate model for blurring degradation into the original EM/MPM method in order to reduce necking We also introduce a new prior model called minimum area increment to reduce necking Current model in EM/MPM has smoothing parameter β (a) Original image (b) Ground truth (c) Original (d) Original EM/MPM (β=3.0) EM/MPM (β=.2) 3

4 SEM Imaging Modes Secondary Electrons (SE): Due to SE s low energy, they can escape only from a thin surface layer of a few nanometers. In this mode, blurring degradation can be modeled with a 2D blurring filter. X BSE PE SE AE BSE Backscattered Electrons (BSE) : Information depth in BSE mode is deeper than in SE mode. If we capture electrons with small energies below kev, we can make the exit depth of BSE have the same order as of SE. Therefore, we can conclude that the interactions for low-energy electrons can be modeled with a 2D filter while the interactions for high-energy electrons can be modeled with 3D blurring filter. R ctron range Ele Diffusion cloud of electron range R for normal incidence of the primary electron (PE). L. Reimer. Scanning Electron Microscopy: Physics of Image Formation and Microanalysis, 2 nd Edition. Springer-Verlag, Berlin, 998 4

5 2D and 3D Blurring Filter Coefficients 2D filter coefficients : The lateral number of generated SE can be modeled as an exponential. ated SE D filter coefficients : We propose 3D filter which has coefficients as below: Number of gener Lateral distance from impact point [nm] The lateral distribution of generated SE (Monte Carlo simulation, silicon, 5kV) 2 2 Günter Wilkening, Ludger Koenders. Nanoscale Calibration Standards And Methods: dimensional and related measurements in the micro and nanometer range, st Edition, WILEY-VCH, Weinheim,

6 Original Image Models for EM/MPM Use the Markov Random Field as the prior model Use the Gaussian distribution Use Bayes rule to combine the these two models into the posterior distribution function data term regularization term 6

7 EM/MPM Segmentation Use the Maximizer of the Posterior Marginal (MPM) criterion as the optimization objective. Minimizes the expected number of misclassified pixels. Use the Expectation/Maximization (EM) algorithm to estimate model parameters. - The unknown parameter vector contains means and variances for the Gaussian image model. 7

8 2D JDS(Joint Deconvolution/Segmentation) Method : Definition We define label field x, observed image y and blurring vector h. - Let the set of all lattice point S be [,,M] 2 and the order of the pixel of the label field x and the observed image y be raster scan order as below: - We can make the blurring matrix H having window size (2W + ) (2W + ) be a vector h using raster scan order, so that 8

9 2D JDS Method : Image Model & Posterior Model 3 Image Model : Posterior Model: 3 D.W. Kim and M.L. Comer, Joint deconvolution/segmentation of microscope image of materials, in IEEE Statistical Signal Processing Workshop, Ann Arbor, MI, USA, August

10 2D JDS Method : EM algorithm In EM iteration we can get closed form solution of variance. But for the mean, we get L linear equations from which we can obtain estimates of the means 0

11 3D JDS Method : Definition We define label field x n and observed image y n of the n-th slice in a stack of image. And we define 3D blurring vector h 3D. y T y n x n : label field of the n-th image y n : observed n-th image. h 3D : blurring vector having coefficient h 3D (s,s 2,m) y

12 3D JDS Method : Image Model & Posterior Model New Image Model : New Posterior Model: 2

13 Results : Test Sequence Slice66(Bottom) Slice67 Slice68 Slice69 Slice70(Top) Series of five René 88 DT images. The light-colored phase is γ' the gray matrix is γ. We applied our method from the bottom image to the top. 3

14 Results : 3D Blurring Image Model (a) Original image (b) Ground Truth (c) Original EM/MPM (β=3.0) Slice 70 PMP = 5.35% (d) blurring 3D JDS image EM/MPM model (β=3.0, ω=0.52, ω=0.0, ω=0.5, ω=0.20, ω=0.25, ω=0.30, ω=0.35, ω=0.40, ω=0.45, ω=0.50, ω=0.55, ω=0.60, ω=0.65, ω=0.70, ω=0.75, ω=0.80, ω=0.85, ω=0.90, ω=0.95, ω=.00, ω=.05, ω=.0, δ δ =0.5) PMP = 4.08% (e) Preprocessed Image (f) Preprocessed EM/MPM MATLAB deconvlucy (β=3.0) (ω=2 2.0) PMP = 4.76% PMP (percentage of misclassified pixels) 4

15 MAI(Minimum Area Increment) To further reduce object necking, we propose a minimum area increment constraint. This assigns a penalty for the merging of two or more large objects Connecting point: A point where two or more disconnected areas of the same class exist in a pre-defined neighborhood around the point 4-neighbor neighborhood 2-neighbor neighborhood Consider a 4-neighbor configuration: The center pixel in the lftfi left figure is not a connecting point; the center pixel in the right figure is a connecting point 2 class example 0 0 : class 0 : class Minimum area increment window size w s = x r Not connecting point when either x r = 0 or x r Connecting point when either x r = 0 or 5

16 Area Increment Measuring Function Area increment measuring function g ws w s,r(x r) : The increase in area of the largest-area region in a window of size w s w s around pixel location r if the class label assigned to pixel r is x r, - If one class is a background class (assume this is class 0), then we let g ws,r(0)=0 for all r -Ifpixelris not a connecting point then g ws,r(x r )=0 Consider the following 3-class example, with class 0 a background class 3 class example blank : class 0 : class 2 : class 2 Minimum area increment window size w s = 5 x r g 2,r () = 0 g 2,r (2) = x r 2 2 g 2,r () = 4 g 2,r (2) = x r 2 2 g 2,r () = 0 g 2,r (2) = x r g 2,r () = 9 g 2,r (2) = 0 6

17 New prior model: MRF and MAI We propose new prior model by incorporating MAI constraint into existing MRF prior model as below: To make proposed prior model more effective, we applied SA(simulated annealing) scheme. We gradually increase the β value of the classes which have no necking problem. 7

18 Results : MAI (a) Original image Slice 07 (b) Ground Truth (c) Original EM/MPM (β=3.0) 30) PMP = 637% 6.37% (d) 2D JDS method with no MAI MAI τ =.5 w s = 7 2-neighbor 23th 24th 25th 26th 27th 28th 29th 30th EM iteration β(0) = 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3.0, β() = 3.0 PMP =2.97% 23th 24th 25th 26th 27th 28th 29th 30th EM iteration β(0) = 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3.0, β() = 3.0 PMP=2.75% 8

19 Results : MAI (a) Original image (b) Ground Truth (c) Original EM/MPM Slice 70 (β=3.0) 30) PMP = 535% 5.35% (d) 2D JDS method with no MAI MAI τ =.5 w s = 7 2-neighbor 23th 24th 25th 26th 27th 28th 29th 30th EM iteration β(0) = 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3.0, β() = 3.0 PMP=4.35% 23th 24th 25th 26th 27th 28th 29th 30th EM iteration β(0) = 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3.0, β() = 3.0 PMP=3.70% 9

20 3D EM/MPM with JDS Image Model : Posterior Model: 20

21 2D EM/MPM VS 3D EM/MPM 2D EM/MPM 3D EM/MPM y T y n Image Model y pixel of next frame Prior Model pixel of previous frame 3D EM/MPM needs a large amount of memory. So we apply 3D EM/MPM method for 3 frames and save the result of the middle frame and then move to the next 3 frames which are one frame shifted from the previous 3 frames. 2

22 Results : 3D JDS with 3D EM/MPM NiAlCr slice 027 ground truth 3D JDS with 2D EM/MPM ( =.5, ω =03 0.3, δ =0 0., PMP = 6.3%) 3D JDS with 3D EM/MPM ( =.5, =0.75, ω =03 0.3, δ = 0., PMP = 6.03%) Rene88 slice70 ground truth 3D JDS with 2D EM/MPM ( =.5, ω = 0.5, δ = 0.5, PMP = 4.38%) 3D JDS with 3D EM/MPM ( =.5, = 0.75, ω = 0.5, δ = 0.5, PMP = 4.24%) 22

23 Results : 3D EM/MPM with JDS and MAI NiAlCr slice number to 59 (59slices, Image Size 94 X 49 pixels) EM/MPM (β=.5) Running time 4 : 8sec 4 Intel i7 CPU 2.4GHz, Memory 8GB 3D JDS &MAI2DEM/MPM (β=.5, ω = 0.3, δ = 0., τ =.5) Running time : 2260sec (28x) 3D JDS & MAI with 3D EM/MPM (β=.5, ω = 0.3, δ = 0., τ =.5) Running time : 6638sec (82x) 23

24 Results : 3D EM/MPM with JDS and MAI Rene88 slice number 43 to 88 (46slices, Image Size 94 X 49 pixels) EM/MPM (β=.5) Running time : 63sec 3D JDS &MAI2DEM/MPM (β=.5, ω = 0.5, δ = 0.5, τ =.0) Running time : 762sec 3D JDS & MAI with 3D EM/MPM (β=.5, ω = 0.5, δ = 0.5, τ =.0) Running time : 575sec 24

25 Conclusions In this research, we propose a blurring model to improve pixel mis- classification originating from blurring in SEM images. The proposed method incorporates physical modeling of electron interactions into a blurring image model We also propose a new prior model including minimum area increment constraint tand apply it with SA scheme. In addition, we apply JDS and MAI in the 3D EM/MPM. Experimental results demonstrate that the proposed methods can be used to reduce necking in the segmentation of microscopeimages of materials 25

26 Thank you

27 Appendix B: EM estimation for the new image model (/4) The EM algorithm is an iterative procedure. At each iteration expectation step and maximization step are performed. In the expectation step the following function is computed. In the maximization step, we can estimate θ(p) which maximize Q(θ(p), θ(p-))

28 Appendix B: EM estimation for the new image model (2/4) Similarly, by differentiating with parameter σ k we can get

29 Appendix B: EM estimation for the new image model (3/4) Therefore,

30 Let Appendix B: EM estimation for the new image model (4/4) then we can get