Computer Aided Engineering Applications 5. Machine Vision

Transcription

1 Computer Aided Engineering Applications 5. Machine Vision 5.1 Introduction 5.2 The camera model 5.3 Spatial filtering 5.4 Segmentation 5.5 Blob analysis 5.6 Classification 5.7 Depth perception Engi6928 -Fall 2014

2 5.1 Introduction Machine visionis the technology and methods used to provide imaging-based automatic inspection and analysis. Applications include : Inspection of parts, guidance, process control. Digital Image processing is the process of conditioning and extracting information from digital images.

3 5.1 Introduction Applicable to many fields RGB Images UV Images IR Images Ultrasound Images SAR Images MRI Images

4 5.1 Introduction Structure of an Industrial Machine vision system Image Acquisition Filtering 10 NUT x,y,z Transformation Segmentation Blob analysis Object detection Metric estimation

5 5.2 Camera Model The Pin hole camera model CCD/CMOS image sensor {C} {W} Pixel Position: This transformation looses the depth information Intensity:

6 5.2 Camera Model Intensity The value of a pixel position {I} xpx Finding intensity of f(0,0) in Matlab ypx >> f=imread([pwd'\fasteners_2_gray.pgm']); >> close all >> imshow(f) >> f(1,1) ans= 120 RGB images have multiple channels. i.e. an intensity vector for each pixel position >> f=imread([pwd'\fasteners_2.bmp']); >> f_info=imfinfo([pwd'\fasteners_2.bmp']); >> f_info.bitdepth ans = 24 >> size(f) ans = >> f(1,1,:) ans(:,:,1) = 118 ans(:,:,2) = 122 ans(:,:,3) = 114 Intensity levels : 2^24 Pixel size: 1040 x 1392 Channels: 3 (RGB) RGB values

7 5.2 Camera Model Image Transformations 1. Translation Clear; close all; clc f=imread([pwd'\fasteners_2_gray.pgm']); f_info=imfinfo([pwd'\fasteners_2_gray.pgm']); subplot(1,2,1) imshow(f); m=size(f,2); %width n=size(f,1); %height %1. translate T=[1 0100; ; 001]; High level function imtranslate(f,[tx ty]); for uy=1:n for ux=1:m u=[uxuy1]'; up=uint16(t*u); if up(1)>0 && up(2)>0 && up(1)<m && up(2)<n g(up(2),up(1))=f(uy,ux); end end end subplot(1,2,2) imshow(g);

8 Image Transformations 5.2 Camera Model 2. Rotation 3. Scale High level function imrotate(f,angle); High level function imscale(f,scale);

9 5.3 Spatial filtering Intensity transformations-changing the image intensity level at a location with a mathematical relation which does not depend on the neighbouring pixels Spatial filtering -Changing the image intensity level at a location with a mathematical relation which depends on intensity of neighbouring pixels. 1. Image Negative %Intensity Transformation - Image Negative f=imread([pwd'\fasteners_2_gray.pgm']); g=2^8-1-f; imshow(g);

10 2. Gamma correction 5.3 Spatial filtering %% Gamma correction f=imread([pwd'\fasteners_2_gray.pgm']); g=uint8(1*double(f).^(0.8)); subplot(1,2,2) imshow(g); 3. RGB to gray %% RGB to gray f=imread([pwd'\fasteners_2.bmp']); subplot(1,2,1) imshow(f); g(:,:)= * f(:,:,1) * f(:,:,2) * f(:,:,3) ; % g= rgb2gray(f); % high level function subplot(1,2,2) imshow(g); RGB Gray

11 5.3 Spatial filtering 4. RGB to HSV %% RGB to hsv f=imread([pwd'\fruits.jpg']); subplot(2,2,1) imshow(f); g = rgb2hsv(f); h = g(:, :, 1); % Hue image. s = g(:, :, 2); % Saturation image. v = g(:, :, 3); % Value (intensity) image. subplot(2,2,2) imshow(h) subplot(2,2,3) imshow(s) subplot(2,2,4) imshow(v) Separates image intensity (luma) from image color (chroma). Processing in the luma channel is robust towards lighting changes. Intensity

12 5.3 Spatial filtering A spatial filter is viewed as a convolution (A weighted sum of a window) using a convolution kernel.

13 5.3 Spatial filtering 5. Average filter (Low pass filter) %% spatial filter - Averaging (box filter) f=imread([pwd'\fruits.jpg']); close all h = ones(40,40) / 40^2; g = imfilter(f,h); imshow(g) 6. High Pass filter %% spatial filter - High Pass f=imread([pwd'\fruits.jpg']); close all h = -ones(5,5); h(3,3)= 5^2; g = imfilter(f,h); imshow(g)

14 Image histogram 5.4 Segmentation A bar plot where the x axis (bins) are different intensity levels. Y axis plots the number of pixels having the intensity level x. %% Gray level histogram close all f=imread([pwd'\fasteners_2_gray.pgm']); m=size(f,1); n=size(f,2); g_hist=zeros(1,256); for i=1:m for j=1:n g_hist(f(i,j)+1)=g_hist(f(i,j)+1)+1; end end bar([1:256],g_hist) axis([ ^5]) % imhist(f) % high level function 10 x pixels with Gray level 74 X = 74 Y =

15 Gray to binary conversion 5.4 Segmentation The histogram exhibits 2 gray levels. A threshold can be used to generate binary images %% Gray to binary close all f=imread([pwd'\fasteners_2_gray.pgm']); M=size(f,1); N=size(f,2); Thresh= 100; for i=1:m for j=1:n if f(i,j)<thresh g(i,j)=1; else g(i,j)=0; end end end g=logical(g); % data type conversion to bool %g=im2bw(f,thresh/2^8); % high level function imshow(g); 10 x X = 74 Y =

16 5.4 Segmentation Otsu`s Thresholding algorithm A method to generate the threshold value automatically. The method finds the threshold which maximizes the between class variance. 10 x 104 %% Otsu's method 8 close all 7 6 f=imread([pwd'\fasteners_2_gray.pgm']); 5 threshold = graythresh(f); 4 g = 1-im2bw(f,threshold); 3 imshow(g) X = 74 Y =

17 5.4 Segmentation Otsu`s Thresholding algorithm 1. Calculate probability distribution from histogram 2. For each threshold value calculate between class variance 3. Select the threshold which maximizes the between class variance 10 x

18 Example 5.4 Segmentation

19 5.4 Segmentation Connected component labelling Separates the binary image to a set of segments. Each pixel belongs to a segment if it satisfies a pixel connectivity condition. Input binary image 4 connectivity labelling 8 connectivity labelling

20 5.4 Segmentation Connected component labelling %% Connected components close all f=imread([pwd'\fasteners_2_gray.pgm']); threshold = graythresh(f); g = 1-im2bw(f,threshold); cc = bwconncomp(g,8); %find 8 neighbor connected components L = labelmatrix(cc); %label the image pixel with the indices of the components RGB = label2rgb(l); %convert the label values to unique colors imshow(rgb) cc = Connectivity: 8 ImageSize: [ ] NumObjects: 13 PixelIdxList: {1x13 cell} Because of noise of the binary image, undesired segments may arise

21 Noise removal 5.4 Segmentation Remove background connected components which are small. %% Noise removal - bwareaopen close all f=imread([pwd'\fasteners_2_gray.pgm']); 㛀 Ѯ threshold = graythresh(f); g = im2bw(f,threshold); g = bwareaopen(g, 5000); %remove background connected compenentswhich have less than 5000px imshow(1-g)

22 5.5 Blob analysis Blob analysis quantifies different segment descriptors to help with object identification. Depending on the application the segment descriptors would require to be: Translation/ rotation invariant: Area, major axis length, 㛀 Ѯ perimeter, generalized moments Scale invariant: number of holes, complexity = perimeter^2/ area, Invariant moments

23 5.5 Blob analysis Matlabcommand regionpropscalculates many segment descriptors. %% Blob anlysis- reigion props clear;clc;close all f=imread([pwd'\fasteners_2_gray.pgm']); g = im2bw(f,graythresh(f)); g = 1-bwareaopen(g, 5000); imshow(g) cc = bwconncomp(g,8); s = regionprops(cc,'all'); hold on for i=1:cc.numobjects bbox=s(i).boundingbox; rectangle('position', bbox,'edgecolor','b') center=s(i).centroid; text(center(1),center(2),[num2str(i)],'color','r'); end Euler number = number of objects number of holes

24 5.5 Blob analysis Generalized moments can be used to represent different descriptors of a segment. In the case of a digital image of size nby mpixels, the generalized moment is given by: M ij m n = x= 1 y= 1 i j xy f ( x, y ) For binary images the function f(x,y) takes a value of 1 for pixels belonging to class object and 0 for class background.

25 5.5 Blob analysis M ij = n m x= 1 y= 1 x i y j f ( x, y) X i j M ij Area Y Moment 64 of Inertia 93

26 X = M M Y = 5.5 Blob analysis The center of mass of a region can be defined in terms of generalized moments as follows: M M The moments of inertia relative to the center of mass: 䃀 2 2 M01 M10 M02 = M02 M20 = M20 M11= M M M Principal axis: TAN2θ = M M 11 M M10M M 00 01

27 5.5 Blob analysis

28 5.5 Blob analysis Generalized Moments M ij m n = x= 1 y= 1 Central Moments µ ij m n x= 1 y= 1 i j xy f( x, y) i j = ( x x)( y y) f( x, y) Normalized central moments µ ij ηij = ( i+ j + 1) 2 M00 Hu`s Invariant moments 䫠 ѫ Xc=M10/M00; Yc=M01/M00; %Central moments mu=0; for i=1:n for j=1:m mu=mu+ (j-xc)^p*(i-yc)^q*f(i,j); end end %Normalized Central moments nu=mu/m00^(1+(p+q)/2);

29 5.5 Blob analysis function [Hu]=hu_moments(f) function[m,mu,nu]=all_moments(f,p,q) m=size(f,2); n=size(f,1); [M20 mu20 nu20]=all_moments(f,2,0); [M02 mu02 nu02]=all_moments(f,0,2); [M11 mu11 nu11]=all_moments(f,1,1); [M12 mu12 nu12]=all_moments(f,1,2); [M21 mu21 nu21]=all_moments(f,2,1); [M30 mu30 nu30]=all_moments(f,3,0); [M03 mu03 nu03]=all_moments(f,0,3); 䫠 ѫ Hu(1)=nu20+nu02; Hu(2)=(nu20-nu02)^2+4*nu11^2; Hu(3)=(nu30-3*nu12)^2 + (3*nu21+nu03)^2; Hu(4)=(nu30+ nu12)^2 + (nu21+nu03)^2; Hu(4)=(nu30-3*nu12)*(nu12+nu30)*((nu30+nu12)^2-3*(nu21+nu03)^2) + (3*nu21-nu03)*(nu21+nu03)*(- (nu21+nu03)^2+3*(nu12+nu30)^2) ; m=size(f,2); n=size(f,1); %Moments M=0; for i=1:n for j=1:m M=M+ j^p*i^q*f(i,j); end End %Central Moments %Normalized Moments

30 4.5 Blob analysis Area Major axis 䫠 ѫ Complexity Slenderness Hu`smoments , , , , , , , , , , , , Transformation and Scale Invariant

31 5.6 Classification Classifiersare used for object identification in machine vision systems. An object has a set of features which is termed as a feature vector. The classifier attempts to identify which classthe 浐 ѫ feature vector belongs to. Supervised learning is the process of tuning the classifier using a training data set.

32 5.6 Classification 1-feature-n-class classifier Normalize the calculated features. Sort the image segments according to the selected feature. Assign classes in the ascending order!only applicable when all classes are present 浐 ѫ Better approach: Assign class according the closeness to the object in the original image (Nearest neighbour)

33 5.6 Classification Nearest Neighbour classifier (m-feature-n-class classifier) 1. Generate feature vectors for each object in the training dataset 2. Normalize the feature vector 3. Train the classifier (assign class mean) oo o o o Feature 2 Test Data x Assign to nearest class Feature 1 1. Generate the feature vector of the test object 2. Normalize using the trained parameters 3. Find the nearest class Training Data

34 5.6 Classification Nearest Neighbour classifier Cc1 : connected components of training image Classes1: Assigned classes of training image Feature_vector1: feature vectors of training image Feature_vector1_norm: Normalized feature vectors of training image Cc2 : connected components of test image Feature_vector2: feature vectors of test image Feature_vector2_norm: Normalized feature vectors of test image Weight : weight given to each feature Classes2 : nearest class of each feature vector (found by the following code) classes2=[]; for j=1:cc2.numobjects for i=1:cc.numobjects distance_measure(i)=norm((feature_vector2_norm(j,:)-feature_vector1_norm(i,:))*weight(i)); end [minval Id]=min(distance_measure); classes2(j)=classes1(id); end

35 GUI improvements 5.6 Classification Euler number checks (produce warnings for inconsistent matches) Invariant features (incorporate Hus moments) User Visualizations and warnings (allow user to 浐 ѫ clearly verify results and adjust critical parameters)

36 5.6 Classification SURF feature matching matched points 1 matched points 2 clear clc close all I1=imread([pwd'\Fasteners_1.bmp']); I2=imread([pwd'\Screw_6.bmp']); I1=rgb2gray(I1); I2=rgb2gray(I2); points1 = detectsurffeatures(i1); points2 = detectsurffeatures(i2); 浐 ѫ matched points 1 matched points 2 [f1, vpts1] = extractfeatures(i1, points1); [f2, vpts2] = extractfeatures(i2, points2); indexpairs = matchfeatures(f1, f2) ; matchedpoints1 = vpts1(indexpairs(:, 1)); matchedpoints2 = vpts2(indexpairs(:, 2)); figure; showmatchedfeatures(i1,i2,matchedpoints1,matchedpoints 2); legend('matched points 1','matched points 2');

37 5.7 Depth perception 1. 3D vision Range sensors 浐 ѫ Time of flight cameras Combined camera and laser scanner r measured by range sensor

38 5.7 Depth perception 2. 3D vision Structured light 浐 ѫ

39 5.7 Depth perception 3. 3D vision Stereo vision/ Multi camera systems 浐 ѫ

40 5.7 Depth perception 4. 3D vision Known 3D object 浐 ѫ

41 Objective: to automate the inspection of LCD modules in order to improve quality control Vision Based Inspection of Liquid Crystal Display (LCD) modules One step in the implementation of a Six-Sigma Program ( 3.4 defects per million parts ) The inspection must be completed within 30 seconds for 10 predetermined LCD patterns System can learn new LCD modules without modifying software 浐 ѫ

42 Vision Based Inspection of LCD modules: System Components Pulnix camera with macro lens High frequency fluorescent light sources Coreco Banditintegrated image acquisition and VGA accelerator Software developed using with WiT graphical programming environment in combination with Microsoft VB 浐 ѫ Memorial University of Newfoundland 11/22/2014

43 Vision Based Inspection of Liquid Crystal Display (LCD) modules 浐 ѫ Original Image Showing Error in Alignment

44 Vision Based Inspection of Liquid Crystal Display (LCD) modules 姀 ѵ Thresholding Operation Image Subtraction with respect to an image with no segments illuminated

45 Vision Based Inspection of Liquid Crystal Display (LCD) modules 姀 ѵ Blob Analysis Reference Points are Identified

46 Vision Based Inspection of Liquid Crystal Display (LCD) modules 姀 ѵ Image Rotation and Translation

47 Vision Based Inspection of Liquid Crystal Display (LCD) modules 姀 ѵ Pixel by Pixel Image Subtraction from Reference Image Thinning Operator

48 Vision Based Inspection of Liquid Crystal Display (LCD) modules 姀 ѵ Blob Analysis

49 References Digital image processing using MATLAB -Gonzalez, R. C., Woods, R. E., & Eddins, S. L. (2004). Upper Saddle River, NJ Jensen: Prentice Hall,. Multiple view geometry in computer vision. Hartley, Richard, and Andrew Zisserman. Cambridge university press, 姀 ѵ