Equation recognizer and calculator. By Daria Tolmacheva

Transcription

1 Equation recognizer and calculator By Daria Tolmacheva

2 Introduction Allow cameras to solve equations Start with simple equations 1 operator (+,, /, *) 2 variables (1,2,3,4,5,6,7,8,9,0) Assumptions Only one equation per image 11pt. Calibri Font

3 Previous Work and Background Document Image Understanding Skew detection, noise filtering, segmentation Decomposition into blocks Semantic recognition or logical layout Skew Detection Curved surfaces Hough transform Parallel straight lines

4 Project Description Step 1: creating templates Image containing all the required characters make_templates.m code: I = imread('templates.png'); threshold = 90; I1 = im2bw(i, 0.35); figure, imshow(i1, []); L = bwlabel(~i1); blobs = regionprops(l);

5 for i = 1:size(blobs,1) rectangle('position', blobs(i).boundingbox, 'EdgeColor', 'r'); a = sprintf('%d', blobs(i).area); text(blobs(i).centroid(1),blobs(i).centroid(2), a,'color', 'b'); box = blobs(i).boundingbox; minx = round(box(1)); maxx = minx + round(box(3)); miny = round(box(2)); maxy = miny + round(box(4)); subimage = I1(minY:maxY, minx:maxx);

6 [row, col] = size(subimage); if row > col diff = row - col; add_col = floor(diff/2); squareimg = ones(row,row); squareimg(:, add_col:(add_col+col-1)) = subimage; resize_blob = imresize(squareimg, [25 25]); elseif row < col diff = col - row; add_row = floor(diff/2); squareimg = ones(col,col); squareimg(add_row:(add_row + row-1),:) = subimage; resize_blob = imresize(squareimg, [25 25]); else resize_blob = imresize(subimage, [25 25]); offscale = min(min(resize_blob)); resize_blob = resize_blob - offscale; maxoffscale = max(max(resize_blob)); resize_blob = resize_blob / maxoffscale; resize_blob = uint8(255*resize_blob); imshow(resize_blob); imsave

7 Project Description Step2: Get each character from equation image Use region props on binary image: I1_grey = rgb2gray(i1); %figure, imshow(i1_grey, []); threshold = 90; I1 = im2bw(i1_grey, 0.35); s = strel('disk', 1); I1 = imclose(i1, s); I1 = imopen(i1, s); L = bwlabel(~i1); blobs = regionprops(l); blobs_center_x = zeros(1,size(blobs,1)); blobs_center_y = zeros(1,size(blobs,1),1); for i=1:size(blobs,1) blobs_center_x(:,i) = blobs(i).centroid(1); blobs_center_y(:,i) = blobs(i).centroid(2); [sb,ix] = sort(blobs_center_x); for i=1:size(blobs,1) rectangle('position', blobs(i).boundingbox, 'EdgeColor', 'r');

8 Project Description Step3: Prepare eigenfaces from templates and mean image(eggn510 L18 PCA) %calcualate the mean of tempaltes m=uint8(mean(templates,2)); %subtract off mean from all templates templates_mean = templates - uint8( single(m)*single( uint8(ones(1,size(templates,2)) ) )); %calculate eigenfaces L=single(templates_mean)'*single(templates_mean); [V,D]=eig(L); PC=single(templates_mean)*V; %calculate image signatures signatures = zeros(size(templates,2), 14); for i=1:size(templates,2); signatures(i,:)=single(templates_mean(:,i))'*pc; % Each row is an image signature

9 Project Description Step4: Process blobs and match them against eigenfaces: (EGGN510 L18 PCA) create subimage for each of the blob to compare it against templates for i=1:size(blobs,1) rectangle('position', blobs(i).boundingbox, 'EdgeColor', 'r'); %a = sprintf('%d', blobs(i).area); %text(blobs(i).centroid(1),blobs(i).centroid(2), a,'color', 'b'); box = blobs(ix(i)).boundingbox; minx = round(box(1)); maxx = minx + round(box(3)); miny = round(box(2)); maxy = miny + round(box(4)); subimage = I1(minY:maxY, minx:maxx); [row, col] = size(subimage);

10 if row > col diff = row - col; add_col = floor(diff/2); squareimg = ones(row,row); squareimg(:, add_col:(add_col+col-1)) = subimage; resize_blob = imresize(squareimg, [25 25]); elseif row < col diff = col - row; add_row = floor(diff/2); squareimg = ones(col,col); squareimg(add_row:(add_row + row-1),:) = subimage; resize_blob = imresize(squareimg, [25 25]); else resize_blob = imresize(subimage, [25 25]); offscale = min(min(resize_blob)); resize_blob = resize_blob - offscale; maxoffscale = max(max(resize_blob)); resize_blob = resize_blob / maxoffscale; resize_blob = uint8(255*resize_blob); reshape_blob = reshape(resize_blob,img_size,1)-m; reshape_weighted = single(reshape_blob)'*pc; scores = zeros(1, size(signatures,1)); for j=1:size(templates,2) % calculate Euclidean distance as score scores(j)=norm(signatures(j,:)-reshape_weighted,2); [C,idx] = sort(scores, 'asc'); matches(i) = idx(1);

11 Project Description Step5: Match templates with scores and evaluate equation: if matches(1)== 1 val1 = 1; elseif matches(1) == 2 val1 = 2; elseif matches(1) == 3 val1 = 3; elseif matches(1) == 4 val1 = 4; elseif matches(1) == 5 val1 = 5; elseif matches(1) == 6 val1 = 6; elseif matches(1) == 7 val1 = 7; elseif matches(1) == 8 val1 = 8; elseif matches(1) == 9 val1 = 9; elseif matches(1) == 10 val1 = 0;

12 Project Description Another Important Point: Images of equation from arbitrary point: Plot a line through blobs centroid and get a transform from that: figure, imshow(i1, []),hold on plot(blobs_center_x, blobs_center_y); coefficients = polyfit(blobs_center_x,blobs_center_y,1); newy = polyval(coefficients, blobs_center_x); plot(blobs_center_x,blobs_center_y,'*',blobs_center_x,newy,' :')

13 Project Description

14 Results Templates didn t seem to match Reasons: bad templates, need bigger dimensions, better alignment, higher precision

15 Future work Make bigger templates with better precision Recognize more complicated equations with multiple operators and multiple digit numbers Recognize more than one equation from image Find a better way to deal with noise Filtering Picking out only important blobs

16 Work Cited [1] M.Ceci, M.Beradi, D.Malerba, Relational Data Mining and ILP for document image understanding, Applied Artificial Intelligence, Taylor & Francis Group, LLC, 21: [2] I.Guyon, R.M.Harlick, J.J.Hull, Data Sets for OCR and Document Image Understanding Research, Handbook of Character Recognition and Document Image Analysis, pp World Scientific Publishing Company, 1997 [3] S.Lu, B.Chen, C.C.Ko A Partition Approach for the Restoration of Camera Images of Planar and Curled Document, Image and Vision Computing, Vol.24 Issue 8, Pages , Electrical and Computer Engineering Department, National University of Singapore, Aug.2006 [4] A.Nakhmani, A.Tannenbaum, A New Distance Measure Based on Generalized Image Normalized Cross Correlation for Robust Video Tracking and Image Recognition, Pattern Recognition Letters, Vol.34 Issue 3, Pages , February 2013 [5] R. Cattoni, T.Coianiz, S.Messelodi, C.M.Modena, Geometric Layout Analysis Techniques for Document Image Understanding: a Review, Povo. Trento. Italy, January 1998