EQUATIONS can at times be tedious to understand

Transcription

1 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE Augmented Reality Equation Plotte Salman Naqvi, Uzai Sikoa Digital Image Pocessing EE 368/ CS 232 Abstact Gaphical Visualization of an equation can often help us bette undestand undelying tends of eal vaiables. Moeove, visualizing it quickly as we come acoss it in a jounal aticle o textbook is often desied in ou scholaly eadings. Typing out that complex equation linealy into a numeical computing envionment, e.g. MATLAB, can be timeconsuming and hassle-pone. With the high-esolution camea, fast-pocessing, and cheap cellphones available to us, it is highly beneficial to have an equation ecognize and gaph plotte application in ou phones. Such a nifty tool can help younge students woking on building thei intuition to be able to visualize simple equations quickly in thei thoughts. Hee, we ty to develop such a tool fo the Andoid platfom. We leveage the OpenCV Andoid libay to pefom client side image pocessing and MATLAB Image Pocessing tool to pefom sleek mathematical chaacte template matching and equation fomation. The application can ead in any pinted equation and plots on sceen the gaph of an explicit equation composed of exponentials, polynomials, tigonometic functions and simple deivatives and integals. Keywods Recusive Segmentation, Binaization, Chaacte Recognition, Tesseact, Template Matching, Keypoint and Edge Detection, Andoid Development. 1 INTRODUCTION EQUATIONS can at times be tedious to undestand and moe so, difficult to quickly visualize the undelying tend of. The situation wosens if an equation is a complicated mixtue of polynomials, exponentials, and tigonometic functions, involving algebaic factions and exponents. Fotunately, moden numeical computation scipting languages, such as MATLAB, can help us quickly visualize such equations. Howeve, in case of limited access to MATLAB o to avoid witing out lage equations linealy such that they ae undestandable by the compute, we need a tool that can snap a pictue of the equation pinted on a pape and in nea eal-time pocess and etun the associated gaph on a cell phone sceen. Moeove, the gaph eading pocess can be made inteactive again by using the image pocessing techniques, and the use can view the gaph at diffeent pespective and ead values off of it. This entie pocess involves a numbe of image pocessing challenges. Fist one is the mathematical equation ecognition. Text ecognition has been widely studies in the past and efficient hand witing U. Sikoa, and S. Naqvi ae with the Depatment of Electical Engineeing, Stanfod Univesity, Stanfod, CA, usikoa, shnaqvi@stanfod.edu Manuscipt completed June 5, 2013 detection is still an aea of active eseach. On the topic of mathematical chaactes ecognition, a numbe of suvey papes published in the past decade o so, [5] and [6], sum up the developments quite well. Moe ecently, thee has been even geate wok being done in the aea of intelligent ecognition and etieval of mathematical expessions [3]. The second pocess involves a fast axes detection and plotting with pespective and skew coection. We implement these two main segments of image pocessing to develop a complete use-oiented cell phone application which can achieve its goal efficiently and vey fast. We talk about the algoithm flow chat in the next section. 2 APPLICATION FLOWCHART Ou equation plotte tool is oganized into a seies of states that ae called into activity depending on which action needs to be completed in the wokflow. An oveview of the states is shown in Figue 1. The application stats in the Captue Equation mode. In this stage, the use captues the image of the equation they would like to plot. The use can etake the image and theefoe, the state estats. Othewise, the application moves into the Equation Analysis stage. Hee, the image is sent to the seve unning MATLAB. Image segmentation, and math chaacte ecognition pat of the image

2 2 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE 2013 pocessing takes place at this stage. The application then eceives the conveted equation sting and displays it fo the use next in the Quey Use stage. At this stage, the use has the option to edit the sting and/o appove it fo the seve to etun the coodinates vectos fo the plot. The application then moves into the Axis Detection state. Hee, the emaining of the image pocessing takes place including the canny edge detection, dilation, keypoint detection, and axis otation estimation. Afte the application has analyzed the camea peview fames and detected the coodinate axes on which to plot, we move on to the next state, Plot Points. At this point, the application calculates the location of the coodinates on the phone sceen, daws the points, and etuns to the Axis Detection state. Duing this loop between Axis Detection and Plot Points, the use has the capability to stat a new equation pocess and thus the use can etun to the Captue Equation state. The state machine appoach used in this application geatly simplifies the integation of diffeent functions and pocesses as needed. a PHP scipt to upload the image data to a file at the seve. This scipt then waits fo the MATLAB to geneate an output file at which stage, it etuns the contents of that file as an http sting. We also ensue that the image is not bigge than 640x480 pixels which limits the file size to unde 175kB fo fast cell to seve communication. We also equie use to ensue that only the equation is captued by the image and no suounding clutte (text, othe equations, etc) ae included in the image. This ensues that all egions segmented by ou algoithm ae actually pat of the equation. We can also include the functionality of allowing the use to cop the image afte taking the pictue so that only the desied equation aea is extacted and sent to the seve. Figue 2 shows an example captued image which we will use as a efeence figue thoughout this pape. Equation Analysis Captue Equation Quey Use Fig. 2: Image captued by the Andoid phone 3.2 Plot Points Axis Detection Fig. 1: The state machine diagam fo the wokflow of the Andoid application S ERVER S IDE Image Captue The pocess begins with the image captue by the cellula phone at the client s end. Once a decently focused, unifomly lit and closed up view pictue of an math equation has been taken, use posts the images to the MATLAB seve. This post initiates Binaization We need to binaize the image well so that the only egions in the image ae the math chaactes and then we can segment each individually. Otsu s method can give a easonable theshold level but with faily distubed backgound, we will see alot of backgound noise getting binaized as well. One technique is to use adaptive Otsu s method which finds the local thesholds by sliding a window acoss the entie image. Howeve, fo ou ealtime application this method is vey slow. Thus we instead implement the backgound subtaction fom the image in ode to emove any non-unifom lighting effects befoe we apply Otsu s method to find a global theshold. Backgound is detemined by dilating the gay scale image with a 24x24 disc shaped stuctuing element. Howeve, a simple dilation gives blocky esult which hampes a good binaization. Thus, we

3 NAQVI et al.: AUGMENTED REALITY EQUATION PLOTTER 3 instead use a ank filte to blu out any chaactes fom the image in a way that the esulting backgound etains its smooth blending acoss pixels. Afte tial on a numbe of lighting and image captue distubance cases, we found that a ank of 160th maximum pixel woks well. This backgound subtacted image looks as good as the binaized image 4 and it is then binaized vey efficiently using Otsu s method gay level theshold. Fig. 3: Gayscale Backgound afte 160th ode Rank Filte Afte binaization, we smooth the cones of the chaactes, by an eosion opeation followed by a dilation. This is done to impove the text ecognition esults by template matching late on. We also pefom an opening opeation to emove any small salt and peppe noise that might have been picked natually duing pictue taking pocess. to ight in the binaized image. We use the vetical pojection to find out whee the white spaces ae that sepaate the chaactes. Gadient zeo cossings of the image pojection ae used to detect the egion edges, and these egions ae extacted as subimages. Howeve, ou job is not done yet, because ou equation might have factions, in which case, we will have to sepaate the numeato and denominato fist and then to pase out individual chaacte of the numeato and denominato espectively. This will theefoe equie egion sepaation in vetical dimension on each of the hoizontally sepaated egion. We also have to do so methodically so that we keep the ight ode of numeato followed by the denominato. Afte the vetical sepaation of faction, we will attempt a hoizontal sepaation again to sepaate the numeato egions. Moeove, thee can be nested factions inside the numeato that we should sepaate. In summay, we wite a ecusive algoithm that keeps on sepaating in y fist to sepaate any lines, then in x to sepaate egions hoizontally, then in y to sepaate any factions within each egion, and then it epeats until no moe sepaation in x o y is possible. Figue 5 depicts the pocess visually. Fig. 5: Segmenting the Image Recusively Fig. 4: Binaized Image afte applying Global Otsu s theshold 3.3 Recusive Image Segmentation Afte the binaization, we can clealy visually distinguish the individual math chaactes. And if the pixels ae high enough and the image was focused well without any motion bluing, each chaacte is clealy sepaated fom the othe. We then begin sepaating the chaactes hoizontally, going fom left 3.4 Exponentiation and Faction Detection Once all the egions ae sepaated and saved as single chaacte images, we iteate ove all egions and compae the y-centoids of each egion with the one peceding it and the one following it. If the y-centoid of the cuent egion is lowe than that of the peceding one, then it maks the beginning of exponentiation, and we check the beginning of exponent label in that image s data stuctue. Moeove, if the y-centoid of the cuent egion is highe than that of the peceding egion then we

4 4 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE 2013 check the end of exponent label in that egion s data stuctue. Afte all the egions have been labeled, we individually place the ˆ (hat) opeato followed by an open paenthesis at the stat of exponent image egion, and a close paenthesis at the end of exponent image egion. 3.5 Image Popeties and Resizing It will also be useful in the chaacte ecognition phase to know a numbe of image popeties of all the segmented chaacte images. In paticula as we shall descibe late, we ae most inteested in knowing the following popeties of each egion: 1) Aspect Ratio, which is the atio of the egion s majo axis length to the mino axis length, 2) Solidity, which is the atio of the numbe of egion pixels to the numbe of pixels in the egion s convex hull, 3) Numbe of Holes in the image, which ae obtained by subtacting fom the numbe of connected components inside an image the image s Eule numbe, 4) Centoid s y-coodinate, which gives the y location of the image s cente of mass and is helpful in detemining the sepaation of chaactes into diffeent lines, 5) Fist Hole s Centoid s y-coodinate, which will be helpful in distinguishing between vaious simila chaactes like e and 6. We stoe these popeties along with the image in a data stuctue which is stoed in the chaacte image database fo late use duing the ecognition step. But befoe stoing the image, we extact its tight bounding box and then esize it to 64x64 pixel, fo small disk space usage and easy template matching late on. The esult segmented set of subimages is shown in Figue 6. Fig. 6: Segmented egions esized with exponents and factions detected pediction model be tained to pocess these. Hand witten chaactes also need to be clealy sepaated and decently consistent in shape and size with the epetitions. SVM is a supevised machine leaning algoithm developed at the National Taiwan Univesity which analyzes data input and ecognizes patten. Fo us howeve, the task is simple because ou application is tageted towads ecognizing pinted equations appeaing in a textbook, couse notes, o jounal aticles. Thus ou fist plan of attack to accomplish easonable OCR is to use an open souce, fee OCR engine, called Tesseact. 4.1 Tesseact Output Tesseact is an feely available OCR engine that was oiginally developed at the HP Labs and is cuently being suppoted by Google [1]. It is known to pefom well in genealized cases of text ecognition whee chaactes ae limited to alphabets and digits and thee is a patten of alphabet occuence modeled by Makov chains. It employs many othe tools to ecognize text than the simple template matching and it needs to be tained fo any othe chaacte set othe than standad english lettes and numbes. In ou case, since ou goal was to be able to detect, exponentials, tigonometic functions, polynomials, simple deivatives, and geneal math symbols, ou chaacte set was lage and so alien fo Tesseact that the output was eally embaassing. Moeove, we also allowed fo diffeent fonts, which all cumulatively caused Tesseact to give vey poo esults to the expession shown ealie. The esult in which y, =,ˆ, and e wee not ecognized is shown below. 72(x6(2) + 2x + 26(x))/8(x6(3) + 5x) Fig. 7: Tesseact Output to ou example image Hence, we use an in-house template matching algoithm combined with post-pocessing of esults which geneates emakably bette esults. 4 OCR, OPTICAL CHARACTER RECOGNITION We ty two main methods fo ecognizing the segmented math chaactes images and we compae thei shotcomings. As we discussed ealie, ecognizing hand-witten equations is moe challenging and equies that a Suppot Vecto Machine (SVM) 4.2 Template Matching Database Fist, we set up a template matching database by witing down all possible chaactes that we expect to see in the types of equations we would like to solve. These amount to 35 chaactes. We also allow fo equations to be witten in 5 commonly

5 NAQVI et al.: AUGMENTED REALITY EQUATION PLOTTER 5 used fonts, including Aial, Times New Roman, Couie, Cambia Standad, and Cambia Math. The database is fed into the algoithm in the fom of an image containing of all chaactes witten in all fonts sepaated in diffeent lines. The algoithm uns a ecusive image segmentation to quickly pase out bounding boxes of all chaactes in the pope ode, computes all the helpful image popeties as discussed ealie fo all chaactes, and then esizes the images to 64x64 templates. The pocess is extemely fast and obust, and makes the template matching database easily expandable. Figue 8 shows the image used to populate the database. Fig. 8: Binaized image of the 35 chaactes with 5 font styles fed into the algoithm fo foming the template database Afte having the database setup, we use the segmented subimages fom the equation image to find the best matched template in the database, which has an associated ASCII chaacte also stoed in the database. Since both the template images and the equation segmented images ae of size 64x64 pixels, it is easie to compute thei matix dot poduct. The template giving the maximum dot poduct is taken to be the best match. We show the template matching text ecognition esult fo ou example case below. y = (xˆ(2) + 2x + 2ˆ(x))xe(xˆ(s) + 5x) Fig. 9: Template Matching Output to ou example Smat Chaacte Fixing based on Image Popeties We notice that cetain chaactes ae matched incoectly to a template in the database. In paticula, x was mistaken with a / (fowad slash), while 3 was mistaken with an s. This confusion between vaious chaactes can esult if the chaactes appeaing in the image ae skewed (skew can esult duing image captue), pixelated (pixelation can esult if fame was not zoomed in enough), blued (due to instability duing camea opeation), o pooly captued due to impope lighting conditions. Howeve, we can tackle with a decent amount of confusion by futhe filteing false positives using template and segmented images egion popeties. Fist, we an the algoithm on moe than a dozen images of diffeent equations and font styles, and unde diffeent lighting, zoom, and skew conditions. We figue out what chaacte pais ae most likely to be confused and then apply smat chaacte fixing. This is shown in the Confusion Matix plotted in Figue 10: Recognized Chaactes Database Chaactes. o c at n i s l e ( ) x y πd ^= - +* / / *+ - =^ dπ y x ( ) e l s i n t a c o. Fig. 10: Confusion Matix manually constucted by finding false positive in ove 2 dozen cases Confusion matix would keep on becoming moe dense and matue as we apply the ecognition on moe and moe divese cases. And as it populates futhe, we will get moe infomation about emedies to adapt in coecting false positives. Based on the cuent state of the confusion matix, we adapted moe than 2 dozen post-ecognition checks to coect the false esults. Fo instance, 8 is the biggest culpit which not so supisingly gets matched with many chaacte images. Thus, we deals with its set of confusion pai as follows: If the ecognized chaacte is 8, but the oiginal image has 1 hole, then if the y-centoid of the hole is above the y-centoid of the egion then ecognize it as e, othewise ecognize it

6 6 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE 2013 as 6, and if thee wee no holes in the oiginal image and the AR is little bit highe than than of 8, then ecognize it as 5, othewise in case of small AR, ecognize it as s. With smat fixing in place of majo chaactes such as 8, (, ), 6, l, etc, ou ecognition esults impove as shown below. Now, x is coectly eplaced with /, but s was still wongly eplaced with a 5. We would need to acquie moe image featues to distinguish such simila chaactes in the futue. Fo now, we get the emaining poblems with the ecognition fixed manually by sending the ecognized equation shown below back to the use. y = (xˆ(2) + 2x + 2ˆ(x))/e(xˆ(5) + 5x) Fig. 11: Recognized Equation afte Smat Fixing 4.3 Equation pasing in MATLAB Recognizable Fomat Afte use coects the equation and sends it back, ou sting pasing algoithm kicks in to put the standad lineaized equation into a fom that is ecognized by a MATLAB o Mathematica. Majo changes to the sting in this stage includes: adding. befoe the multiplication, division, o exponentiation with a independent vaiable, adding multiplication sign befoe the paentheses, eplacing e ( with an exp(, d/dx( with dif f( fo diffeentiation, etc. Ready to solve equation looks like this: y = ((x.ˆ(2) + 2. x + 2.ˆ(x)))./(exp(x.ˆ(5) + 5. x)); Fig. 12: MATLAB Recognizable equation as Retuned by the Pasing algoithm MATLAB solves the equation fo a given ange of x values, which use can also specify, and intelligently chooses the y-limits to show the impotant plot featues within the ange of x, and sends back to the client, the sting of x, y coodinates and thei espective anges. 5 CLIENT SIDE 5.1 Coodinate Axis Detection Once the seve has pocessed the use coected equation chaacte sting, the use is pesented with the camea peview fame. At this stage in the Andoid application, the plot is ceated and ovelaid onto the image of the use povided coodinate axes. The application must continually locate the axes and plot the points as the image of the axes shifts and otates in the peview fame. We implemented the image pocessing on the phone using the OpenCV [2] Java API. The image pocessing is done between peview fames, and theefoe, fo the application to emain stable (and not fustating to the use), the image pocessing must conclude quickly to minimize lag. Theefoe, this adds a limitation to the complexity of the image pocessing opeations. The goal of the coodinate axes detection system is to locate the endpoint coodinates of the two axes. Fom these endpoints, the plot aea, oigin, otation, and scale of the plot can be detemined. We stat the detection pocess by fist extacting the gayscale intensity image fom the camea peview image. The Andoid camea povides images which ae in the YUV colospace. It follows that we can simply use the Y data fom the fame and set it as the gayscale image. We then feed this gayscale image into OpenCV s Canny edge detecto to extact the edges of the coodinate axes. The Canny edge detecto povides a good binaization of the image if thee ae no extaneous edges in the camea s field of view such as pinted text o stay maks. The pimay issue with allowing extaneous blobs in the image is that they might be candidates fo the possible endpoints of the two lines. These maks can be emoved afte Canny edge detection, using a egion labelling and egion popeties algoithm. Howeve, as we will see below, OpenCV s implementation of this algoithm slows down the coodinate detection too geatly to be utilized. The next stage of the detection pocess is to feed the Canny edge detected image into a closing opeation to fill in the edges. The Canny edge detecto [8] etuns an image with the outline of the axis. We use a dilation and eosion opeation implemented by OpenCV using a stuctuing element that is a 20 by 20 pixel squae. The esult is an image with a egion that connects the two sides of each of the edges. Finally, we use a Hais keypoint detecto to detect the cones in the image. Typically, this will esult in two keypoints being detected at each endpoint of the line. In addition to these points, keypoints ae detected along the edges of the lines due to iegulaities in the dawn line o due to

7 NAQVI et al.: AUGMENTED REALITY EQUATION PLOTTER 7 stay maks that wee left ove afte the binaization pocess. We detect the endpoints of the lines by just picking the outemost keypoints (keypoints with the maximum x, minimum x, maximum y, and minimum y). Theoetically, on a clean pape with no majo stay maks, this pocess will wok unless the use has otated the axis too fa. In this case, some of the outlying keypoints will be the same, and the application will not be able to plot the points. Picking these fou keypoints allows the application to calculate the position of the oigin. The oigin is detemined by finding the intesection of the two lines fomed by the fou keypoints. A simple atio of deteminants easily solves the system of two linea equations. The next step is to stat plotting points. ρ = θ + φ (3) a = cos ρ (4) b = sin ρ (5) Whee is the distance fom the oigin of the new point, ρ is the angle the point subtends fom the hoizontal diection on the phone, and a and b ae the coodinates of the point in the phone s coodinate system. L x and L y can be calculated by finding the distance between the two end keypoints and the oigin. Each point the MATLAB seve esponds with can be tansfomed using this method, with anothe check to detemine if the point is in the positive y quadants o the negative, as the calculation of ρ would be a subtaction instead of an addition. Fig. 13: The coodinate tansfomation that needs to be pefomed to plot points coectly on the phone sceen. Figue 13 shows the coodinate tansfomation that needs to occu afte the axes ae detemined to calculate the new coodinates of a point given its coodinates in the plot ceated by MATLAB. The oiginal coodinate system, (xy) gets tansfomed into the detected coodinate system epesented by (x y ). The coodinate system is scaled and otated by φ. Howeve, the point s location in the phone s coodinate system needs to be calculated in ode to daw the point accuately. This system is epesented by the dotted lines. If the length L x and L y ae povided along with the coodinates of the new points, the new coodinates can be detemined as follows: = ( al x L x ) 2 + ( ) bl 2 y (1) L y θ = actan bl y/l y al x/l x (2) 6 EXPERIMENTAL RESULTS 6.1 Equation Recognition Algoithm was tested on seveal types of equations and unde vaious illumination conditions fo taking equation s image. We see convincing esults in all these attempts. Unfotunately a pefect output is fa fom eal at this stage; mostly atleast one chaacte is ecognized wong o paentheses ae not closed. Fotunately, the use intevention at this stage solves the emaining issues. Fo ou detected esults, we exploe futhe the eason of getting poo template matching fo cetain types of equations. Fo ou efeence example, we plot a histogam of the distibution of matched chaactes to the diffeent font styles in Figue 14. One not so obvious, but easily justifiable obsevation, is that chaactes get wongly matched to Aial font style alot and hadly to the Times New Roman. This is to be expected because Aial font chaactes have no complicated edges and have most thick egions. Ou smat chaacte fixing jumps in to the escue and we get eally got post-pocessing esults. Figue 15 shows the esults of the algoithm applied to a numbe of diffeent equation images and the coesponding esults. We notice that most bad esults come in when we ae tying to match chaacte pais: s, 8, e, c; i, l, t, ; and 6, (, 9, ) etc. Again, as seen in the figue; most matching is to the Aial font style chaactes, although all the images used had

8 8 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE Distibution of Chaacte Matching to Fonts Fequency ation, and Hais keypoint detecto, was chosen afte implementing seveal diffeent appoaches and examining the tadeoff between quality of detection and pocessing time. Since the eal time plotting aspect of the application needed to be pocessed with minimal lag time fom the actual peview fames, the appoach we would choose would have to be both quick and effective enough l n h d Mat bia St Roma Couie a i b Cam es New Cam Tim Aia Fig. 14: Distibution of Chaactes matched to the diffeent fonts in the Template Database equations witten oiginally in the Cambia Math style. Smat Chaacte Fixing: Distibution of Chaacte Matching to Fonts 3 Fequency y=e(.5x)c0s(4x) Template Matching: y=8(.5x)608(4x9 0 h Mat ia Std oman Couie bia Camb New R Cam es Tim l Aia Distibution of Chaacte Matching to Fonts 5 Template Matching: Fequency 4 y=2x3+x y=2x3+x2+4 0 Cam h td Mat oman Couie ia S bia Camb New R es Tim 6 Template Matching: 5 Fequency y=sin(c0s(.5x)) l Aia Distibution of Chaacte Matching to Fonts y=81n(6086.5x9) 1 0 h td Mat oman Couie ia S bia Camb New R Cam es Tim 25 l Aia Distibution of Chaacte Matching to Fonts y=81n6x^(2)+2x+2^(x)0* an(2x9^(-1x8^(6x^(3)+5x9) Fequency y=sin(x^(2)+2x+2^(x))*tan(2x)^(-1/e^((x^(3)+5x)) 0 h td Mat oman Couie ia S bia Camb New R Cam es Tim l Aia Fig. 15: Results of applying OCR on diffeent types of equations 6.2 Coodinate Axis Detection The appoach that we decided to use fo the final application using a Canny edge detecto, close ope- Fig. 16: Table of pocessing times as measued by the Andoid OS duing an axis detection and plot points cycle. Figue 16 shows the timing esults fo the thee appoaches we tested. Each test was pefomed using the same hand dawn coodinate axes to be detected. The fist appoach used backgound subtaction by subtacting the oiginal gayscale image with a dilated vesion. The flat dilation was pefomed using a stuctuing element shaped like a cicle with a diamete of 25 pixels. Then adaptive thesholding was used with a block size of 10, and the lagest egion was kept. The endpoints of the lines could be detemined by finding the outlying bight pixels. This pocessing was completed in 1320ms on the Motoola DROID phone. The second appoach we tested was using the Canny edge detecto with an OpenCV function that povided the coodinates of the endpoints of the lines it detected. This method completed in 2632ms. Finally, we tested the cuent appoach (explained in the Methods above). This pefomed in 978ms. All thee appoaches wee able to detect the axes. Howeve, the lage delays caused by the two fome appoaches pointed to using the last appoach as the final appoach. Figue 17 shows the oiginal camea fame containing the coodinate axis to be detected. Afte pefoming the Canny edge detection algoithm, the output image is shown in Figue 18. This image shows the detected edges as a line of pixels. As can be seen, the binaization pocess also emoves

9 NAQVI et al.: AUGMENTED REALITY EQUATION PLOTTER 9 Fig. 20: Hais keypoints detected Fig. 17: Oiginal gayscale camea peview. Fig. 18: Output of the Canny edges algoithm. Fig. 21: Resulting plot the backgound illumination vaiation. Figue 19 shows the esult of the closing opeation. Thee is now only one egion in the image. Next, the Hais keypoint detecto is used to find the endpoints of the axis. Figue 20 shows the keypoints that wee detected. Since the lines ae not pefectly staight, the Canny edge, and closing opeations leave small cones that the keypoint detecto picks up. This is not an issue because of ou outlie keypoint selecto. Figue 21 is a photogaph of the phone sceen displaying the plot that was geneated using the points etuned by the seve. As shown, the plot displays coectly ovelaid on the dawn coodinate axes Fig. 19: Afte applying close opeation. D ISCUSSION Coodinate Axis Detection The esults fo the axis detection appoach ae pomising. Fo elatively clean images, the pocess woks as expected. Howeve, the appoach does not take into account outlying egions and theefoe the axes would not be accuately detected. The appoach of egion filteing with the Canny edge detecto and Hais keypoint detecto could emedy this issue. Howeve, the delay additional pocessing intoduces is an issue that must be dealt with. Fo elatively newe phones, the pocessing powe upgade possibly could limit the time lag caused by the pocessing and moe sophisticated detection could be pefomed.

10 10 DIGITAL IMAGE PROCESSING COURSE PROJECT, JUNE 2013 If futue inteactivity featues of the application ae taken into consideation, the appoach used hee fo axis detection might be beneficial. It is both fast and accuate enough to detect the axis, but in addition, keypoints detected within the bounding box aound the plot aea defined by the limits of the axis ae not possible axis endpoints. Theefoe, maks made in the inteio of the plot do not affect the detection. An additional featue of the application could take advantage of this by allowing the use to daw points in the inteio of the plot aea. These could be detected as egions o possible keypoints that do not fall nea the axis. The coodinates of these use defined points could be povided fo futhe plot infomation. 8 FUTURE WORK This tool has many useful extensions and many beneficial applications. As a fist next step, we can wok on making the eal-time plotting of the gaph insensitive to pespective distotion along with skewness. We would also like to make the OCR insensitive to image captue, in paticula, making it insensitive to skew and pespective distotion. We can use the RANSAC MATLAB Toolbox fom the Gaphics and Media Lab of Lomonosov Moscow State Univesity [7] to compute RANdom SAmple Consensus (RANSAC) that ts to the top and the baseline of the equation in the image. In this way we can find the coected bounding box of the image and apply the pojective affine tansfom to deskew and pespective compensate the image. Also, to impove the chaacte ecognition in the cuent setup, we would like to make a moe detailed confusion matix and encode moe classifies fo minimizing false positives. We would also like to allow fo including the handwitten equation ecognition capability by taining an SVM as discussed ealie. Anothe goal of the poject that we stopped shot of achieving was to add the inteactive plot eading capability, whee use can put a dot within the axes pemises on the pape, and the tool can ovelay on the sceen the eading of the gaph at that location. Finally, being able to visualize 3D gaphs would be a gem of an accomplishment fo this type of a poject. seve. We get eally nice esults with elatively simple template matching scheme with elatively low numbe of post-matching image classifies. ACKNOWLEDGMENTS The authos would like to thank Pof. Bend Giod fo teaching them nifty image pocessing tools to solve many useful eal-wold poblems. They ae also indebted to the helpful Teaching Assistants, David Chen and Matt Yu, who was eve-eady to discuss poject ideas and povide citical feedback. Moeove, vaious intellectual discussions held with Sam Tsai, gaduate student at the SCIEN Lab, wee eally mind-opening and helpful fo cafting pathway fo chaacte ecognition. They also extend thei thanks to fiend, Patick Blaes, fo helping out in setting up the seve. REFERENCES [1] Tesseact Optical Chaacte Recognition Engine. Available: [2] Itseez, Andoid OpenCV, [Online]. Available: [3] R. Zanibbi and D. Blostein, Recognition and etieval of mathematical expessions, Intenational Jounal on Document Analysis and Recognition (IJDAR), vol. 15, pp , [4] S. S. Tsai, H. Chen, D. Chen, R. Vedantham, R. Gzeszczuk and B. Giod, Mobile visual seach using image and text featues, in Signals, Systems and Computes (ASILO- MAR), 2011 Confeence Recod of the Foty Fifth Asiloma Confeence on, [5] K. Chan and D. Yeung, Mathematical Expession Recognition: A Suvey, Intl J. Document Analysis and Recognition, vol. 3, no. 1, pp. 3-15, Aug [6] D. Blostein and A. Gbavec, Recognition of Mathematical Notation, Handbook of Chaacte Recognition and Document Image Analysis, pp , Wold Scientific, [7] Gaphics and Media Lab RANSAC Matlab Toolbox. Available: machineleaning/ansactoolbox [8] Canny, J., A Computational Appoach To Edge Detection, IEEE Tans. Patten Analysis and Machine Intelligence, 8(6):679698, CONCLUSION We have successfully showed that nea eal-time pinted explicit mathematical equation plotting can be achieved using Andoid OpenCV and MATLAB