Sketch-Based User Interface for Inputting Graphic Objects on Small Screen Devices

Transcription

1 Sketch-Based User Interface for Inputtng Graphc Objects on Small Screen Devces Lu Wenyn, Xangyu Jn 2, Zhengxng Sun 2 Department of Computer Scence, Cty Unversty of Hong Kong Tat Chee Avenue, Kowloon, Hong Kong SAR, Chna csluwy@ctyu.edu.hk 2 State Key Laboratory for Novel Software Technology, Nanjng Unversty, Nanjng 20093, PR Chna jxy@graphcs.nju.edu.cn Abstract. For small screen devces, such as PDAs, whch totally depend on a pen-based user nterface, tradtonal menu-selecton/button-clckng based user nterface becomes nconvenent for graphcs nputtng. In ths paper, a novel sketch-based graphcs nputtng user nterface s presented. By sketchng a few consttuent prmtve shapes of the user-ntended graphc object, the canddate graphc objects n the shape database are guessed and dsplayed n a ranked lst accordng to ther partal structural smlarty to what the user has drawn. The user can then choose the rght one from the lst and replace the sketchy strokes wth the exact graphc object wth proper parameters, such as poston, sze and angle. Ths user nterface s natural for graphcs nput and s especally sutable for schematc desgn. Introducton Currently, most graphcs nput/edtng systems, ncludng Mcrosoft Offce, Photo- Draw, Vso, and many CAD systems, ask users to nput graphc objects usng mouse/keyboard wth lots of toolbar buttons or menu tems for selecton. Ths clumsy user nterface s nconvenent when drawng graphc objects on small screen devces, such as PDAs, for there s no room to accommodate so many toolbar buttons and menu tems. The most convenent and natural way for human bengs to draw graphcs s va a pen-based nterface, that s, usng pens to draw sketches, just lke on a real sheet of paper. Moreover, t s even better to recognze and convert the sketchy curves drawn by a user to rgd and regular shapes mmedately. Ths s because, wth the onlne graphcs recognton as an mmedate and useful feedback, the user can realze errors or napproprateness earler and therefore draw dagrams more perfectly. In ths paper, we dscuss the ssues of nputtng composte graphc objects usng the sketch-based user nterface. When drawng a pre-defned graphc object, the user tends to dvde the graphc objects nto several prmtve shapes, whch are usually convex polygons (users usually do not regard a concave shape as an entre shape and tend to nput them n several dfferent parts) or ellpses. The user can nput a prm-

2 tve shape n a sngle-stroke or n several consecutve strokes. In our proposed user nterface, we dscover latent prmtve shapes among user-drawn strokes and show the regularzed shape on the screen mmedately. Users can then adjust the recognton and regularzaton results by the suggeston of the system. Ths mmedate feedback strategy makes the user nteracton smoother and humanstc. Moreover, t has an extra advantage to reduce nner-stroke and nter-stroke noses, whch are usually ntroduced by the non-profcency or un-professonalsm of the user. After beng recognzed and regularzed, prmtve shapes, whch belong to one graphc object, are grouped together accordng to ther spatal and temporal relatonshps. They are then segmented and combned to form an object skeleton. Partal structural smlartes are calculated between the user-drawn graphc object and the canddates n the database. The graphc objects that are the most smlar to what the user has drawn are suggested to the user n a ranked lst for the user to choose and confrm. In ths way, the user does not need to manually go to menus to fnd and select what he/she wants. Hence, t s a more effcent, convenent, and natural way to nput composte graphc objects. We have mplemented ths user nterface for composte graphc object nputtng n our on-lne graphcs recognton system Smart Sketchpad []. Experments have shown the effectveness and effcency of the shape smlarty measurement and the naturalness and convenence of the user nterface. 2 Related Works Very few research works have been done on such on-lne graphcs recognton. Zeleznk et al. [2] have nvented an nterface to nput 3D sketchy shapes by recognzng the defned patterns of some nput 2D graphcs that correspond to certan sketchy sold shapes. Some sketchy nput nterfaces such as the one developed by Gross and Do [3], are manly for conceptual and creatve layout desgns. Fonseca and Jorge and ther group [4][5] have mplemented an on-lne graphcs recognton tool that can recognze several classes of smple shapes. But ther recognton approach s based on global area calculaton. Ths smple rule can hardly dstngush ambguous shapes such as pentagon and hexagon and therefore cannot acheve hgh recognton precson generally. The sketch recognton work reported by Arvo and Novns [6] s manly for 2D on-lne graphcs recognton. Ther approach contnuously morphs the sketchy curve to the predcted shape whle the user s drawng the curve. However, ther man purpose focuses on the user studes of such sketch recognton system. Moreover, ther recognton approach only handles two smplest classes of shapes (crcles and rectangles) drawn n sngle strokes. Ths s not adequate for a real software tool that can be used for nputtng most classes of dagrams. Anyway, ther prelmnary concluson shows that such nterface s partcularly effectve for rapdly constructng dagrams consstng of smple shapes. Hence, n order to provde the capablty to nput more complex dagrams, t s necessary to extend the sketchy graphcs recognton approach to handle more complex and composte shapes.

3 3 The Proposed Approach The mnmum nput unt of a graphc object n an on-lne graphcs recognton system s a stroke, whch s the trajectory of the pen movement on a tablet wth the pen-tp touchng the tablet between the tme when the pen-tp begns to touch the tablet and the tme when the pen-tp was lfted up from the tablet. Several consecutve strokes can consttute a prmtve shape, whch can be a straght-lne segment, an arc segment, a polygon, or an ellpse, etc. Composte graphc objects are more complex shapes that consst of several prmtve shapes. The objectvty of on-lne graphc object recognton s to convert the nput strokes nto the user-ntended graphc objects. In our approach, we frst recognze the nput strokes nto prmtve shapes. The latest nput prmtve shapes (based on the assumpton that the component of the same object must be nputted consecutvely) are then grouped together accordng to ther locaton vcnty as a query example. A content-based graphcs retreval procedure then starts n the object database based on partal structural smlartes between the query example and those predefned composte graphc objects n the database. The most smlar composte graphc objects are dsplayed n a ranked lst for the user to choose. If the user cannot fnd hs/her ntended objects, he/she can contnue to draw other components untl the ntended shape appears n the lst and can be dragged and dropped to the drawng area. In order to avod too much ntrusve suggestons of composte shapes of low smlarty, only those canddates whose smlartes are above a threshold wll be dsplayed for suggestons. Relevance feedback based on the component s nput sequence s also employed to rase the system s performance. There are four major stages n our proposed approach: prmtve shape recognton, prmtve shapes groupng, partal structural smlarty calculaton, and relevance feedback. We wll dscuss them n detal n the followng subsectons. 3. Prmtve Shape Recognton The frst stage of our approach s prmtve shape recognton. In ths stage, an nput sketchy lne s frst recognzed as a known prmtve shape type (whch can be a lne segment, an arc segment, a trangle, a quadrangle, a pentagon, a hexagon, or an ellpse). The recognzed shape s then regularzed to the most smlar rgd one that the user mght ntend to draw. The entre prmtve shape recognton process s dvded nto four sub-processes: pre-processng, feature extracton, closed-shape recognton, and shape regularzaton. 3.. Pre-Processng Due to non-profcency or un-professonalsm, the sketchy lne for an ntended shape nput s usually very cursve/unshaped and free-formed. For example, wthout usng a ruler, a straght lne drawn by a drafter s not so straght f measured strctly no matter how much attenton the drafter s payng to the drawng operaton. More often, the sketchy lne s not properly closed. Hence, the sketchy lne s not sutable for feature extracton drectly. Pre-processng s frst done to reduce all knds of noses. The preprocessng stage ncludes four sub-processes: polygonal approxmaton, agglomerate

4 ponts flterng, end pont refnement, and convex hull calculatng. Many ntermedate ponts on the sketchy lne are redundant because they le (approxmately) on the straght-lne segment formed by connectng ther neghbours. These ponts can be removed from the chan so that the sketchy lne can be approxmately represented by a polylne (an open polygon) wth much fewer crtcal vertces. We apply the algorthm developed by Sklansky and Gonzalez [7] to mplement polygonal approxmaton n ths paper. Due to the shaky operatons caused when the pen-tp touches the tablet and when t s lfted up, there are often some hooklet-lke segments at the ends of the sketchy lnes. There mght also be some crclet at the turnng corner of the sketchy lne. These noses usually reman after polygonal approxmaton. Agglomerate ponts flterng process s ntroduced to reduce these noses. The man dea of ths process les n the dfference of pont densty. Polylne segments, whch have a hooklet or crclet, usually have much hgher pont densty than the average value of the whole polylne. The task of agglomerate ponts flterng s to fnd such segments and use fewer ponts to represent the segment. Because t s dffcult for the user to draw a perfectly closed shape, the sketchy lne s usually not closed or forms a cross near ts endponts. In other words, t has mproper endponts. These mproper endponts are great barrers for both correct shape recognton and well regularzaton. For a sketchy lne that has cross endponts, we delete ts extra ponts to make t properly closed. For a sketchy lne that s not closed, we extend ts endponts along ts end drectons and make t closed. After that t can undergo other processng as f t were prevously closed. The sketchy lne the user draws s often very cursve, and mght also be concave. These noses have strong mpact on the later feature extracton stage. We employ the classcal algorthm developed by Graham [8] to obtan the convex hull of the vertces set, whch s used to represent ts orgnal lne and therefore remove those noses. Expermental results show that usng convex hull nstead of the orgnal nput stroke helps to rase the precson of prmtve shape recognton n general cases, although t mght ntroduce extra noses. After pre-processng, lne segments and arc segments can be dstngushed from other closed-shapes by ntutve rules. A closed-shape s then represented by a polygon and needs further classfcaton nto more elaborate types (ncludng trangle, quadrangle, pentagon, hexagon, and ellpse). Feature extracton and closed-shape recognton are key processes to fulfll ths objectve Feature Extracton and Closed-Shape Recognton We regard all polygons havng the same vertex number as the same shape type. For nstance, damonds, squares, rectangles, trapezods, and parallelograms are all regarded as quadrangles. Hence, the feature we used for recognton must be rrelevant to the polygon s sze, poston, and rotaton. Thereby, we employ the turnng functon [9] to obtan the feature vector of the polygon representaton of the sketchy stroke. Turnng functon Θ A (s) measures the angle of the counter-clockwse tangent as a functon of the arc-length s, starng from a reference pont O on a polygon A s boundary. Thus Θ A (0) s the angle v of the tangent at O from the x-axs, as n Fg.. Θ A (s)

5 accumulates the turnng angles (whch s postve f the turnng s left-hand and negatve otherwse) as s ncreases. Θ(s) v v+2π O v 0 s Fg.. A polygon and ts turnng functon Our defnton of turnng functon s a lttle dfferent from the commonly used one [9]. We see that the prevous defnton s dependent on both the traversal drecton of the polygon and the reference orentaton. We scaled the polygon so that ts permeter length s. We use the tangent at O as the reference orentaton such that Θ A (0)=0 and determne a traversng drecton such that all turnng angles are postve. Hence, our turnng functon Θ A s a monotonous ncreasng one from [0, ] nto [0, 2π]. If an m-dmensonal feature vector s needed, we equally dvde the boundary of the polygon nto m peces (m=20 n ths paper). Each element of the feature vector s the turnng degrees accumulated n a correspondng pece of the polygon, as s defned n Eq. (). ϕ f V = ϕ + 2π ϕ 0 f ϕ < 0 mod m where ϕ = Θ' A ( ) Θ' A ( ). m m After feature extracton, we classfy closed-shapes nto more elaborate prmtve shape types. A mult-class classfer based on support vector machnes [0] s used to assgn each nput stroke to ts correspondng prmtve shape type Shape Regularzaton After shape type s known, fttng s employed to adjust the shape parameters (e.g., three vertces for a trangle) such that the recognzed shape can best ft ts orgnal closed curve. We employ two basc types of fttng processes: ellpse fttng and polygonal fttng. For ellpse fttng, we frst determne ts axes orentatons by fndng the egenvectors of the covarance matrx of the samplng ponts along the orgnal curve at equ-length steps. The stroke s enclosng rectangle whose edges are parallel to the axes s used to determne the centre and axes lengths of the ellpse. For N-edge polygonal fttng, we frst fnd ts nscrbed N-edge polygon that has the maxmal area. As a result, we cut the orgnal stroke nto N peces by the vertces of ths nscrbed polygon. By fndng the lnear regresson result of each pece, the edges of the optmzed N-edge polygon can be obtaned. The nput shape that the user has drawn cannot precsely match the one he/her ntends to nput. Rectfcaton process s employed to make the shape very smlar to the ()

6 one that the user has n mnd. Ths process s currently rule based, ncludng two subprocesses called vertcal/horzontal rectfcaton and rgd shape rectfcaton. The shape s gradually regularzed to a rgd shape as regular as possble followng the arrowheads n Fg.2. trangle Horzontal and vertcal regularzaton ellpse quadrangle rght trangle crcle parallelogram trapezod sosceles trangle damond rectangle equlateral trangle square Fg.2. Shape rectfcaton 3.2 Prmtve Shapes Groupng After prmtve shape recognton, we group together the recognzed and regularzed prmtve shapes that the user has already drawn to guess hs/her ntended composte graphc object. Based on the assumpton that the user tends to draw an object n consecutve strokes, we only group together the latest drawn prmtve shapes accordng to ther locaton vcnty. Denote a group as SG, whch conssts of k prmtve shapes (S, S 2, S k ). Denote ψ as a functon to get the center pont of the boundng box of a shape or a shape group. Defne Ds(x, y) as a functon to get the dstance between two ponts. We have the followng defntons: d = σ max = k = ( Ds( ψ ( SG), ψ ( s ))) d Ds ( ψ ( SG ), ψ ( where d represents the compactness of components. The smaller d s, the more compact these components are. σ represents the mbalance between dfferent components. The larger σ s, the larger the possblty that a component does not belong to ths object. At ntalzaton, the shape group SG=φ. Then, we add prmtve shapes from the latest drawn one and follow the reverse nput sequence untl one of the four condtons s met:. No ungrouped shapes avalable, s )) (2) (3)

7 2. d > d max, 3. σ > σ max, or 4. The boundng box of SG s bgger than a gven threshold. Thus the ultmate SG s the shape group we guess for the user-ntended shape. Fg.3 llustrates an example of prmtve shapes groupng. The red lne segment s the latest nput shape, and t can be grouped together wth two quadrangles. The crcle cannot be grouped n because t would make the combned shape mbalanced (the σ value would exceed the threshold). Fg.3. Prmtve shapes groupng 3.3 Partal Structural Smlarty Calculaton After prmtve shape groupng, the combned shape group can also be regarded as a composte object. Ths object (the source object) and each canddate object (the destnaton object) n the database are then compared for smlarty assessment. Snce the nputted shape mght be n an ncomplete form, the smlarty s not symmetrc between the source and destnaton objects. It s a partal smlarty measured from the source to the destnaton objects. For example, f the source object (an ncomplete object) s a part of the destnaton object (a complete object), we thnk they are hghly smlar because the source object can be completed later. However, f the source object contans certan components that do not exst n the destnaton object, we thnk ther smlarty should be very low, no matter how smlar other correspondng parts are. Frst, we normalze the source object and the canddate object nto a square of 00*00 pxels. Then we calculate the partal structural smlarty n two stages. In the frst stage, prmtve smlarty s only measured based on matched prmtve shapes. In the second stage, these matched prmtve shapes are removed from both the source and the desgnaton objects. More elaborate shape smlarty s then adjusted accordng to the smlarty of the topologcal structures of the remaned parts Smlarty between Prmtve Shapes An ntutve way to calculate the smlarty between the source object and the destnaton object s to compare ther components one-by-one based on dentcal shape types and relatve postons. The overall smlarty can be acqured through a weghted sum of smlarty of ther matched components. We defne Average Pont Shftng (APS) for two prmtve shapes f ther types are dentcal through the followng rules:. If both are lne segments, denote them as P -P 2 and P 3 -P 4. The APS of two lne segments s defned as

8 APS L = ( Mn( Ds( P, P3 ) + Ds( P2, P4 ), Ds( P, P4 ) + Ds( P2, P3 )) (4) 2 2. If both are arc segments, denote them as P -P 2 -P 2 and P 3 -P 34 -P 4, where P 2 and P 34 are the mddle pont of an arc. The APS of two arc segments s defned as APS A = ( APS L + Ds( P 2, P 34 )) (5) 2 3. If both are n-polygons (n s the number of vertexes), denote ther vertex lsts (n the same traversal order) as P 0, P, P n- and Q 0, Q, Q n- respectvely. The APS between two polygons s defned as n n APS P = Mn( Ds( P, Q ( + j) mod n)) (6) j= 0 n = 0 4. If both are ellpses, the APS s defned as the APS P between ther boundng boxes. The partal structural smlarty between two composte objects s then defned based on APS. Denote the component set of the source object as S, whch has m components S, S 2, S m. Denote the component set of the destnaton object as D, whch has n components D, D 2, D n. If m=0, the smlarty s defned as. If m>n, the smlarty between them s 0. Otherwse (n m), we create a mappng j form [, m] to [, n], whch satsfes the followng two condtons:. For each [, m], S and D j() are of the same shape type. 2. For, 2 [, m] and 2, j( ) j( 2 ). Enumerate all these possble mappngs and denote them as a set J. The partal structural smlarty between the source object and the destnaton object can be obtaned by m APS( S, D j( ) ) Sm prm ( S, D) = Max( ) (7) j J m = 00 2 where 00 2 s the length of the dagonal of the normalzed area Smlarty between Object Skeletons Admttedly, dfferent users may have dfferent opnons n decomposng a composte object nto prmtve shapes. Even the same user may change hs deas from tme to tme. E.g., for the composte graphc object called envelope, a user may nput t by drawng a pentagon and a trangle. However, he may also nput t by drawng a pentagon and two lne segments. See Fg.4 for llustratons of two ways to draw the envelope. Although the ultmate combned objects look totally the same, they cannot match each other through the prevous strategy presented n Sect Moreover, t s extremely dffcult to enumerate all the possble combnatoral ways to draw a composte graphc object, n both spatal and temporal consderatons. Hence, we also need to fnd other ways to measure smlarty between two composte objects.

9 = + = + + Fg.4. A composte graphc object may be drawn n several dfferent ways Object skeleton s a graph we use to represent the topologcal structure of the composte object. In order to acqure the skeleton of the object, we frst generate an ntal canddate skeleton and then merge redundant nodes and edges to obtan the ultmate one. See Fg.5 for llustraton. Node generaton s frst done based on the followng rules:. All vertces of polygons are canddate nodes, such as Pont ~7. 2. All end ponts of lne and arc segments are canddate nodes, such as Pont 8 and The mddle ponts of arc segment are canddate nodes. 4. All of the cross ponts (of two dfferent prmtve shapes) are canddate nodes, such as Pont 0 and (a) the destnaton object n the database (some prmtve shapes has been removed) (b) the source object that the user has nput (some prmtve shapes has been removed) (c) the ntal skeleton for (b) (d) the ultmate skeleton for (b) Fg.5. Object skeleton acquston After nodes generaton, these canddate nodes dvde the polygon and lne segment nto several peces. Each pece can be logcally represented by an edge (a lne segment). If a node s dstance to an edge s under a threshold, we move ths node on the edge (choose the most nearest pont on the lne segment to replace the node) and splt the edge nto two peces. E.g., n Fg.5 (c), node 5 splts the edge between node and 2 nto two peces. After gettng the ntal skeleton, those nodes that are close enough are merged. Edges must be correspondngly adjusted to adapt to the new nodes. All fully overlappng edges should be merged nto one. E.g., n Fg.5 (c), node 4 and 6 are combned nto node 4. And node 7and 3 are combned nto node 3. Thus, the edge between node 6 and 7 s then elmnated. After these processes, we can get the ultmate object skeleton, as llustrated n Fg.5 (d). The object skeleton s represented by a graph consstng of a node set and an edge set. The smlarty between object skeletons s calculated based on ther topologcal structures. Denote the node set of the source object skeleton as NS, whch has m elements. Sort these nodes accordng to the sum of ther x coordnate and y coordnate descendent, denote them as NS, NS 2, NS m. Denote the node set of the destnaton object skeleton as ND, whch has n elements. Use the smlar way sort them n a lst, denote them as ND, ND 2, ND n. Defne the edge set of the source object as ES and

10 the edge set of the destnaton object as ED. Defne logcal edge set of the destnaton object LED={(, 2 ) (ND, ND 2 ) ED,, 2 [, n]}. If m=0, the smlarty s defned as. If m>n, the smlarty between these two skeletons s 0. Otherwse (n m), create a mappng j from [, m] to [, n] whch satsfes the followng two condtons.. For each, 2 [, m] and 2, j( ) j( 2 ). 2. For each, 2 [, m] and < 2, j( )-j( 2 )< ε, where ε s an nteger threshold. Defne the logcal mapped edge set under mappng j as LME j ={(j( ), j( 2 )) (ND, ND 2 ) ED and, 2 [, m]}. Thus we can defne the followng three metrcs:. Average Node Shftng (ANS) 2. Edge Precson (EP) ANS j = m m = Ds( NS, ND 00 2 j( ) ) (8) LME j LED EPj = (9) LME j 3. Edge Recall (ER) LME j LED ER j = (0) LED Enumerate all these possble mappng j to form a set denoted as J. The smlarty between two object skeletons s defned as Sm S, D) = Max( w ( ANS ) + w EP + w ER ) k ( j 2 j 3 j j J Where w, w 2, w 3 >0 and w +w 2 +w 3 =. The computatonal complexty of ths algorthm depends on ε. If ε=0, the number of all possble mappng s C. If ε n, the m n m number of all possble mappng s P n. Hence, the computatonal complexty of ths m m algorthm s between C and P n n. The larger ε s, the larger the computatonal cost s, and more accurate result can be obtaned The Overall Smlarty Calculaton The computaton complexty of smlarty calculaton between object skeletons s very hgh for a real-tme system. Hence, we must reduce the node number to save calculaton tme. We fnd users have common senses to group some parts of a composte graphc object nto one prmtve shape. For nstance, users usually draw some specal prmtve shape types (E.g., ellpse, hexagon, etc.) n a sngle stroke and seldom dvde them nto peces (multple strokes). The same stuaton s for some solated prmtve shapes, whch do not have ntersecton wth other parts of the object. These prmtve shapes can be separated from other strokes of both the source and the destnaton objects and frst compared by our prmtve shape comparson strategy. Then, the remaned parts of both objects can be compared through our object skeleton comparson strategy. Thus the node number of object skeleton can be much reduced. De- ()

11 note the source object as S and the destnaton object as D. Denote the separated prmtve shapes of S and D as S prm and D prm respectvely. Denote the remaned parts of S and D are S rem and D rem respectvely. The overall smlarty between S and D s a lnear combnaton of the smlartes between S prm and D prm and that of S rem and D rem, as defned n Eq. (2). 0, f Sm ( S, D) = k Sm prm where k, k 2 >0 and k + k 2 =. Sm ( S prm prm ( S, D prm prm, D prm ) = 0 ) + k Sm ( S 2 k or rem, D Sm ( S rem k ), rem, D rem ) = 0 otherwse (2) 3.4 The User Interface and Relevance Feedback We mplement the proposed approach n our SmartSketchpad [] system. After only a few components of a composte object are drawn, the smlartes between the nput source object and those canddates n the database are then calculated usng the partal structural smlarty assessment strategy. The most smlar objects are dsplayed n a ranked canddate lst (accordng to the descendng order of smlartes) n the smart toolbox for the user to choose. If the user cannot fnd hs/her ntended shape, he/she can contnue to draw other components untl the ntended one appears n the lst. In order to avod too much ntrusve suggeston of composte shapes of low smlarty, only the frst ten objects whose smlartes are above a threshold (whch s 0.3 currently) are dsplayed n the smart toolbox for suggestons. The smart toolbox wll not gve any suggeston f less than two components are drawn. E.g., f a user wants to nput a graphc object called bear, he/she can frst draw ts face and an ear by sketchng two crcles. But the ntended object does not appear n the smart toolbox. Then the user contnues to draw ts rght ears by sketchng a smaller crcle. Ths tme, the wanted object appears and s ranked as the 5th n the smart toolbox for selecton. If the user dd not notce ths, he/she can contnue to draw ts left eye by sketchng a lne segment. # Input Sm Rank Canddates sorted n descendng smlarty ID: # of nput components Sm: Smlarty of the ntended object Input: The nput components Rank: Rank of the ntended object Fg.6. Input a composte graphc object by sketchng a few consttuent prmtve shapes Ths tme, only three objects are lsted and the wanted object s ranked as the frst n the smart toolbox. See Fg.6 for llustraton. However, the partal structural smlarty calculaton strategy we have proposed s not perfect yet. Durng the drawng processes, rrelevant objects may obtan hgher ranks n the object lst. Sometme, users have to draw the ntended object completely so that t could be seen n the lst. Or,

12 even after the entre object s drawn, t stll does not appear n the smart toolbox. In ths case, the user has to clck the more button contnually to fnd hs/her ntended object. In order to avod such stuaton, relevance feedback s employed to mprove both nput effcency and accuracy. Because each user has hs/her drawng style/preference, a specfc user tends to nput the same object through a lst of components of fxed consttuton (e.g., two ellpses and a trangle) and fxed nput tme sequence (e.g., ellpse-trangle-ellpse). Hence, the nput prmtve shape lst offers much more nformaton that we have not utlzed n our partal structural smlarty calculaton. Each tme when the user drags an object from the smart toolbox, the current grouped nput prmtve shape lst wll be saved as a word ndex for ths ntended object wth a probablty. E.g., a lst rectangle-trangle may be used for ndex of envelope and arrow box. The probablty can be easly obtaned from the relatve tmes when the ntended object s selected for ths word. Hence, for an nputted source object and a canddate object n the database, we not only can acqure ther shape smlarty usng the partal structural smlarty calculaton strategy, but also can acqure the probablty (ntally, all possbltes are set to 0) that the canddate object s just the ntend one. We lnearly combne them wth dfferent weghts to obtan a new metrc for smlarty comparson. Therefore, f a user always chooses the same canddate object for a specfc nput prmtve shape sequence, the probablty of ths canddate object must be much hgher than that of other canddate objects. Obvously, through ths relevance feedback strategy, the ntended object wll obtan a hgher rank than before. 4 Performance Evaluatons In ths experment, we created 97 composte graphc objects for expermentaton n SmartSketchpad. All these objects are composed of less than ten prmtve shapes, as shown n Fg.7. The weghts and thresholds we used are ε=20, w =0.4, w 2 =0.3, w 3 =0.3, k =k 2 =0.5.We randomly selected 0 objects (whose ID s 73, 65, 54, 88, 22, 5, 2, 8, 8, and 76) and draw these objects as queres. We recorded the ntended object s rank varaton together wth the drawng steps. Results of four objects are shown n Fg.8. The horzontal coordnate s the nput steps. In each step, only one component s drawn untl the object s properly fnshed. The vertcal coordnate s the rank of the ntended object. In most cases, the ntended object wll appear n the smart toolbox (ranked n the frst 0) after only a few components are drawn. Take Object 73 as an example. The user can nput ths object n 6 steps. After four components are drawn, the ntended object s rank s 4. Thus the user can drectly drag t form the smart toolbox. Averagely (of the ten objects we have tested), after 85.7% components of the ntended object are drawn, t wll be shown as the frst one n the smart toolbox. In order to see t n the smart toolbox (top 0), only 67.0% of the ntended object needs to be drawn. And the rato s 68.7% for top 5 and 72.0 for top 3. The user does not need to draw the complete sketch for a composte graphc object. Only a few sketchy components are suffcent for the system to predct the user s ntent. Hence, ths user nterface provdes a natural, convenent, and effcent way to nput complex graphc objects.

13 Fg.7. The composte graphc objects we have created for expermentaton Object 73 Object Object 76 Object 5 Fg.8. The ntended object ranks (vertcal axes) and nput steps (horzontal axes) 5 Summary We proposed a novel user nterface to nput composte graphc objects by sketchng only a few of ther consttuent smple/prmtve shapes. The nputted shape s smlartes to those objects n the database are updated n the drawng process and those promsng composte objects are suggested to the user for selecton and confrmaton. By dong so, the user s provded wth a natural, convenent, and effcent way to

14 nput composte graphc objects and can get rd of the tradtonal graphc nput nterface based on menu selecton and ths user nterface s especally sutable for small screen devces. 6 Acknowledgement The work descrbed n ths paper was fully supported by a grant from Cty Unversty of Hong Kong (Project No ) and partally supported by a grant from Natonal Natural Scence Foundaton of Chna (Project No ). 7 References. Lu, W., Qan, W., Xao, R., and Jn, X.: Smart Sketchpad An On-lne Graphcs Recognton System. In: Proc. of ICDAR200, Seattle (200) Zeleznk, R.C., Herndon, K.P., Hughes, J.F.: SKETCH: An Interface for Sketchng 3D Scenes. In: SIGGRAPH96, New Orleans (996) Gross, M.D., Do, E.Y.L.: Ambguous Intentons: A Paper-lke Interface for Creatve Desgn. In: Proc. of the 9th Annual ACM Symposum on User Interface Software and Technology, Seattle, WA (996) Albuquerque, M.P., Fonseca, M.J., Jorge, J.A.: Vsual Language for Sketchng Document. In: Proc. IEEE Sym. on Vsual Languages (2000) Fonseca, M.J., Jorge, J.A.: Usng Fuzzy Logc to Recognze Geometrc Shapes Interactvely. In: Proc. 9th IEEE Conf. on Fuzzy Systems, Vol. (2000) Arvo, J., Novns, K.: Flud Sketches: Contnuous Recognton and Morphng of Smple Hand-Drawn Shapes. In: Proc. of the 3th Annual ACM Symposum on User Interface Software and Technology, San Dego, Calforna (2000) 7. Sklansky, J., Gonzalez,V.: Fast Polygonal Approxmaton of Dgtzed Curves. Pattern Recognton 2 (980) Graham, R.L.: An Effcent Algorthm for Determnng the Convex hull of A Fnte Planar Set. Informaton Processng Letters (4) (972) Arkn, E.M., Chew, L.P., Huttenlocher,D.P., Kedem, K., Mtchell, J.S.B.: An Effcent Computable Metrc for Comparng Polygonal Shapes. IEEE Trans. on PAMI 3(3) (99) Vapnk, V.: The Nature of Statstcal Learnng Theory. Sprnger-Verlag, New York (995)