Online Graphics Recognition: State-of-the-Art

Transcription

1 Online Graphics Recognition: State-of-the-Art Liu Wenyin Dept of Computer Science, City University of Hong Kong

2 Outline Introduction: Problem and Motivation Primitive Shape Recognition Composite Graphic Object Recognition Document Recognition and Understanding Performance Evaluation and User Study Summary Open problems

3 The Problem Input graphic objects by freehand sketching on tablet Instant and continuous recognition when the strokes are added determine type/class & parameters (& semantics) (optionally) beautify appearance find (due to feedback) and correct error earlier Similar problem: handwriting

4 Motivation Driven by Pen-based interaction devices Ubiquitous computing Natural, humanistic, & convenient UI TabletPC heating up the market of handwriting software e.g., Microsoft and IBM are developing their own handwriting software

5 Motivation: Why Sketch-based? Current UI for Graphics Input Mouse clicks on menus & buttons Disadvantages of Mouse and Menus Inconvenient: too many clicks; hard to remember/find Unnatural: interruptive, unsuitable for quick tasks/ideas Unsuitable for small screen devices Graphics Input UI for Creative Design by Sketching Sketching with a pen is a mode of informal, perceptual, direct interaction: especially good for creative design tasks Continuous, non-interruptive, natural ways for interaction Productivity, instant feedback for quick correction/position

6 Humanistic Sketch-based UI: Design Principles Same way as human: informal, fast/instant feedback, ambiguity, creativity... Machine adapts to human, and not vice versa Efficient and Intelligent Guess user s intention and do accordingly Supported by graphics recognition: sketchy to regular Personalized Learn user s drawing styles, habits, preferences from user actions (as virtual feedback)

7 Graphic Document Graphic m Visual Token Objects Text Composite Graphic Object Primitive Shape 2..m Stroke 1..m

8 Graphic Objects Stroke--trajectory of a pen movement during touching a tablet minimal unit of user input, represented by a chain of points Primitive Shape--simple shape: closed/open, single/multi-strokes limited # of classes: e.g., triangles, ellipses, or line/arc segments Composite Graphic Object--consisting of 2+ primitive shapes assumption: input components of one object consecutively & adjacently Visual Token--component of a graphic document just like words in a text document composite graphic objects or free/single primitive shapes Graphic Document--complete document for a purpose composed of visual tokens semantics: defined by the tokens, their parameters & spatial relations

9 Recognition Tasks/Stages 1. Primitive shape (or stroke) recognition: simultaneous recognition immediate recognition 2. Composite graphic object recognition: simultaneous recognition immediate recognition 3. Document recognition and understanding: simultaneous recognition immediate recognition

10 Primitive/Stroke Recognition Determine type & parameters (pos, size, & orientation) Regularize/beautify to the most common form (params) because it is intents of most users In most cases, immediate recognition: recognize after it is completely input Single or multiple strokes link multiple strokes first Gestures: visual commands (e.g., undo/re-do, remove) represented by primitive shapes & need this level recognition

11 Examples of Editing Gestures SILK (Landay and Myers, IEEE Computer 2001) Different groups developing different set of gestures Standardization is necessary

12 Simultaneous Recognition Fluid Sketches (Arvo and Novins, UIST2000) while a freehand stroke is being input guess/suggest what the user is intending to draw simultaneous or immediate feedback

13 Fluid Sketches (Arvo & Novins)

14 Primitive Recognition: 4 Stages Stroke curve pre-processing Shape classification Shape fitting Shape regularization/beautification

15 Stroke Pre-Processing: Problem Input: a freehand stroke Output: a refined polyline Requirement: the output is similar to the input freehand stroke but with some necessary perfection (noise reduction) Link multiple strokes or segment a single stroke if necessary

16 Pre-Processing Hooklet Circlet Polygonal approximation with ε = 1.0 pixel Polygonal approximation with ε = 5.0 pixels The sketchy line before processing The sketchy line after processing The sketchy line before processing After pulling the end points After deleting extra points

17 Shape Classification: Problem Input: the refined polyline Output: the type id of a basic shape class: e.g., line, triangle, quadrangle, pentagon, hexagon, ellipse, or free curve Requirement: correctness: the type (output) is of the user s intent

18 Shape Classification Based on features extracted from the stroke s vector polyline (or image) represent the stroke Many pattern recognition methods can be used: Rule-Based Approaches Neural-Network-Based Approaches SVM-Based Approaches etc. e.g., Ernesto Tapia and Raul Rojas (ICDAR 2003)

19 Features Used in Recognition Corners can be found by speed (Davis 2002; Calhoun et al. 2002) curvature Turning angle functions (Arkin et al. 1991) Attraction force model (Jin et al. PG2002) Stroke order and direction especially for composite objects Domain-specific or independent knowledge

20 Corners: Speed & Curvature Davis (2002) Calhoun et al. (2002)

21 Turning Angle Functions Arkin et al. (IEEE T-PAMI 1991) T(s) v v+2p O v 0 1 s

22 Attraction Force Model Jin, Liu, Sun & Sun (2002) Inner angle of attracted point Inner angle of the attracting point Distance between the two points α f ( A, B) = 2 βdis ( A, B) B A (a) C B A B B A C C C (b) (c) (d)

23 Decision Making Rule-based: # of corners (or vertices) Construction of Classifiers SVM (can be used for incremental learning) One-against-one structure: n(n-1)/2 classifiers One-against-all structure: n classifiers Max-win scheme Training with samples Neural Network

24 Shape Fitting: Problem Input: the type id the stroke (original and refined polyline) Output: the fitted shape (characterized by parameters) Requirement: the output has the lowest average distance to the input stroke

25 Shape Fitting Polygonal Fitting: (a) (b) Ellipse Fitting: the axis orientation the center point (c) y (d) x the axis orientation (a) (b)

26 Shape Regularization: Problem Input: the fitted shape Output: the regularized shape (characterized by parameters) Requirement: the output is similar to the original freehand stroke but also appears in its most beautiful form: e.g., conforming as much as possible to connectedness, perpendicularity, congruence, and symmetry, intended by the user. Also referred to as beautification (Igarashi et al. 1997)

27 Shape Regularization Inner-Shape Regularization Inter-Shape Regularization

28 Inner-Shape Regularization Equilateral Rectification Edges and axes Parallelism Rectification Edges Special Angle Rectification 90, 30, 45, 60, 120, etc. Horizontal/Vertical Rectification Edges, axes, diagonals

29 Inner-Shape Rectification Rules Fitted Shape triangle ellipse quadrangle right triangle circle parallelogram trapezoid isosceles triangle diamond rectangle equilateral triangle square

30 Inter-Shape Rectification Affected by neighbors in a document Size Rectification Position/orientation Rectification Alignment Intersection Tangency Concentric

31 Composite Object Recognition: Problem After recognizing the current stroke Combine the current shape with previous ones Based on their sequential & spatial relationship assumptions: consisting of 2+ primitive shapes input components consecutively & adjacently Determine or predict the type & parameters of the composite object Regularization or beautification

32 Composite Object Recognition: Approaches Classifier-based approaches: decision tree Fonseca and Jorge (2000): fuzzy Peng, Sun, Liu, & Cong (GREC2003) Similarity-based approaches Based on similarities of component and constraints Representation: ARG(Li 2000), RAG(Lladós 2001) Relational distance metric (Shapiro 1993) Directional shape similarity (Liu et al. 2001) Ernesto Tapia and Raul Rojas (GREC2003)

33 Directional Composite Similarity Principles for composite similarity metrics Partial (for partial match) Structural/topological Stroke-number free Stroke-order free Used in our demo system (SmartSketchpad)

34 Scenario for Composite Input for partial input match:

35 Document Recognition and Understanding: Problem Analyze the connections and relationship among the elements Obtain and represent the semantics in current (part/whole) drawing as one document Beautify and re-display it into a neat layout Comparison to offline document recognition similar to engineering drawings but more cursive for sketches more regular for engineering drawings

36 Document Recognition and Understanding: Applications Mainly for quick design 2D diagrams: GUI: Landy & Myers (2001), Caetano et al. (2002) UML diagrams: Blostein et al. (2002) 3D object input: Igarashi et al. (SIGGraph1999): TEDDY Lipson and Shpitalni (2002) Hsu and Lee (1994): 2.5D animations

37 Sketch Input for a Dog in Animation Fabian Di Fiore & Frank Van Reeth (2002) A Multi Level Sketching Tool for Pencil&Paper Animation

38 Document Recognition and Understanding Approaches Gross (1994, 1996): Sketch-A-Sketch detect & maintain spatial relationship (constraints) represented as binary predicates: e.g., concentric, contains, connects, overlap by the bounding box, size, & starting-ending points by-product: learn composite objects from examples Pinto-Albuquerque et al. (2000): DocSketch syntax as a fuzzy relational adjacency grammar visual syntax analyzer

39 Prototype Systems ASSIST (Alvarado and Davis IJCAI2001), MIT SketchIT: Stahovich (1996), MIT, CMU SILK: Landay & Myers (2001), UC Berkeley & CMU Teddy: Igarashi et al. (SIGGraph1999), U-Tokyo Tivoli: Pedersen et al. (CHI1993), Xerox PARC EsQUIsE: Pierre Leclercq (GREC2003) SmartSketchpad: Liu et al. (2001, 2002)

40 ASSIST A Shrewd Sketch Interpretation & Simulation Tool MIT AI Lab Christine Alvarado s Master Thesis (2000) Alvarado and Davis (IJCAI-2001) Each new stroke triggers three stage process: Recognition: generate all possible interpretations primitive stroke recognition & composite object (device) recognition Reasoning: score each interpretation Resolution: select the current best consistent interpretation Gesture recognition: arrows and pointing

41 SketchIT Conceptual Design for CAD (mechanical engineering) instead of precise design Thomas F. Stahovich (1996), MIT PhD Thesis Stahovich, Davis & Shrobe (AAAI-1997) QC-space for representing interaction among mechanical parts Calhoun, Stahovich, et al. (2002) semantic network based recognizer for multi-strokes Stahovich, Davis & Shrobe (AI-1998) generate multiple new designs from a sketch

42 Experiments of Primitive Shape Recognition

43 Database of Composite Objects In this experiment, we created 97 composite graphic objects. All these objects are composed of less than ten primitive shapes. The weights and thresholds we used areε =20, w 1 =0.4, w 2 =0.3, w 3 =0.3, k 1 =k 2 =0.5. We randomly selected 10 objects (whose ID is 73, 65, 54, 88, 22, 5, 12, 81, 18, and 76) and draw these objects as queries. In most cases, the intended object will appear in the smart toolbox (ranked in the first 10) after only a few components are drawn.

44 User Study and Performance Evaluation Object 73 Object Object 76 Object 5

45 Sketches for Evaluating Different UIs (a) sketch1 (b) sketch2

46 Drawing Time for Different Sketches Using Different UIs #User ID Drawing Time for Sketch 1 (s) Drawing Time for Sketch 2 (s) Sketch-based Traditional Sketch-based Traditional Average

47 Open Problems Complex editing gestures recognition and editing-related applications up to several hundred different gestures Composite object recognition for large object set: for graphics input up to 10,000 master objects in MS VISIO user and domain adaptation Semantic level understanding for creative/conceptual design reasoning & prediction of the user s intentions

48 Where to Find Papers? 2002 AAAI Spring Symposium Series--Sketch Understanding Chairs: Tom Stahovich, James Landay, Randy Davis Online Proc.: ACM Annual Conference on Human Factors in Computing Systems (SIGCHI) ACM Annual Symposiums on User Interface Software and Technology (UIST) GREC and ICDAR SIGGRAPH

49 Summary Brief survey of online graphics recognition Problems, Approaches, and Applications Supportive for pen-based UI improve user productivity convenient for creative tasks: quick design ideas users unanimously prefer the sketch-based UI

50 Thank You! Contact Liu Wenyin See some of my research work at The survey paper can be found at