PEN-BASED METHODS FOR RECOGNITION AND ANIMATION OF HANDWRITTEN PHYSICS SOLUTIONS

Transcription

1 PEN-BASED METHODS FOR RECOGNITION AND ANIMATION OF HANDWRITTEN PHYSICS SOLUTIONS Salman Cheema PhD Dissertation Interactive Systems and User Experience Research Cluster of Excellence Department of EECS University of Central Florida 21 nd October 2014

2 Outline Background Selected related work Initial Prototypes Current Prototype Informal Evaluation Conclusion Q&A

3 Fundamental Research Question Can the answer to a physics problem be used to animate the diagram, with a positive impact on student learning?

4 Problems to be Solved Methods to acquire student solutions Methods to parse and construct models from student solutions Methods for checking student solutions Generalized animation framework Techniques for measuring learning effect

5 Pen Interaction is Good for Tutoring Natural and Transparent Abowd 1999 Faster than typing (for math at least) Anthony et al 2005, 2007 May reduce cognitive load Sweller1994

6 Diagrams are also Important Design Process Ullman et al 1990, Gross & Do 1996, Tversky 1999, Kavakli & Gero 2001 Science Education Larkin & Simon 1987, Wai et al 2009, Ainsworth et al 2011 Spatial vs Sequential Representation [Larkin & Simon 1987] Spatial skills correlate with STEM proficiency Increased Engagement & Improved Learning Ainsworth 2011

7 Animation has Potential People already visualize systems in motion Shepard 1978, Clement 1994, Johnson-Laird 1998 Especially mechanical systems [Hegarty 2004] Pulleys for example Mental Animation can be piecemeal or global Good overview [Hegarty 2004]

8 Putting it all together A new class of intelligent tutoring systems is needed Pen Interaction for natural input Understand problem statements and student solutions Provide general-purpose animation support Provide feedback about correctness of student solutions

9 Challenges Pen interaction is easy Hardware support widely available Can also approximate on tablets Problem statements Analysis may not be difficult Body of NLP work Problem statements can yield descriptions of scenarios Initial conditions

10 Sample Problem Statement

11 Solution

12 Student solutions are complex Enormous variation in solution steps and diagrams Under- or over-defined diagrams Missing or extraneous information Diagrams are almost never precise Precise definition necessary for animation Text phrases or fragments Require semantic analysis

13 Selected Related Work

14 Relevant Research Areas Sketch Recognition Corner finding Stroke grouping Low level techniques High level techniques Pen-based Systems for Tutoring Statics, Logic, Set Theory, etc Animation and Traditional Tutoring Systems

15 Selected Pen-based Systems MathPad 2 (LaViola and Zeleznik, 2004) PhysicsBook (Cheema & LaViola, 2010, 2012) Mechanix (Valentine et al., 2012) Kara (2008) PenProof (Jiang et al., 2010) Newton s Pen I & II (Lee et al., 2007, 2012)

16 Related Work : Animation & Tutoring Andes Physics Tutoring System (Van Lehn et al., 2005) AutoTutor (Graesser et al ) Algebra Cognitive Tutor (Anderson et al 1995) Anthony et al (2012) Crayon Physics Newton s Playground

17 Existing tools are limited Specialized for specific tasks Often have no animation capabilities Do not use answer for animation if they do, it is not easy ( e.g. in Mathematical Sketching)

18 Recap of Initial Progress Three Proof of Concept Prototypes ( )

19 Contributions Multiple granularities of input Recognition of low-level diagram elements Annotations to provide context Used answer for animation Custom 2D physics engine for animation

20 Limitations A small subset of kinematics problems was supported Concepts related to f=ma Animation system was difficult to extend to other domains No understanding of problem statement Only used part of the solution No method to model solution

21 Current Prototype Contributions Architectural improvements Improved recognition pipeline Uses a range of simulators Tailored for specific types of physics problems Problem statement is used A method to model a solution as a deductive proof

22 Current Prototype Screenshot

23 Recognition

24 Ink Stroke Preprocessing Ink Stroke : Sequence of 2D points Cusp : Area of high curvature in an ink stroke Preprocess to remove noise Enumerate Cusps IStraw (Xiong and LaViola, 2010) Add to Math Recognizer (StarPad) Based on MathPaper (Zeleznik and LaViola, 2008)

25 Recognition Workflow (detailed)

26 Support for 9 Diagram Elements

27 Bottom-up Recognition Phase Unistroke and multistroke recognition heuristics Cluster annotations Tags Arrows Equations Initial (localized) Beautification Align vertical/horizontal edges Fix point ordering

28 Unistroke and Multistroke Recognition

29 Top-down Recognition Phase Identify problem domain via NLP List of possible simulators Process annotations and associated math domain-specific rules encoded in simulators Assign reasonable initial conditions Domain-specific beautification Localized and/or global On-demand Uses the QuickDraw framework

30 The QuickDraw Beautification Framework

31 QuickDraw overview Constraint-based Beautification Works with line segments and circles Assign canonical ordering (left-right, top-bottom) Infer a list of constraints from recognized diagram elements Vertical, horizontal, equal length, parallel, perpendicular, concentric, tangent, touch, collinear, etc Use novel beautification algorithm to generate precise diagram

32 Worked Example: Square Recognition 4 line segments Canonical ordering (left-right, top-bottom) Inferred Constraints 2 vertical and 2 horizontal lines Same length Vertical lines are parallel Horizontal lines are parallel Connected path Same perpendicular distance same between horizontal and vertical line segments

33 Worked Example (cont d) Based on ordering Compute the slope of left line segment Compute slopes of all other line segments Read an endpoint from the sketch Yields intercept Read length from sketch Beautify left line segment Beautify top line segment Beautify bottom line segment Beautify right line segment

34 Beautification Algorithm A = set of attributes of all elements B = Empty Set While ( A is not empty) If an attribute a i is computable using attributes in B Compute its value by using associated constraint else Select highest ranked a i from A Read its value from sketch B += {a i }, A-= {a i } Construct beautified elements from attributes in B

35 QuickDraw Demo

36 Diagrams tested with QuickDraw

37 Use in Current Prototype Intervals Pulleys Vertical/Horizontal segments/springs/wires/edges Touch constraints Wires/springs/pulley endpoints Shapes Touch constraints between shapes Example: shape resting on surface

38 Animation Runtime

39 Multiple Simulators Currently contains 4 simulators Freefall Kinematics Friction 1-D Elastic Collisions Equilibrium Hooks for custom behavior Lasso + Tap to associate own mathematics New simulators can easily extend system

40 Individual Simulator Design

41 Capabilities Simulator Elements Annotations Supported Problems Freefall Kinematics Circles, Polygons, Pulleys, Springs Arrows, Dotted Lines, Intervals Free Fall, Free-hanging Springs, Projectiles, Pulleys, Doodling Friction Circles, Polygons, Pulleys, Springs, Line Segments, Polylines Arrows, Dotted Lines, Intervals Sliding Contact, Inclined Planes, Kinetic and Static Friction Momentum Circles, Polygons, Pulleys, Springs, Line Segments Arrows, Dotted Lines 1-D Elastic Collisions Equilibrium Circles, Polygons, Wires Arrows, Dotted Lines Simple Equilibrium Problems, Tension Problems, Objects held with breakable wires

42 Solution Checking Proposed Method. Not fully implemented

43 Solution Checking Construct a solution graph and find a path through it General representation of a deductive argument or proof Singley 1990, Matsuda & VanLehn 2004 Leaf-nodes are givens or endpoints Links between vertices represent premises supporting that step In our case, Proof is mathematical Axioms are physics principles

44 Example

45 Analysis of Checking Algorithm Worst case O(n 2 ) time complexity Perfect Handwriting recognition is required Must know exact logical ordering of solution steps Do not support chained expressions Do not support text fragments Incomplete Implementation Need to integrate mathematics package for verifying correctness of arithmetic manipulations

46 Demo

47 Informal Evaluation

48 Informal Evaluation 4 participants (3M, 1F) Ages Task1: sketch spring system and view graphs Task2: solve simple kinematics problem

49 Results

50 Conclusion & Future Work

51 Problems to be Solved (initial slide) Methods to acquire student solutions Chose pen-based Interaction for natural interaction Methods to parse and construct models from student solutions Have made contributions with new recognition and beautification techniques Improvements needed to ensure stability Methods for checking student solutions Proposed graph-based method for this purpose Incomplete and needs testing Generalized animation framework Easily extensible animation framework Some interaction metaphors need to be devised to support full range of animation Techniques for measuring learning effect Major area of future work

52 Categories of Animation Initial conditions must always be known to begin animating Open-Ended (animate forever) Inspect/Reason/Solve for something that affects animation Unpredictable event happens, student reasons about event Time-of-Interest Inspect/Reason/Solve for something at different points in time Point-of-Interest Inspect/Reason/Solve for something at different points in trajectory

53 Supported Animation Capabilities Can infer motion Can define time limit (associate t=x with canvas) Can define distance limit (using intervals) Support limited unpredictable events 1-D collisions in momentum Rigid body collision in open-ended animations

54 Limitations in Animation Don t have a good way to define multiple points in time and/or trajectory Don t have a good way to indicate quantity of interest with each interesting point Don t have a method to glean implicit information from problem statement No support for piecemeal animation.

55 Summary Sketch-based physics tutoring systems have been developed before Broader goals than previous attempts Animation is primary focus

56 Summary (cont d) Four prototypes were constructed Investigated different methods for animation Current technique using multiple simulators holds most promise Contributions in sketch recognition and beautification Contributions in solution modeling Analysis of problems suggests new research directions interaction metaphors and animation behaviors

57 Acknowledgments Funding Sources NSF CAREER award IIS NSF Awards IIS and CCF

58 Publications Salman Cheema, Sumit Gulwani, and Joseph LaViola. Quickdraw: Improving drawing experience for geometric diagrams. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 12, pages , New York, NY, USA, ACM Salman Cheema and Joseph LaViola. Physicsbook: A sketch-based interface for animating physics diagrams. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces, IUI 12, pages 51 60, New York, NY, USA, ACM Salman Cheema and Joseph J. LaViola, Jr. Applying mathematical sketching to sketch-based physics tutoring software. In Proceedings of the 10th International Conference on Smart Graphics, SG 10, pages 13 24, Berlin, Heidelberg, Springer-Verlag Salman Cheema and Joseph J. LaViola, Jr. Towards intelligent motion inferencing in mathematical sketching. In Proceedings of the 15th International Conference on Intelligent User Interfaces, IUI 10, pages , New York, NY, USA, ACM

59 Questions