Boxes, Inkwells, Speech and Formulas DRAFT

Transcription

1 Boxes, Inkwells, Speech an Formulas DRAFT Richar J. Fateman March 0, 2006 Introuction This paper sets out some esigns for entering mathematical formulas into a computer system. An initial approach to this task suggests that the previous moel, namely writing mathematics on paper or chalkboar, shoul lea to a natural computer system using a stylus for writing on a tablet. For feeback an for presentation such as in this paper, we use the typesetting capabilities of Knuth s TEX system to show how properly typeset expressions might appear. We use TEX here to show our esign for an interactive input scheme, uner implementation. For this to work, an interactive system must make expressions appear approximately in the same sequences as illustrate, on a computer isplay. The soli color boxes that appear in the incomplete forms are intene as invitations for the user to continue writing out a formula, continuing from within one of those boxes. Think of them as virtual inkwells. For example, an attempt to write a superscript must begin by ipping the stylus (or mouse) in the superscript inkwell. An attempt to write an operan ajacent to an existing must begin in that inkwell. Initiating writing elsewhere on the screen will have no proper ink an will not contribute to the formula entry. We also point out that speaking the terms, rather than writing them, may provie more accurate communication. At this point we suggest you look ahea a page or two to see some pictures of inkwells.. Why Inkwells? This ink-well-base constraine input provies an obvious basis for cooperation between the human entering a formula an the supporting computer program. The program promises to try to parse the formula into something meaningful; the user must be able to express the intent of the writing using the provie paths that can be interprete successfully. Experience with other programs written without such constraints [8], but using a free-format input emonstrates substantial problems. The user (especially one who is unfamiliar with or uncomfortable using a stylus or mouse) will generally not be able to write glyphs of the proper size, an will not precisely position them. Other programs [7] repeately analyze the sequence of strokes to fin new an possibly better interpretations of the markings. This reinterpretation may convert correctlyrecognize symbols into incorrect ones. On the other han, since the input is unstructure, one can go back an insert extra parentheses or other markers as necessary. We make a simplifying assumption that the input will always procee roughly left-to-right, with occasional iversions up an own. The appearance of inkwells will force this irection. The strategy for the stylus once it has been inke, is critical. The user is not constraine to keep the stroke insie the inkwell. Once there is ink, a lower-case escener letter like p woul be expecte to rop below the inkwell.

2 The user oes not have to write small letters: once the stylus has ink, it can be use to write any size characters. (We coul also zoom in on the expression.) The treatment of secon an subsequent strokes can be unifie in two ways: If they are rawn close together in time, they are part of the same character. Thus a multiplestroke K or E woul have to be written so that the interval between the strokes is brief. A new (large) rawing pa can be isplaye so that anything within that pa is part of the same collection. This woul allow a user to write more extensive content for a box. Instea of writing ax + b in 4 separate steps, it coul be written in one. The Microsoft writing-pa application which can be ocke anywhere on the screen is attractive to serious users. The alternative write anywhere esign appears to be trickier to use, but this is a human-factors issue neeing stuy. As inicate below, speech input may be appropriate. If we can accurately recognize spoken eks square sign square of eks as x 2 sin 2 x then we have a potential for rapi input. Consier the symbol which is unicoe (hex) You might speak it as neither a subset of nor equal to [3], but reliable recognition of the hanwritten glyph, given the number of similar symbols, woul be pretty ifficult. The sizes an spacing of some objects in the formulas will have to aapt as the formula is written. These growing objects inclue ivie bars, parentheses an other brackets, matrix elimiters, integral signs, proucts, summations. The size of variables will epen not on how large a user writes them, but their positions relative to other symbols. The rules for TEX provie for careful evaluation of appropriate spacing an size, with the possibility of user-specifie exceptions. The context (isplay or in-line) as well as global irectives, can also affect size. A functional escription of TEX layout [4] may be more useful a guie as to how to re-implement this than the original escription or coe..2 Inkwells may be speech cues It is not require to enter material in these boxes by writing. Once a position is selecte, it may be preferable to speak the glyph. As an example, a vertical bar may be nearly the same as a slash mark / or the number, or a number of other glyphs that look similar. A vertical stroke may also be part of any number of multistroke symbols such as K or P. A horizontal line might, among many possibilities, be minus or ivies, or part of =. Speaking removes some ambiguity. It may also be faster as a consequence. Saying bol small tee to prouce t may be quicker an more accurate than trying to write a symbol which might easily be confuse with +. There seems to be no intuitive embolen written notation. (In the past, instructions given to an ol-fashione human typesetter tene to require unerlining with color pencils). A simple speech application that allows recognition of symbols spoken as (for example) Greek capital gamma seems to be quite accurate, at least for a small set of alternatives. Once one begins to use special symbols, the nee for isambiguation of apparently similar glyphs increases. A hanwriting recognizer aske to istinguish similar isolate characters woul not be reliable, an even if one coul be constructe to make such istinctions, it woul require an exceptionally careful human writer:, <, an, L an even α. Whereas left angle bracket, less than, an angle, to use the TEX names, are rather ifferent spoken utterances. There are thousans of spoken symbols that coul be taught to the computer system. Names nee not be unique; any of several common names might be use for the same glyph. The human writer woul not have to signify fonts by writing in Fraktur or Blackboar glyphs. A combination of speech an hanwriting may be most helpful. Our working hypothesis at this moment is that in a multimoal input metho base on ink-well constraints relies primarily on position information for which we can use a mouse. The hanwriting recognition can be own-playe if the symbol specification 2

3 buren is (mostly) carrie by speech. We woner how natural this might be to someone who comes to this task with a preconception that mathematics is normally hanwritten. The listing of MathML entities running from variations on the letter A through greater-than-or-equal, slante, ot insie to script letter z has 293 glyphs. This oes not however inclue all font variations or moifiers such as bol, unerline, etc. It oes not correspon irectly to TEX notation, but the latter is extensible through font-construction programs..3 Checking Display Preliminaries In orer for you to rea this paper, we must first be assure that the system that is being use to view or print it isplays the right size boxes in our TEX macros relative to the usual typeset material. The re rectangles below shoul approximate the sizes of the A an the x. (The same macro is use for each rectangle, but the expansion of the macro epens on the context within the TEX expression). A x X Here are some samples showing expressions an their transcriptions with pieces replace by rectangles. The geometrical arrangements shoul be similar. A y xb = X B A C = X x y + z A + B = A typeset example In orer to be concrete, we will use a running example for several sections. Here is one sample expression, isplaye as TEX typesets it: y 2 + z 2 + x x 2 Constrain by Leftmost Entry of Expressions The principal rule we nee to provie for our users is that whenever possible expressions shoul be written from left to right. This is the normal irection, but occasionally one makes mistakes. We allow the user to select an expression previously entere, an write over it perhaps with a much more elaborate expression, but not all previously written sub-expressions can be selecte (for example, in a sum of five items, select the mile three.) 2. Templates for Super/Subscript an Right Let us begin the entry of a particular expression, in pieces, using our first esign. The general expression template allows for a superscript an a subscript, an a position to its immeiate right. Only rarely are all three fille, so this may seem like overkill for proviing templates, an to the user both complicate an clumsy. y 2 + z 2 + x x 3

4 Enter the symbol x, then 3, 2 in any sequence. Here we enter 2 then 3. x x 2 x 3 2 In this esign both the 2 an the 3 might themselves have super or subscripts, accoring to this template. Again, most of the confusion evaporates at the next step where we enter the +. Then the system can abanon some of the extraneous boxes an retypeset: Let us skip 5 steps to reach: x x x If we introuce a part of a notation that is always compoun, like (balance) brackets or parentheses, we isplay the match immeiately. Parentheses can grow epening upon what is subsequently place within them. In the case of an integral, the of the x is immeiately inserte: x The next two steps will orinarily be contracte to one if the user is able to write 45 in one box. If the user pauses for a longer time (lets the ink ry ), the system has no recourse but to try to make sense of what has been receive so far. We return to this point in a later section: how much can be written in one box? x x Next, rawing a ivie bar makes a ifference. In our system, the user must learn to raw a ivie bar (same as a minus sign) prior to rawing the numerator or enominator. This is consistent with our view that the user must generally enter the Leftmost expression. (An alternative is to write the numerator an then eit it into a fraction) We continue: (this automatic placement of the x is problematical for x x, an alternative notation. Perhaps we nee integral: no x option as a menu selection, as well as other integral-like variations.) A template with a possible ivie-bar everywhere is far more cluttere than the sub/super/right template, an seems unacceptable. 4

5 We continue in this way to eventually reach: y 2 + z 2 + x x The only way we coul make this sequence even more cluttere might be to allow expressions to grow to the left, above, below, as well as pre-sub an pre-superscripts. 2.2 Clearing some clutter Here is another esign. We leave off the subscript template boxes an require that a writer who nees to prouce x 2 writes simply x an later goes back (e.g. click with right mouse button within the glyph bouning box) to choose from a menu: {elete, copy, eit, subscript, font, size}. Choosing subscript then opens up a subscript template. A selection metho (where exactly oes one point?), as well as appropriate menu items for compoun expressions requires careful esign. One metho we have use requires selecting α + β by clicking insie the box for the expression, but outsie the boxes for the three components. Another metho uses a rectangular lasso to contain some or all of the selection. The menu available for a selection epens on what is selecte: an integral sign woul have ifferent menu properties from symbolic atoms, or from the sum of two terms. It seems that a programming type structure woul be almost inevitable, at least for most sophisticate users. We coul eclare (within a elimite sub-scope of the formula entry program) that all use of specific symbols have special attributes. For example we coul eclare that A shoul always be set in bol as A, the sequence of two letters sn shoul be set in Roman as sn, an symbols otherwise not in the hanwriting lexicon (or ifficult to istinguish from other symbols) shoul be entere in istinct forms. That is, one might really want to write A B as A vee B, since the letter v (hanwritten) is not istinct from. One coul also specify temporary shorthans like D mapping to b 2 4ac, or long-hans like ot to represent the perio, or cots to represent three centere ots each of which woul otherwise be har to recognize in hanwritten form. (See section on speech for more iscussion on this ambiguity.) Returning to our constraint-box esign, if we eliminate the subscript template box by efault, our example above becomes simpler to view. Introucing subscripts then requires the user to go to an eit menu to make the x into an x 2 or use some kin of macro-expansion that makes the utterance xsub2 into a subscripte x.) Here s how it might look: Eit (using a metho not shown here) the x to make it x 2, either now or later. x x 2 x 3 2 At the point where we enter the +, the system has only one extraneous box, an then it has no extraneous boxes at all. x x x y x y 2 x y 2 + 5

6 x y 2 + z x y 2 + z 2 x x x x x y 2 + z 2 + x x. Comparison of the entire sequence makes it even clearer that removing the subscripts from most templates makes the overall construction more comprehensible. 2.3 Speech, again Remember our suggestion for speech input: the symbol above coul be written as vee but perhaps more easily spoken as logical or. We have just begun to experiment on the hypothesis that speech input generally to the computer coul actually remove the nee for some of these macro commans an also make the symbol recognition far more accurate. The point is particularly clear when the hanwritten glyphs are problematical. Speaking ot may be a key to recognizing what might otherwise be a (small) piece of essentially ranom ink. 2.4 Other improvements Another possibility is to use color coing to keep track of what is likely to be the main course (most likely continuation) of the expression. Thus we coul illustrate that we expect that the user to use (one of) the blue boxes next. We coul even avance the point of insertion to the blue box so that a continuation of speaking or typing woul procee automatically to the next blue box. Or voice commans coul navigate among the boxes. x x 3 x 3 4 x

7 x 34 + x sin x In this example, speaking eks up three four own plus eks sign eks woul be acceptable. There are a host of features that may make sense, especially for serious users, beyon simple hanwriting an speech. Here are some options: We coul ask users up front: Do you expect to enter any subscripte variables? If so, what are their names? Do they have subscripts or other marks? (This context an other similar information below coul be save, restore, eite, etc.) We coul ask about unusual notations, especially those that might be easily confuse in hanwriting like <> versus rangle? How woul you prefer to notate these? How o you plan to speak their names? Are they constraine as in the typical bracketing of parentheses? We shoul provie a metho for aing new templates, such as a template that starts out with equation numbers on the left ( ). Another template that might be use in tables of integrals, woul have spaces for sie-conitions restricting the omain of some of the extra parameters, cross reference to relate formulas, an even a reference to the literature. Once an expression is entere (or compute), there is a potential nee for alterations. The esign for such eiting is not covere in this ocument. We shoul explore this further [2]. It shoul be possible to scroll a large expression, zoom in or out of expressions for ease in writing or reaing, an it shoul be possible to fol expressions over more than one line. This last issue is consierably more ifficult to aress aequately, an may be best ignore for lack of a goo automate solution. A goo multi-line expression coul be entere on several lines or in a matrix by a skille typesetter. We have written as though a ocument consists entirely of a single formula. This is, of course, unlikely 2. A useful entry system shoul allow intersperse orinary text both in-line an in complete paragraphs surrouning isplaye mathematics. More specialize combinations of text an formulas as in theorems or proofs shoul be accomoate. 2.5 How much can be written in one box? It is clear that the tiny boxes are insufficient space to write characters; that is why we refer to them as inkwell. But it woul be convenient to insert many characters in a box: in fact, as many as can be conveniently written or spoken. Thus e sin ax woul most plausibly be written in only two boxes, at least if the whole exponent can be written before the ink ries. One technique use in Microsoft Office s hanwriting interface is to provie a separate writing pa, as oppose to the write anywhere moe. (The major recognition avantage is that the user/computer have a baseline on which to write/recognize. The major isavantage is that you probably want to use the stylus as a pseuo-mouse, too. Dual use is confusing.) In our case a writing pa coul be implemente by a local pop-up winow pa which allows more written material. How large shoul this pa be? (Shoul it grow?) How long shoul it last before the ink ries? Shoul it always appear? 2 Except in spy novels where the formula is secret. 7

8 These are elicate issues but resolving them may not be important. In our experience, a hanwritten version of a + b sin θ is har to recognize regarless of where it is written. A result of at b sino is typical. Even the untraine speech program in Microsoft Office comes out pretty close: a+be sign ata. A more restricte speech grammar we are eveloping (Fall, 2003) to emonstrate applications of the Speech SDK seems to work rather more accurately, given a subset of wors or phrases typical of mathematics, incluing stanar alphabetic characters. 3 More examples It is tempting to provie special treatment for known functions. That is, we can assert that uttering the symbol sin immeiately prouces a template which provies for three new boxes: the power of the sin, its argument, an the (optional) continuation. That is one coul create sin 2 (xy) by starting with this template: sin ( ). This template suggests (a) the parentheses are not optional. (b) The next atum to be entere shoul be the argument of the sin. (c) The power an the continuation to the right are optional or seconary after the argument is fille in. The ownsie to this is that it becomes impossible to typeset sin x without parentheses. Whether this is an issue or not is a matter of taste, about which it is ifficult to legislate. A complete implementation shoul provie isplay of glyphs that grow, such as brackets an matrices. See how TeX hanles them: ( ) (a + b) c + + sin (x + y) e /(+x) We can aress the nee of someone who writes a + bc + an subsequently ecies to enclose this expression in parentheses as (a+b)c+ by eiting operations, since it woul otherwise violate our proceural guielines of starting from the left an proceeing to the right. One technique is essentially to use another piece of canvas to write a ( ), then select the insie a + b out of an existing form, an past it into the. The complete scenario is thus more elaborate than just overwriting an expression with some parentheses. 4 Comparison with Other Programs The lengthy A Survey of User Interfaces for Computer Algebra Systems [6] provies a broa perspective on the topic prior to about 998. The historical highpoints inclue, in our view (R. Anerson s early thesis, W. Martin s program, MS equation eitor, recent tour-e-force programs such as Milo/PacificTech, Expressionist/Theorist/Livemath, an Soiffer s work in Mathematica). Recent progress in hanwriting or speech input is not inclue. 4. Keyboar input It is har for us to be critical of keyboar input of mathematics: we have been using this metho for many years, an there are several programs that provie excellent support for the higher levels of expertise. We first point out some, perhaps obvious, points against the keyboar for math input. Math is inherently locally two-imensional as x 2 an therefore is a mismatch to the unaorne keyboar, which provies a locally linear string. By this we mean that the two-imensional page layout of text is one by a global operation of inserting new line characters. Given that a pen or a mouse is use for geometric positioning, forcing the user to switch to the keyboar to provie the content is inconvenient. 8

9 In favor of keyboars, we have these consierations: Hanwriting (or speech) is not always accurate an efficient. If errors cannot be hanle in the same moe, the user may be force to learn about keyboar entry in orer to use it for corrections. Once the keyboar is introuce either for content or correction, the esign can easily go a step further an use it (via arrow keys or other coes) as a replacement for the mouse/stylus selection. This is a possibility: using the tab key to circulate sequentially among possible selectable boxes is a convention with filling in HTML forms. One program (Mathematica) uses control+space to rop out of a construction; TeXmacs uses the rarrow key similarly. With such a convention an a few clues as to how to navigate aroun a matrix or a, a skille typist can o very well in positioning. Making use of macro efinitions for common subexpressions or frameworks, a common tool for the expert in TEX can provie even more efficiency. We mention a few examples of keyboar input (although the [6] provies much more). TeXmacs is a free open-source eiting system [5], inspire by the emacs eitor an TEX. It is not implemente using either one, however. TeXmacs (once it is successfully installe) allows you to interactively type (an see as immeiate feeback) almost anything in TEX, as well as various extensions to images or other objects. TeXmacs allows alternative methos. For example you can choose from a menu a fraction template or alternatively type \frac in place or the shortcut em alt-f, to then be shown an in-place template for numerator an enominator. The non-menu approach is preferable since you can keep your fingers on the keyboar. (TeXmacs has keyboar shortcuts, but this coul presumably be ae.) To change from text to math an enter a fraction a b you type 8 characters: $ alt-f return a own-arrow b right-arrow $. TeXmacs uses font erive from the excellent TEX repertoire. Occasionally (most often when first starting up with a new installation) it pauses to compute new font sizes as they are neee for the first time. TeXmacs is a highly competent text typesetting system, but as a math system it is esigne to interface with other systems, an oes quite well orinarily. There are issues with conflicting system constraints on interfaces, hanling errors, etc. One of the more popular computer algebra systems, Mathematica [9], provies for keyboar/mouse input. The stanar input is a linear keyboar input format, but recent versions of the system have ae 2-D input formatting which can be accesse by menu or by keyboar escapes. The running WYSIWYG input/output is not as elegant in appearance as the TeXmacs system, but is similar in that it refers back to a similar repository of mathematical symbols from TEX. Mathematica generally assumes you are in math moe so to enter a fraction a b you can type a control-/ b control-space. Comman completion is available in either systems so you can avoi typing some long names. In each of these systems you can move a cursor to the left into expressions. TeXmacs provies both visual feeback (boxes) as well as a brief written escription in the lower right corner of your location in an expression (e.g. hints of the containing operators of the selection point). The Mathematica program places some emans on the input expressions that are not entirely commonplace, such as enclosing function arguments in square brackets, an using a special for the ifferential marker in integrals. (It is possible to type material in Traitional form with conventional appearance: this can sometimes be properly interprete an eite, but frankly traitional form is ambiguous: a(r + s) means one thing if a is a function, but if a is a constant it inicates invisible multiplication). Expressions entere into computer algebra systems like Mathematica ten to get simplifie an rearrange, sometimes to a isconcertingly ifferent form, an sometimes to great benefit. The TeXmacs program is generally less concerne about the meaning of expressions than Mathematica because TeXmacs oes not have to fit every utterance into a particular computer algebra moel of representation an evaluation. This can be use to avantage when especially when efining new notations. Certainly consiering the price, TeXmacs eserves careful scrutiny as a mathematics entry program. For Winows users, TeXmacs currently requires an installation of Cygwin, an in my experience takes some filing to get to work. It can export files as TEX although that is not the native form for TeXmacs. 9

10 We shoul also mention the equation-entry keyboar system that probably requires minimal filing to install (for Microsoft Wor/Office users): the Microsoft equation eitor an its upgrae to MathType ( from Design Science). This program is capable of goo typeset results, although it too seems to be ifficult to use without interspersing significant mouse activity between keystrokes. The esign is not any more keyboar-hostile that others: many of the menu choices can be reache by combination keystrokes or reache by navigating menus by arrows. Every program requires some effort for o symbols; the palettes may be extensible, an as unicoe becomes more easily available, the symbol set can presumably have thousans of glyphs in ifference fonts an sizes. Conclusion: We can t count out the keyboar as an important input evice, especially for people who are using avance notations an can type accurately. Furthermore, it is likely that unusual notations an glyphs will be introuce first via keyboar input. By the same token, the steep initial learning curve is a barrier for naive users. 4.2 Other Math+Speech Efforts What appears to be a primitive version of speaking math is emboie in a program Mathtalk using a version of the software Dragon Naturally Speaking. The emonstrations are not encouraging, but the fact that it works with MacKichan Scientific Notebook is promising. (A iscouraging aspect is the suggestion in the emos that a military-style alphabet is use: thus x might be spoken as elta x-ray. It is not clear if this is essential. In our own experiments, single-letter recognition is somewhat unreliable with the letters {b,, t, p} sometimes confuse. Recognition of phrases like y by x is far more reliable.) 5 Conclusions an Further Work We have suggeste a novel hanwritten technique which isplays all possible continuations of a formula, writing it left to right. The user is constraine to raw ink from these continuation spots. This feeback promotes the relatively painless construction of well-forme formulas. Seconly, we have suggeste that, as a minimal effort to take avantage of multimoal input, iniviual symbols or short phrases inserte by speech input can be synchronize with hanwriting or pointing. It may in fact be easier to replace the entering of most hanwritten symbols by speaking them. Other uses of speech might inclue using voice commans to eit a symbol or make an alternative recognition choice, or to give higher-level commans like begin bolface. We complete a emonstratin prototype; actually two.) The first by Kevin Lin (SKEME) emonstrates the boxing an isplay, an is written in Lisp. The stubs for speech an hanwriting were not fully evelope. A system calle Math Speak an Write, with a somewhat less evolve isplay, but one linke to both speech an hanwriting was written by three stuents, is of sufficient robustness that we offer a ownloa of it, Continue improvement of this system leas us to go back to basics an fin a way to recognize math symbols without extensive training on a per-user basis, an also voice recognition, with error correction, using a context-free grammar that escribes math. We hope to be able to test our esign against other systems [, 8, 7, 5] for use in two areas: proucing ocuments, an as an alternative (to typing) as a moe of input for computer algebra systems. Finally, we feel it appropriate to point out that even recognition of forms syntactically, an reucing them to an apparently unambiguous linear form oes not imply that the recognition program has a complete semantic moel of the utterance. The meaning behin a notation is necessarily context epenent. One of my favorite examples occurs in the use of x in a publishe integral table entry for the formula (a+bx)/(c+ x)x. 0

11 6 Acknowlegments This research was supporte in part by NSF grant CCR aministere through the Electronics Research Laboratory, University of California, Berkeley. Stuent assistents were supporte in part by an Unergrauate Research Opportunities grant from the UC College of Engineering an the Haas Scholars fun. We are aware of the irony that a Berkeley project shoul avocate constraine speech input, as oppose to free speech. References [] J. Arvo, FFES: Freehan Formula Entry System. [2] S. Dooley. Eiting mathematical content an presentation markup in interactive mathematical ocuments, Proc. ISSAC 2002, (Lille, France) [3] R. Fateman. How can we speak math? [4] R. Heckman an Reinhar Wilhelm. A Functional Description of TEX s Formula Layout, J. Functional Programming 7(5), (997) heckmann/oc.html [5] Joris van er Hoeven. TeXmacs ( [6] Norbert Kajler, Neil Soiffer. A Survey of User Interfaces for Computer Algebra Systems. J. Symbolic Computing 25(2): (998) [7] N. E. Matsakis. Recognition of Hanwritten Mathematical Expressions, MS report, MIT viola/research/publications/matsakis-ms-99.pf (999) also emo at [8] M. Suzuki, Infty Project. suzuki/inex-e.html (inclues links to ownloa of emo) [9] S. Wolfram et. al. Mathematica,