Dynamic Aural Browsing of MathML Documents via VoiceXML

Transcription

1 Dynamic Aural Browsing of MathML Documents via VoiceXML Hemambar Reddy, Gopal Gupta Department of Computer Science University of Texas at Dallas Richardson, TX 75025, USA Arthur I. Karshmer Department of Information Technology University of South Florida Lakeland, FL 33810, USA Abstract Understanding and internalizing complex mathematical expressions is a difficult task for visually impaired individuals. For visually impaired individuals, understanding mathematical expressions entails being able to browse the expression, i.e., navigating the expression and hearing the whole expression or its sub-parts repeatedly. Designing aural browsers that allow such navigation is a difficult task. In this paper we present a VoiceXML-based approach for aurally navigating mathematical expressions coded in MathML. In our approach a MathML document is translated into a VoiceXML document via the XSLT transformation language. The VoiceXML document is further augmented so that users can attach speech labels (termed voice anchors by us) at various points in the document while browsing. These speech labels or voice anchors can then be used by the listener to dynamically navigate the expression during the browsing session. A prototype system based on these ideas has been implemented and tested. 1 Introduction We consider the problem of communicating mathematical expressions to blind individuals. Communicating the structure of large mathematical expressions to blind individuals is a difficult problem that has received attention recently [6, 7, 11, 8]. A linear reading of large expressions can result in the reader getting lost in the details. It is considerably easier to understand a formula visually because visual representations convey the structure of a formula. Blind readers do not have access to the two dimensional display of mathematics that the sighted users have that makes the visualization of mathematical expression easier. The linear encoding of a complex expression in MathML, Latex, or the Nemeth or Marburg code may be so long that by the time a user gets to the later parts of the expression, they might have forgotten the earlier part. In this paper we consider the design of a browser for mathematical expressions that leverages the VoiceXML technology. We assume that the input documents to be browsed are coded in MathML. MathML is an XML based mark-up for tagging mathematics documents and has gained wide acceptance. Similarly, VoiceXML [2] is an XML based markup language designed for creating audio/voice based documents. The main goal of VoiceXML is to bring the advantages of Web-based development and content delivery to interactive voice response applications [2]. Just as a web browser renders HTML documents visually, a VoiceXML

2 interpreter renders VoiceXML documents aurally. The VoiceXML technology has been widely adopted in the telecom industry for automated handling of calls. In our approach mathematical expressions coded in MathML are mechanically translated to VoiceXML via XSLT [14], a transformation language for XMLs. The VoiceXML document can then be heard with a voice browser by a visually impaired individual. However, merely hearing the VoiceXML document is not enough, since VoiceXML provides very little navigational control to a user while listening to a document. In fact, navigational control in VoiceXML, at present, is completely controlled by the author of the VoiceXML document [12]. To make navigation more interactive and under listener's control, we further enhance the VoiceXML document with voice anchors. VoiceXML documents enhanced with voice anchors allow listeners to place speech labels on sub-expressions of a mathematical expression. These speech labels can be used by listeners to move around the various sub-expressions via simple voice utterances and voice commands. Thus, the mathematical formula can be browsed much more effectively. Note that in our approach, in contrast to other browsers for Math formulas (such as Math Genie [6]), the listener interacts with the documents and aurally browses math expressions through voice input rather than through keyboard input. 2 Translating MathML to VoiceXML Print mathematics uses a complex system of two-dimensional symbolic notations. The challenge in putting mathematics on the Web is to represent notation and content in such a way that it can be unambiguously understood by software programs that search, index, display and manipulate mathematics. MathML has been designed with this aforementioned goal of representing, encoding and communicating mathematical insights to the users. Thus, MathML is an important step in publishing mathematics on the Web. However, mathematics on the web is generally inaccessible through aural means (e.g., for blind individuals). Our work endeavors to make MathML accessible aurally via VoiceXML using XSLT to perform the translation. MathML elements can be categorized into three types: Presentation elements, Content elements and Interface elements. Presentation elements (i.e., notational elements such as a row, a superscript, subscript, etc.) are used to determine placement of various entities during display. Content markup elements correspond to a wide variety of operators, relations and named functions, and capture the meaning of math expressions. Both presentation and content markup represent math using nested structures, and thus can be recursively decomposed and processed. Consider a very simple math expression: Its MathML encoding is shown below:

3 <math> -- Root Node for MathML Document <mrow> -- Horizontally aligned elements <mrow> <mrow> <mo> </mo> -- For Marking operators <mrow> <msup> -- For Superscripts in Expressions <mi>z</mi> -- For Displaying the Variables <mi>n</mi> </msup> <mo> </mo> <mi>erf</mi> <mo> </mo> <mo>(</mo> <mrow> <mi>a</mi> <mo> </mo> <mi>z</mi> </mrow> <mo>)</mo> </mrow> </mrow> </mrow> </mrow> </math> Since MathML documents are inaccessible to blind individuals, we wish to translate them to VoiceXML so that they can be aurally browsed to help understand the math expressions contained in them. We accomplish this via a translator written in XSLT. Note that the XSLT translator we have developed converts the presentation form of MathML to VoiceXML. 3 Enhancing VoiceXML for Dynamic Control VoiceXML [2] is a mark-up language designed to make Internet content and information accessible via voice and audio. With VoiceXML, not only information becomes more accessible to visually impaired individuals, it also becomes accessible through other media (such as cellphones) and at times when visual medium cannot be used (such as while driving a car). Considerable work has been done in this direction, which includes the design of the VoiceXML [2] language and voice browsers to display VoiceXML documents. VoiceXML is the HTML of the voice-based Web, the open standard mark-up language for voice applications. Numerous commercial vendors such as IBM [4], TellMe [3] and BeVocal [1] provide voice browsers that can be used to play VoiceXML documents. These browsers can be accessed through a microphone and speakers of a computer (e.g., IBM Websphere), or through the phone (e.g., Tellme Studio). The association of these browsers to VoiceXML can be compared to the association of HTML to popular text browsers such as Netscape and Internet Explorer. Just as a user can interact with a HTML page, he/she can also interact with a VoiceXML page. The notable features of VoiceXML are:

4 1. Recognition of spoken/dtmf (Dual Tone Multiple Frequency) input. DTMF refers to pressing telephone keys. 2. Assigning spoken input to variables in the document and making decisions based on the assigned values to the variables. 3. Playing synthesized speech, audio files with the help of Text-to-Speech (TTS) converter. 4. Hyper-linking to other documents/other areas of the same document as in HTML. 3.1 Limitations of VoiceXML A VoiceXML document consists of a series of forms, each of which is aurally played one after the other. The document writer can specify the order in which the various forms should be played. In addition, the VoiceXML document can be written in such a way that user's input (through voice) can be taken to determine which way the control should flow. VoiceXML restricts the (voice) inputs from the listener to a set of predefined phrases called the grammar. This grammar is usually a simple list of words specified in the VoiceXML document. To provide more sophisticated navigation, the developer of the VoiceXML may prompt the user (via audio menus) for the name of the form they wish to revisit when the document ends. The author can express more complex navigation scenarios; however, the flow of control is always determined by the document author. If the document writer fails to anticipate an interaction scenario and leaves it unspecified in the document, the listener is deprived of the particular interaction. Hence, the onus of foreseeing all possible interaction scenarios falls on the author of the document. This is clearly a limitation, as listeners/users have no control over navigation, other than what the document writer allows them. For example, if the writer decides that the user must listen to a long dialog, the user has no choice but to listen. Likewise, say, while browsing a chemistryrelated document, if the document author decides that the information about the chemical chlorine must be played after the information about the chemical ether, then the listener cannot instruct the browser to play them in the opposite order, or to skip one of them, etc. Another limitation of VoiceXML is that the grammar (really a list of words) in a VoiceXML document specifies all the words that the listener may possibly speak while giving the input. Any word spoken outside of this grammar will not be recognized. This is because current voice recognition technology cannot recognize an arbitrary utterance from an arbitrary user. 3.2 Dynamic Navigation with Voice Anchors The focus of our work is to provide greater control to the listeners, so that they can navigate through VoiceXML documents with considerable degree of freedom. Our approach follows two basic guidelines: (i) the user should have the ability to interrupt or interact at any given time during aural browsing; (ii) the user should have means to hear exactly what they want and skip what they don't want. To allow the user greater control of the navigation, we introduce the concept of dynamic voice anchors. Voice anchors permit listeners to have complete control of navigation during aural browsing, thus allowing complex interactions to take place through audio/voice. A very simple way to provide the listener of a document more control is to permit them to utter certain pre-defined words (with pre-defined semantics) to move back and forth in the document.

5 For example, uttering skip should move the control to the next dialog, while uttering back should move it to the previous one, etc. However, full-fledged dynamic navigation of VoiceXML necessitates that the listener be granted freedom to label a portion of the document with an audio identifier of their choice and later (during the same browsing session) return to that labelled portion by uttering the identifier. These identifiers, or voice anchors, are similar to HTML anchors, except that HTML anchors are placed by the document's author and not by the user. Currently, VoiceXML does not support such dynamic voice anchors. Allowing users to define voice anchor will require supporting grammars in VoiceXML documents that are defined at runtime, i.e., at the time the document is served. It will also require modifications to VoiceXML browsers to have them support such dynamic grammars. As mentioned earlier, current voice recognition technology cannot recognize an arbitrary utterance spoken by an arbitrary user and convert it to text. Current technology can only recognize arbitrary utterances, as long as the set of such utterances is fixed in advance, as is done in current VoiceXML browsers. Thus, arbitrary voice anchors cannot be supported; however, we outline a number of reasonable solutions to support voice anchors, given this limitation [12]. The most successful solution is based on requiring listeners to spell the voice anchor name. Since each utterance is a letter, it can be easily recognized. The letters can then be assembled into an anchor name, and dynamically added to the grammar while browsing. So, later during the current browsing session when the anchor name is spoken (as a full word) it will be recognized and the voice browser will switch to the section of the document where the anchor was placed. The most appealing aspect of this approach is that any word can be chosen for an anchor name. Ideally, the voice browser should be modified to make these approaches for placing anchors work, however, by dynamically generating new VoiceXML documents through CGI programs called in VoiceXML pages we are able to implement our solutions without having to modify a voice browser. A description of these implementation techniques for supporting listener controlled dynamic navigation can be found elsewhere [12]. Once anchors have been placed in the document, more sophisticated navigation strategies can be specified. One can go a step further and design voice-commanded scripting languages that permit users to verbally program sophisticated navigation strategies [16]. The VoiceXML documents are enhanced on-the-fly so that they can accept dynamic voice anchors enabling listeners to have more control while browsing. Enhancements to the VoiceXML document allow listeners to give voice commands to the voice browser. The user can then skip to the next dialog in the VoiceXML document, repeat the current dialog again, go to the beginning/end of the VoiceXML document, as well as dynamically place anchors at various points in the document and then access those points later in an interactive fashion. In the context of browsing mathematics, if a mathematical expression is coded in VoiceXML, then listeners can associate voice anchors with its various sub-expressions while listening. The sub-expressions can be revisited merely by uttering the anchor corresponding to the subexpression that the user desires to re-visit.

6 4 XSLT for Translating MathML to VoiceXML EXtensible Stylesheet Language Transformation is a language for transforming documents in one XML to another. Transformations expressed in XSLT describe rules for transforming a source tree into a target tree, where the trees correspond to XML descriptions. The transformation is expressed as a set of template rules. Each template rule associates a pattern which matches nodes in the source tree with a sequencing operator which can be evaluated to produce part of a result tree. Thus, the template rules define the criteria for selecting elements from the source tree and provide instructions for transforming the selected content to create the result tree. Each template rule constructs a result tree fragment of markup and content. An XSLT transformation's result tree is created when all the result tree fragments are combined in the order specified to form a completed whole. XSLT can be thought of as a high level formal language for describing tree transformations that restructure content to create a new format. XSLT relies on a parser to convert the XML document into a tree structure, which XSLT then manipulates. We used the Saxon system, an XSLT processor which internally uses the SAX parser [15]. We have developed XSLT template rules for transforming MathML to VoiceXML. Visualizing a complex mathematical expression generally requires that the listener browse through it several times. However, using current tools (including current screen readers), the only option (s)he has is to hear the entire expression repeated again and again. Instead a better approach would be for the listener to interactively listen to the various sub-expressions repeatedly. The XSLT transformer developed by us converts MathML documents to VoiceXML documents. Appropriate description text is added so that when the VoiceXML document is aurally browsed then, the math expression can be completely understood. Moreover, the transformer divides complex expressions into a number of forms in VoiceXML to facilitate the anchor-based dynamic navigation discussed in the previous section. Consider the mathematical expression below. A fragment of the XSLT code that translates the above equation is shown below. The code below transforms the math expression above to a VoiceXML document which is broken up into five forms, each corresponding to a sub-expression and named with an integer. A fragment of the VoiceXML document generated is shown in the next page. <?xml version = "1.0"?> <xsl:stylesheet version ="1.0" xmlns:xsl=" xmlns:mml=" <xsl:output method = "xml" omit-xml-declaration="yes" /> <xsl:template match ="mml:math"> <vxml version="1.0">

7 <form id="init"> <field name = "input"> <xsl:apply-templates/> </field> </form> <form id ="end"> <block> Thanks for using the system </block> </form> </vxml> </xsl:template> <xsl:template match="mml:mo"> <xsl:choose> <xsl:when test="text() = '(' or text() = ')' "> <xsl:variable name="mowithparen" select = "count (preceding::mml:mo[text() = '(' or text() = ')'])"/> <xsl:choose> <xsl:when test="text() = ')' and not(following::mml:mo[text() = ')'])"> <noinput> <goto next = "end"/> </noinput> </xsl:when> <xsl:otherwise> <noinput> <goto next = "{$mowithparen}"/> </noinput> </xsl:otherwise> </xsl:when> </xsl:choose> <xsl:otherwise> <prompt><xsl:apply-templates/></prompt> </xsl:otherwise> </xsl:choose> </xsl:template> </xsl:stylesheet> XSLT document needs to include an XML version declaration (just like any other XML document). Namespace-prefixed elements enable an XSLT processor to distinguish them from other kind of markup in the style sheet. It is declared within stylesheet's root element, which is either xsl:stylesheet or xsl:transform. The version attribute needs to be included between the root element and namespace-prefix declaration. The xsl:template element is used to encapsulate template rules. The value of its match attribute contains an XML Path written as an XPath (a language for addressing parts of

8 an XML document) statement that is used as a pattern to select source elements for processing. Whenever mml:mo is matched by the template, it checks to see if '(' or ')' occurs in the expression and correspondingly groups each such delimited sub-expression in a distinct form. Each form is given a unique id (an integer). The apply-templates element signals to an XSLT processor to look for other template rules and apply them at that point in the style sheet's structure. The result of each rule combines to form the result tree of the completed transformation. The resulting VoiceXML document is output by the XSLT processor which takes as input the MathML document and the XSLT document. Part of the VoiceXML output produced by the XSLT program for the math expression shown earlier is given below. <?xml version = "1.0"?> <vxml version = "2.0"> <form id = "1"> <field name = "input"> <prompt> integral of</prompt> <prompt> z</prompt> <prompt> to the power of</prompt> <prompt> n</prompt> <prompt> times</prompt> <prompt> e</prompt> <prompt> to the power of</prompt> <prompt> b</prompt> <prompt> times</prompt> <prompt> z</prompt> <noinput> <goto next ="#2" /> <noinput> </field> </form> <form id = "5"> <field name = "input"> <prompt> times</prompt> <prompt> erf</prompt> <prompt> function</prompt> <prompt> (</prompt> <prompt> a</prompt> <prompt> times</prompt> <prompt> z</prompt> <prompt> )</prompt> <noinput> <goto next ="#end" /> </noinput> </field> </form> <form id = "end"> <block> Thanks for using the system </block> </form> </vxml>

9 5 Implementation All the components of a system that allows users to interactively browse mathematical expressions using VoiceXML have been implemented and integrated [13]. The MathML to VoiceXML transformation module is written in XSLT, as described earlier, and can handle almost entire MathML. The mathematical expression in the VoiceXML document is broken up into several forms so as to facilitate its dynamic navigation. The VoiceXML document is then further enhanced by another module we have developed [12] that provides users the ability to dynamically attach voice anchors at any arbitrary place in the VoiceXML document during aural browsing. The enhanced document can then be played on a voice browser and interactively navigated in a dynamic fashion. Navigation is accomplished via voice commands. Users can repeat sub-expressions, skip sub-expressions, jump to the beginning or end of the expression, as well as go backward to the previous sub-expressions. In addition, users can attach voice anchors to any sub-expression during browsing, and later if they wish to revisit that sub-expression (in the same browsing session), they merely have to utter the anchor name and the browser will jump to the sub-expression where the anchor was placed. If multiple sub-expressions have been given the same voice anchor name, then all the sub-expressions with that anchor will be linearly read out (cumulative anchor [12,13]). Our prototype implementation is quite efficient, and can handle almost entire print mathematics. Expressions that are as long as 3 pages in print have been successfully translated and interactively browsed. 6 Related Work Aural browsing of mathematics has received a lot of attention recently. Gillan et al have performed a number of experiments with blind and blind-folded subjects to understand how humans understand complex expressions [5]. Raman has developed the AsTeR system that can be used to hear math expressions coded in Latex [11]; however, all input is through keyboard, and aural browsing is not as sophisticated. The most notable system that has been developed for aural browsing of mathematics is the Math Genie [6]. Math Genie permits aural browsing of math expressions. The Math Genie is a custombuilt, stand-alone tool that is based on a very general architecture [7]. It takes a MathML file as input and allows a blind user to aurally navigate the parse tree of the math expression. The output of the Math Genie is in audio, however, users have to issue commands via the key board (e.g., using the arrow keys) to navigate the expression tree. Parts of the expressions can be abstracted away in Math Genie, allowing listeners to hear skeleton of the expression first; listeners can progressively ask for more and more details. The biggest difference between the Math Genie and our work is that we leverage existing off-the-shelf technology (XSLT, VoiceXML) to develop an aural browser for math expressions. Also, all commands for interaction purposes can be given through voice in our system. The Math Genie does not have the ability to label an expression with anchors, unlike our system. However, our system does not have the ability to abstract away sub-expressions. Thus, the Math Genie tool and our system allow different types of aural browsing facilities. The two can be regarded as complementary.

10 7 Conclusion and Future Work In this paper we presented a system for aurally browsing math expressions coded in MathML. Our system converts MathML to VoiceXML with the help of an XSLT transformer. It then augments the VoiceXML document with machinery that allows listeners to dynamically place voice anchors. The voice anchors can then be used to dynamically navigate the math expression coded in VoiceXML. The system has been developed and used to aurally browse very complex expressions. Future work includes building the ability to abstract away sub-expressions (as in Math Genie) as well as to output the audio in accordance to the Nemeth Mathspeak convention [9]. Future work also includes designing verbal scripting languages for orally programming navigational strategies for browsing mathematical documents [10]. References [1] Bevocal, Inc. [2] VoiceXML specification. [3] Tell Me Studio Voice Solutions [4] IBM Websphere. ttp://www-3.ibm.com/software/webservers/ [5] D. Gillan, P. Barraza, A. Karshmer, S. Pazuchanics. Cognitive Analysis of Equation Reading: Application to the Development of the Math Genie. ICCHP pp [6] A. Karshmer, C. Bledsoe, P. B. Stanley. The Architecture of a Comprehensive Equation Browser for the Print Impaired. ICCHP pp [7] A. Karshmer, G. Gupta, D. Gillan. Architecting an Auditory Browser for Navigating Mathematical Expressions. ICCHP pp [8] A. Karshmer, G. Gupta, et al. Reading and Writing Mathematics: The MAVIS Project. In Behavior and Information Technology (1):2-10. [9] A. Nemeth. Mathspeak. [10] M. Nichols, G. Gupta. Voice-commanded Scripting Language for Programming Navigation Strategies on-the-fly. These Proceedings. [11] T.V. Raman. Audio System for Technical Readings. Ph.D. Dissertation, Cornell University, [12] H. Reddy, N. Annamalai, G. Gupta. Listener Controlled Dynamic Navigation of VoiceXML Documents. In 2004 International Conference on Computer Helping People (ICCHP), Springer Verlag. pp [13] H. Reddy. Listener Controlled Dynamic Navigation of VoiceXML. MS Thesis. University of Texas at Dallas, [14] World Wide Web Consortium, XSL Transformations (XSLT). Version 1.0 W3C Recommendation November 16, [15] The SAXON XSLT and XQuery Processor.