Tools and Toolkits for Voice and Animated Character based Interventions. Overview

Transcription

1 Tools and Toolkits for Voice and Animated Character based Interventions Timothy Bickmore & Daniel Schulman College of Computer & Information Science Northeastern University Tim Overview Dialogue based Interventions IVR languages & tools ECA tools Dan Dtask & LiteBody 1

2 Dialogue based Behavioral Informatics Current status of IVR systems Speech Recognition: accuracy of 85 90%+ Speech Synthesis: high quality for short utterances Dialog Management: stronglymanaged dialog flow Language Processing: Very limited Language Generation: Very trivial 2

3 GC_START Are you going to get some [more] exercise today? (time2bed > 2) GC_17 GC_2 (null) X again? if REL & know sport no yes GC_1 (time2bed<2) No (null) Great. I'm going to play a sport. Yes GC_18 GC_3 Yep GC_4 Which one? What kind of exercise are you going to do? Something else. How long do you plan to play for? GC_16 GC_19 I'm going to go for a walk. I'm going to workout at the gym. Are you going to workout tomorrow? What kind of exercise? (TEXTENTRY) I can't GC_7 GC_5 Great. How long do you plan to go for? GC_6 (above exp) (below exp) Great. How much aerobic exercise do you plan to do? Great. How long do you plan to go for? GC_21 (at exp) You shouldn't try to do so Do you think you can go for X minutes? much so soon... How about Do you think you can increase your time a little today/tomorrow? X minutes this time? Can you keep up the same time as yesterday/etc? OK No, I really want to. GC_9 Yes GC_8 No (null) GC_22 OK, but you should try to increase gradually.. Where are you going to walk? MotivateDuration if REL & know location GC_20 (illness injury) GC_11 GC_19 Is it becuase of your illness/ GC_10 Are you going to go injury? No GC_13 with X again? No (no illness/injury) Yes Are you going to (location X) again? yes No yes (if REL & know buddy) MotivateToExercise GC_12 (null) (else) (loner) GC_14 Are you going to gowith anyone? Yes No GC_15 Who? GC_END 4/27/2011 Directory assistance Taxi bookings Stock transactions Remote banking Travel reservations Auto attendants Pizza ordering And Current IVR Apps State of the Art: Scripted Interactions Scripts written by teams of experts Represented as flow charts or Augmented transition networks Implemented as state machines / ATNs / VXML GetCommitment 3

4 Voice XML What is it? Standardized language for specifying voice dialogs Output: Prerecorded audio and text to speech (TTS) Input: Touch tone keys and Automatic Speech Recognition (ASR) Extension of XML Designed to interact with web based applications History Phone Web project by AT&T Research Lucent and AT&T have incompatible dialects of Phone Markup Language So, VoiceXML Forum created with AT&T, Lucent, Motorola, and IBM Team develops VoiceXML 0.9, a first pass at standardization 2000 VoiceXML 1.0 was created and submitted to World Wide Web Consortium (W3C) 2001 VoiceXML 2.0 by W3C s Voice Browser Working Group 4

5 Big Picture: The phone as a web browser 5

6 VXML is a kind of XML Tags and body <cmu> <welcome>welcome to CMU! </welcome> <ecom> <welcome>welcome to the E-commerce!</welcome> </ecom> </cmu> zero or more attributes <welcome accent= texan >Welcome</welcome> <welcome accent= pittsburgh >Welcome</welcome> Tag with no body <breath/> A simple example <?xml version="1.0" encoding="utf-8"?> <vxml version="2.1"> <form> <field name= goingtobeach" type="boolean"> <prompt> Are you going to Daytona Beach this year? </prompt> <filled>ohh... <if cond= goingtobeach">that s great! <goto next= goingdocument.vxml" /> <else /> bummer maybe next year. <goto next= notgoingdocument.vxml" /> </if> </filled> </field> </form> </vxml> 6

7 A simple example ASR grammar & TTS prompt <?xml version="1.0" encoding="utf-8"?> <vxml version="2.1"> <form> <field name= goingtobeach" type="boolean"> <prompt> Are you going to Daytona Beach this year? </prompt> <filled>ohh... <if cond= goingtobeach">that s great! <goto next= goingdocument.vxml" /> <else /> bummer maybe next year. <goto next= notgoingdocument.vxml" /> </if> </filled> </field> </form> </vxml> A simple example Execution conditioned on ASR recognition <?xml version="1.0" encoding="utf-8"?> <vxml version="2.1"> <form> <field name= goingtobeach" type="boolean"> <prompt> Are you going to Daytona Beach this year? </prompt> <filled>ohh... <if cond= goingtobeach">that s great! <goto next= goingdocument.vxml" /> <else /> bummer maybe next year. <goto next= notgoingdocument.vxml" /> </if> </filled> </field> </form> </vxml> 7

8 A simple example if/else with goto <?xml version="1.0" encoding="utf-8"?> <vxml version="2.1"> <form> <field name= goingtobeach" type="boolean"> <prompt> Are you going to Daytona Beach this year? </prompt> <filled>ohh... <if cond= goingtobeach">that s great! <goto next= goingdocument.vxml" /> <else /> bummer maybe next year. <goto next= notgoingdocument.vxml" /> </if> </filled> </field> </form> </vxml> Commercial IVR Tools IBM WebSphere Voice Toolkit 8

9 Commercial IVR Tools Audium Studio Commercial IVR Tools Voxeo Designer 9

10 Commercial IVR Tools & Services VoiceXML based Solutions Directory at: Embodied Conversational Agents Automated Interventions (not computer mediated communication) 10

11 Natural, intuitive modality provides rapid acceptance Emulate human face to face conversation Focus on nonverbal communicative behavior gaze, posture, gesture, etc. Motivation: Acceptance & Usability 11

12 Motivation: Trust Building Agents can emulate human relationship building behavior to build and maintain trusting, working relationships. Motivation: Clients with Low HealthLiteracy Face to face communication, in conjunction with written instructions, is best However, most professionals have extremely limited time to spend with a customers 12

13 Current / Past Projects NIH/National Cancer Institute R01 Oncology clinical trial adherence R01 Exercise & UV for Cancer Prevention (URI) R21 Exercise Promotion for Older Bilingual Latino Adults (Stanford) NIH/National Institute on Aging R01 Clinic based Walking Promotion for Older Adults (BMC) NIH/National Heart, Lung, and Blood Institute R01 Hospital Discharge Patient Education (BMC) NIH/National Library of Medicine R21 Exercise & Diet promotion R21 Exercise promotion on mobile devices AHRQ Web based post discharge medication adherence/screening (BMC) Preconception care for young African American Women (BMC) NSF Exercise promotion for older adults 13

14 Embodied Conversational Agents Dialogue Engine options: Keyword match rules HTN/ATN (VoiceXML) Planner Output: typically TTS & animation Input options: Multiple choice Typed text Speech Embodied Conversational Agents Output Toolkits 14

15 Embodied Conversational Agents Standardization Efforts SAIBA Framework Situation / Agent / Intention / Behavior / Animation BML / FML Kopp, S., B. Krenn, et al. (2006). Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. Intelligent Virtual Agents. Marina Del Rey, CA. BML Synchronization Points 15

16 Xtranormal State Microsoft Agent Beginning with Windows 7, Microsoft Agent will not be included or supported in future versions of the Microsoft Windows operating system. We encourage Microsoft Agent application developers and redistributors to evaluate their activities in light of this decision. 16

17 4/27/2011 Oddcast SitePal Scalable, web based, 2D (Flash based) characters. Limited dialog / logic support. sitepal.com SmartBody (BML support) USC Institute for Creative Technologies 17

18 3D Modeling & Animation Poser (Smith Micro) Maya etc. Dialogue Engines Eliza/ALICE (e.g. alice.pandorabots.com) Latent Semantic Analysis Virtual People Factory (Virtual Experience Research Group, U. Florida) VoiceXML CEA 2018/ DISCO web.cs.wpi.edu/~rich/cetask/ 18

19 Dtask & Litebody Developed under NLM grant to provide open source tools for web delivered, ECA based interventions in behavioral medicine Demonstrate use of standards Ontologies & task models for exericse and fruit&vegetable promotion LiteBody web based ECA framework Dtask plan based dialog engine 19