An Overview of Speech Based Interfaces

Size: px

Start display at page:

Download "An Overview of Speech Based Interfaces"

Steven Webb
5 years ago
Views:

1 An Overview of Speech Based Interfaces PART II: Applications and Business Cases Dr. Jean Hennebert, HES-SO Overview PART I Fundamentals of Speech Technologies PART II Applications and Business Cases Introduction Speech as a source of information Speech production process Speech signal analysis Overview of Speech Technologies Audio Coding Speaker Recognition Text to Speech Systems Speech Recognition 2 Text-to-speech providers and applications Dictation Systems Example in the medical domain Telephony Dialog Systems Automatic indexation of audio and video documents Mobile Services translation, car systems Conclusions - What s next?

2 TEXT-TO-SPEECH PROVIDERS AND APPLICATIONS Commercial TTS providers Text-to-speech synthesis Nuance Vocalizer 5 Acapela group Voice as a service Loquendo Emotion in the TTS SVOX Focus on car systems 4

3 TTS applications Over the phone information applications Environments where hands are busy Cars: vehicule information, voice guiding in GPS Warehouse: stock management, parcel delivery Sight disabled / blind people Web pages automated readings W3c recommendations exist Legal aspects enforce accessibility for blinds to public services Places where visual display would be too expensive or too intrusive Museum Train Places where there is a sense of emergency about the information Train stations 5 DICTATION SYSTEMS FOR VERTICAL SEGMENTS

4 Dictation systems Unconstrained dictation is difficult You need To train the system to your voice To train yourself to the system To speak the standard version of the language A quite environment 3 times faster as a novice keyboard user Not worthwhile if you are a good keyboard user Dictation systems for vertical markets Reducing the vocabulary search space makes the application more viable For legal domain For medical domain

grammar Provide phonetic transcriptions for unknown words 2.

5 Typical medical installation 1. Collect documents from the doctor make the system learn the vocabulary and grammar Provide phonetic transcriptions for unknown words 2. Train the system on the voice of the doctor 3. Repeat the operation if needed Companies are coaching this procedure TELEPHONY DIALOG SYSTEMS

6 Classification of automatic speech recognition systems Application control - Speaker (in)dependent - Runs on PDA - Web navigation,... - A couple hundred words Signal Quality Dictation System - Speaker dependent - Runs on laptop - Word processing,... - Up to 50K words CPU 11 Keyword recognition - Word-print based - Runs on mobile phones - Voice dialing appl.,... - A few dozen words Server-side ASR - Speaker independent - Runs on big servers - Dialog based appl. - Up to 10K words Dialog Machines Dialog Machine IVR system Dialogs Information V-commerce DTMF TTS ASR SV Automatic Attendant IVR = Interactive Voice Response DTMF = Dual Tone Multiple Frequency TTS = Text-to-Speech ASR = Automatic Speech Recognition SV = Speaker Verification 12

7 !"#$%&'()*$#'+*'!#,-.' 13 Difficulties (1/2)! Due to telephony channel! limited bandwidth, channel variability! environmental noise! Due to service constraints! Speaker independence! Barge-In capability! Due to the language! homophone, ambiguities! coarticulation! Due to the user! Next slide! 14

language skills Technology rejection Users should be 15 educated and motivated Strategy to face these difficulties

8 Difficulties (2/2) Difficulties (2/2) Not used to this new Human Computer Interface High expectations Easily frustrated No discipline Short-term auditive memory Hesitations, fillers, breathings Phone from any place Poor language skills Technology rejection Users should be 15 educated and motivated Strategy to face these difficulties 1. Reduce the vocabulary search space 2. Split the inputs through directed dialogs 3. Plan for fall-back strategies Operator pool

9 Dialog design Finite State Automaton Dialog Module Start End 17 Dialog State Prompt When do you want to fly? Grammar dates.grammar Prompt Design (1/4) Open questions Example: How may I help you? Trigger many different answers. Make grammar design more difficult (see later). Make search space larger thus decreasing speech recognition accuracy; Are more user-convenient and implement fast dialogs. Closed questions Example: Chose one of the following option: check plane arrival time or book a flight Trigger few different answers. Make grammar design easier; Make search space smaller this increasing speech recognition accuracy; Are less user-convenient and make long dialogs. 18

10 Prompt Design (2/4) The risk associated to a dialog state influences the prompt design. Information versus Transaction High user tolerance Large vocabulary No need for operator assistance Low user tolerance Medium vocabulary Need for operator assistance Open questions Closed questions Risk 19 Prompt design (3/4) Low risk Welcome to United Airlines. I will help you getting information on flight schedules. For what flight would you like information? Flight from Boston to San Francisco arriving approximately at 6 PM tomorrow. We have two flights corresponding to your request. Flight number 1, leaving from Boston to San Francisco, departure tomorrow at 3 PM and arriving... High risk Welcome to MyBank. Please say your 7 digits identification number , is that correct? Yes What would you like to do: check your balance or transfer money? Transfer money Transfer money. From which account would you like to transfer money. My checking account From your checking account, is that correct? Yes How much would you like to transfer? 20

11 Prompt Design (4/4) Prompt design follows usability design rules Low cognitive load Efficiency Accuracy Graceful error recovery Clarity 21 Grammar design (1/4)! A grammar defines the set of phrases/words that the caller can speak to interact with the system;! A grammar is tied to each dialog state;! A grammar is intimately designed with the dialog question; ^ Large grammars come with open questions; ^ Small grammars come with closed questions;! Grammars can be static or dynamic 22

12 Grammar design (2/4)! Grammars are defined using Grammar Specification Languages (Nuance GSL, GRXML, ): Example with Nuance GSL OR [... ] AND (... ) OPTIONAL?... REPETITIVE +... EXAMPLE: yes/no grammar [ yes no ] [ (yes?please) (no?(thank you)) ] 23 Grammar design (3/4).YES_NO (?[um uh] [([ (yes?(it?[sure certainly] is)) (it?[sure certainly] is) yup yeah okay sure (you got it) (?(?yes that's) [right correct]) ]?[please thanks (thank you)]) {return(yes)} ([ nope (absolutely not) (no?[[(it isn't) (it's not) (it is not)] way]) [(it isn't) (it's not) (it is not)] (?(?no that's) [wrong (not [correct right]) incorrect ] ) ]?[thanks (thank you)]) {return(no)} ] ) ways to say yes or no

13 Grammar design (4/4) Grammar design is iterative Start with a reasonable a priori coverage of potential answers Collect production logs Tune the grammar to cover up coverage and remove un-probable answers VoiceXML 2.0 VoiceXML 2.0: markup language designed for creating audio dialogs that feature synthesized speech, digitized audio, recognition of spoken and DTMF key input, recording of spoken input, telephony, and mixed initiative conversations. In W3C Voice Browser activity VoiceXML

14 Comparison to HTML «VoiceXML = language for writing Web pages you interact with by listening to spoken prompts and jingles, and control by means of spoken input.» «HTML was designed for visual Web pages and lacks the control over the user-application interaction that is needed for a speech-based interface. With speech you can only hear one thing at a time (kind of like looking at a newspaper with a times 10 magnifying glass). VoiceXML has been carefully designed to give authors full control over the spoken dialog between the user and the application. The application and user take it in turns to speak: the application prompts the user, and the user in turn responds.» From Dave Raggett, W3C 27 System architecture HTML IP" users! IP" PSTN" IP" Application server Content database PBX! Voice platform VXML Lexicon Grammars Waves 28

15 Voice platform PBX VoiceXML interpreter Speech resources Telephony web ASR TTS SV Voice platform 29 VoiceXML features VoiceXML documents describe: spoken prompts (synthetic speech) output of audio files and streams recognition of spoken words and phrases recognition of touch tone (DTMF) key presses recording of spoken input control of dialog flow telephony control (call transfer and hang-up) 30

16 Cloud speech services providers Full remote voice xml platform Speech recognition services Speech synthesis services Providers: Some telephony voice-activated application examples

Directory Assistance 33 Directory Assistance Business Driver Human operators are costly Call centers are difficult to manage turn rate Clients are shifting to less costly technologies (web) Solution

17 Directory Assistance 33 Directory Assistance Business Driver Human operators are costly Call centers are difficult to manage turn rate Clients are shifting to less costly technologies (web) Solution Dedicated text-to-speech engine to pronounce correctly names and cities Multilingual system Cost = 1 tenth of operator cost However: a large part of the cost is in the marketing of the phone number 34

PHONE BANKING UBS 35 Business Driver 6 Mio calls to the branches with the question: account balance Reduction of the number of branches Consequence " more calls in contact center " more agents 600 K

18 PHONE BANKING UBS 35 Business Driver 6 Mio calls to the branches with the question: account balance Reduction of the number of branches Consequence " more calls in contact center " more agents 600 K e-banking customers only 300 K active 2.8 Mio retail customers The Solution Natural language understanding software 150 Ports - Multilingual (German, English, French, Italian) Secure call transfer to call center agent Seamless integration in multi-channel infrastructure NEW Relies only on spoken commands Highways: «summary», «A12» Cantons: «Tessin» Passes: «Nüfenen Pass» Train: «summary» Up-to-date information from ViaSuisse Pre-recorded speech elements and text-to-speech Ideal for hand-free phones in cars

0848-football Subscription to sms services SNOWPHONE [0900 11 0900]

ski ressort Available in German, French & English Collaboration with

19 0848-football Subscription to sms services SNOWPHONE [ ] Delivers up-to-date snow information for all Swiss ski ressorts Just name a ski ressort Available in German, French & English Collaboration with Seilbahnen Schweiz SNOWPHONE Bei Anruf Schnee! EXCELSIS Business Technology 38

20 Yellow pages on the phone Will probably follow the same business model as Google ads Voice mail to text Transform voice mail to or sms SMS or to voice For the blinds For car drivers

21 Other known systems Airline reservation Phonebanking Ikea, UPS, AOL, Daimler Chrysler, Ford, General Electric, LG, Nokia, SBC, United Airlines, Verizon and Vodafone. 41 Demos Airline reservation Phonebanking 42

Celebrity voices are sometimes used to create a persona, image of

22 Persona creation Concept of Sub-Servant Alien Sub-servant: it is a human-like service Alien: the service do not replace a human Celebrity voices are sometimes used to create a persona, image of the enterprise offering the service AUTOMATIC INDEXING OF AUDIO DOCUMENTS

23 Classification of automatic speech recognition systems Application control - Speaker (in)dependent - Runs on PDA - Web navigation,... - A couple hundred words Signal Quality Automatic Indexing Dictation System - Speaker dependent - Runs on laptop - Word processing,... - Up to 50K words CPU Keyword recognition - Word-print based - Runs on mobile phones - Voice dialing appl.,... - A few dozen words Server-side ASR - Speaker independent - Runs on big servers - Dialog based appl. - Up to 10K words 45 Applications Audio indexing for search engines Text query for broadcast video Audio query Automatic subtitles Off-line Direct Copyright infringement detection 46

24 Audio indexing for search engines Google Audio Indexing Gaudi limited to political channels 47 48

25 49 Nouveaux métiers Perroquet Souffleur Correcteur 50

26 MOBILE PHONE SYSTEMS CAR SYSTEMS Speech synthesis Is going mobile iphone demo Coupling machine translation with speech synthesis 52

27 Speech recognition on smart phones We reach almost the same capacities as on desktop machines Constraining the vocabulary makes application viable dictation on mobile is showing up

28 Constrained dictation for vertical segments ARTISTIC APPLICATIONS

Speech to musical arrangement Demo from Microsoft Pitch detection Voice prosody pattern modeling Automatic arrangement http://research.microsoft.

29 Speech to musical arrangement Demo from Microsoft Pitch detection Voice prosody pattern modeling Automatic arrangement TTS technologies for extending voice capabilities New voice creation Castrato Farinelli Mix of countertenor (male) and soprano (female) Timber homogenisation Celebrity voice dubbing

30 Conclusions Speech technologies is a large market Maybe not in generalistic dictation systems Rather in TTS, vertical segment dictation, speech recognition over telephony lines Equivalent of html for voice activated dialog systems: VoiceXML Speech as a service is showing up TTS Speech recognition Speech applications are coming in the market of mobile and embedded systems 59 Further Readings M. Cohen et al., «Voice User Interface Design», Addison Wesley, 2004, ISBN

An Overview of Speech Based Interfaces

An Overview of Speech Based Interfaces PART II: Applications and Business Cases Dr. Jean Hennebert, HES-SO Overview PART I Fundamentals of Speech Technologies MScBA PART II Applications and Business Cases