INFORMATION ACCESS VIA VOICE. dissertation is my own or was done in collaboration with my advisory committee. Yapin Zhong

Transcription

1 INFORMATION ACCESS VIA VOICE Except where reference is made to the work of others, the work described in this dissertation is my own or was done in collaboration with my advisory committee. Yapin Zhong Certificate of Approval: Kai H. Chang Professor Computer Science and Software Engineering Juan E. Gilbert, Chair Assistant Professor Computer Science and Software Engineering Dean Hendrix Associate Professor Computer Science and Software Engineering Stephen L. McFarland Acting Dean Graduate School

2 INFORMATION ACCESS VIA VOICE Yapin Zhong A Dissertation Submitted to The Graduate Faculty of Auburn University In Partial Fulfillment of the Requirements for the Degree of Doctor Philosophy Auburn, Alabama December 19, 2003

3 INFORMATION ACCESS VIA VOICE Yapin Zhong Permission is granted to Auburn University to make copies of this dissertation at its discretion, upon the request of individuals or institutions and at their expense. The author reserves all publication rights. Signature of Author Date Copy sent to: Name Date ii

4 2003 YAPIN ZHONG All Rights Reserved iii

5 DISSERTATION ABSTRACT INFORMATION ACCESS VIA VOICE Yapin Zhong Doctor of Philosophy, December 19, Typed Pages Directed by Dr. Juan E. Gilbert This dissertation concentrates on the problem of designing and developing a spoken query retrieval (SQR) system to access large document databases via voice. The main challenge is to identify and address issues related to the adaptation and scalability of integrating automatic speech recognition (ASR) systems and information retrieval (IR) systems. Additionally, the mechanics of designing an effective and efficient speech user interface (SUI) pose yet another significant challenge, especially since the aim is to facilitate voice queries of large document databases. The resulting system should enable users to access large document databases effectively and efficiently. Furthermore, its language model should be capable of adapting to updates of the document databases. In this research, a framework allowing information access to large document databases via voice is presented and several approaches designed to cope with the issues of adaptability, scalability, effectiveness and efficiency are described in detail. Through iv

6 experiments performed on the TREC-9 document dataset, the performances of the new approaches were evaluated and their potential was demonstrated. v

7 ACKNOWLEDGMENTS The author would like to express his deep gratitude to his advisor, Dr. Juan E. Gilbert, for his patient guidance, valuable advice, and continued encouragement throughout his studies. Sincere thanks are also due to his two graduate committee members, Dr. Kai H. Chang and Dr. Dean Hendrix, for their reviewing and advising efforts. In addition, the author would like to thank Celeste German for her reviewing and valuable comments. Finally, deepest thanks to author s wife, Weihong Hu, for her help while conducting the experiment and constant support. vi

8 vii

9 Information Access via Voice Yapin Zhong Dept. of Computer Science & Software Engineering Auburn University

10 Outline Motivation Background Research Challenges A Framework of Spoken Query Retrieval Experiments and Research Findings Conclusions 9/17/2003 2

11 Motivation A very large part of the world population does not have access to either computers or the Internet Very tiny visual interfaces make users feel quite uncomfortable Blind or partially-sighted users are not able to access information visually 9/17/2003 3

12 Background Two categories: spoken document retrieval (SDR) and spoken query retrieval (SQR) In SDR, written queries are used to search speech archives for relevant speech information SQR uses spoken queries to retrieve relevant textual information 9/17/2003 4

13 Desktop User Phone User PDA User TCP/IP Telecommunications Infrastructure Mobile Phone User Voice Server 9/17/2003 5

14 Voice Server Nuance Voice Platform VoiceXML Interpreter & Controller Speech Recognition Audio Recording DTMF Speech Synthesis Audio Playback Databases Telecommunications Infrastructure 9/17/2003 6

15 Automatic Speech Recognition arg max W P( W X ) = arg max W P( X W ) P( W ) User Speech Parameterization Acoustic Models Pattern Matching Language Models Transcriptions 9/17/2003 7

16 Three major properties in SQR Spoken Spoken queries are usually very short Spoken Spoken queries usually need a very large vocabulary Query Query processing is required to be in close to real time 9/17/2003 8

17 Research Challenges A lack of adaptability and scalability in integrating language models between ASR and document Information Retrieval systems Difficult to meet the critical aspect of real time user expectations A general lack of effectiveness and efficiency in designing speech user interfaces 9/17/2003 9

18 The Language Size Vs. The time Consumed Time (sec) Vocabulary Size 9/17/

19 The Language Size Vs. the Word Recognition Error (WRE) Rate WRE (percent) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Vocabulary Size 9/17/

20 Query Coverage Compared to the Language Size Vocabulary Size (k) Query Coverage [FM02] (percent) /17/

21 Problems in SQR Interfaces Speech is transient but graphics are persistent Speech is invisible Speech is asymmetric 9/17/

22 A Framework of SQR Design Principle System Architecture Context-Aware Language Model Bisecting K-MedioidsK Method Voice Navigator Information Verbalization 9/17/

23 Design Principle Integrating ASR systems with existing IR systems, but not simply combined by the way of input/output protocol [FII02] Language models that will enable both adaptability and scalability so as to satisfy the document retrieval requirements for large databases Effective and efficient SQR user interfaces 9/17/

24 System Architecture Speech Interface Voice Portal Backend Speech Telecom Network Internet Voice Server Voice Navigator Query Functions Result Access Functions L M Internet L M A IR DB 9/17/

25 Context-Aware Language Model (CALM) Speech A Context-Aware Language Model ASR Active LM IR LM Agent DB General LM 9/17/ SG SG

26 Procedure to construct CALM Preprocess and Index the collected documents Represent each document with a vector Cluster the collected documents into certain groups Represent the center of each group with an important document Construct the CALM with a set of important keywords from the centered document 9/17/

27 9/17/

28 Document Clustering Analysis Assign a set of documents to the different groups based on their similarity closely associated documents tend to be relevant to the same requests document clustering should result in more effective, as well as more efficient, retrieval Hierarchical and partitioning clustering 9/17/

29 Bisecting K-MedioidsK (BKMdd) A medioid representative An objective function to control the iterative optimization: J = 2 i= 1 n j = 1 A two-phase clustering method d 2 ij 9/17/

30 BKMdd The bisecting phase: Set iter = 0; Repeat Compute by using (2.2.1); Assign V old = V; 2 n 2 Compute the new medioid set by using J = d ij; i = 1 j = 1 Until ( = or iter = MAX_ITER) The K-MedioidsK phase: Set K; Repeat Until K 1 ( M ) = M M 2 d ij j= 1 Pick a cluster to split by using ; Find two sub-clusters by using Bisecting phase; 9/17/

31 Voice Navigator (VN) Category System Query Results Browsing Functions Help Main menu Try again Exit (Goodbye) By source By field Literal Response Cooperative Response Read by title Read by abstract Previous Next Repeat Stop Save 9/17/

32 VN Dialogues Main Menu Dialogue Query Dialogue Results Dialogue Save Dialogue 9/17/

33 Diagram of the VN Welcome Message Main Menu Dialogue Query Dialogue Results Dialogue Confirm Save Dialogue Confirm MyLibrary 9/17/

34 Information Verbalization The use of computer supported, auditory interactions to amplify understanding of abstract and/or large data Literal Response (LR) Cooperative Response (CR) Mixed Intelligent Response (MIR) Cluster-based Intelligent Response (CIR) 9/17/

35 MIR Strategy MIR Combine LR and CR strategies to present all documents in the ranked results set one by one if there is no response from the user. If the system receives any responses from the user, MIR will stop the current presentation immediately, then process the action quickly 9/17/

36 CIR Strategy The results are clustered before present action The results are ranked within each cluster The top ranked documents are selected from each cluster The number of the documents to be presented is usually five but never beyond nine 9/17/

37 Experiments and Research Findings Experimental protocol Data collection methods Participants and procedure Evaluation Metrics 9/17/

38 Experimental Protocol Materials Large document databases: Data Set: OHSUMED ( ) Number of documents 348,566 Size of collection in Mb Average document length 195 Average document length (unique terms) 111 9/17/

39 Data Collection Methods Dialogue recordings System logs User surveys 9/17/

40 Participants and Procedure 39 college level students Read the instructions Access the system Search documents Fill out the survey 9/17/

41 Evaluation Metrics Spoken query metrics Task success metrics Interface efficiency and quality metrics User satisfaction 9/17/

42 Spoken Query Metrics Percent of Queries Female/Male Native/Second Under/Grad 9/17/

43 Numbers of Users, Queries, and Terms Total Number of Participants Total Number of Spoken Queries Average Number of Spoken Queries per User Number of Unique Queries Total Number of Spoken Terms Total Number of Uniquely Spoken Terms Mean Number of Terms /17/

44 Users by Number of Queries SQR: 70% of users more than a single query Excite: 67% of users had one and only query Spoken query modification was a strong trend No. of users No. of queries per user 9/17/

45 Spoken Queries & Documents Reviewed Significant association between the spoken queries and the document reviewed Documents/Queries Subjects Documents saved/viewed Queries 9/17/

46 Queries by Number of Terms SQR: More than 60% of the queries less than or equal to 2 terms. the mean of terms was 2.66 Excite: the mean of Web search terms was 2.21 No. of Queries No. of Terms in Query 9/17/

47 Term Frequency Distribution 24 terms covers 82.8% of all queries Term Frequency Query Terms 9/17/

48 The CALM Coverage The CALM consisted of 389 terms which represent at least a 93% coverage of all the most important terms found in all documents There were 138 unique spoken queries covering 99 unique terms There were 8 unique terms that were spoken by participants that did not appear in the CALM The experimental coverage of the CALM was 93.10% 9/17/

49 Task Success Metrics The mean of Documents Found was 1.54 Significant as a function of users experience No significant difference between the MIR &CIR strategy Significantly negatively correlated to Barge Ins, Query Terms Mis-recognition, and Word Recognition Error (WRE) 9/17/

50 Interface Efficiency Metrics System Turns and User Turns were positively associated with Query Term Mis-recognitions and Barge Ins Elapsed Time was positively associated with Query Term Mis-recognitions, Barge Ins, and Spoken Queries 9/17/

51 Interface Qualitative Data ASR rejections were positively associated with WRE, Barge Ins, and Spoken queries 40.00% 35.00% 30.00% 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% ASR Rejections Query Terms Mis-... Timeouts Help Requests Barge Ins 9/17/

52 Word Recognition Error (WRE) The WRE rate for the entire set of spoken query utterances was 6.87% 12% 11.29% 10% 8.74% WRE 8% 6% 6.70% 7.05% 5.10% 5.90% 4% 2% 0% Female/Male Native/Second Language Under/Grad 9/17/

53 User Satisfaction The average user satisfaction rating was 82% based on 31 satisfactory factors User Satisfaction (percent) 88% 86% 84% 82% 80% 78% 76% 74% 72% 0.77 TTS ASR Task Ease System Response Expected Behavior Interactive Pace User Expertise Future Use 0.78 satisfactory factors 9/17/

54 Task Success & Results Presentation Strategies Task Success was significantly positively related to Results Presentation Strategy No significant difference in the Task Success measure as a function of the Results Presentation Strategy 9/17/

55 Users Satisfaction Perceptions with Touchtone Experience No significant difference between subjects who had touchtone and no touchtone experience Percentage 90.00% 88.00% 86.00% 84.00% 82.00% 80.00% 78.00% 76.00% 74.00% 72.00% 70.00% TTS ASR Ease of Use System Response Expected Behavior Interactive Pace User Expertise Future Use Task Success User Satisfaction Presentation Results Touchtone Experience No Touchtone Experience 9/17/

56 Users Satisfaction Ratings with SUI experience No significant difference as a function of subject SUI experience Percentage % 90.00% 80.00% 70.00% TTS ASR Ease of Use System Response Expected Behavior Interactive Pace User Expertise SUI Experience No SUI Experience Future Use Task Success User Satisfaction Presentation Results 9/17/

57 Conclusion Achieved a high user satisfaction rating Achieved a high Task Success rating Performed well with regard to interface efficiency Kept its promise of an improved interface quality The CALM was developed with a high coverage 9/17/

58 Contributions Identified and addressed the issues and constraints that the effects of a language model on the ASR performance Addressed the issues related to SUI that arise in performing SQR tasks Studied and investigated document clustering techniques and VoiceXML technologies 9/17/

59 Contributions (Cont d) Proposed an architecture of SQR systems for large document databases Defined a Context-Aware Language Model (CALM). A document clustering technique was employed to build such a CALM Defined an effective and efficient dialogue framework to facilitate SQR tasks Proposed two information verbalization strategies to present the retrieval results. 9/17/

60 Future Research Study how to combine a visual and verbal user interface to interact with the user to enhance the user s information access Document summarization may improve the user s information access Temporal and contextual factors may affect the user s information needs 9/17/

61 Thanks Questions? Comments? 9/17/

62 References [FII02] Fujii,, A., Itou, K., and Ishikawa, T. A Method for Open-Vocabulary Speech-Driven Text Retrieval. Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP2002), pp , 195, July [FM02] Franz, Alexander, Milch Brian. Searching the web by voice. Proceedings of the 19th International Conference on Computational Linguistics (COLING), pages , 1217, /17/