TALK project work at UCAM

Transcription

1 TALK project work at UCAM Matt Stuttle August 2005 Web: mns25 Dialogs on Dialogs

2 Overview TALK project Goals Baseline system the Hidden Information State (this) system Proposed system architecture The Speech Understanding (SU) Agent Confidence scores HVS parser Current and future work Dialogs on Dialogs 1

3 Project Goals: TALK project: Tools for Ambient Linguistic Knowledge Implement Information State Update (ISU) approaches Develop in-car and in-home multimodal dialog systems Learning strategies for the ISU approach Integrate multimodality and multilinguality Partners include Edinburgh, Gothenburg, DFKI, Saarbrucken, Seville, Bosch, Linguamatics and BMW Dialogs on Dialogs 2

4 TALK project: UCAM Project Workpackages: WP1: Unifying Multimodality and Multilinguality New Speech Understanding components Developing notion of multimodal dialog act Closely coupling parser, recogniser and dialog manager WP4: Adaptivity and Learning Reinforcement learning approaches (Partially Observable) Markov Decision Process dialog managers User modelling approaches Dialogs on Dialogs 3

5 Baseline System: Current baseline system built with Edinburgh: Based on SACTI Tourist Information domain Built using: DIPPER dialog manager Open Agent Architecture ATK speech recogntion using n-gram language models Word spotting parser Currently features: Implicit and explicit confirmation Fragmentary clarifications Simple fixed dialog plan (slot filling) Overanswering Multimodal output Dialogs on Dialogs 4

6 Proposed HVS System Overview System Architecture Speech Understanding Speech In ATK Parser N best list of hypothesised user dialogue acts Dialogue Manager Turn Manager Buffer this Data Base Speech Out Festival NLG System dialogue acts Four agents: Speech Understanding Dialogue manager Turn Manager Speech Generation Dialogs on Dialogs 5

7 Proposed HVS System Overview Hidden Information State model Operates as a MDP over Belief State B = P (S d, S u ) where S u =user state and S d =dialog state Similar advantages to POMDP for modelling uncertainty but like an MDP can be optimised using on-line methods Belief state is discrete approximation to continuous state space Approximation is a forest of most likely states User state expands according to application specific rules User state captures domain ontology Actions A s are based on a linear function approximation of Q(B, A s ) Dialogs on Dialogs 6

8 Speech Understanding Agent Speech Understanding Agent Speech In ATK HVS Parser A u,1 A A u,2 u,3 N best list of hypothesised user dialogue acts ATK sends N best word lists from lattices A A... u,4 u,n Parser generates dialogue acts continuously Some of UCAM s proposed work focuses on developing the SU agent. Uses ATK and the Hidden Vector State (HVS) Parser. Outputs an n-best list of user dialog acts with scores. Operates asynchronously, forwards dialog acts to the Turn Manager agent. Dialogs on Dialogs 7

9 Speech Understanding (SU) Agent Motivation Asynchronous, so does not impose a turn-taking model - will allow barge-in. Will output the n-best dialog acts Use of HVS and ATK will allow statistical confidence measures to be implemented HVS model can be incrementally trained on-line with system data. Multimodal inputs can be investigated later (WP1 work) Dialogs on Dialogs 8

10 Confidence Scores Current implementation: ATK currently outputs confidence scores based on recognition scores relative to a background model Current scores do not contain much information about correctness of hypothesis Recent work has enabled use of n-best lists and lattice generation in ATK Need to generate very long n-best lists to get alternative parse hypotheses. Dialogs on Dialogs 9

11 Confidence Scores Proposed alternative: Generate word posteriors from lattice for each word position Will generate more hypotheses from parser Deletions/insertions handled by NULL label Can generate alternatives for each word in the best hypothesis Word posteriors have been shown to be more informative Can develop strategies based on fragmentary clarification more easily Currently working on generating information on-the-fly Calculating word posteriors requires forward and backward pass through the (complete) lattice Dialogs on Dialogs 10

12 ATK Outputs I need see the that s near nearer theatre here Lattice Is that i see that s nearer is that nearer i see that s near here is the theatre i need the theatre the nearer I 0.7 Is 0.3 theatre see 0.6 null need that s 0.5 the a 0.2 that nearer 0.7 null theatre 0.1 near 0.2 here N-Best Confidence Network Dialogs on Dialogs 11

13 The Hidden Vector State (HVS) model for Semantic Parsing Overview Trade-off between flat-concept and hierarchical models Flat-concept HMM model extended by expanding each state States encode stack of push-down automata Model is right branching encodes hierarchical context avoids tractability issues of a PCFG approach Dialogs on Dialogs 12

14 Parse Tree generated by the Hidden Vector State Model TOLOC ON DUMMY CITY DATE SE sent_start I want to return to Dallas on Thursday sent_end DUMMY TOLOC CITY TOLOC ON DATE ON SE Parse tree of semantic concepts is right branching Tree is represented in vector state sequence below Dialogs on Dialogs 13

15 The Hidden Vector State (HVS) model for Semantic Parsing Motivation HVS is a statistical semantic parser Fits well into UCAM s plan for a statistical dialogue modelling approach Gives n-best acts together with probability/confidence scores HVS model is data-driven allows on-line/incremental adaptation/training requires only lightly supervised data HVS can be adapted to work on-the-fly will allow barge-in can use more complicated turn-manager Dialogs on Dialogs 14

16 Training and Parsing with the Hidden Vector State Model I want to return to Dallas on Thursday domain specific lexical classes 3 training parsing HVS-Model HVS-Parser abstract semantic annotation for each utterance TOLOC ON DUMMY CITY DATE SE sent_start I want to return to DUMMY TOLOC Dallas CITY TOLOC on ON Thursday DATE ON sent_end SE Dialogs on Dialogs 15

17 Current work Training HVS and word-spotting model on SACTI data for subset of tasks. Encapsulate HVS parser as an ATK component Make speech understanding (ATK+HVS) OAA agent Modify ATK to produce word-posterior networks with associated confidence scores Dialogs on Dialogs 16

18 Future plans (SU) Agent Combine ATK and HVS probabilities for a parser confidence score Adapt HVS approach to work with this data format Current parser results using HVS and word-spotting techniques are relatively poor SACTI data is not ideal: will perform a new data collection to bootstrap parser and language models Collection will feature abstract prompts of all features desired in final system May also do some collections with baseline system Plan to use a real town (Cambridge?) for new data collection with a much larger database extracted from Google maps/yell.com Dialogs on Dialogs 17