Lightly Supervised Learning of Procedural Dialog Systems. Svitlana Volkova Pallavi Choudhury, Chris Quirk, Bill Dolan, Luke Zettlemoyer

Transcription

1 Lightly Supervised Learning of Procedural Dialog Systems Svitlana Volkova Pallavi Choudhury, Chris Quirk, Bill Dolan, Luke Zettlemoyer

2 Motivation Procedural dialog systems aim to assist users with a range of goals: guide visitors through a museum (Traum et al., 2012) teach students physics (Dzikovska et al., 2011) enable interaction with a health care system (Morbini et al., 2012) Limitations: require expensive manual engineering substantial domain-specific task knowledge Building scalable dialogue systems in an automated way with light supervision

3 Intent-Oriented Dialog U: I want to add page numbers and a title S: Top or Bottom of the page? U: Top S: Please select page design from the templates (*System shows drop down menu*) U: *User selects from menu* S: Enter header or footer content U: C.V. S: Task completed.

4 Our Approach I. Learn procedural dialog systems from task-oriented textual resources ü I. Query logs ü II. Instructional text II. Use light non-expert supervision III. No help of knowledge engineers or domain experts!

5 Data: Query to Help Page Mapping Queries Help Pages ü help with page numbers in the header ü how to number some pages ü word adding page numbers ü how do you add numbering word format footer sections ü how ü break do you page add a page number on the right and ü how header do I on page the break left ü manual page break ü inserting a manual page break ü how do you insert a manual page break ü a ü manual adding page background break is also known as ü manual ü add page line background break ü how to add a background ü delete background ü how do you delete the color background ü erase ü inserting background comments ü deleting ü insert background a comment ü how to insert a comment in word ü how to add comments ü insert comment in word 2007 ü how to delete comments in word ü delete comments function

6 Data I. Instructional Text Topic Goal Goal Instructions Other sources of instructional text: recipes that describe how to cook meals software help web pages that describe how to achieve goals and

7 Data II. Query Logs Search query logs = User intent statements Queries mapped to Microsoft Office Help Pages High frequency pain points (>1K queries per page) Other sources of user intents: Crowdsourcing or popular applications This is not dialog data! No sequences of utterances! Data omits slot fillers e.g., for the goal add a border color, style typically left unspecified

8 Key Steps I. Learning to convert the textual knowledge into procedural knowledge q Building Dialog Trees from Instructions II. Learning a dialog manager using classificationbased approach q Understanding Initial Dialog Turns q Understanding Subsequent Dialog Turns

9 Building Dialog Trees from Instructions Input: help pages and queries Output: Step 1. Extraction for every page extract a topic, goals and instructions Step 2. User Action Classification classify user actions and identify system actions construct dialog trees

10 Input: Instructions and Queries Queries ü help with page numbers in the header ü how to number some pages ü microsoft word adding page numbers ü how do you add numbering in ms word ü word format footer sections ü how do you add a page number on the right and header on the left ü insert page numbers not available ü how to insert page numbers on both side of the pages Goal 1 Topic Goal 2 Instructions

11 Output: A Dialog Tree Add and format page numbers Add header or footer content to a preformatted page number design t 1 q t Include Page or Page X of Y with the page number q g g 1.1 g 1.2 g 1.3 g 1.4 a u : Binary a s : Top of Page or Bottom of Page? a a u : Selection a s : Popup a a s : Enter header or footer context? a u : Input a q a Execute

12 Step 1. Extraction Office Help Page Interaction points = Actions t 1 g 1.1 i i i g 1.2 i i ADD AND FORMAT PAGE NUMBERS CREATE A HEADER OR FOOTER THAT CONTAINS ONLY THE PAGE NUMBER 1. On the Insert tab, in the Header & Footer group, click Page Number. 2. Click Top of Page or Bottom of Page, depending on where you want page numbers to appear in your document. 3. Choose a page number design from the gallery of designs. INCLUDE PAGE OR PAGE X OF Y WITH THE PAGE NUMBER 1. On the Insert tab, in the Header & Footer group, click Page Number. 2. Click Top of Page, Bottom of Page, Page Margins, or Current Position depending on where you want page numbers to appear. None Binary Selection None Selection

13 Step 2. User Action Classification Data (manually labeled): 30 topics with >1K queries, 76 goals and 246 instructions, ~ 3 instructions per goal p(a u i, θ) = eθ φ(a u,i) a u eθ φ(a u,i) Classify user actions automatically with limited annotation effort Features Accuracy Baseline 1: Majority 0.53 Baseline 2: Heuristic 0.64 Ngrams 0.89

14 Key Steps I. Learning to convert the textual knowledge into procedural knowledge þ Building Dialog Trees from Instructions II. Learning a dialog manager using classificationbased approach q Understanding Initial Dialog Turns q Understanding Subsequent Dialog Turns

15 Understanding Initial Dialog Turns Input: dialog trees with topics, goals, actions and queries new query q Output: an initial dialog state s 0 that maximizes ( ( predict the deepest relevant node in the dialog tree

16 Query To Dialog State Mapping t 1 queries: page number numbering pages how do you number pages how do you do page numbering p(s 0 q, θ) = e i θ iφ i (s 0,q) s 0 e i θ iφ i (s 0,q) g 1.1 queries: number page with text page numbering with text prefix how do you put your name and a page number on each page a 1.1 queries: headers page number inserting page number in header page numbers on bottom of page page numbering at bottom page g 1.1 g 1.2 a a a t 1 g 1.3 g 1.4 Execute q 1 : how to number pages q 2 : add content with page number q 3 : set a page number in a footer

17 Results: Understanding Initial Dialog Turns Lexical features: Query n-grams + stems + no stop words State features: TFIDF 10 best scores and query q overlap with: QueryOverlap: a set of queries from a node QueryHistoryOverlap: queries from the parent nodes Data: ~1K query to node mappings (80/20 splits) 5-10 queries per every topic, goal and less for action nodes Feature Types Topic Goal Action Baseline 1: Random Baseline 2: TFIDF 1 Best Lexical (L) n-grams + stop words + stems L + TFIDF 10Best + QueryOverlap All above + QueryHistoryOverlap

18 Understanding Subsequent Dialog Turns Input: a query q, system action from previous dialog state and a new query q Output: a new dialog state s at any level in the dialog tree: (

19 Dialog State Update Classification Start APPEND S: Top of Page or Bottom of Page? g 1.1 g 1.2 a U: I want to add page numbers and a title ADD AND FORMAT PAGE NUMBERS S: Please select page design from the templates t 1. S: Enter header or footer content. g 1.4 a a U: Top U: Select U: CV t 13 g Execute INSERT A SECTION BREAK U: I want to add page numbers and a title S: Top or Bottom of the page? U: Top S: Please select page design from the templates (*System shows drop down menu*) U: *User selects from menu* S: Enter header or footer content U: C.V. S: Task completed.

20 Dialog State Update Classification Start OVERRIDE g 1.1 U: I want to add page numbers and a title g 1.1 g 1.2 a U: add page X of Y ADD AND FORMAT PAGE NUMBERS t 1.. g 1.4 t 13 g INSERT A SECTION BREAK U: I want to add page numbers and a title U: add page X of Y a a Execute

21 Dialog State Update Classification APPEND p(s q,q,a s θ)= e i θ iφ i (s,q,a s,q) s e i θ iφ i (s,q,a s,q) Start OVERRIDE RESET U: I want to add page numbers and a title ADD AND FORMAT PAGE NUMBERS t 1 U: I want to add a section break INSERT A SECTION BREAK t 13 U: I want to add page numbers and a title S: Top of Page or Bottom of Page? g 1.1 g 1.2 a S: Please select page design from the templates. g 1.4 g S: Top or Bottom of the page? U: Top S: Please select page design from the templates (*System shows drop down menu*) U: I want to add a section break U: Top a a Execute

22 Dialog State Update Classification p(s q,q,a s θ)= e i θ iφ i (s,q,a s,q) s e i θ iφ i (s,q,a s,q) Start RESET OVERRIDE APPEND S: Top of Page or Bottom of Page? g 1.1 U: I want to add page numbers and a title g 1.1 g 1.2 a U: add page X of Y ADD AND FORMAT PAGE NUMBERS S: Please select page design from the templates t 1. S: Enter header or footer content U: I want to add a section. break g 1.4 a a U: Top U: Select U: CV t 13 g Execute INSERT A SECTION BREAK U: I want to add page numbers and a title S: Top or Bottom of the page? U: Top S: Please select page design from the templates (*System shows drop down menu*) U: *User selects from menu* S: Enter header or footer content U: C.V. S: Task completed.

23 Dialog State Turn Simulation Data: 1.5K Simulated Dialog Turns Previous state s to new query q pairs For every node/state n: sample q ~ (q n 1 q mn ) get system action a s sample a new node n using prior sample q ~ (q n 1 q n m ) Update Prior Reset Override Append

24 Results: Understanding Subsequent Dialog Turns Lexical features: query n-grams + stems + no stop words State features: query q overlap with QOverlap + POverlap: prev. query and system prompt SQOverlap: queries from prev. state s S QOverlap: queries from new. state s Feature Types Topic Goal Action Append Override Lexical (L) L + QOverlap + POverlap L + S QOverlap L + QOverlap + SQOverlap L + SQOverlap + S QOverlap

25 Key Steps I. Learning to convert the textual knowledge into procedural knowledge þ Building Dialog Trees from Instructions II. Learning a dialog manager using classificationbased approach þ Understanding Initial Dialog Turns þ Understanding Subsequent Dialog Turns Does it really work?

26 Small Scale User Study Select 8 goals, 48 interactions total Ask 6 users to rate the system on a scale from 1 to 5 User Satisfaction Av. Score ± Std. Dev. Overall satisfaction 3.42 ± 0.49 Questions were relevant 4.17 ± 0.41 System managed ambiguity 4.00 ± 0.89 Objective Measures Rates User achieved task 83% System asked irrelevant questions 19% System asked redundant questions 18%

27 User Study Error Analysis Missed key information in the input and asked redundant questions: U: I want the background color to be yellow S: Choose a color for the document U: Yellow S: Task completed Failed to correctly identify the intent and asked an irrelevant question: U: I want to change the font face and size S: Choose the default font U: No, reduce the size of the text S: *System Restarts*

28 Conclusions Construct dialog systems with light supervision automatically given only textual resources: instructional text search query click logs Future Work: Scale to new domains! Increase the complexity of the dialog manager Explore the range of possible textual knowledge sources that could be incorporated Enable end users to author new goals by writing procedural instructions in NL

29 Thank you! Contacts: Svitlana Volkova Pallavi Choudhury Chris Quirk Bill Dolan Luke Zettlemoyer

30 Office Help Pages (Topics) and Queries t 1 t n and q 1 q m Goals and Instructions q 1... q k and i 1... I l Dialog trees f 1... f k Query Paraphrases Queries Topics Goals Instructions Dialog Trees help with page numbers in the header how to number some pages adding page numbers how do you add numbering q 1 q m t 1 g 1 g k i 1 i l i 1 f 1 i l how to set a font size to stick in word default font how to set the font how do you change default font and font size in office? q 1 q m t n g 1 g k i 1 i l i 1 i l f n

31 Dialog State Update: From Topic Node q: how do you number pages q : add a background Start q : set a page number in a footer q : insert a comment.. t 1 t 2.. t 21.. g 1.1 g 1.2 g 21.1 g 21.3 g 1.4 q : insert numbers in margins q : add a comment redo a Execute a a a Execute RESET APPEND

32 Dialog State Update: From Goal Node Start q : enter text and page numbers q : add a border.. t 1 t 3.. t 6.. q: inserting page x of y g 1.1 g 1.3 g 1.4 q : setting a default font g 6.1 g 6.4 q : page x of y in footer a q : set default font and style a a a Execute RESET OVERRIDE APPEND Execute

33 Dialog State Update: From Action Node Start t 1 t 13.. q : make a degree symbol.. t 21.. q : insert page numbers g 1.1 g 1.2 g 21.1 g 21.2 g 1.4 q : comment in q: headers a document page number q : page number design a q : changing initials in a comment a a a a RESET OVERRIDE APPEND Execute Execute