An Ensemble Dialogue System for Facts-Based Sentence Generation

Transcription

1 Track2 Oral Session : Sentence Generation An Ensemble Dialogue System for Facts-Based Sentence Generation Ryota Tanaka, Akihide Ozeki, Shugo Kato, Akinobu Lee Graduate School of Engineering, Nagoya Institute of Technology, Japan

2 Introduction l Neural networks-based dialogue model has several problems.! Not informative response such as I don t know! Inconsistent response with real world facts l Combining multiple facts-based models allows the response to be more informative and diverse. 1

3 Pre-processing of Facts 1. Categorize facts into 2 types based on HTML tags! Subject Facts! "#$% : enclosed by,! Description Facts! &'"( : enclosed by not enclosed by any tags 2. Select facts highly related to the context Subject Facts! "#$% Facts 1 <title> Justice League </title> 2 From Wikipedia, the free encyclopedia 3 For other uses, see Justice League 4 <h1> Story </h1> N <p> The film was </p> 1 Justice League 2 Story. K Cast Description Facts! &'"( 1 From Wikipedia, the free encyclopedia 2 For other uses, see Justice League. L The film was 2

4 Pre-processing of Facts 1. Categorize facts into 2 types based on HTML tags 2. Select facts highly related to the context! Select 10 facts using the cosine similarity of the word2vec between facts and the context Context Justice League filming wraps in London. 1 Justice League cosine similarity 2 Superhero. select K Spiderman 1 Justice League is one of the most 2 Justice League is a 2017American superhero L The film was. * This study uses K=L=10 3

5 Ensemble Dialogue System l Dialogue system combining 3 proposed modules! Select the final response by feeding all the candidates Reranker Select Response Retrieve Generate FR DB Facts Subject Facts Description Facts MHRED Dialogue Data Memory-augmented HRED (MHRED) Sentence selection module with Facts Retrieval (FR) Reranker 4

6 Ensemble Dialogue System l Dialogue system combining 3 proposed modules! Select the final response by feeding all the candidates Reranker Select Response Retrieve Generate FR DB Facts Subject Facts Description Facts MHRED Dialogue Data Memory-augmented HRED (MHRED) Sentence selection module with Facts Retrieval (FR) Reranker 5

7 Memory-augmented HRED (MHRED) l Generate a response conditioned on a previous context and facts MHRED : HRED [Serban et al., 15] + MemN2N [Sukhbaatar et al.,15 ] 6

8 Memory-augmented HRED (MHRED) l Generate a response conditioned on a previous context and facts Facts Encoder Decoder Hierarchical Recurrent Encoder (HRE) 7

9 Facts Encoder l Select facts to be injected in responses l Map facts to a continuous representation paragraph sub-header header title final hidden state of HRE 8

10 Ensemble Dialogue System l Dialogue system combining 3 proposed modules! Select the final response by feeding all the candidates Reranker Select Response Retrieve Generate FR DB Facts Subject Facts Description Facts MHRED Dialogue Data Memory-augmented HRED (MHRED) Sentence Selection Module with Facts Retrieval (FR) Reranker 9

11 Sentence selection with facts Retrieval (FR) 1. Construct DB 2. Response Selection Justice League filming wraps in London. I think it is better than Spiderman. Context Response DB < Query, Response > 10

12 Sentence selection with Facts Retrieval (FR) 1. Construct DB 2. Response Selection I think it is better than Spiderman. I love Marvel movie. Justice League is directed by Snyder. Subject Facts 1 Justice League 2 Superhero. output up to 10 responses Hit!! K Spiderman duplicate words search entries Spiderman Marvel Justice League 11

13 Ensemble Dialogue System l Dialogue system combining 3 proposed modules! Select the final response by feeding all the candidates Reranker Select Response Retrieve Generate FR DB Facts Subject Facts Description Facts MHRED Dialogue Data Memory-augmented HRED (MHRED) Sentence selection module with Facts Retrieval (FR) Reranker 12

14 Reranker (1/2) l Reranker sorts candidates by feeding all the results of the MHRED and FR, and then selects the best response l Classify whether a candidate is positive or negative as a response using the XGBoost candidates 13

15 Reranker (2/2) l Dataset (created by ourselves using the distributed dataset)! Positive Examples (44449 pairs) Context Response pairs selected with the high response score in the dialogue dataset! Negative Examples (44449 pairs) Context Response pairs generated from the positive examples, changing sentence length, order or topic randomly l Features! Candidate : Length, Fluency, etc.! Last Utterance - Candidate pair : Word sim, N-gram sim, etc.! Context - Candidate pair : Topic sim 14

16 Experiments l DSTC7 Dataset! Dialogue Reddit (2011/ /11) Train : dialog, Dev : dialog, Test : dialog! Facts Articles extracted from web sites such as Wikipedia l Evaluation Metrics! Automatic NIST, BLEU, METEOR : word overlap metrics div [Li et al., 15] : diversity metric! Human (5-point Likert scale) Appropriateness : conversationally appropriate and relevant to the previous turns. Informativeness : informative response that is relevant to the user input and has potential utility. 15

17 (Submitted) Models for Comparison l S2S! Seq2seq [Vinalys et al, 15] l HRED! HRED [Serban et al, 16] l HRED-F l MHRED-F l MHRED-F5-R l MHRED-F15-R F: Facts Penalty l Ensemble! proposed model R: Reranker 5, 15: Beam width 16

18 Automatic Evaluation l Ensemble performs better than other models! Combining multiple systems is effective 17

19 Automatic Evaluation l Ensemble performs better than other models! Combining multiple systems is effective l MHRED-F performs on the diversity score notably! Generate diverse responses on the new domain 18

20 Automatic Evaluation lensemble performs better than other models! Combining multiple systems is effective l MHRED-F performs on the diversity score notably! Generate diverse responses on the new domain l Improve word-overlap scores by introducing the Reranker! Reranker tends to select natural responses close to human 19

21 Human Evaluation l Official Baseline! Baseline(constant): Only I don t know what you mean. Valid response to any context! baseline(random): Random sampling from the dialogue data. Response is usually high fluent because of human-made l Our model beats baseline models 20

22 Examples (MHRED) UserA : UserB : Model Response Ensemble (MHRED) 21

23 Examples (FR) UserA : UserB : Model Response Ensemble (FR) 22

24 Example of Reranking UserA: UserB : Model Response Rank MHRED MHRED FR 1 2 worst 23

25 Conclusion l Proposed an ensemble dialogue system with facts! Consists of the MHRED FR and Reranker! Generate more diverse and informative responses than a sole model l Future Work! Extend an end-to-end learning for multiple systems simultaneously 24

26 Thank You Special thanks to Track2 Organizers