An Automatic Method for Summary Evaluation Using Multiple Evaluation Results by a Manual Method

Transcription

1 An Autoatic Method for Suary Evaluation Using Multiple Evaluation Results by a Manual Method Hidetsugu Nanba Faculty of Inforation Sciences, Hiroshia City University Ozuka, Hiroshia, Japan nanba@its.hiroshia-cu.ac.jp Manabu Okuura Precision and Intelligence Laboratory, Tokyo Institute of Technology 4259 Nagatsuta, Yokohaa, Japan oku@pi.titech.ac.jp Abstract To solve a proble of how to evaluate coputer-produced suaries, a nuber of autoatic and anual ethods have been proposed. Manual ethods evaluate suaries correctly, because huans evaluate the, but are costly. On the other hand, autoatic ethods, which use evaluation tools or progras, are low cost, although these ethods cannot evaluate suaries as accurately as anual ethods. In this paper, we investigate an autoatic evaluation ethod that can reduce the errors of traditional autoatic ethods by using several evaluation results obtained anually. We conducted soe experients using the data of the Text Suarization Challenge 2 (TSC-2). A coparison with conventional autoatic ethods shows that our ethod outperfors other ethods usually used. 1 Introduction Recently, the evaluation of coputer-produced suaries has becoe recognized as one of the proble areas that ust be addressed in the field of autoatic suarization. To solve this proble, a nuber of autoatic (Donaway et al., 2000, Hirao et al., 2005, Lin et al., 2003, Lin, 2004, Hori et al., 2003) and anual ethods (Nenkova et al., 2004, Teufel et al., 2004) have been proposed. Manual ethods evaluate suaries correctly, because huans evaluate the, but are costly. On the other hand, autoatic ethods, which use evaluation tools or progras, are low cost, although these ethods cannot evaluate suaries as accurately as anual ethods. In this paper, we investigate an autoatic ethod that can reduce the errors of traditional autoatic ethods by using several evaluation results obtained anually. Unlike other autoatic ethods, our ethod estiates anual evaluation scores. Therefore, our ethod akes it possible to copare a new syste with other systes that have been evaluated anually. There are two research studies related to our work (Kazawa et al., 2003, Yasuda et al., 2003). Kazawa et al. (2003) proposed an autoatic evaluation ethod using ultiple evaluation results fro a anual ethod. In the field of achine translation, Yasuda et al. (2003) proposed an autoatic ethod that gives an evaluation result of a translation syste as a score for the Test of English for International Counication (TOEIC). Although the effectiveness of both ethods was confired experientally, further discussion of four points, which we describe in Section 3, is necessary for a ore accurate suary evaluation. In this paper, we address three of these points based on Kazawa s and Yasuda s ethods. We also investigate whether these ethods can outperfor other autoatic ethods. The reainder of this paper is organized as follows. Section 2 describes related work. Section 3 describes our ethod. To investigate the effectiveness of our ethod, we conducted soe exainations and Section 4 reports on these. We present soe conclusions in Section 5. 2 Related Work Generally, siilar suaries are considered to obtain siilar evaluation results. If there is a set of suaries (pooled suaries) produced fro a docuent (or ultiple docuents) and if these are evaluated anually, then we can estiate a anual evaluation score for any suary to be evaluated with the evaluation results for those pooled suaries. Based on this idea, Kazawa et 603 Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages , Sydney, July c 2006 Association for Coputational Linguistics

2 al. (2003) proposed an autoatic ethod using ultiple evaluation results fro a anual ethod. First, n suaries for each docuent,, were prepared. A suarization syste generated suaries fro docuents. Here, we represent the i th suary for the j th docuent and its evaluation score as x and y, respectively. The syste was evaluated using Equation 1. n scr( x) = w y Si( x, x ) + b (1) i= 1 j= 1 j The evaluation score of suary x was obtained by suing paraeter b for all the subscores calculated for each pooled suary, x. A subscore was obtained by ultiplying a paraeter w j, by the evaluation score y, and the siilarity between x and x. In the field of achine translation, there is another related study. Yasuda et al. (2003) proposed an autoatic ethod that gives an evaluation result of a translation syste as a score for TOEIC. They prepared 29 huan subjects, whose TOEIC scores were fro 300s to 800s, and asked the to translate 23 Japanese conversations into English. They also generated translations using a syste for each conversation. Then, they evaluated both translations using an autoatic ethod, and obtained W H, which indicated the ratio of syste translations that were superior to huan translations. Yasuda et al. calculated W H for each subject and plotted the values along with their corresponding TOEIC scores to produce a regression line. Finally, they defined a point where the regression line crossed W H = 0.5 to provide the TOEIC score for the syste. Though, the effectiveness of Kazawa s and Yasuda s ethods were confired experientally, further discussions of four points, which we describe in the next section, are necessary for a ore accurate suary evaluation. 3 Investigation of an Autoatic Method using Multiple Manual Evaluation Results 3.1 Overview of Our Evaluation Method and Essential Points to be Discussed We investigate an autoatic ethod using ultiple evaluation results by a anual ethod based on Kazawa s and Yasuda s ethod. The procedure of our evaluation ethod is shown as follows; (Step 1) Prepare suaries and their evaluation results by a anual ethod (Step 2) Calculate the siilarities between a suary to be evaluated and the pooled suaries (Step 3) Cobine anual scores of pooled suaries in proportion to their siilarities to the suary to be evaluated For each step, we need to discuss the following points. (Step 1) 1. How any suaries, and what type (variety) of suaries should be prepared? Kazawa et al. prepared 6 suaries for each docuent, and Yasuda et al. prepared 29 translations for each conversation. However, they did not exaine about the nuber and the type of pooled suaries required to the evaluation. (Step 2) 2. Which easure is better for calculating the siilarities between a suary to be evaluated and the pooled suaries? Kazawa et al. used Equation 2 to calculate siilarities. Si( x, x where ) x in( x x = (2), x ) x x indicates the nuber of discourse units 1 that appear in both x and x, and x represents the nuber of words in x. However, there are any other easures that can be used to calculate the topical siilarities between two docuents (or passages). As well as Yasuda s ethod does, using W H is another way to calculate siilarities between a suary to be evaluated and pooled suaries indirectly. Yasuda et al. (2003) tested DP atching (Su et al., 1992), BLEU (Papineni et al., 2002), and NIST 2, for the calculation of W H. However there are any other easures for suary evaluation. 1 Rhetorical Structure Theory Discourse Treebank. Id=LDC2002T07 Linguistic Data Consortiu

3 3. How any suaries should be used to calculate the score of a suary to be evaluated? Kazawa et al. used all the pooled suaries but this does not ensure the best perforance of their evaluation ethod. (Step 3) 4. How to cobine the anual scores of the pooled suaries? Kazawa et al. calculated the score of a suary as a weighted linear su of the anual scores. Applying regression analysis (Yasuda et al., 2003) is another ethod of cobining several anual scores. 3.2 Three Points Addressed in Our Study We address the second, third and fourth points in Section 3.1. (Point 2) A easure for calculating siilarities between a suary to be evaluated and pooled suaries: There are any easures that can calculate the topical siilarities between two docuents (or passages). We tested several easures, such as ROUGE (Lin, 2004) and the cosine distance. We describe these easures in detail in Section 4.2. (Point 3) The nuber of suaries used to calculate the score of a suary to be evaluated: We use suaries whose siilarities to a suary to be evaluated are higher than a threshold value. (Point 4) Cobination of anual scores: We used both Kazawa s and Yasuda s ethods. 4 Experients 4.1 Experiental Methods To investigate the three points described in Section 3.2, we conducted the following four experients. Exp-1: We exained Points 2 and 3 based on Kazawa s ethod. We tested threshold values fro 0 to 1 at intervals. We also tested several siilarity easures, such as cosine distance and 11 kinds of ROUGE. Exp-2: In order to investigate whether the evaluation based on Kazawa s ethod can outperfor other autoatic ethods, we copared the evaluation with other autoatic ethods. In this experient, we used the siilarity easure, which obtain the best perforance in Exp-1. Exp-3: We also exained Point 2 based on Yasuda s ethod. As a siilarity easure, we tested cosine distance and 11 kinds of ROUGE. Then, we exained Point 4 by coparing the result of Yasuda s ethod with that of Kazawa s. Exp-4: In the sae way as Exp-2, we copared the evaluation with other autoatic ethods, which we describe in the next section, to investigate whether the evaluation based on Yasuda s ethod can outperfor other autoatic ethods. 4.2 Autoatic Evaluation Methods Used in the Experients In the following, we show the autoatic evaluation ethods used in our experients. Content-based evaluation (Donaway et al., 2000) This easure evaluates suaries by coparing their content words with those of the huanproduced extracts. The score of the contentbased easure is obtained by coputing the siilarity between the ter vector using tf*idf weighting of a coputer-produced suary and the ter vector of a huan-produced suary by cosine distance. ROUGE-N (Lin, 2004) This easure copares n-gras of two suaries, and counts the nuber of atches. The easure is defined by Equation 3. Count atch ( gran ) S R gran S ROUGE N = (3) Count( gra ) S R gran S where Count(gra N ) is the nuber of an N-gra and Count atch (gra N ) denotes the nuber of n- gra co-occurrences in two suaries. ROUGE-L (Lin, 2004) This easure evaluates suaries by longest coon subsequence (LCS) defined by Equation 4. u LCS ( ri, C) i= i ROUGE L = (4) where LCS U (r i,c) is the LCS score of the union s longest coon subsequence between reference sentences r i and the suary to be evaluated, and is the nuber of words contained in a reference suary. N 605

4 ROUGE-S (Lin, 2004) Skip-bigra is any pair of words in their sentence order, allowing for arbitrary gaps. ROUGE-S easures the overlap of skip-bigras in a candidate suary and a reference suary. Several variations of ROUGE-S are possible by liiting the axiu skip distance between the two in-order words that are allowed to for a skip-bigra. In the following, ROUGE-SN denotes ROUGE-S with axiu skip distance N. ROUGE-SU (Lin, 2004) This easure is an extension of ROUGE-S; it adds a unigra as a counting unit. In the following, ROUGE-SUN denotes ROUGE-SU with axiu skip distance N. 4.3 Evaluation Methods In the following, we elaborate on the evaluation ethods for each experient. Exp-1: An experient for Points 2 and 3 based on Kazawa s ethod We evaluated Kazawa s ethod fro the viewpoint of Gap. Differing fro other autoatic ethods, the ethod uses ultiple anual evaluation results and estiates the anual scores of the suaries to be evaluated or the suarization systes. We therefore evaluated the autoatic ethods using Gap, which anually indicates the difference between the scores fro a anual ethod and each autoatic ethod that estiates the scores. First, an arbitrary suary is selected fro the 10 suaries in a dataset, which we describe in Section 4.4, and an evaluation score is calculated by Kazawa s ethod using the other nine suaries. The score is copared with a anual score of the suary by Gap, which is defined by Equation 5. n scr'( xkl ) ykl k = 1 l= 1 Gap = (5) n where x kl is the k th syste s l th suary, and y kl is the score fro a anual evaluation ethod for the k th syste s l th suary. To distinguish our evaluation function fro Kazawa s, we denote it as scr (x). As a siilarity easure in scr (x), we tested ROUGE and the cosine distance. We also tested the coverage of the autoatic ethod. The ethod cannot calculate scores if there are no siilar suaries above a given threshold value. Therefore, we checked the coverage of the ethod, which is defined by Equation 6. The nuber of suaries evaluated by the ethod Coverage = (6) The nuber of given suaries Exp-2: Coparison of Kazawa s ethod with other autoatic ethods Traditionally, autoatic ethods have been evaluated by Ranking. This eans that suarization systes are ranked based on the results of the autoatic and anual ethods. Then, the effectiveness of the autoatic ethod is evaluated by the nuber of atches between both rankings using Spearan s rank correlation coefficient and Pearson s rank correlation coefficient (Lin et al., 2003, Lin, 2004, Hirao et al., 2005). However, we did not use both correlation coefficients, because evaluation scores are not always calculated by a Kazawabased ethod, which we described in Exp-1. Therefore, we ranked the suaries instead of the suarization systes. Two arbitrary suaries fro the 10 suaries in a dataset were selected and ranked by Kazawa s ethod. Then, Kazawa s ethod was evaluated using Precision, which calculates the percentage of cases where the order of the anual ethod of the two suaries atches the order of their ranks calculated by Kazawa s ethod. The two suaries were also ranked by ROUGE and by cosine distance, and both Precision values were calculated. Finally, the Precision value of Kazawa s ethod was copared with those of ROUGE and cosine distance. Exp-3: An experient for Point 2 based on Yasuda s ethod An arbitrary syste was selected fro the 10 systes, and Yasuda s ethod estiated its anual score fro the other nine systes. Yasuda s ethod was evaluated by Gap, which is defined by Equation 7. s( xk ) yk k= Gap = 1 (7) where x k is the k th syste, s(x k ) is a score of x k by Yasuda s ethod, and y k is the anual score for the k th syste. Yasuda et al. (2003) tested DP atching (Su et al., 1992), BLEU (Papineni et al., 2002), and NIST 3, as autoatic ethods used in their evaluation. Instead of those ethods, we

5 tested ROUGE and cosine distance, both of which have been used for suary evaluation. If a score by Yasuda s ethod exceeds the range of the anual score, the score is odified to be within the range. In our experients, we used evaluation by revision (Fukushia et al., 2002) as the anual evaluation ethod. The range of the score of this ethod is between zero and 0.5. If the score is less than zero, it is changed to zero and if greater than 0.5 it is changed to 0.5. Exp-4: Coparison of Yasuda s ethod and other autoatic ethods In the sae way as for the evaluation of Kazawa s ethod in Exp-2, we evaluated Yasuda s ethod by Precision. Two arbitrary suaries fro the 10 suaries in a dataset were selected, and ranked by Yasuda s ethod. Then, Yasuda s ethod was evaluated using Precision. Two suaries were also ranked by ROUGE and by cosine distance and both Precision values were calculated. Finally, the Precision value of Yasuda s ethod was copared with those of ROUGE and cosine distance. 4.4 The Data Used in Our Experients We used the TSC-2 data (Fukushia et al., 2002) in our exainations. The data consisted of huan-produced extracts (denoted as PART ), huan-produced abstracts (denoted as FREE ), coputer-produced suaries (eight systes and a baseline syste using the lead ethod (denoted as LEAD )) 4, and their evaluation results by two anual ethods. All the suaries were derived fro 30 newspaper articles, written in Japanese, and were extracted fro the Mainichi newspaper database for the years 1998 and Two tasks were conducted in TSC-2, and we used the data fro a single docuent suarization task. In this task, participants were asked to produce suaries in plain text in the ratios of 20% and 40%. Suaries were evaluated using a ranking evaluation ethod and the revision ethod evaluation. In our experients, we used the results of evaluation fro the revision ethod. This ethod evaluates suaries by easuring the degree to which coputer-produced suaries are revised. The judges read the 4 In Exp-2 and 4, we evaluated PART, LEAD, and eight systes (candidate suaries) by autoatic ethods using FREE as the reference suaries. original texts and revised the coputer-produced suaries in ters of their content and readability. The huan revisions were ade with only three editing operations (insertion, deletion, replaceent). The degree of the huan revision, called the edit distance, is coputed fro the nuber of revised characters divided by the nuber of characters in the original suary. If the suary s quality was so low that a revision of ore than half of the original suary was required, the judges stopped the revision and a score of 0.5 was given. The effectiveness of evaluation by the revision ethod was confired in our previous work (Nanba et al., 2004). We copared evaluation by revision with ranking evaluation. We also tested other autoatic ethods: content-based evaluation, BLEU (Papineni et al., 2001) and ROUGE-1 (Lin, 2004), and copared their results with that of evaluation by revision as reference. As a result, we found that evaluation by revision is effective for recognizing slight differences between coputer-produced suaries. 4.5 Experiental Results and Discussion Exp-1: An experient for Points 2 and 3 based on Kazawa s ethod To address Points 2 and 3, we evaluated suaries by the ethod based on Kazawa s ethod using 12 easures, described in Section 4.4, as easures to calculate topical siilarities between suaries, and copared these easures by Gap. The experiental results for suarization ratios of 40% and 20% are shown in Tables 1 and 2, respectively. Tables show the Gap values of 12 easures for each Coverage value fro 0.2 to 1.0 at 0.1 intervals. Average values of Gap for each easure are also shown in these tables. As can be seen fro Tables 1 and 2, the larger the threshold value, the saller the value of Gap. Fro the result, we can conclude for Point 3 that ore accurate evaluation is possible when we use siilar pooled suaries (Point 2). However, the nuber of suaries that can be evaluated by this ethod was liited when the threshold value was large. Of the 12 easures, unigra-based ethods, such as cosine distance and ROUGE-1, produced good results. However, there were no significant differences between easures except for when ROUGE-L was used. 607

6 Coverage Measure Table 1 Coparison of Gap values for several easures (ratio: 40%) Average R R R R R-L R-S R-S R-S R-SU R-SU R-SU Cosine Threshold Sall Large Coverage Measure Table 2 Coparison of Gap values for several easures (ratio: 20%) Average R R R R R-L R-S R-S R-S R-SU R-SU R-SU Cosine Threshold Sall Large Exp-2: Coparison of Kazawa s ethod with other autoatic ethods (Point 2) In Exp-1, cosine distance outperfored the other 11 easures. We therefore used cosine distance in Kazawa s ethod in Exp-2. We ranked suaries by Kazawa s ethod, ROUGE and cosine distance, calculated using Precision. The results of the evaluation by Precision for suarization ratios of 40% and 20% are shown in Figures 1 and 2, respectively. We plotted the Precision value of Kazawa s ethod by changing the threshold value fro 0 to 1 at 0.05 intervals. We also plotted the Precision values of ROUGE- 2 as dotted lines. ROUGE-2 was superior to the other 11 easures in ters of Ranking. The X and Y axes in Figures 1 and 2 show the threshold value of Kazawa s ethod and the Precision values, respectively. Fro the result shown in Figure 1, we found that Kazawa s ethod outperfored ROUGE-2, when the threshold value was greater than The Coverage value of this point was In Figure 2, the Precision curve of Kazawa s ethod crossed the dotted line at a threshold value of The Coverage value of this point was To iprove these Coverage values, we need to prepare ore suaries and their anual evaluation results, because the Coverage is critically dependent on the nuber and variety of pooled suaries. This is exactly the first point in Section 3.1, which we do not address in this paper. We will investigate this point as the next step in our future work. 608

7 precision threshold value Kazawa's ethod R-2 Figure 1 Coparison of Kazawa s ethod and ROUGE-2 (ratio: 40%) 1 Table 3 Gap between the anual ethod and Yasuda s ethod Ratio Average 20% 40% Cosine R R R R R-L R-S( ) R-S R-S R-SU( ) R-SU R-SU precision threshold value Exp-4: Coparison of Yasuda s ethod with other autoatic ethods We also evaluated Yasuda s ethod by coparison with other autoatic ethods in ters of Ranking. We evaluated 10 systes by Yasuda s ethod with ROUGE-3, which produced the best results in Exp-3. We also evaluated the systes by ROUGE and cosine distance, and copared the results. The results are shown in Table 4. Kazawa's ethod R-2 Figure 2 Coparison of Kazawa s ethod and ROUGE-2 (ratio: 20%) Exp-3: An experient for Point 3 based on Yasuda s ethod For Point 2 in Section 3.2, we also exained Yasuda s ethod. The experiental result by Gap is shown in Table 3. When the ratio is 20%, ROUGE-SU4 is the best. The N-gra and the skip-bigra are both useful when the suarization ratio is low. For Point 4, we copared the result by Yasuda s ethod (Table 3) with that of Kazawa s ethod (in Tables 1 and 2). Yasuda s ethod could accurately estiate anual scores. In particular, the Gap values of by ROUGE-2 and by ROUGE-3 are saller than those produced by Kazawa s ethod with a threshold value of 0.9 (Tables 1 and 2). This indicates that regression analysis used in Yasuda s ethod is superior to that used in Kazawa s ethod. Table 4 Coparison between Yasuda s ethod and autoatic ethods Ratio Average 20% 40% Yasuda Cosine R R R R R-L R-S( ) R-S R-S R-SU( ) R-SU R-SU As can be seen fro Table 4, Yasuda s ethod produced the best results for the ratios of 20% and 40%. Of the autoatic ethods copared, ROUGE-4 was the best. 609

8 As evaluation scores by Yasuda s ethod were calculated based on ROUGE-3, there were no striking differences between Yasuda s ethod and the others except for the integration process of evaluation scores for each suary. Yasuda s ethod uses a regression analysis, whereas the other ethods average the scores for each suary. Yasuda s ethod using ROUGE-3 outperfored the original ROUGE-3 for both ratios, 20% and 40%. 5 Conclusions We have investigated an autoatic ethod that uses several evaluation results fro a anual ethod based on Kazawa s and Yasuda s ethods. Fro the experiental results based on Kazawa s ethod, we found that liiting the nuber of pooled suaries could produce better results than using all the pooled suaries. However, the nuber of suaries that can be evaluated by this ethod was liited. To iprove the Coverage of Kazawa s ethod, ore suaries and their evaluation results are required, because the Coverage is critically dependent on the nuber and variety of pooled suaries. We also investigated an autoatic ethod based on Yasuda s ethod and found that the ethod using ROUGE-2 and -3 could accurately estiate anual scores, and could outperfor Kazawa s ethod and the other autoatic ethods tested. Fro these results, we can conclude that the autoatic ethod perfored the best when ROUGE-2 or 3 is used as a siilarity easure, and a regression analysis is used for cobining anual ethod. References Robert L. Donaway, Kevin W. Druey and Laura A. Mather A Coparison of Rankings Produced by Suarization Evaluation Measures. Proceedings of the ANLP/NAACL 2000 Workshop on Autoatic Suarization: Takahiro Fukushia and Manabu Okuura Text Suarization Challenge/Text Suarization Evaluation at NTCIR Workshop2. Proceedings of the Second NTCIR Workshop on Research in Chinese and Japanese Text Retrieval and Text Suarization: Takahiro Fukushia, Manabu Okuura and Hidetsugu Nanba Text Suarization Challenge 2/Text Suarization Evaluation at NTCIR Workshop3. Working Notes of the 3rd NTCIR Workshop Meeting, PART V: 1 7. Tsutou Hirao, Manabu Okuura, and Hideki Isozaki Kernel-based Approach for Autoatic Evaluation of Natural Language Generation Technologies: Application to Autoatic Suarization. Proceedings of HLT- EMNLP 2005: Chiori Hori, Takaaki Hori, and Sadaoki Furui Evaluation Methods for Autoatic Speech Suarization. Proceedings of Eurospeech 2003: Hideto Kazawa, Thoas Arrigan, Tsutou Hirao and Eisaku Maeda An Autoatic Evaluation Method of Machine-Generated Extracts. IPSJ SIG Technical Reports, 2003-NL-158: (in Japanese). Chin-Yew Lin and Eduard Hovy Autoatic Evaluation of Suaries Using N-gra Co- Occurrence Statistics. Proceedings of the Huan Language Technology Conference 2003: Chin-Yew Lin ROUGE: A Package for Autoatic Evaluation of Suaries. Proceedings of the ACL-04 Workshop Text Suarization Branches Out : Hidetsugu Nanba and Manabu Okuura Coparison of Soe Autoatic and Manual Methods for Suary Evaluation Based on the Text Suarization Challenge 2. Proceedings of the Fourth International Conference on Language Resources and Evaluation: Ani Nenkova and Rebecca Passonneau, Evaluating Content Selection in Suarization: The Pyraid Method. Proceedings of HLT-NAACL 2004: Kishore Papineni, Sali Roukos, Todd Ward and Wei-Jing Zhu BLEU: A Method for Autoatic Evaluation of Machine Translation. IBM Research Report, RC22176 (W ). Keh-Yih Su, Ming-Wen Wu, and Jing-Shin Chang A New Quantitative Quality Measure for Machine Translation Systes. Proceedings of the 14 th International Conference on Coputational Linguistics: Sione Teufel and Hans van Halteren Evaluating Inforation Content by Factoid Analysis: Huan Annotation and Stability. Proceedings of EMNLP 2004: Kenji Yasuda, Fuiaki Sugaya, Toshiyuki Takezawa, Seiichi Yaaoto and Masuzo Yanagida Applications of Autoatic Evaluation Methods to Measuring a Capability of Speech Translation Syste. Proceedings of the Tenth Conference of the European Chapter of the Association for Coputational Linguistics: