Text Summarization using Neural Network Theory

Transcription

1 Iteratioal Joural of Computer Systems (ISSN: ), Volume 03 Issue 07, July, 206 Available at Simra Kaur Jolly, Wg Cdr Ail Chopra 2 Departmet of CSE, Ligayas Uiversity, Faridabad Haryaa, Idia 2 Assistat Professor, MRIU, Faridabad, Haryaa, Idia Abstract The idea of eural is based o the belief that workig of huma brai by makig the right coectios ca be copied usig silico ad wires as livig euros ad dedrites. The huma brai is composed of billio erve cells called euros. They are coected to each other cells by Axos. Stimuli from exteral eviromet or iputs from sesory orgas are accepted by dedrites. These iputs create electric impulses, which quickly travel through the eural etwork. A euro ca the sed the message to other euro to hadle the issue or does ot sed it forward. It is a ew techique for summarizig corpus of articles usig a eural etwork. A eural etwork is traied to lear the relevat characteristics of seteces that should be icluded i the summary of the article. The eural etwork is the modified to geeralize ad combie the relevat characteristics apparet i summary seteces. Fially, the modified eural etwork is used as a filter to summarize ews articles. Sice digitally stored iformatio is more ad more available, users eed suitable tools able to select, filter, ad extract oly relevat iformatio. Text summarizatio ca be categorized ito two approaches: extractio ad abstractio. This paper focuses o extractio approach. The goal of text summarizatio based o extractio approach is setece selectio. Oe of the methods to obtai the suitable seteces is to assig some umerical measure of a setece for the summary called setece weightig ad the select the best oes amog them. A summary text is a derivative of a source text codesed by selectio ad/or geeralizatio o importat cotet. Query-focused summaries eable users to fid more relevat documets more accurately, with less eed to cosult the full text of the documet. Extractive summarizatio methods try to fid out the most importat topics of a iput documet ad select seteces that are related to these chose cocepts to create the summary. This paper is a study of eural used for extractive summarizatio, amely, Neural Network. Keywords: Adaptive clusterig, feature fusio, eural etworks, pruig, text summarizatio. I. INTRODUCTION Earlier, a large amout of research papers ad books have bee digitally stored. However, the storage media to store such a large database was very expesive. Therefore the cocept of automatic summarizatio was itroduced to store the iformatio about papers ad books i limited storage space. But due to large amout of iformatio available o the web, there is a eed to represet each documet by its summary to save time ad effort for searchig the correct iformatio. I summary, the followig are the importat reasos i cotext of automatic text summarizatio: ) A summary or abstract saves time 2) A summary or a abstract facilitate documet selectio ad literature searches. 3) It improves documet idexig efficiecy 4) Machie geerated summary is free from bias 5) Customized summaries ca be useful i questio aswerig systems where they provide persoalized iformatio. 6) The use of automatic or semi-automatic summarizatio by commercial abstract services may allow them to scale the umber of published texts they ca evaluate. Automatic documet summarizatio is extremely helpful i tacklig the iformatio overload problems. It is the techique to idetify the most importat pieces of iformatio from the documet, omittig irrelevat iformatio ad miimizig details to geerate a compact coheret summary documet. There are differet types of summarizatio approaches depedig o what the summarizatio method focuses o to make the summary of the text [2]. ) Abstract vs. Extract summary - Abstractio is the process of paraphrasig sectios of the source documet whereas extractio is the process of pickig subset Ease of Use of seteces from the source documet ad presets them to user i form of summary that provides a overall sese of the documets cotet. 2) Geeric vs. Query-based summary - Geeric summary do ot target to ay particular group. It addresses broad commuity of readers while Query or topic focused queries are tailored to the specific eeds of a idividual or a particular group ad represet particular topic. 3) Sigle vs. Multi-documet summary - Sigle documet summary provide the most relevat iformatio cotaied i sigle documet to the user that helps the user i decidig whether the documet is related to topic of iterest. With the explosio of the World Wide Web ad the abudace of text material available o the Iteret, text summarizatio has become a importat ad timely tool for assistig ad iterpretig text iformatio. The Iteret provides more iformatio tha is usually eeded. Therefore, a twofold problem is ecoutered: searchig for relevat documets through a overwhelmig umber of documets available, ad absorbig a large quatity of relevat iformatio. Summarizatio is a useful tool for selectig relevat texts, ad for extractig the key poits of each text. Some articles such as academic papers have accompayig abstracts, which make them easier to decipher their key poits. However, ews articles have o such accompayig summaries, ad their titles are ofte 535 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

2 ot sufficiet to covey their importat poits. Therefore, a summarizatio tool for ews articles would be extremely useful, sice for give ews topic or evet, there are a large umber of available articles from the various ews agecies ad ewspapers. Because ews articles have a highly structured documet form, importat ideas ca be obtaied from the text simply by selectig seteces based o their attributes ad locatios i the article. We propose a machie learig approach that uses artificial eural etworks to produce summaries of arbitrary legth ews articles. A eural etwork is traied o a corpus of articles. The eural etwork is the modified, through feature fusio, to produce a summary of highly raked seteces of the article. Through feature fusio, the etwork discovers the importace (ad uimportace) of various features used to determie the summary-worthiess of each setece. A. Restricted Boltzma Machie: is a stochastic eural etwork (that is a etwork of euros where each euro has some radom behavior whe activated). It cosist of oe layer of visible uits (euros) ad oe layer of hidde uits. Uits i each layer have o coectios betwee them ad are coected to all other uits i other layer (Fig. ). Coectios betwee euros are bidirectioal ad symmetric. This meas that iformatio flows i both directios durig the traiig ad durig the usage of the etwork ad those weights are the same i both directios.maitaiig the Itegrity of the Specificatios II. Figure. Restricted Boltzma Machie TEXT SUMMARIZATION WITH NEURAL NETWORKS I this techique, firstly, each documet is first coverted ito seteces. Each setece is represeted as a vector [f, f2... f7], composed of 7 features. Cosiderig a machie learig approach, that uses artificial eural etworks to produce summaries of ews articles. News articles have a highly structured documet form; importat ideas ca be obtaied from the text simply by selectig seteces based o their attributes ad locatios i the article. A eural etwork is traied o a corpus of articles. The eural etwork is the modified, through feature fusio, to produce a summary of highly raked seteces of the article ie most frequetly occurrig seteces. Through feature fusio, the etwork discovers the relevace of various features used to determie the summary-worthiess of each setece [4]. Seve Features of a Documet ) f Paragraph follows title 2) f2 Paragraph locatio i documet 3) f3 Setece locatio i paragraph 4) f4 First setece i paragraph 5) f5 Setece legth 6) f6 Number of thematic words i the setece 7) f7 Number of title words i the setece Feature f5, setece legth, is useful for filterig out short seteces such as datelies ad author ames commoly foud ews articles. It is also aticipated that short seteces are ot to be icluded i summaries. Feature f6, the umber thematic words, idicates the umber of thematic words i the setece, relative to the maximum possible. It is obtaied as follows: from each documet, remove all prepositios, ad reduce the remaiig words to their morphological roots. Abbreviatios ad Acroyms The resultat cotet words i the documet are couted for occurrece. The top 0 most frequet cotet words are cosidered as thematic words. This feature determies the ratio of thematic words to cotet words i a setece. This feature is expected to be importat because terms that occur frequetly i a documet are probably related to its topic. Therefore, a high occurrece of thematic words i saliet seteces is expected. Fially, feature f7 idicates the umber of title words i the setece, relative to the maximum possible. It is obtaied by coutig the umber of matches betwee the cotet words i a setece, ad the words i the title. This value is the ormalized by the maximum umber of matches. This feature is expected to be importat because the importace of a setece may be affected by the umber of words i the setece also appearig i the title. These features may be chaged or ew features may be added. The selectio of features plays a importat role i determiig the type of seteces that will be selected as part of the summary ad, therefore, would ifluece the performace of the eural etwork. There are maily two differet phases i this system. ) Neural Network Traiig: The first phase of the process ivolves traiig the eural etworks to lear the types of seteces that should be icluded i the summary. This is accomplished by traiig the etwork with seteces i several test paragraphs where each setece is idetified as to whether it should be icluded i the summary or ot. This is doe by a huma reader. The eural etwork lears the patters iheret i seteces that should be icluded i the summary ad those that should ot be icluded. A three-layered feed forward eural etwork is used, which has bee prove to be a uiversal fuctio approximator. It ca discover the patters ad approximate the iheret fuctio of ay data to a accuracy of 00 percetages, as log as there are o cotradictios i the data set. The eural etwork cosists of seve iput-layer euros, six hidde-layer euros, ad oe output-layer euro. A cojugate gradiet method where the eergy fuctio is a combiatio of error fuctio ad a pealty fuctio is used. The goal of traiig is to search for the global miima of the eergy fuctio. The additio of the pealty fuctio drives the associated weights of uecessary coectios to very small values while stregtheig the rest of the coectios. Therefore, the uecessary coectios ad euros ca be prued without affectig the performace of the etwork. Oce the etwork has leared the features that must exist i summary seteces, there is a eed to 536 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

3 discover the treds ad relatioships amog the features that are iheret i the majority of seteces. This is accomplished by the feature fusio phase, which cosists of two steps: ) elimiatig ucommo features; ad 2) collapsig the effects of commo features. The coectios havig very small weights after traiig ca be prued without affectig the performace of the etwork. As a result, ay iput or hidde layer euro havig o emaatig coectios ca be safely removed from the etwork. I additio, ay hidde layer euro havig o abuttig coectios ca be removed. This correspods to elimiatig ucommo features from the etwork 2) Setece Selectio: Oce the etwork has bee traied, prued, ad geeralized, it ca be used as a tool to filter seteces i ay paragraph ad determie whether each setece should be icluded i the summary or ot. This phase is accomplished by providig cotrol parameters for the radius ad frequecy of hidde layer activatio clusters to select highly raked seteces. The setece rakig is directly proportioal to cluster frequecy ad iversely proportioal to cluster radius. Oly seteces that satisfy the required cluster boudary ad frequecy are selected as high-rakig summary seteces. Features of Each documet are coverted ito a list of seteces. Each setece is represeted as a vector [f, f2... f7], composed of 7 features. Seve Features of a Documet f f2 f3 f4 f5 f6 f7 Paragraph follows title Paragraph locatio i documet Setece locatio i paragraph First setece i paragraph Setece legth Number of thematic words i the setece Number of title words i the setece Features f to f4 represet the locatio of the setece withi the documet, or withi its paragraph. Feature f5, setece legth, is useful for filterig out short seteces such as datelies ad author ames commoly foud i ews articles. We also aticipate that short seteces are ulikely to be icluded i summaries [3]. Feature f6, the umber of thematic words, idicates the umber of thematic words i the setece, relative to the maximum possible. III. TEXT SUMMARIZATION PROCESS A. Proposed Deep Learig Approach Text summarizatio techique is divided ito two approaches extractive ad abstractive. But due to the limitatio of atural laguage geeratio techiques i geeratig the abstractive summary geerally extractive approach is used for summarizatio. For summarizig the text there is a eed of structurig the text ito certai model which ca be give to RBM as iput. First of all i text summarizatio the text documet is preprocessed usig various prevalet preprocessig techiques ad the it is coverted ito setece matrix defied over a vocabulary of words. This structured matrix each row will work as a iput to our RBM (Fig. 2). After gettig the set of top priority word from the RBM the iput query, setece vector ad high priority word output is compared to geerate the extractive summary of the text documet. B. Preprocessig To make the documet light (ot cotaiig uwated words) preprocessig of the text documet for structurig is doe by applyig various techiques developed by the liguist. There are myriads of techique by which we ca reduce the desity of text documet. I this study we are usig the followig techiques. C. Part of Speech Taggig Figure 2. block diagram Part of speech taggig is the process of markig or classifyig the words of text o the basis of part of speech category (ou, verbs, adverb, adjectives) they belog. Varieties of algorithms are there to perform the POS 537 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

4 taggig like hidde Markova models, usig dyamic programmig. D. Stop Word Filterig Stop words are the words which are filtered out prior to or after the preprocessig task geerally there is o specific rule o aparticular word to be stop word, it is completely subjective depeds upo the situatio. I our coditio we cosiderig words like a, a, i by as stop word ad filters this word from the origial documet. Stop word filterig is the stadard filterig i text miig applicatios. E. Stemmig Aother importat techique we eed to apply is stemmig. Stemmig is process of brigig the word to its base or root form for example usig words sigular form istead of usig the plural (usig boys as boy), removig the ig from verb (chagig doig to do). There are umber of algorithms, geerally referred as stemmers, are there that ca be used to perform the stemmig. F. Feature Vector Extractio After reducig the desity of documet is structured ito a matrix. A setece matrix S of order *v is cotaiig the features for every setece of a matrix. For very iformative summarizatio we are extractig four features of a setece of text documet viz similarity with title, relative positio of setece, term weight of words formig seteces, cocept-extractio of setece. Setece matrix row vector represets the setece which is makig the documet ad colum vector cotais the etry for these extracted features. G. Feature Computatio A setece is cosidered importat if it s similar to the title of text documet. Here similarity is cosidered o the basis of occurrece of commo words i title ad setece. A setece has good feature score if it has maximum umber of words commo to the title. The ratio of the umber of words i the setece that occur i title to the total umber of words i the title helps to calculate the score of a setece for this feature. It is calculated by: f s t t S = Set of words of setece T = Set of words of title st = Commo words i setece ad title of documet H. Positioal Feature Positioal value of a setece is also extracted. A setece is relevat or ot ca also be judged by its positio i the text. To calculate the positioal score of setece we are cosiderig the followig coditios: f2 =, if setece is the startig setece of the text f2 = 0, if setece comes i the middle paragraphs of Text f2 =, if setece comes i the last of the text I. Term Weight This is aother very importat feature to be cosider for summarizatio of text. Here by term weight we simply mea the term frequecy ad its importace. This is the most stadard feature cosidered i various atural laguage processig tasks. The frequecy here is the term frequecy which reflects the importace of a word i a documet, it simply tells umber of times a word appears i the text. The term frequecy of a word will be give by tf(f,d) where f is the frequecy of the word ad d is text the documet. The total term weight is calculated by computig tf(f,d) ad idf for a documet. Here idf refers to iverse documet frequecy which simply tells about whether the term is commo or rare across all documets. It is obtaied by dividig the total umber of documets by the umber of documets cotaiig the term ad the takig the log of that quotiet. The idf is give by: idf t, D log D d D : t d where, D is the total umber of documets, D: td, it is the umber of documets where term t appears. The total term weight is give by tf*idf which ca be calculated by: tf * idf (t,d,d) = tf (t,d )* Idf (t,d) f 3 =tf * idf. J. Cocept Feature The cocept feature from the text documet is extracted usig the mutual iformatio ad widowig process. I widowig process a virtual widow of size k is moved over documet from left to right. Here we wat to fid out the co-occurrece of words i same widow ad it ca be calculated by followig formula: P(w i, w j ) MI(w i, w j ) log 2 P(w i ) * P(w j ) where, P(w i, w j )- Joit probability that both keyword appeared together i a text widow. P(w i )- probability that a keyword w i appears i a text widow ad ca be computed by: f s t t U = Set of words of setece V = Set of words of title st = Commo words i setece ad title of documet P(w i ) sw t sw i = The umber of widows cotaiig the keyword w i sw 538 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

5 sw = Total umber of widows costructed from a text documet The setece matrix geerate by above steps is: S T K. Setece Matrix P Tw C S2 f f 2 f 3 f S Here setece matrix S = (s, s 2,..s ) where s i = (f, f 2,..f 4 ), i<= is the feature vector. L. Deep Learig Algorithm The setece matrix S = (s, s2,..s ) which is the feature vector set havig elemet as si which is set cotais the all the four features extracted for the setece si. Here this set of feature vectors S will be give as iput to deep architecture of RBM as visible layer. Some radom values is selected as bias Hi where i =,2 sice a RBM ca have at least two hidde layer. The whole process ca be give by followig equatio: S s,s 2...s where, s i = (f,f 2,..f 4 ), i<= where is the umber of seteces i the documet. Restricted Boltzma machie cotais two hidde layers ad for them two set of bias value is selected amely H 0 H : h 0, h, h H h h 0, h,h H 2...h These set of bias values are values which are radomly selected. The whole operatio of Setece matrix is performed with these two set of radomly selected value. The whole operatio with RBM starts with givig the setece matrix as iput. Here s,s 2,..s are give as iput to RBM. The RBM geerally have two hidde layers as we metioed above. Two layers are sufficiet for our kid of problem. To get the more refied set of setece features. RBM works i two step. The iput to first step is our set of setece matrix, S = (s,s 2,..s ), which is havig the four features of setece as elemet of each setece set. Durig the first cycle of RBM a ew refied setece matrix set: s ',...s s ',s '' 2 The above expressed s is geerated by performig: P(w i ) sw t /sw sw i = The umber of widows cotaiig the keyword w i sw = Total umber of widows costructed from a text documet s ',...s s ',s '' 2 The above expressed s is geerated by performig: s i h i Durig step 2 the same procedure will be applied to this obtaied refied set to get the more refied setece matrix set with H ad which is give by: s" s",s" 2,...s" After obtaiig the refied setece matrix from the RBM it is further tested o a particular radomly geerated threshold value for each feature we have calculated. For example we select threshold thr c as a threshold value for the extracted cocept-feature. If for ay setece f 4 <thr the it will be filtered ad will become member of ew set of feature vector. Step. s,s 2,s [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] ' ' ' Step 2. s,s 2,s ց ւ s i h i (H 0 ) ' ' ' s ' (s,s 2,s ) [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] ց ւ s i h i (H ) (s '' '' '' s '',s ),s 2 M. Optimal Feature Vector Set Geeratio I the first part we have obtaied a good set of feature vectors by Deep learig algorithm. I this phase we will fie tue the obtaied feature vector set by adjustig the weight of the uits of the RBM. To fie tue the feature 539 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

6 vector set optimally we use back propagatio algorithm. Back propagatio algorithm is well kow method to adjust the deep architecture to fid good optimum feature vector set for the precise cotextual summary of text. The deep learig algorithm i this phase uses cross-etropy error to fie tue the obtaied feature vector set. The cross-etropy error for adjustmet is calculated for every feature of the setece.for example term weight feature of the setece will be recostruct by usig followig formula: [ v f v log f v v ( f v )log( f v )] f v = The t f value of v th word f v^ = The t f value of recostructio I this way all three features will be optimized. N. Summary Geeratio I summary geeratio phase, the obtaied optimal feature vector set is used to geerate the extractive summary of the documet. For summary geeratio first task is obtaiig the setece score for each setece of documet. Setece score is obtaied by fidig the itersectio of user query with the setece. After this step rakig of the setece is performed ad the fial set of seteces for text summary geeratio defiig the summary is obtaied. O. Rakig of Setece This is the fial step to obtai the summary of text. Here rakig of the setece is performed o the basis of the setece score obtaied i previous step. The seteces are arraged i descedig order o the basis of the obtaied setece score. Out of these seteces top-n seteces are selected o the basis of compressio rate give by the user. To fid out umber of top seteces to select from the matrix we use followig formula based o the compressio rate. It is give by: N C N S 00 IV. RESULTS AND ANALYSIS We used 50 ews articles from the Iteret with various topics such as techology, sports, ad world ews to trai the etwork. Each article cosists of 6 to 89 seteces with a average of 29 seteces. The etire set cosists of,435 seteces. Every setece is labeled as either a summary setece or a with a average of 6 seteces per article. After the feature fusio phase, f (Paragraph follows title) ad f 4 (First setece i the paragraph) were removed as well as two hidde layer euros. The removal of f feature is uderstadable, sice most of the articles did ot have sub-titles or sectio headigs. Therefore, the oly paragraph followig a title would be the first paragraph, ad this iformatio is already cotaied i feature f 2 (paragraph locatio i documet). The removal of f 4 feature idicates that the first setece i the paragraph is ot always selected to be icluded i the summary. We the used the same 50 ews articles as a test set for the modified etwork. The accuracy of the modified etwork raged from 94% to 00% with a average accuracy of 96.4% whe compared to the summaries of the huma reader. That is, the etwork was able to select all seteces that were labeled as summary setece i most of the articles. However, the etwork missed oe to two seteces i six of the articles ad selected oe setece that was ot labeled as summary i each of five articles. The performace of the text summarizatio process depeds heavily o the style of the huma reader ad to what the huma reader deems to be importat to be icluded i the summary. A huma reader ad the modified etwork summarized 0 differet ews articles, idepedetly. The performace of the modified etwork was 96% accurate whe compared with the huma reader s summaries. For each article, 4 to 9 seteces were selected by the etwork as summary seteces with a average of 5 seteces. Both the huma reader ad the modified etwork summarized 7 of the articles exactly the same. For two of the articles, the modified etwork missed oe setece i each article; ad for oe of the articles, the modified etwork icluded a setece that was ot selected by the huma reader. N s = Number of seteces i documet C = Compressio rate [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] [f,f 2,f 3,f 4] ց ւ s i h i (H ) s '' '' '' '' (s,s 2,s ) V. CONCLUSIONS Our text summarizatio method performed well o the test paragraphs with accuracy of 96%. The selectio of features as well as the selectio of summary seteces by the huma reader from the traiig paragraphs plays a importat role i the performace of the etwork. The etwork is traied accordig to the style of the huma reader ad to which seteces the huma reader deems to be importat i a paragraph. This, i fact, is a advatage our approach provides. Idividual readers ca trai the eural etwork accordig to their ow style. I additio, the selected features ca be modified to reflect the reader s eeds ad requiremet. 540 Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206

7 REFERENCES [] Ms. Soali. B. Maid, Research Paper o Basic of Artificial Neural Network, Iteratioal Joural o Recet ad Iovatio Treds i Computig ad Commuicatio, Volume: 2 Issue:, pp: [2] P.B. Baxedale, Machie-Made Idex for Techical Literature: A Experimet, IBM Joural of Research ad Developmet, vol. 2(4), pp , 958. [3] J. Kupiec, J. Pederso ad F. Che, A Traiable Documet Summarizer, Proceedigs of the 8th Aual Iteratioal ACM SIGIR Coferece o Research ad Developmet i Iformatio Retrieval, Seattle, Washigto, pp , 995. [4] M. Porter, A algorithm for suffix strippig, Program, vol. 4(3), pp , 980. [5] H.P. Luh, The Automatic Creatio of Literature Abstracts, IBM Joural for Research ad Developmet, vol. 2(2), pp , 958. [6] M.R. Hestees ad E. Stiefel, Methods of cojugate gradiets for solvig liear systems, Joural of Research of the Natioal Bureau of Stadards, vol. 49, pp , 952. [7] I. Mai, Automatic Summarizatio, Joh Bejamis Publishig Compay, pp , 200. [8] W.T. Chuag ad J. Yag, Extractig setece segmets for text summarizatio: a machie learig approach, Proceedigs of the 23rd Aual Iteratioal ACM SIGIR Coferece o Research ad Developmet i Iformatio Retrieval, Athes, Greece, pp , [9] C-Y. Liu ad E. Hovy, Idetifyig Topics by Positio, Proceedigs of the 5th Coferece o Applied Natural Laguage Processig (ANLP- 97), Seattle, Washigto, pp , 997 [0] K. Sparck Joes, A Statistical Iterpretatio of Team Specificity ad its Applicatio i Retrieval, Joural of Documetatio, vol. 28(), pp. -2, Iteratioal Joural of Computer Systems, ISSN-( ), Vol. 03, Issue 07, July, 206