Time Expression Recognition Using a Constituent-based Tagging Scheme

Transcription

1 Track: Web Conen Analysis, Semanics and Knowledge Time Expression Recogniion Using a Consiuen-based Tagging Scheme Xiaoshi Zhong and Erik Cambria School of Compuer Science and Engineering Nanyang Technological Universiy, Singapore {xszhong,cambria}@nu.edu.sg ABSTRACT 1) We find from four daases ha ime expressions are formed by loose srucure and he words used o express ime informaion can differeniae ime expressions from common ex. The findings drive us o design a learning mehod named TOMN o model ime expressions. TOMN defines a consiuen-based agging scheme named TOMN scheme wih four ags, namely T, O, M, and N, indicaing he consiuens of ime expression, namely Time oken, Modifier, Numeral, and he words Ouside ime expression. In modeling, TOMN assigns a word wih a TOMN ag under condiional random fields wih minimal feaures. Essenially, our consiuen-based TOMN scheme overcomes he problem of inconsisen ag assignmen ha is caused by he convenional posiion-based agging schemes (e.g., BIO scheme and BILOU scheme). Experimens show ha TOMN is equally or more effecive han sae-of-he-ar mehods on various daases, and much more robus on cross-daases. Moreover, our analysis can explain many empirical observaions in oher works abou ime expression recogniion and named eniy recogniion. 3) 2) 4) Sepember/B 2006/L 1/B Sepember/I 2006/L (a) BILOU ag assignmen: Sepember in differen posiions wihin labeled ime expressions is assigned wih differen ags of U, B, L, or I. The inconsisen ag assignmen reduces he predicive power of Sepember, and his conradics he finding ha ime okens can differeniae ime expressions from common ex. Sepember/T 3) 2006/T Sepember/T 1) Sepember/T 2006/T 4) 1/N Sepember/T 2006/T 2) (b) TOMN ag assignmen: Sepember in differen posiions wihin labeled ime expressions is consisenly assigned wih he same ag of T. The consisen ag assignmen proecs he Sepember s predicive power. Figure 1: BILOU and TOMN ag assignmen during raining. BILOU scheme is based on he posiion wihin he chunk, while TOMN scheme bases on he consiuen of he chunk. Here inconsisen ag assignmen is defined as ha during raining, a word is assigned wih differen ags because he word appears in differen posiions wihin labeled chunks.2 ACM Reference Forma: Xiaoshi Zhong and Erik Cambria Time Expression Recogniion Using a Consiuen-based Tagging Scheme. In Proceedings of The Web Conference 2018 (WWW 18). ACM, New York, NY, USA, 10 pages. hps://doi.org/ / Sepember/U 2006/B Sepember/L The findings moivae us o design a learning mehod named TOMN o model ime expressions. Specifically, TOMN defines a consiuen-based agging scheme named TOMN scheme,1 consising of four ags, namely T, O, M, and N, indicaing he consiuens of ime expression, namely Time oken, Modifier, Numeral, and he words Ouside ime expression. Time okens include he words ha explicily express informaion abou ime, such as 2006, monh, and Sepember. Modifiers are he words ha modify ime okens and appear around hem; for example, las modifies monh in las monh. Numerals are he ordinals and numbers (excep he year like 2006 ). TOMN models ime expressions under condiional random fields (CRFs) [19] wih only a kind of pre-ag feaures and he lemma feaures; i assigns a word wih a TOMN ag. TOMN scheme can keep he ag assignmen consisen during raining and herefore overcomes he problem of inconsisen ag assignmen. (In a supervised learning procedure, ag assignmen occurs in feaure exracion during raining and in ag predicion. We focus on he raining sage o analyze he impac of ag assignmen.) The loose srucure by which ime expressions are formed exhibis in wo aspecs. Firs, many ime expressions consis of loose collocaions. For example, he ime oken Sepember can form a ime expression by iself, or forms Sepember 2006 by anoher ime INTRODUCTION Time informaion plays an increasingly imporan role in daa mining, informaion rerieval, and naural language processing [1, 8]. Researchers from hese fields have devoed remendous effor o specify sandards for ime expression annoaion [14, 18, 30, 32], build annoaed corpora for ime expression [18, 27, 31], and recognize ime expressions from free ex [5 7, 11, 42, 43, 45]. We analyze four daases (i.e., TimeBank [31], Gigaword [28], WikiWars [27], and Twees [47]) for he characerisics of ime expressions and have wo imporan findings abou heir organizaion and consiuen words. Firs, ime expressions are formed by loose srucure, wih more han 53.5% of disinc ime okens appearing in differen posiions wihin ime expressions. Second, ime okens can differeniae ime expressions from common ex; more han 91.8% of ime expressions have a leas one ime oken while no more han 0.7% of common ex conain ime okens. This paper is published under he Creaive Commons Aribuion 4.0 Inernaional (CC BY 4.0) license. Auhors reserve heir righs o disseminae he work on heir personal and corporae Web sies wih he appropriae aribuion. WWW 18, April 23 27, 2018, Lyon, France 2018 IW3C2 (Inernaional World Wide Web Conference Commiee), published under Creaive Commons CC BY 4.0 License. ACM ISBN /18/04. hps://doi.org/ / TOMN denoes our mehod and TOMN scheme denoes he agging scheme ha TOMN defines. 2 The definiion of inconsisen ag assignmen can be generalized as ha during raining, a uni in differen labeled insances is assigned wih differen ags for some reason while he uni should be consisenly assigned wih he same ag. The uni of ineres can be a word, a relaion, a webpage, or a group of words as a whole, ec. 983

2 oken appearing afer i, or 1 Sepember 2006 by a numeral before i and anoher ime oken afer i. Second, some ime expressions can change heir word order wihou changing heir meanings. For example, Sepember 2006 and 2006 Sepember express he same meaning. The convenional agging schemes like BILOU [33] are based on he posiion wihin he chunk, namely a Uni-word chunk, and he Beginning, Inside, and Las word of a muli-word chunk. Under BILOU scheme, a word ha appears in differen posiions wihin labeled ime expressions is assigned wih differen ags; for example, he above Sepember can be assigned wih U, B, L, or I. See Figure 1(a). The inconsisen ag assignmen causes difficuly for saisical models o model ime expressions. Firs, inconsisen ag assignmen reduces he predicive power of lexicon, and his conradics he finding ha ime okens can differeniae ime expressions from common ex. Second, inconsisen ag assignmen migh cause he problem of ag imbalance. Our TOMN scheme insead is based on he consiuen of he chunk (i.e., Time oken, Modifier, and Numeral) and assigns he same consiuen word wih he same ag, regardless of is frequency and is posiions wihin ime expressions. Under TOMN scheme, for example, he above Sepember is consisenly assigned wih T. See Figure 1(b). Wih consisen ag assignmen, TOMN scheme avoids he poenial ag imbalance and proecs ime okens predicive power. We evaluae TOMN agains five sae-of-he-ar mehods (i.e., HeidelTime [39], SUTime [9], SynTime [47], ClearTK [4], and UW- Time [21]) on hree daases (i.e., TE-3 [42], WikiWars [27], and Twees [47]). 3 Experimens show ha TOMN is equally or more effecive han he sae-of-he-ar mehods, and much more robus on cross-daase performance. Experimens also show he advanage of TOMN scheme over he posiion-based agging schemes. In addiion, we find ha he named eniies in CoNLL03 English NER daase [34] demonsrae common characerisics similar o he ime expressions (see Secion 6 for deails). The finding suggess ha he problem of inconsisen ag assignmen widely exiss in he posiion-based agging schemes and ha our idea of defining consiuen-based agging scheme should be widely effecive for general eniies. In he fuure we will furher develop ha idea. To summarize, we make he conribuions as follows. We analyze four diverse daases for he characerisics of ime expressions and have wo imporan findings abou heir organizaion and consiuen words. We discover a fundamenal problem underlying in he posiionbased agging schemes: inconsisen ag assignmen. To overcome ha problem we define a consiuen-based agging scheme o model ime expressions. Our mehod provides an idea o model arge eniies based on heir consiuens. We conduc experimens on various daases, and he resuls show he effeciveness, efficiency, and robusness of our mehod compared wih sae-of-he-ar mehods. The resuls also show he advanage of our consiuen-based agging scheme over he posiion-based agging schemes. The analysis in his work can help explain many empirical resuls and observaions repored in oher works abou ime expression recogniion and named eniy recogniion. 3 We follow [47] no o use he Gigaword daase in our experimens because is labels are auomaically generaed by oher aggers bu no he ground ruh. 2 RELATED WORK Time expression idenificaion aims o auomaically idenify ime expressions from free ex and i can be divided ino wo subasks, namely recogniion and normalizaion. In his paper we focus on he recogniion. Mehods for ime expression recogniion can be caegorized ino rule-based mehods and learning-based mehods. Rule-based Mehods. Rule-based mehods like TempEx, GUTime, HeidelTime, and SUTime mainly handcraf deerminisic rules o idenify ime expressions. TempEx and GUTime use boh handcrafed rules and machine-learn rules o resolve ime expressions [26, 44]. HeidelTime manually designs rules wih ime resources o recognize ime expressions [39]. SUTime designs deerminisic rules a hree levels (i.e., individual word level, chunk level, and ime expression level) for ime expression recogniion [9]. A recen ype-based ime agger, SynTime, designs general heurisic rules wih a oken ype sysem o recognize ime expressions [47]. TOMN uses he oken regular expressions, similar o SUTime [9] and SynTime [47], and furher groups hem ino hree oken ypes, similar o SynTime. While SynTime furher defines 21 oken ypes for he consiuen words of ime expression, TOMN uses he hree general oken ypes ha are helpful for a learning mehod o connec he words wih low frequencies o he words wih high frequencies. And TOMN leverages saisical informaion from enire corpus o improve he precisions and alleviae he deerminisic role of deerminisic rules and heurisic rules. Learning-based Mehods. Learning-based mehods in TempEval series mainly exrac feaures from ex (e.g., characer feaures, word feaures, synacic feaures, and semanic feaures), and on he feaures apply saisical models (e.g., CRFs, logisic regression, maximum enropy, Markov logic nework, and suppor vecor machines) o model ime expressions [4, 15, 23, 41]. Besides he sandard mehods, Angeli e al., and Angeli and Uszkorei exploi an EM-syle approach wih composiional grammar o learn laen ime parsers [2, 3]. Lee e al. leverage combinaory caegorial grammar (CCG) [36] and define a collecion of lexicon wih linguisic conex o model ime expressions [21]. Unlike [4, 15, 17, 23, 41] which use he sandard feaures, TOMN derives feaures according o he characerisics of ime expressions and uses only a kind of pre-ag feaures and he lemma feaures; which can enhance he impac of he significan feaures and reduce he impac of he insignifican feaures. Unlike [2, 3, 21] which use fixed srucure informaion, TOMN uses he loose srucure informaion by grouping he consiuen words of ime expression under hree oken ypes, which can fully accoun for he loose srucure of ime expressions. More imporanly, TOMN models ime expressions under a CRFs framework wih a consiuen-based agging scheme, which can keep he ag assignmen consisen. Time Expression Normalizaion. Mehods for ime expression normalizaion are mainly based on rules [4, 15, 23, 39, 41, 44]. Since he rule mehods are highly similar, Llorens e al. sugges o consruc a large shared knowledge base for public use [22]. Lee e al., Angeli e al., and Angeli and Uszkorei combine grammar rules and machine learning echniques o normalize ime expressions [2, 3, 21]. TOMN focuses on he recogniion and leaves he normalizaion o hose highly similar rule mehods or he fuure work. 984

3 Table 1: Daase saisics ( imex denoes ime expression.) Daase #Documens #Words #Timex TimeBank ,418 1,243 Gigaword 2, ,309 12,739 WikiWars ,468 2,671 Twees ,199 1,127 Table 2: Percenage of disinc ime okens and modifiers ha appear in differen posiions wihin ime expressions Daase BIO Scheme BILOU Scheme Time Token Modifier Time Token Modifier TimeBank Gigaword WikiWars Twees TIME EXPRESSION ANALYSIS Daases. We analyze he ime expressions from four daases: TimeBank, Gigaword, WikiWars, and Twees. TimeBank is a benchmark daase and consiss of 183 news aricles [31]. Gigaword consiss of 2,452 news aricles wih auomaically annoaed ime expressions [28]. WikiWars is consruced by collecing war aricles from Wikipedia [27]. Twees consiss of 942 wees colleced from Twier [47]. The four daases cover comprehensive daa (Time- Bank, Gigaword, and Twees) and specific domain daa (WikiWars) as well as formal ex (TimeBank, Gigaword, and WikiWars) and informal ex (Twees). Table 1 summarizes he daase saisics. 3.1 Findings Alhough he four daases differ in source, domain, corpus size, and ex ype, heir ime expressions demonsrae some common characerisics. We find such wo common characerisics of ime expressions abou heir organizaion and consiuen words. Finding 1. Time expressions are formed by loose srucure; more han 53.5% of ime okens appear in differen posiions wihin ime expressions. We find ha ime expressions are formed by loose srucure and he loose srucure exhibis in wo aspecs. Firs, many ime expressions consis of loose collocaions. For example, he ime oken Sepember can form a ime expression by iself, or forms Sepember 2006 by anoher ime oken appearing afer i, or 1 Sepember 2006 by a numeral before i and anoher ime oken afer i. Second, some ime expressions can change heir word order wihou changing heir meanings. For example, Sepember 2006 can be wrien as 2006 Sepember wih he same meaning. From he poin of view of he posiion wihin ime expressions, he Sepember may appear as he (i) beginning or (ii) inside word of ime expression when ime expressions are modeled by BIO scheme; or i may appear as (1) a uni-word ime expression, or he (2) beginning, (3) inside, (4) las word of a muli-word ime expression when ime expressions are modeled by BILOU scheme. Table 3: Percenage of ime expression s consiuens ha appear in ime expressions (P imex ) and in common ex (P ex ) Daase Time Token Modifer Numeral P imex P ex P imex P ex P imex P ex TimeBank Gigaword WikiWars Twees Table 2 repors he percenage of disinc ime okens and modifiers ha appear in differen posiions wihin ime expressions. Disinc here means ignoring he word varians and frequencies during couning; for example, monh, monhs, and mhs are reaed he same and are couned only once. Differen posiions means he wo differen posiions under BIO scheme and a leas wo of he four differen posiions under BILOU scheme. For each daase, under BIO scheme, more han 53.5% of disinc ime okens appear in differen posiions; and under BILOU scheme, more han 61.4% of disinc ime okens appear in differen posiions. The number of modifiers ha appear in differen posiions is more han 27.5%. When BIO or BILOU scheme is applied o model ime expressions, he appearance in differen posiions leads o inconsisen ag assignmen, and he inconsisen ag assignmen causes difficuly for saisical models o model ime expressions. We need o explore an appropriae agging scheme (see Secion 4.1 for deails). Finding 2. Time okens can differeniae ime expressions from common ex while modifiers and numerals canno. Table 3 repors he percenage of ime expression s consiuen words appearing in ime expressions (P imex ) and in common ex (P ex ). Common ex here means he whole ex wih ime expressions excluded. P imex is defined by Equaion (1) and P ex is by Equaion (2), where T {ime oken, modi f ier, numeral}. #imex ha conain T P imex (T ) = (1) #oal imex #okens ha are T P ex (T ) = (2) #oal okens From he second column of Table 3 we can see ha more han 91.8% of ime expressions conain a leas one ime oken; he percenage 91.8% is consisen wih he one analyzed by Zhong e al. [47]. (Some ime expressions wihou ime oken depend on oher ime expressions; for example, 95 depends on 100 days in 95 o 100 days. ) By conras, he hird column shows ha no more han 0.7% of common ex conain ime okens. This indicaes ha ime okens can differeniae ime expressions from common ex. On he oher hand, he las four columns show ha on average, 32.1% of ime expressions and 21.1% of common ex conain modifiers and 24.9% of ime expressions and 4.1% of common ex conain numerals. This indicaes ha modifiers and numerals canno differeniae ime expressions from common ex. Looking a he Twees daase, we can see ha he P imex of he ime okens (96.0%) is relaively high while he P imex of he modifiers (21.4%) and numerals (18.8%) are much lower han he ones of oher daases. This indicaes ha in Twier people end o use ime expressions wih fewer modifiers and numerals. 985

4 3.2 Properies In addiion o our findings, Zhong e al. also conclude some characerisics abou ime expressions from he same four daases [47]. The characerisics concluded by Zhong e al. are helpful for our analysis, so we briefly describe hem here and repor hem as ime expressions properies. 4 Propery 1. Time expressions are very shor. More han 80% of ime expressions conain no more han hree words and more han 90% conain no more han four words. [47] Time expressions across differen daases follow a similar lengh disribuion and on average, ime expressions conain abou wo words. For he one-word ime expressions, he percenage of which in TimeBank is 40.3%; in Gigaword he percenage is 53.9%; in WikiWars i is 36.2%; and in Twees i is 62.9%. Propery 2. Only a small group of ime-relaed keywords are used o express ime informaion. [47] There are only abou 70 disinc ime okens used in individual daase, regardless of he corpus sizes and he number of ime expressions, and only 123 disinc ime okens across differen daases. Tha means ime expressions are highly overlapped a heir ime okens wihin individual daase and across differen daases. Propery 3. POS informaion could no disinguish ime expressions from common words, bu wihin ime expressions, POS ags can help disinguish heir consiuens. [47] Among he op 40 par-of-speech (POS) ags in ime expressions (10 4 daases), 37 ags whose percenage over he corresponding ags of he whole ex is lower han 20%. 4 TOMN: TIME-RELATED TAGGING SCHEME Figure 2 shows he overview of TOMN, including hree pars: TOMN scheme, TmnRegex, and ime expression recogniion. TOMN scheme is a consiuen-based agging scheme wih four ags. Tmn- Regex is a se of regular expressions for ime-relaed words. Time expressions are modeled under a CRFs framework wih he help of TmnRegex and TOMN scheme. 4.1 TOMN Scheme Finding 1 suggess us o explore an appropriae agging scheme o model ime expressions. We define a consiuen-based agging scheme named TOMN scheme wih four ags: T, O, M, and N; hey indicae he consiuens of ime expression, namely Time oken, Modifier, and Numeral, and he words Ouside ime expression. Convenional agging schemes like BIO 5 [35] and BILOU 6 [33] are based on he posiion wihin he chunk. BIO refers o he Beginning, Inside, and Ouside of a chunk; BILOU refers o a Uni-word chunk, and he Beginning, Inside, Las word of a muli-word chunk. TOMN scheme insead is based on he consiuen of he chunk, indicaing he consiuens of ime expression. Following we use BILOU scheme, he newer version, as represenaive of he convenional posiion-based agging schemes for analysis. 4 One of Zhong e al. concluded properies is incorporaed ino our Finding 2. 5 The BIO scheme in his paper denoes he sandard IOB2 scheme described in [35]. 6 The BILOU scheme is also widely known as he IOBES scheme. T (ime oken) M (modifier) N (numeral) O (ouside imex) TOMN Scheme Time Token Modifier Numeral TmnRegex Raw Tex Feaure Exracor TOMN Pre-ag Feaures Lemma Feaures CRFs-based Tagger Annoaed Tex Figure 2: Overview of TOMN. Top-lef side shows he TOMN scheme, consising of four ags. Boom-lef side is he TmnRegex, a se of regular expressions for ime-relaed words. Righ-hand side shows he ime expression modeling, wih he help of TmnRegex and TOMN scheme. Using BILOU scheme for ime expression recogniion leads o inconsisen ag assignmen. (A ypical supervised learning procedure involves ag assignmen in wo sages; one is in feaure exracion during raining and he oher is in ag predicion. We focus on he raining sage o analyze he impac of ag assignmen under differen kinds of agging schemes.) Finding 1 shows ha ime expressions are formed by loose srucure, exhibiing in loose collocaions and exchangeable order. Under BILOU scheme, boh of loose collocaions and exchangeable order lead o he problem of inconsisen ag assignmen. Suppose Sepember, Sepember 2006, 2006 Sepember, and 1 Sepember 2006 are four manually labeled ime expressions in raining daa. During feaure exracion, hey are assigned as Sepember/U, Sepember/B 2006/L, 2006/B Sepember/L, and 1/B Sepember/I 2006/L (see Figure 1(a)). The four Sepember have he same word (he word iself) and express he same meaning (he ninh monh of he year), bu because hey appear in differen posiions wihin ime expressions, hey are assigned wih differen ags (i.e., U, B, L, and I). The inconsisen ag assignmen causes difficuly for saisical models o model ime expressions. Firs, inconsisen ag assignmen reduces he predicive power of lexicon. A word assigned wih differen ags causes confusion o model he word. If a word is assigned wih differen ags in equal number, hen he word iself canno provide any informaion useful o deermine which ag should be assigned o i. Reducing he predicive power of lexicon indicaes reducing he predicive power of ime okens, and his conradics Finding 2 which shows ha ime okens can differeniae ime expressions from common ex. Second, inconsisen ag assignmen may cause anoher problem: ag imbalance. If a ag of a word dominaes in raining daa, hen all he insances of ha word in es daa will be prediced as ha ag. For example, 1 Sepember 2006 can be wrien as Sepember 1, 2006 in some culures. If he raining daa are colleced from he syle of 1 Sepember 2006 in which mos Sepember are assigned wih I, hen i is difficul for he rained model o correcly predic he daa colleced from he syle of Sepember 1, 2006 in which Sepember should be prediced as B. 986

5 TOMN scheme insead overcomes he problem of inconsisen ag assignmen. TOMN scheme assigns a ag o a word according o he consiuen role ha he word plays in ime expressions. Since our TmnRegex well defines he consiuen words of ime expression (see Secion 4.2) and same word plays same consiuen role in ime expressions, herefore, he same word is assigned wih he same TOMN ag, regardless of is frequency and is posiions wihin ime expressions. For example, TOMN scheme assigns he above four ime expressions as Sepember/T, 2006/T Sepember/T, Sepember/T 2006/T, and 1/N Sepember/T 2006/T (see Figure 1(b)). The four Sepember have he same ag of T and saisical models need only o model hem as T, wihou any confusion. Wih consisen ag assignmen, TOMN scheme avoids he poenial ag imbalance and proecs ime okens predicive power. Besides, TOMN scheme models a word by fewer ags han BILOU scheme does. BILOU scheme ypically models a ime oken by four ags (i.e., U, B, L, or I) and models a modifier/numeral by five ags (i.e., U, B, L, I, or O), while TOMN scheme models a ime oken by one ag (i.e., T) and models a modifier/numeral by wo ags (i.e., M or N if he modifier/numeral appears inside ime expressions and O if i appears ouside ime expressions). Compared wih BILOU scheme, TOMN scheme reduces he complexiy of he rained model. 4.2 TmnRegex Propery 2 indicaes a small group of words ha are used in ime expressions. TOMN uses hree ime-relaed oken ypes, namely ime oken, modifier, and numeral, o group hose words. The hree oken ypes are consisen wih hree of he above four ags (i.e., T, M, and N), and are similar o he ones defined in SynTime [47]. Time okens explicily express informaion abou ime, such as year (e.g., 2006 ), monh (e.g., Sepember ), dae (e.g., ), and ime unis (e.g., monh ). Modifiers are he words ha modify ime okens and appear around hem; for example, he wo modifiers he and las modify he ime oken monh in he las monh. Numerals include ordinals and numbers, excep hose ha are recognized as year (e.g., 2006 ). Token ypes are defined on okens hemselves and are no necessarily relevan o heir conex. For example, 2006 alone expresses ime informaion, so i is a ime oken; alhough he 1 in 1 Sepember 2006 implies he day, iself alone does no express ime informaion, so i is a numeral. The hree oken ypes wih he words hey group form a se of oken regular expressions, and he se of oken regular expressions is denoed by TmnRegex. TmnRegex is consruced by imporing oken regular expressions for is ime oken, modifier, and numeral from SUTime. 7 Like SynTime [47], TmnRegex collecs from SUTime only he regular expressions a he level of oken. TmnRegex conains only 115 disinc ime okens, 57 modifiers, and 58 numerals, wihou couning he words wih changing digis. 4.3 Time Expression Recogniion Time expression recogniion mainly consiss of wo sages: (1) feaure exracion and (2) model learning and agging. When exracing feaures we se a guideline ha he feaures should be able o help differeniae ime expressions from common ex and help build connecions among ime expressions. 7 hps://gihub.com/sanfordnlp/corenlp/ree/maser/src/edu/sanford/nlp/ime/rules Feaure Exracion. The feaures we exrac include wo kinds: TOMN pre-ag feaures and lemma feaures. When exracing feaures we use w i o denoe he i-h word in ex. TOMN Pre-ag Feaures. Finding 2 shows ha ime okens can differeniae ime expressions from common ex while modifiers and numerals canno, herefore, how o leverage he informaion of hese words becomes crucial. In our consideraion, hey are reaed as pre-ag feaures under TOMN scheme. Specifically, a ime oken is pre-agged by he ag of T, a modifier is pre-agged by M, and a numeral is by N; oher common words are by O. The assignmen of pre-ags is conduced by simply looking up he words a TmnRegex. The las four columns of Table 3 indicae ha modifiers and numerals consanly appear in ime expressions and in common ex. To disinguish where a modifier or numeral appears, we conduc a checking for he modifiers and numerals (he words wih pre-ag of M or N (M/N)) o record wheher hey direcly or indirecly modify any ime oken. Indirecly here means a M/N ogeher wih oher M/N modifies a ime oken; for example, in las wo monhs, las (M) ogeher wih wo (N) modifies monhs (T). The checking is a loop searching relying on he ime okens. For each ime oken we search is lef side wihou exceeding is previous ime oken and search is righ side wihou exceeding is following ime oken. When searching a side of a ime oken, if encouner a M/N, hen record he M/N and coninue searching; if encouner a word ha is no M/N, hen sop he searching for his side of his ime oken. Afer he checking, hose M/N ha modify ime okens are recorded; for example, he wo in wo monhs is recorded while in wo apples is no. The checking is reaed as a feaure for modeling. For he pre-ag feaures we exrac hem in a 5-word window of w i, namely he pre-ags of w i 2, w i 1, w i, w i+1, and w i+2. For he checking feaure we consider only he curren word w i. In he raining phase we consider he TOMN pre-ag feaures for only he words wihin labeled ime expressions. In he es phase he TOMN pre-ag feaures are exraced for all he words in ex. Lemma Feaures. The lemma feaures include he word shape of w i in a 5-word window, namely he lemmas of w i 2, w i 1, w i, w i+1, and w i+2. If w i conains changing digi(s), hen we se is lemma by is oken ype. For example, he lemma of 20:16 is se by TIME. We use five special lemma for he words wih changing digis: YEAR, DATE, TIME, DECADE, and NUMERAL. The lemma feaures can help build connecions among ime expressions; for example, he wo differen words 20:16 and 19:25:33 are conneced a TIME. The lemma feaures are exraced for all he words in ex in boh of he raining phase and he es phase. We do no consider he feaures of characers nor word varians because hey canno help build connecions among ime expressions bu hur he modeling; for example, Sep. and Sepember express he same hing bu compuer does no rea hem as he same hing. We also do no consider he POS feaures nor oher synacic feaures. Propery 3 indicaes ha POS ags canno help differeniae ime expressions from common ex, and experimens confirm ha POS ags do no improve he performance. On he oher hand, Finding 1 shows ha ime expressions are formed by loose srucure, which ogeher wih Propery 3 suggess ha oher synacic feaures (e.g., synacic dependency) ha rely on POS ags and fixed linguisic srucure canno provide exra useful informaion 987

6 Track: Web Conen Analysis, Semanics and Knowledge Table 4: Feaures of word w i used for modeling 5 EXPERIMENTS We conduc experimens on hree daases, namely TE-3, WikiWars, and Twees, o evaluae TOMN agains five sae-of-he-ar mehods, namely HeidelTime (wih Colloquial seing for Twees), SUTime, SynTime, ClearTK-TimeML (shor as ClearTK ), and UWTime. 1. TOMN pre-ags of w i in a 5-word window, namely he pre-ags of w i 2, w i 1, w i, w i+1, and w i+2 2. If w i is a M or N, hen check wheher i direcly or indirecly modifies any ime oken 3. Lemmas of w i in a 5-word window 5.1 Experimen Seing Daases. The hree daases used in our experimens are TE-3, WikiWars, and Twees. TE-3 uses he TimeBank corpus as raining se and he Plainum corpus as es se. TimeBank consiss of 183 news aricles and Plainum consiss of 20 new aricles; hey are comprehensive corpora in formal ex and described in TempEval-3 [42]. WikiWars is a specific domain daase in formal ex, consising of 22 English Wikipedia aricles abou famous wars [27]. Twees is a comprehensive daase in informal ex, wih 942 wees ha conain ime expressions [47]. For WikiWars and Twees, we follow previous works [21, 47] o se heir raining ses and es ses. The performance of a mehod (rule-based or learning-based) on a daase is repored on he daase s es se. On/O Sepember/T 1/N,/M 1939/T,/O sae/o in/o 1939/T./O (a) T, M, and N words ogeher form a ime expression. For/O he/m firs/n 10/N monhs/t of/m 1915/T,/O Ausria/O... (b) T, M, and N words ogeher form a ime expression.... in/o a/m few/m days/t and/m weeks/t respecively/o./o (c) Linker and separaes wo ime expressions. Baseline Mehods. We evaluae TOMN agains five sae-of-hear mehods, including hree rule-based mehods, HeidelTime, SUTime, and SynTime, and wo learning-based mehods, ClearTK and UWTime. HeidelTime [39] and SUTime [9] use predefined deerminisic rules and achieve he bes resuls in relaxed mach while ClearTK [4] uses a CRFs framework wih BIO scheme and achieves he bes resul in sric mach in TempEval-3 [42]. UWTime uses combinaory caegorial grammar (CCG) o predefine linguisic srucure for ime expressions and achieves beer resuls han HeidelTime on TE-3 and WikiWars daases [21]. SynTime uses a se of general heurisic rules and achieves good resuls on TE-3, WikiWars, and Twees daases [47]. SynTime has wo versions, a basic version and an expanded version. Because he expanded version requires exra manual annoaion for each daase, for fair comparison, we use he basic version o ensure ha he oken regular expressions used in SynTime and TOMN are comparable. We/O could/o do/o i/o in/o 95/N o/m 100/N days/t./o (d) Linker o separaes wo ime expressions. Figure 3: Examples of ime expression exracion. The label indicaes ime expression. for a CRFs-based learning mehod, which already considers he dependency, o differeniae ime expressions from common ex. We herefore do no use hose synacic feaures in our model. Feaure Values. For he TOMN pre-ag feaures we separae he feaures wih binary values. The heory of scales of measuremen suggess ha non-ordinal aribues should be ransformed o separae dimensions [37]. TOMN pre-ags are non-ordinal, herefore, each of TOMN pre-ags as well as he checking feaure works as a separae feaure. For he lemma feaures we follow he radiional use o incorporae muliple values under a feaure. Table 4 summarizes he feaures ha are used for ime expression modeling. Typically up o 11 feaures are exraced for a word. Evaluaion Merics. We repor he resuls in Precision, Recall, and F 1 under sric mach and relaxed mach by using he evaluaion oolki10 of TempEval-3 [42]. Sric mach means exac mach beween he exraced ime expressions and he ground ruh while relaxed mach means ha here exiss overlap beween hem Model Learning and Tagging. TOMN models ime expressions hrough he feaure vecors under a CRFs framework [19]. In implemenaion, we use Sanford Tagger8 for he lemma feaures and use CRFSuie9 wih defaul seing for model learning. For he agging, each word is assigned wih one of TOMN ags, namely T, O, M, or N. Noe ha he TOMN scheme is used in feaure exracion as a kind of feaures and in sequence agging as labeling ags. 5.2 Experimen Resuls Table 5 repors he performance of TOMN and baseline mehods. Among he 18 measures, TOMN achieves 13 bes or second bes resuls. I is beer han SynTime which achieves 10 bes or second bes resuls, and much beer han oher baselines which achieve a mos 4 bes or second bes resuls. For each measure, TOMN achieves eiher bes resuls or comparable resuls. Especially for he F 1, TOMN performs he bes in sric F 1 on Twees and in relaxed F 1 on WikiWars; for oher F 1, TOMN performs comparably (mos are wihin 0.5% difference) o he corresponding bes resuls. Time Expression Exracion. Afer sequence agging, hose T, M, and N words (or non-o words) ha appear ogeher are exraced as a ime expression. See Figure 3(a) and 3(b). A special kind of modifiers, i.e., he linker o, -, or, and and separaes he non-o words ino parallel ime expressions. See Figure 3(c) and 3(d) TOMN vs. Baseline Mehods. We furher compare TOMN wih he rule-based mehods and he learning-based mehods. 8 hp://nlp.sanford.edu/sofware/agger.shml 9 hp:// 10 hp:// 988

7 Table 5: Performance of TOMN and baseline mehods. We make bold he bes resuls and underline he second bes. Some resuls are repored direcly from he sources where he resuls are publicly available. Daase TE-3 WikiWars Twees Mehod Sric Mach Relaxed Mach Pr. Re. F 1 Pr. Re. F 1 HeidelTime[40] SUTime[10] SynTime[47] ClearTK[4] UWTime[21] TOMN HeidelTime[38] SUTime SynTime[47] ClearTK UWTime[21] TOMN HeidelTime SUTime SynTime[47] ClearTK UWTime TOMN TOMN vs. Rule-based Baselines. On TE-3 and Twees, TOMN achieves comparable resuls wih SynTime. On WikiWars, TOMN achieves he F 1 wih 2.0% o 2.3% absolue increase over SynTime. This indicaes ha compared wih SynTime, TOMN is equally effecive on comprehensive daa and more effecive on specific domain daa. The reason is ha he heurisic rules of SynTime are greedy for recalls a he cos of precisions, and he cos is expensive when i comes o specific domain daa. TOMN insead leverages saisical informaion from enire corpus, which may miss he rare ime expressions bu helps recognize ime expressions more precisely; especially in specific domain daa, he saisical informaion significanly improves he precisions a lile cos of recalls. For HeidelTime and SUTime, excep he sric F 1 on WikiWars, TOMN ouperforms hem on all he daases, wih up o 15.3% absolue increase in recalls and up o 12.0% absolue increase in F 1. The reason is ha he deerminisic rules of HeidelTime and SUTime are designed in fixed manner, which lacks flexibiliy [47]. TOMN vs. Learning-based Baselines. Excep he sric F 1 on WikiWars, TOMN ouperforms ClearTK and UWTime on all hree daases in all he recalls and F 1. Especially on TE-3 and Twees daases, TOMN improves he recalls by a leas 9.8% in sric mach and a leas 5.1% in relaxed mach, and improves he F 1 by a leas 8.5% in sric mach and a leas 3.1% in relaxed mach. The reasons are ha he fixed linguisic srucure predefined in UWTime canno fully capure he loose srucure of ime expressions, he BIO scheme used in ClearTK reduces he predicive power of ime okens, and some of heir feaures (e.g., synacic dependency) acually hur he modeling. For he sric F 1 on WikiWars, TOMN performs slighly worse han he wo learning-based mehods because like SynTime, TOMN follows TimeBank and SynTime o exclude he preposiions (excep of ) from ime expressions while some ime expressions in WikiWars include hese preposiions. Table 6: Cross-daase performance on he es se of TE-3. Training indicaes he daase whose raining se is used for raining. Color background indicaes single-daase resuls. Training TE-3 WikiWars Twees Mehod Sric Mach Relaxed Mach Pr. Re. F 1 Pr. Re. F 1 ClearTK UWTime TOMN ClearTK UWTime TOMN ClearTK UWTime TOMN Table 7: Cross-daase performance on es se of WikiWars Training TE-3 WikiWars Twees Sric Mach Relaxed Mach Mehod Pr. Re. F 1 Pr. Re. F 1 ClearTK UWTime TOMN ClearTK UWTime TOMN ClearTK UWTime TOMN Table 8: Cross-daase performance on he es se of Twees Training TE-3 WikiWars Twees Sric Mach Relaxed Mach Mehod Pr. Re. F 1 Pr. Re. F 1 ClearTK UWTime TOMN ClearTK UWTime TOMN ClearTK UWTime TOMN Cross-daase Performance. We conduc cross-daase experimens and compare TOMN wih he learning-based mehods ha require raining. Specifically, a mehod is rained on he raining se of one daase and hen esed on he es ses of oher daases. Since he hree daases used in our experimens are quie diverse, he cross-daase experimens on he hree daases herefore can evaluae a learning mehod s robusness. Table 6 repors he crossdaase performance on TE-3; Table 7 repors he performance on WikiWars; and Table 8 on Twees. For comparison, Table 6, 7, and 8 also repor he performance on single-daase; single-daase here means he raining se and he es se belong o he same daase. The single-daase resuls are repored direcly from Table 5. On (he es se of) TE-3, TOMN achieves a leas 84.0% in sric F 1 and a leas 93.4% in relaxed F 1. (See he rows of WikiWars 989

8 and Twees in Table 6.) On Twees, TOMN achieves a leas 85.5% and 94.3% respecively. (See he rows of TE-3 and WikiWars in Table 8.) I significanly ouperforms ClearTK and UWTime. On WikiWars, TOMN achieves comparable resuls wih ClearTK and UWTime in relaxed mach bu performs worse han UWTime in sric mach; especially when rained on Twees, TOMN achieve only 63.0% in sric F 1, 7.5% lower han he one of UWTime. (See he rows of TE-3 and Twees in Table 7.) Twees conains many shor ime expressions (62.9% one-word ime expressions) and uses fewer modifiers and numerals in ime expressions while WikiWars includes quie a few long ime expressions (only 36.2% one-word ime expressions) and some descripive ime expressions. For hese reasons, TOMN rained on Twees canno fully recognize he long and descripive ime expressions in WikiWars. UWTime insead predefines linguisic srucure, which conribues significanly o exac recogniion of hose long and descripive ime expressions. Look a he single-daase and cross-daase performance in relaxed mach. TOMN achieves similar performance, regardless of which daase i is rained on; in relaxed F 1, TOMN achieves abou 93.9% on TE-3, abou 91.6% on WikiWars, and abou 94.8% on Twees. By conras, ClearTK and UWTime perform well on singledaase bu much worse on cross-daase; especially on Twees, heir relaxed F 1 drops from a leas 87.0% when rained on Twees o a mos 80.3% when rained on oher daases. This indicaes ha TOMN is much more robus han ClearTK and UWTime. The robusness of TOMN can be explained by Finding 2 and Propery 2. Finding 2 indicaes ha ime okens are capable of predicing ime expressions and Propery 2 indicaes ha ime expressions highly overlap a heir ime okens wihin individual daase and across differen daases. Tha means, he ime okens from one daase can help recognize he ime okens from oher daases. Therefore, in erms of relaxed mach, he cross-daase performance should be comparable o he single-daase performance Facor Analysis. We conduc experimens o analyze he impac of he TOMN scheme as labeling ags and he feaures used in TOMN. The resuls are repored in Table 9. Impac of TOMN Labeling Tags. To analyze he impac of he TOMN scheme as labeling ags, we keep all he feaures unchanged excep change he labeling ags from TOMN scheme o BIO scheme o ge a BIO sysem and o BILOU scheme o ge a BILOU sysem. The BIO and BILOU sysems use he same TOMN pre-ag feaures and lemma feaures ha are used in TOMN. 11 The ag assignmen of BIO and BILOU schemes during feaure exracion in he raining sage follows heir radiional use; for example, a uni-word ime expression is assigned wih B under BIO scheme while i is assigned wih U under BILOU scheme. When exracing ime expressions from agged sequence in he es sage, we adop wo sraegies. One sraegy follows heir radiional use in which ime expressions are exraced according o he ags of words; for example, a U word under BILOU scheme is exraced as a ime expression. The oher sraegy follows he one used for TOMN in which he non-o words ha appear ogeher are exraced as a ime expression. The radiional sraegy is denoed by rad while 11 The BIO and BILOU schemes can be exraced wih oher feaures, bu we using he BIO and BILOU schemes here is o conduc conrolled experimens o analyze he impac of TOMN scheme as labeling ags. So we exrac he same feaures in TOMN o he BIO and BILOU schemes. Table 9: Impac of facors. BIO denoes he sysems ha replace TOMN labeling ags by BIO ags while BILOU denoes he sysems ha replace by BILOU ags. rad indicaes he radiional sraegy for exracion while nono indicaes he non-o sraegy. indicaes he kind of feaures removed from TOMN; PreTag denoes he TOMN pre-ag feaures and Lemma denoes he lemma feaures. Daase TE-3 WikiWars Twees Mehod Sric Mach Relaxed Mach Pr. Re. F 1 Pr. Re. F 1 TOMN BIO r ad BIO nono BILOU r ad BILOU nono PreTag Lemma TOMN BIO r ad BIO nono BILOU r ad BILOU nono PreTag Lemma TOMN BIO r ad BIO nono BILOU r ad BILOU nono PreTag Lemma he non-o sraegy is by nono. The resuls of he BIO and BILOU sysems are repored as BIO and BILOU in Table 9. We can see ha he non-o sraegy performs almos he same as he radiional sraegy, and he BIO sysems achieve comparable or slighly beer resuls compared wih he BILOU sysems. The reason is ha ime expressions on average conain abou wo words (see Propery 1); in ha case, BILOU scheme is reduced approximaely o BLOU scheme and BIO scheme is changed approximaely o BLO scheme. Beween BLOU scheme and BLO scheme here is only sligh difference; and under he impac of inconsisen ag assignmen and TOMN pre-ag feaures, his sligh difference affecs slighly o he performance. Following we do no disinguish BILOU scheme from BIO scheme and do no disinguish non-o sraegy from radiional sraegy; he four mehods of BIO r ad, BIO nono, BILOU r ad, and BILOU nono are simply represened by BILOU. On TE-3 and WikiWars, TOMN significanly ouperforms BILOU. TOMN achieves he recalls ha are 7.0% o 14.5% absolue higher han hose of BILOU and achieves he F 1 ha are 5.0% o 11.4% absolue higher han hose of BILOU. The reason is ha he loose collocaions and exchangeable order in ime expressions lead BILOU scheme o suffer from he problem of inconsisen ag assignmen; TOMN scheme insead overcomes ha problem. On Twees, TOMN and BILOU achieve similar performance; he difference beween heir performance ranges wihin 1% in mos measures. The reason is ha 62.9% of ime expressions in Twees are one-word ime expressions and 96.0% of ime expressions conain 990

9 Table 10: Running ime ha TOMN and he learning-based mehods cos o go hrough a whole process (uni: seconds) Mehod TE-3 WikiWars Twees ClearTK UWTime 864 1, TOMN ime okens (see Propery 1); which means he one-word ime expressions conain only he ime okens. In ha case, TOMN scheme is reduced approximaely o TO scheme and BILOU scheme is reduced approximaely o UO scheme. Then UO scheme becomes a consiuen-based agging scheme in which U indicaes he ime oken. I is equivalen o TO scheme. (BIO scheme is reduced approximaely o BO scheme in which B indicaes he ime oken. Then BO scheme is equivalen o TO scheme as well as UO scheme.) Impac of TOMN Pre-ag Feaures. To analyze he impac of TOMN pre-ag feaures, we remove hem from TOMN. Afer hey are removed, alhough mos of he precisions increase and even reach highes scores, all he recalls and F 1 drop dramaically, wih absolue decreases of 10.4% o 32.9% in recall and 4.8% o 19.1% in F 1. Tha means TOMN pre-ag feaures significanly improve he performance and confirms he predicive power of ime okens. The resuls also validae ha pre-ag is a good way o use hose lexicon. Impac of Lemma Feaures. When lemma feaures are removed, he performance in relaxed mach on all he daases is affeced slighly. The reason is ha he TOMN pre-ag feaures provide useful informaion o recognize ime okens. The sric mach on TE-3 and WikiWars decreases dramaically, which indicaes ha he lemma feaures heavily affec he recogniion of modifiers and numerals. The sric mach on Twees is affeced lile because wees end no o use modifiers and numerals in ime expressions Compuaional Efficiency. HeidelTime and SUTime run nearly in real ime; SynTime in real ime. Table 10 repors he running ime ha TOMN and he learning-based mehods cos o go hrough a whole process (including raining and es) on he hree daases on a Mac OS lapop (1.4GHz Processor and 8GB Memory). We can see ha TOMN is much more efficien han ClearTK and UWTime. Consider only he es, TOMN runs in real ime. 6 DISCUSSION The analysis of ime expressions can explain a lo of empirical observaions repored in oher works abou ime expression recogniion. UzZaman e al. repor ha using an exra large daase does no improve he performance [42]; Behard repors ha using TimeBank alone performs beer han using TimeBank and AQUAINT daases ogeher [4]; and Filannino e al. repor ha feaures of gazeeers, shallow parsing, and proposiional noun phrases do no conribue significan improvemen [15]. These observaions can be explained by he findings and properies illusraed in Secion 3. Finding 1, 2 and Propery 2, 3 ogeher sugges ha addiional gazeeers, large corpus, and more daases provide no furher useful informaion bu repeaed ime okens and heir loose combinaions, and ha hose synacic feaures canno provide exra useful informaion for a CRFs-based learning mehod o model ime expressions. The analysis of agging schemes can explain he empirical observaions repored in oher works abou he impac of he BIO (or IOB2) and BILOU (or IOBES) schemes in named eniy recogniion (NER). Rainov and Roh repor ha BILOU scheme ouperforms BIO scheme on MUC-7 and CoNLL03 NER daases [33]; Dai e al. repor ha IOBES scheme performs beer han IOB2 scheme in drug name recogniion [13]. When looking a heir resuls, however, we find ha he improvemens are raher sligh, mos of hem are wihin 1%; and in some cases, BIO scheme performs beer han BILOU scheme. Lample e al. confirm ha hey do no observe significan improvemen of IOBES scheme over IOB2 scheme on CoNLL03 NER daase [20]. These observaions can be explained by our analysis of agging schemes in Secion 4.1 and Basically, BIO and BILOU schemes are based on he posiion wihin he chunk and implicily assume ha arge eniies should be formed by fixed srucure and even fixed collocaions. Bu eniies as par of language are acually flexible. When applied o eniy recogniion, he BIO and BILOU schemes would more or less suffer from he problem of inconsisen ag assignmen. We analyze he named eniies in CoNLL03 (English NER) daase [34] as an example. We find ha for each of he CoNLL03 raining, developmen, and es ses, more han 53.7% of disinc words appear in differen posiions wihin named eniies; more han 93.7% of named eniies each has a leas one word no appearing in common ex; and he named eniies on average conain 1.45 words, wih 63.2% one-word named eniies. The percenage 53.7% is similar o he one of disinc ime okens in ime expressions (53.5%; see Finding 1); he 93.7% is similar o he one of ime expressions ha conain ime okens (91.8%; see Finding 2); and he lengh disribuion is similar o he one of ime expressions in Twees (see Propery 1). Tha means named eniies demonsrae common characerisics similar o ime expressions. When modeling named eniies, like modeling ime expressions, he BIO and BILOU schemes would eiher suffer from he problem of inconsisen ag assignmen or be roughly equivalen if hey were reduced o he consiuen-based BO and UO schemes. In eiher case, he difference beween he wo schemes impacs slighly. When analyzing he CoNLL03 daase (which conains four eniy ypes: PER, LOC, ORG, and MISC), we find ha some named eniies are annoaed wih differen eniy ypes. In he raining se, for example, Wimbledon is annoaed 4 imes wih LOC, 8 imes wih ORG, and 18 imes wih MISC. Such named eniies (including several polysemy) in he raining se, developmen se, and es se reach relaively high percenage of respecive 6.9%, 4.4%, and 6.5%. The inconsisen annoaion and inconsisen ag assignmen may be able o explain why mos sae-of-he-ar NER mehods achieve he F 1 a around 94.5% on he developmen se and around 91.5% on he es se [12, 20, 24, 25, 29, 33, 46], and why more han 10 years effor improves he F 1 by only 0.8% on he developmen se (from 2003 s 93.9% [16] o curren 94.7% [25]) and by only 2.9% on he es se (from 2003 s 88.7% [16] o curren 91.6% [12]). The wo inconsisency problems seem o limi he upper bound of he performance on developmen se a near 94.5% and he one on es se a near 91.5%. This suggess ha o furher improve he performance on curren CoNLL03 daase wih curren mehods is difficul and unreliable. Insead of coninuing o fine-une curren mehods, we should ry o correc he inconsisen annoaion and address he problem of inconsisen ag assignmen. 991