A Neighborhood Relevance Model for Entity Linking

Transcription

1 A Neighborhood Relevance Model for Entity Linking Jeffrey Dalton University of Massachusetts 140 Governors Drive Aherst, MA, U.S.A. Laura Dietz University of Massachusetts 140 Governors Drive Aherst, MA, U.S.A. ABSTRACT Entity Linking is the task of apping entions in docuents to entities in a knowledge base. One of the crucial tasks is to identify the disabiguating context of the ention, and joint assignent odels leverage the relationships within the knowledge base. We deonstrate how joint assignent odels can be approxiated with inforation retrieval. We build on pseudo-relevance feedback and use the source corpus to build an neighborhood relevance odel that we show is ore effective than local odels for ranking KB entities. Our results deonstrate that siple text based features cobined with a supervised Learning to Rank odel result a odel that atches or outperfors the top perforing syste on in-kb accuracy in the TAC KBP entity linking task. 1. INTRODUCTION Entity linking is iportant because ost inforation on the web is unstructured text in the for of news, blogs, forus, and icroblogs such as Twitter and Facebook. A key challenge is to link these unstructured text docuents to the Web of Data. Entity linking bridges the structure gap by linking entions of entities in free text to Wikipedialike knowledge bases, in which entities are inter-linked and further are associated with free text. Entity links enable navigation between docuents and entities, and to related docuents by the induced link structure. Entity linking is a fundaental building block that supports a wide variety of extraction, suarization, and data ining tasks. For exaple, starting with an entity, the links to docuents where it is entioned can be used to identify sources for extracting relevant facts, such as a person s nae, who they are arried to, or where they work. The ajor challenge in entity linking is uncertainty. An entity ention in text ay be abiguous for a wide variety of reasons: ultiple entities share the sae nae (e.g. Michael Jordan), entities are referred to incopletely (e.g. Justin for Justin Bieber), by pseudonys or nicknaes (Christopher George Latore Wallace is also known as The Notorious Perission to ake digital or hard copies of all or part of this work for personal or classroo use is granted without fee provided that copies are not ade or distributed for profit or coercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific perission and/or a fee. Copyright 20XX ACM X-XXXXX-XX-X/XX/XX...$ B.I.G.), and are often abbreviated (e.g. UW for the University of Wisconsin as well as University of Washington). The entity linking proble has been studied over several years in the TAC Knowledge Base Population venue with the following task definition: Proble Entity Linking: Given a query string q in a docuent, predict the entity c in the knowledge base which this string represents, or NIL if no such entity is available. A typical entity linking syste consists of four steps: 1) query expansion, 2) candidate generation, 3) entity ranking, and 4) handling NIL cases. The goal of the first two steps is to achieve a high-recall set of Wikipedia entities. Given the candidate set, ost effective approaches, e.g., [17, 4, 18], leverage contextual entities as disabiguating evidence in step 3. The downside is that the candidate set of step 2) is acquired by a pipeline of heuristics, resulting in arbitrary large candidate sets on the order of hundreds of entities for abiguous atches, where the consequences of the interplay between step 2 and step 3 are not well aligned. We advocate an inforation retrieval approach that uses one probabilistic odel to approach steps 1-3. State-of-theart entity linking ethods only eploy IR to a inor degree, where this work pushes the boundaries to axiize the use of IR ethods. The foralis of graphical odels allows us to ground our work on odels fro both Inforation Extraction and Inforation Retrieval. For a given query, a good candidate entity fulfills three properties: The naes atch, textual context of the query is contained the article text, and naed entities surrounding the query are reflected in entities neighboring the answer entity. Further notions of contextual siilarity can be included, but throughout the paper we focus on nae variants, phrases, and neighboring entities to odel the user intent for the query. Starting with an underlying graphical odel, corresponding query odel, query analysis and indexing coponent are derived. Entity linking provides soe unusual challenges. The typical IR setting addresses short queries by using relevance odels to add ore ters to the query odel. In entity linking, the query is ebedded in a docuent, providing an abundance of context which could be included. However, not all context is equally helpful, either because of abiguity, heterogeneity in topic, or spurious collocations. Consider the exaple ABC shot the TV draa Lost in Australia. with the task of linking ABC to Aerican Broadcasting Copanies, Inc. The naed entity span Australia is not relevant for the true answer. It ight actually isguide the process to link to the wrong entity Australian Broadcasting

2 Corporation Television. We introduce the neighborhood relevance odel to estiate the salience of context with the goal of filtering and weighting (as opposed to expanding) the query odel. The neighborhood relevance odel is based on ideas of pseudorelevance feedback and latent concept expansion to leverage collocation evidence across other siilar docuents in the corpus. Our ain contributions are: An unsupervised approach to entity linking based upon the Markov Rando Field inforation retrieval odel that provides copetitive perforance out-of-the-box. A unified retrieval based approach to linking cobining candidate generation and ranking in a single retrieval fraework, with ore than 95% recall in the highest ranked 25 entities. uery-specific approach for identifying salient neighboring entities using external and across-docuent evidence based on relevance feedback Deonstrating the benefits of the entity neighborhood relevance odel in cobination with a supervised learning to rank fraework. 2. BACKGROUND AND RELATED WORK 2.1 Related Work Early work on entity linking was done by Bunescu and Pasca [2] and Cucerzan [3] to link entions of topics to their Wikipedia pages. In contrast to their odels, we focus on text based ranking features and do not use Wikipediaspecific features such as category hierarchies, disabiguation pages, and extracted concepts. Our work is related to that of Gottipati and Jiang [8] who apply a language odeling approach to entity linking. They expand the original ention query with contextual inforation fro the language odel of the query docuent. We use the local weighting as a starting point for estiating the entity salience and copare against it as a baseline. Entity linking has been studied in a variety of recent venues. At INEX the Link the Wiki task explored autoatically discovering links that should be created in a Wikipedia article [10]. More recently, it is one of the principle tasks studied at the ongoing Text Analysis Conference Knowledge Base Population track (TAC KBP). Ji et al. [12, 11] provide an overview of the recent systes and approaches. This includes state-of-the-art in approaches to entity linking, such as an updated Cucerzan syste [4] which consistently perfors at or near the top of the rankings. Instead of linking individual entions one at a tie, recent work [3, 18, 21, 13, 9] focuses on linking the set of entions, M, that occur in the local docuent d. These odels perfor collective (or joint) inference over the entions in docuent to identify a coherent assignent of KB entries to entions. In our work we leverage the set of entions M in the docuent as context in an inforation retrieval odel. One iportant way that our work differs is that we focus on identify salient entity entions in the context, because entions in the docuent ay be spurious or only tangentially related if the docuent contains ultiple topics. As the trend for joint resolution of entities within docuents and clustering NIL entries increases, a relevant related task is cross-docuent coreference resolution, where the goal is to deterine whether two entions of an entity refer to the sae instance across all docuents. Work in this field was done by Bagga and Baldwin [1] and Gooi and Allan [7] who used entity language odels built fro the context the entities occur in to disabiguate abiguous entity entions. In this work we do not focus on clustering all entions in text across docuents, instead we focus on linking entions of entities in a single docuent using cross-docuent evidence to a knowledge base. 2.2 Graphical Models and Factor Graphs Graphical odels provide the atheatical fraework for foralizing intuition on how available data and requested quantities are connected. Casting data and quantities of interest as rando variables X = X 1, X 2,... X n, dependencies between two (or ore) variables are encoded by factor functions φ that assign a non-negative score to each joint configuration x of their variables. The configuration of all variables is scored by the likelihood function, which is represented by the noralized product over all factors. This paper akes use of undirected graphical odels, which are also called Markov Rando Fields. In these odels, the likelihood L( x) of the configuration x has an alternative log-linear representation over the cliques y of variables y x exp { θ y, f( y) }. X in the graphical odel, L( x) = 1 Z Here.,. represents the inner product of f, feature vector of the clique, and its paraeter vector θ y. Z refers to the arginal probability over all possible configurations to ensure that L satisfies the laws of a probability. The paraeter vector can be either hand set or learned fro training data with discriinative optiization ethods. Graphical odels can be nested, in which case the likelihood function L of the inner graphical odel can be used as a factor φ or a weighted feature f in the outer odel. The odel can be used to derive predictions x by axiizing the likelihood or equivalently the log likelihood as x = arg ax x log L( x). 2.3 Graphical Models in IR: Sequential Dependence Model Markov Rando Fields are widely used in inforation retrieval and ost unigra, n-gra, and ter dependence odels can be expressed as a graphical odel, involving rando variables for the query ters q 1, q 2,... q n and a docuent d. Each docuent is then scored according to the likelihood function L(d, q 1, q 2,... q n). The sequential dependence odel is a retrieval odel for a ulti word query. Each of the query ters q 1, q 2,... q n is cast as a rando variable together with a docuent d. The odel akes use of three classes of factors: ter, bigra, and windowed bigra, where factors of the sae class are sharing the sae paraeter θ. The odel includes a ter factor φ t (q i, d) between each query ter and the docuent variable. For each pair of adjacent query ters q i, q i+1 it includes a bigra factor φ o (q i, q i+1, d) and an windowed bigra factor φ u (q i, q i+1, d). Each docuent d is then scored according to log L(d, q 1,..., q n) log φ t (q i, d)+ i log φ o (q i, q i+1, d) + log φ u (q i, q i+1, d) i i

3 The graphical odel paradig allows each of the factors φ to arise fro feature vectors f of arbitrary length. However, the original work of Metzler at al. [16] uses only a single scalar feature per factor so that the inner products default to a scalar ultiplication. Factor φ t is induced by the feature that represents the Dirichlet-soothed log-probability p(q i d) of the single ter q i in d. Given the Dirichlet soothing paraeter µ and docuent ter frequency n qi,d, docuent length n,d, and collection ter frequency n qi, and nq i, collection size n (, ) it is given by p(q i, d) = log n q i,d+µ n (, ). n,d +µ Further it uses the score of ordered bigras for log φ o (q i, q i+1, d) = θ o #1(q i, q i+1) and unordered bigras within a window of eight ters for log φ u (q i, q i+1, d) = θ u #uw8(q i, q i+1). We are going to use the open source retrieval engine Galago, 1 which is part of the Leur project. Galago coes with the ipleentation of the sequential dependence odel above, acessible via the operator #seqdep(q 1,..., q n). Galago further allows to nest retrieval odels via the #cobine operator to score docuents geoetric ean interpolation. For odels M i and interpolation weights λ i, the likelihood function L(d) = i Mi(d)λ i is accessible via syntax #cobine:1=λ 1:2=λ 2:...:n=λ n(m 1 M 2... M n). 2.4 Graphical Models in IE: Candidate-based Neighborhood Model Markov rando fields are equally popular in the inforation extraction counity. Early approaches to entity linking [5], use a graphical odel with single factor φ e (q, c). Each of the candidates c are scored by log L(c, q) θ, f(q, c) where the feature vector includes a variety of siilarity functions between the query string, and the article s title, redirect, anchor text, as well as TF-IDF weighted cosine siiliarity between ters in the query docuent and the Wikipedia article. The paraeter θ is trained discriinatively with a learning to rank approach. Ratinov [19] extended this basic odel by explicitly incorporating contextual entity entions, each with a respective set of candidates z. The idea is that entities which are entioned in the sae docuents are also likely to be linked on Wikipedia. Therefore, if each contextual ention is linked to its correct candidate z, the links between KB entries z and the candidates entries c for the query will reveal the true answer c. This intuition is odeled in the likelihood function of Equation 1, which requires two copatibility easures: One copatibility easure between entions in the text to KB entries φ e, as well as a copatibility easure aong KB entries φ ee. L(c) = φ e (q, c) ( φ e (, z) φ ee (z, c) dz) (1) As the task is to link only the query ention, the contextual entity links are arginalized out by integration over z. The dilea is that linking s to z s requires to solve the entity linking proble as part of the solution. Therefore the proble has to be addressed by joint inference which in this case does not have a closed-for solution, and therefore require approxiate inference. 3. UERY MODEL 1 In this section we close the gap between the graphical odels for entity linking as developed in IE counity and the graphical odels for inforation retrieval. One shortcoing of Ratinov s odel is that it requires generated candidate sets for the query and contextual entions which, with current ethods, is not only tie consuing, but can also result in very large sets of candidates z that need to be integrated over in Equation 1. Another issue is that not all contextual entions are equally relevant for the query, as we argued above, soe are spurious or isleading. We address both issues in the following. 3.1 Neighborhood uery Model We deonstrate how the retrieval engine can be used to optiize Equation 1 whenever factor functions φ e and φ ee can be expressed as query operators. The consequence is a tight integration of the candidate generation (step 2, in the pipeline) with the entity ranking (step 3), optiizing over all possible candidates on Wikipedia at once. The key insight is to solve the integral over z (cf. Equation 2), with sart preprocessing and indexing: The Wikipedia snapshot is transfored so that the article of entity c is enriched with inforation about the contextual entities and their KB counterparts z. The factor φ e (, c) can therefore be directly optiized within the retrieval odel fraework. ˆ φ e (, c) = φ e (, z) φ ee (z, c) dz (2) L(c) = φ e (q, c) φ e (, c) (3) With the introduction of the factor φ e, Equation 1 is rewritten as Equation Relevance-weighted Neighborhood uery Model As pointed out before, not all contextual entities are equally relevant. For each contextual entity the salience for disabiguating query q is denoted by ρ q(), ranging on a scale between 0 and 1. If the salience ρ q() is 0, we want to reove the effect of φ e (, c) on the likelihood function. Based on the geoetric ean, which is the natural choice for probabilities, we achieve the weighting with the geoetric interpolated odel of Equation 4. L(c) = φ e (q, c) ( ) ρq() φ e (, c) Notice, that the unweighted odel follows as a special case where all saliences are 1. We want to further introduce paraeters λ and λ M that allow the trade-off between the direct siilarity of the query and candidate as expressed by φ e (q, c) and the aggregated influence of the contextual entities. Exploiting that the sortorder induced by L is invariant with respect to logariths, we cast the optiization in log-space as in Equation 5. log L(c) = λ logφ e (q, c)+λ M (4) ( ) ρ q() log φ e (, c) (5)

4 3.3 Extended Context in uery Model We further study variations on the relevance-weighted neighborhood query odel given in Equation 5. Nae variances v of the query string can be extracted fro the query docuent, to add robustness to the entity linking inference. This is especially iportant if the query string is an acrony or an abiguous reference to the entity. However, the nae variance extraction ay be less reliable, which is expressed in an additional trade-off paraeter λ V. We also incorporate non-entity context in the for of surface phrases of the sentence that surround the query ention or one of the entioned nae variances. The sentence context s is balanced with the paraeter λ S, and the copatibility with the candidate answer c is expressed in the factor φ se (s, c). The resulting optiization criterion of the candidate answer c for the query odel given the query q, V nae variants v, S contextual phrases s, and M contextual entity entions is given in Equation 6. log L(c) = λ logφ e (q, c) (6) + λ V 1 log φ e (v, c) V v + λ S 1 log φ se (s, c) S + λ M 1 M s ( ) ρ q() log φ e (, c) 3.4 Joint Inference with Galago Using log-linear odels for factors φ with features that are readily available in the Indri and Galago query languages, we can leverage the retrieval engine to optiize Equation 6. This is possible because the weighted sus with weights λ and ρ are expressed with the #cobine operator (cf. Section 2.3), with subodels that express φ(x) = exp { θ, f(x) }. Inspecting the inner product θ, f(x) = i θifi(x) reveals another #cobine operator with weights θ i acting on the features f i(x). If all features f i(x) can be expressed in the Galago query language, Equation 6 can be directly optiized inside the search engine. For instance we could use a feature vector for f e (q, c) that separates scores of q in the Wikipedia title field, redirect field, and anchor text field. For feature vector f e (, c) we could separate scores of in the article full text fro the titles of in- and outlinks. But for siplicity we use feature vectors with only a single entry, the sequential dependence odel score, for all factors f se (s, c), f e (q, c), f e (, c). With these feature functions, the optiization criterion of Equation 6 is equivalent to the following Galago query. 4. NEIGHBORHOOD RELEVANCE MODEL In the previous section we introduced a query odel containing relevance weighted entity entions. We now discuss ethods for estiating these weights ρ q() in an unsupervised anner. As previously entioned, even unabigious entions are not necessarily useful for disabiguation. We introduce a odel for deterining the iportance of these neighborhood salience weights ρ() using pseudorelevance feedback [15]. The idea is that a neighbor is iportant if it occurs frequently in the context of the query #cobine:0=λ :1=λ V :2=λ S :3=λ M ( (7) ) #seqdep(q) #cobine(#seqdep(v 0)... #seqdep(v V )) #cobine(#seqdep(s 0),..., #seqdep(s S)) #cobine:0 = ρ( 0) :... k : ρ( 0)( ) #seqdep( 0),..., #seqdep( k ) ention within the docuent as well as across other docuents that are topically related. The approach is based on the assuption that these pseudorelevant docuents ention the sae target entity. In other words, the goal is to identify docuents containing pseudocoreferent entions. We use these pseudo-docuent entions to deterine the strength of association between entities in the neighborhood. If a neighboring ention in the query docuent is not relevant, it will only be contained in few or none of the pseudo-relevant docuents. If it represents salient disabiguation context, it is assued to occur in any docuents of the retrieved set. 4.1 Local Docuent Neighborhood Model We can estiate beliefs about the salience of the entity neighborhood fro the source docuent. This technique was used by Gottipati and Jiang [8] to build a ultinoial language odel of entity entions fro the query docuent d q with occurrence count n,dq. We refer to this siple estiation technique as the local odel. ρ local n,dq q () = (8) n,d q Gottipati also tested weighting schees that incorporating distance, but found that these did not significantly iprove the results. We find that whenever the query is not the ain focus of the query docuent, any contextual entities are not relevant for disabiguation and ay actually lead to worse perforance (see experiental evaluation). 4.2 Across-docuent Neighborhood Relevance Model We suggest the neighborhood relevance odel which estiates entity saliences ρ fro across-docuent evidences. Having identified the query string q, with nae variants v, and neighborhood, and using the local docuent saliences ρ local, we search for coreferent entions in pseudo-relevant docuents we call the pseudo-coreferent entions. We use the query odel given in Equation 6 to retrieve pseudo-relevant docuents d fro the source corpus. L(d), the retrieval score under the query odel, represents its relevance to the query q. Given a set of pseudo-relevant docuents D, we can approxiate the docuent relevance L(d) probability with. In cobination with a ultinoial language odel, based on occurrence counts n,d of d D L(d ) entions in the pseudo-relevant docuents d, the neighborhood relevance odel estiates salience weights ρ nr q () as follows.

5 ρ nr q () = 1 d D L(d ) d D n,d L(d) (9) n,d In other words, the salience of a ention in the neighborhood is expressed by accuulating relative retrieval probabilities of docuents according to how often they contain the ention. Typically relevance odels are used to expand the query with new ters. This odel is capable of introducing new entity entions that are not contained in the query docuent. However, since the context of the query docuent is already very rich, a preliinary experient deonstrated that it is better to use the relevance odel to reduce and weight the context found in the query docuent. 5. KB BRIDGE: ENTITY LINKING SYSTEM In this section we describe KB Bridge, our inforation retrieval based entity linking syste which is ipleented using the Galago search engine and the MRF-IR retrieval fraework. The linking syste links entions in the query docuent to knowledge base entities. The ranking of the entities is a two-stage process. First, entities are ranked using the Galago retrieval odel described in Equation 7. We then optionally, in the second stage, the ranking is refined with a supervised learning to rank odel e.g. RankLib 2. The final step is NIL handling which deterines if the ention is in the knowledge base or whether it is unknown. 5.1 Knowledge Base Representation Our syste addresses text-driven knowledge bases in which each entity is associated with free text, where links between entities are extracted fro hyperlinks directly or via relation extraction systes. Wikipedia is one representation of such a knowledge base, but our syste works as well on other knowledge bases with full text data. In order to efficiently search over very large knowledge bases containing illions or even billions of entries we use a full-text retrieval syste. For the knowledge base experients we describe below, we index the full text of Wikipedia article, the title, redirects, Freebase nae variations, and internal anchor text, web anchor text. The cobination of both text and structured etadata in the search index allows the execution of rich contextual query odels. We can further utilize field indices to efficiently incorporate coplex feature vectors in Section??. For estiating the neighborhood relevance odel, we index a larger unstructured corpus, preferrably with siilar characteristics as the query docuents. This allows the syste to estiate the neighborhood relevance odel weights ρ using topically siilar docuents fro pseudo relevance feedback. 5.2 Docuent Analysis The first step in linking is to identify the query span q and to find disabiguating contextual inforation for the query odel introduced in Section??: nae variants v, contextual sentences s, and and other entity spans in the neighborhood. If entities of type person, organization, or location are the ain focus of the linking effort, Naed Entity Recognition tools, such as fro UMass s factorie [14] and Stanford 2 vdang/ranklib.htl CoreNLP [6] provide useful spans to derive query entions q, nae variants v, and neighboring entities. The KB Bridge syste is not liited to entities of these types, it can link any kind of KB entity, as long as a corresponding span detector is available. For the nae variants, v, the syste identifies siilar spans within the docuent that are likely to be coreferent, such as Steve to Steve Jobs or IOC to International Olypic Coittee. The goal is to identify alternative naes that are less abiguous than the query ention. We use the within-docuent coreference tool fro UMass s factorie, together with capitalized word sequences that contain the query string (ignoring capitalization and punctuation for the atching) to extract nae variants v. All reaining spans are used as the neighborhood. Fro the set of coreferent entions, we extract the sentences s they occur within. After reoving stopwords, casing and punctuation they represent non-ner context such as verbs, adjectives, and ulti-word phrases. 5.3 KB Entity Ranking Next, the inforation fro docuent analysis is used to build a query odel given in Equations 6 and 7 to rank the entries in the knowledge base. Our syste supports all feature vectors for which entries are expressable in Galago query notation. To deonstrate generality for cases where rich eta data is absent, we use a feature representation where every factor is associated with a single feature, which represents the score of the sequential dependence odel. We use this query odel both for the relevance odel and for retrieving KB entities. To estiate the neighbor relevance odel, we retrieve docuents fro the background corpus, using local salience weights ρ local. Fro the retrieved docuent set D, the neighborhood relevance saliences ρ nr are estiated using Equation 9. Finally, the query odel with updated salience weights ρ nr () is executed against the knowledge base to retrieve KB entities c which are optial according to Equation Learning to Rank KB entries To further iprove the ranking, we leverage supervised achine learning in a learning to rank (LTR) syste which re-ranks the retrieved set of KB entities. This ranking can eploy ore expensive textual feature coparison which are infeasible to score on the entire collection. For ranking, the syste uses the LabdaMART odel, a type of gradient boosted decision tree that is state-of-the-art in ranking and captures non-linear dependencies in the data. There are hundreds of features used in the reranking step. A description of the sets of features used in the odel is found in Table 1. We note that the features used in the odel are text-based and do not utilize the type inforation of NER types and Wikipedia categories. 5.5 NIL Handling The NIL handling coponent deterines if the top ranked KB entity for a ention should be linked or if there is no atch in the knowledge base in which case NIL should be returned. We return NIL, if the supervised score of the top ranked entity is below a threshold τ. The NIL threshold τ is tuned on the training data. ueries for which NIL is returned should be clustered, however it has been shown that

6 Feature Set Type Description Character Siilarity q, v Lower-cased noralized string siilarity: Exact atch, prefix atch, Dice, Jaccard, Levenstein, Jaro-Winkler Token Siilarity q,v Lower-cased noralized token siilarity: Exact atch, Dice, Jaccard Acrony atch q Tests if query is an acrony, if first letters atch, and if KB entry nae is a possible acroyn expansion Field atches q, v Field counts and query likelihood probabilities for title, anchor text, redirects, alternative naes fields Link Probability q, v p (anchor KB entry) - the fraction of internal and external total anchor strings targeting the entity Inlink count docuent prior Log of the nuber of internal and external links to the target KB entry Text Siilarity docuent Noralized text siilarity of docuent and KB entity: Cosine with TF-IDF, KL, JS, Jaccard token overlap Neighborhood text siilarity docuent Noralized neighborhood siilarity: KL Divergence, Nuber of atches, atch probability Neighborhood link siilarity docuent Neighborhood siilarity with in/out links: KL divergence, Jensen-Shannon Divergence, Dice overlap, Jaccard Rank features retrieval Raw retrieval log likelihood, Noralized posterior probability, 1/retrieval rank Context Rank Features retrieval retrieval scores for each contextual coponents: q, v, s, nr, local Table 1: Features of the query ention and candidate Wikipedia entity. siple heuristics achieve strong results. We assign NILs to the sae cluster whenever their top ranked candidates are the sae, otherwise they are kept in their own singleton cluster. For the special case of an TAC KBP entity linking syste, we notice that the reference knowledge base is a subset of the full Wikipedia. We exploit this fact by further returning NIL whenever the top ranked Wikipedia entity is not contained in the reference knowledge base. 6. EXPERIMENTAL EVALUATION 6.1 Setup We base our experiental evaluation on four data sets fro the TAC KBP entity linking copetition fro Over the years, the TAC organizers and the Linguistic Data Consortiu cae up with evaluation queries with varying characteristics both in ters of abiguity (average unique entions per entity) and variety (average nuber of entities per ention) Data The TAC KBP Knowledge Base was constructed fro a dup of English Wikipedia fro October 2008 containing 818,741 entries. The source collection includes over 1.2 illion newsire docuents, approxiately 500 thousand web docuents and hundreds of transcribed spoken docuents. Across all years there are a total of 12,130 queries. We use all queries with odd nubered query IDs as training data, and the even queries for evaluation. We inspected the distribution of the queries in the split and the NIL to in-kb as well as the type distribution (Per/Org/GPE) of the results are preserved. The training set contains 6043 queries, 3034 with a ground truth entity c and 3009 NIL queries. The evaluation set contains 6087 queries with 3058 NIL and 3029 in-kb. This training set is used to learn paraeters of our query odel, as well as paraeters of the supervised re-ranker. For evaluation, we use even nubered query IDs on a year-by-year basis Linking corpus: 2012 Wikipedia For evaluating a large-scale text retrieval approach to linking, we use a ore recent dup of Wikipedia that includes the full-text of the article along with other structured etadata including redirects, disabiguation links, outgoing links, anchor text, and full-text. We use a Freebase dup of the English Wikipedia fro January 2012, which contains over 3.8 illion articles including link and full text inforation. In addition, we use the Google Cross-Wiki dictionary[20] for external web link inforation. We derive a apping between our snapshot and the official TAC KBP knowledge base using title atches and article redirects. 6.2 Methods We first evaluate which kinds of context are ostly benefitial to be integrated into Equation 6: the query q, the nae variants v, the sentences s surrounding the query or nae variants, as well as the neighborhood context. Which subset is included is indicated by, V, S, or M in the ethod prefix. For ethods that include neighborhood context, we study different estiation ethods for the salience ρ() of each neighbor. This includes the local docuent odel by Gottipati [8] (indicated by local), and our neighborhood relevance odel (indicated by the suffix nr). We use the ethod based on the query string (), and the cobination of query and nae variants (V), as well as the context weighting local context (VM local) as baselines. Our suggested ethods are VSM prf and VM prf, the full query odel with neighborhood relevance weighting with and without sentences. For each of the copared ethods, we train a separate set of λ paraeters on the training training data using a coordinate ascent learning algorith. For instance, the resulting VSM nr odel the estiated paraeters are: λ = 0.321, λ V = 0.293, λ S = 0.155, and λ M = Context Entity Ranking We first perfor an intrinsic evaluation of the ranking ethods on the in-kb queries. We easure the ranking effectiveness in identifying the correct KB entity. In this experient we study the effectiveness of different cobinations of context for disabiguation. Table 2 presents the ranking results in ters of the ean reciprocal rank etric (MRR). It deonstrates that the all best ethods include the neighborhood relevance weighting schee (nr), and the suggested ethods VM nr and VSM nr are significantly better than the V baseline. The only exception is in 2010, when the queries are easier only the M nr ethod is statistically significant. The ethod VM nr ethod is significantly better than the weighting fro Gottipati (VM local) local weighting for the years 2009, 2010, and 2011, with no significant difference in We hypoth-

7 Method V M nr * 0.825* M VM nr 0.795* * 0.715* VM local 0.784* * VS * VSM nr 0.792* * 6* VSM local 0.780* * 0.719* all context 0.786* * 0.735* P V VM VM_nr VSM_nr all V VM VM_nr VSM_nr all V VM VM_nr VSM_nr all V VM VM_nr VSM_nr all Table 2: Ranking results on TAC by year with varying context ethods with ean reciprocal rank (MRR). The best results for each year are highlighted in bold. Results that are statistically significant with α = 5% over the V baseline are indicated with *. esize that this is because the queries in 2012 are significantly ore difficult and therefore the quality of the feedback docuents is likely significantly lower. Figures 1 visualizes the ablation study for the suggested ethod. Figure 1a shows the cuulative iproveents as context is added (VM) and weighted with neighborhood relevance (VM nr). We observe that adding sentences context does not significantly iprove perforance, and we do not report the sentence context results in subsequent experients. Figure 1b details the individual contributions of context coponents. Coparing the ethod which adds nae variants V versus the ethod which incorporates weighted context (M nr) without nae variants, reveals that the context is equally helpful as nae variants expansion, if weighted with the neighborhood relevance odel. This is especially useful when no high quality nae variants are extractable fro the text, as is the case in inforal text fro social edia. 6.4 Learning to Rank Once candidates are ranked, we can further refine the unsupervised retrieval ranking using supervised learning to rank. We use the features describe previously in Table 1 and apply the learning to rank odel to our best current odel, VM nr. The results of this are shown in Table 3. The results show that significant iproveent over the unsupervised baseline is possible using ore expensive text features cobined with supervised learning. While all years iprove, the results for 2012 are still well below the other years, indicating that the difficulty of these queries. The distribution of the correct answers is iportant. For linking, only getting the top answer correct or not is considered. However, down strea systes using the entity linking inforation could benefit fro getting the correct answer in a sall nuber of results. Figure 2 visualizes the distribution of the answer in the ranked results. The results show that all the ethods return 90% of entities in the top 10 results. After learning to rank refineent is perfored 95% recall is achieved using only the top 5 results. 6.5 TAC KBP results In previous sections we focused on the ranking of in-kb entities only. In this section, we evaluate the ranking as part of the entire linking pipeline described in Section??. P V M_local M_nr (a) Cuulative V M_local M_nr V M_local M_nr (b) Individual Contributions V M_local M_nr Figure 1: Ablation study for the suggested ethod in ters of Precision at 1. Method VM nr VM nr LTR Table 3: Learning to rank refineent results with ean recipocal rank (MRR). All LTR results are statistically significant with α = 5% over the unsupervised VM nr average recall V VM_nr VM_nr LTR Figure 2: Recall at rank cutoff k cutoff rank k For this experient we use the icro-averaged accuracy because we do not focus on NIL clustering. The results are reported in Table 4. We observe that the unsupervised retrieval VM nr perfors well, above the edian 2012 and copetitive in previous years. The supervised odels iprove effectiveness. When the NIL score threshold is applied we observe that the in-kb effectiveness decreases, but the overall accuracy nubers increase because of a greater decrease in false positive links. The results show that the

8 in KB NIL all in KB NIL all in KB NIL all in KB NIL all VM nr VM nr LTR VRM nr LTR NIL Best Perforer Table 4: TAC Entity Linking perforance in icro-avg accuracy. VM nr with LTR reranking and the nil threshold applied outperfor the top syste in 2009 and are copetitive with the best perforing syste in subsequent years. 7. CONCLUSION In this paper we propose an approach to entity linking based upon the Markov Rando Field inforation retrieval odel (MRF-IR). We focus on the task of ranking knowledge base entities. The inforation retrieval syste uses only text-based features without exploiting knowledge fro Wikipedia. We deonstrated how joint neighborhood odels can be expressed within the MRF-IR fraework. Further, we proposed a neighborhood relevance odel (NRM) that uses relevance feedback to identify salient entity entions in the context of the query docuent. Our experients on the TAC KBP entity linking data show that the neighborhood relevance odel outperfors or is en par with other contextual odels. We also deonstrated that a learning to rank odel using text based features outperfors the current best perforing systes on in-kb ranking, and even a very siple NIL handling strategy results in overall nubers that are coparable to the state-of-the-art entity linking systes. Acknowledgeents This work was supported in part by the Center for Intelligent Inforation Retrieval and in part under subcontract # fro SRI International, prie contractor to DARPA contract #HR C Any opinions, findings and conclusions or recoendations expressed in this aterial are those of the authors and do not necessarily reflect those of the sponsor. 8. REFERENCES [1] Ait Bagga and Breck Baldwin. Entity-based cross-docuent coreferencing using the vector space odel. In Coputational Linguistics, [2] Razvan Bunescu and Marius Pasca. Using encyclopedic knowledge for naed entity disabiguation. In European Chapter of the Association for Coputational Linguistics (EACL-06), [3] S. Cucerzan. Large-Scale Naed Entity Disabiguation Based on Wikipedia Data. EMNLP, [4] S. Cucerzan. Tac entity linking by perforing full-docuent entity extraction and disabiguation. Proceedings of the Text Analysis Conference, [5] Mark Dredze, Paul McNaee, Delip Rao, Ada Gerber, and Ti Finin. Entity disabiguation for knowledge base population. In International Conference on Coputational Linguistics, [6] J.R. Finkel, T. Grenager, and C. Manning. Incorporating non-local inforation into inforation extraction systes by gibbs sapling. In Annual Meeting on Association for Coputational Linguistics. Association for Coputational Linguistics, [7] Chung H. Gooi and Jaes Allan. Cross-Docuent coreference on a large scale corpus. In Daniel and Sali Roukos, editors, HLT-NAACL, [8] Swapna Gottipati and Jing Jiang. Linking entities to a knowledge base with query expansion. In EMNLP, [9] Johannes Hoffart, Mohaed A. Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weiku. Robust disabiguation of naed entities in text. In EMNLP, [10] Darren W. Huang, Yue Xu, Andrew Trotan, and Shloo Geva. Focused access to XML docuents. chapter Overview of INEX 2007 Link the Wiki Track, pages Springer-Verlag, [11] Heng Ji and Ralph Grishan. Knowledge base population: successful approaches and challenges. In HLT, [12] Heng Ji, Ralph Grishan, and Hoa Dang. Overview of the TAC2011 knowledge base population track. In Text Analysis Conference, [13] Sayali Kulkarni, Ait Singh, Ganesh Raakrishnan, and Souen Chakrabarti. Collective annotation of wikipedia entities in web text. In KDD, [14] Andrew McCallu, Karl Schultz, and Saeer Singh. Factorie: Probabilistic prograing via iperatively defined factor graphs. In NIPS, [15] D. Metzler and W.B. Croft. Latent concept expansion using arkov rando fields. In Proceedings of the 30th annual international ACM SIGIR conference on Research and developent in inforation retrieval, pages ACM, [16] Donald Metzler and W. Bruce Croft. A arkov rando field odel for ter dependencies. In Proceedings of the 28th annual international ACM SIGIR conference on Research and developent in inforation retrieval, SIGIR 05, pages , New York, NY, USA, ACM. [17] S. Monahan, J. Lehann, T. Nyberg, J. Plyale, and A. Jung. Cross-lingual cross-docuent coreference with entity linking. Proceedings of the Text Analysis Conference, [18] L. Ratinov, D. Roth, D. Downey, and M. Anderson. Local and global algoriths for disabiguation to wikipedia. In ACL, 2011.

9 [19] L. Ratinov, D. Roth, D. Downey, and M. Anderson. Local and global algoriths for disabiguation to wikipedia. In ACL, [20] Valentin I. Spitkovsky and Angel X. Chang. A Cross-Lingual dictionary for english wikipedia concepts. In Conference on Language Resources and Evaluation, [21] Veselin Stoyanov, Jaes Mayfield, Tan Xu, Douglas W. Oard, Dawn Lawrie, Ti Oates, and Ti Finin. A context-aware approach to entity linking. In AKBC-WEKEX, 2012.