Biomedical literature mining for knowledge discovery

Transcription

1 Biomedical literature mining for knowledge discovery REZARTA ISLAMAJ DOĞAN National Center for Biotechnology Information National Library of Medicine

2 Outline Biomedical Literature Access Challenges in Literature Search User interactions with PubMed Author name disambiguation Click-words identification Disease name recognition Relationship extraction

3 Biomedical Literature Access Welcome to PubMed PubMed comprises more than 20 million citations for biomedical articles from MEDLINE and life science journals. Citations may include links to full-text articles from PubMed Central or publisher web sites. 20 million citations 5,200 journals Diverse topics: microbiology, delivery of health care, nutrition, pharmacology, environmental health and more Categories: anatomy, organisms, diseases, psychiatry, psychology physical sciences and more

4 Word Weight PubMed abstracts Blast Nucleotide sequences Protein sequences Blast

5

6 PubMed: the busiest NCBI database

7 Daily PubMed Usage PubMed Queries Abstract Retrievals Full text Retrievals Average 2.2 Million 2.7 Million 900 Thousand Average/User Islamaj Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

8 Cumulative number of publications Number of publications related to breast cancer in PubMed 250, , , ,000 50, Publication year

9 Challenges in literature search Rapid growth of biomedical literature Breakdown of disciplinary boundaries Garg, et al., Lost in publication: Half of all renal practice evidence is published in non-renal journals, Kidney international, 2006

10 Goal: improving access to literature PubMed oriented projects: Log analysis of user search needs and habits Author name disambiguation Concept recognition for linking related data Methods development in text mining Natural language processing techniques Statistical and machine learning methods

11 Outline Biomedical Literature Access Challenges in Literature Search User interactions with PubMed Author name disambiguation Click-words identification Disease name recognition Relationship extraction

12 General user interactions with PubMed

13 PubMed log analysis First large-scale investigation of its kind Data: one month data including 23M user sessions Analyzed millions of queries and clicks Identified search topics in 10,000 queries Islamaj Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

14 User behavior characteristics Time on the site Number of searches Number of clicks on PubMed citations Number of full text requests Number of words in a query Types of queries Position of clicks Number of returned results

15 The first ranked search result is the most clicked March Islamaj 1, 2011Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

16 The first ranked search result is the most clicked (true for every page) March Islamaj 1, 2011Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

17 User click trend Most (>80%) clicks happened in top 20 positions.

18 PubMed search result size 9% 6% 15% 0 Result % % 1, ,000 10,000 18%

19 Users are less likely to click if number of search results is large March Islamaj 1, 2011Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

20 Users are more likely to reformulate their search if number of search results is large March Islamaj 1, 2011Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

21 Highlights of PubMed users search behavior Results of query analysis Average number of queries issued by a user per day 4.05 Average number of words in a PubMed query 3.54 Median number of citations returned per query 44 Results of click through analysis Queries that do not retrieve any results 15 % Queries that were followed by another query 47 % Abstract views followed by full text of the same article 29 % Average number of abstract or full text articles requested (clicked) by a user per day 3.57 Islamaj Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

22 Query annotation results (10,000 queries) March Islamaj 1, 2011Dogan R, et al. (2009) Understanding PubMed user search behavior through log analysis. Database. Vol. 2009: bap018;

23 Most common associations Categories Examples Author name + Citation Rezarta Islamaj Dogan, 2009 Disorder + Medical Procedure breast cancer mammography Disorder + Gene/Protein brca1 breast cancer Gene/Protein + Biological process nfkb activation Disorder + Chemical/Drug cold aspirin

24 Understanding User s Query Crucial for a search engine to address the information need of a user. Example: AUTHOR NAME DISAMBIGUATION

25 36% of PubMed queries contain an author name

26 Author Identification Articles sharing the same author name Individual authors

27 ,024 2,048 4,096 8,192 16,384 32,768 Average number of articles per author Million articles Million Unique Author Names 3.3 Million Author Names have multiple articles 100,000 10,000 1, Number of articles associated with an author name

28 Average Number of Articles per Author Number of authors 10,000,000 1,000, ,000 10,000 1, Number of articles associated with an author name

29 Identifying articles penned by the same author We designed a machine learning process to learn the difference between same author papers and different author papers in PubMed Author Name

30 Features Set (MEDLINE citation fields) For each pair of articles: Title of the article Abstract of the article Co-authors MeSH Terms First Author Affiliation Journal Name Publication Date Chemical compounds Grant information

31 Classifier Feature weight Feature Analysis

32 Author Identification B A C E D -3 Compute pair-wise score

33 Author Identification B A C E D Compute pair-wise probability

34 Hierarchical clustering A B C D E A B,C D E A,B,C D E

35 Average number of articles per author Original Namespace Estimated Individual Authors Space 100,000 10,000 1, Number of articles associated with an author name

36 Number of authors Original Namespace Estimated Individual Authors Space 10,000,000 1,000, ,000 10,000 1, Number of articles associated with an author name

37 Understanding User s Query Crucial for a search engine to address the information need of a user. Example: CLICK-WORDS IDENTIFICATION

38 56% of the queries in PubMed contain only content words

39 Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Islamaj Dogan R and Lu Z. (2010) Click-words: learning to predict document keywords from a user perspective. Bioinformatics 2010

40 Users generate/build signals for the papers they access Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Query Words Islamaj Dogan R and Lu Z. (2010) Click-words: learning to predict document keywords from a user perspective. Bioinformatics 2010

41 Author keywords: Arp2/3 complex cofilin FRAP lamellipodium migration MeSH terms: Actin Capping Actin-Related Protein 2-3 Complex/metabolism* Actins/metabolism* Adaptor Proteins, Signal Cell Line, Tumor Cortactin/metabolism Mice PMID: arp cortactin actin cofilin Click words: actin arp2 cofilin cortactin Top five TF-IDF words: arp2 actin cofilin lamellipodium complex

42 TF-IDF words TF-IDF is a statistical measure used often in information retrieval to identify the most relevant terms of a document compared to the whole collection We can use the TF-IDF weight to select the most important words for an article

43 Data Set Training Data Set Evaluation Dataset PubMed Log Data Two months Six months User queries 100 Million 333 Million Abstract clicks 130 Million 329 Million User sessions 51 Million 144 Million PubMed articles 47,609 11,237 Total click-words 101,377 22,663 Top five TF-IDF words 237,155 62,310 Click-words/article Highly accessed article: Received on average one click per user per day

44 Data Set Training Data Set Evaluation Dataset PubMed Log Data Two months Six months User queries 100 Million 333 Million Abstract clicks 130 Million 329 Million User sessions 51 Million 144 Million PubMed articles 47,609 11,237 Total click-words 101,377 22,663 Top five TF-IDF words 237,155 62,310 Click-words/article Random baseline: Break-Even precision recall point: 0.429

45 Learning Method We built a machine learning model to identify users click words from top weighted TF-IDF words Wide margin classifier with Huber Loss function 5-fold cross validation Evaluation: Break Even precision recall Ranking analysis for each article

46 Word itself is not enough We could built a learning model using only the words, but this would not be sufficient, New words in new articles would be a problem The set of words that appears as a click for one article but not for another article would be confusing We would not be able to rank articles based on solely this feature We need context

47 Click-word features Word and its neighbors Part of speech tag (MEDPOST) MetaMap semantic type Location in the abstract Part of phrase Abbreviation TF-IDF rank

48 Training dataset 5-fold Cross Validation Evaluation dataset top 5 TF-IDF words Classification Model Mean Average Precision Break- Even Precision Recall ROC Prec@1 Random selection TF-IDF weight Click word Model Random selection TF-IDF weight Click word Model

49 Click word A content word is a word in a query that results in a click for the article Automatic identification important for effective document retrieval User choice: the word that most users prefer to access a particular article Increases chances that an article receives better visibility.

50 Click-word characteristics Have high TF-IDF weight Occur several times Appear in Title Are Nouns and Verbs Are part of phrase Are meaningful concepts Are abbreviated terms

51 Neighbor words L3 L2 L1 WORD R1 R2 R3 Management Background Treatment Diagnosis Patients Chronic Background Role Management Stem Breast Acute Human Factor Virus Syndrome Cancer Receptor Family Infection Related Signaling Deficient Regulates Cells Virus Syndrome Cancer Receptor Family Infection Review Clinical Nf Micrornas Including Agents

52 Outline Biomedical Literature Access Challenges in Literature Search User interactions with PubMed Author name disambiguation Click-words identification Disease name recognition Relationship extraction

53 Understanding User s Query Crucial for a search engine to address the information need of a user. Example: DISEASE NAME RECOGNITION

54 20% of the queries in PubMed contain a disease name

55

56 Text retrieval in clinical data A very challenging problem Training data: 349 patient records Testing data: 477 patient records Data comes from i2b challenge Annotated for concept and relationship Three different hospitals

57 Extracting medical concepts from patient records Medical Concept Problem Treatment Test Example Sentence On admission, the patient was found to have a mild fever, myalgias, and arthralgias that were relieved by Tylenol. Infectious Disease was consulted and recommended doxycycline to cover both organisms. Pending labs included wound, bacterial, and fungal cultures and serologies for Bartonella, Francisella, Yersinia, EBV,

58 Example: There was some concern that the patient may have a partial biliary obstruction and the patient was sent for a magnetic resonance cholangiopancreatography to further evaluate the biliary system. TEST Conducted for PROBLEM

59

60 Concept recognition Concept Exact span evaluation Inexact span evaluation Precision Recall F-measure Precision Recall F-measure Problem Treatment Test Overall Best i2b2 system

61 Relationship identification model Representation: Five, not necessarily consecutive, context-blocks. Separate bag-of-words models. Context-blocks SVM features: Assertion UMLS concept identifiers UMLS semantic types Baseline: Naïve bag-of-words SVM model using the same features

62 Relationship identification Relationship Relates Conducted Reveals Given Not Given Improves Causes Worsens Naïve bag-ofwords SVM model Context-blocks SVM model Context-blocks + feature selection Features in the best model (Words +) CUI Assertion - CUI, Assertion, SemType CUI SemType CUI, Assertion, SemType Assertion

63 End-to-end model Results of relationship identification Annotated concepts Predicted concepts Prior feature After feature Prior feature selection selection selection Improves After feature selection Worsens Causes Given Not Given Reveals Conducted Relates Overall weighted average

64 Text retrieval in clinical data Why was the patient admitted to the hospital? What tests were done? Which ones were effective? Which medication worsened his condition? What problems were created as a result? What happens next?

65 Text retrieval in clinical data Who will follow the patients? Which medication is to be taken regularly? What dosage/duration? How can we use the knowledge in medical summaries to help doctors and nurses when they evaluate/treat new patients?

66 Question Answering Knowledge Discovery INFORMATION EXTRACTION SYNTHESIS

67 Summary PubMed Logs analysis is currently used to improve retrieval quality and direct future development of the site Click-words summarize the wisdom of the crowds. They are specific for every article, and can be predicted using the learned characteristics Disease sensor will be useful linking related disease resources in PubMed

68 Research interests Knowledge discovery on biomedical literature and other text resources Turning knowledge into health care: text mining on systematic reviews Identifying weak links in genomic sequences that affect function

69 Thank you

70 ACKNOWLEDGEMENTS This research was supported by the Intramural Research Program of the NIH, National Library of Medicine. W. John Wilbur Zhiyong Lu Lana Yeganova Won Kim Wanli Liu Don Comeau Natalie Xie Larry Smith Aurelie Neveol Craig Murray Sun Kim Vahan Grigoryan