Structured Element Search for Scientific Literature

Transcription

1 Structured Element Search for Scientific Literature Ryan McDonald Natasha Noy, Matthew Burgess, Magda Procopiuc George Brokos, Polyvios Liosis, Dimitris Pappas, Ion Androutsopoulos

2 What Scientists Rarely Do

3 What Scientists Do A Lot

4 What Scientists Do A Lot Rank/Extract Normalize/Harmonize Manual

5 Structured Element X Y A 1 3 B 2 4 Tables Figures Captions Equations References Etc.

6 Structured Element Search... Structured Component Database/Index Normalize/Harmonize

7 Why does Google care? to organize the world s information and make it universally accessible and useful.

8 Why does Google care? Research driven search: Scientists; Students; Journalists; Lawyers, Policy Makers; etc.

9 Why does Google care? Research driven search: Scientists; Students; Journalists; Lawyers, Policy Makers; etc.

10 Part 1: Layout parsing and SE Extraction X Y A 1 3 B 2 4

11 Intuition: Visual cues make detection obvious

12 Object Detection

13 Use Object Detection Ren et al go/faster-rcnn Render each page as flat image Use Faster RCNN to classify bounding boxes

14 Experiments Training/dev data: annotated ~11,000 pages with table, figure, captions Evaluation Used PDFFigures2 to pre-populate Biology; Physics; Neuroscience; Computer Science; Economics; Random Only English 1176 pages of computer science (from PDFFigures2 paper) 847 pages from top conferences in different fields randomly samples from Google Scholar 1585 pages in German sampled from Lecture Notes in Informatics 473 pages in Chinese sampled from multiple journals (mostly information science) Metric - bounding box precision/recall Precision = intersection(gold_bb, pred_bb) / pred_bb Recall = intersection(gold_bb, pred_bb) / gold_bb

15

16 Parsing Tables

17 Pubmed OA Corpus M pairs

18 1. Extract Caption/Table pairs from PDFs using object detection Procedure 2. pdftotext/ocr to find words in caption bounding box Table II. TNF/LT- a Sensitivity of L929r2 Cells Transfected with neo r Alone, or Combined with TNF of LT- a Genes 3. Find table in XML whose caption has lowest edit distance <table> <caption> Table II. TNF/LT- a Sensitivity of L929r2 Cells Transfected with neo r Alone, or Combined with TNF of LT- a Genes </caption> <thead> <tr> <th> Sensitivity to </th> </tr> </table>

19 Related Work Heuristics Machine Learning He et al. 2017; Schreiber et al small studies on single domain Pubmed OA corpus Luong et al. 2012; Constantin et al use OCR features, incl spatial information Object Recognition / Deep Learning PDFFigures (Clark et al. 2015/2016) Lopez et al 2011 Praczyk and Nogueras-Iso 2013 Ammar et al training data for table/figure detection Parsing tables Little work on parsing tables from PDF/images From HTML a rich set of methods (Penn et al and others); tabular vs.

20 Part 2: Ranking Structured Component Database/Index

21 Document = text from structured element Caption OCR/PDFBox d figure/table info References in text to structured element

22 Ad-hoc Retrieval / Relevance Ranking Relevance Score Relevance Score Interaction-based Relevance-based Query Document Qu ery ent cum Do

23 (Deep) Ad-hoc Retrieval / Relevance Ranking Relevance-based DSSM (Huang et al. 2013) CDSSM (Gao et al. 2014) ARC-I (Hu et al. 2014) Interaction-based DeepMatch (Lu and Li 2013) ARC-II (Hu et al. 2014) MatchPyramid (Pang et al. 2016) DRMM (Guo et al. 2016) PACRR (Hui et al. 2017) DeepRank (Pang et al. 2017) Use Query-Doc term similarity matrices Similar to AoA, BiDAF, etc. for Reading Comp. QA

24 Deep Relevance Matching Model (DRMM) Guo et al 2016

25 DRMM Guo et al. Document-aware Term Encodings Histogram of bucketed cosine similarities between query term and document terms K buckest = Fixed-width input -- AGNOSTIC TO QUERY-DOC LENGTH We found this did not work at all -- very unstable Relevance Score Gating: Linear layer over IDF/w2v Term Score Aggregation Term Gating Works well CONS: Can t be trained end-to-end Not context-sensitive

26 Attention-Based ELement-wise (ABEL-DRMM)

27 POoled-SImilariTy (POSIT-DRMM) MULTI-VIEW -- Can also be done for ABEL-DRMM

28 POSIT Example

29 Training and evaluating a model Don t really have query -> SE data Test models using ad-hoc retrieval benchmarks BioASQ document ranking data query to title/abstract title/abstract length ~= caption/text length Trec Robust04 query to title/topic-description

30 Results: BioASQ 5B (batch 2-5, batch 1 as dev)

31 Results: Robust04 (5 fold cross validation)

32 BioASQ Challenge Can only return maximum 10 documents and 10 snippets Does not have to be 1 snippet per document Prelim results -- humans will judge all system outputs On dev data a microbiologist (my wife) says about ~50-60% of top 10 docs relevant Document Ranking (1st on 4 of 5 batches) F-measure MAP Our best Best non-aueb Snippet Selection (1st on 4 of 5 batches) F-measure MAP Our best Best non-aueb Only batches 2-5 MindLab and aueb-nlp-5 were not in batch 1

33 Summary Layout/table parsing Vanilla object recognition and encoder-decoder work Less brittle than analyzing parsed PDFs Ranking Carefully constructed deep models can be SOTA, even in small training regimes

34 Thanks!