Table Identification and Information extraction in Spreadsheets

Transcription

1 Table Identification and Information extraction in Spreadsheets Elvis Koci 1,2, Maik Thiele 1, Oscar Romero 2, and Wolfgang Lehner 1 1 Technische Universität Dresden, Germany 2 Universitat Politècnica de Catalunya, Spain

2 Spreadsheets in the Realm of Big Data IMPORTANT SOURCE OF INFORMATION Used in various domains Content generated by experts CHALLENGES No description of the structure High degree of autonomy Implicit Information Information Extraction: Data Structure Implicit Metadata RDF XML JSON DB 2

3 Proposed Solution A PROCESSING PIPELINE KDIR-IC3K 16 CCIS Book Series CAiSE 17 Classify Cells Revise Predictions Identify Tables Cell Features Region Features Heuristics Layouts 3

4 Proposed Solution A PROCESSING PIPELINE Classify Cells Revise Predictions Identify Tables Future Work Evaluation Dataset 4

5 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 5

6 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 6

7 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 7

8 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 8

9 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 9

10 Cell Classification* UNDERSTANDING THE DATA Work at the cell granularity: - Omit empty cells - Omit floating objects Use 5 layout categories (labels): Header Derived Metadata Data Attributes A B C D E 1 2 Number of Items (in Units) Sold per Region 3 Item 4 Location Adapter Total Mouse Monitor 5 VGA HDMI 6 Europe 7 Spain France Asia 10 China Vietnam Item Keyboard is omitted. Check next sheet. * Koci et al.: A machine learning approach for layout inference in spreadsheets. KDIR IC3K 16 10

11 Supervised Classifications of Cells COLLECTING FEATURES Binary and numerical >200 Selected 43 for evaluation Random Forest CLASSIFICATION Evaluated 5 classifiers Random Forest most accurate tree T cell features Type String Is Bold Font No Underline Center Aligned Default Fill Color Features Thick Left Border Not Aggregation Not Referenced Right is Empty Top Same Style Best Results for Data and Header 11

12 A PROCESSING PIPELINE Classify Cells Revise Predictions Identify Tables Future Work Evaluation Dataset 12

13 TIRS Framework* Table identification and Reconstruction HEURISTIC PROCEDURE Identify tables from the results of cell classification A procedure of 8 heuristics tasks Table 1 Table 3 Pre- Process TIRS Worksheet of Classified Cells Table 2 * Koci et al.: Table Identification and Reconstruction in Spreadsheets. CAiSE 17 13

14 Grouping Cells NOTIONS Work with larger structures Decrease complexity LABEL REGIONS (LRS) Adjacent Cells Same Label M H H H H H H H H A A D D D D B A D D D D B A A D D D D B A D D D D B M 14

15 Grouping Cells NOTIONS Work with larger structures Decrease complexity LABEL REGIONS (LRS) Adjacent Cells Same Label M H A D B M D B 15

16 Grouping Cells NOTIONS Work with larger structures Decrease complexity LABEL REGIONS (LRS) Adjacent Cells Same Label M H H H H H H H H A A D D D D B A A! D D D B A D! D D D D B A D D D D B M 16

17 Grouping Cells NOTIONS Work with larger structures Decrease complexity LABEL REGIONS (LRS) Adjacent Cells Same Label Rectilinear polygons M H H H H H H H H A A D D D D B A A! D D D B A D! D D D D B A D D D D B M 17

18 Grouping Cells NOTIONS Work with larger structures Decrease complexity LABEL REGIONS Adjacent Cells Same Label Rectilinear polygons MINIMUM BOUNDING RECTANGLE An approximation for LRs Extensively used for spatial problems (0,0) M H H H H H H H H A A D D D D B A A! D D D B A D! D D D D B A D D D D B M 18

19 Constructing Tables - Preliminaries CONSTRAINS FOR PAIRING HEADER WITH DATA 1) A Header is on the top of Data 2) The distance between them is the smallest 3) Their projections overlap significantly: D1 H overlap(projection D, projection(f)) max(projection_length D, projection_length(f)) > θ - Overlap ratio measures this significance - For the evaluation θ (empirically) set to 0.5 D3 D4 D2 19

20 Constructing Tables - Preliminaries CONSTRAINS FOR PAIRING HEADER WITH DATA 1) A Header is on the top of Data 2) The distance between them is the smallest 3) Their projections overlap significantly: H overlap(projection D, projection(f)) max(projection_length D, projection_length(f)) > θ - Overlap ratio measures this significance - For the evaluation θ (empirically) set to 0.5 D2 20

21 First 5 Heuristic Steps TABLE PATTERNS There are 5 patterns of tables Each corresponds to one heuristic task σ i N σ j M overlap(projection Hi, projection(dj)) max σ i N projection_length(hi), σ j M projection_length(dj) > θ H H 1 H 2 H D D One to One Many to One D H H 1 H 2 H break! D 1 D 2 D 1 D 2 One to Many Many to Many Breaker 21

22 First 5 Heuristic Steps TABLE PATTERNS There are 5 patterns of tables Each corresponds to one heuristic task A D A 1 D A 1 D 1 One to One A 2 Many to One A 2 D 2 Many to Many A 1 D 1 A D A D 2 One to Many Breaker! 22

23 Heuristics Steps 6-8 HANDLE UNPAIRED & REMOVE INCONSISTENCIES Assign regions In-between tables 23

24 Heuristics Steps 6-8 HANDLE UNPAIRED & REMOVE INCONSISTENCIES Assign regions In-between tables Detect overlapping tables 24

25 Heuristics Steps 6-8 HANDLE UNPAIRED & REMOVE INCONSISTENCIES Assign regions In-between tables Detect overlapping tables Give regions to nearest table 25

26 Heuristics Steps 6-8 HANDLE UNPAIRED & REMOVE INCONSISTENCIES Assign regions In-between tables Detect overlapping tables Give regions to nearest table 26

27 Evaluation: Table Orientations One Table Vertical Horizontal Both Precision 76% 62% 23% 45% Recall 83% 58% 24% 32% 27

28 Evaluation: Number of Misclassified Cells None Few(1-3) Bunch(4-7) Many(8- ) Precision 82% 56% 60% 53% Recall 82% 53% 53% 57% 28

29 Overall Results 65% 63% Precision Recall 29

30 A PROCESSING PIPELINE Classify Cells Revise Predictions Identify Tables Future Work Evaluation Dataset 30

31 Revising Classification Prediction* Load Standardize Re-label!! Identify * Koci et al.: Cell Classification for Layout Recognition in Spreadsheets. Chapter at Communications in Computer and Information Science (CCIS) book series. (Currently In Press) 31

32 Revising Classification Prediction M H H H H H H H H A A D D D D B A A! D D D B A D! D D D D B A D D D D B M A A D D D D A A! D D D A D! D D D D A D D D D M 32

33 Revising Classification Prediction IRREGULAR REGIONS HINT MISCLASSIFICATIONS A A D D D D A A! D D D A D! D D D D A D D D D M 33

34 Revising Classification Prediction IRREGULAR REGIONS HINT MISCLASSIFICATIONS Make all strictly rectangular Small regions are suspicious A A D D D D A A! D D D A D! D D D D A D D D D M?? Standardize!?? 34

35 Revising Classification Prediction IDENTIFY TRUE NEGATIVES A supervised machine learning task Use the context from the neighborhood???? Similarity Dissimilarity 35

36 Revising Classification Prediction IDENTIFY TRUE NEGATIVES A supervised machine learning task Use the context from the neighborhood Similarity Dissimilarity Influence? 36

37 Revising Classification Prediction IDENTIFY TRUE NEGATIVES A supervised machine learning task Use the context from the neighborhood Similarity Dissimilarity Influence Absence? 37

38 Revising Classification Prediction IDENTIFY TRUE NEGATIVES A supervised machine learning task Use the context from the neighborhood Similarity Dissimilarity Influence Absence? 38

39 Revising Classification Prediction IDENTIFY TRUE NEGATIVES A supervised machine learning task Use the context from the neighborhood!! 39

40 Revising Classification Prediction PREDICT TRUE LABEL (I.E., RELABELING) A supervised machine learning task Use the context from the neighborhood!! Similarity Dissimilarity Influence Absence 40

41 Revising Classification Prediction PREDICT TRUE LABEL (I.E., RELABELING) A supervised machine learning task Use the context from the neighborhood Similarity Dissimilarity Influence Absence 41

42 Revising Classification Prediction PREDICT TRUE LABEL (I.E., RELABELING) A supervised machine learning task Use the context from the neighborhood Similarity Dissimilarity Influence Absence 42

43 Revising Classification Prediction OVERALL PROCESS Load Standardize Re-label!! Identify 43

44 Revising Classification Prediction EVALUATION RESULTS Successfully identified almost half of misclassified (total 1,237) Relabeling approach needs further improvements Identifying Relabeling 44

45 A PROCESSING PIPELINE Classify Cells Revise Predictions Identify Tables Future Work Evaluation Dataset 45

46 The Evaluation Dataset THREE CORPORA OF SPREADSHEETS 288 Worksheets Tables FUSE ENRON EUSES TOTAL FUSE ENRON EUSES TOTAL Barik, T., Lubick, K., Smith, J., Slankas, J., & Murphy-Hill, E. FUSE: A Reproducible, Extendable, Internet-Scale Corpus of Spreadsheets. MSR 2015 Hermans, F., & Murphy-Hill, E. Enron's Spreadsheets and Related s: A Dataset and Analysis. ICSE 2015 Fisher, M., & Rothermel, G. The EUSES Spreadsheet Corpus: a Shared Resource for Supporting Experimentation With Spreadsheet Dependability Mechanisms. ACM SIGSOFT Software Engineering Notes 2005 (Vol. 30, No. 4, pp. 1-5) 46

47 The Evaluation Dataset ANNOTATION TOOL 47

48 A PROCESSING PIPELINE Classify Cells Revise Predictions Identify Tables Future Work Evaluation Dataset 48

49 Future Work GRAPH REPRESENTATION M H H A D B M D B D H Right-Left 0.2 D Top-Bottom 1.0 M 49

50 Future Work GRAPH REPRESENTATION M H split H A D B M D B D H D M 50

51 Future Work GRAPH REPRESENTATION M H H A D B M D B D H D M 51

52 Future Work Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function 52

53 Future Work Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Plugin Results Features Web Service 53

54 Future Work FUTURE WORK Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Plugin Web Service 54

55 Future Work FUTURE WORK Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Plugin Web Service 55

56 Future Work FUTURE WORK Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Cell classification using Recurrent Neural Networks posts/ Understanding-LSTMs/ 56

57 Conclusions & Future Work FUTURE WORK Table 1 Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Cell classification using Recurrent Neural Networks Schema Extraction from identified tables Table 3 Table 2 57

58 Conclusions & Future Work FUTURE WORK Table 1 Graph-based Table Identification (target ECIR) - Node clustering - Guided minimum-cut - Fitness function Demo: a plugin for Excel to extract information Cell classification using Recurrent Neural Networks Schema Extraction from identified tables Table 3 Table 2 58

59 Related Work 1. Chen, Z., & Cafarella, M. (2013, August). Automatic Web Spreadsheet Data Extraction. In Proceedings of the 3rd International Workshop on Semantic Search Over the Web (p. 1). ACM. 2. Eberius, J., Werner, C., Thiele, M., Braunschweig, K., Dannecker, L., & Lehner, W. (2013, October). DeExcelerator: A Framework for Extracting Relational Data from Partially Structured Documents. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management (pp ). ACM. 3. Adelfio, M. D. and Samet, H. (2013). Schema Extraction for Tabular Data on the Web. VLDB 13, 6(6): Cunha, J., Saraiva, J., & Visser, J. (2009, January). From Spreadsheets to Relational Databases and Back. In Proceedings of the 2009 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation (pp ). ACM. 5. Abraham, R., & Erwig, M. (2007). UCheck: A Spreadsheet Type Checker for End Users. Journal of Visual Languages & Computing, 18(1), Shigarov, Alexey O. (2015). "Table Understanding Using a Rule Engine. Expert Systems with Applications, 42(2): Swidan, A., & Hermans, F. (2017, June). Semi-automatic Extraction of Cross-Table Data from a Set of Spreadsheets. In International Symposium on End User Development (pp ). Springer, Cham. 59

60 Thank You!

61 Region Spatial Arrangements 61

62 Constructing Tables - Preliminaries CONSTRAINS FOR PAIRING ATTRIBUTES WITH DATA 1) Attributes are on the left of Data 2) The distance between them is the smallest 3) Their projections overlap significantly: overlap(projection D, projection(f)) max(projection_length D, projection_length(f)) > θ A D - Overlap ratio measures this significance - For the evaluation θ (empirically) set to

63 Constructing Tables - Preliminaries CONSTRAINS FOR PAIRING ATTRIBUTES WITH DATA 1) Attributes are on the left of Data 2) The distance between them is the smallest 3) Their projections overlap significantly: H overlap(projection D, projection(f)) max(projection_length D, projection_length(f)) > θ A D - Overlap ratio measures this significance - For the evaluation θ (empirically) set to