Center for Reflected Text Analytics. Lecture 2 Annotation tools & Segmentation

Size: px

Start display at page:

Download "Center for Reflected Text Analytics. Lecture 2 Annotation tools & Segmentation"

Geoffrey Powell
5 years ago
Views:

1 Center for Reflected Text Analytics Lecture 2 Annotation tools & Segmentation

2 Summary of Part 1 Annotation theory Guidelines Inter-Annotator agreement Inter-subjective annotations Annotation exercise Discuss disagreements with your neighbor Improve annotation guidelines University of Stuttgart 2

3 Annotation Tool Support Tools can support the annotation process at various stages Managing multiple annotators Assign documents to annotate Supervise their progress Analyse disagreements Display disagreements (only) Calculate quantitative IAA (κ) Create a gold standard Make decisions on disagreements Record final decisions Usable tools: See handout University of Stuttgart 3

4 Segmentation University of Stuttgart 4

5 Segmentation Tool Download = University of Stuttgart 5

6 Segmentation Abstract definition No meaning of a segment implied The task of separating a text into multiple parts ( segments ) Segmentation according to various criteria based on Structure (chapters, acts, letters, speeches) Linguistics (sentences, paragraphs) Narrative content (scenes, time, place) Content (topics under discussion) No generic criterion covering multiple research questions University of Stuttgart 6

7 Segmentation Viewpoints Focus on segments Spans of text Focus on segment boundaries Positions in a text Views are equivalent we will switch between them when appropriate Segment 1 Segment 2 Segment 3 Segment 4 Segment Boundary Segment Boundary Segment Boundary University of Stuttgart 7

8 Entities + Segments = Networks Mary Peter Paul Co-Occurrence Network University of Stuttgart 8

9 Entities + Segments = Networks Slightly more abstract description Segmented text with the appearing entities {A, B}, {A, B, B, B, A}, {A, C} A B C Convert into an (quadratic) adjacency matrix Diagonal is typically uninteresting Matrix is symmetric A 2 1 B 2 0 C 1 0 Create network A node is created for each row (or cell) A An edge is created for each cell, weighted according to cell value B C University of Stuttgart 9

10 Segmentation Annotation Theoretically Segments can be annotated just like entity references Both cover sequences of words Appropriate annotation guidelines would define when to annotate segments Practically Segmentation criterion closely tied to research question No reasonable generic abstraction layer That works for multiple research questions and/or text corpora Single texts only contain a few segments Much more annotated texts needed for any kind of automatisation University of Stuttgart 10

11 Segment Annotation Tool Web-based UI Beta-Software Automatic annotation through rules and tools Entity annotation Stanford Named Entity Recognizer (Finkel et al., 2005) Only proper names, no descriptive noun phrases Rules (regular expressions) to specify the entity references Segment annotation Rules (regular expressions) to specify the segment boundaries Unsupervised segmentation algorithm (TextTiling; Hearst, 1994) Network export Gephi University of Stuttgart 11

12 Gephi Network Tool Free and open source Wide range of metric, filter and layout algorithms Network editing (e.g., merge nodes) Plugins Export into static images University of Stuttgart 12

13 demo University of Stuttgart 13

14 Regular Expressions Useful text processing skills 101 A powerful way to describe sets of character sequences Many search tools support REs, and all programming languages do Looks cryptic, but is quite systematic REs on slides/handout are marked in forward slashes / / for readability they don t need to be typed in the tool Basics Many regular characters stand for themselves The RE /a/ finds occurrences of the character a Sequences of characters stand for sequences of themselves The RE /the/ finds occurrences of the string the University of Stuttgart 14

15 Regular Expressions Basics Many regular characters stand for themselves The RE /a/ finds occurrences of the character a Sequences of characters stand for sequences of themselves The RE /the/ finds occurrences of the string the Meta characters ( quantifiers ) are applied on the previous character?: previous character optional (0-1 times) /them?/ finds both the and them +: Previous character one or more times /ab+/ finds ab, abb, abbb, The kleene star * finds the previous character zero or more times /ab*/ finds a, ab, abb, abbb, University of Stuttgart 15

16 Regular Expressions Alternations and Character Classes /(re1 re2)/ finds everything that finds either re1 or re2 /(good better best)/ finds comparative and superlative forms of the adjective good /great(er est)?/ finds comp. and sup. forms of great The question mark makes the suffixes optional We can mark alternatives on character level in square brackets: [ ] /[Tt]he/ finds upper and lower case forms of the Square brackets support ranges of characters /[A-Z]/ finds upper-case characters (beware: locale) /[0-9]/ finds digits University of Stuttgart 16

17 Regular Expressions Special cases and exceptions The dot. matches everything /a.*b/ finds everything that begins with a and ends with b Escape character: Backslash In order to find a dot, we need to prevent its special meaning /.*\.doc/ finds everything that ends on.doc (e.g., filenames) University of Stuttgart 17

18 Regular Expressions Real examples Chapter 10. /Chapter [0-9]+\./ Chapter V. (Roman numbers) /Chapter [IVXCM]+\./ Beware: Possible over-matching Dates: MAY 22., AUGUST 23. /[A-Z]+ [0-9]+\./ Beware: Possible over-matching University of Stuttgart 18

19 TextTiling Hearst (1994) Unsupervised segmentation algorithm, developed for expository texts Compares lexicon in a window left and right of a target sentence gap step size = 3 n n+1 sentence boundary window size = v 1 = v 7 2 = dist(v 2 1, v 2 ) = 0 9 d n University of Stuttgart 19

20 TextTiling Hearst (1994) Unsupervised segmentation algorithm, developed for expository texts Compares lexicon in a window left and right of a target sentence gap n n+1 sentence boundary d n d n+3 University of Stuttgart 20

21 TextTiling Hearst (1994) More powerful algorithms are available E.g., topic segmentation Clear adaptation possibilities How to create word vectors? Which words are included (function/content words)? Which value is represented in the vector (frequency, tf*idf, information, ) How to calculate similarity/distance? Cosine, manhattan, But: Evaluation is hard No gold standard available Different expectations University of Stuttgart 21

22 Hands-On Session 2 Go to Load a text of your liking (it s better if you are familiar with it) Add entity references by applying the Stanford NER system Make a brief check, if the important entities are included ( Passepartout, for instance, is not) You can add specific names by specifying regular expressions Add reasonable segment annotations Export a GEXF file and load it into Gephi Play with various options and see how the network changes University of Stuttgart 22

A bit of theory: Algorithms

A bit of theory: Algorithms There are different kinds of algorithms Vector space models. e.g. support vector machines Decision trees, e.g. C45 Probabilistic models, e.g. Naive Bayes Neural networks, e.g.