Teamware: A Collaborative, Web-based Annotation Environment. Kalina Bontcheva, Milan Agatonovic University of Sheffield

Transcription

1 Teamware: A Collaborative, Web-based Annotation Environment Kalina Bontcheva, Milan Agatonovic University of Sheffield

2 Outline Why Teamware? What s Teamware? Teamware for annotation Teamware for quality assurance and curation Teamware for defining workflows, running automatic services, managing annotation projects Outlook 2

3 From Annotation Tools to Collaborative Annotation Workflows We have lots and lots of tools and algorithms for annotation; what we need is 1.methodological instead of purely technological 2.multi-role instead of single role 3.assistive instead of autonomous 4.service-orientated, not monolithic 5.usable by non-specialists GATE Teamware Research users in several EU projects External users at IRF and Matrixware Interest from other commercial users as well 3

4 GATE Teamware: Annotation Workflows on the Web GATE Teamware is: Collaborative, social, Web 2.0, has behaviour mining using Machine Learning Parallel and distributed (using web services) Scalable (via service replication) Workflow based with business process integration via BPEL 4

5 Teamware Layer Cake User Interface Layer Language Engineer User Interface GATE Developer UI Data Curation User Interface Document Browser Annotation Diff UI ANNIC UI Manual Annotation User Interface Schema Annotation UI Ontology Annotation UI Teamware Executive Layer Authentication And User Management Workflow Management Services Layer GATE Document Service GATE Annotation Services GATE Machine Learning API GATE Ontology Service 5

6 Division of Labour: A Multi-role Methodology (Human) Annotators - labour has to be cheap! Bootstrap annotation process with JAPE rules or mixed-initiative learning Curators (or super-annotators) Reconcile differences between annotators, using IAA, AnnDiff, curator UI Manager Defining annotation guidelines and schemas Choose relevant automatic services to pre-process Toolset including performance benchmarking, progress monitoring tools, small linguistic customisations Define workflow, manage annotators, liaise with language engineers and sys admins Sys admin Setup the Teamware system, users, etc. Language engineer Uses GATE Developer to create bespoke services and deploy online 6

7 Teamware: Manual Annotation Tool 7

8 Manual Annotation Process Annotator logs into Teamware Clicks on Open Annotation Editor Requests an annotation task (first button) Annotates the assigned document When done, presses the Finish task button If wants to save work and return to this task later Save button, then close the UI. Next time a task is requested, the same document will be assigned, so it can be finished Depending on the project setup, it might be possible to reject a document and then ask for another one to annotate (Reject button) 8

9 Hands-on Open a web browser and Teamware Login using you user name (from your reg.pack): <cics-account-id>-annotator Open the annotation UI Try requesting tasks, editing annotations, saving your work, asking for another task, etc. This is what Teamware looks like to a human annotator 9

10 Teamware for Curators Still being developed, so UI is in transition Identify if there are differences between annotators using IAA Inspect differences in detail using AnnDiff Edit and reconcile differences if required New curator UI in Teamware under development Currently available in Developer 10

11 IAA: Do my annotators agree? 11

12 IAA: Results 12

13 IAA: Recap The IAA on IE tasks, such as named entity recognition, should be measured using f-measure across all annotators For classification tasks, use Kappa to measure IAA For details, see the evaluation lecture and the GATE user guide 13

14 AnnDiff: Finding the differences 14

15 Where are these in Teamware? Only visible to curators and their managers Resources/Documents menu Select the corpus worked on Iterate through each document Run IAA and AnnDiff, as required Try for yourself: Login as <cics-user-name>-curator Corpus: annie-demo The first or second documents 15

16 Forthcoming curator facilities Have a corpus-level view of IAA Extended AnnDiff to allow easy reconciliation of the differences between 2 annotators Currently prototyped in Developer Will be made available in Teamware soon 16

17 New AnnDiff in Developer 17

18 Beyond Pair-wise Reconciliation AnnDiff only handles 2 sets of annotations at a time we often need more! Towards an in-place, content-based reconciliation interface 18

19 Current UI Prototype 19

20 Teamware for Managers Defining workflows Running annotation projects Tracking progress 20

21 Teamware Workflows Whole process is controlled by a workflow manager Workflow may be simple: Give the document to a human annotator Information curator checks a sample of documents for QC or more complex Invoke one or more web services to produce automatic annotations Pass each document to 2 annotators Information curator to quickly check level of agreement between the annotators and reconcile any differences Annotated documents used to train an ML model When model is good enough, start making suggestions to the annotators 21

22 Workflow Templates 22

23 Defining new workflows Select Projects/WF Templates Opens the WF wizard Choose which services you want to run Choose whether you want manual annotation, how many annotators per doc, 23

24 Setting up a Manual Annotation Project Upload the schemas Upload the documents Define the Workflow template Run the project, choosing the corpus, the annotators, curators, etc. DEMO! 24

25 Setting up an Automatic Annotation Project Configure the web service(s) Define the Workflow template Run the project, choosing the corpus DEMO! 25

26 Semi-automatic Projects Just combine the two sets of steps 26

27 Teamware: Monitoring Project Progress 27

28 Outlook Teamware is still under active development Many features subject to change If you d like further information or to try it with your data for a particular project, please contact Hamish and Kalina 28