IBM Watson Application Developer Workshop. Watson Knowledge Studio: Building a Machine-learning Annotator with Watson Knowledge Studio.

Transcription

1 IBM Watson Application Developer Workshop Lab02 Watson Knowledge Studio: Building a Machine-learning Annotator with Watson Knowledge Studio January 2017 Duration: 60 minutes Prepared by Víctor L. Fandiño IBM Global Business Partners

2 Overview You can use Watson Knowledge Studio (WKS) to create a machine-learning model that understands the linguistic nuances, meaning, and relationships specific to your industry or to create a rule-based model that finds entities in documents based on rules that you define. To become a subject matter expert in a given industry or domain, Watson must be trained. You can facilitate the task of training Watson with Watson Knowledge Studio. With Watson Knowledge Studio you can deliver meaningful insights to users by deploying a trained model in other Watson cloud-based offerings and cognitive solutions, including AlchemyLanguage, Watson Discovery service and Watson Explorer. Watson Knowledge Studio provides easy-to-use tools for annotating unstructured domain literature, and uses those annotations to create a custom machine-learning model that understands the language of the domain. The accuracy of the model improves through iterative testing, ultimately resulting in an algorithm that can learn from the patterns that it sees and recognize those patterns in large collections of new documents. The following diagram illustrates how it works Based on a set of domain-specific source documents, the team creates a type system that defines entity types and relation types for the information of interest to the application that will use the model. A group of two or more human annotators annotate a small set of source documents to label words that represent entity types, words that represent relation types between entity mentions, and to identify coreferences of entity types. Any inconsistencies in annotation are resolved, and one set of optimally annotated documents is built, which forms the ground truth. Watson Application Developer Workshop 1

3 The ground truth is used to train a model. The trained model is used to find entities, relations, and coreferences in new, never-seen-before documents. Additionally, you can build a rule-based model with Watson Knowledge Studio. Watson Knowledge Studio provides a rules editor that simplifies the process of finding and capturing common patterns in your documents as rules. You can then create a model that recognizes the rule patterns, and deploy it for use in other services. Objectives The objectives of this lab is to provide an overview on how to build a machinelearning annotator in WKS, covering the following tasks: Create projects Create type systems Create document sets Add dictionaries Create tasks for human annotators Create dictionary-based annotators and machine-learning annotators Deploy the machine-learning annotator to AlchemyLanguage a compare the results Prerequisites In the Labs Preparation Guide: Getting Started with IBM Watson APIs & SDKs you have instructions to get an IBM Bluemix and Watson Knowledge Studio account. Also, you will need Postman for testing the deployed annotator. For this lab, use the latest version of Chrome or Firefox web browsers. For the best performance, use a screen resolution of at least 1024x1280. Note: in this lab you will be working in your own Watson Knowledge Studio instance with the administrator role (ADMIN). That means that you are the only member of the annotator component team. A real project always requires multiple human 2 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

4 annotators in addition to an administrator or project manager. In this case you will be performing all the tasks. Because you are the only human annotator, you will not be able to analyze inter-annotator agreement and adjudicate conflicts in annotated documents, which is always an important part of the annotator development workflow. For information about team members in Watson Knowledge Studio and how to create users, see the section Assembling the team in the WKS Knowledge Center. Creating a project A project defines all of the resources that are required to create a machine-learning annotator, including training documents, the type system, dictionaries, and annotations that are added by human annotators. For more information about project creation, see Creating a project. 1. Log in to Watson Knowledge Studio with your administrator ID 2. In the WKS main page, click Create Project. 3. Give the project a name. You cannot change the project name later, so choose a short name that reflects your domain content or the purpose of the annotator component. You can specify a longer description, which can be changed later. In this lab, we will name the project wadwwks 4. Identify the language of the documents in your project. The documents that you add to the project, and the dictionaries that you create or import, must be in the language that you specify. In this example, documents will be in English. The selected language cannot be changed latter 5. In the Component Configuration, leave the Default Tokenizer. The default tokenizer is more advanced than the dictionary-based tokenizer; it uses machine-learning to identify the tokens in the source documents based on the statistical learning it has done in the language of the source documents. It identifies tokens with more precision because it understands the more natural and nuanced patterns of language. The dictionary-based tokenizer identifies tokens based on language rules. The selected tokenizer cannot be changed latter. See Tokenizers for more details. 6. In the Project Manager Selection, you have the option to add project managers to the project (the administrator can add or remove project managers later by editing the project). Only the names of people that you Watson Application Developer Workshop 3

5 assigned to the project manager role from the User Account Management page for the instance are displayed. Since you have access only to a single user ID, no names will be available, so this entry will be empty 7. When you are ready, click Create. The project will be created and you will be directed to the project Type System configuration. To change the project description or add or remove project managers later, an administrator can edit the project. 8. Sample Screen shot 4 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

6 Creating a type system You will now learn how to import and modify a type system within Watson Knowledge Studio. You must create or import a type system before you begin any annotation tasks. See Type systems for more information about this topic. 9. Download the en-klue2.zip file to your computer. This file contains an example KLUE type system for English documents 10. Within your project, click Type System in the banner or the navigation menu. On the Type System page, click Import 11. Select the en-klue2.zip file from your computer and click Import. The imported type system is displayed in the table Watson Application Developer Workshop 5

7 entity types and 2,177 relation types should be imported. You can browse the type system. You can also edit an entity type. For instance, locate the MONEY entity type. In the Action section click Edit and in the Roles column delete the role AWARD. Click Save Adding documents for annotations After you finish making changes to the type system, you can begin adding documents to your project. You will now learn how to add documents to a project in Watson Knowledge Studio that can be annotated by human annotators. See Adding documents to a project for more information about adding documents. 13. Download the documents-new.csv file to your computer. This file contains example documents suitable for importing. 14. Within your project, click Documents in the banner or the navigation menu. On the Documents page, click Import Document Set 6 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

8 15. Select the documents-new.csv file from your computer and click Import. The imported file is displayed in the table. The imported document set should contain 14 documents. You can click the document set in the table to access a browse the content of each document in the set. They contain news about computing technologies and companies. At this point, as a Project Manager, you are now ready to divide the corpus into multiple document sets and assign the document sets to different human annotators. Since you are the only user in the instance, you will create a single annotation set. Creating annotation sets An annotation set is a subset of documents from an imported document set that you assign to a human annotator. The human annotator annotates the documents in the annotation set. To later use inter-annotator scores to compare the annotations that are added by each human annotator, you must assign at least two human annotators to different annotation sets. You must also specify that some percentage of documents overlap between the sets. Note: In a real project, you would create as many annotation sets as needed, based on the number of human annotators working in the project. In this lab, you will create just one annotation set that you will also annotate. For more information about annotation sets, see Creating and assigning annotation sets. Watson Application Developer Workshop 7

9 16. Within your project, click Documents in the banner or the navigation menu 17. Click Create Annotation Sets. The Create Annotation Sets window opens. By default, this window shows the base set (containing all documents), as well as fields where you can specify the information for a new annotation set 18. Select your name in the Annotator list and provide a name for the set. Notice that you could add more sets (and a human annotator for each one), which is a more realistic situation in a business environment. In the case of more than one set, the Overlap field specifies the percentage of documents in the base set to be included in all of the new sets, so they can be annotated by all annotators and you can compare the results. Since you only have one set, the overlap has no effect 8 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

10 19. Click Generate The new annotation set is created and now appear in the Annotation Sets tab of the Documents page. Adding a dictionary Dictionaries are used in WKS for pre-annotating text when creating a machinelearning annotator. You will now learn how to add a dictionary to a project in Watson Knowledge Studio. For more information about dictionaries, see Adding Watson Application Developer Workshop 9

11 dictionaries to a project. 20. Download the dictionary-items-organization.csv file to your computer. This file contains dictionary terms in CSV format, suitable for importing into a Watson Knowledge Studio dictionary 21. Within your project, click Dictionaries in the banner or the navigation menu 22. Click the Add icon to add a dictionary Note: Do not click the Import icon, which is used to import a dictionary that you want to use as-is. For the lab, you will create a new editable dictionary and then import terms into it. 23. In the Name field, type Test dictionary. Click Save to create the (empty) dictionary. The new dictionary is created and automatically opened for editing 24. In the dictionary pane, click Import. In the Import Dictionary Entries window, select the dictionary-items-organization.csv file from your 10 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

12 computer and then click Import. 24 terms in the file are imported into the dictionary. Each term represents an organization 25. Click Add Entry to create a new term. An editable row is added at the top 26. In the Surface Forms column, type IBM and International Business Machines Corporation on separate lines (when you begin to type a new surface form, a space is added below for an additional surface form). Leave the radio button next to IBM selected, indicating that this surface form is the lemma. In the Part of Speech column, select Noun. Click Save After you create a dictionary, you can use it to speed up human annotation tasks by pre-annotating the documents. Pre-annotating with a dictionary-based annotator You will now learn how to use a dictionary-based annotator to pre-annotate documents in Watson Knowledge Studio. For more information about preannotation with dictionary-based annotators, see Pre-annotating documents with the Dictionary pre-annotator. 27. Within your project, click Annotator Component in the banner or the navigation menu. You can see different ways to pre-annotate documents Watson Application Developer Workshop 11

13 28. Under the description of the Dictionary Pre-annotator type, click Create this type of pre-annotator. The Dictionary Mapping window opens. 29. The list of entity types you previously imported when creating the type system appears. You now have to associate each dictionary that you want the dictionary pre-annotator to use, with the entity type that matches the type of the dictionary terms. You must map at least one dictionary before you can run the pre-annotator. Map the ORGANIZATION entity type to the Test dictionary dictionary you created previously: Click Edit for the ORGANIZATION entity type name. Choose the dictionary from the list 30. Click the plus sign beside the dictionary name to add the mapping, and then click Save 31. Click Create and then select Create & Run from the drop-down menu 12 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

14 32. On the Run Annotator page, click the check boxes to select the document set that you created earlier in the lab (not including the base set) 33. Click Run The documents in the selected set are pre-annotated using the dictionary annotator you created. The annotator component is added to the Annotator Component page; you could later use the same annotator to pre-annotate additional document sets by clicking Run. Creating an annotation task In this section, you will learn how to use annotation tasks to track the work of human annotators in Watson Knowledge Studio. For more information about Watson Application Developer Workshop 13

15 annotation tasks, see Creating an annotation task. 34. Within your project, click Human Annotation in the banner or the navigation menu. On the Human Annotation page, click Add Task 35. Specify the details for the task: In the Title field, type Test. In the Deadline field, select a date in the future 36. Click Create 37. In the Add Annotation Sets to Task window, click the check boxes to select the document set you created previously. This specifies that the document set must be annotated by the assigned human annotators as part of this task. Remember that for this lab you only have one human annotator and the corresponding annotation set. In a real scenario, you will have multiple annotation sets assigned to different human annotators in your project 38. Click Create Task 14 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

16 39. Click the Test task to open it. You can use this view to view the progress of human annotation work, calculate the inter-annotator agreement scores, and view overlapping documents to adjudicate annotation conflicts Annotating documents In this section, you will learn how to use the Ground Truth Editor to annotate documents in Watson Knowledge Studio. For more information about human annotation, see Annotation with the Ground Truth Editor. 40. In the Test task you created in the previous section, click Annotate next to the Annotation Set 1 annotation set. The Ground Truth Editor opens, showing you a preview of each document in the document set. The Ground Truth Editor opens in a new browser tab, showing you a preview of each document in the document set Watson Application Developer Workshop 15

17 41. Scroll to the Technology - gmanews.tv document and click to open it for annotation. Note that the term IBM has already been annotated with the ORGANIZATION entity type; this annotation was added by the previous dictionary pre-annotator process. This pre-annotation is correct, so it does not need to be modified 16 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

18 42. You will now annotate a mention. Click the Mentions icon to begin annotating mentions. In the document body, select the text Thomas Watson 43. In the list of entity types, click PERSON. The entity type PERSON is applied to the selected mention 44. Click the Relations icon to begin annotating relations Watson Application Developer Workshop 17

19 45. Select the Thomas Watson and IBM mentions (in that order). To select a mention, click the entity-type label above the text 46. In the list of relation types, click founderof. The two mentions are connected with a founderof relationship 47. Click the Completed option from the menu, and then click the Save icon to confirm, and then click Close 48. In the list of documents click Submit All to submit the documents for approval. Once confirmed, you can see that the status of all documents is Completed Note: In a real project, you would create many more annotations and complete all of the documents in the set before submitting. 49. Close the Ground Truth Editor 50. Back in the Human Annotation Tasks window, click the Refresh button. You can see now that the Annotation Set 1 is in Submitted status 51. Mark the check box near to Annotation Set 1; you will see that an Accept and Cancel buttons appear. Click Accept. You have now promoted the 18 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

20 document set to ground truth Note: In a real situation you will have several annotation sets reviewed by different human annotators. You will have to compare their work to determine whether different human annotators are annotating overlapping documents consistently. In that situation, Watson Knowledge Studio calculates inter-annotator agreement (IAA) scores by examining all overlapping documents in all document sets in the task, regardless of the status of the document sets. The IAA scores show how different human annotators annotated mentions, relations, and coreference chains. It is a good idea to check IAA scores periodically and verify that human annotators are consistent with each other. For example, a human annotator could have defined the relation between IBM and Thomas Watson as founderof and another one as employedby. The IAA scores will reflect this situation that you will have to analyse and discuss with the annotators to adjudicate conflicts. This is something you can do in the annotation task. For this simple example in the lab, you are the single annotator and a minimum set of human annotations have been done, so no conflicts are present and the annotation set status is completed (the status should be In Conflict if any overlapping is detected when you select and accept the document sets). Creating and deploying a machine-learning annotator After you have promoted the documents to ground truth, you can use them to train the machine-learning annotator. When you create a machine-learning annotator, you select the document sets that you want to use to train it. You also specify the percentage of documents that are to be used as training data, test data, and blind data. Only documents that became ground truth through approval or adjudication can be used to train the machine-learning annotator. Watson Application Developer Workshop 19

21 52. On the Annotator Component page, click Create Annotator. On the Machine Learning Annotator pane, click Create this type of annotator 53. Select the document sets that you want to use for creating a machinelearning annotator. Click the check mark next to Annotation Set1. Use the default values for creating your testing, training, and blind data. Then, click Next. Accept to reuse the current dictionary mapping in the next window and click Train & Evaluate 20 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

22 Note: Training might take more than ten minutes, or even hours, depending on the number of human annotations and the total number of words across documents. 54. You are now back to the Annotator Component window where you can see the training progress of your annotator 55. After the machine-learning annotator is trained, you can export it or you can view detailed information on its performance by clicking Details. On the Model Settings tab, you have access to the Train / Test / Blind sets where you can view the documents that human annotators worked on. You can click View Decoding Results to see the annotations that the trained machine-learning annotator created on that same set of documents 56. On the Statistics tab, you can view details about the precision, recall and F1 scores for the machine-learning annotator. You can view these scores for mentions, relations, and coreference chains by using the radio buttons. You can analyse performance by viewing a summary of statistics for entity types, relation types, and coreference chains. You can also analyse statistics that are presented in a confusion matrix. The confusion matrix helps you compare the annotations that were added by the machine-learning annotator to the annotations in the ground truth. Watson Application Developer Workshop 21

23 Note: In this tutorial, you annotated documents with only a single dictionary for organizations. Therefore, the scores you see are 0 or N/A for most entity types. The numbers are low, but that is expected, because you did not do any human annotation or correction. 57. On the Versions tab, you can take a snapshot of the annotator and the resources that were used to create it (except for dictionaries and annotation tasks). For example, you might want to take a snapshot before you change the annotator. If the statistics are poorer the next time you run it, you can promote the older version and delete the version that returned poorer results. Also, if you want to make your annotator available to other Watson applications, you must create at least one version of the annotator. This allows you to deploy one version, while you continue to improve the current version. The option to deploy does not appear until you create at least one version 58. On the Versions tab, click Take Snapshot. Provide a description of the snapshot 22 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

24 59. Once the snapshot has been created, click Deploy in the Action section in the same line that the snapshot. Select AlchemyLanguage as the service to deploy the model to and click Next 60. In the Deploy Model window, introduce your AlchemyLanguage API key. This is the same one you used in the previous AlchemyLanguage lab (you can get it from your IBM Bluemix dashboard, where you should have your AlchemyLanguage service). Click Deploy 61. A confirmation window indicating the deployment to the AlchemyLanguage service has started will appear. In that window, you will have your model Id, the one you should use when invoking the Watson Application Developer Workshop 23

25 AlchemyLanguage methods with your annotator. Copy this model Id for use it later. Click Ok 62. You can check the status of the deployment in the Action section close to the snapshot. Depending on the model, the deployment can take some time to complete. Once you see that the status is available, you are model is ready to be used by the AlchemyLanguage methods Using the annotator in AlchemyLanguage (optional) After you train a machine-learning annotator, you can use it to pre-annotate new documents that you add to the corpus and you can make it available to other Watson applications, like AlchemyLanguage, Watson Discovery service or Watson Explorer. See Using the machine-learning model to learn how to deploy your annotators to these IBM Watson applications. Now that you have deployed your model, you can test it with AlchemyLanguage. Verify first that the deployment has finished. We are now going to extract named entities with the AlchemyLanguage GetRankedNamedEntities method. 63. Open a Postman session. Create a new POST HTTP request with the following parameters: 24 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio

26 Method Endpoint apikey outputmode POST <your api Key> xml emotion 1 sentiment 1 knowledgegraph 1 model text <your model Id> NCR, which counts later IBM Thomas Watson as one of its early employees, said its products and services account for more than $400 billion in annual commerce and 23 billion consumer selfservice transactions 64. Verify that in the Body section of the request, the option x-www-formurlencoded is selected 65. Click Send to make the request. You should get an XML output similar to the following one Watson Application Developer Workshop 25

27 66. IBM is recognized as organization and Watson as a person, the entities you trained your annotator for. 67. If you try the same request, but now removing the model parameter, you should realize that the entities extracted are much more generic 68. You can try the same exercise now with the AlchemyLanguage GetTypeRelations method. You will see that Thomas Watson is identified as founderof IBM. Try again removing the model parameter and compare the results End of Lab 26 Lab02 Building a Machine-learning Annotator with Watson Knowledge Studio