KMX Patent Analytics

Transcription

1 Training Exercises November 4, 2011 Project KMX Patent Analytics Author R. v.d. Berg, R. Argentini, I. Project Code A1 Timmermans Client Philips IP&S Document Status Final Date November 4, 2011 Document Version 3.0 Copyright c Treparel Information Solutions B.V., Delft, The Netherlands. All Intellectual Property Rights on the content of this document are explicitly reserved by Treparel Information Solutions B.V. No part of this document may be copied or made public by means of press, photocopy, microfilm audio or videotape or by whatever means, nor to be stored in an electronic retrieval system without prior written permission by Treparel Information Solutions B.V.

2 Contents 1 Import a document set Open the application Create a home folder Import the document sets Create a workspace Opening dataset Settings Create a binary classifier Create a binary classifier Improve the classifier Export the results Show classification statistics and ROC plot Perform a multi-class classification Perform a compound classification Separate one class into subclasses Examining the landscape view and brushing Using the Landscape view & Brushes to create a classifier Using coloring and filtering November 4, 2011 Treparel ii

3 Introduction R Important text is marked:r R Text displayed on the screen is shown in bold: text R Input, that is text you should type, is shown emphasized: text November 4, 2011 Treparel 1

4 1. Import a document set 1 Import a document set In this exercise you will learn how to import a document set into KMX Patent Analytics. Imported data sets can then be subjected to processing using text-mining techniques. 1.1 Open the application Start the application by clicking on the logo on the desktop or by selecting: Start All Programs KMX Patent Analytics EE KMX Patent Analytics. You will be logged on automatically. 1.2 Create a home folder After the login process is completed, the object browser will open. The object browser is a hierarchical file browser-like environment where all the datasets and classifiers are stored. Let us first create a folder to hold all of your data. Navigate to your folder (the folder named after you), by double-clicking folders or clicking the + in front of them. Make sure that your folder is expanded. Right-click on the name of your folder and select Create Folder. Enter Training and press enter. 1.3 Import the document sets We will now upload the data sets needed for this course. Select File Import data set (Excel and CSV) from the menu. Select the file Mouse vs Mouse.xls from your file system. You will be prompted for a place where the data set will be stored. Open your home folder. Select the folder labelled Training and press OK. After the file has been uploaded and imported, it will appear in your Training folder. Repeat the above steps for the other datasets: Infectious Diseases.csv Diagnostic Technology.xls Diagnostic Technology - X-ray.xls November 4, 2011 Treparel 2

5 2. Create a workspace 2 Create a workspace In this exercise you will learn how to create a workspace. A workspace is a preprocessed set of documents that can be used as input for text mining functions. A workspace is created from a data set. You have already imported all the data sets used in this training in the previous exercise. 2.1 Opening dataset To start creating a workspace, do the following: Open the folder Training. Open the data set Mouse vs Mouse.xls by either double-clicking it or selecting Open from the context menu. A new window will open titled Select a folder. workspace. Here we select where we want to save the Click on the folder Training. Press OK at the bottom of the window 2.2 Settings A new tabbed window opens titled Settings. Under tab Metadata fill in the Object Name: Mouse workspace Fill in the Purpose: training exercise Click on the tab labeled Features If necessary, expand the tree by clicking on the + sign Here we can select the sections of the documents that we wish to include in our text mining operations. For this exercise, select the document titles (stored in the column TIDW) and the patent claims (stored in the column CLAIMS). Select feature TIDW by clicking on it and then change the slider setting at the bottom from 0 to 1 by using the associated slider. If we were to set the weight to 2 for example, this feature would have a double weight. Do the same for CLAIMS: set the slider to 1 Let us adjust some of the text processing options: Click on the tab labeled Text Processing Verify that the language is set to English Words present in the blacklist are discarded when the documents are processed. The default blacklist settings are suitable for processing most English documents. Let s add a word to the blacklist just to see how it is done: November 4, 2011 Treparel 3

6 2. Create a workspace Patent claims have a tendency to start with the words what is claimed is. Place the cursor at the end of the blacklist and type the word claimed to add it to the list. Click on the tab SVM Parameters verify that the parameters are set respectively on 100 for the number of significant words per document and at 750 for the Feature Vector Length Click on Accept Your workspace is ready. You will find the workspace in the folder training. November 4, 2011 Treparel 4

7 3. Create a binary classifier 3 Create a binary classifier In this exercise we will create a binary classifier. This is a classifier that can discriminate between two classes. 3.1 Create a binary classifier Double click on workspace Mouse workspace that we just created. The workspace opens in a new window. In this exercise we will build a binary classifier that separates the patents concerning biological mice from those concerning computer mice. We will call the biological mice patents Positive and the others Negative. Set the label model to binary. You will find the selection box in the lower-right portion of the window. We will now label the documents by looking at the titles. R Label five biological mouse documents as Positive by clicking on the label in the label column. This will change the label from None into Positive. You will now notice a tick mark in the L column. This indicates that the document will be used as a Learning Document for creating the classifier. Label five computer mouse documents as Negative by double-clicking on the label column once. You can now create the first classifier. Make sure that the Classification window is shown (use the tabs at the bottom of the window). Click on the button Create Classifier. This might take a few seconds. At the bottom left side of the window, activate the option labeled Session Objects. When the classifier has been created, it will appear in this tab. It will have an automatically assigned name of the form svm_00000_rulesx. Let us rename the classifier into a more convenient name. Clicking on the classifiers with the right mouse button. Select the Rename from the menu. Type in the new name, e.g. class1test. Usually, there is a tick mark next to Also classify after training; this causes it to automatically use the newly created classifier to assign a score to all documents. However, we can now also use our classifier to classify the documents in the workspace manually. Select the classifier in the Session Objects panel and click on button Classify Now. November 4, 2011 Treparel 5

8 3. Create a binary classifier The program classifies the documents using the classifier you just created. This might take a few seconds. The classification process is finished once you see scores appearing in the Score column. These scores, between 1 and 100, indicate how well the documents match the classification. Click on the header of the Score column to sort the results in ascending order. Click again on the header of the Score column to sort the results in descending order. Select from the menu: View Frequency distribution plot or press on the Frequency distribution plot button on the toolbar. You will see a histogram showing the obtained classification results. 3.2 Improve the classifier We are now going to improve the classifier by making use of the suggestion system. R The KMX Patent Classifier comes with a suggestion system that helps you to improve your classifiers and your classification results by indicating documents that should be included in the classifier as additional learning documents. These documents have a tick mark under the column S, which stands for Suggestion. Find the documents with a tick mark in the S column, either by sorting the documents by score and scrolling half-way through the results or by ordering the documents by the S column itself. Look at the titles of these documents and label them either Positive or Negative. If you do not know the answer, leave the label as None. Create the second (improved) classifier by clicking on the button labeled Create Classifier. This might take a few seconds. You will find the new classifier in the Session Objects panel. Rename this classifier into class2test. Classify the documents using the class2test classifier. Verify that the Frequency Distribution histogram has changed. If you are uncertain about how to perform any of the above tasks, feel free to page back through this tutorial. You can repeat this process as many times as needed, using the suggestion system to identify documents that should be labeled to improve the classification results. Suppose we are happy with the second classifier. We now want to save the classifier so that it will be available in future sessions. R Session objects are NOT saved when the application is closed. This means that you should always save classifiers you want to retain for future sessions. Right-clicking the classifier in the Session Objects panel. Select the option Save Object. November 4, 2011 Treparel 6

9 A new window will open titled Select a folder. classifier. 3. Create a binary classifier Here we select where we want to save the Click on the folder Training. Press OK at the bottom of the window. This classifier is now saved permanently to this folder in the object hierarchy. Once saved, this classifier is moved to the Object Browser tab. Select the Object Browser tab. Open the Training folder. Select the class2test classifier. Press the Classify Now button. 3.3 Export the results It is often desirable to save the results of the classification operation for further processing. Press the Save results to CSV button on the toolbar or choose from the menu File Save results to CSV. Save the resulting file in a convenient place in your file-system. This file can be opened for example in Excel and can be used for further analysis. 3.4 Show classification statistics and ROC plot After a binary classification, the classification statistics in the view menu are available. The classification statistics are only available in binary labeling mode. A ROC plot is constructed based on cross-validation, the user no longer has to define a separate test set. R Cross-validation requires at least 10 labeled training documents, of which at least three positive and three negative. Binary classification statistics (Precision, Recall, F1 and the ROC plot) are computed on the basis of five-fold cross-validation. This means that the training set is divided into five "packages" of approximately equal size. One package is kept aside while a classifier is built on the documents contained in the other four. The documents in the hold-out package are classified using this classifier. The process is then repeated for all packages, yielding classification scores for all documents in the training set. These scores are used to compute the classification statistics. Press the Classification statistics button on the toolbar or choose from the menu Analysis Classification statistics. This will present the ROC plot for these specific training documents. The blue line is the expectation value of the ROC plot. The area between the grey lines is the 95% confidence interval. The closer the grey lines are to the blue line the higher the certainty of the classification forecast. If the uncertainty margin is unacceptably large, you can reduce it by increasing the number of training documents. In addition to the ROC plot, the classification statistics window will also show other key performance metrics such as the precision P, the recall R and the F1 measure obtained for a certain threshold value. We are now done with the Mouse vs Mouse data set. November 4, 2011 Treparel 7

10 3. Create a binary classifier Close the workspace. November 4, 2011 Treparel 8

11 4. Perform a multi-class classification 4 Perform a multi-class classification We will now create a multi-class classifier. This is a classifier that can discriminate between multiple (i.e. more than two) classes. For this exercise we will use non-patent literature data extracted from the MEDLINE abstract database. Create a workspace from the Infectious Diseases data set using the procedure outlined in Exercise 1. Under the tab feature selection set the slider from 0 to 1 for TIDW, AB and AU Double click on workspace you just created to open the workspace window Make the Classification tab visible Set the Label mode to Free In this exercise we will look for documents related to the following three viral illnesses: the hemorrhagic fever Ebola, the pulmonary disease SARS and the lethal avian influenza variant H5N1. Let us start labeling the documents: First start with Ebola. Look for the documents that are dealing with Ebola by looking at the title and typing ebola in the label column. Label at least ten documents as ebola Label at least ten documents as sars Label at least ten documents as h5n1 To find new documents quickly you can use the Search function: Activate the search function by clicking in the keyword search window, at the top right. Type your keyword and press enter. The documents matching this query are shown, the rest is hidden. If the keyword occurs in the title, this will be highlighted. In the Full document tab, the keyword is also highlighted. You can use the obvious search terms ebola, sars and h5n1. You can also try and use the broader influenza to find more documents about h5n1. Click Create classifier to create a multi-class classifier, in this case for three classes Rename the classifier This will give you the classification result showing you for each document the score for the 3 chosen classes: Ebola, H5N1 and SARS. You can now see for example that certain documents have a high score for Ebola (>80) and a low score for SARS (<20). This indicates that this particular document deals with Ebola rather than with SARS. Some documents, such as abstracts of Preparing for a global pandemic editorials, can have high scores for all three classes. November 4, 2011 Treparel 9

12 4. Perform a multi-class classification Select from the menu View Frequency distribution plot or press the Frequency distribution plot button on the toolbar. The graph you see shows aggregated results for the three classes. You can improve the classifier by adding training documents. Improve the classifier R The suggestion system is not enabled for multi-class classification. This means that you will need to identify additional training documents to improve the classifier. Save the results to a CSV file. Close the workspace. November 4, 2011 Treparel 10

13 5. Perform a compound classification 5 Perform a compound classification In exercise 3 and exercise 4 we have seen how to create classifiers for binary and multi-class classification. R Multi-class classification is a convenient way to divide a data set into multiple classes, but it comes with two significant drawbacks: 1. The suggestion system cannot be applied to multi-class classification, making the iterative improvement processes more burdensome. 2. Documents labeled as positive for one class are automatically assumed to be negative for the other classes. This is not always strictly the case. Even where this is the case it makes it impossible to optimize the classification of a certain class while keeping the results for the other classes constant. For cases where accurate independent multi-class classification is required, KMX Patent Analytics has the option to combine multiple binary classifiers into a single compound multi-class classifier. In this exercise we will create a compound classifier to classify patents regarding diagnostic technologies: ECG, ultrasound and x-ray. Create a workspace from the Diagnostic Technologies dataset Save the workspace under training Under tab Metadata fill in the Object Name, e.g. exercise 4 Under tab Feature Selection set ruler to 1 for fields: TIDW, CLAIMS Under tab Text Processing choose English Under tab SVM Parameters set number of words to 100 and feature vector length to 750 Press Accept Double click to open the created workspace. Set the Label Mode to Binary. Start with creating a binary classifier for the category ultrasound. Choose ten positive documents (use keywords like ultrasound and echo to quickly find matching documents) Choose ten to twenty negative documents (use keywords like x-ray and electro to quickly find matching documents) Rename the classifier to ultrasound Use the classifier to classify all documents in the workspace Improve the classifier by labeling the suggested documents Create a new classifier and rename it to ultrasound2 Classify the documents again and look at the results November 4, 2011 Treparel 11

14 5. Perform a compound classification If you are unsure about how to execute any of these steps, feel free to look at the description of exercise 3. We now need to clear the labels associated with the documents. Press the Clear labels button on the toolbar or use the menu: Edit Clear labels. Repeat the steps above to create the other two binary classifiers: Create a binary classifier for X-Ray Create a binary classifier for ECG Now we can assemble the binary classifiers into a single compound classifier. First we create a folder to house the binary classifiers. Go to the Classification tab and select the User objects option. Create a new folder within the training folder by selecting the training folder, right click the mouse and choose option Create New Folder from the menu. Rename this folder to Diagnostics Classifier Then we save the classifiers into this folder. Go to the Session objects option. Save all three improved classifiers to the new folder Diagnostics Classifier by right clicking on the classifiers and choosing Save Session Object(s). We can now use the folder itself as a multi-class classifier: Go to the User objects tab. Select the folder Diagnostics Classifier. Click Classify Now. The documents in your workspace will be classified according to the created compound classifiers. Examine the results and save them in your training directory. We can display the classification scores for all documents for all classes graphically using a parallel coordinates visualization. Display the parallel coordinates visualization by selecting the menu option View Parallel coordinates or by pressing the Parallel coordinates button on the toolbar. November 4, 2011 Treparel 12

15 6. Separate one class into subclasses 6 Separate one class into subclasses One of the advantages of compound classifiers over multi-class classifiers is that they can be adjusted after creation. In the previous exercise we have created a single binary classifier for the x-ray class. However, upon closer inspection some documents turn out to be not strictly about the use of X-ray for diagnostic purposes, but about x-ray technology in general. In this exercise we will create an additional x-ray technology binary classifier and add it to our compound classifier. Create a workspace using the Diagnostic Technologies - X-Ray dataset. Use the same settings as we used for exercise 4. This data set contains only the x-ray documents from exercise 5. We created this data set by exporting the results of exercise 4 and throwing away all rows with scores for x-ray below 50. We will now create a binary classifier for X-ray technology in order to separate the X-technology documents from the X-ray diagnostics documents Choose ten positive documents concerning X-ray technology. Choose ten negative documents concerning X-ray diagnostics. Create the classifier. Rename the classifier to x-ray tech Use the classifier to classify all documents in the workspace Improve the classifier by labeling the suggested documents Create a new classifier Rename this classifier to x-ray tech2 Classify the documents again and look at the results These documents share quite a bit of vocabulary. Depending on the learning documents selected it is possible that you will require multiple iterations to achieve satisfying separation. If necessary, iterate again. We have now created a classifier to discriminate between x-ray diagnostics and x-ray technology. Save the improved classifier X-ray tech2 (or X-ray tech3/4 if more iterations were required) into the folder of the compound classifier that we created in exercise 4. Classify this workspace with the compound classifier Verify that we have a Score column for all four classes. Now we can apply this classifier to the original data set. Close the workspace. Open the Diagnostic Technologies workspace. Classify this workspace with the compound classifier Look at the results and save them in your training directory. November 4, 2011 Treparel 13

16 7. Examining the landscape view and brushing 7 Examining the landscape view and brushing The previous exercises have given us a thorough understanding of how to build binary, multi-class and compound classifiers. The common factor in building these classifiers was initial selection of training documents and then refine it based on the suggestion system or on user inspection. KMX Patent Analytics also supports landscaping, based on the Least Squares Projection technique. This technique attempts to retain the neighborhood relations of documents in high dimensional feature space when projecting these documents onto a two dimensional surface. Landscaping allows us to explore the dataset and see if the training documents we are chosing are actually suitable training documents or documents that could be considered noise or outlier documents. We start with the relatively easy Mouse vs Mouse dataset. Double click on workspace Mouse vs Mouse workspace created earlier (in exercise 2) to open it. The workspace window will now open. If the workspace window is opened for the first time no visualizations will be present. Generate a projection by either clicking the message... Click here to generate a projection, or pressing the Generate Projection on the toolbar, or by selecting Analysis Generate Projection from the menu. Figure 1: Projection generated from the Mouse vs Mouse workspace The resulting visualization will resemble the picture in figure 1 Each document is visualized as a small filled circle. You can zoom in by using the scroll wheel on your mouse or drag the visualization by pressing the scroll wheel. Some other options are present in the visualization context menu. As can be observed in the picture the data set appears to be distributed in two cloudlike clusters of documents. Let us examine the documents to see if the data in the two clusters are actually documents about biologicial or computer mouses. The term annotations already give us a first insight in the topics of the clusters. November 4, 2011 Treparel 14

17 7. Examining the landscape view and brushing Place the mouse pointer with the tip on a document and wait for the annotation window to appear (you do not have to click) The terms that appear in the annotation window represent the document ID (accession number), the title and the five most important stemmed terms representing the document. If these five terms are not enough to define the document as either biological or computer related you can turn on the full document view. Figure 2: Document view of a mouse patent Press the Full document button on the toolbar or by selecting View Full document from the menu. Use the mouse to click on a document. Examine the contents of the Full document view. A computer mouse patent is illustrated in figure 2. You probably notice that the crosshair changes. This happens because by clicking on the document you select it. This selection is propagated to all views that also contain this document. KMX Figure 3: The brushes window Patent Analytics supports multiple brushes. These brushes are managed in the brushes window, as illustrated in figure 3, this window shows the brushes that are present. You can add brushes and change the brushes names to your liking. A brush can be saved to file by selecting the Save brush option from the context menu of the brushes window. The resulting file will be a.brush file with the document-id of the brushed documents. This file can be loaded (Load brush) at a later point when the created selection is needed again. R The default mode is selection. You can brush documents by switching to a brush mode by selecting Brushes Paint (2), for example. R You can deselect documents in the current brush by holding the <CTRL> button and by clicking or dragging the mouse over the documents. You can clear an entire brush by using the context menu of that brush in the brushes window. Deselect all selected documents by using either method. Close the dataset view November 4, 2011 Treparel 15

18 8. Using the Landscape view & Brushes to create a classifier 8 Using the Landscape view & Brushes to create a classifier By using the mouse and the brushes we have an interactive way to visually identify and inspect groups of documents. We can use this interactive feature to create classifiers by selecting documents to label. Let us use this interactive feature to create a classifier for the Infectious Diseases dataset used in exercise 4. Open the Infectious Diseases workspace created in exercise 4. Make sure a projection exists (see exercise 7). Identify the most important clusters of documents Create a brush for each of the classes present (e.g. Ebola, SARS, H5N1) Select appropriate documents (e.g. five each) for each of the created brushes We can use the created selections to create a binary classifier for each of the classes. We do this by using the context menu-option label brush of the brushes window in combination with the Classification tab. If the type of classifier is set to binary the user gets a selection dialog to specify if the training documents are positive, negative or none (no training document). If the classifier-type is set to free the user gets a pop-up dialog to enter a label. The default label is the name of the brush. Select a brush, e.g. Ebola, and label this brush as positive Select the other two brushes and label these brushes negative Switch to the Classification tab and create a classifier Save the created classifier with a descriptive name, e.g. Ebola, in your training directory Normally we would use the suggestion system to optimize the classifier by finding additional training documents. For the purposes of this excercise we will use the unoptimized classifiers to save time. Repeat this process for the other two classes, H5N1 and SARS. We now have three binary classifiers one for each of the classes present. We use these classifiers to create a compound classifier. In the Classification tab switch to the User objects Create a folder, e.g. Infectious Diseases Compound, in your training directory Drag the created classifiers to this directory We can now use the folder as a multi-class classifier. Go to the User objects option Select the folder just created Click Classify now. November 4, 2011 Treparel 16

19 9. Using coloring and filtering 9 Using coloring and filtering By coloring and filtering the set of documents, we can highlight and isolate portions of the dataset for further examination and reporting. Open the Infectious Diseases workspace created in exercise 4. Classify the documents using the (unoptimized) compound classifier created in exercise 8. Let us see how the documents were classified. Click on the Landscaping tab Display the coloring frame by selecting the menu option View Coloring or by pressing the Coloring button on the toolbar. Enable coloring by checking Enable coloring by variable value in the coloring frame. Select the coloring variable Infectious Diseases Compound. The documents in the landscaping window will now be colored according to their classification using the compound classifier. Select the Color labels tab from the bottom of the coloring frame. The coloring frame will now show the possible values of the classification field (Ebola,H5N1 and SARS) and their associated colors. Select the Colormap tab from the bottom of the coloring frame. Choose a different color map (e.g. ice) and observe the result in the landscaping window. We are only interested in documents published before the first documented SARS case on 27 November Display the filtering frame by selecting the menu option View Filters or by pressing the Filters button on the toolbar. Press Add Filter to add an empty filter. Double-click on the empty Variable field in the new filter. appear. A drop-down selection will Select DA. This field encodes the publication date as an eight-character text string. Double-click on the operator field. Select the less than filter. Double-click on the value field. Enter the date " ". The documents about SARS will disappear from the landscaping window. We are also only interested in documents that mention the WHO (World Health Organization) in the abstract. Press Add Filter to add an empty filter. Select AB as Variable. November 4, 2011 Treparel 17

20 9. Using coloring and filtering Enter "WHO" as Value. Leave the Operator as contains. Let us remove the date filter: Select the first filter. Press Remove Filter. We can now save the landscaping window as an image suitable for reports and presentations. Make the Landscaping tab visible and right-click on the visualization. Select Save widget to file. Select a location, a name and a file format (PNG, JPG or BMP). Press Save. We can save the color labels to a separate file that can be included in the presentation or report as a legend. Select the Color labels tab from the bottom of the coloring frame. Right-click on the color labels. Select Save all Labels. Select a location, a name and a file format (PNG, JPG or BMP). Press Save. To include the images directly into a document (e.g. Microsoft Word or Powerpoint) we use the clipboard instead of saving the images to a file. Select the option Copy all labels to clipboard from the context menu of the Color labels tab, then paste the image at the appropriate place in your document. November 4, 2011 Treparel 18