SKOS Shuttle. (Welcome) Tutorial TEM Text Extraction Management. May 2018

Transcription

1 SKOS Shuttle (Welcome) Tutorial TEM Text Extraction Management May 2018 This tutorial illustrates How to extract in SKOS Shuttle new concepts out of free text and to add them to a thesaurus Table of Contents TERM EXTRACTION PROCESS... 2 UPLOAD CORPUS... 3 REVIEWING CORPUS DOCUMENTS... 8 EXTRACTING TERMS REVIEWING TERMS DELTA EXTRACTIONS USING TERMS IN THESAURUS RESTRICTIONS Semweb LLC Copyright All rights reserved 1

2 SKOS Shuttle provides you with a simple but powerful text engine where you can upload documents which are scanned for new terms which you can then decide to drag into concepts for a thesaurus. TERM EXTRACTION PROCESS This paragraph introduces you to the definitions used to work with text and with some theory elements of term extraction. (Free) text is (usually) enclosed in (text) documents (e.g. PDF or WORD). A collection of documents, usually containing text (possibly) in one language is called a (text) corpus. Once you have a corpus, what you might be interested in, is to use it as a basis for extracting terms. In SKOS Shuttle a text corpus is analyzed to extract (compound) terms up to 5 words each and to weight them according with the TF/IDF method. More in detail, TF is a short for term frequency IDF is a short for inverse document frequency From Wikipedia: TF/IDF is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf-idf value increases proportionally to the number of times a word appears in the document and is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general. Nowadays, tf-idf is one of the most popular term-weighting schemes; 83% of text-based recommender systems in the domain of digital libraries use tf-idf. SKOS Shuttle will weight each term using separately tf and idf values. The process discussed here can be factorized into basically 5 steps: Upload corpus Review corpus Extract terms Review terms Use terms in thesaurus In SKOS Shuttle a corpus can be a file, a zip file (to upload) or a URL with a depth radius (to crawl). Since SKOS Shuttle uses Apache Tika to read documents, the (text) documents understood by SKOS Shuttle are exactly those processed by Tika, e.g. PDF, EXCEL, WORD, XML. Since a corpus is normally related to a language (e.g. English), SKOS Shuttle uses Tika s language recognizer to automatically detect the language of each uploaded document and activate / deactivate it according with its language matching 1. The returning arrows should indicate that the next step can make necessary a change in the previous one, e.g. reviewing the corpus might induce you to rebuild the corpus using some modified parameters. Of course several steps can be jumped to generate more effectively a better corpus. 1 For instance, for an English corpus, English documents will be activated, documents with another recognized language deactivated. Semweb LLC Copyright All rights reserved 2

3 UPLOAD CORPUS As introduced in the previous chapter, a corpus is either a zipped collection or a collection of web pages. Start the TEM module by clicking on the orange menu bar 2. An (empty) list of corpora is displayed (Picture 1) where you can start adding and configuring your corpora. Picture 1: Empty list of corpora The tab Collections contains the corpus management itself. The tab Extractions shows the extracted terms, once you started the term extraction function. Create a new corpus by clicking on the + sign on the right of the word Corpora. A new corpus record is created (Picture 2) and opened in edit mode. 2 If clicking on that icon just a description of TEM appears, this means your user has not been enabled for TEM. Ask Semweb to do it. Semweb LLC Copyright All rights reserved 3

4 Picture 2: Editing a corpus (zoom to better see details) In short: The corpus record allows you to define name, language and definition in a first shot (pls. save!) Then, it shows you the corpus status e.g., the number of active / not active documents in the corpus. The further section 1. Load Corpus (URL or File) allows you to upload your text corpus. Once you have a corpus with a good status (e.g. with some enabled documents) you can associate (2.) one or several thesauri to that corpus. Furthermore, launch the term extraction by pressing the button Extract terms in section 3. Launch Analysis. Section 4. allows you to configure periodic crawling actions based on a URL and stored corpus parameters. Pls. refer to another tutorial to see how 3. Works. Start the edit by pressing in section 1. and remember to press the save button to keep your corpus data stored once you think you are done. In Detail: Section Name of Corpus Give here a name to remember the function of that corpus. Once you gave a name you can already save the corpus record for further use. Section Language Here you can select the language of the corpus to be used to determine which documents should be active. Currently the following languages are foreseen to be processed in SKOS Shuttle: de en es fr German English Spanish French Section Description Give here a verbose description, why you created this corpus (for later). Semweb LLC Copyright All rights reserved 4

5 Section Status The status of the corpus is an indication, how many documents are currently (up)loaded and which of those documents are active. Active documents participate to the term extraction (see below). The display of this section is changed / updated after each document upload or each crawl. Section 1. Load Corpus (URL or file) This section allows you to locate and upload your text corpus and to specify the following details. New corpus: Check if you whish to see a separate extraction. If unchecked, the extracted terms will be added to the latest extraction. Compounds: instruct SKOS Shuttle to construct compound terms with up to 5 substantives. Min.w.len: minimum length of a term (word) to be extracted URLs or s in documents Nowadays a document can contain several URL s or addresses which are de-facto non-terms. SKOS Shuttle let you choose among three options, how to process them: as text a URL / address is treated as a term discarded a URL / address is recognized and deleted from document text (forgotten) collected a URL / address is deleted from document text and collected in a separate file. Stopwords file Stopwords are words which you want to delete from document text. However they can be parts of accepted compound terms. Therefore, stopwords are still useful. You can upload here your stopwords file (one line, one word) and decide whether or not that file be used in one extraction (later). The stopwords file is used by SKOS Shuttle to eliminate them in case they come up as single terms in the extraction. Method a) Zip - Upload Once you have a single document ora zip 3 folder with your documents, locate the file on your file system using the button in the section Load Corpus at bottom position and press the orange button to load the corpus. After a short while SKOS Shuttle will write below the button the result it could accomplish uploading those documents. (Picture 3). Picture 3: After uploading a zip file with a corpus The following document types can currently be uploaded (and processed) according to TIKA 1.17: 3.zip,.Z,.gz are valid zip extensions. Semweb LLC Copyright All rights reserved 5

6 Method b) Crawling the web If the text corpus should be gathered from the web, SKOS Shuttle allows you to provide a URL (in the topmost field) and some recursion control parameters to limit the crawling process. Crawling the web can result in a huge amount of visited documents (lots of time and space), therefore it must be limited using a depth and a limit. Method b) can be configured to be run automatically by SKOS Shuttle (see section 4.). When you hover on the URL field, further crawl configuration fields (see below) are displayed. The depth defines how deep the crawler is allowed to go starting from the URL, the limit should prevent the crawler for each URL to crawl an indefinite amount of documents. Non-text-document like javascript, images, videos or css files should be excluded from the crawling process. As soon as you hover on the URL field, further fields appear: Depth TopN4D DOC names blacklist DOC names whitelist DOC content filter How should text be filtered defines how deep SKOS Shuttle should crawl defines the limit of crawled documents for each URL Defines regular expressions to be matched in order not to load a document with a matching URL (each line one expression) Defines regular expressions to be used to reprioritize documents in the limited crawling process on the basis of their matching URL s (each line one expression) Defines regular expressions to be used to reprioritize documents in the limited crawling process on the basis of their matching content (each line one expression) String of characters to be eliminated during the first text extraction (after Tika) in order to get cleaner terms. Picture 4: Crawling a corpus Picture 5: Crawling results Picture 4 gives an example of parameterization to crawl a website using a depth of 1 and a limit of 10 documents each opened URL. Furthermore, the DOC names blacklist contains some expressions to avoid crawling pages, whose URL s match one of those expressions. DOC name whitelist reprioritize those pages having e.g. linkedin.com in their URL and from those pages, only those should be (up)loaded, which contains the term ontolog (ontology, ontologies) in their text. The Crawl result is shown in Picture 5 below the button with the label Press here to upload corpus. Note: You might get another crawling result (more or less documents) if you try this example at another time. Crawling depends also on how the web is currently organized (e.g. delays on page reads). After you crawled / uploaded your text corpus, you press the save button and consider Picture 7. Semweb LLC Copyright All rights reserved 6

7 Section 2. Associate Thesauri Is one or are more than one thesauri associated, SKOS Shuttle will check extracted terms against that or those associated thesauri. Extractions will then contain only terms, which were not found in none of the associated thesauri. To associate thesauri with the corpus or to change the assotiacion, simply click on the symbol inside the corpus record. A small selection dialog is displayed (picture xxx). Select (or deselect) there one or more thesauri and confirm. Picture 6: Thesaurus Association Panel In our example we will use corpus documents in the medicine and will select MeSH (Medical Subject Headings Thesaurus) as associated thesaurus and confirm the selection. The result is a corpus record (Picture 7) which is bound to that thesaurus. Semweb LLC Copyright All rights reserved 7

8 REVIEWING CORPUS DOCUMENTS Picture 7 shows a corpus record having 4 enabled documents. Picture 7: Uploaded text corpus Once you have created a text corpus, you can proceed via to review it (Picture 8). Picture 8: Reviewing corpus documents In the review panel a list of uploaded corpus documents is displayed together with their recognized languages and some control elements to activate/deactivate them from being taken as basis for a term extraction (see below). On this panel documents can be viewed, selected, deleted, filtered in order to quickly adjust your extraction basis. The upper right section of the panel shows in a more comfortable way the same black/white lists as in the corresponding corpus record of Picture 4. The review panel (below) allows you also to see/read the single document. Click on the link to see the document in an iframe on the right, on D to see the document in the browser 4 and on T to see its extracted text. Picture 9 and Picture 10 show a document view (original and text). 4 provided you have predisposed your browser to render document formats inline, or a tab is opened. Semweb LLC Copyright All rights reserved 8

9 Picture 9: Reviewing original document Picture 10: Reviewing text of document Reviewing a text corpus can lead to delete or deactivate documents, adding more expression to e.g. a blacklist, and finally to re-crawl the web content with the adapted parameters. Semweb LLC Copyright All rights reserved 9

10 EXTRACTING TERMS Once you have a valid corpus with at least one active document, the button Extract terms in the section 2. Launch Analysis gets orange. This means, that you can now start a term extraction (Picture 11). The only two parameters to set (or to be aware of) before launching a term extraction are New extraction: a fully new dated (basis) extraction is built. If not, terms are added to the latest one. Delta extraction: terms are inserted only if not already present in the basis extraction. Case sensitive: terms are compared in lower case, discarding doubles which differ only by case. Update tf/idf value: once you modify the corpus, the computed tf/idf values of earlier extractions become invalid. Checking this box will tell SKOS Shuttle to recalculate the values for every extraction 5. Picture 11: Term extraction results Picture 11 shows a tiny statistic done by SKOS Shuttle at the end of the launched extraction. In this case, 1234 terms were extracted (including combinations in compound terms having max 3 words each). Since it is the first extraction at all, no terms were updated. See available extractions by pressing the button See extraction or by changing in the tab Extractions (Picture 12). Note: 1. If you associate thesauri with a corpus and start an extraction, SKOS Shuttle will automatically fetch all (current) concept labels from the associated thesauri (via SPARQL), in order to exclude extracted terms found in the associated thesauri. These labels will then be stored in a one day timed cache in SKOS Shuttle, so within that day all launched extractions will find the thesaurus labels stored in the cache and the extraction will take just the quick text processing time The extraction statistic here said, that out of 189 extracted terms, 40 terms were found in the MeSH thesaurus (plausible outcome). 5 In Version this feature is not implemented. 6 As an example, to fetch labels are downloaded and stored in ca. 3 minutes. Semweb LLC Copyright All rights reserved 10

11 REVIEWING TERMS Picture 12: First segment of 553 extracted terms The Extraction tab shows two lists. On the left a list of (extraction) dates and on the right the terms of the selected dated extraction with its term suggestions. From here you can review your terms, delete or modify them. Terms can be as well sorted using the TF (sort by TF values), IDF (sort by IDF values) or Extracted terms (sort alphabetically). A filtered selection of terms can be visualized using the term filter. Picture 13: Sorting / filtering controls To modify a term, click on the term itself. A small edit field appears. Modify and save using depicted in Picture 14. button as Picture 14: Editing a term Semweb LLC Copyright All rights reserved 11

12 DELTA EXTRACTIONS As mentioned before, a delta-extraction is an extraction created in relation to another extraction. SKOS Shuttle can create a basis extraction (a full one), then successively delta-extractions containing just the delta, e.g. the new terms extracted. Every delta-extraction relies on one-and-only-one basis- or deltaextraction in a chain. In order to show the power of delta-extraction, we consider crawling the following url s one after the other. It is left to you whether you let reconstruct a new corpus at each crawl run. 1) 2) 3) using Depth=1 and TopN4D=10, no blacklist, no filters. Every crawl run delivers 3 documents, including the document behind the URL. After each crawling please exclude (uncheck) in the review panel (shown in pictures 8 or 9) the two documents which are not the one with the crawled URL, leaving checked the document distinguished by the crawl URL (e.g. This will produce a good corpus with less noise and will help showing the advantage of delta-extractions. We repeat the operations Crawl and Delta-Extract for each of the URL s in 1-3 (above). The resulting pictures illustrate the outcome. Please note how many terms were excluded from the extraction because they were found in the MeSH thesaurus. Ignored terms are those which contained so many special characters, that they caused an internal problem and were then discarded. Semweb LLC Copyright All rights reserved 12

13 Picture 15: Crawling /environment/ and creating a basis extraction Picture 16: Basis extraction for /environment/ Semweb LLC Copyright All rights reserved 13

14 Picture 17: Crawling /health_medicine/allergy/ and creating a delta extraction Picture 18: Delta extraction for /health_medicine/allergy/ Semweb LLC Copyright All rights reserved 14

15 Picture 19: Crawling /health_medicine/ birth_control/ and creating a delta extraction Picture 20: Delta extraction for /health_medicine/birth_control/ Each delta-extraction shows just terms which are not in the previous extraction. A checkbox on a deltaextraction shows all terms of the current plus the underlying extractions. E.G. if you hover on the checkbox Semweb LLC Copyright All rights reserved 15

16 of the topmost delta-extraction (Picture 20), both underlying extractions will be highlighted. Clicking on that checkbox shows terms of all involved extractions. Picture 21: Delta extraction hovering on topmost checkbox highlights related extractions Picture 21 shows the result on clicking on that checkbox i.e., all terms in that chain of extractions. Picture 22: Delta extraction showing all terms in the extraction chain Delta-extractions can be used for easily discovering new terms while still preserving all terms. They are particularly important for periodic crawling (pls. see extra tutorial on this subject), since at each point in time a URL (or a set of URL s) can be crawled, new terms extracted and easily analyzed. Semweb LLC Copyright All rights reserved 16

17 USING TERMS IN THESAURUS Once one or more terms extractions are available, you can maintain your thesaurus by adding some of these extracted terms. To see the extracted terms with your thesaurus, open the terms panel by clicking on in the details tab while positioned on some concept in your thesaurus. Picture 23: Using terms to maintain a thesaurus Here the suggested terms appear as you already saw in Picture 12 but with an additional information: the term status. A small colored ball on the right of each term shows whether that term is already contained in the RDF material (labels, definitions, notes) of the current thesaurus. Green ball means: term is not (yet) used in thesaurus, you are free to use it, a gray ball means, that that term is already contained in the thesaurus. Hovering on a gray ball shows on a tooltip on which parts of the thesaurus that term is Semweb LLC Copyright All rights reserved 17

18 already contained / mentioned. Move the whole panel to a preferred place by dragging on the left side part of the panel (the part containing the extractions dates). How to add a term to the thesaurus to some field (labels, definitions, relations,... new concepts): 1) By normally typing the same text as the term in the appropriate place 2) By manually dragging and dropping the term into that place Note: after you have added a term to a field, the status of the term is updated. RESTRICTIONS Working with text corpora can be quite expensive in storage. Since SKOS Shuttle is basically a very low cost service, it must pose itself some limits in order to ensure everybody the use of this module. According to the used tariff, SKOS Shuttle allows its users to process at no cost or very low cost the following quantities of corpora: Tariff Maximal length of uploaded zip file Maximal (sum) of length of crawled documents Free 50MB 50MB Silver 100MB 100MB Gold 300MB 300MB Platinum 1GB (ca. 3.3x 300MB) 1GB Iridium 10GB (ca. 33 x 300MB) 10GB Onsite Unlimited Unlimited Notes: 1) Uploading a zipped corpus must underlie to the maximal uploading file limit of 300MB, however in case of more space you can upload several time different zipped corpora. 2) In case of exceeding tariff limit while uploading or crawling, SKOS Shuttle will gracefully stop uploading and issue a clear warning. 3) Documents with many pictures take unnecessarily lots of space, the less pictures, the better 4) Most efficiently the ideal upload consists of text files 5) Uploading or crawling a corpus as well as extracting and weighting terms are quite CPU-expensive operations. SKOS Shuttle protects itself from overloading by allowing a fixed number of such operations at the time among its users. Should you request an Extraction synchronically to other users, you might be warned to wait a little before you submit it again. Semweb LLC Copyright All rights reserved 18

19 Congratulations! You are done in this tutorial you learned how to use and understand SKOS Shuttle s TEM (Term Extraction Management). Questions? Improvements? Just contact us or on Semweb LLC Copyright All rights reserved 19