Text Analytics User Guide

Transcription

1 Text Analytics User Guide 13/09/ :35:00 Author: ALLEN Robin

2 Text Analytics User Guide i Changes to this document Date Modification 07/08/2018 Second draft of content up to start of examples. 31/07/2018 Cont. interface. 13/07/2018 Same formatting allowed as in IUCLID Functionalities. Started adding visual description of interface. Text-only examples. 18/05/2018 Text only 14/05/2018 First version.

3 Text Analytics User Guide ii Table of Contents Changes to this document... i Table of Contents... ii Table of Figures... iii Table of Tables... iii 1. Introduction Choosing search criteria Quick first search Technical aspects of searching Language analysis Language detection Stemming Stop words Search Phrases Searching IUCLID fields Searching IUCLID attachments Search options Search scope Highlight Total highlights Highlights length Collapse optional metadata Search results Search results The flat view (default) Search results - More options Search results More options - Hierarchy Search results More options - Document Search results More options - Dossier metadata Search results More options - Download (text) Search results More options - Download (original) Search results The aggregated view Show flat view for a single dossier Search options for the aggregated view Aggregate by Hyperlinks Export Worked examples of searches Simple keyword search (level 1) Intermediate level searching (level 2) Advanced search (level 3) - the query language of text analytics... 20

4 Text Analytics User Guide iii Table of Figures Figure 1: Example simple search of type (1)... 2 Figure 2: An example of a search result opened in the user interface of IUCLID... 3 Figure 3: The search options under the menu when the type of view is the default Flat... 7 Figure 4: Highlights of a search result... 8 Figure 5: Example of properties shown for a search result in the default (flat) view for a IUCLID attachment Figure 6: More options in the search result Figure 7: Example output of the option Hierarchy for a search result Figure 8: Selecting either the flat or aggregated views for search results Figure 9: Search results in the aggregate view, aggregated by dossier UUID Figure 10: Go back to the aggregate view after viewing flat results for a single dossier Figure 11: Aggregated view - options Figure 12: Copy a link to IUCLID, to share it with a colleague Table of Tables Table 1: Detectable languages... 4 Table 2: Supported file types of IUCLID attachments... 6 Table 3: The fields exported for a search... 19

5 Text Analytics User Guide 1 1. Introduction Text analytics (TA) is a search engine that searches the database of an installation of IUCLID. It provides complementary search functionality to that available via the standard IUCLID user interfaces. A search can read through nearly all data that a user can enter in to IUCLID. This includes data entered in to fields, and the content of attachments. The search results state where within the instance of IUCLID a match was found, and contain hyperlinks that open a java client user-interface of IUCLID at the search result. TA has been designed to return search results quickly enough to allow users to browse the data, honing search criteria as they go along. The search criteria accepted by the TA range in complexity from simple keywords, right up the use of the TA s own query language, for use when highly specific information is to be selected from large databases. The user interface of TA is viewed using a web-browser. Consult your local administrator for the address. The default addresses for the web-based interfaces of IUCLID and the TA, are shown below: of iuclid>/iuclid6-web of iuclid>/ta It is possible to have the standard IUCLID interface, and the interface of TA open at the same time Choosing search criteria There are three main ways of searching, which in order of simplicity are: 1. Enter text that is related to the intended search results, just as you would in an internet search engine such as Google. This approach is used in the example shown below; 2. Enter a search term in text, but also add extra criteria such as wildcards, Boolean logic, and/or groupings; 3. Use the custom query language of Text Analytics. The feature Query suggest can be used to help build a query. If you need to do only a simple search, or you are not that familiar with the terminology and data structure of IUCLID, start with method (1). If you have more specific information about what you are looking for, for example you want to exclude certain search results, try (2). In that case it is recommended that you check the syntax described later in this manual. Finally, if you are looking for something highly specific, or you want to control exactly what type of data is found, create your own query using the query language of Text Analytics (3). Bear in mind that the query language is specific to Text Analytics, and requires the ability to construct queries manually by referring to the documentation of the language in this manual. This process may be aided by the feature Query suggest which can be used to create a query, or parts of a query, which can then be refined, and/or combined with each other. The methods defined above (1-3) are described in detail in subsequent sections of this document. To help get you started, some examples are given below in order of increasing complexity.

6 Text Analytics User Guide Quick first search To get you started and to provide some context for reading the rest of the manual, this section contains an example of a search of type 1 as described above. It also shows how to view a search result in the user-interface of IUCLID. After going through this example, you can either jump straight to the examples later in this document, or continue reading, at least as far as the examples. To perform anything but the most basic of searches it is recommended to continue reading this manual. It will save you time compared to clicking around in the interface unaided. First, enter some search terms appropriate to your data in to the box that has a magnifying glass icon to its right, and then click on the icon. In the example shown below two words have been entered: skin irritation. The width of the interface affects the location of some fields. To see the same layout as the one below, reduce the width of the browser page to match it. Figure 1: Example simple search of type (1)

7 Text Analytics User Guide 3 Legend for Figure 1: 1. Enter search terms 2. Click on the magnifying glass icon 3. Note that the first search result is a field in IUCLID, as opposed to an attachment. 4. Note that the field is in an endpoint study record in section of IUCLID. The field is labelled Endpoint, under the heading Administrative data. 5. Follow this link to open the user-interface of IUCLID to display the parent IUCLID document of the search result, as shown below. Figure 2: An example of a search result opened in the user interface of IUCLID Legend for Figure 2: 1. The value in the field that matched the search; 2. The UUID of the parent document of the search result. Note that the document UUIDs match between the link in the TA, and the information panel in IUCLID. Also note that the dossier UUID in IUCLID matches with the value shown with a blue background in the interface of TA, which is also a link to IUCLID.

8 Text Analytics User Guide 4 2. Technical aspects of searching TA does not search the IUCLID database directly, but searches an index it creates periodically. The date and time of the most recent indexing is stated in the user interface above the search criteria box. In the example shown above it is :00. TA searches within only IUCLID dossiers. Therefore, for example, a substance dataset entity that is outside a dossier is not searched. A read-only copy of that same substance dataset within a dossier, is searched. A search can be conducted either within the values of IUCLID fields, or within the text of attachments. The default is to search both, but this can be changed from the search options, as shown in Figure 3. Aspects of the two types of searching are explained in the subsections below. Searches are insensitive to case Language analysis Whilst the TA is creating an index, it analyses the text, and carries out various transformations with the aim of making the searches faster, and more convenient. Some practical results of this are described below Language detection The language analyser tries to automatically detect the language in the indexed text using a set of rules that describe each language it knows. The detectable languages are stated in the table below. If the none of the languages in the table is detected, English is set by default. Detected languages are stored in the index, and are therefore searchable. Table 1: Detectable languages Language English German Spanish Danish French Swedish Finnish Greek Hungarian Italian Norwegian Portuguese Romanian Russian Code EN DE ES DA FR SV FI EL HU IT NO PT RO RU

9 Text Analytics User Guide Stemming The text is broken up in to individual words and stemmed, which means that once the language has been detected, the rules of that language are used to find the stems of the words. This is also applied to search terms, so the search automatically matches different forms of a word. For example, the stem of assessment is assess, so the search term assessment matches assess, assessment, assessed etc Stop words Certain words are left out of the index because they are considered as not containing information that adds to the usefulness of the search. These are referred to as stop words. In English they include some, but not all, conjunctions and prepositions. For example, and, at, are stop words, but over is not. The stop words depend on the detected language. A stop word in one language is not necessarily a stop word in another language. Therefore, a search result that has a detected language other than the language of the search terms, will most likely not be relevant to the search. For example, the word to is a stop word in English, but not in either German or Spanish Search Phrases If a phrase is entered, any of the words can match, and in any order. To search for an exact phrase, place it in double quotes Searching IUCLID fields IUCLID fields are the places in the IUCLID user-interface in to which users can enter data. For example, they include free text fields, pick lists, numbers, checkboxes, dates, and numeric ranges. There are three types of IUCLID field: 1. Free text 2. Pick list 3. Range, date, number, checkbox etc. Language detection is done for only the first two types of field. For the third, English is set. The properties of a search result are described in Figure Searching IUCLID attachments A IUCLID attachment is a file that is attached to a IUCLID document. The file can be attached either within a field, or to a whole IUCLID document. In the latter case, the file is accessed in IUCLID via the Attachments tab in the Information panel, as shown in the example below.

10 Text Analytics User Guide 6 TA extracts text from IUCLID attachments, and then indexes it. When a search is carried out, it is the index that is searched. TA can index only certain types of file, which are listed in the table below. Table 2: Supported file types of IUCLID attachments File type or application PDF Microsoft Word Microsoft Excel HTML (web page) RTF File extension pdf doc, docx xls, xlsx html, htm rtf In the case of PDF files, if the file contains a scan, TA extracts the text using optical character recognition (OCR). The scan must have a resolution of least 300 dpi. The properties of a search result for an attachment are described in Figure Search options The menu for search options is accessible via the black arrowhead icon indicated in the figure below. These options affect all searches.

11 Text Analytics User Guide 7 Figure 3: The search options under the menu when the type of view is the default Flat Search scope Select whether the search looks for matches inside fields, attachments, or both at the same time.

12 Text Analytics User Guide Highlight The highlight feature is available only for a search result found in an attachment. ## free text##? A highlight is a short piece of text that contains the matching text of a search. Matching words are shown in bold. A search can be set up to return one or more highlights per search result. The first highlight is shown in the search result in the field matched content. If more than one highlight is available, the complete set can be viewed by clicking on the link more matched content. An example is shown below in which a simple search was carried out in attachments for the phrase Appraisal of the Safety of Chemicals in Foods. Figure 4: Highlights of a search result The first highlight is Listings e.g. Chemical Inventories. Clicking on the link more matched content shows all the highlights. Note that the search found instances of the words Chemical and Chemicals even though the search phrase entered in to the TA did not contain the word Chemical. This happens because in this case, the search engine searches for the stems of the words in the search phrase. For more information about that behaviour, see section Total highlights The maximum number of highlights that are displayed (0-10). A value of zero displays all the text within a search result. In the example shown above, the value is 3.

13 Text Analytics User Guide Highlights length The number of characters per highlight (20-500). In the example shown above, the value is Collapse optional metadata Tick the box to have the optional metadata hidden by default in the search results. Viewing the data can be toggled on and off from within the search results. 3. Search results A search result is created for every match between the search criteria, and either a IUCLID field, or an attachment. Various parameters are recorded for a search result, which are then displayed within the search result. For example, the UUID of the IUCLID parent document of the search result. The format of the display of a search result depends on the type of view being used. There are two types of view, which are described in subsequent sections: 1) Flat (default): An individual search result is shown for every match between the search criteria and a dossier. If there is more than one match for a particular dossier, the dossier appears in multiple search results. 2) Aggregated: Search results are grouped by dossier. It is recommnded to first familiarize yourself with the flat view before progressing to the aggregated view Search results The flat view (default) A search result is shown in the user interface in a box with a pale blue border and header. The various parameters dislayed for a search result are described in an annotated example below in Figure 5. The width of the interface affects the location of some fields. To see the layout as it is shown below in Figure 5, set the width of the browser page to match that of the figure.

14 Text Analytics User Guide 10 Figure 5: Example of properties shown for a search result in the default (flat) view for a IUCLID attachment

15 Text Analytics User Guide 11 Key to Figure 5: No. Name Remark 1 Dossier UUID This is the UUID of the dossier that contains the IUCLID parent document of the search result. It is a hyperlink to the instance of IUCLID that contains the document. Following the link opens a IUCLID client window that displays the dossier. 2 Type of data source This can be either IUCLID field for a field in IUCLID, or IUCLID attachment for a file that is attached anywhere within a dossier. For IUCLID attachment, specfic parameters are shown in the search result, as indicated below by an asterisks*. 3 Parent entity type The type of IUCLID entity that contains the data source in (2). In the example above it is a Substance. 4 Value / Content / Matched content The value of a field, or a highlight of the text, that matched in the search. The link More matched content points to the highlights of the search result. Highlights are explained in section More options These are options for how to display the search result, and related document(s). The options are described below in section 3.2 Search results - More options. 6 Parent entity UUID This is the UUID of the IUCLID parent document of the search result. It is a hyperlink to the instance of IUCLID that contains the document. Following the link opens a IUCLID client window that displays the IUCLID document. 7 Filename* The filename of the attachment. 8 Label The name of the IUCLID field that contains the data which matched the search criteria. If the match is an attachment that is not in a field but is attached to a whole document, the value is Attachments. That is the case in the example shown above. For another example, see section Size* The file size of the attached file. 10 OCR* This indicates whether Optical Character Recognition (OCR) was used on an attachment. 11 Mime* The media type (MIME) of the attached file. 12 Detected language Text Analytics tries to automatically detect the language in the text being searched using a set of rules that describe each language it knows. This is the most prevalent language detected in the document that contains the search result. See entry 15 below.

16 Text Analytics User Guide Order in section Some flexible records in IUCLID can contain more than one data block. Order in section is the position of the particular matching data block down a list of several blocks. The third one down would have a value Order in section = IUCLID type The data type in IUCLID for the matching value. Examples include: Textarea, Open list, Half-bounded with closed list decimal. 15 IUCLID path The path to the matching value in IUCLID. This value can be displayed within the IUCLID interface by pressing the function key F Optional metadata This toggles the visibility of the values shown in the footer of the search result under the horizontal bar. The default can be set under the search options described in section 2.4 Search options. 17 All languages This property is shown only for field search results. This is a list of the languages detected in the search result, with the fraction of the total stated in parentheses after the language code. For example, half English and half Finnish is EN(0.5)FI(0.5). See entry 12 above Search results - More options Under the button labelled More options there are additional options for how to display the search result, and the related document(s). Some options are presented only when the search result is an attachment, as shown in the figure below. The box with the blue background at the top right of a search result indicates whether the search result is a IUCLID field, or a IUCLID attachment. Figure 6: More options in the search result

17 Text Analytics User Guide Search results More options - Hierarchy This option shows where the parent document of the search result resides in the IUCLID document hierarchy. It shows the documents that refer to it; even if indirectly. The hierarchy is presented with the search result at the top of the window. The referring documents are shown progressing down the window, which is up the IUCLID hierarchy. The highest level in the hierarchy is the dossier. Each unique path through the IUCLID hierarchy to the search result is shown. Multiple unique paths are separated by a thick dashed horizontal line. An example is shown in the figure below for a search result whose parent document is a Flexible record. The hierarchical structure is: Dossier > Substance > Flexible record Figure 7: Example output of the option Hierarchy for a search result Key to Figure 7: 1. The parent document of the search result. This is a document that contained text in a field that was matched by the search criteria.

18 Text Analytics User Guide A IUCLID document that refers directly to the parent document. In this example it is a Substance referring to a Flexible record in IUCLID section 1.2 Composition. 3. At the foot of the window is the dossier that contains the search result. 4. The dashed line is a separator between two unique paths through the IUCLID hierarchy to the parent document of the search result. All the unique paths are shown. The UUIDs of the documents are clickable hyperlinks to the instance of IUCLID that contains the documents. Following a link opens a IUCLID client window that then displays the document Search results More options - Document Displays the IUCLID parent document of the search result. The text of the IUCLID document is presented as a web page Search results More options - Dossier metadata This is for the presentation of dossier metadata obtained from external systems. If no such metadata is available to the instance of Text Analytics, this option can be ignored Search results More options - Download (text) This applies only to a search result that is an attachment. It is the text that was extracted from the attachment either directly, or using OCR. If the text came from OCR, as indicated by the OCR parameter for the search result, this text may be used to check the accuracy of the OCR. For example, mistakes in the OCR may explain why the attachment does not match the search criteria it is expected to match Search results More options - Download (original) This applies only to a search result that is an attachment. It is the original of the attached file. The same file is available as a link within the web page displayed by the option Document Search results The aggregated view The Aggregated view shows the search results grouped by the dossier that contains them. To switch from the default Flat view to the Aggregate view, click on the button indicated by the red arrow in the figure below. This button can also be used to switch back from the Aggregate to the Flat view.

19 Text Analytics User Guide 15 Figure 8: Selecting either the flat or aggregated views for search results In the Aggregated view, search results that occurred in the same dossier are shown surrounded by a box with a pale blue border and header. The UUID in the header is that of the dossier, which is a hyperlink to IUCLID. The search results are shown grouped within the IUCLID hierarchy as follows: dossier > parent document (entity) > search result (matching field or attachment) For each dossier, the number of each document and search result displayed within it is limited to two. An example of the aggregate view is shown below.

20 Text Analytics User Guide 16 Figure 9: Search results in the aggregate view, aggregated by dossier UUID Key to Figure 9: 1. The dossier UUID. This is a hyperlink to IUCLID. 2. Switch to a flat view for only this dossier. See below in section for more information. 3. The dossier metadata. The fields shown here are determined by the search options described below in Figure This is the first of several documents in the dossier that contain a match. A maximum of two documents are shown per dossier. 5. This is the UUID of the IUCLID document. It is not a hyperlink.

21 Text Analytics User Guide This is the first of several search results in the same IUCLID document. A maximum of two are shown per document. 7. This is the second of several documents in the dossier that contain a match. 8. This is the first of several search results in the second IUCLID document Show flat view for a single dossier The search results within a particular dossier can be shown in the flat view by clicking on the button labelled Flat view, in the right hand side of the header for the dossier, as can be seen in point (2) in Figure 9. To go back to the aggregate view for all the dossiers in the original search, click on the link Back to aggregate view, which is at the upper left of the interface, as shown below. Figure 10: Go back to the aggregate view after viewing flat results for a single dossier Search options for the aggregated view Both the Flat and Aggregated views have a set of options that are accessible under the black arrowhead icon to the right of the toggle button. The options specific to the Aggregated view are shown below.

22 Text Analytics User Guide 18 Figure 11: Aggregated view - options Aggregate by The default value is Snapshot UUID which groups search results by the dossier UUID Hyperlinks Where a search result contains a hyperlink to IUCLID, the format of the URL is: /ta/api/iuclid/client/documents/ <parent UUID or parent IUCLID document>/ <UUID of dossier> To share such links with your colleagues who have access to the same instance of IUCLID, send the link to them, for example by . To copy the address in Windows, right-click on the link, and select copy shortcut, as shown below. Bear in mind that if your colleague is accessing the data in an instance of IUCLID running on a different server from your own, the first line shown above in the link format must be edited accordingly.

23 Text Analytics User Guide 19 Figure 12: Copy a link to IUCLID, to share it with a colleague 3.5. Export Clicking on the button labelled Export, exports the search query and information about the results of the search. The format is MS Excel. The search query is output in its own worksheet. The fields exported in a separate worksheet are listed below in table Table 3. Table 3: The fields exported for a search Fields Value Label Order in section Parent entity UUID Parent entity Type Snapshot UUID (Dossier UUID) Field Type Remarks Field Phrase Field Detected language All languages Attachments Label Order in section Filename MIME Size Checksum Remarks Attachment UUID Parent entity UUID / Content group Parent Entity Type Snapshot UUID (Dossier UUID) Detected language Attachment type Is Attachment Scanned Content EC number

24 Text Analytics User Guide Worked examples of searches The following examples serve as an introductory tutorial to TA. After working through the examples in order, you may be able to perform the search you need. If not, search through the manual for what you are trying to do. If you intent to use TA regularly to do anything other than simple searches, it is recommended to read carefully through all of this manual in order. Text to be entered in to the search box is shown below in bold like this Simple keyword search (level 1) To execute a query, enter a search term in to the field at the left of the magnifying glass icon, and then click on the magnifying glass button. Find all entities and documents that contain the whole word permethrin : permethrin 4.2. Intermediate level searching (level 2) Find all entities and documents that contain a word that begins with permethr : permethr* Find all entities and documents that contain the sequence of letters permethr, and the phrase "piperonyl butoxide": *permethr* AND "piperonyl butoxide" 4.3. Advanced search (level 3) - the query language of text analytics Query Suggest cannot create the query described in this example, but it can be used to create most of it. The next step is to consult the documentation of the query language to find out how to edit the output of Query Suggest, so that is carries out the required search. Find dossiers of type REACH Registration tonnes in which an endpoint study record in section Short-term toxicity to fish contains a value of other: in the field Rationale for reliability incl. deficiencies. The text associated with the value of other: contains the phrase old. This text was entered into the text box that lies immediately to the right of the field in the IUCLID interface. Text can be entered in to that box only after the value other: has been selected for the field. The parameters to select in Query Suggest under Fields are: Submission type Subject type Field label Field value REACH Registration tonnes SUBSTANCE 6.1 Aquatic toxicity->6.1.1 Short-term toxicity to fish.administrative data.rationale for reliability incl. deficiencies Other:

25 Text Analytics User Guide 21 The search query is: joinlevel:dossier{entity.submission_type:r_ AND entity.dossier_subject_type:substance} AND field.label:"6.1 Aquatic toxicity->6.1.1 Shortterm toxicity to fish.administrative data.rationale for reliability incl. deficiencies" AND field.value:"other:" The question is, how to look for the value old in text associated with the field other. The options available for a search criterion applied to a field are shown under the tab Searchable fields, in the section Searchable field properties. This contains a list of all the properties of a field that can be searched. Note that the value of the field is also a property. In this example we want to search in the text that was entered in to the box that lies immediately to the right of the field in the IUCLID interface. This text is considered to be a property of the field. The name of the property is other_text. The general syntax in the query language to search within a field is: field.<property>:<search term> In this example we have: property=other_text search term=*old* Therefore, the criterion is: field.other_text:*old* The search criterion shown above is actually applying two search criteria at once. Firstly that other: was selected in the drop-own menu of the field, and the secondly that the phrase old was entered in to the box that lies immediately to the right of the field in the IUCLID interface. The wild card * has been used to find any instances of the phrase old, which includes words such as older. The query generated by Query Suggest for the previous example already contains a search criterion that is applied to the value of the field. Therefore, no new criterion need be added, instead it is enough to edit the existing one as follows: AND field.value:"other:" -> AND field.other_text:*old* Thus the query for this example is: joinlevel:dossier{entity.submission_type:r_ AND entity.dossier_subject_type:substance} AND field.label:"6.1 Aquatic toxicity->6.1.1 Shortterm toxicity to fish.administrative data.rationale for reliability incl. deficiencies" AND field. other_text:*old* Breaking the query in to its component parts we have: joinlevel:dossier{entity.submission_type:r_

26 Text Analytics User Guide 22 The data that satisfy the search criteria must be present in a IUCLID entity of type dossier, which is of type REACH Registration tonnes AND entity.dossier_subject_type:substance} The data that satisfy the search criteria must be present in a IUCLID entity of type substance AND field.label:"6.1 Aquatic toxicity->6.1.1 Short-term toxicity to fish.administrative data.rationale for reliability incl. deficiencies" The data that satisfy the search criteria must be present in a IUCLID field of type Shortterm toxicity to fish.administrative data.rationale for reliability incl. deficiencies AND field. other_text:*old* The IUCLID field must have a property named other_text that contains the phrase old somewhere in its value.