Orange3-Textable Documentation

Transcription

1 Orange3-Textable Documentation Release 3.0a1 LangTech Sarl Dec 19, 2017

2

3 Contents 1 Getting Started Orange Textable Description Installation Widgets Text Import Segmentation Processing Table Construction Conversion/Export Tutorials 49 4 Indices and tables 51 i

4 ii

5 NB: this documentation is still work in progress. Until it s completed, the user is advised to consult the documentation of Textable v2.0 Contents: Contents 1

6 2 Contents

7 CHAPTER 1 Getting Started 1.1 Orange Textable Getting Started Orange Textable is an open-source add-on bringing advanced text-analytical functionalities to the Orange Canvas visual programming environment (itself open-source). It has been designed and implemented on behalf of the department of language and information sciences (SLI) at the University of Lausanne and it essentially enables users to build data tables on the basis of text data, by means of a flexible and intuitive interface. Orange Textable offers the following features: text data import from keyboard, files, or URLs support for various encodings, including Unicode standard preprocessing and custom recoding (based on regular expressions) segmentation and annotation of various text units (letters, words, etc.) ability to extract and exploit XML-encoded annotations automatic, random, or arbitrary selection of unit subsets unit context examination using concordance and collocation tables calculation of frequency and complexity measures recoded text data and table export This documentation is divided into 3 main sections (see detailed contents below): The Getting Started section provides an overview of the features in Orange Textable and gives relevant information on how to download and install the add-on and load your data. The Widgets section offers an exhaustive explanation of all widgets in the Orange Textable s interface. It also includes practical examples of usage. 3

8 The Tutorial section shows how to use Orange Textable to solve more complex and interesting problems in text data analysis. The purpose of this part of the documentation is to inspire you to use Orange Textable for more advanced research Widgets Text Import Segmentation Processing Table Construction Conversion/Export Tutorials 1.2 Description Orange Textable is an open-source add-on bringing advanced text-analytical functionalities to the Orange Canvas visual programming environment (itself open-source). It has been designed and implemented on behalf of the department of language and information sciences (SLI) at the University of Lausanne and it essentially enables users to build data tables on the basis of text data, by means of a flexible and intuitive interface. Orange Textable offers the following features: text data import from keyboard, files, or URLs support for various encodings, including Unicode standard preprocessing and custom recoding (based on regular expressions) segmentation and annotation of various text units (letters, words, etc.) ability to extract and exploit XML-encoded annotations automatic, random, or arbitrary selection of unit subsets unit context examination using concordance and collocation tables calculation of frequency and complexity measures recoded text data and table export This documentation is divided into 3 main sections (see detailed contents below): The Getting Started section provides an overview of the features in Orange Textable and gives relevant information on how to download and install the add-on and load your data. The Widget Catalog section offers an exhaustive explanation of all widgets in the Orange Textable s interface. It also includes practical examples of usage. The Tutorial section shows how to use Orange Textable to solve more complex and interesting problems in text data analysis. The purpose of this part of the documentation is to inspire you to use Orange Textable for more advanced research. 4 Chapter 1. Getting Started

9 1.3 Installation Orange Textable is available as an add-on in the Orange data mining software package. In order to install Orange Textable, you first need to download Orange. A detailed installation guide is available here < _. After sucessfully downloading Orange, open the program, click the Options button and select Add-ons. Tick the box in front of the Orange-Textable option and then click the OK button. The Orange Textable tab should appear in the list on the left side of the program after having exited and restarted the program Installation 5

10 6 Chapter 1. Getting Started

11 CHAPTER 2 Widgets 2.1 Text Import Text Field Import text data from keyboard input. Signals Inputs: Text data Segmentation containing the text to be edited Outputs: Text data Segmentation covering the input text Description The Text Field widget enables the input of data by using a keyboard. It emits a segmentation containing a single unannotated segment covering the whole string. It can also be used to manually edit a previously imported string (imported from a text file or an URL). 1. The editable text field. Standard editing functions (copy, paste, cancel, etc.) apply. 7

12 8 Chapter 2. Widgets

13 2. By clicking Send, changes are committed to the output of the widget. Alternatively, tick Send automatically and changes will be committed at every modification. The text field s content is normalized in three ways: converted to Unicode, subjected to canonical Unicode-recomposition technique, various forms of line endings (in particular rn and r) are converted to a single form (namely n). 3. Information about the output: the length of the segmentation output in characters. Examples The first example is used to demonstrate how to input text for further processing by using the Text Field widget. The widget is fairly easy to use. Place the widget on the canvas by choosing it from the list of widgets in the add-on or use the right-click to click on the canvas and select the widget from there. Double-click on it to open it and input the text by using your keyboard. Standard editing functions apply and you can copy and paste text from other sources as well. When finished, click the Send button (alternatively, tick Send automatically) and display the segmented data through any of the compatible outputs. We simply entered the text string This is a very simple example and chose to display it in the Display widget. widgets/textimport/images/textfield-example-1.png Our second example is used to demonstrate how to use the Text Field widget to manually edit imported strings of texts. We loaded Jane Austen s Pride and Prejudice by using the URLs widget and because the text about Project Gutenberg had no use for us (and our further research) we decided to remove it by editing the input content in the Text Field widget. widgets/textimport/images/textfield-example-2.png Text Files Import data from raw text files. Signals Inputs: Message 2.1. Text Import 9

14 JSON Message controlling the list of imported text files Outputs: Text data Segmentation covering the content of imported text files Description This widget is designed to import the contents of one or several text files. It outputs a segmentation containing a (potentially annotated) segment for each imported file. The imported textual content is normalized in several ways: converted to Unicode, subjected to canonical Unicode-recomposition technique, stripped from the utf8 byte-order mark and various forms of line endings (in particular rn and r) are converted to a single form (namely n). The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. The basic interface The basic interface of the widget enables the input of a single file. 1. Select a file. The selected file later appears in the File Path text field. 2. Specify the encoding of the file by using the encoding drop-down menu. 3. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the widget s output at every modification. 4. Information about the output: the number of characters in a single segment contained in the output segmentation, or the reasons why no segmentation is emitted (no input data, encoding issue, etc.). 10 Chapter 2. Widgets

15 The advanced interface The advanced interface of the widget enables the import of several files in a determined order. The user can decide to annotate and encode each file accordingly. 1. Select the input files as well as their encoding, determine the order in which they appear in the output segmentation, and (optionally) assign annotations. The list of imported files with additional information about the corresponding annotations and encodings appears at the top of the window. 2. Modify the selection of sources by: (a) changing the order in which the files appear in the output: Move Up, Move Down (b) deleting individual files from the list: Remove (c) clearing the list of all files: Clear All (d) import a file list in JSON format and add it to the previously selected sources: Import List (e) export the source list in a JSON file: Export List 3. The Browse button opens a selection dialog. After having selected one or more files in this dialog and validated the choice by clicking on Open, the files appear in the File path(s) field and can be added to the list by clicking on the Add button. It is also possible to type the complete paths of the files directly in the text field, separating the paths corresponding to the successive files with the string (space + slash + space). 4. Select the Encoding. 5. Assign an annotation by specifying its key in the Annotation key field and the corresponding value in the *Annotation value field. 6. Specify the label affected to the output segmentation. The Import filenames with key checkbox enables the program to create for each imported file an annotation whose value is the file name (as displayed in the list) and whose key is specified by the user in the text field on the right of the checkbox. Similarly, the button Autonumber with key enables the program to automatically number the imported files and to associate the number to the annotation key specified in the text field on the right. 7. By clicking Send changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 8. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Example For the purpose of this example, we have decided to make use of the advanced interface and explore the additional options offered by it. We opened the Text Files widget and selected the advanced settings option. We imported several text files and annotated them. We got our data from the Project Gutenberg website and chose to focus on annotating Shakespeare s works by genre. We selected Romeo and Juliet, The Merchant of Venice and Shakespeare s Sonnets and annotated them according to type (tragedy, comedy, poetry). We then displayed our results in the Display widget URLs Fetch text data online Text Import 11

16 12 Chapter 2. Widgets

17 2.1. Text Import 13

18 Signals Inputs: Message JSON Message controlling the list of imported URLs Outputs: Text data Segmentation covering the content of imported URLs Description The URLs widget is designed to import the contents of one or several internet locations (URLs) to the Orange Canvas softaware. Its output is a segmentation containing a (potentially annotated) segment for the content of each selected URL. The imported textual content is subject to normalization. The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. Basic interface The basic version limits the number of imported URLs to one. 1. Input the selected URL and select the encoding of its content. 2. By clicking Send changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 14 Chapter 2. Widgets

19 3. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Advanced interface The advanced version of the URLs widget allows the user to import the content of several URLs in a determined order; each URL can moreover be associated to a distinct encoding and specific annotations. The emitted segmentation contains a segment for the content of each imported URL. 1. The Sources section allows the user to select the imported URLs as well as their content s encoding, to determine the order in which they appear in the output segmentation, and optionally assign annotations. The list of imported URLs with additional information about corresponding annotations and encodings appears at the top of the window. 2. The buttons on the left side of the Sources section allow the user to modify their selection by: (a) changing the order in which the URLs appear in the output: Move Up, Move Down (b) deleting individual URLs from the list: Remove c) clearing the list of all URLs: Clear All c) import a list of URLs in JSON format and add it to the previously selected sources: Import List d) export the source list in a JSON file: Export List 3. Add new URLs to the list. They must first be input in the field before they can be added to the list by clicking on the Add button. If you wish to add several URLs simultaneously, you must separate them by the string / (space + slash +space). 4. Select the Encoding. 5. Assign an annotation by specifying its key in the Annotation key field and the corresponding value in the Annotation value field. These three parameters (encoding, key, value) will be applied to each URL appearing in the URLs field at the moment of their addition to the list with Add. 6. Specify the label affected to the output segmentation. The Import URLs with key checkbox enables the program to create for each imported URL an annotation whose value is the file name (as displayed in the list) and whose key is specified by the user in the text field on the right of the checkbox. Similarly, the button Auto-number with key enables the program to automatically number the imported URLs and to associate the number to the annotation key specified in the text field on the right. 7. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 8. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Example For the purpose of this example, we have decided to make use of the advanced interface and explore the additional options offered by it. We opened the URLs widget and selected the advanced settings option. We imported several URLs and annotated them. We got our data from the Project Gutenberg website and chose to focus on annotating Shakespeare s works by genre. We selected Romeo and Juliet, The Merchant of Venice and Shakespeare s Sonnets and annotated them according to type (tragedy, comedy, poetry). We then displayed our results in the Display widget Text Import 15

20 16 Chapter 2. Widgets

21 2.2 Segmentation Processing Preprocess Basic text preprocessing. Signals Inputs: Segmentation Segmentation covering the the text that should be preprocessed Outputs: Text data Segmentation covering the modified text Description The **Preprocess* widget enables simple text preprocessing and creates a modified copy of the input content. The user can modify case (lower and upper) or replace accentuated characters with their non-accentuated equivalents. 1. Change case and convert every character to lower or upper case. 2. Replace accentuated characters with their non-accentuated equivalents Segmentation Processing 17

22 3. Copy all the annotations of the input segmentation to the output segmentation. 4. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. 5. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. Example We used the Text Field widget to input a short text. We then decided to preprocess it in the Preprocess wiget and display it in the Display widget Recode Custom text recoding using regular expressions. Signals Inputs: 18 Chapter 2. Widgets

23 Segmentation Segmentation covering the text that should be recoded Message JSON Message controlling the list of substitutions Outputs: Recoded text data Segmentation covering the recoded text Description The Recode widget creates a modified copy of the input segmentation. The modifications applied are defined by substitutions, namely pairs composed of a regular expression and a replacement string. The interface of the Recode widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. Basic Interface The basic version if the widget is limited to the application of a single substitution. 1. Specify the regular expression. Use regular expressions to reach more data or just enter a word you wish to subtitute. 2. Specify the replacement string. If the replacement string is left empty, the text parts identified by the regular expression will simply be deleted Segmentation Processing 19

24 3. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 4. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Advanced Interface The advanced interface allows the user to define several substitutions and to determine the order in which they should be applied. 1. The Substitutions section allows the user to select the substitutions applied to each successive input segment and to determine their application order. The list of substitutions with corresponding regular expressions, replacement strings and additional options associated with the regular expression used appears at the top of the window. 2. The buttons on the left side of the Sources section allow the user to modify his selection by: changing the order in which the substitutions are applied: Move Up, Move Down deleting individual substitutions from the list: Remove clearing the list of all substitutions: Clear All import a list of substitutions in JSON format and add it to the previously selected sources: Import List export the list of substitutions in a JSON file: Export List 3. Define a new subsitution by using RegEx. 4. Define the replacement string. 5. Control the application of the corresponding options to the regular expression: Ignore case, Unicode dependent, Multiline, Dot matches. 6. Add a new substitution to the list. 7. Copy every annotation of the input segmentation to the output segmentation. 8. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 9. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted Example For the purpose of this example, we have decided to adapt our American spelling for our readers that prefer British English. We mostly focused on the difference in spelling the affix -ize (or -ise that is preferred in British English). We used the Text Field widget to input several words that use the American way of spelling and then segmented the input segmentation into words. Lastly, we used the advanced interface of the Recode widget and replaced all occurences of the affix -ize with the affix -ise Merge Merge two or more segmentations. 20 Chapter 2. Widgets

25 2.2. Segmentation Processing 21

26 Signals Inputs: Segmentation (multiple) Any number of segmentations that should be merged together Outputs: Merged data Merged segmentation Description The Merge widget merges two or more segmentations and outputs them to its output channel. 1. Create for each input segmentation an annotation whose value is the segmentation label and whose key is specified by the user in the text field on the right of the checkbox. 2. Automatically number the output segments and associate the number to the annotation key specified in the text field on the right. 3. Copy every input segmentation annotation to the output segmentation. 4. Fuse several distinct segments with the same addresses into a single segment. 5. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 6. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. 22 Chapter 2. Widgets

27 Example We used the Text Field widget to input texts in three different languages. We renamed them according to the language that was used, merged them together by using the Merge widget and decided to import the labels with the key Language. We then displayed our results in the Display widget and by doing that we created ourselves a handy dictionary, which we then saved to our computer Segment Subdivide a segmentation using regular expressions. Signals Inputs: Segmentation 2.2. Segmentation Processing 23

28 Segmentation that should be further segmented Message JSON Message controlling the list of regular expressions Outputs: Segmented data Segmentation containing the newly created segments Description The Segment widget subdivides a segmentation into a series of new segments by using regular expressions. Alternatively, it can also operate based on a desciption of the separators that appear in-between te segments. The user is also able to create annotations for the output segments. The interface of the widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. Basic Interface The basic interface of the widget only enables the application of a single regular expression. 1. Use the drop-down menu and define the segment type you need. There are several available options: Segment into letters, Segment into words, Segment into lines, and Use a regular expression. By choosing the latter, you need to input your own RegEx. 2. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 3. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. 24 Chapter 2. Widgets

29 Advanced interface In its advanced version, the widget enables the user to define several regular expressions and to determine the order in which they should successively be applied to each segment of the input segmentation. The user can also specify if a given regular expression describes the form of the targeted segments (Tokenize mode) or rather the form of the separators in-between these segments (Split mode). 1. Select the RegExes applied to each input segment and determine their application order. The list of RegExes with their associated modes, actual expressions, corresponding annotations and the options associated with those expressions appears on top of the window. 2. The buttons on the left side of the RegExes section allow the user to modify his selection by: changing the order in which the RegExes are applied: Move Up, Move Down deleting individual RegExes from the list: Remove clearing the list of all RegExes: Clear All import a list of RegExes in JSON format and add it to the previously selected sources: Import List export the list of RegExes in a JSON file: Export List 3. Select segmentation mode: If you wish to use a regular expression to describe the form of the targeted segments, use the Tokenize mode. Tokenization is breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens (an individual occurrence of a linguistic unit in speech or writing). If you wish to use a regular expression to describe the form of the separators in-between these segments, use the Split mode. 4. Specify the regular expression you wish to add. Optionally, also specify the annotation key and value. Control the application of the corresponding options to the regular expression: Ignore case, Unicode dependent, Multiline, Dot matches. 5. Add the new regular expression to the list. 6. Define the output segmentation label: Auto-number with key: automatically number the output segments and associate the number to the annotation key specified in the text field on the right. Import annotations: copy the annotations of each input segment to the corresponding outpus segments. Fuse duplicate segments: fuse several distinct segments with the same address into a single segment. 7. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 8. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted Example - For the purpose of this example, we used the Text Field widget to input Shakespeare s Sonnet 130. We then connected it to the Segment widget and used a regular expression (s+[^.!?]*[.!?]) to divide the poem into sentences. We later displayed it in the Display widget Select Select a subset of segments in a segmentation Segmentation Processing 25

30 26 Chapter 2. Widgets

31 2.2. Segmentation Processing 27

32 Signals Inputs: Segmentation Segmentation out of which a subset of segments should be selected Outputs: Selected data (default) Segmentation containing the selected segments Discarded data Segmentation containing the discarded segments Description The Segment widget enables the user to select a subset of segments in segmentation. The selection is made by using regular expressions and the user can select whether to include or exclude the selected segments from the output segmentation. The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. The advanced version offers different methods of selection - Regex, Sample, or Threshold, whereas the basic version only offers specifying the selection by using regular expressions. The basic version of the widget also does not include regular expression options, auto-numbering is disabled and annotations are copied by default. Method: Regex 1. Select a mode of selection: Include or Exclude the selected segment. 2. Specify the regular expression. Optionally, also specify the annotation key and value. Control the application of the corresponding options to the regular expression: Ignore case, Unicode dependent, Multiline, Dot matches 3. Select whether you wish to auto-nummber the segmentations and if you wish to copy the annotations from the input segmentation to the ouput of the widget. 4. By clicking Send changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 5. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Method: Sample Select the segments of the input segmentation with a random sampling process. 1. Choose in which way to express the wanted size for the sample. Count: the size of the sample will be expressed directly in the number of segments. Proportion: the sizze of the sample will be expressed in percentage of input segments 2. Select whether you wish to auto-number the segmentations and if you wish to copy the annotations from the input segmentation to the ouput of the widget. 3. By clicking Send changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 28 Chapter 2. Widgets

33 2.2. Segmentation Processing 29

34 30 Chapter 2. Widgets

35 4. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Method: Threshold widgets/segmentationprocessing/images/select-treshhold-st Only the segments whose content (or annotation value for a given key) has a frequency in the segmentation that is comprised between given bound are retained from the input segmentation. 1. Select an annotation key from the input segmentation. 2. Choose the way in which to express the minimal and maximal frequency limits: Count: the limits wll be expressed in absolute frequencies Proportion: the limits will be expressed in percentages 3. Select whether you wish to auto-number the segmentations and if you wish to copy the annotations from the input segmentation to the ouput of the widget. 4. By clicking Send changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 5. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Example THRESHOLD NE DELA - javi rdeč error Example Intersect In-/exclude segments based on another segmentation. Signals Inputs: Segmentation (multiple) 2.2. Segmentation Processing 31

36 Segmentation out of which a subset of segments should be selected ( source segmentation), or containing the segments that will be in-/excluded from the former ( filter segmentation) Outputs: Selected data (default) Segmentation containing the selected segments Discarded data Segmentation containing the discarded segments Description The Intersect widget enables the user to search through multiple segmentations and find segments that intersect. Those segments can then be included or excluded from the output segmentation. The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. The basic interface The basic interface allow the user to specify whether to include or exclude the intersecting segments from the output segmentation. 1. Select the widget mode you wish to use and include or exclude segments from the output segmentation. 32 Chapter 2. Widgets

37 2. Use the drop-down menu to select the source and the filter segmentation. The latter is mostly used when trying to filter information deemed irrelevant for research. 3. By clicking on the Send button, changes are committed to the output of the widget. Alternatively, use Send automatically and the segmentation will be modified on every change. 4. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted The advanced interface 1. Select the widget mode you wish to use and include or exclude segments from the output segmentation Segmentation Processing 33

38 2. & 4. Use the drop-down menu to select the source and the filter segmentation. The latter is mostly used when trying to filter information deemed irrelevant for research. 3. & 5. A Filter/Source annotation key dropdown menu. If a given annotation key of the source/filter segmentation is selected, the corresponding annotation value (rather than content) types will condition the in-/exclusion of the source segmentation segments. 6. Automatically number the segments from the output segmentation and associate the number to the annotation key specified in the text field on the right. 7. Copy every annotation from the input segmentation to the output segmentation. 8. By clicking on the Send button, changes are committed to the output of the widget. Alternatively, use Send automatically and the segmentation will be modified on every change. 9. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted Example For the purpose of this example, we have decided to explore the ratio between the lines spoken by male characters and the lines spoken by female characters in Shakespeare s Romeo and Juliet. We used the Extract XML widget to extract all speaker tags (<SPEAKER>) and preprocessed the data because it was not all capitalized. We then divided the file into line segments. We used the Text Field widget to input the names of female characters in Romeo and Juliet and then segmented the segmentation into words. By using the Intersect widget and information about the segments sent to the widget s output we came to the conclusion that the ratio is approximately 2:1 in favor of male character lines Extract XML Create a new segmentation based on XML markup. Signals Inputs: Segmentation Segmentation covering XML data based on which a news segmentation will be created Outputs: Extracted data Segmentation containing the segments corresponding to extracted XML elements Description The **Extract XML* widget is used to search portions corresponding to a specific XML element type in an input segmentation. It creates a segment for each occurence of this element. If such an occurence is distributed among several segments of the input segmentation, the widget will create as many segments in the output segmentation. Every attribute from the extracted elements is automatically converted in and annotation in the output segmentation. The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. 34 Chapter 2. Widgets

39 2.2. Segmentation Processing 35

40 Basic interface The basic interface only allows the extraction of a single type of element at a time; however, it extracts every occurrence of this element, including those embedded in other occurrences of the same type. 1. Delete the XML tags embedded within the extracted XML elements, if any. The extracted elements will potentially be decomposed in several segments corresponding to portions of their content, which are separated by the deleted XML tags. In the XML field, you select which element types are to be sought. 2. By clicking on the Send button, changes are committed to the output of the widget. Alternatively, use Send automatically and the segmentation will be modified on every change. 3. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted Advanced Interface The advanced version of the Extract XML widget allows the user to configure the XML element extraction. 1. Indicate the XML element type which should be sought. 2. Assign to each output segment an annotation whose key is the text contained in the field immediately on the right and whose value is the name of the XML element extracted by the widget. 3. Exclude XML tags embedded within the extracted XML elements from the output segmentation. The extracted elements will potentially be decomposed in several segments corresponding to portions of their content which are separated by the excluded XML tags. 4. By not choosing this option, a single segment is created (as opposed to several). 5. Determines the behavior of the widget in the very particular case where (a) elements of the extracted type are (exactly) embedded in one another, (b) they have different values for the same attribute, (c) the Remove markup option is selected and (d) the Fuse duplicates option (section Options) as well.if the Prioritize shallow attributes option is selected, the value of the element closest to the surface will be kept. 36 Chapter 2. Widgets

41 2.2. Segmentation Processing 37

42 6. Limit the extraction by specifying the conditions bearing on attributes on the extracted elements. These conditions are expressed in the form of regular expressions that the given attribute values must match. Use the buttons on the right side of the widget to delete the selected condition or completely empty the list. Specify the attribute in question and the corresponding regular expression (RegEx). Manage the application of the corresponding options to the regular expression. Add the newly created condition to the list. 7. Specify the output segmentation label: Auto-number with key: automatically number the segments of the output segmentation Import annotations: copy annotations from the input segmentation to the output segmentation Merge duplicate segments: fuse distinct segments with the same address 8. By clicking Send, changes are communicated to the output of the widget. Alternatively, tick Send automatically and changes will be communicated to the output at every modification. 9. Information about the number of segments in the output segmentation or the reasons why no segmentation is emitted. Example We downloaded a set of Shakespeare s plays marked up in XML and used the Text Files widget to import two files into Orange. We then extracted all stage directions by typing <STAGEDIR> in the XML element field. We are now free to explore how stage directions in a comedy differentiate from those in a tragedy. images/extractxml-example.png Display Display or export the details of a segmentation. Signals Inputs: Segmentation Segmentation to be displayed or exported Outputs: 38 Chapter 2. Widgets

43 Bypassed segmentation (default) Exact copy of the input segmentation Displayed segmentation Segmentation covering the entire string displayed in the widget s interface Description The Display widget is used to display or export the content of an inputted segmentation on screen. The user can choose to annotate the segments or just output them without any modification. The widget is available in two versions, according to whether or not the Advanced Settings checkbox is selected. Basic interface The widget formats the input segmentation in HTML and displays for each segment its number, its complete address (string index, start and end positions) as well as its annotation. 1. Directly show a particular segment by using Go to segment Segmentation Processing 39

44 2. Information about the output. The Display widget does not possess a Send option, because its main feature is the visualization of the input segmentation s content and annotations. Advanced interface The advanced interface enables custom formatting and provides the user with the means to produce a personalized rendering of his data. The formatting of each segment is determined by a string which the user enters in the Format field. 1. Tick the box in front of Apply custom formatting to personalize the rendering of your data: The Header section enables you to specify the string that will be inserted at the beginning of the formatted segmentation. Determine the format of each segment. The string entered in the Format field can contain a text that will be reproduced as it is in the rendered output, as well as references to variables to insert in the output. These references take the following general form: %(variable_name)format. The string entered in the Segment delimiter section will be inserted between each segment of the formatted segmentation. The sequence n is used for line breaks and the sequence t is used for tabulation. The Footer section enables you to specify the string that will be inserted at the end of the formatted segmentation. 2. Export the displayed segmentation (standard HTML or user-defined format). You can select the encoding and save the formatted content in a separate file. By selecting the Copy to clipboard option the displayed segmentation can be pasted in another application (utf-b encoding is used by default). 3. Send to file. 4. Information about the number of segments in the widget s output. If the checbox in front of Apply custom formatting is not ticked, the Navigation section is enabled and the user can view particular segments by using the *Go to segment control. Examples We first used the Project Gutenberg website to download Shakespeare s Romeo and Juliet in the plain text format. We imported the file to Orange Canvas by using the Text Files widget. We edited the text in the Text Field widget and then divided the segmentation into segments according to lines by using the Segment widget. We then chose to display our data in the Display widget. To illustrate another use of this widget we decided to make use of the Export option. We took advantage of the text from the example above and exported the newly created segmented file to our computer. 2.3 Table Construction Count Count segment types. 40 Chapter 2. Widgets

45 2.3. Table Construction 41

46 Signals Inputs: Segmentation (multiple) Segmentation whose units constitute the units to be counted or the contexts in which the units will be counted Outputs: Pivot Crosstab Table displaying the absolute frequency of units Description widgets/tableconstruction/images/count-stamped.png Example Length Compute the (average) length of segments. Signals Inputs: Segmentation (multiple) Segmentation whose segments constitute the units of length measurement, the contexts whose lengths will be measured, or the units over which length will be averaged Outputs: Textable table Table in the internal format of Orange Textable 42 Chapter 2. Widgets

47 Description widgets/tableconstruction/images/length-stamped.png Example Variety Measure the variety of segments. Signals Inputs: Segmentation Segmentation whose segments constitute the units of variety measurement, or the contexts in which variety will be measured Outputs: Textable table Table in the internal format of Orange Textable Description widgets/tableconstruction/images/variety-stamped.png Example Cooccurrence Measure cooccurrence between segment types Table Construction 43

48 widgets/tableconstruction/icons/cooccurence.png Signals Inputs: Segmentation (multiple) Segmentation whose segments constitute the units subject to measurement of their cooccurrence or the contexts in which unit cooccurrence will be measured Outputs: Pivot Crosstab Table displaying the cooccurrence of units in the defined context Description widgets/tableconstruction/images/cooccurrence-stamped.png Example Context Explore the context of segments. Signals Inputs: Segmentation (multiple) Segmentation containg the key segments whose context will be examined or the segments which serve to define these contexts Outputs: Textable table 44 Chapter 2. Widgets

49 Table displaying the concordance of key segments or their collocations Description widgets/tableconstruction/images/context-stamped.png Example Category Build a table with categories defined by segments content or annotations. Signals Inputs: Segmentation (multiple) Segmentation whose segments constitute the basis for category extraction Outputs: Textable table Table displaying the extracted categories Description Example 2.4 Conversion/Export Convert Convert, transform, or export Orange Textable tables Conversion/Export 45

50 Signals Inputs: Textable table Table in the internal format of Orange Textable Outputs: Orange Table (default) Data in the standard Orange data table format (possibly transformed) Textable Table Table in the internal format of Orange Textable (possibly transformed) Segmentation Segmentation containing the output table in tab-delimited format Description widgets/conversionexport/images/convert-stamped.png Example Message Parse JSON data in segmentation and use them to control other widgets. Signals Inputs: Segmentation Segmentation containing a single segment with the JSON data to be parsed Outputs: Message JSON Message object that can be sent to other widgets 46 Chapter 2. Widgets

51 Description widgets/conversionexport/images/message-stamped.png Example 2.4. Conversion/Export 47

52 48 Chapter 2. Widgets

53 CHAPTER 3 Tutorials 49

54 50 Chapter 3. Tutorials

55 CHAPTER 4 Indices and tables genindex modindex search 51