IBM Watson Explorer Community Edition IBM

Transcription

1 IBM Watson Explorer Community Edition IBM

2 Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at copytrade.shtml. The following terms are trademarks or registered trademarks of other companies: Adobe is a registered trademark of Adobe Systems Incorporated in the United States, and/or other countries. Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows and Windows NT are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. The United States Postal Service owns the following trademarks: CASS, CASS Certified, DPV, LACSLink, ZIP, ZIP + 4, ZIP Code, Post Office, Postal Service, USPS and United States Postal Service. IBM Corporation is a non-exclusive DPV and LACSLink licensee of the United States Postal Service. Other company, product or service names may be trademarks or service marks of others. Copyright IBM Corporation US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents Chapter 1. Installing Watson Explorer Community Edition Providing access to the local file system from Watson Explorer Community Edition Chapter 2. Getting started with Watson Explorer Community Edition Running Watson Explorer Community Edition Chapter 3. Administering content Collections Add a collection Collection fields Collection facets Enrichment Part of Speech annotator Named Entity Recognition annotator Sentiment Analysis annotator Dictionary annotator Supervised Multi-labeling classifier Datasets Add a dataset Datasets Crawlers Add a crawler Importers Converter pipeline Converters Chapter 4. Mining content Collection detail view Guided Analysis view Chapter 5. Watson Explorer Community Edition API Appendix. Installing Watson Explorer Community Edition on a computer with Azure Active Directory Notices Copyright IBM Corp iii

4 iv IBM Watson Explorer Community Edition

5 Chapter 1. Installing Watson Explorer Community Edition IBM Watson Explorer Community Edition gives you the power of IBM Watson Explorer in an application that you can run on your Microsoft Windows or Macintosh computer. You can download and install Watson Explorer Community Edition and quickly start searching and analyzing your data. Watson Explorer Community Edition runs inside Docker, a container platform. The Watson Explorer Community Edition setup program installs Docker if necessary and then downloads the IBM Watson Docker image, which runs inside Docker. Prerequisites Your computer must have virtualization enabled. The operating system requirements are as follows. Microsoft Windows platforms v Windows 10 Education v Windows 10 Enterprise v Windows 10 Pro Macintosh platforms v Mac OS X or later Installing Watson Explorer Community Edition Download and run IBM Watson Explorer Setup. A wizard will guide you through the installation process. You should be aware of the following issues during the installation process. v IBM Watson Explorer Setup will install Microsoft Hyper-V (on Windows platforms) and Docker, if they are not already installed. This may cause your computer to reboot or require a new sign in one or more times during the installation process. After a reboot or sign in, double-click the IBM Watson Explorer icon on the desktop and the installation will continue. v If you computer uses Azure Active Directory for sign-in authentication, see Installing Watson Explorer Community Edition on a computer with Azure Active Directory, on page 39 for additional steps required when installing Watson Explorer Community Edition. When the installation is complete, a Results screen is displayed where you can launch the Content Miner or the Admin Console. When you start Watson Explorer Community Edition from now on, the Results screen is displayed immediately. Uninstalling Watson Explorer Community Edition You can uninstall the IBM Watson Docker image by clicking More info > Uninstall. If required, you can also uninstall IBM Watson Explorer and Docker by using the uninstall process of your operating system. Copyright IBM Corp

6 Providing access to the local file system from Watson Explorer Community Edition Some operations in IBM Watson Explorer Community Edition require access to your local file system. You can grant this access in Watson Explorer Community Edition. Procedure 1. On the Results screen of Watson Explorer Community Edition, click More info > Mount. 2. Select a source folder to mount. 3. Enter the path to be used to access the source folder after /mnt/ in the Mount location. 4. Click to confirm your choice. 5. You can now choose additional folders or cllick Mount Folders to finish. Results The folders are now available within Watson Explorer Community Edition. You may be requested to provide local account credentials so that you can access the mounted folders. When you refer to this folder, you use the mount file path (/mnt/...). 2 IBM Watson Explorer Community Edition

7 Chapter 2. Getting started with Watson Explorer Community Edition Watson Explorer Community Edition introduces components that are the natural and cumulative evolution from basic search, through advanced natural language processing (NLP) and machine learning (ML) based cognitive computing. This evolution has the goal of delivering information and business insights faster and with more accuracy. Watson Explorer combines search and content analytics to help users find and understand the unstructured information they use to work more efficiently and make better, confident decisions at the point of impact. Watson Explorer enables: v Cognitive exploration: Securely connect to, search, and explore all of your organization's data, regardless of format or where it is stored or managed (find what you know), v Cognitive insights: Gain insights from unstructured dark data with advanced content analytics using hypothesis-free discovery (discover what you don t know), v Cognitive advice: The system provides advice in the next steps of the discovery process. So how do you do that? With Watson Explorer Community Edition we introduce two new components: v To administer your content, Watson Explorer Admin Console is intended for administrators who will create datasets and collections for the use of analysts, define annotators and classifiers, and launch crawlers to index collections. v To mine your content Watson Explorer Content Miner is the tool analysts use to analyze and create collections. Administering content Please see Chapter 3, Administering content, on page 5 for details on the Watson Explorer Admin Console. Mining content Please see Chapter 4, Mining content, on page 25 for details on the Watson Explorer Content Miner. Running Watson Explorer Community Edition You can run IBM Watson Explorer Community Edition by using the IBM Watson Explorer application or directly from your browser. When you run the IBM Watson Explorer application, it offers you the choice of launching the Content Miner or the Admin Console. When you click one of these, IBM Watson Explorer Content Miner or IBM Watson Explorer Admin Console opens in a browser window. You can also open these windows directly in your browser by typing or into Copyright IBM Corp

8 the browser address bar. This method also allows you to run a version of Watson Explorer Community Edition that is installed on a different computer in your network. Note: When you launch Content Miner or Admin Console, you will sometimes observe a browser error. This is because the web server that serves the content has not completely started. In this case, wait a few seconds and refresh your browser. 4 IBM Watson Explorer Community Edition

9 Chapter 3. Administering content You can add, modify, and delete collections, enrichments, and datasets from the IBM Watson Explorer Admin Console. Collections A collection contains the entire set of sources that users can query at a time. When you create a collection, you specify options that apply to the entire collection. Enrichments Enrichments are annotators and classifiers that specify specific parts of the data to be used for analysis. Datasets A dataset defines data that is used by a collection. Crawlers and importers are used to extract data from data sources and add to a dataset. Collections A collection contains the entire set of sources that users can query at a time. When you create a collection, you specify options that apply to the entire collection. Add a collection You can add a collection using the IBM Watson Explorer Admin Console. On the Create Collection page, there are two options available for collection settings. Default Setting This setting creates a collection using the data you specify. It also uses standard enrichment, field, and facet definitions. Custom Setting This setting create a collection for which you specify the data, enrichments, fields, and facets. Add a collection using the default setting Creating collections with the default setting allows you to quickly start analyzing data using commonly used settings for enrichment. Before creating your collection, create a dataset that contains the data you wish to analyze. For more information, see Datasets on page 9. When you specify the default setting, the following settings are used for the enrichment, fields, and facets. Enrichment The enrichment uses the body text content field and the Part of Speech annotator. Fields Facets The default date field is date_added and the default title field is Document URI. Facets are used with default labels. Copyright IBM Corp

10 Creating a collection using default settings When you create a collection, you specify the name, description, and select Default Setting as the collection template. On the next page, you specify which dataset to use. You can also upload.csv files or crawl a filesystem. You then confirm your settings and you can now run indexing. Add a collection using the custom setting Creating collections with the custom setting gives you complete control over all components of the collection. Before creating your collection, create a dataset that contains the data you wish to analyze. For more information, see Datasets on page 9. You can also create custom annotators and classifiers, if desired. For more information, see Enrichment. Creating a collection using custom settings When you create a collection, you specify the name, description, and select Custom Setting as the collection template. On the next page, you specify which dataset to use. You can also upload.csv files or crawl a filesystem. On the following pages of the wizard, you can specify enrichments, fields, and facets. You then confirm your settings and you can now run indexing. Collection fields A document is composed of fields, which are more specific pieces of information. You use fields to specify the metadata to use in your analysis. On the Specify metadata fields you specify the default date and title fields. You can also override the default selection of metadata fields. Collection facets A facet is an unit of analysis. You analyze the unstructured content with facets and various statistics. Specifying meaningful labels for each facet is very important for your successful analysis. The difference between facets and fields is that multiple facets may be generated from a field. For example, a field (body, IBM Corporate headquaters is in Armonk, New York, United States ) may have multiple facets such as (noun, IBM ) and (verb, is ). The facets listed are produced by the annotators, classifiers, and metadata fields in the collection. You can modify the facet labels. Enrichment Enrichment is a process to analyze unstructured text content and extract significant data for analysis. Enrichment uses annotators and classifiers to perform this work. On the Enrichment page for a collection, you can select annotators and classifiers to use. 6 IBM Watson Explorer Community Edition

11 Annotators Annotators extract words and phrases from a dataset that satisfy the rules of the annotator being used. There are three built-in annotators available to all collections. Part of Speech The Part of Speech annotator extracts words and phrases from the given text and marks these extractions as annotations. For more information, see Part of Speech annotator. Named Entity Recognition The Named Entity Recognition annotator extracts person names, locations, and organizations from the given text and marks these extractions as annotations. For more information, see Named Entity Recognition annotator on page 8 Sentiment Analysis The Sentiment Analysis annotator extracts phrases and expressions which convey sentiment, and marks these extractions as annotations. For more information, see Sentiment Analysis annotator on page 8. There is also an annotator that you can create. Dictionary The Dictionary annotator uses a dictionary that contains words meaningful to your business. To improve text analysis, you can specify equivalent terms and different grammatical forms of the words. You can also map the words in a dictionary to a facet. To use a dictionary annotator, you must create or upload a dictionary containing your terms. For more information, see Dictionary annotator on page 8. Classifiers A classifier classifies documents in a dataset into categories. There is one of type of classifier that you can create. Supervised Multi-labeling The Supervised Multi-labeling classifier classifies documents into pre-defined classes. This classifier uses machine learning to determine how to classify documents. For more information, see Supervised Multi-labeling classifier on page 9. Part of Speech annotator The Part of Speech annotator extracts words and phrases using built-in linguistic analysis and default rules. These extractions are marked as annotations. The following annotations are made to the text. v Part of Speech Noun - General Noun, Others Verb, Adjective, Adverb, Conjunction, Interjection, Numeral v Phrase Constituent Noun Phrase - Noun Sequence, Modified Noun, Prep Noun Chapter 3. Administering content 7

12 Predicate Phrase - Predicate with Adverb, Noun - Predicate, Verb - Noun Conjunction Phrase - Resultative Conjunction, Contradictory Conjunction The following languages are supported. v Arabic, Czech, Danish, German, English, Spanish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Romanian, Russian, Slovak, Turkish, Chinese Named Entity Recognition annotator The Named Entity Recognition annotator extracts person names, locations, and organizations from the given text and marks these extractions as annotations. The following annotations are made to the text. v Named entity Person, Location, Organization The following languages are supported. v German, English, Spanish, French, Hebrew, Japanese, Chinese Sentiment Analysis annotator The Sentiment Analysis annotator extracts phrases and expressions which convey sentiment, and marks these extractions as annotations. The following annotations are made to the text. v Sentiment Analysis Phrase - Positive, Negative Expression - Positive, Negative Target (English and Japanese languages only) - Positive, Negative The following languages are supported. v Czech, English, Spanish, Hebrew, Japanese, Dutch, Romanian, Russian, Turkish, Chinese Dictionary annotator The Dictionary annotator uses a dictionary that contains words meaningful to your business. To improve text analysis, you can specify equivalent terms and different grammatical forms of the words. You can also map the words in a dictionary to a facet. After you create a dictionary annotator, you add dictionaries in the Dictionary tab. You can upload a list of words or type them manually in the user interface. In the Basic Settings tab, you specify the part of speech and language of the dictionary, and also optionally specify an associated facet. In the Word List tab, you can add or remove words and also specify other words with the same meaning as a word. 8 IBM Watson Explorer Community Edition

13 Datasets Supervised Multi-labeling classifier The Supervised Multi-labeling classifier classifies a document into pre-defined classes and give it labels representing assigned classes to it. A set of classes into which the documents are classified is defined by providing training data, which is a set of documents having correct labels. The document classification predicts types, topics, or user-defined categories of a document, based on a set of parameters trained with user-provided training data. The current document classifier implementation of Watson Explorer is classified as supervised multi-labeling. The word supervised means that this classifier uses supervised learning, a type of machine learning method that requires training data to learn how to recognize classes of documents based on their metadata and keywords in text. On the other hand, the word multi-labeling means it may give zero or more labels to a single document. Suppose we are going to apply the multi-labeling to news articles having labels representing typical topics such as politics, economy, sports, social, and science. As we can imagine, a single article may contain more than one topics, so multi-labeling suits to predict their topics. Training data is a set of examples where each example is a pair of a document and its expected labels. Below is a concrete example of a training example. [Input Document] "If you want to play baseball on the moon, you have to take into account the difference of gravity from the earth." [Expected Labels] sports, science A standard procedure for applying supervised multi-labeling to new documents is as follows: 1. Identify a use case and typical examples of an input document. 2. Define a set of possible labels. 3. Prepare training data. 4. Create and train a labeler (a component that does the supervised multi-labeling) using the training data. 5. Evaluate the accuracy of multi-labeling. 6. Deploy a trained model as a classifier instance to annotate labels to new documents. A dataset is a container where documents for searching and analysis are stored. You can add crawlers and importers to the dataset to import your data. Once your data is loaded into the dataset, you can add it to a collection for analysis. Add a dataset You add a dataset using the IBM Watson Explorer Admin Console. To add a dataset, click Add dataset on the Watson Explorer Admin Console home screen. After you create a dataset, you can populate the dataset with the following actions. 1. Create crawlers and importers to import the data for analysis. For more information, see Crawlers on page 10 and Importers on page 19. Chapter 3. Administering content 9

14 2. Modify the converter pipeline that converts the import documents into a usable form for analysis. For more information, see Converter pipeline on page Specify the date format, time zone, and locale on the Settings tab. Datasets A dataset is a container where documents for searching and analysis are stored. You can add crawlers and importers to the dataset to import your data. Once your data is loaded into the dataset, you can add it to a collection for analysis. Crawlers You configure crawlers for the different types of data that you want to include in a collection. A single collection can contain any number of crawlers. A crawler has two primary functions. When you configure a crawler, the discovery processes determine which sources are available in a data source. After you start a crawler, the crawler copies data from the data sources to a data store so that the parser can prepare the content for indexing and searching. The following crawlers are available in IBM Watson Explorer Community Edition. Box crawler The Box crawler crawls files stored on Box. Box ( is a cloud-based data storage repository. JDBC database crawler The JDBC database crawler crawls JDBC databases. UNIX file system crawler The UNIX file system crawler crawls file systems that have been mounted in Docker. This crawler can crawl Microsoft Windows file systems as well as UNIX or Linux file systems. Web crawler The Web crawler crawls web sites. Add a crawler After you create a dataset, you can add a crawler. After you click Add crawler on the Overview tab of your dataset, you select a crawler type. You now enter the configuration properties for the crawler. These properties vary depending on the crawler type. Go to the topic listed below for your crawler type to find out which configuration properties are needed. v Configuration properties for a Box crawler on page 11 v Configuration properties for a JDBC database crawler on page 12 v Configuration properties for a UNIX file system crawler on page 15 v Configuration properties for a Web crawler on page 16 After entering the configuration parameters, click Create to create the crawler. You will now see the page for your crawler. There are five tabs on this page. Overview On this tab, you can start the crawler using the crawl method specified in the configuration properties. The process of the crawl is displayed on this 10 IBM Watson Explorer Community Edition

15 page. You can also specify a different crawl method by clicking selecting a crawl method. The crawl methods are described here. Start normal crawl Crawl new, modified, and deleted content. Start quick crawl Crawl new and modified content. Start full crawl Crawl all content Refresh Configuration This tab displays the configuration parameters and you can modify them, if necessary. Field mappings After you crawl a data source, data source fields are automatically mapped to dataset fields, if possible. This tab shows the results of this mapping. You can modify the mapping of individual data source fields, as well as adding or removing data source fields. Schedules You can create schedules that specify when the crawler is run. Status You can view the status of each crawl space and converter pipeline. Configuration properties for a Box crawler The Box crawler crawls files stored on Box. Box ( is a cloud-based data storage repository. The Create crawler: Box screen is where you enter the configuration parameters for a box crawler. Crawler Properties Crawler name The name of the crawler. Alphanumeric characters, hyphens, underscores, and spaces are allowed. Crawler description A description of the crawler. Advanced properties Maximum document size The maximum size is 131,071 bytes. When the crawler session is started Specifies which content to crawl. Data Source Properties Client ID The private key specified by the Box App configuration. Client Secret The client secret specified by the Box App configuration. and Chapter 3. Administering content 11

16 Path to Private Key The path to the location of the private key that is part of the key pair generated for communication with box.com. The private key must be in DER or PEM format with either a.der or.pem extension. The folder that contains the private key must be mounted so it is available. For information on how to mount folders, see Providing access to the local file system from Watson Explorer Community Edition on page 2. Private Key Passphrase The passphrase required to decrypt the key if the private key is an encrypted file. Key ID The public key ID generated by Box. Enterprise ID of the Box account The Enterprise ID of the Box account. Comments Enable this option to crawl comments. Tasks Enable this option to crawl tasks. Crawl space Properties Crawl space Box Content type filter Select Included filter or Excluded filter. File content type to include exclude The file content types to include or exclude. Extension filter Select Included filter or Excluded filter. File extensions to include exclude The file extensions to include or exclude. Automatic code page detection Enable this option when the code page is unknown. Crawler plug-in Enable the crawler plug-in Enable this option when you use the crawler plug-in. Plug-in class name The class name for the crawler plug-in. Plug-in class path The JAR file location of the crawler plug-in. The folder that contains the JAR file must be mounted so it is available. For information on how to mount folders, see Providing access to the local file system from Watson Explorer Community Edition on page 2. Configuration properties for a JDBC database crawler The JDBC database crawler crawls JDBC databases. The Create crawler: JDBC database screen is where you enter the configuration parameters for a box crawler. 12 IBM Watson Explorer Community Edition

17 Crawler Properties Crawler name The name of the crawler. Alphanumeric characters, hyphens, underscores, and spaces are allowed. Crawler description A description of the crawler. Advanced properties Time to wait between retrieval requests The time is expressed in milliseconds. Maximum number of active crawler threads The maximum number of active crawler threads. Maximum number of documents to crawl The maximum number of documents to crawl. Maximum document size The maximum size is 131,071 bytes. When the crawler session is started Specifies which content to crawl. Data Source Properties JDBC database type The type of database. JDBC driver name The JDBC driver class name. For example, com.ibm.db2.jcc.db2driver or com.microsoft.sqlserver.jdbc.sqlserverdriver. JDBC driver classpath The location of the JDBC driver. For example, /mnt/jdbc/sqljdbc4.jar. The folder that contains the JAR file must be mounted so it is available. For information on how to mount folders, see Providing access to the local file system from Watson Explorer Community Edition on page 2. Database URL The database URL. Some examples are shown here. DB2 UDB: jdbc:db2://localhost:50000/sample Oracle: jdbc:oracle:thin:@localhost:1521:sample SQL Server: jdbc:sqlserver://localhost:1433;databasename=sample User name The user name to crawl the source. Password The password associated with the user name. Crawl space Properties You can find and add multiple crawl spaces for a JDBC database. For instructions, see Finding and adding crawl spaces in a JDBC database on page 14. Crawler plug-in Enable the crawler plug-in Enable this option when you use the crawler plug-in. Chapter 3. Administering content 13

18 Plug-in class name The class name for the crawler plug-in. Plug-in class path The JAR file location of the crawler plug-in. The folder that contains the JAR file must be mounted so it is available. For information on how to mount folders, see Providing access to the local file system from Watson Explorer Community Edition on page 2. Finding and adding crawl spaces in a JDBC database: You can define multiple crawl spaces in a JDBC database. Before you begin You must have entered the data source properties for the database. Procedure 1. Click Find and Add under Crawl space Properties. 2. In the Find and Add crawl spaces window, click Fetch children to populate the drop-down list with a list of possible crawl spaces. 3. Click on an item in the drop-down list, and then click Add and Close to select it. You are now returned to the page for your crawler. 4. Click, and then click Edit to enter the crawl space properties. The following properties can be set. All of the properties are optional. Crawl space name The crawl space name. SQL WHERE clause to limit the data to crawl Specify the SQL WHERE clause to limit the data to crawl. For example, Size < 1000 and Date > Primary key The primary key of the table. The value in the column specified as the primary key is used as a part of document ID. The crawler can automatically detect which column can be unique based on the table definition. However, if more than column can be used, you should select that column. Column with large binary or text content The key of the target database table to be handled as a content field. The following column types are supported. v java.sql.types.binary v java.sql.types.blob v java.sql.types.clob v java.sql.types.longvarbinary v java.sql.types.varbinary v java.sql.types.nclob If there is only one column in the target database table whose type is in the above list, this key is automatically detected. If there are multiple 14 IBM Watson Explorer Community Edition

19 columns, you need to select one of them because the JDBC database crawler does not support multiple content fields. MIME type of large binary or text content The MIME type of the content fields. You can force the MIME type to parse the contents. For example, application/pdf. Field to use for detecting updates This field is used to determine when documents are to be recrawled. Automatic code page detection Enable this option when the code page is unknown. What to do next You can repeat the above procedure to define any number of crawl spaces. Configuration properties for a UNIX file system crawler The UNIX file system crawler crawls file systems that have been mounted in Docker. This crawler can crawl Microsoft Windows file systems as well as UNIX or Linux file systems. The Create crawler: UNIX file system screen is where you enter the configuration parameters for a box crawler. Crawler Properties Crawler name The name of the crawler. Alphanumeric characters, hyphens, underscores, and spaces are allowed. Crawler description A description of the crawler. Advanced options Time to wait between retrieval requests The time is expressed in milliseconds. Maximum number of active crawler threads The maximum number of active crawler threads. Maximum number of documents to crawl The maximum number of documents to crawl. Maximum document size The maximum size is 131,071 bytes. When the crawler session is started Specifies which content to crawl. Crawl space Properties You can find and add multiple crawl spaces for a file system. For instructions, see Finding and adding crawl spaces in a file system on page 16. Crawler plug-in Enable the crawler plug-in Enable this option when you use the crawler plug-in. Plug-in class name The class name for the crawler plug-in. Chapter 3. Administering content 15

20 Plug-in class path The JAR file location of the crawler plug-in. The folder that contains the JAR file must be mounted so it is available. For information on how to mount folders, see Providing access to the local file system from Watson Explorer Community Edition on page 2. Finding and adding crawl spaces in a file system: You can define multiple crawl spaces in a file system crawler. Procedure 1. Click Find and Add under Crawl space Properties. 2. In the Find and Add crawl spaces window, click Fetch children to populate the drop-down list with a list of possible crawl spaces. 3. Click on an item in the drop-down list, and then click Add and Close to select it. You are now returned to the page for your crawler. 4. Click, and then click Edit to enter the crawl space properties. The following properties can be set. All of the properties are optional. Crawl space name The crawl space name. Level of subdirectories to crawl In the drop-down list you can specify the current directory only, all subdirectories, or a specific number of subdirectories to crawl. Field to use as the document date Document modification date By default, the modified dates of the crawled documents is stored as the native field ModifiedDate. Document crawl date Assigns the crawled dates to the native field ModifiedDate. Extension filter Select Included filter or Excluded filter. File extensions to include exclude The file extensions to include or exclude. Automatic code page detection Enable this option when the code page is unknown. What to do next You can repeat the above procedure to define any number of crawl spaces. Configuration properties for a Web crawler The Web crawler crawls web sites. The Create crawler: Web screen is where you enter the configuration parameters for a web crawler. 16 IBM Watson Explorer Community Edition

21 Crawler Properties Crawler name The name of the crawler. Alphanumeric characters, hyphens, underscores, and spaces are allowed. Crawler description A description of the crawler. Advanced properties User agent The user agent name to use when crawling. address The address of the user of the crawler. Time to wait between retrieval requests The time is expressed in milliseconds. Maximum document size The maximum size is 131,071 bytes. Minimum recrawl interval The minimum recrawl interval in days. Maximum recrawl interval The maximum recrawl interval in days. Ignore robots.txt Select to ignore allow and deny rules in robots.txt. Data Source Properties Start URL Specify the URLs where the crawler has to start crawling. Rules to Crawl Domains Specify the domain names that you want to allow or forbid the crawler to crawl. Domain names are case sensitive and the wildcard character (*) can occur anywhere in the domain name. The order of the rules is significant. The crawler applies the first rule that matches a candidate URL. The default rule, forbid domain *, forbids all Web crawling and must occur last in the list of domain rules. Some examples are shown here. v forbid domain (excludes the entire domain) v allow domain *.ibm.com (crawls any domain that ends in ibm.com) v allow domain server*.ibm.com:443 (crawls only port 443 on IBM domains that begin with "server") Rules to Crawl HTTP Prefixes Specify the HTTP and HTTPS prefixes that you want to allow or forbid the crawler to crawl. The wildcard character (*) can occur one or more times in the URL The order of the rules is significant. The crawler applies the first rule that matches a candidate URL. Some examples are shown here. v allow prefix (crawls pages in the public directory on this domain) Chapter 3. Administering content 17

22 v forbid prefix (excludes all other directories on this domain) Advanced crawler properties For advanced use only. Crawl space Properties You can find and add multiple crawl spaces for a web crawler. For instructions, see Finding and adding crawl spaces from web sites. Finding and adding crawl spaces from web sites: You can define multiple crawl spaces in a file system crawler Procedure 1. Click Add under Crawl space Properties. 2. On the Crawl space Properties page there are four types of settings to choose from. Each setting has its own set of parameters. v v v BASIC This setting is used for basic authentication. User name The user name of the user for authentication. Password The password of the user for authentication. Root URL The root URL to crawl. PROXY This setting is used for proxy access. User name The user name of the user for authentication. Password The password of the user for authentication. Proxy server domains Specify information about domains that are served by an HTTP proxy server. For the domain names, you can type an asterisk (*) to crawl all domains, or type a leading asterisk to crawl domains that match a pattern. For example, * or *server1.bar.com. Proxy server host name or IP address The proxy server host name or IP address. Proxy server port number The proxy server port number. NTLM This setting is used for NTLM authentication. User name The user name of the user for authentication. Password The password of the user for authentication. 18 IBM Watson Explorer Community Edition

23 Root URL The root URL to crawl. NTLM domain name The NTLM domain name. v NTLM host name The NTLM host name. FORM This setting is used for form-based authentication. Request type DIRECT Do not fetch login page. INDIRECT Fetch login page and fill parameters in login form. URL Pattern Scope to apply form base authentication. Login page Required for INDIRECT request type. Form name Required for INDIRECT request type. Form action Form action URL to submit form. Method to submit the form Select the POST or GET methods. Form parameters List of form parameter in <key=value> format. Importers An importer imports the data from a file directly into your dataset. The only importer currently available is the CSV Importer. Creating an importer 1. When you add an importer, you first give it a name and a description. 2. Upload the CSV files to import. You can upload multiple CSV files here. 3. On the parsing page, specify CSV parsing options such as the character set format, delimiters, headers (if any) and escape characters. The preview at the bottom of the page is updated according to selected parsing options. 4. Map CSV columns. Choose columns that you want to import and specify names of dataset fields where those columns are imported to. You can choose which columns to import, rename mapped dataset fields, and select the data type of each field. 5. You can now import the data by clicking Save. If Start importer now is checked, import will start immediately. Chapter 3. Administering content 19

24 Converter pipeline Converters convert data retrieved by crawlers into data that is suitable for indexing. Each converter can handle input that has its supported content types. In processing, the converter may change the content type for output. Data retrieved by crawlers can be converted by multiple converters in a converter pipeline. Converters run in the order found in the converter pipeline. Converters The Converter pipeline tab for a dataset lists, in order, the converters that have been selected for that dataset by IBM Watson Explorer Community Edition. You can make changes to the converters by clicking following actions are available. Edit Insert Up Down next to a converter. The Edit the settings for this converter. Some converters have settings that you can modify. These are described in Converters. Insert a new converter. Move this converter up one position. Move this converter down one position. Delete Delete this converter. Converters This topic lists the available converters and describes their settings, if any. Binary File Converter Parses a binary file. Settings Time to wait before a request times out Specify the time in seconds Content Type Converter Adds correct content type for metadata extracted by crawler. Settings None. CSV Converter Parses CSV file. Settings Separators Specify separators. For example,,;. Quote Specify a quotation character. For example, ". Starting line as a header Read the starting line as a header, which enables the column names to be mapped to index fields. Encoding Converter Detects the text encoding type and converts it. Settings None. 20 IBM Watson Explorer Community Edition

25 Field Filter Configures filters for parsing field values in different ways, such as combining field values into a single multiple-value field so that the values can be analyzed as a single facet. auxiliarysettings A list of field filters that you have created. For instructions, see Adding filters to a field filter on page 22. File Extension To Mime Type Converter Guesses content type by file extension. Settings Keep original mimetype Specify whether or not to keep the original mimetype. Ignore capital letters in file extensions Specify whether or not to ignore the capital letters in the file extensions. File extension to mimetype Specify the file extension to mimetype mapping. For example,.csv to text/csv. Guess Content Converter Guesses content type by file ccontent. Settings GZIP (gunzip) Converter Extracts.gz files. Settings None. HTML Converter Parses.html files. Settings Max bytes examined Maximum number of bytes examined by this converter. Keep original mimetype Specify whether or not to keep the original mimetype. Type override Specify type overrides For example, orgtype1 to newtype1. Meta names to output The content of the meta tags that match this set of wildcard expressions will be output. Body XPath When extracting the text of the page, this XPath expression can be used to extract one or more starting points on the page. If there are no matches for the XPath expression, the top of the page will be used instead. Tags to strip Any tag that matches this xpath expression will be discarded along with any subnodes it contains. This Chapter 3. Administering content 21

26 JSON Converter Parses JSON files. Settings None. Tar Converter Extracts.tar files. Settings None. Truncation Converter Truncates text fields. Settings ZIP Converter Extracts.zip files. Settings None. expression is the match on an xsl:select node. If a tag matches both the tags to strip and the tags to keep, the output is undefined. Tags to keep Any tag that matches this xpath expression will be included in the indexed text (without any attributes). Any tag that does not match either this xpath or the Tags to strip xpath will be removed but its children will be processed. This expression is the match on an xsl:select node. If a tag matches both the tags to strip and the tags to keep, the output is undefined. Include ALT text This flag enables or disables the inclusion of ALT text on images and anchor tags. Title XPath When extracting the title of the page, this XPath expression can be used to specify a specific location. If there are no matches for the XPath expression, the default HTML TITLE tag will be used instead. Disable title extraction If this is selected, titles will not be extracted from the HTML to create an extra "title" content. This is useful when the title is provided through external meta-data. Max Character Length Specifies the truncation limit. Adding filters to a field filter You can add multiple filters in order to parse field values in custom ways. To add a filter, click Add in the Field Filter. On the next screen, enter the following information. Filter name A unique name for the filter. 22 IBM Watson Explorer Community Edition

27 Index field The name of the field to apply this filter to. Condition Choose the precondition for this filter from the drop-down list. Click Add under Filters to specify the action of this filter. Choose an action from the drop-down list. Depending on the item, you may need to enter an additional parameter such as a target field name. Click OK to complete the field definition. Chapter 3. Administering content 23

28 24 IBM Watson Explorer Community Edition

29 Chapter 4. Mining content Collection detail view The IBM Watson Explorer Content Miner is the tool analysts use to analyze and create collections. If your administrator has created collections, you will see these on the home page. As well, Watson Explorer Community Edition comes with three collections you can begin mining. You can immediately begin an analysis of any collections on the page, you can edit details of those collections, and you can create new collections, classifiers, and custom annotators. For information about the collections, enrichments, facets and fields you can work with in Watson Explorer Content Miner, see the following: v Collections on page 5 v Enrichment on page 6 v Collection facets on page 6 v Collection fields on page 6 What do you want to analyze? The purpose of this view is to collect your interests and clues to suggest mining actions in the Guided Analysis view on page 26. Figure 1. Collection detail view In the query input line: v Specify query text and facets to analyze. v Type ahead lists all facets and facet values. v Keep blank to analyze whole document sets. v Click the magnifier icon to move to Guided Analysis view. Facets: You can click facets to add them to your query. Or change the facet type from the drop-down list (right side of view). You can show facets and values in the dashboard, metadata facets and values, or you can create a custom dashboard. Date range: Show a time series chart for date facets to specify a date range to analyze. Saved analysis: Resume a query from the save time. FAQ Q: How to go back to collections or collection detail page. A: From breadcrumbs. Copyright IBM Corp

30 Guided Analysis view Q: Can I skip the collection detail page? A: User can bookmark both Collection Detail page and Guided Analysis View page. Q: When I try to analyze a facet whose type is Number or Date from Custom Analysis, why are the results are always zero? A: This is current limitation. For Date type, please use timeseries, trends and topic analysis instead. The purpose of the Guided Analysis view is to let you get insight by just following the suggested actions. With this view, you can analyze the target collection without knowledge of text mining through the Guided Analysis experience. Watson Explorer suggests available actions for the current context. You simply select the action, and then Watson Explorer automatically analyzes the content and shows the results. When you find the words and phrases in which you are interested, just select them to narrow down the results, and then actions are suggested again. Guided Analysis view consists of these components: v Mining graph v Custom analysis on page 29 v Saved analysis on page 31 v Settings v Toolbar on page 31 v And there are three views within the larger Guided Analysis view: Documents view on page 32 Timeseries view on page 32 Dashboard view on page 33 Mining graph The following mining graph represents the following mining steps: All documents in the collection + Location:Manhattan + (Client Segment:Silver Card Member AND Product Line:Ice cream). The All Documents node and current query node have document counts. 26 IBM Watson Explorer Community Edition

31 Figure 2. Mining graph v Nodes A circle node represents an analysis query and its result document set. For example, Product Line: Ice cream. The magnifier nodes represent actions you can take based on context. For example: Show Documents. The left-most single node represents "All Documents" of this collection. Node type: - All Documents: shows a count of all documents and the collection name. - Facet query node: Represents facet query and date range query. Shows facet name and values. If multiple values are specified, comma separated values are displayed. Conjunction is "OR". - Conjunction node: Represents AND or OR. - Text query node: Represents all other query text. The node that represents the current analysis context is bold, and an arrow icon is displayed under the node to indicate that this node is current analysis target. Note that you can show analysis results from the node context menu even when that node is not the current context (see Context menu for graph node). v Link A node is added when user narrows down the document. Each link between nodes represents this relationship. A dotted link represents suggested actions and its analysis target document set. v Suggested Actions Actions are suggested for the current context node. Actions include: Analyze cause or characteristic: - Automatically examines various facets defined in the category tree, and suggests characteristic words and its facets. - Not available for All Documents node. Chapter 4. Mining content 27

32 Analyze trend and anomaly: - Invokes a topic and trends analysis for the current context node. This action is suggested only when the current context node is a facet query node and when there is a date facet. - When there is no date facet, this action is not displayed. Analyze sentiment reputation: - Invokes a correlation analysis for a sentiment facet (positive/negative target, expression, phrase facets). Custom Analysis: - Shows Custom Analysis view (if the Custom Analysis view is visible, this action is not suggested). Show Documents: - Shows Documents view for you to see current documents in a result set. v Context menu for graph node 28 IBM Watson Explorer Community Edition

33 Click a graph node to display its context menu. Set Current Context: Set this node as the current analysis context. Show Past Analysis Results: Show all the analysis results you invoked for this clicked node. Add Query Text: Show query input to add query text to the current query node. Show Documents: Show the Documents view for this clicked node. Show Timeseries: Show the Timeseries view for this clicked node. Delete: Delete this node (for a node that is the child of a conjunction, only the last three actions are available). Custom analysis Chapter 4. Mining content 29

34 v Analysis Type: Select analysis types from the drop-down. Choices are Correlation, Topic, Trends, and Pairs. A correlation analysis provides a facet table (used when one facet is selected or the result has a long label), a word cloud (used when multiple facets are selected), or a map (state map only; only used when the facet name is "state"). A topic or trends analysis provides a time series as an overview line chart. You can show all date ranges in index and current selected range. You can resize, and move to change selected range. Use a heatmap view to see highlighted cells at a glance. To see a bar chart, click on the toggle arrow icon. The pairs analysis helps you to identify a high correlation of facet values from two selected facets. The Content Miner requires two sets of search results to calculate a correlation. Accordingly, you select two facets that represent the two search result sets that you want to explore. For example, if you pick a pairs analysis, and then select Client Age and Product Line facets (and click Analyze), we can see a results such as the following image. 30 IBM Watson Explorer Community Edition