Type of Submission: Article Title: Watson Explorer REST API Tutorial #1 Subtitle: Java Programming. Keywords: WEX, WCA, analytics, Watson

Transcription

1 Type of Submission: Article Title: Watson Explorer REST API Tutorial #1 Subtitle: Java Programming Keywords: WEX, WCA, analytics, Watson Prefix: Mr. Given: Kameron Middle: A. Family: Cole Suffix: Job Title: Watson L2 Support Engineer Bio: Kameron Cole has been involved with Watson Content Analytics technology, formerly known as TAKMI, for 10 years. Kameron has designed and implemented large-scale Watson Content Analytics Solutions all over the world. He holds the first United States Patent in this technology, including three additional patents. Company: IBM Watson Photo filename: Abstract: One of the most powerful offerings in the Watson Content Analytics middleware is its robust REST API. Requiring no runtime and no compilation, REST code can be integrated with any programming that has access to Hypertext Transfer Protocol, providing access to real-time Watson cognitive analytical analysis. This first article lays the REST foundation, and offers three Java options for processing results.

2 Watson Explorer REST API Tutorial #1 Java Programming Contents Introduction...4 Search API: /search...4 Search Single collection, sorted by date (most recent first)...5 Search on another collection, also sorted by date...6 Search on two collections at once, combined results ordered by date...7 FireFox RestClient...8 Admin API: /pear...8 Java Programming Choices for processing returned results: String atom JSON Extended json Example: Using the Admin API to retrieve Processing Engine Archives Appendix X: Java StringWriter() code sample xx Listing X: atom code sample xx Listing X: json code sample xx Listing X: json code sample xx... 23

3 Author Date Version Kameron Cole July 28, Vijai Gandikota Mamatha Nagaraju

4 Introduction Representational State Transfer (REST) has gained widespread acceptance across the Web a simpler alternative to SOAP- and Web Services Description Language (WSDL)-based Web services. Adoption of REST by mainstream Web 2.0 service providers including Yahoo, Google, and Facebook who have deprecated or passed on SOAP and WSDL-based interfaces The REST API consists of two categories of APIs: APIs for search and text mining tasks The Search REST API is available on a search application infrastructure and listens on the search application port, which by default is port 8393 if it's an embedded server install. name>:<search application port>/api/v10/ for the Search REST API APIs for administration tasks The Administrative REST API is available on index servers and uses the same port number as the administrative console, which by default is 8390 if it's an embedded server install. name>:<administrative console port>/api/v10/ for the Administrative REST API When WebSphere Application Server installation option is selected, the default port is 80 for both API sets. You can change these port numbers when you install the product. Search API: /search All queries for the /search REST API have the format: hostname:port/api/v10/search In all cases, you will need to signify a collection, and, of course, a query. Some other useful parameters include:

5 sortkey some parametric field to be used to sort the results pagesize this is actually the number of results you want returned output this would be with XML using atom formatting, or json. If absent, atom is assumed Search Single collection, sorted by date (most recent first) e&pagesize=4

6 Search on another collection, also sorted by date e&pagesize=4

7 Search on two collections at once, combined results ordered by date CollectionB&query=cancer& sortkey=date&pagesize=10

8 FireFox RestClient A very useful development tool for viewing formatted REST responses is the RestClient from Firefox: Once this plugin is installed, you can view nicely formatted results in your Firefox browser. The next examples will demonstrate the RestClient. Admin API: /pear For analytics and data mining applications, there are a number of APIs for working with Processing Engine Archives (.pear extension). They have the canonical form: name>:<administrative console port>/api/v10/admin name>:<administrative console port>/api/v20/admin As with all REST calls, XML and JSON are available as return types. You can specify "&output=application/xml" to get XML and "&output=application/json" to get a return value in JSON.

9 Important: for Amin 1.0 API calls you need the parameters &api_username=<username>&api_password=<password> For the Admin 1.0 API, we can get a list of installed pears with /pear?method=getlist&api_username=<username>&api_password=<password> For the Admin API v2.0, there are two steps required: 1) First we must get a token, which can be then stored, and reused throughout the session. This provides better security

10 name>:<administrative console port>/api/v20/admin/login?username=username&password=password 2) Now we can use this token to call the /pear API /system/parse/pear/list&securitytoken=<security token obtained above>

11 Java Programming Java programming of REST has the additional step of establishing the HTTP transport. This can be done by the use of the Apache HttPClient package. NOTE: THERE HAVE BEEN SIGNIFICANT CHANGES TO THIS PACKAGE RECENTLY. THE CODING EXAMPLES PRESENTED ARE FROM V4.3 To invoke the /search API, we create the URL: private static final String URL = " The HttpClient has change a great deal. This is the code to instantiate a client that will close after the method exits. CloseableHttpClient httpclient = HttpClients.createDefault(); The previous method methods have been replaced by corresponding Objects. HttpPost httppost = new HttpPost(URL);

12 To add parameters, we create a name-value pair List List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("collection", "CollectionA CollectionB")); nvps.add(new BasicNameValuePair("query", "cancer")); nvps.add(new BasicNameValuePair("sortKey", "date")); nvps.add(new BasicNameValuePair("pageSize", "4")); The basic parameters are the collection ID (this is not the collection name), and the query. We have added the sortkey of date and pagesize of 4. Recall that pagesize is the equivalent of count. With the list parameters the final URL is construction and executed as an entity, following the classic command pattern of the Gang of Four ("Design Patterns: Elements of Reusable Object-Oriented Software", by Gamma, Helm, Johnson, Vlissides). httppost.setentity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response = httpclient.execute(httppost); The entity is responsible for getting the content: HttpEntity entity = response.getentity(); At this point, we need to decide how we want to get the content, which can vary. Choices for processing returned results: String The simplest approach is to use a StringWriter to capture the entire response as a String. HttpEntity entity = response.getentity(); InputStream is = entity.getcontent(); StringWriter writer = new StringWriter(); IOUtils.copy(is, writer, "UTF-8"); String results = writer.tostring(); System.out.println(results); The only special thing is this code is the use of Apache Commons IOUtils The output is straightforward:

13 atom Since atom is the default response, we will show a simple way to capture the results in a meaningful way. We are using the Rome extensions for Java We begin by capturing the content as a SynFeed Object: HttpEntity entity = response.getentity(); InputStream is = entity.getcontent(); SyndFeed feed = null; InputSource source = new InputSource(is); SyndFeedInput input = new SyndFeedInput(); feed = input.build(source); We are going to do a little extra Java now. Although the REST APIs allows us to sort the results by parametric values JSON Search results can be returned using json technology. To get the results returned as json, you will need to add a parameter to the search string: output=application/json In the Java code we have been working on, it would be like this:

14 The basics units of json responses, in terms of Java, are JsonObjects, and JsonArrays. In the return string, JsonObject are within a curly brace, and arrays are within a square bracket: Continuing to use the HttpClient code as outlined above, we want to get the response body. However, the first line in the response will NOT be understood by the json utilities. This is because it is the Http status: try { String status = response.getstatusline().tostring(); System.out.println(response.getStatusLine()); Output: HTTP/ OK There is a special method in the HttClient API (EntityUtils) to get the body, without the status line, as a String: CloseableHttpResponse response = httpclient.execute(httppost); HttpEntity entity = response.getentity(); String bodyasstring = EntityUtils.toString(response.getEntity()); Once we have the body as a String, we can parse it with a JsonParser. This takes a basic java.io.reader as an argument, which reads the string: Reader reader = new StringReader(bodyAsString); JsonParser jr = Json.createParser(reader); This parser divides the json message into events (like a SAX xml parser). These are javax.json.stream.jsonparser.event Event event = null; The first event in the response is "es_apiresponse":{"es_evaluationtruncation":false,"es_querye valuationtime":1,"es_totalresults":30676,"es_totalresultstyp e":"estimated","es_numberofavailableresults":500,"es_numbero festimatedresults":30676,"es_query":[{"role":"request","sear chterms":"cancer"

15 We will find this first Event and stop on it: // Advance to "es_apiresponse" key while(jr.hasnext()) { event = jr.next(); if(event == Event.KEY_NAME && "es_apiresponse".equals(jr.getstring())) { event = jr.next(); Now we can print out all the values of the es_apiresponse object: // Output contents of "es_apiresponse" object while(event!= Event.END_OBJECT) { switch(event) { case KEY_NAME: { System.out.print(jr.getString()); System.out.print(" = "); case VALUE_FALSE: { System.out.println(false); case VALUE_NULL: { System.out.println("null"); case VALUE_NUMBER: { if(jr.isintegralnumber()) { System.out.println(jr.getInt()); else { System.out.println(jr.getBigDecimal()); case VALUE_STRING: { System.out.println(jr.getString()); case VALUE_TRUE: { System.out.println(true); default: { event = jr.next(); finally{system.out.println("eof"); The output looks like this:

16 es_evaluationtruncation = false es_queryevaluationtime = 0 es_totalresults = es_totalresultstype = estimated es_numberofavailableresults = 500 es_numberofestimatedresults = es_query = role = request searchterms = cancer EOF Extended json Example: Using the Admin API to retrieve Processing Engine Archives int pearlov = 0; values count // I am declaring an integer to keep a track of the list of while ((line = reader.readline())!= null) { // for each line returned by WCA System.out.println(line); // first I am printing the line for reference pearlov = pearlov + 1; // I am increasing the count JsonReader jsonreader = Json.createReader(new StringReader("{")); // I am creating a dummy JsonReader JsonObject json = jsonreader.readobject(); // I am creating a JsonObject [NOTE: Not JSONObject. The methods available are different in both] jsonreader = Json.createReader(new StringReader(line)); // Now I am reading in my line which is a response back from WCA for the list of pears // But WCA doesnt return multiple lines for each pear. It only gives back one json line. peararray = jsonreader.readarray(); // So I create an array (JsonArray peararray;) and read the array into it from the JsonReader object int len = peararray.size(); // I check the length of the array System.out.println("The array has size = " + len); // Then for each element in the array for(int i=0; i<len;i++){ JsonObject jsonobject = peararray.getjsonobject(i);

17 // I first create a JsonObject [Again note not JSONObject] String ID = jsonobject.getstring("id"); etc etc. // I get one of the elements in the row that I want. System.out.println(ID); // And then I print it. Or store it to the database finally{ This produces the following output: [{"ID":"PearId1","NLQConfigurationGroups":[],"classpath":null,"installDir":"\/g pfs\/bi\/wcadata\/data\/pearsupport\/pearid1","maincompdesc":"com.ibm.watson.wc a.wca35_enron_00_pear.xml","name":"enron35_01","type":null,{"id":"pearid2","nl QConfigurationGroups":[],"classpath":null,"installDir":"\/gpfs\/bi\/wcadata\/da ta\/pearsupport\/pearid2","maincompdesc":"com.ibm.watson.l2_pear.xml","name":"g ENE00","type":null,{"ID":"PearId3","NLQConfigurationGroups":[],"classpath":nul l,"installdir":"\/gpfs\/bi\/wcadata\/data\/pearsupport\/pearid3","maincompdesc" :"com.ibm.watson.l2_pear.xml","name":"gene01","type":null,{"id":"pearid4","nlq ConfigurationGroups":[],"classpath":null,"installDir":"\/gpfs\/bi\/wcadata\/dat a\/pearsupport\/pearid4","maincompdesc":"com.ibm.watson.l2_pear.xml","name":"cl oudgenomics2","type":null,{"id":"pearid5","nlqconfigurationgroups":[],"classpa th":null,"installdir":"\/gpfs\/bi\/wcadata\/data\/pearsupport\/pearid5","mainco mpdesc":"com.ibm.watson.l2_pear.xml","name":"cloudgenomics3","type":null] The array has size = 5 PearId1 PearId2 PearId3 PearId4 PearId5 Appendix X: Java StringWriter() code sample xx package com.ibm.watson.rest.search.ordered; import java.io.ioexception; import java.io.inputstream; import java.io.stringwriter; import java.util.arraylist; import java.util.iterator; import java.util.list; import java.util.treeset; import org.apache.commons.io.ioutils; import org.apache.http.httpentity; import org.apache.http.namevaluepair;

18 import org.apache.http.client.entity.urlencodedformentity; import org.apache.http.client.methods.closeablehttpresponse; import org.apache.http.client.methods.httppost; import org.apache.http.impl.client.closeablehttpclient; import org.apache.http.impl.client.httpclients; import org.apache.http.message.basicnamevaluepair; import org.apache.http.util.entityutils; import org.xml.sax.inputsource; import com.sun.syndication.feed.synd.syndfeed; import com.sun.syndication.io.syndfeedinput; public class OrderedSearchQueryString { private static final String URL = " /** args */ public static void main(string[] args) throws Exception { CloseableHttpClient httpclient = HttpClients.createDefault(); HttpPost httppost = new HttpPost(URL); List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("collection", "CollectionA CollectionB"));; nvps.add(new BasicNameValuePair("query", "cancer")); nvps.add(new BasicNameValuePair("sortKey", "date")); nvps.add(new BasicNameValuePair("pageSize", "4")); httppost.setentity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response = httpclient.execute(httppost); try { System.out.println(response.getStatusLine()); HttpEntity entity = response.getentity(); InputStream is = entity.getcontent(); StringWriter writer = new StringWriter(); IOUtils.copy(is, writer, "UTF-8"); String results = writer.tostring(); System.out.println(results); EntityUtils.consume(entity); finally { response.close(); private static void writeresults(inputstream is) { StringWriter writer = new StringWriter();

19 try { IOUtils.copy(is, writer, "UTF-8"); catch (IOException e) { // TODO Auto-generated catch block e.printstacktrace(); String results = writer.tostring(); System.out.println(results); Listing X: atom code sample xx package com.ibm.watson.rest.search.ordered; import java.io.ioexception; import java.io.inputstream; import java.io.stringwriter; import java.util.arraylist; import java.util.iterator; import java.util.list; import java.util.treeset; import org.apache.commons.io.ioutils; import org.apache.http.httpentity; import org.apache.http.namevaluepair; import org.apache.http.client.entity.urlencodedformentity; import org.apache.http.client.methods.closeablehttpresponse; import org.apache.http.client.methods.httppost; import org.apache.http.impl.client.closeablehttpclient; import org.apache.http.impl.client.httpclients; import org.apache.http.message.basicnamevaluepair; import org.apache.http.util.entityutils; import org.xml.sax.inputsource; import com.sun.syndication.feed.synd.syndfeed; import com.sun.syndication.io.syndfeedinput; public class OrderedSearchQueryATOM { private static final String URL = " /** args */ public static void main(string[] args) throws Exception {

20 CloseableHttpClient httpclient = HttpClients.createDefault(); HttpPost httppost = new HttpPost(URL); List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("collection", " CollectionA CollectionB")); //nvps.add(new BasicNameValuePair("collection", "ncbi")); nvps.add(new BasicNameValuePair("query", "cancer")); nvps.add(new BasicNameValuePair("sortKey", "date")); nvps.add(new BasicNameValuePair("pageSize", "4")); httppost.setentity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response = httpclient.execute(httppost); try { System.out.println(response.getStatusLine()); HttpEntity entity = response.getentity(); InputStream is = entity.getcontent(); //Rome stuff for reading atom SyndFeed feed = null; TreeSet<String> tree = new TreeSet<String>(); Iterator<String> iterator = tree.iterator(); InputSource source = new InputSource(is); SyndFeedInput input = new SyndFeedInput(); feed = input.build(source); List entries = feed.getentries(); int ctr=0; while (ctr<entries.size()) { String value = entries.get(ctr).tostring(); tree.add(value); ctr++; System.out.println("Set: "+tree); // writeresults(is); EntityUtils.consume(entity); finally { response.close(); private static void writeresults(inputstream is) { StringWriter writer = new StringWriter(); try { IOUtils.copy(is, writer, "UTF-8"); catch (IOException e) { // TODO Auto-generated catch block e.printstacktrace();

21 String results = writer.tostring(); System.out.println(results); Listing X: json code sample xx package com.im.watson.rest.admin; import java.io.bufferedreader; import java.io.reader; import java.io.stringreader; import java.util.arraylist; import java.util.list; import javax.json.json; import javax.json.jsonarray; import javax.json.jsonobject; import javax.json.jsonreader; import javax.json.stream.jsonparser; import javax.json.stream.jsonparser.event; import org.apache.http.namevaluepair; import org.apache.http.client.entity.urlencodedformentity; import org.apache.http.client.methods.closeablehttpresponse; import org.apache.http.client.methods.httppost; import org.apache.http.impl.client.closeablehttpclient; import org.apache.http.impl.client.httpclients; import org.apache.http.message.basicnamevaluepair; import org.apache.http.util.entityutils; public class AdminPEARListJson { private static final String URL = " /** args */ public static void main(string[] args) throws Exception { CloseableHttpClient httpclient = HttpClients.createDefault(); HttpPost httppost = new HttpPost(URL); List <NameValuePair> nvps = new ArrayList <NameValuePair>();

22 nvps.add(new BasicNameValuePair("method", "getlist")); nvps.add(new BasicNameValuePair("api_username", "user")); nvps.add(new BasicNameValuePair("api_password", "password")); nvps.add(new BasicNameValuePair("output", "application/json")); httppost.setentity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response = httpclient.execute(httppost); //pear?method=getlist&api_username=<username>&api_password=<password> try { String status = response.getstatusline().tostring(); String bodyasstring = EntityUtils.toString(response.getEntity()); String line = ""; JsonArray peararray; BufferedReader reader = new BufferedReader(new StringReader(bodyAsString)); int pearlov = 0; reader.readline())!= null) { System.out.println(line); while ((line = pearlov = pearlov + 1; JsonReader jsonreader = Json.createReader(new StringReader("{")); JsonObject json = jsonreader.readobject(); jsonreader = Json.createReader(new StringReader(line)); peararray = jsonreader.readarray(); int len = peararray.size(); " + len); for(int i=0; i<len;i++){ System.out.println("The array has size = JsonObject jsonobject = peararray.getjsonobject(i);

23 String ID = jsonobject.getstring("id"); System.out.println(ID); finally{ Listing X: json code sample xx package com.ibm.watson.rest.search.ordered; import java.io.reader; import java.io.stringreader; import java.util.arraylist; import java.util.list; import javax.json.json; import javax.json.stream.jsonparser; import javax.json.stream.jsonparser.event; import org.apache.http.namevaluepair; import org.apache.http.client.entity.urlencodedformentity; import org.apache.http.client.methods.closeablehttpresponse; import org.apache.http.client.methods.httppost; import org.apache.http.impl.client.closeablehttpclient; import org.apache.http.impl.client.httpclients; import org.apache.http.message.basicnamevaluepair; import org.apache.http.util.entityutils; public class OrderedSearchQueryJSON { private static final String URL = " /** args */ public static void main(string[] args) throws Exception {

24 CloseableHttpClient httpclient = HttpClients.createDefault(); HttpPost httppost = new HttpPost(URL); List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("collection", "CollectionA CollectionB")); nvps.add(new BasicNameValuePair("query", "cancer")); nvps.add(new BasicNameValuePair("sortKey", "date")); nvps.add(new BasicNameValuePair("pageSize", "4")); nvps.add(new BasicNameValuePair("output", "application/json")); httppost.setentity(new UrlEncodedFormEntity(nvps)); CloseableHttpResponse response = httpclient.execute(httppost); try { // String status = response.getstatusline().tostring(); String bodyasstring = EntityUtils.toString(response.getEntity()); Reader reader = new StringReader(bodyAsString); JsonParser jr = Json.createParser(reader); Event event = null; // Advance to "es_apiresponse" key { while(jr.hasnext()) { event = jr.next(); if(event == Event.KEY_NAME && "es_result".equals(jr.getstring())) event = jr.next(); // Output contents of "es_apiresponse" object while(event!= Event.END_OBJECT) { switch(event) { case KEY_NAME: { System.out.print(jr.getString()); System.out.print(" = "); case VALUE_FALSE: { System.out.println(false); case VALUE_NULL: { System.out.println("null"); case VALUE_NUMBER: { if(jr.isintegralnumber()) {

25 System.out.println(jr.getInt()); else { System.out.println(jr.getBigDecimal()); case VALUE_STRING: { System.out.println(jr.getString()); case VALUE_TRUE: { System.out.println(true); default: { event = jr.next(); finally{system.out.println("eof");