Simple Data Source Crawler Plugin to Set the Document Title

Size: px

Start display at page:

Download "Simple Data Source Crawler Plugin to Set the Document Title"

Merilyn Shepherd
6 years ago
Views:

1 Simple Data Source Crawler Plugin to Set the Document Title IBM Content Analytics 1

2 Contents Introduction... 4 Basic FS Crawler behavior Using the Customizer Filter to Modify the title Field Using a crawler plugin to set the title Type A Web Crawler, Pre-fetch Plugin Web Crawler, Post-Parse Plugin Data Source Crawler, Type A Data Source Crawler, Type B Debugging crawler plugins Getting the content Use Java Regex to pick the first line in the document Setting Existing Index Fields Results Appendix A: Plugin Source Code Appendix B: Source code for Java Regex testing

3 Author Date Version Kameron Cole January 6,

Introduction The title of a document is not as obvious to a machine, as it is to a human. Consider this Word doc Clearly, I want This is the Title to be the title!

4 Introduction The title of a document is not as obvious to a machine, as it is to a human. Consider this Word doc Clearly, I want This is the Title to be the title! I have selected the Title Font Style, after all! Even Windows wants to call this document This is the Title note what Windows chooses as the defualt filename when I click Save 4

5 Strangely enough, though, Windows does NOT actually set the title Property to anything: 5

6 Of course, I could set it myself 6

7 How many users of Word even know that this is possilbe? Further, I can always save the document with a completey different filename, or, even different title Property 7

8 Basic FS Crawler behavior. In my test collection, I have created two txt files: The contents of the files look like: 8

9 Note the the defaul behavior of any file system crawler is to use the file name as the title: We can explore this in the crawler s metadata Fields. Note that both _$FileName$_ and _$Title$_ are mapped to index Fields, and are avaialbe for fielded search. 9

10 If we do a fielded search for each field, we get the same results: 10

11 11

12 For demonstration purposes, we will put both Fields into the summary: 12

$remove the file extension and the dot /\.$

13 Using the Customizer Filter to Modify the title Field I can use this regular expression in a Filter in the Application Customizer to remove the file extension and the dot /\..*$// 13

14 Using a crawler plugin to set the title A simple use case might be that the first line of each document is the title at least is it more likely to be the title that the filename. The Title Filter above will not work for this scenario. There are two other options: a Post-parse Crawler Plugin and a UIMA Annotator. Of these, the Crawler Plugin will be the best for performance, simplicity, and manageability. Type A There are several types of crawler plugins: Web Crawler, Pre-fetch Plugin Web Crawler, Post-Parse Plugin Data Source Crawler, Type A Crawler types: Content Integrator Content Manager DB2 JDBC database Notes Quickr for Domino Seed list File system Data Source Crawler, Type B Crawler types: Agent for Windows file systems crawler BoardReader crawler Case Manager crawler Exchange Server crawler FileNet P8 crawler SharePoint crawler 14

15 In this article, we will focus on a Windows File System crawler, which is Type A. It is always a good idea to begin with looking at the sample application, MyCrawlerPlugin.java, in $ES_INSTALL_ROOT/samples/dscrawler Here are the fundamental points for creating this type of crawler plugin: Extend com.ibm.es.crawler.plugin.abstractcrawlerplugin. The AbstractCrawlerPlugin class is an abstract class. Implement the following methods: o o o o o o o init() ismetadataused() iscontentused() activate() deactivate() term() updatedocument() The init, activate, deactivate, and term methods are implemented to do nothing. The ismetadataused method and iscontentused method are implemented to return false by default. See note below for important consequeces. The updatedocument method is an abstract method, so you must implement it. For name resolution, use the ES_INSTALL_ROOT/lib/dscrawler.jar file. Debugging crawler plugins The only way to do debugging is to write to the product logs. This is done via a custom logging utility: /** Logger */ private static final PluginLogger logger; static { PluginLogger.init(PluginLogger.LOGTYPE_OSS,PluginLogger.LOGLEVEL_INFO); logger = PluginLogger.getInstance(); /** End Logger **/ Here are the available methods to write to the logs /* Testing Logging Statements*/ logger.info("this is info."); 15

16 logger.warn("this is warning."); logger.error("this is error."); /* End Testing Logging Statements */ It is important to note that the default logging settings for a Collection are set to WARNING/ERROR. This means that if you leave the settings as default, you will not see the above statement "This is info." To show informational messages in the collection log file, open the administration console. Select the collection, click Actions > Logging > Configure log file options, and then select All messages for the type of information to log and trace. After you stop and restart the crawler session, informational messages appear in the collection log file. All Logger statements will show up in a new, separate log, called <collection_id>_<date_stamp> Getting the content The updatedocument() method takes as an argument a pointer to the content of each document, in a CrawledData public CrawledData updatedocument(crawleddata crawleddata) To get the actual contents, you call: List<Content> orgcontentlist = crawleddata.getoriginalcontents(); Note that a List is required, since the API supports compound documents (there may be more than one document in a content item). The result List returned by getoriginalcontents method cannot be modified. If you need to modify the content, you must create a new list containing all contents to be indexed. List<Content> contentlist = new ArrayList<Content>(); if (orgcontentlist.isempty()) logger.error("orgcontentlist is empty"); for (int i = 0; i < orgcontentlist.size(); i++) { contentlist.add(orgcontentlist.get(i)); In this article, we do not modify the contents. However, as a best practice, we will still create the copy. 16

17 Note: even though you do not modify the contents, you must set the iscontentused() method to return true (see above). If you do not, the crawleddata.getoriginalcontents() will return an empty list. public boolean iscontentused() { return true; Note: If you use the Eclipse New Class wizard to generate unimplemented methods, iscontentused() will NOT get auto-generated. Use Java Regex to pick the first line in the document We use the following regex declaration to pick the first line in any document: private static final String TITLE_PATTERN = "^\\A(.*)"; private Pattern titlepattern = Pattern.compile(TITLE_PATTERN); We will pass the entire content as a String into the following method: private String gettitle(string doc) { List<String> result = new ArrayList<String>(); matcher = titlepattern.matcher(doc); while (matcher.find()) { result.add(matcher.group(1)); return result.get(0); It is critical to insure that the String content is copied exactly, so as to preserve the line feeds and carriage returns. The traditional Java method of using readline() would actually skip everything in the document, except the textual content. Therefore, you need to use the read() method, and get each character: // Now read the content into a String 17

InputStream in = null; BufferedReader br = null; StringBuilder sb = new StringBuilder(); try { int value = 0; for (Content content : contentlist) { if (contentlist.isempty()) logger.

18 InputStream in = null; BufferedReader br = null; StringBuilder sb = new StringBuilder(); try { int value = 0; for (Content content : contentlist) { if (contentlist.isempty()) logger.error("contentlist is empty"); // Get InputStream object to ready the content. in = content.getinputstream(); br = new BufferedReader(new InputStreamReader(in)); while ((value = br.read())!= -1) { char c = (char) value; sb.append(c); contentstring = sb.tostring(); in.close(); catch (IOException ioe) { throw new CrawlerPluginException(ioe); title = gettitle(contentstring); Setting Existing Index Fields There is one method to set Index fields (the methods are distinct for datasource and web crawler (wc): You must first get the Fields: List<FieldMetadata> metadatalist = crawleddata.getmetadatalist(); if (!(metadatalist == null)) { 18

19 for (int i = 0; i < metadatalist.size(); i++) { logger.warn("field name: " + metadatalist.get(i).getfieldname() + "\n"); logger.warn("field value: " + metadatalist.get(i).getvalue() + "\n"); logger.warn("\n"); else logger.error("metadatalist is empty"); It is very important to note that, even though the Metadata Fields are all mapped to Index Fields, the CrawlerPlugin API only returns the Metadata Field names. In other words, this code logger.warn("field name: "+metadatalist.get(i).getfieldname()+"\n"); logger.warn("field value: "+metadatalist.get(i).getvalue()+"\n"); produces: <OFMsg> "2" " "664"24" title_test" title_test.win_98180.crawlerplugin" WIN-134UMQA234V" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $Title$ Therefore, our main code must search for the name, $Title$, not title: // Set the title Field if (!(metadatalist == null)) { for (int i = 0; i < metadatalist.size(); i++) { if (metadatalist.get(i).getfieldname().contentequals(" $Title$ ")) { metadatalist.get(i).setvalue(title); logger.warn("new Field value: " + metadatalist.get(i).getvalue() + "\n"); 19

20 Results Here you can see that the filename remains unchanged, but the title is now set to the first line of the documents: 20

21 Here is the logging output: 21

22 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 This is warning. <OFMsg> "1" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 This is error. <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $Directory$ <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value: E:\rawdata\title_test <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $FileName$ <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value: doc1.doc <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 22

23 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $Extension$ <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value:.doc <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $ModifiedDate$ <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value: <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $FileSize$ 23

24 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value: 494 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field name: $Title$ <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 Field value: doc1.doc <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 This is the title for doc1. <OFMsg> "2" " "4020"25" title_test" com.ibm.es.crawler.plugin.logging.pluginlogger.java"241"3 New Field value: This is the title for doc1. 24

25 Appendix A: Plugin Source Code /** * */ package crawlerplugins.typea; import com.ibm.es.crawler.plugin.abstractcrawlerplugin; import com.ibm.es.crawler.plugin.crawleddata; import com.ibm.es.crawler.plugin.crawlerpluginexception; import java.io.bufferedreader; import java.io.bufferedwriter; import java.io.ioexception; import java.io.inputstream; import java.io.inputstreamreader; import java.io.outputstream; import java.io.outputstreamwriter; import java.util.arraylist; import java.util.list; import java.util.regex.matcher; import java.util.regex.pattern; import com.ibm.es.crawler.plugin.content; import com.ibm.es.crawler.plugin.fieldmetadata; import com.ibm.es.crawler.plugin.logging.pluginlogger; import java.util.arraylist; import java.util.list; import java.util.regex.matcher; import java.util.regex.pattern; /** p8admin * */ public class TitleCrawlerPlugin extends AbstractCrawlerPlugin { /** Logger */ private static final PluginLogger logger; static { PluginLogger.init(PluginLogger.LOGTYPE_OSS, PluginLogger.LOGLEVEL_INFO); logger = PluginLogger.getInstance(); 25

26 /** End Logger **/ private static final String TITLE_PATTERN = "^\\A(.*)"; private Pattern titlepattern = Pattern.compile(TITLE_PATTERN); String title = ""; private Matcher matcher; public TitleCrawlerPlugin() { super(); public boolean iscontentused() { return true; public void init() throws CrawlerPluginException { /* * [Tips] If your crawler plug-in module requires something to do for * initialization, add the code here. [Example] Get JDBC connection for * your local system. connection = * DriverManager.getConnection("jdbc::db2::xxxx); */ public boolean ismetadataused() { /* * [Tips] If your crawler plug-in module updates both metadata and * security tokens, returns true. If your crawler plug-in module updates * security tokens only, returns false. */ return true; public void term() throws CrawlerPluginException { /* * [Tips] If your crawler plug-in module requires something to do for * termination, add the code here. [Example] Close JDBC connection for * your local system. connection.close(); */ return; 26

27 /* * (non-javadoc) * * com.ibm.es.crawler.plugin.abstractcrawlerplugin#updatedocument(com.ibm.es *.crawler.plugin.crawleddata) public CrawledData updatedocument(crawleddata crawleddata) throws CrawlerPluginException { /* Testing Logging Statements */ logger.info("this is info."); logger.warn("this is warning."); logger.error("this is error."); /* End Testing Logging Statements */ // print metadata List<FieldMetadata> metadatalist = crawleddata.getmetadatalist(); if (!(metadatalist == null)) { for (int i = 0; i < metadatalist.size(); i++) { logger.warn("field name: " + metadatalist.get(i).getfieldname() + "\n"); logger.warn("field value: " + metadatalist.get(i).getvalue() + "\n"); logger.warn("\n"); else logger.error("metadatalist is empty"); String contentstring = ""; List<Content> orgcontentlist = crawleddata.getoriginalcontents(); /* * The result list by getoriginalcontents method can not be modified. * Create a new list containing all contents to be indexed. */ List<Content> contentlist = new ArrayList<Content>(); if (orgcontentlist.isempty()) logger.error("orgcontentlist is empty"); 27

28 for (int i = 0; i < orgcontentlist.size(); i++) { contentlist.add(orgcontentlist.get(i)); // Now read the content into a String InputStream in = null; BufferedReader br = null; StringBuilder sb = new StringBuilder(); try { int value = 0; for (Content content : contentlist) { if (contentlist.isempty()) logger.error("contentlist is empty"); // Get InputStream object to ready the content. in = content.getinputstream(); br = new BufferedReader(new InputStreamReader(in)); while ((value = br.read())!= -1) { char c = (char) value; sb.append(c); contentstring = sb.tostring(); in.close(); catch (IOException ioe) { throw new CrawlerPluginException(ioe); title = gettitle(contentstring); logger.warn(title); // Set the title Field if (!(metadatalist == null)) { for (int i = 0; i < metadatalist.size(); i++) { if (metadatalist.get(i).getfieldname().contentequals(" $Title$ ")) { metadatalist.get(i).setvalue(title); logger.warn("new Field value: " + metadatalist.get(i).getvalue() + "\n"); 28

29 return crawleddata; private String gettitle(string doc) { List<String> result = new ArrayList<String>(); matcher = titlepattern.matcher(doc); while (matcher.find()) { result.add(matcher.group(1)); return result.get(0); Appendix B: Source code for Java Regex testing package test; import java.util.arraylist; import java.util.list; import java.util.regex.matcher; import java.util.regex.pattern; public class RegexTest { private Matcher matcher; //Alternative private static final String TITLE_PATTERN = "^\\A(.*)"; private Pattern titlepattern = Pattern.compile(TITLE_PATTERN); 29

30 //private static final String TITLE_PATTERN = "/^(.*)$/m"; //private static final String TITLE_PATTERN = "\\A.*"; public static void main(string[] args) { String data = "Testing... \n" + "Address 1: 88 app 2/8\n" + "superman taman, puchong\n" + "36100, Malaysia\n" + "Address 2: abc" + "testing end"; String doc = "First Line is Always the Title \n"+ "After that, there are many lines of text which \n"+ "represent the body of the article"; RegexTest obj = new RegexTest(); //List<String> list = obj.getaddress(data); String title = obj.gettitle(doc); System.out.println("Test Data : "); System.out.println(doc + "\n"); System.out.println("Title : " + title); private String gettitle(string doc){ List<String> result = new ArrayList<String>(); matcher = titlepattern.matcher(doc); while (matcher.find()) { result.add(matcher.group(1)); return result.get(0); 30

Developing, Compiling, and Running a Non-Web Data Source Crawler Plug-in with IBM Content Analytics with Enterprise Search 3.0

Developing, Compiling, and Running a Non-Web Data Source Crawler Plug-in with IBM Content Analytics with Enterprise Search 3.0 Vijai Gandikota April 2013 Contents Introduction... 1 Understanding the code