Content analysis with Apache Tika

Size: px

Start display at page:

Download "Content analysis with Apache Tika"

Rosaline Hamilton
5 years ago
Views:

1 Content analysis with Apache Tika or -

2 Main challenge Lucene index 2

3 Other challenges Licenses Dependencies Efforts breaking up Custom solution limits 3

4 What is Tika? Another Indian Lucene project? No. 4

5 What is Tika? It is a Toolkit Doing What? Where? How? Detection Metadata Various documents Using existing parser libs Extraction Structured text content 5

6 Current coverage 77 Mime types (+ 36 aliases) 179 glob patterns 18 magic patterns 66 Supported Types 15 Parsers 6

7 A brief history of Tika Sponsored by the Apache Lucene PMC Graduating 0.2 Release March 07 Incubator December 07 1 Release 7

Tika organization 7 committers Issue Tracking https://issues.apache.

8 Tika organization 7 committers Issue Tracking se/tika Mailing Lists tika-dev@incubator.apache.org Changing after graduation 8

9 Getting Tika and contributing Download Build Have fun 0.1-incubatingsrc (tar.gz) mvn install svn.apache.org mvn eclipse:eclipse 9

10 Tika Design The Parser interface Document input stream XHTML SAX events Document metadata Parser implementations 10

11 The Parser interface void parse(inputstream stream, ContentHandler handler, Metadata metadata) throws IOException, SAXException, TikaException; Document + Metadata parse() XHTML + Metadata 11

12 Tika Design The Parser interface Document input stream XHTML SAX events Document metadata Parser implementations 12

13 Document input stream InputStream Not Readable IOException InputStream Not Parsable TikaException InputStream OK PARSED! 13

14 Tika Design The Parser interface Document input stream XHTML SAX events Document metadata Parser implementations 14

15 XHTML SAX events <html xmlns=" <head> <title>...</title> </head> <body>... </body> </html> parse() SAX events ContentHandler 15

16 Why XHTML? Reflect the structured text content of the document Not recreating the low level details For low level details use low level parser libs 16

17 ContentHandler (CH) and Decorators (CHD) XHTMLContentHandler BodyContentHandler TextContentHandler CHD that Produces XHTML events CHD that only passes the body to a CH CHD that only passes characters to a CH 17

18 Tika Design The Parser interface Document input stream XHTML SAX events Document metadata Parser implementations 18

19 Document metadata Metadata.RESOURCE_NAME_KEY The name of the file or resource that contains the document Metadata.CONTENT_TYPE According to the content type the document was parsed to Metadata.TITLE If the document format contains an explicit title field Metadata.AUTHOR If the document format contains an explicit author field 19

20 more metadata: HPSF Apache POI: HPSF TITLE - Title SUBJECT - Subject AUTHOR - Author KEYWORDS - Keywords COMMENTS - Comments TEMPLATE - Template LAST_SAVED - Last Saved By REVISION_NUMBER - Revision Number LAST_PRINTED - Last Printed LAST_SAVED - Last Saved Time/Date LAST_SAVED - Last Saved Time/Date PAGE_COUNT- Number of Pages WORD_COUNT- Number of Words CHARACTER_COUNT - Number of Characters APPLICATION_NAME - Name of Creating Application 20

21 Tika Design The Parser interface Document input stream XHTML SAX events Document metadata Parser implementations 21

22 Parser implementations 3 Party Libraries PDFBox Apache POI Tika Parsers PDFParser OfficeParser 22

23 The AutoDetectParser Encapsulates all Tika functionalities Can handle any type of document Type Detection Parsing 23

24 Type Detection MimeType type = types.getmimetype( ); Magic markers in prefix type.matchesxml(data) magic.eval(data) Resource Name MimeType type = patterns.matches(name) Metadata Metadata.CONTENT_TYPE Default Type application/octet-stream 24

25 tika-mimetypes.xml An example: Gzip <mime-type type="application/x-gzip"> <magic priority="40"> <match value="\037\213" type="string offset="0" /> </magic> <glob pattern="*.tgz" /> <glob pattern="*.gz" /> <glob pattern="*-gz" /> </mime-type> 25

26 Supported formats Excel Word Power Point PDF Plain Text RTF Outlook Gzip Bzip2 XML HTML Images Java class & jar MP3 OpenDocument Tar ZIP others 26

27 A really simple example InputStream input = MyTest.class.getResourceAsStream("testPPT.ppt"); Metadata metadata = new Metadata(); ContentHandler handler = new BodyContentHandler(); new OfficeParser().parse(input, handler, metadata); String contenttype = metadata.get(metadata.content_type); String title= metadata.get(metadata.title); String content = handler.tostring(); 27

28 Future Goals OCR Speech Recognition XMP Integration Parser Configurability 28

29 Who uses Tika? Apache Nutch Apache Jackrabbit Apache Droids Apache UIMA 29

This tutorial is designed for all Java enthusiasts who want to learn document type detection and content extraction using Apache Tika.

This tutorial is designed for all Java enthusiasts who want to learn document type detection and content extraction using Apache Tika. About the Tutorial This tutorial provides a basic understanding of Apache Tika library, the file formats it supports, as well as content and metadata extraction using Apache Tika. Audience This tutorial