DB2 Information Integrator OmniFind Edition

Transcription

1 IBM Software Group DB2 Information Integrator OmniFind Edition Samson Tai Product Introduction Center, AP South 2004 IBM Corporation

2 IBM is Focusing on Search and Text Analysis 200+ Researchers >200 patents in categorization & clustering, taxonomy generation & maintenance, intention analysis, summarization, machine translation, text search Institute for Search and Text Analysis (ISTA) Co-Leaders in SWG and Research Set strategy in the area of search, text mining and analysis Software Group Development Laboratories Lotus, WebSphere Portal Almaden and Haifa Research Labs Data Management Bethesda, Boeblingen, Silicon Valley, Yamato, 2

3 DB2 Information Integrator OmniFind Edition Delivers the best results with sub-second response Sophisticated relevancy algorithms for corporate content Scales for large collections or enterprises 500K documents and above 1000s of concurrent users Fits easily into enterprise applications Java APIs Document level security Eases administration and maintenance Proof of concept in hours not days Analysis features all under-the-covers Information Integrator OmniFind Edition 3

4 Proven Quality, Scale and Robustness on IBM Intranet w3 A test bed to bring the right product to market Employee Profile Search (skills ): XML records 30 K update pushes daily with 2 to 3 hour turnaround Production since March 2004 Quality preferred 2:1 over prior technology Scale -- 80K queries/day with sub-second response over 7M pages Indexes Processes 7 M unique pages 10,000 websites 20 K per document 80 K queries/day 7 K queries/hour peak Stressed to 10x higher Robustness % availability since Sept. 24x7 operation 4

5 OmniFind Client SIAPI Search Runtime Application Server Crawlers for Web for NNTP for Websites Newsgroups Information Integrator Main Index Delta Index Global Analysis Indexer RDS Raw Data Store for CM for Notes for Exchange Content Manager Notes / Domino MS Exchange Store Parser Tokenizer for Windows File for Unix File Windows File System Unix File System Admin Console Controller Data Listener API Custom Crawler 5

6 Asynchronous model Search Runtime Indexer Parser Data source Crawlers/Parser/Indexer/Search runtime run asynchronously in parallel: Crawlers run continuously (Web and News) or on a schedule basis Parser runs continuously Indexer runs on a schedule basis (Full/Delta) Search runtime runs continuously This model allows great scalability and availability: as long 1 search runtime is up, the system delivers search results, even if crawlers/parser/indexer are down OmniFind has a local copy (store) of the content to be searched, which can create inconsistencies This implies a lag time before changes in a data source (content updated/removed, security credentials modified) are reflected in the search results. Add/remove URL feature coming for the web crawler 6

7 Crawlers Retrieves/fetches content from each specific source, acting like a regular client, going through the standard APIs/protocols, for each data source: Web browser like Internet Explorer to reach websites (via HTTP and HTTPS) Lotus Notes for Domino servers (via Notes RPC or IIOP) Application issuing SELECT statement for relational databases (via JDBC) Etc 2 types of crawlers: 24/7 (web and newsgroups) Manual or scheduled start (Domino, relational databases, MS Exchange, DB2 CM, filesystems, WebSphere PDM) Crawl space: What content do I want to search? Where should OmniFind start and stop? Store compressed content in DB2 with metadata I/O intensive task (network and disk) 7

8 Web Crawler IBM Software Group DB2 information management software High Performance and Manageability Designed to crawl thousands of Intranet sites Follows all HTML links to discover webpages pages retrieved per second Automatic crawl interval adjustment based on webpages change rates Site based crawl speed control politeness settings Web space tuning by URL pattern or IP address Intranet Crawl thread Crawl thread Crawl thread Crawl thread Crawl Buckets Persistent Site DB Web Pages Enterprise Search Collections Web Crawler 8

9 DB2 Information Integrator Crawler Supports DB2 or federated (nickname) tables DB2 and Information Integrator federated sources (DB2 zos, Informix, Oracle) Each record identified by URI Mapping columns to searchable field Mapping CLOB and BLOB columns to searchable content DB2 DB Table Server: db2.ibm.com:6001 Database: ClaimDB, Table: Accident ID (Pri. Key) Description Customer Date Data (CLOB) Car Accident John Smith 07/01/20003 Crashed at.. URI Metadata db2://db2.ibm.com:60001/claimdb/accident/id/ <Metadata Language="en"> <CommonMetadata DataSource="DB2" DatasourceName= Claim DB"... </DatasourceSpecificMetadata> </Metadata> Enterprise Search Collections Content Mr. Smith had an accident where his car crashed head to head with.. 9

10 DB2 Content Manager Crawler Supports Crawling DB2 CM Each record uniquely identified by PID Mapping attributes to searchable field Document content is filtered by the parser for text indexing DB2 CM Server: cm.ibm.com:80 Item Type: Automobile Attribute Category Title Doc Type Creator Date CarInfo Car Accident PDF John Smith 07/01/20003 Content PDF Document URI Metadata Content cm://cm.ibm.com:80/automobile/<pid> <Metadata Language="en"> <CommonMetadata DataSource= CM" DatasourceName= Automobilie"... </DatasourceSpecificMetadata> </Metadata> PDF Document Enterprise Search Collections 10

11 Parser Runs continuously Consumes content stored in DB2 by the crawlers Transforms the various document formats (HTML, XML, PDF, Word, Powerpoint, etc ) into an intermediate common text form ( including some metadata like author, font ) Possibility of mapping specific fields (HTML and XML) Understand the structure of the text (URL, metadata, paragraph, sentence, words) Processed text is kept in a store 11

12 Document Processing! $! "# $ $ % % 12

13 Steps in the document processing Identify document format Identify codepage and convert text to unicode Separate text from markup data =>Parsing Identify document language Segment the stream of text to lexical units =>Tokenization, Lemmatization 13

14 Document Format and Codepage Look for document name extension Codepage included in the document Analyze meta information from the crawler Note: The more you specify about a document collection in its crawler, the better the search quality 14

15 Separate Text from Markup With the specific parser for the document format extract the text information Document: <html> <title>java Documentation</title> <body> This is the body text.</boy></html> Text data: Java Documentation This is the body text. 15

16 Language Identification Monolingual detection assumption: each document has one language Automatic language detection suggests a language with a specific confidence Otherwise default language is used Document language can be used a query time to search only documents of a specific language Note: Document language is mandatory for follow up text analysis 16

17 Tokenization IBM Software Group DB2 information management software Based on natural language analysis technologies Language identification IBM LanguageWare multi-lingual tokenization and stemming Support 20+ Simple, Complex, and Ideographic languages White space and n-gram tokenizer for doc in other languages dictionnary Lexical Analysis Tokens Multi-word expressions and Character Normalization New-York city Washington-based résumé New York Washington resume Morphology - roots, decomposition, stemming, lemmas running Lagerregal run Lager regal 17

18 Segmentation/Tokenization Break down text in distinct lexical units I don t plan to go to the meetings I do not plan to go to the meeting Dictionary based segmentation for supported languages Non-dictionary based segmentation for all other 18

19 Dictionary based Word detection Define what can be found Sentence detection Needed for phrase and adjacency search Lemmatization Associate a word to its canonical form houses => house organized => organize 19

20 Non Dictionary based White space segmentation Use white spaces between words as delimiter N-gram segmentation Treat overlapping sequences of n characters as one word Bigram: ABCD => AB BC CD Sufficient for many retrieval tasks Used for languages without white space delimiters 20

21 Additional Normalization after linguistic processing finding when searching for Case normalization : USA => usa Umlaut expansion : schön => schoen Accent removal : é => e Other diacritics removal : ç => c Ligature expansion : Æ => ae 21

22 Categorization Rule based categorizer Leverage Pre-defined taxonomy in WebSphere Portal Search Engine 2700 business categories Tooling to add and modify categories Tokenized Documents Rule Based Categorizer URL Rules: Contents Rules: +thinkpad T30 T40 WebSphere Portal Pre-defined categorizer (Eureka!) & Taxonomy Manager - Computers - -- Computer Hardware -- Computers Systems -- pseries Servers -- Thinkpads -- Computer Storages -- Computer Software Document with assigned categories Enterprise Search Collection 22

23 High Performance and Scalable Index Build The index lists every tokens, and for each token, the list of documents it appears in and its positions in the document Stems are indexed as well as the original word Documents are sorted by static ranking Process millions of documents per hour Delta index for fast turnaround of changes Highly compressed The index is approximately half the size of the store Building the Index is an I/O bound process. 23

24 Global analysis IBM Software Group DB2 information management software Main index with global ranking order (static or document-based ranking): Web based Ranking: Link and Anchor text Analysis Date based ranking: for newsgroup and document currency Process Anchor text (in HTML links, the text that describes the page pointed to) <A HREF= >IBM</A> Link analysis: how many sites link to the each document Associate scopes to each documents based on rules defined by users (ex: America, Asia-Pacific, HR, etc ) Removes duplicates/obsolete documents 24

25 Search runtime Timeout to traverse the index, ordered by static rank, the best results found are being returned if not completed (90% of requests take less than 2sec) Predefined links (use stats for Frequent searches) Search and Index API talks to a servlet running on WebSphere Caching of previous search results 25

26 Highly relevant results generation How to change my intranet password linux +portal Queries WAS DB2 download Query processing Remove Stop Words Stemming Select Ranking Model (Informational vs. Navigational Queries) Fetch Cache and Index in memory Dynamic ranking: Generate and rank intermediate results Factors: Lexical Affinities Term location, Font Frequency Construct final results with dynamic document summary from the store Anchor text 26

27 End-user Search Workflow 1 The User types in a search query Search Application (Portal ) Search API Search Runtime Delta Index Main Index Store 2 The User wants to display a document listed in the search results Web server News server Custom Viewer Application Server Domino 27

28 dynamic summary predefined link lexical affinities Information Integrator stemming site collapse stop-word elimination 28

29 Search query syntax + - ( ) operators +jaguar car I had a dream Fielded/metadata search author:david Parametric search #SALARY::>15000< Language filtering $language::en Document type filtering $doctype::pdf Scope, category filtering Source filtering $source::nntp Web: site: (only returns pages on url:hr (only returns pages where the URL contains hr) link: (returns pages pointing to 29

30 Security IBM Software Group DB2 information management software Role based administration security Collection Security Document Level Security User Login Enterprise Search Query: Security Token: Search Portlet Salary Increase HR Group, Mgr Application Authentication and Collection Authorization e.g. Employee Profile Collection Administration ES Admins WebSphere WPS Security/LDAP Portal Search and security filtering Crawler Crawler Collection Admins HR Group, Mgr Security Tokens per doc (public, assigned, crawled from field) Data Source: e.g. Employee Profile Domino DB 30

31 w3.ibm.com Index and search over 8 million unique pages Over 7 Million unique URLs Over 7,000 websites About 20K per document 30-40K searches per day Response times well under 0.5 Sec 7 days to crawl the entire intranet Main index built every day Delta index built every 2 hours Crawled data is approx 100GB Index is approx. 50GB 31

32 Capability Planning - Application Considerations Data Sources Local and Wide Area Network Bandwidth Load on mission critical data source servers Number of Documents and Collections Collection can hold index for multiple data sources A collection is a searchable unit for applications Average size and number of documents in collections Refresh rate for content updates Search Applications Performance Number of concurrent users Average search results per query Average response time 32

33 Software requirements Red Hat Linux Advanced Server 3.0 Update 1 IBM Java Runtime Environment (JRE) SR2 DB2 UDB v8.1 + fixpak 5 WebSphere Application Server v5.1 WebSphere Application Server Network Deployment v5.1 Optional: Lotus Domino to access Notes databases (NSF files) via Notes RPC (not required if using IIOP) DB2 Content Manager Toolkit to access DB2 CM 33

34 Configurations 1-node configuration Mid-range departmental collections (few million documents) 4-nodes configuration Enterprise Scale (tens of millions of documents) Load balancing and Failover for 24x7 operations Search runtime 1 Crawlers Parser Indexer Controller Search runtime Dispatcher Search runtime 2 Gigabit Ethernet Parser Indexer Controller Crawlers 34

35 Demo 35