Microsoft Indexing Service: A Technical Overview. Introduction

Transcription

1 Microsoft Indexing Service: A Technical Overview 1 Introduction What is Microsoft Indexing Service? What are the key interfaces for developers? How much architecture do you need to know? 2 1

2 Microsoft Indexing Service Feature overview Architecture: Indexing Architecture: Queries Programming: Using ActiveX Data Objects Programming: Advanced Techniques Installation and Management 3 Features Full text searches Multiple document formats searches Document properties searches date, author, subject, file size Advanced search operators boolean, numeric, NEAR Wild-card searches Web-based forms 4 2

3 Features (Cont.) Integrated with Internet Information Server and with file system Replaces FastFind old MS Office feature Indexing Service is notified when documents are added or changed Security integration Indexing as a background task Supports multiple languages 5 Questions? 6 3

4 Architecture: Indexing The indexing process Catalogs Scanning Filtering Word Breaking Normalization Index Types: Transitory and Persistent 7 How Indexing Works Group documents in scope into the catalog Scan documents for content and properties Write documents to catalog Filter document content Divide document content into words Documents are normalized 8 4

5 Index Server Catalogs Top-level unit: index and cached properties Default catalog named Web Each catalog has directories and properties folders Adding catalogs? 9 Scanning Documents Full vs. incremental scanning Full scan identifies all documents After addition of new directory As part of serious error recovery Incremental for modified documents Done on every startup Done after file change notification lost due to high update rate 10 5

6 Scanning Documents Scans can be forced from Index Server Manager Do after removing filter or word breaker Do after changing filter registry info 11 Filtering Identify different file formats Adobe provides a PDF filter Extract portions of text Recognize language shifts Mark them so correct word breaker and normalizer used Handle embedded objects 12 6

7 Word Breaking Some languages use whitespace and punctuation for word breaks, but other languages do not. Index data stored in Unicode Avoid code page and DBCS issues Seven supported languages on install English, French, German, Spanish, Italian, Dutch, Swedish 13 Normalization Remove noise words Manage capitalization and punctuation Generally clean up output from the word breaker 14 7

8 Types of Indexes Three types of indexes Transitory, RAM-based word list index Persistent shallow Master index 15 RAM-Based Word List Index Non-noise words stored in this list Small indexes MaxWordLists defaults to 20 Merge and write to shadow index when too many RAM word lists Recreated every time index server starts 16 8

9 Persistent Indexes Maximum of 255 Shadow index: merge of word lists and/or shadow index into single index Master index Master merge process Merge master index with all shadow indexes Shadows and original master index deleted when done. 17 Questions? 18 9

10 Architecture: Queries Query components Scope of query Query restrictions Query types Fuzzy Queries 19 Query Components Scope: Range of documents to search Restrictions: Content or properties to search for Viewable results 20 10

11 Scope of Query Directory path or paths on the server Correspondence with virtual directories Scope can consider directory subsets NNTP roots as well as HTTP roots can be searched 21 Query Restrictions Words, Phrases, and Complete Sentences Document Types (Such as *.DOC and *.TXT) Textual Properties (i.e, abstract or author field) Boolean Operators Ranked by Match Quality 22 11

12 Query Restrictions (Cont.) Wild-Card Operators (* and?) Numeric operators against constants (such as dates) Proximity of Words and Phrases 23 Query Types Standard query: searches for exact text entered Free text query: Isolates significant terms in the supplied text. Use the free text query checkbox on the sample forms. Free text isolates nouns and noun phrases, then removes connecting terms 24 12

13 Fuzzy Queries Queries that provide only part of the text sought in conjunction with wild-card operators Linguistic inflection and stemming: past, present, and future tenses. Distance affects result ranking 25 Questions? 26 13

14 Programming with ADO What is OLEDB? What is ADO? Catalogs as Relational Databases A Little More Architecture Indexing Service Object Types Programming Basics SQL For Indexing Service 27 Catalogs as Relational Databases You populate RDBMS tables Indexing Service populates catalogs One catalog per internet-based service One catalog per virtual web site Atomic part = index entry Keyword and document reference Two tables: Word Lists, Saved Indexes 28 14

15 A Little More Architecture Properties Important to this product Over 40 of them Most interesting are size, dates, location Characterizations Document abstracts You set the size (usually words) First renderable contents (HTML = <BODY>) 29 Indexing Service Object Types Query Utility AdminIndexServer CatAdm ScopeAdm 30 15

16 SQL For Indexing Service Example: SELECT DocTitle, Path, FileName, Characterization, Size FROM SCOPE() WHERE CONTAINS ( SearchText ) Use Friendly Name or Property name as the field name Can t use SELECT * except with views 31 FROM Clause Predefined views Alias for field groups FILEINFO, FILEINFO_ABSTRACT, EXTENDED_FILEINFO, WEBINFO, EXTENDED_WEBINFO, SSWEBINFO, SSEXETENDED_WEBINFO Custom views (Not covering details) Using a scope 32 16

17 Using a SCOPE FROM SCOPE() FROM SCOPE ( D:\users\joef ) FROM SCOPE ( SHALLOW TRAVERSAL OF D:\users\joef ) Multiple scopes Virtual directories vs. physical directories 33 The WHERE Clause ARRAY CONTAINS NEAR: Within 50 words FORMSOF FREETEXT: Phrases LIKE: Wildcards MATCHES: Regular expressions 34 17

18 Programming Basics User input mapping to SQL queries is a bit awkward Catalog as Data Source for connection SQL language extensions 35 ADO Query Example Set up Set objquery = CreateObject("IXSSO.Query") Set objresults = CreateObject("ADODB.Recordset") 36 18

19 Advanced Techniques File System Techniques Admin Helper Query Helper Make ADO a bit easier ISAPI Extensions IFilter Improved Query Languages for File System Techniques 3.0 indexes file system as well as web content No full rescan of NTFS5 file system Search dialog uses indexing service Faster than command prompt 38 19

20 AdminHelper AdminIndexHelper Index server management CatAdm Manages Catalog definition (What is in the catalog? ScopeAdm Manages a scope definition for a catalog object 39 ISAPI Extensions What is an ISAPI extension? Good for use from pure HTML A little outdated, but performance is excellent *.HTX = HTML extension *.IDA = Internet Data Administration *.IDQ = Internet Data Query 40 20

21 IFilter Used for custom filters (example: PDF files) DLL to expose IFilter interface GetChunk GetText or GetValue depends on CHUNK_STATE Register PersistentHandler for file name extension 41 Three Query Languages 2.0 Query Language Extensible SQL queries 42 21

22 Installation 43 Managing Indexing Service Indexing Management built into Computer Management Console See handouts 44 22

23 More Info Index Server Newsgroup Quite Active default.asp?icp=technet&slcid=us&newsgr oup=microsoft.public.inetserver.indexserver 45 References Professional Active Servers Pages 3.0, WROX Press Microsoft OLEDB Provider for Microsoft Indexing Service Indexing Service Documentation