Let Your (APEX) Application Find Everything "Fuzzy Search" with Oracle Text

Transcription

1 Let Your (APEX) Application Find Everything "Fuzzy Search" with Oracle Text Carsten Czarski Consulting Member of technical Staff Oracle Application Express Oracle Deutschland B.V. & Co KG

2 Search challenges in Business Applications Need to search based on different criteria Need fuzzy / error-tolerant searching Need performance also on huge amounts of data Need an easy interface 2

3 3

4 4

5 Oracle TEXT 7

6 Oracle TEXT: What is that? Fulltext index engine within the Oracle Database Introduced somewhen in the time of Oracle7 Part of all database editions no separate license required Provides linguistic and fuzzy search features Searches documents and relational data

7 Let's start simple Search documents 9

8

9 Search Documents 1. Have documents in your table XMLTYPE, VARCHAR2, CLOB, BLOB Filtering available for binary data (Office, PDF) 2. Create a text index on top of your table create index ix_text_index on docs_tab (doc) indextype is CTXSYS.CONTEXT 3. Create the APEX report select * from docs_tab where contains(doc, :P1_SEARCH) > 0

10 Full text search with SQL: Functionality SQL CONTAINS-Function Relevance-Ranking based on token frequency: SCORE Visualization Markup, Highlighting "Keyword-in-Context" select score(1), doc from docs_tab where CONTAINS( doc, 'Software AND Oracle', 1 )>0

11 Oracle TEXT Queries Combine Query Terms... where contains(text, 'Hund AND Katze') >0 Use Wild Cards... where contains(text, 'Hu%d AND Kat_e') >0 Powerful Name Searching... where contains(text, 'NDATA(name, Hunt)') >0 Fuzzy matching... where contains(text, '?Hunt') >0 Multi Lingual Stemming... where contains(text, '$läuft') >0

12 Oracle TEXT Queries NEAR Operator... where contains(text, 'near(hund, Katze), 4') >0 Search within Sections or Paragraphs (XML!)... where contains(text, 'Hund WITHIN TITEL') >0 Thesaurus support... where contains(text, 'SYN(Hund,[thes]' >0)

13 Oracle TEXT Indexing process Filter Sectioner Lexer AUTO_FILTER Plain Text HTML XML News/ Basic Whitespace Themen Engine Decompounding alternate spelling

14 Oracle TEXT: Index Data Storage Tabelle DR$... $I DR$... $K DR$... $N DR$... $R DR$... $P DR$... $S DR$... $E DR$... $G DR$... $D Inhalt Token table: Contains all Tokens and information about their occurance in documents Mapping table ROWID DOCID "Negativ list": Contains all deleted DOCIDs Mapping table DOCID ROWID Substring index to support "%{TOKEN}" queries SDATA table: contains structured data for queries with <, > or BETWEEN Oracle12c: XQuery fulltext contains XML namespace information Oracle12c: Near Realtime Index Oracle12c: Save Copy feature

15 Configure the Oracle TEXT Index: Preferences select distinct pre_class, pre_owner from ctx_preferences; PRE_CLASS PRE_OWNER DATASTORE ODD_EDITIONS STORAGE US STORAGE SH DATASTORE CTXSYS DATASTORE US...

16 Lexer Preferences: An example for the German language Typical Preference settings for Netherlands begin CTX_DDL.create_preference('MY_LEXER','BASIC_LEXER'); CTX_DDL.set_attribute('MY_LEXER', 'MIXED_CASE', CTX_DDL.set_attribute('MY_LEXER', 'BASE_LETTER', CTX_DDL.set_attribute('MY_LEXER', 'BASE_LETTER_TYPE', CTX_DDL.set_attribute('MY_LEXER', 'INDEX_STEMS', end; 'NO'); 'YES'); 'GENERIC'); 'YES');

17 OK, we can search documents, now let's move on Search table data 19

18 Oracle TEXT for relational data...?

19 Oracle TEXT Index on plain relational data...? Oracle TEXT is here to index documents Oracle TEXT allows various datastores (where do documents come from?) DIRECT_DATASTORE (indexed column) is the default FILE_DATASTORE, URL_DATASTORE for external sources MULTICOLUMN_DATASTORE grabs data from other table columns USER_DATASTORE for a "PL/SQL" source We can turn relational data to a "virtual" document with PL/SQL

20 How does a USER_DATASTORE work...? A PL/SQL procedure collects the data to be indexed Table(s) PL/SQL Code Oracle TEXT Index

21 Create the PL/SQL procedure for the USER_DATASTORE

22 Register the PL/SQL procedure for Oracle TEXT

23 Declare Search Sections

24 Declare Index Parameters and finally create Text Index

25 PL/SQL based USER_DATASTORE: Some hints and tips You have multiple interfaces Return VARCHAR2 FAST! Return a CLOB Return a BLOB Oracle TEXT will "filter" like the output (PDF or Office documents) Procedure will be called during index creation - for each table row Be Fast - write the most efficient PL/SQL and SQL you can You know, you will always be below 32K...? Use VARCHAR2, not CLOB! Using CLOB? Initialize one temp LOB, hold in a package and re-use Write fast, not nice code, i.e. generate XML by String concatenation.

26 28

27