Full Text Search Throwdown. Bill Karwin, Percona Inc.

Transcription

1 Full Text Search Throwdown Bill Karwin, Percona Inc.

2 In a full text search, the search engine examines all of the words in every stored document as it tries to match search words supplied by the user.

3 StackOverflow Test Data Data dump, exported December million Posts = 8.18 GB

4 StackOverflow ER diagram searchable text

5 The Baseline: Naive Search Predicates

6 Some people, when confronted with a problem, think I know, I ll use regular expressions. Now they have two problems. Jamie Zawinsky

7 Accuracy issue Irrelevant or false matching words one, money, prone, etc.: SELECT * FROM Posts WHERE Body LIKE '%one%' Regular expressions in MySQL support escapes for word boundaries: SELECT * FROM Posts WHERE Body RLIKE '[[:<:]]one[[:>:]]'

8 Performance issue LIKE with wildcards:!select * FROM Posts WHERE title LIKE '%performance%'! OR body LIKE '%performance%'! OR tags LIKE '%performance%'; 49 sec POSIX regular expressions: 7 min 57 sec!select * FROM Posts WHERE title RLIKE '[[:<:]]performance[[:>:]]'! OR body RLIKE '[[:<:]]performance[[:>:]]'! OR tags RLIKE '[[:<:]]performance[[:>:]]';

9 Why so slow? CREATE TABLE TelephoneBook (! FullName VARCHAR(50)); CREATE INDEX name_idx ON TelephoneBook! (FullName); INSERT INTO TelephoneBook VALUES! ('Riddle, Thomas'),! ('Thomas, Dean');

10 Why so slow? Search for all with last name Thomas SELECT * FROM telephone_book WHERE full_name LIKE 'Thomas%' uses index Search for all with first name Thomas SELECT * FROM telephone_book WHERE full_name LIKE '%Thomas' can t use index

11 Because: B-Tree indexes can t search for substrings

12 FULLTEXT in MyISAM FULLTEXT in InnoDB Apache Solr Sphinx Search Trigraphs

13 FULLTEXT in MyISAM

14 FULLTEXT Index with MyISAM Special index type for MyISAM Integrated with SQL queries Indexes always in sync with data Balances features vs. speed vs. space

15 Insert Data into Index (MyISAM) mysql> INSERT INTO Posts SELECT * FROM PostsSource; time: 33 min, 34 sec

16 Build Index on Data (MyISAM) mysql> CREATE FULLTEXT INDEX PostText! ON Posts(title, body, tags); time: 31 min, 18 sec

17 Querying SELECT * FROM Posts WHERE MATCH(column(s)) AGAINST('query pattern'); must include all columns of your index, in the order you defined

18 Natural Language Mode (MyISAM) Searches concepts with free text queries:!select * FROM Posts WHERE MATCH(title, body, tags ) AGAINST('mysql performance' IN NATURAL LANGUAGE MODE) LIMIT 100; time with index: 200 milliseconds

19 Query Profile: Natural Language Mode (MyISAM) Status Duration starting checking permissions Opening tables init System lock optimizing statistics preparing FULLTEXT initialization executing Sending data end query end closing tables freeing items cleaning up

20 Boolean Mode (MyISAM) Searches words using mini-language:!select * FROM Posts WHERE MATCH(title, body, tags) AGAINST('+mysql +performance' IN BOOLEAN MODE) LIMIT 100; time with index: 16 milliseconds

21 Query Profile: Boolean Mode (MyISAM) Status Duration starting checking permissions Opening tables init System lock optimizing statistics preparing FULLTEXT initialization executing Sending data end query end closing tables freeing items cleaning up

22 FULLTEXT in InnoDB

23 FULLTEXT Index with InnoDB Under development in MySQL 5.6 I m testing m1 Usage very similar to FULLTEXT in MyISAM Integrated with SQL queries Indexes always* in sync with data Read the blogs for more details:

24 Insert Data into Index (InnoDB) mysql> INSERT INTO Posts SELECT * FROM PostsSource; time: 55 min 46 sec

25 Build Index on Data (InnoDB) Still under development; you might see problems: mysql> CREATE FULLTEXT INDEX PostText! ON Posts(title, body, tags); ERROR 2013 (HY000): Lost connection to MySQL server during query

26 Build Index on Data (InnoDB) Solution: make sure you define a primary key column `FTS_DOC_ID` explicitly: mysql> ALTER TABLE Posts CHANGE COLUMN PostId `FTS_DOC_ID` BIGINT UNSIGNED; mysql> CREATE FULLTEXT INDEX PostText! ON Posts(title, body, tags); time: 25 min 27 sec

27 Natural Language Mode (InnoDB) Searches concepts with free text queries:!select * FROM Posts WHERE MATCH(title, body, tags) AGAINST('mysql performance' IN NATURAL LANGUAGE MODE) LIMIT 100; time with index: 740 milliseconds

28 Query Profile: Natural Language Mode (InnoDB) Status Duration starting checking permissions Opening tables init System lock optimizing statistics preparing FULLTEXT initialization executing Sending data end query end closing tables freeing items cleaning up

29 Boolean Mode (InnoDB) Searches words using mini-language:!select * FROM Posts WHERE MATCH(title, body, tags) AGAINST('+mysql +performance' IN BOOLEAN MODE) LIMIT 100; time with index: 350 milliseconds

30 Query Profile: Boolean Mode (InnoDB) Status Duration starting checking permissions Opening tables init System lock optimizing statistics preparing FULLTEXT initialization executing Sending data end query end closing tables freeing items cleaning up

31 Apache Solr

32 Apache Solr Formerly known as Lucene, started 2001 Apache License Java implementation Web service architecture Many sophisticated search features

33 DataImportHandler conf/solrconfig.xml:... <requesthandler name="/dataimport" class="org.apache.solr.handler.dataimport.dataimporthandler"> <lst name="defaults"> <str name="config">data-config.xml</str> </lst> </requesthandler>...

34 DataImportHandler conf/data-config.xml: <dataconfig> <datasource type="jdbcdatasource" driver="com.mysql.jdbc.driver" url="jdbc:mysql://localhost/testpattern?useunicode=true" batchsize="-1" user="xxxx" password="xxxx"/> <document> <entity name="id" query="select PostId, ParentId, Title, Body, Tags FROM Posts"> </entity> </document> </dataconfig> extremely important to avoid buffering the whole query result!

35 DataImportHandler conf/schema.xml:... <fields> <field name="postid" type="string" indexed="true" stored="true" required="true" /> <field name="parentid" type="string" indexed="true" stored="true" required="false" /> <field name="title" type="text_general" indexed="false" stored="false" required="false" /> <field name="body" type="text_general" indexed="false" stored="false" required="false" / > <field name="tags" type="text_general" indexed="false" stored="false" required="false" / > <field name="text" type="text_general" indexed="true" stored="false" multivalued="true"/ > <fields> <uniquekey>postid</uniquekey> <defaultsearchfield>text</defaultsearchfield> <copyfield source="title" dest="text"/> <copyfield source="body" dest="text"/> <copyfield source="tags" dest="text"/>...

36 Insert Data into Index (Solr) command=full-import time: 14 min 28 sec

37 Searching Solr +performance time: 79ms Query results are cached (like MySQL Query Cache), so they return much faster on subsequent execution

38 Sphinx Search

39 Sphinx Search Started in 2001 GPLv2 license C++ implementation SphinxSE storage engine for MySQL Supports MySQL protocol, SQL-like queries Many sophisticated search features

40 sphinx.conf source src1 {! type = mysql! sql_host = localhost! sql_user = xxxx! sql_pass = xxxx! sql_db = testpattern! sql_query = SELECT PostId, ParentId, Title,!! Body, Tags FROM Posts! sql_query_info = SELECT * FROM Posts \!! WHERE PostId=$id }

41 sphinx.conf!index test1 {! source = src1! path = C:\Sphinx\data }

42 Insert Data into Index (Sphinx)! C:\Sphinx> indexer.exe -c sphinx.conf.in --verbose test1 Sphinx release (r3309) using config file 'sphinx.conf'... indexing index 'test1'... collected docs, MB sorted Mhits, 100.0% done total docs, bytes total sec, bytes/sec, docs/sec total 11 reads, sec, kb/call avg, msec/call avg total 542 writes, sec, kb/call avg, 5.7 msec/ call avg Execution time: s time: 8 min 20 sec

43 Querying index $ mysql --port 9306 Server version: release (r3309) mysql> SELECT * FROM test1 WHERE MATCH('mysql performance'); id weight rows in set (0.02 sec)

44 Querying index mysql> SHOW META; Variable_name Value total 1000 total_found 7672 time keyword[0] mysql docs[0] hits[0] keyword[1] performance docs[1] hits[1] time: 13ms

45 Trigraphs

46 Trigraphs Overview Not very fast, but still better than LIKE / RLIKE Generic, portable SQL solution No dependency on version, storage engine, thirdparty technology

47 Three-Letter Sequences! CREATE TABLE AtoZ (! c!! CHAR(1),! PRIMARY KEY (c));! INSERT INTO AtoZ (c) VALUES ('a'), ('b'), ('c'),...! CREATE TABLE Trigraphs (! Tri! CHAR(3),! PRIMARY KEY (Tri));! INSERT INTO Trigraphs (Tri) SELECT CONCAT(t1.c, t2.c, t3.c) FROM AtoZ t1 JOIN AtoZ t2 JOIN AtoZ t3;

48 Insert Data Into Index my $sth = $dbh1->prepare("select * FROM Posts") or die $dbh1->errstr; $sth->execute() or die $dbh1->errstr; $dbh2->begin_work; my $i = 0; while (my $row = $sth->fetchrow_hashref ) { my $text = lc(join(' ', ($row->{title}, $row->{body}, $row->{tags}))); my %tri; map($tri{$_}=1, ( $text =~ m/[[:alpha:]]{3}/g )); next unless %tri; my $tuple_list = join(",", map("('$_',$row->{postid})", keys %tri)); my $sql = "INSERT IGNORE INTO PostsTrigraph (tri, PostId) VALUES $tuple_list"; $dbh2->do($sql) or die "SQL = $sql, ".$dbh2->errstr; if (++$i % 1000 == 0) { print "."; $dbh2->commit; time: 116 min 50 sec $dbh2->begin_work; } space: 16.2GiB } print ".\n"; $dbh2->commit; rows: 519 million

49 Indexed Lookups!SELECT p.* time: 46 sec FROM Posts p JOIN PostsTrigraph t1 ON! t1.postid = p.postid AND t1.tri = 'mys'!

50 Search Among Fewer Matches!SELECT p.* time: 19 sec FROM Posts p JOIN PostsTrigraph t1 ON! t1.postid = p.postid AND t1.tri = 'mys' JOIN PostsTrigraph t2 ON! t2.postid = p.postid AND t2.tri = 'per'

51 Search Among Fewer Matches!SELECT p.* time: 22 sec FROM Posts p JOIN PostsTrigraph t1 ON! t1.postid = p.postid AND t1.tri = 'mys' JOIN PostsTrigraph t2 ON! t2.postid = p.postid AND t2.tri = 'per' JOIN PostsTrigraph t3 ON! t3.postid = p.postid AND t3.tri = 'for'

52 Search Among Fewer Matches!SELECT p.* time: 13.6 sec FROM Posts p JOIN PostsTrigraph t1 ON! t1.postid = p.postid AND t1.tri = 'mys' JOIN PostsTrigraph t2 ON! t2.postid = p.postid AND t2.tri = 'per' JOIN PostsTrigraph t3 ON! t3.postid = p.postid AND t3.tri = 'for' JOIN PostsTrigraph t4 ON! t4.postid = p.postid AND t4.tri = 'man'

53 Narrow Down Further!SELECT p.* time: 13.8 sec FROM Posts p JOIN PostsTrigraph t1 ON! t1.postid = p.postid AND t1.tri = 'mys' JOIN PostsTrigraph t2 ON! t2.postid = p.postid AND t2.tri = 'per' JOIN PostsTrigraph t3 ON! t3.postid = p.postid AND t3.tri = 'for' JOIN PostsTrigraph t4 ON! t4.postid = p.postid AND t4.tri = 'man' WHERE CONCAT(p.title,p.body,p.tags) LIKE '%mysql%'! AND CONCAT(p.title,p.body,p.tags) LIKE '%performance%';

54 And the winner is... Jarrett Campbell

55 Time to Insert Data into Index LIKE expression FULLTEXT MyISAM FULLTEXT InnoDB Apache Solr Sphinx Search Trigraphs n/a 33 min, 34 sec 55 min, 46 sec 14 min, 28 sec 8 min, 20 sec 116 min, 50 sec

56 Insert Data into Index (sec) LIKE MyISAM InnoDB Solr Sphinx Trigraph

57 Time to Build Index on Data LIKE expression FULLTEXT MyISAM FULLTEXT InnoDB Apache Solr Sphinx Search Trigraphs n/a 31 min, 18 sec 25 min, 27 sec n/a n/a n/a

58 Build Index on Data (sec) n/a n/a n/a LIKE MyISAM InnoDB Solr Sphinx Trigraph

59 Index Storage LIKE expression FULLTEXT MyISAM n/a 2382 MiB FULLTEXT InnoDB? MiB Apache Solr Sphinx Search Trigraphs 2766 MiB 3355 MiB MiB

60 Index Storage (MiB) LIKE MyISAM InnoDB Solr Sphinx Trigraph

61 Query Speed LIKE expression FULLTEXT MyISAM FULLTEXT InnoDB Apache Solr Sphinx Search Trigraphs 49,000ms - 399,000ms ms ms 79ms 13ms 13800ms

62 Query Speed (ms) LIKE MyISAM InnoDB Solr Sphinx Trigraph

63 1000 Query Speed (ms) LIKE MyISAM InnoDB Solr Sphinx Trigraph

64 Bottom Line build insert storage query solution LIKE expression k-399k ms SQL FULLTEXT MyISAM 31:18 33: MiB ms MySQL FULLTEXT InnoDB 25:27 55:46? ms MySQL 5.6 Apache Solr n/a 14: MiB 79ms Java Sphinx Search n/a 8: MiB 13ms C++ Trigraphs n/a 116: GiB 13,800ms SQL

65 Final Thoughts Third-party search engines are complex to keep in sync with data, and adding another type of server adds more operations work for you. Built-in FULLTEXT indexes are therefore useful even if they are not absolutely the fastest. Different search implementations may return different results, so you should evaluate what works best for your project. Any indexed search solution is orders of magnitude better than LIKE!

66 New York, October 1-2, 2012 London, December 3-4, 2012 Santa Clara, April 22-25, 2013 /live

67 Expert instructors In-person training Custom onsite training Live virtual training

68

69 Copyright 2012 Bill Karwin Released under a Creative Commons 3.0 License: You are free to share - to copy, distribute and transmit this work, under the following conditions: Attribution. You must attribute this work to Bill Karwin. Noncommercial. You may not use this work for commercial purposes. No Derivative Works. You may not alter, transform, or build upon this work.