Thinking Beyond Search with Solr Understanding How Solr Can Help Your Business Scale. Magento Expert Consulting Group Webinar July 31, 2013

Transcription

1 Thinking Beyond Search with Solr Understanding How Solr Can Help Your Business Scale Magento Expert Consulting Group Webinar July 31, 2013

2 The presenters Magento Expert Consulting Group Udi Shamay Head, Expert Consulting Group Steve Kukla Business Solution Architect, Expert Consulting Group Kirill Morozov Application Architect, Expert Consulting Group July 31,

3 Today s agenda What is Apache Solr? Business Use Cases for Scale Supporting Initial Catalog Growth Supporting Growing Traffic Supporting Substantial Catalog Growth Supporting A Real-Time Catalog Key Points to Remember Q&A July 31,

4 What is Apache Solr? July 31,

5 What is Apache Solr? General Solr Overview Solr Separate application installed on its own server, or on an existing server in the environment depending on business needs. Solr uses schema configuration files which can be found in Magentto/lib/Apache Magento communicates with Solr via HTTP/XML Searching options configured via the Magento admin panel July 31,

6 What is Apache Solr? Solr the Search Platform Better text-based searching provides a better customer experience More relevant fuzzy searching* Faceted searches Search corrections Out of the box type-ahead* Response caching for better performance *Requires customization to leverage at 100% July 31,

7 What is Apache Solr? What Makes Solr Powerful Solr is more than a search engine because Most data customers see is handled by Solr instead of MySQL July 31,

8 What is Apache Solr? What Makes Solr Powerful Solr is more than a search engine because Most data customers see is handled by product_id MySQL (EAV) attribute_id product_id attribute_value attribute_id product_id attribute_name Solr instead of MySQL Solr uses a simpler data structure Solr (No EAV) product_id attribute_name attribute_value July 31,

9 What is Apache Solr? What Makes Solr Powerful Solr is more than a search engine because Most data customers see is handled by Solr instead of MySQL Solr uses a simpler data structure Solr supports replication which allows it to truly scale for growth Magento Solr Solr Solr Solr Solr July 31,

10 Supporting Initial Catalog Growth July 31,

11 Business Use Case Supporting Initial Catalog Growth Business Background Growing catalog from 10K to 100K SKUs From 1 to 2 stores Problems Increased indexing time Out-dated information on the front-end From 1 to 2+ web nodes / 1 database node Using native Solr Search July 31,

12 Supporting Initial Catalog Growth Problem Increasing Index Footprint 50,000 SKUs 100,000 SKUs Slow Indexing Year 2 2 websites 2 store views 10,000 SKUs 3.5 min* 17.5 min* 35 Min* Control 1 website 1 store view 1.75 Min* 10 Min* 17.5 min* *Expected indexing time July 31,

13 Supporting Initial Catalog Growth Solution Custom Data Import Handler Concept Connects to the database using JDBC Extra data transformations must be written in Java/JavaScript. Uses a prepared xml configuration July 31,

14 Supporting Initial Catalog Growth Data Import Handler Results Results 10 times faster indexing Supports delta-indexing Things to keep in mind Solr knows about its data source May require extra development efforts Extra data transformations must be written in Java/JavaScript July 31,

15 Supporting Growing Traffic July 31,

16 Business Use Case Supporting Growing Traffic Business Background Growing catalog 1,000,000 SKUs Growing traffic: up to 100 requests / second Problem Solr can t handle increasing user concurrency 3 stores 3+ web nodes/ 1 database node Using Data Import Handler July 31,

17 Supporting Growing Traffic Increasing Index Footprint OK 100,000 SKUs 500,000 SKUs 1,000,000 SKUs Indexing delta data handles updates Year 3 3 websites 3 store views 4.75 min* min* 47.5 Min* < 1000 updates/sec Control 2 website 2 store view 3.5 Min* 17.5 Min* 35 min* *Expected indexing time July 31,

18 Supporting Growing Traffic Problem Increased Response Time 100,000 SKUs 30 RPS 500,000 SKUs 60 RPS 1,000,000 SKUs 100 RPS Solr CPU is maxed out Year 3 3 websites 3 store views 80 msec* 100 msec* 120 msec* Control 2 website 2 store view 75 msec* 95 msec* 105 msec* *Expected average response time July 31,

19 Supporting Growing Traffic Solution Solr Replication Concept Separate reading requests Replicate index across multiple nodes Read from multiple servers July 31,

20 Supporting Initial Catalog Growth Solr Replication Results Results Allows Solr to handle read traffic Introduces fail-over Things to keep in mind Requires middle-ware or Magento customization Possible heavy data duplication Extra changes in infrastructure July 31,

21 Supporting Substantial Catalog Growth July 31,

22 Business Use Case Supporting Substantial Catalog Growth Business Background Growing catalog 5,000,000 SKUs 4 stores 4+ web nodes / 1 database node Problems Delta-indexing delays Slow response time Using Data Import Handler Using Solr replication July 31,

23 Supporting Substantial Catalog Growth Problem Increasing Index Footprint Year 4 4 websites 4 store views 1,000,000 SKUs 63.5 min* 2,500,000 SKUs min* 5,000,000 SKUs Min* > 1000 updates/sec Delta indexing delays Control 3 website 3 store view 47.5 Min* Min* min* *Expected indexing time July 31,

24 Supporting Substantial Catalog Growth Problem Increased Response Time 1,000,000 SKUs 100 RPS 2,500,000 SKUs 200 RPS 5,000,000 SKUs 400 RPS Slow response time Year 4 4 websites 4 store views 150 msec* 270 msec* 400 msec* Control 3 website 3 store view 120 msec* 230 msec* 300 msec* *Expected average response time July 31,

25 Supporting Substantial Catalog Growth Solution Index Sharding Concept Distributed search Distributed + Replication (SolrCloud) July 31,

26 Supporting Growing Traffic Index Sharding Results Results Distributed search for faster response time 50 times faster indexing with 5 shards MySQL A B C I D H F G E Things to keep in mind Magento Custom solution Requires Magento customization or middleware introduction Extra changes in infrastructure Solr Shards D E F G H I July 31,

27 Supporting A Real-Time Catalog July 31,

28 Business Use Case Supporting A Real-Time Catalog Business Background Growing catalog 10,000,000 SKUs Business Requirement Always up-to-date index 5 stores 5+ web nodes / 1 database node Data Import Handler SolrCloud and distributed search July 31,

29 Supporting A Real-Time Catalog Solution Listen To The MySQL Bin Log Concept Connect via MySql replication protocol Listen to data-related events MySQL Binlog Replication MySql Slave July 31,

30 Supporting A Real-Time Catalog Solution Listen To The MySQL Bin Log Concept Connect via MySql replication protocol Listen to data-related events MySQL Binlog Replication Listener Extract information from events Manipulate with document in Lucene index Log Parser Solr July 31,

31 Supporting A Real-Time Catalog Listening To The MySQL Bin Log Results A B C Results Replication-like connection Indexes are always up-to-date Things to keep in mind Relatively complex implementation Magento MySQL I D H F G E Bin log D E F Solr Shards G H I July 31,

32 Key Points to Remember July 31,

33 Key Points to Remember Solr helps businesses scale Solr s search capabilities provide a better site experience than MySQL LIKE or Full-text Solr is more than a search platform it is a key for scalability and growth Solr s data import handler keeps Solr performing well as your catalog grows Solr replication helps accommodate growing traffic Solr shards help keep indexing execution time and search response times low for very large catalogs Listening to the MySQL bin log can help facilitate a continuously updating catalog July 31,

34 References Scaling Solr Solr Wiki Type-Ahead Data Import Handler(DIH) Replication Shard Distributed Search MySql Replication listening Change Data Capture Replication Listener (C) Open-Replicator (Java) July 31,

35 Q&A July 31,

36 The presenters Magento Expert Consulting Group Udi Shamay Head, Expert Consulting Group Steve Kukla Business Solution Architect, Expert Consulting Group Kirill Morozov Application Architect, Expert Consulting Group July 31,