Scaling Solr in the Cloud

Transcription

1 Scaling Solr in the Cloud

2 Agenda About the Speakers About Nutshell Nutshell and Solr Solr Resource Usage Scaling Methods Questions and Answers

3 About the Speakers BitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com

4 About Nutshell Web and Mobile CRM application Heavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jquery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios

5 Nutshell and Solr Heavy use of Solr for searching, table views, and de-duplication Used for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users Used for de-duplication: Accounts, Contacts

6 Reads vs Writes Jul 20, :00 AM Jul 20, :00 AM Jul 20, :00 AM Jul 20, :00 PM Jul 20, :00 PM Jul 20, :00 PM Jul 20, :00 PM Jul 21, :00 AM Writes Reads Time Average Read Query Time: 2.11ms

7 Demo

8 Computer Resources CPU Disk I/O RAM Disk Storage

9 CPU Disk I/O RAM Disk Storage

10 Resources Heavy CPU Disk I/O RAM Disk Storage Light

11 Resources Heavy CPU Disk I/O RAM Disk Storage Light

12 Resources Heavy CPU Disk I/O RAM Disk Storage Light

13 Resources Heavy CPU Disk I/O RAM Disk Storage Light

14 Resources Heavy CPU Disk I/O RAM Disk Storage Light

15 Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome

16 Dark Ages Age of Enlightenment Modern Era Today Separate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch

17 Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer

18 50 Dark Ages Age of Enlightenment Modern Era Today 25 0 Auto provisioning Separate Jetty app for each customer

19 Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API

20 Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library

21 Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM

22 Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM

23 Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM

24 Dark Ages Age of Enlightenment Modern Era Today Shared Schema Across Cores Decreases initialization time for each core Decreases memory usage

25 Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display

26 Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn t build indexes

27 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave

28 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers

29 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers

30 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers

31 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager

32 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager

33 Dark Ages Age of Enlightenment Modern Era Today Web MySQL (mastermaster) Solr Slave Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager

34 Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing

35 Dark Ages Age of Enlightenment Modern Era Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing Today

36 Dark Ages Age of Enlightenment Modern Era Today Intelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login

37 Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id

38 Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers

39 Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers

40

41

42 User 8000 User 8000 User 8000 User 8000 User 8000 User User

43

44

45 123456

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61 Scaling Solr in the Cloud auto provisioning 50 Users separate Jetty app for each customer separate Solr core per customer 300 Users fall back to MySQL for table data migrated from Sun JVM to IcedTea JVM shared schema across cores 600 Users solr index field selection splitting reader / writer roles intelligent core spin down / up parititioning of customer to separate Solr servers 7000 Users