Accessibility & Reliability

Transcription

1 Accessibility & Reliability Combining Dovecot and Scality to Win Dan Shain & Jim Perry Cloud Office R&D

2 Who are we? Dan Shain Heads Engineering, Development, and Operations for Rackspace Applications Services Cloud Office Division 30+ years in the IT Realm with focus on large distributed systems Jim Perry Technical Manager for Software Development Responsible for all things Dovecot 25 years in the software development and IT management

3 IN 2014 WE NEEDED TO CHANGE OUR OPERATIONAL PARADIGM

4 Dilemma We Faced Open Source Dovecot with filesystem based storage Current path would increase base product cost by 200% 4500 Hardware Projections CAPEX Costs OPEX Costs Logistical Nightmare Status Quo 4 Footer

5 Dovecot Deployment that was 4,500+ Assets across 3 Datacenters and in 5 physical zones Scaling Limitations Compute and Storage could not scale independently Complicated support processes and scripts Data Protection N+2 Protection for Geo-redundant failures was not reliable and very costly Environment Management Hardware management was a logistical nightmare System Rebalancing was Managed Process Product Impacts On boarding limitations caused by write width Cost basis rising 5

6 Our Use Case Wide range of uses on a multi-tenant platform Multi-tenant environment > 800,000 Domains Wide variety of Domain level settings Usage patterns, although normalized, have large variances Normal Does not cause problems, Abby Normal does we have extremes that break the normal Mailbox Distribution By Message Count Average Messages Sent/Received by Mailbox/Day Accessed Everywhere Mailbox Count Sent Received 6 < 10 < 100 < 500 < 5,000 > 3,000 > 20,000 > 200,000 > 1,000,000 > 2,000,000 Messages per Mailbox 1 2 < 5 < 10 < 20 < 50 < 100 < 200 < 500 < 1,000 < 2,000 < 5,000 < 10,000 >10000

7 Demands Product Team/Customers Engineering/Operations o Happy Customers o Improved cost margins o Value Added Customer Support Contacts o No Down Time o Ensure Deliverability o Zero Data Loss o Faster Feature Delivery o Stay abreast of Vulnerabilities o Reduce Operational Costs to run and maintain the business o Stabilize Capital Expenditures o Ease Supply Train Impacts o Reduce/optimize Datacenter Space o Move to virtualized deployment models 7

8 DovecotPro Scality Solution Why did we chose this solution? Access Patterns Very High Write, High Read, Medium Delete Data after 10 days, stays and stays Customer Activity Profile Business Class Usage (Extremely High Daily Active Percentage) N+2 Architecture Need multi-zone and Geo-Graphic managed solution Fault Segmentation (Hierarchy of failure events) Scaling Must scale compute and storage separately Things WILL Fail Disk Node Rack Row Network Datacenter Maintenance & Upgrade Knowledge of Dovecot Reputation of Vendors 8

9 Deployment Chosen Data Center DoveCot Pro Backend Data Center DoveCot Pro Backend Cass andra Availability Zone I Availability Zone II Cass andra Dovecot/Scality Multi-Geo Location Deployment Load Balancer (Local haproxy) DoveCot Pro Backend sproxyd Load Balancer (Local haproxy) Scality Ring Stretched over 2 Data Centers Load Balancer (Local haproxy) Load Balancer (Local haproxy) DoveCot Pro Backend Cass andra Availability Zone I Availability Zone II Cass andra

10 Optimized Scality Infrastructure Leverage Scality to Optimize Hardware Apollo 4510 platform Tremendous scaling options Gigabyte Cost optimization Normalized and consistent environment Key We worked with Scality to create an optimized Storage Ring Focus on the nature of access patterns as well as sizing Do not overload the use case (We are not creating an S3 like environment) 10 OX Summit

11 Footprint Reduction Over 2,000 Assets Decommissioned to date Avoided ordering over 1,440 Assets and opening 1 new server room Vacating several rooms Reduced Maintenance to less than 35% of Peak (and dropping) Hardware Repairs Progress to Date Q Q Q Q Q Q Q Q Q Sep-16 Dec-16 Mar-17 Jun-17 Sep-17 Dec-17 Mar-18 Jun-18 Sep-18 Dec-18

12 Storage Stability Scality survived hardware outages; Disk and Chassis Handled Massive Cassandra Rebuild Efforts Rack Moves & Room Reconfigurations Only 1 Scality Ring associated customer impacting event Root Cause related to HP Firmware defect Only a handful of special tuning needed to run to: Re-engage disks post hardware oddness Node release of expired data Expansion Scheduled for this Summer

13 CHALLENGES FACED Interplay between DovecotPro, Cassandra & Scality Early adoption woes Initial errors in Dovecot Director Ring Configuration Fault Segmentation Too many customized plugins on our behalf Need to improve path to upgrade Allowance needed for Bake-In Optimist: Someone who figures that taking a step backward after taking a step forward is not a disaster, it's a cha-cha. Tuning of the Ring Overhead Reduction Expanded our internal knowledge base - Robert Brault 13

14 THANK YOU ONE FANATICAL PLACE SAN ANTONIO, TX US SALES: US SUPPORT: Copyright 2018 Rackspace Rackspace Fanatical Support and other Rackspace marks are either registered service marks or service marks of Rackspace US, Inc. in the United States and other countries. Features, benefits and pricing presented depend on system configuration and are subject to change without notice. Rackspace disclaims any representation, warranty or other legal commitment regarding its services except for those expressly stated in a Rackspace services agreement. All other trademarks, service marks, images, products and brands remain the sole property of their respective holders and do not imply endorsement or sponsorship. 14