Resilience Validation

Transcription

1 Resilience Validation Ramkumar Natarajan & Manager - NFT Cognizant Technology Solutions

2 Abstract With the greater complexities in application and infrastructure landscapes, the risk of failure is ever increasing. Building a robust, highly available & fault tolerant software system becomes a challenging task and an absolute necessity in today's world. Even a minute of downtime can cost millions of dollars. What if? analysis Feature Groups Domino effects Major Incidents Myths & Facts Key tenants 2

3 Need of Resilience Validation Is our application resilient to handle unforeseen / unplanned events? Being prepared to maneuver disruptive events has become an essential part of business success in the modern era. $500,000 $150 M $81 M $150 M Outage of 5 Minutes Overall Internet traffic dropped by 40% Outage of 5 Hours Cancelled about 1000 flights Outage of 12 Hours Cancelled about 2100 flights Outage of 2.5 Hours Amazon S3 is used by around 148,213 websites Major Outages and Domino effects 3

4 Business Impacts Impact to customers and strategic partners. Lost Opportunities & Revenue losses. Service Level Agreement Penalties Impact to Employee Productivity. Impact on reputation & quality. Outage costs are in increasing trend as the user base and complexities expands exponentially. 4

5 Myths & Facts MYTHS FACTS 1 Disaster Recovery / Passive system will protect against all kind of failures without any impacts. Impact analysis needs be done to identify any challenges or issues during failover. 2 Active nodes in cluster can handle failures of other nodes / data centers effectively. Failover testing needs to be performed to validate system s ability to handle usage and the burden that additional use may cause on the system. 3 Migrating to cloud make applications resilient Moving to cloud doesn't guarantee resiliency. 4 Off-the-shelf software products (like Oracle, Couchbase) are already tested and certified Testing needs to be done in the client environment to find the limitations & right configurations 5 Resiliency testing is laborious process & requires availability of multidisciplinary team members. Just like other testing, few of the steps like failure simulation can be automated with less or no manual intervention. 5

6 Resilience Tenants Handle hardware & component failures gracefully Manage the load variations without any issues. How an application should be? Rainy day scenarios, introducing CPU, Memory Spikes, node & process failures. Manoeuvre network & downstream latencies efficiently. Network & Downstream system latencies Fault Tolerance Latency Tolerance Load Tolerance Data Tolerance Black Friday, Product Release peak load events. Handle erroneous situations & data that is outside the range Errors, Exceptions & bulk data from downstream systems 6

7 Objectives of Resiliency Validation Identify & fix Single Point of Failure. Know the unknown Engineering, Functional & Performance Flaws. Enable resilient features & provide self healing abilities. Reduce MTTD (Mean Time to Detect) & MTTR (Mean Time To Recovery). Improve high availability (Fall back) & Scalability by embracing next-gen HA technologies. Effort required to restore system health. Resilience Feature Groups High Availability Scalability Switchover Data Recovery Dependency DR Validation CI 7

8 Resiliency Implementation Resilience implementation & validation for different applications and streams were broadly categorized across three planes. Incident Analysis Application, Platform & Operational Readiness FMEA Analysis Architecture & System Review Near Neighbor Analysis Application Resilience vulnerability and prioritization matrix Product Backlog Data Analysis 8

9 Validation - War gaming War gaming is a process of simulating failures / production events and observe the system behavior. War Gaming will help augment operational efficiency and stability through discovery, practice and teamwork. To ensure our customers experience the absolute least amount of impact during unanticipated production events. Validate Fault, load, latency & data tolerance levels of application 9

10 Key Industry Trends Hystrix is a fault tolerance library developed by Netflix to stop cascading failures and enable resilience in the distributed systems Key Features Fault Throttling Circuit Breaker. Fall back 10

11 Cognizant Chaos Framework v v v 11

12 Client: Largest cable company and home Internet service provider in the United States. Services: Cable television, Internet, telephone service and home security to both residential and commercial customers. outage Self recovery Success Stories YTD Incident 20% than 2015 YTD Incident duration (MTTR) 42% than 2015 Total outage by application platform reduced to 5% from 30% More than 10+ outages were avoided due to downstream failures with Fault Throttling Improved the MTTR of downstream system between 5 to 10 minutes from 1-2 hour DEFECT COUNTS Monkeys Executed Defects Found AVAILABILITY LOAD LATENCY DATA 12

13 Benefits Ensure continuous response to end-customers. Resiliency testing will avoid / reduce domino effect. Meets Quality of Service (QoS) beyond BCP and DR. Cost of downtime will be reduced. Shorter up time following a service disruption Customer satisfaction will be improved. 13

14 Takeaways Expect the unexpected. Key tenets of resilient application. Resiliency testing not only addresses downtime - Load, Data, Latency Testing and engineering approaches. Failure simulations can be automated using Chaos Framework. Resilience testing is good to must have. 14

15 References & Appendix Cognizant internal reference links & materials

16 Author Biography Ramkumar Natarajan has over 14+ years of experience with specialization in Performance testing and engineering space. Ram is part of Cognizant QE&A NFT CoE & leads the Resiliency testing initiatives. Ram holds Master s degree in Computer Applications from Madras University. Earlier he has worked with Wipro Technologies & Ramco Systems. 16

17 Question & Answers 17

18 Thank You!!! 18