Achieving Continuous Delivery - Micro Services. - Vikram Gadang

Size: px

Start display at page:

Download "Achieving Continuous Delivery - Micro Services. - Vikram Gadang"

Marcus Preston
6 years ago
Views:

1 Achieving Continuous Delivery - Micro Services - Vikram Gadang

2 Agenda Starting point Observations and lessons learned Architecting for CD Build pipeline strategy Testing strategy Deployment strategy

3 State of affairs Manual process (QAR) - Creating deployable artifacts Manual build process to QA & Prod Manual regression tests followed up by actual feature tests Fix regression bugs and/or feature bugs and repeat build Move build artifact to stage env and repeat manual testing Install the build to production during off-peak hours (2am - 4am)

4 Initial Attempt Use Jenkins as CI server to create a build pipeline Manually triggered build, test & deploy Write test cases that tested functionality end-to-end

5 Observations Long build & test time Most testing involve integration testing Tests not very reliable due to backend dependencies

7 Observations Monolithic architecture Large and complex codebase Hard to scale & deploy Long-term commitment to a technology

8 Observations Code coverage by tests.. 30% or less High cyclomatic complexity. 10 or more per function (too many if/then/else, switch, try/catch statements in code) Entangled code Complex dependency mesh

9 Micro Services Architecture A B C A B A B B D C C E A B C D D E E E Monolithic architecture puts all functionality in one process Microservices architecture breaks down functionality into many processes

10 Monolithic Architecture StoreFront UI Database Catalog Service Profile Service Browser Billing Service WAR Tomcat Remote Services

11 Micro Services Architecture Catalog Service StoreFront UI Database Profile Service Billing Service Remote Service

12 Micro Services - Pros Divide and Conquer. Break down complex problems into smaller parts that can be solved independently Small codebase. Small enough to fit in your head Enables frequent deployments Enables polyglot programming & persistence Enables independent scalability No long-term commitment required. Rewrite over maintain

13 Micro Services - Hard to do Dealing with complexity of the system as a whole Monitoring Deployment in a container or outside a container Avoiding duplication - Hint: move common code to libraries Service discovery Simple!= Easy

14 Basic Requirements for CD Make Dev, QA, Stage & Prod environments similar Zero-downtime deployments Testing and deployment are integral parts of delivery Build quality in Automate everything Build Unit tests Integration tests Code coverage and quality analysis Deployment Acceptance tests

15 Basic Requirements for CD Ability to scale out individual components Stateless systems Convention over configuration Commit early and often. Test early and fail fast All triggers are data-driven. (Measure everything)

16 Delivery Pipeline

17 Implementing the pipeline Started slow, added components over time Build Test DeployDev Test DeployQA Test DeployStage Test Deploy Prod Test Continuously improve the pipeline

18 Delivery Pipeline Build Build & & Unit Test Unit Test Integrati Integration Test on Test Publish Publish Artifact Artifact Deploy Dev Acceptan Acceptance TestTest ce Commit Deploy Deploy QA Dev Acceptan Acceptance TestTest ce Deploy Deploy Stage Dev QA Staging Deploy Deploy Prod Dev Quality Gate Acceptan Acceptance TestTest ce Acceptan Acceptance Test ce Test Production Code Analysis

19 Pipeline Implementation Jenkins - CI Server Gradle - Build tasks Nexus - Repository

20 Pipeline View (Jenkins)

21 Technology Stack Centos 6.4+ JVM 7+ Tomcat 7+ Spring Boot Mongo DB MySQL DB

22 Source Code Strategy Minimize pre-requisites to get started Only JDK 7+ Start with a walking skeleton Metrics and health stats Unit, integration and acceptance tests Security Logging Build and deployment scripts

23 Source Code Strategy Incremental development with frequent commits All commits go into master and are continuously integrated and tested leading to faster feedback Developers pick each others changes immediately Branch for release, branch for feature more often leads to merge/integration hell at end of sprints Feature flags to enable/disable features Write less code

24 More code = more tests public void dopost(httpservletrequest request, HttpServletResponse response) throws ServletException, IOException { StringBuffer jb = new StringBuffer(); String line = null; try { BufferedReader reader = request.getreader(); while ((line = reader.readline())!= null) jb.append(line); } catch (Exception e) { /*report an error*/ } try { JSONObject jsonobject = JSONObject.fromObject(jb.toString()); } catch (ParseException e) { // crash and burn throw new IOException("Error parsing JSON request string"); } // Work with the data... // Return response HistoryResponse historyresp = historyservice.writehistory(history); JSONObject respjson = converttojson(historyresp); response.setcontenttype("application/json"); response.getwriter().write(respjson.tostring()); }

25 Less code is method=post) public HistoryResponse History history ) { return historyservice.writehistory(history); }

26 Testing strategy Unit testing in 5 mins or less - fast feedback Integration testing in data access and remote access layer Acceptance tests after unit & integration tests Automated code review & quality analysis. Builds fails if any of above fail

27 Testing JUnit - Java Unit & Integration Tests Mockito - Mocking framework QUnit - Javascript unit tests Selenium - Acceptance & Web tests JMeter - Acceptance & Performance

28 Testing strategy

29 Real world scenario

30 Code Quality Strategy Enforce minimum code coverage by tests Overall coverage (line & branch) 90% or more New code coverage (line & branch) 90% or more Enforce low cyclomatic complexity (brain overload & testability) 10 per method. (if, while, do, for, &&,,?:, catch, switch, case, return, throw) Enforce code quality Error Handling Style problems Ignoring conventions or skipping unit tests

31 Code Quality Tools Jacoco - Java code coverage Saga - JS code coverage PMD - Code analysis FindBugs - Potential issue finder JDepend - Code complexity SonarQube - Ties it all together Saga JS Code Coverage

32 Measure Code Quality (Jenkins)

33 Measure Code Quality (SonarQube)

34 Measure Code Quality

35 Measure Code Coverage

36 Measure Code Quality

37 Deployment Strategy Automate deployment using Chef Systems built for zero downtime deployment Rollback is just another deployment by a single click Do not make any changes directly in production environment Emergency releases go through standard deployment pipeline

38 Deployment Tools Jenkins - CI Server Nexus - Repository Chef - Deployment

39 Monitoring New Relic - Monitoring platform Http/JMX monitoring - Health and metrics ARGUS/TOGA Java Melody - Collection of stats Elastic Search/Logstash/Kibana - Logs collection

40 Monitoring (New Relic)

41 API Documentation Automate API documentation with Swagger UI

42 API Documentation

43 Takeaways If it hurts, do it more often CD forces you to do the right thing You cannot do it without automating the build, test, deploy and release process Measure everything. Make no assumptions Things go better if dev and ops are friends

44 How long would it take your organization to deploy a change that involved just one single line of code? - Mary & Tom Poppendieck

45 How long would it take your organization to deploy a change that involved just one single line of code? Do you deploy changes at this pace on a repeatable, reliable basis? - Mary & Tom Poppendieck

46 References

47 Questions?

Continuous Testing at Scale

Continuous Testing at Scale TAPOST Conference October 12th 2016, Riga dmitry@buzdin.lv @buzdin Dmitry Buzdin Introduction to Continuous Testing Continuous Delivery Get changes to production in fast and