An Architecture for Self-Organizing Continuous Delivery Pipelines

Size: px

Start display at page:

Download "An Architecture for Self-Organizing Continuous Delivery Pipelines"

Alexia Copeland
5 years ago
Views:

1 An Architecture for Self-Organizing Continuous Delivery Pipelines Master Thesis Final Talk Jan Simon Döring

2 Towards Continuous Delivery 2.0 The next generation Software Delivery Systems 2

3 Continuous Delivery Continuous Delivery is a set of principles and practices to reduce the cost, time and risk of delivering incremental changes to users Humble, Farley 3

4 Current generation Software Delivery Systems VCS build test release defines Delivery Model node {... stage('build') { sh 'mvn clean package' } stage('deploy dev'){ sshagent(['remotecredentials']) { sh "scp target/*.jar root@a:/opt/jenkins-demo.jar" sh "ssh root@${server} nohup java Dserver.port= ${port} -jar /opt/jenkins-demo.jar &" } } } Jenkins 2 Pipeline as code example Delivery System 4

Modeling Problems The build scripts are complicated or complex Problems, causes and solutions when adopting continuous delivery A systematic literature review Laukkanen, Itkonen, Lassenius -

5 Modeling Problems The build scripts are complicated or complex Problems, causes and solutions when adopting continuous delivery A systematic literature review Laukkanen, Itkonen, Lassenius - Information and Software Technology (Feb. 2017) A delivery model change causes a failure of the next build with a probability of 40%. Mining Changes of Build Processes in the Context of Continuous Integration Benedikt Holmes Bachelor Thesis RWTH Aachen (2017) 5

Lassenius Problems, causes and solutions when adopting continuous

6 Build Monolith The build system cannot be modified flexibly New technologies Build Hero New process steps Laukkanen, Itkonen, Lassenius Problems, causes and solutions when adopting continuous delivery A systematic literature review Information and Software Technology (Feb. 2017) 6

7 Challenges Project Evolution - Technical Evolution (e.g. new tools) - Process Evolution (e.g. new process activities) - Organizational Evolution (e.g. new policy) Be flexible! Be maintainable! Modeling Usability - Minimize required knowledge (technical & process) - Support the user Model Simplification! Provide Assistance via Tooling! 7

8 Domain Driven Design Tackling Complexity in the Heart of Software 8

9 Delivery Process Abstractions code compile binary Transformation binary unit test report Assessment binary report decision Promoted binary Quality Gate policy Inspired by J. Hermanns Evolution of Build Artifacts in Continuous Delivery Bachelor Thesis, RWTH Aachen (2015) 9

10 e.g. At most 3 files with LoC > 20 policy transformation assessment Quality gate configuration Git artifact checkout artifact Quality gate artifact compile transform artifact artifact transform artifact Qgate artifact Static analysis e.g. # of.java files with LoC > 20 report #FilesResult = 2 artifact assess assess report report 10

11 Domain Driven Design Core Domain conforms to Delivery Process describes instance of External Delivery Model DeliveryProcess Execution Specification 1..* has Stage Views Command Configuration Model Result Model 1..* defines dependent on Execution Precondition instance of 1..* triggers Execution produces configured by Configuration instance of Result instance of has * Logical Dependency Functional Dependency ExecutionResult Artifact on Assessment Transformation Quality Gate Execution Result: Technical Result (EnvInfo, Logs,..) vs Result: Functional Result Internal Delivery Modell Execution Reasoning Support External Delivery Modell 11

12 Central Design Decisions Conceptual level Artifact activity Artifact Isolation & Autonomy Standardized Interface Service Principles Architecture level Artifact + Config + Selector Command API activity activity - Microservice 12

13 Central Design Decisions Supporting Model Context Artifact Storage Context Generic Model Service External Delivery Model Storage Service Artifact Context Context Service Supporting Self-Organizing Context via Ubiquitous Language of Self-Organizing Context Model Events Process Planner Internal Delivery Model Registry Specification Orchestration Context View Service (CQRS) Process Events Orchestrator Delivery Process Execution Generic Core Execution Events Event Async Communication via Event 13

14 Self-Organization Analyze, generate, adapt and optimize the delivery model by the system itself 14

15 Specification Be explicit about input / output schema Allows for reasoning Specification Config Model Result Model - Required Artifact Types - Parameter Schema 15

16 Self-Organizing Idea: Tackle modeling challenge by automatic planning External Delivery Model Planner Internal Delivery Model Reduce required knowledge Compliance & Best practices Optimization Automatic evolution Model Separation allows for custom (tailored) description languages

17 Pipeline Description Language stages: - name: build transformations: - checkout - compile - jar Stage Definitions transformations: Transformation Definitions - name: checkout service: git-service activity: checkout configuration: repositoryuri: - name: compile service: maven-service activity: compile dependson: - alias: repo ref: p://this/transformations/checkout configuration: workspace: "@repo Assessment Definitions assessments: - name: unittests service: maven-service activity: test dependson: - alias: compile ref: p://this/transformations/compile - alias: repo ref: p://this/transformations/checkout configuration: pom: "@repo" classes: "@compile" qualitygates: Quality Gate Definitions - strategy: auto policies: - name: unittestpolicy interpretation: threshold ref: p://this/assessments/unittests actualvalue: passedrate setpoint: 1 17

18 Planning Operations Add / remove / modify stage Add / remove / modify activity Add dependencies - name: compile service: maven-service activity: compile dependson: - alias: repo ref: p://this/transformations/checkout/workspace configuration: workspace: "@repo" - name: compile service: maven-service activity: compile - name: assemble service: maven-service activity: assemble Semantically equivalent - name: assemble service: maven-service activity: assemble configuration : workspace: "@repo" classes: "@compile" dependson: - alias: repo ref: p://this/transformations/checkout/workspace - alias: compile ref: p://this/transformations/compile/classes 18

19 Visualization Commands Delivery System Management Tool Queries API Gateway Access View Service Internal Model Representation External Model Representation Model Service Orchestrator Infrastructure Event Store Model Importer Event Bus Delivery Process Management Specification Registry Process Planner Task Queue Events Monitoring Service Discovery <<Transformation>> Service... <<Assessment>> Service Artifact Storage Notification External Provider Static Analysis Tool... Deployment Tool... Notification Tool Functional Service (motivated by DDD) Technical Service 19

20 Architecture Dynamics Queries View Service Event Store Delivery System Management Tool Model Events Event Bus Delivery Process Events Execution Events Events Model Service Process Planner Orchestrator External Delivery Model Delivery Process Delivery Process Execution Commands Import phase of Architecture Control Flow Planning phase of Architecture Control Flow Execution phase of Architecture Control Flow Control Flow Data Flow 20

21 Core Delivery Framework Import Planning Execution Model Service Planner + Specification Registry Orchestrator + Services Import: M2M from external to internal delivery model Planning: Adapt & Optimize model - Model-based planning: Consider specified activities - Project-based planning: additionally use project data, other external sources.. Execution: Perform modeled activities

22 Service Blueprint Service Orchestrator TaskQueue <<PollingConsumer>> OrchestratorClient Controller Worker Execution API Receiver Discover & Health API instantiates Concrete Configuration Specification Registry Registry Client Local Registry Specification Model 22

23 Service SDK For Java Spring Services Eases development of new activity services Derives specification Registers specification Provides execution API Orchestrator = "checkout", type = CommandType.TRANSFORMATION) public class GitCheckoutCmd implements Command<GitCheckoutResult> { } private final GitCheckoutProperties properties; private final GitExecutor git; public GitCheckoutCmd(GitCheckoutProps properties, GitClient git) { this.properties = properties; this.git = git; public GitCheckoutResult execute(executionmonitor monitor) { //implement me } 23

24 Visualization Commands Delivery System Management Tool Queries API Gateway Access View Service Model Service Process Planner Orchestrator Infrastructure Event Store Delivery Process Management Specification Registry Quality Gate Service Git Service Maven Service Jmeter Service Docker Service Artifact Storage External Provider

27 Case Study Industry project: IBE Web Service (API Gateway) 65k LoC, ~18 team members Tech: Java EE, Maven Dependencies: Database, Keycloak Tests: Junit, Jmeter Objective: Is the core domain applicable? Can new technologies be integrated easily? What is the impact of Self-Organization?

28 Case Study Delivery Process Transformation Level Git Checkout Project Sources Maven Compile Java Classes QS Gate 1 Java Classes Maven Package Gateway Ear Docker Build Gateway Ear Docker Container Names Docker Provision Quality Gate 2 Gateway Ear Teardown Docker Deploy to Artifactory Assessment Level Project Sources Maven Test Surefire Reports Cobertura Reports Ip + Ports Jmeter Tests Jmeter Reports Conceptual New activity Previously performed manually by web service team 28

29 Case Study Simplify model by automatic mapping Manual Model Planned Model Semantically equivalent LoC = 124 Complexity = 18 LoC = 66 Complexity = 1 Complexity = 1 + Number of dependencies (similar to McCabe) 29

30 Case Study Project Planner Maven Planner Analyzes maven sub-modules Optimization goal: Fail-fast Execution Speedup: -9,14% Fails up to 3.31 times faster Test Failure Seeding

Case Study - Conclusion Core domain applicable? - 100% coverage of process with concepts - No adoptions required Can new technologies be integrated easily?

31 Case Study - Conclusion Core domain applicable? - 100% coverage of process with concepts - No adoptions required Can new technologies be integrated easily? - We added a Jmeter Service & Docker Service - No adoptions to existing services - Service SDK Impacts of Self-Organization - Reduced LoC by ~47%, Less complex - Up to 3,31 times faster feedback - Stakeholder Feedback: Reduced transparency ( magic ) 31

32 Threads to validity Complex System (structural) Implementation Overhead ( first implement, then use ) Too much overhead for small projects 32

33 Future Work Modeling Tools E.g. Graphical or Smart Tooling (auto-completion, recommendation) Improve Validation Smells and anti-pattern detection Policy-based Extended Planning Dynamic (Re-) Planning More planners & multi-target planning Learning from history We are here Delivery Ecosystem 33

34 Summary 34

35 What are next generation Software Delivery Systems? Easier to use! Really flexible! Smart! Robust! Maintainable! 35

CONTINUOUS DELIVERY IN THE ORACLE CLOUD

CONTINUOUS DELIVERY IN THE ORACLE CLOUD Lykle Thijssen Bruno Neves Alves June 7, 2018 NLOUG Tech Experience Amersfoort eproseed Confidential ABOUT US Lykle Thijssen Principal Architect and Scrum Master