Software Continuous Integration & Delivery INCREASING SOFTWARE DEVELOPMENT AGILITY TO SPEED TIME TO MARKET

Transcription

1 DAITAN WHITE PAPER Software Continuous Integration & Delivery INCREASING SOFTWARE DEVELOPMENT AGILITY TO SPEED TIME TO MARKET White Paper Contents Making software development more Agile Moving to a more Agile software development process A roadmap for a more Agile software development process Making software development more Agile Once upon a time, people wrote code and then ran it to find how well it worked. Working individually, the developer was able to iterate quickly until the software was useful to solve a real problem. But as software became more complex, it was broken into separate and independent sub-modules developed more or less in isolation by individual developers. Each sub-module was tested separately, the system was integrated back together, then tested again, staged, released, distributed and deployed. Software development cycles stretched over several months even several years. But now software engineering is now evolving towards shortening the feedback loops and minimizing the time it takes from code being written to being in production. This time, we want to do it with more quality and predictability. The tools to achieve that are processes, automation, and collaboration. Market pressures are forcing this change. Companies are being forced into a faster, nimbler software release mindset. Today, from five to fifty software releases a year is not unusual. With every company effectively becoming a digital company whether they are themselves delivering technology as a product or service, or relying on a technology to make their business run efficiently and successfully so the demand for fast software development time to market to maintain competitive edge is critical. In IBM s 2013 Global CEO study, more than 50% of CEOs surveyed recognized that effective software development is key to achieving a competitive advantage, but only a quarter felt successful at leveraging it. In this context, the concept of Agile development was born. 2 Daitan Group 2016 Accelerating. TM

2 However, in mission-critical environments, advancing Agile in enterprise software development cannot happen at the expense of quality or reliability. So, new methologies have emerged that encompass testing, deployment and delivery of software. These methodologies include new processes, automation, and collaboration practices that mitigate the risk of reduced quality and reliability. While there is no magic recipe for implementing Agile software development, there are good guidelines, best practices, and process frameworks that can be used. We have learned many of them first-hand at Daitan Group. We see the need for each project to have a process to fit its needs, and the needs of the client team. We have seen that it s critical that the process is regularly reviewed and adapted after each iteration using feedback from our own teams, and our client teams. This white paper presents a roadmap for migrating a software development team to a more Agile-centered process, shows some of the best practices we have learned at Daitan, and lists some of tools we have used to make this roadmap a reality. Agile development practices have steadily risen to become a trusted and preferred method of development for software teams everywhere. Using Agile, organizations can respond to market changes faster, deliver higher quality software, and gain a significant competitive edge 1. HARVARD BUSINESS REVIEW ANALYTICS SERVICES 1 Agile Practices: The Competitive Advantage for the Digital Age, Harvard Business Review 3 Daitan Group 2016 Accelerating. TM

3 Moving to a more Agile software development process 1 AGILE REQUIRES A STRATEGIC APPROACH, EFFECTIVE TEAMS AND AN INTELLIGENT USE OF TOOLS & TECHNOLOGY At Daitan Group, we are frequently called upon to migrate existing team structures to a more Agile process, and to model best practices for Agile software development. In so doing, we have learned some important best practices. A strategic approach Agile software development requires a strategic approach In order to be successful in both transitioning to, and improving, an Agile development process a team requires a highly strategic approach to be agreed in advance. In this document we present a roadmap for Agile process steps, but that roadmap should start with a strategy. A strategy would take into consideration the goals or benefits to the business and also the development team and/or the development cycle requirements. For example, automation has a considerable pay-off if there is a goal to reduce the develop-test-release cycles to ensure more frequent releases to end users. Effective team building Agile software development requires effective team building We have learned at Daitan Group that the extent to which Agile is embraced relies on cultivating a fully Agile mindset in the entire team, from a project s executive sponsors, through client engineers and extended team members. Success increases if a project s team members, from the top down, are educated on the benefits that result from the Agile method. Agile requires trust. In the entire team. So creating a relationship with internal and external outsourced teams based on trust and collaboration is critical. It s critical to build teams that will not only use Agile methods, but will also adapt to changes in requirements, scope, and priorities as the project moves forward. This team mindset underpins the success of every Agile project. A culture of fragmented accountability with a focus on individual deliverables or lack of commitment to overall product success cannot succeed in being Agile. 4 Daitan Group 2016 Accelerating. TM

4 Tools & technology Agile software development requires an intelligent use of tools and technology To respond to the needs of Agile development, new tools and technologies have emerged in the last few years including distributed and adaptive project management tools, collaboration and communication tools, automation frameworks, and more. So it s important to select the right tools at the outset of a project, and get the whole team trained and up to speed. Many tools are available as open source making it cost effective to take advantage of these time-saving tools. In this document we have listed the tools we use on a regular basis on our client projects. 5 Daitan Group 2016 Accelerating. TM

5 A Roadmap for a more Agile software development process In the diagram above, we present a roadmap that, when followed, will resuled in increased overall software process agility. The roadmap suggests a path that a team may go on when journeying from a more traditional software development process (such as Waterfall) to a more Agile process that accommodates build automation, test automation, continuous integration and delivery. Improving software process agility allows organizations to see significant changes and benefits independent of where the organization is on the roadmap even before reaching the desired destination. At Daitan Group, we partner with our client organizations to move their software development through all stages shown in the roadmap, so they can take advantage of the benefits of a fully agile process. Our goal is to always maximize development efficiency to speed our clients time-to-market. Moving across the levels involves not only adapting processes and automating them, but also changing organizational culture and structure as needed to fully embrace this approach. Source Code Control and Build Automation are the first steps in managing development processes. Every software development organization larger than a handful of developers will have done these steps. Test Automation allows the develop/test cycle to be repeated more often, increasing speed and efficiency. As process matures, Continuous Integration allows developers to write and test software functionality as a continuous process. Release Automation enables software to be automatically packaged, deployed and tested in an infrastructure that simulates production environments. Continuous Delivery, or the continuous and automated delivery of software to the end user, is the last stage in that evolution process. Typically it will include offering the software in the cloud, so that software delivery does not require updates on the customer premises or in the client equipment. 6 Daitan Group 2016 Accelerating. TM

6 Source Code Control and Build Automation 1 The absolute baseline for any software development process is a source code control system that allows a team of software developers to work together with a consolidated source code repository. This repository must accommodate versioning, and the process for compiling code and generating the executable software must be fully automated. The repository keeps everything needed to build the application, and the team normally operates under common guidelines and rules for branching, committing, merging, and minimizing the friction of code integration. The automated build system compiles, links and does everything else needed to get the application to run correctly for a given target environment. No professional software development today is done without some form of source code control and build automation. As outsource partners, we are regularly called to integrate into existing client source code repository systems. Consequently, our developer teams become familiar with all leading systems. Best practices for Agile source code control & build automation Here are two best practices for source code control and build automation that we have learned to help speed our clients Agile development process: Perform static code analysis Have your build system perform static code analysis such as code coverage (the % of the code base that is covered by tests), code duplication and unused code detection. This will help your code stay clean and less bug prone. Integrate the build tool and Version Control System. Our agile teams configure build systems to allow a build to be triggered every time a developer submits a commit to the main code repository. You can also configure this to happen on a pre-determined basis. Either way, once they are integrated, it becomes far easier to see which code changes caused a failed build. These are source code control and build automation systems we use on a regular basis at Daitan Group: CVS or Subversion (Open Source) ClearCase Perforce Git Bitkeeper Mercurial (Hg) Maven Gradle Ant 7 Daitan Group 2016 Accelerating. TM

7 Test Automation 2 Before Agile test automation, test cases were run manually and then automated through scripting. This resulted in a lack of coverage, a lag between coding and test automation, and caused quality assurance of software to be an expensive and often unsuccessful proposition. When release cycles were lengthy more than a year for example automating tests was less critical. But now, companies want to release to end users multiple times a year. So test automation becomes mandatory to maintain a fast time to market. Rather than testing software manually, test automation employs tools to run test cases without intervention. Output is then compared to what was expected. Agile developers have been experimenting with new approaches to automated testing of software for some time. Test-driven development (TDD), where unit test code is written to define the functionality before the actual code is written is one popular technique. Other related techniques are Behavior Driven Development (BDD) and Design Driven Development (D3). A Quality Assurance (QA) automation framework sets the rules of automation for a specific product. It integrates the function libraries, test data sources, object details and various reusable modules. These components act as small building blocks, which need to be assembled to represent a business process. The framework provides the basis of test automation and simplifies the automation effort. Best Practices for Test Automation Here are some best practices for test automation that we have learned to help speed our Agile development process: Keep the build fast to find failures quickly As your system grows, your build execution time will grow with it. This means that build execution time can take a long time (hours) to complete successfully. To mitigate this, focus on detecting failing test cases as soon as you can. Run tests in parallel If your build takes 2 hours running on a single machine, consider adding one more machine and make each machine run half of your tests. Keep doing this until you maximize efficiency. This method will save you build time, and possibly make room for the build to run other kinds of tests, such as performance tests. Embrace unit testing Unit tests run fast. So cover your code with as many unit tests as possible, using mocking techniques and tools to simulate external services. Cover the scopes that unit tests cannot reach with integration and end-to-end tests. Test in a cloned production environment Testing in a clone of the production environment allows you to anticipate problems that your system may have when deployed. Some bugs only appear when you change environment-related variables like database, operating system and processor architectures. These are the unit test and QA automation end-to-end tools we use regularly at Daitan Group: TestNG xunit Cucumber (BDD, Ruby) Jasmine (JavaScript) Selenium (Open Source, Web Interfaces) FITnesse (Open Source, Integrated tool) WindMill 8 Daitan Group 2016 Accelerating. TM

8 Continuous Integration 3 Continuous Integration is the practice of accelerating commits and the testing of code. In a Continuous Integration environment, developers check-in their work several times a day. A Continuous Integration server monitors those code check-ins and, as often as practical, it checks-out the new code, builds the system and runs unit and integration tests. If there is any failure in the process, the Continuous Integration server alerts the developers. A broken build then gets the top priority attention of the team. Ideally, Continuous Integration pairs continuous builds with Test Automation, to ensure each build hasn t impacted the quality of the code base. To really see benefits, developers must commit to address any code issues right when they happen, and not postpone. In situations where code is tested manually, comprehensive feedback can take days to happen. But when implemented along with Test Automation, it really makes instant code feedback possible. Mandating frequent integration keeps developers from going a wrong direction for too long. However, even when done with manual testing, Continuous Integration is still light years more productive and results in faster time-to-market than the traditional waterfall development, where developers might work for weeks or longer before trying to integrate their work to the main tree, only to find them that the integration faces a large number of fundamental defects. Test automation tools used by Daitan teams include: Jenkins (Open Source) Apache Continuum (Open Source) Travis CI (both commercial and Open Source) Bamboo (Commercial) New generation cloud-based Continuous Integration frameworks we use at Daitan Group include: Semaphore BuildHive In cases where there are multiple teams or cells, allocating at least one person with an end-to-end view of the product to be responsible for QA can further minimize integration problems at the end of each development sprint. Continuous Integration requires comprehensive commitment, and can really test a team s commitment to Agile! But it has huge benefits. Benefits of continuous integration: Continuous Integration does not eliminate software bugs, but it accelerates their discovery and keeps code from going in the wrong direction for too long Developers pay more attention to quality and tests since Continuous Integration failures are very visible and usually block work if there is a broken build The system is always in a state that can be released to users. This provides more flexibility to the business and enables the automation of the release process and delivery to users. With branching, continuous integration, and test automation, teams can be productive and innovative while still protecting code quality. This is Agile development at it s best: delivering working software regularly, with minimal technical debt and without compromising ingenuity. ATLASSIAN, THE AGILE COACH 9 Daitan Group 2016 Accelerating. TM

9 Release Automation & Continuous Delivery 4 Release Automation and Continuous Delivery is the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers on production at any time. You have achieved Continuous Delivery when software can be deployed to customers at the push of a button (and no one panics). Continuous Delivery can usually be approached once Continuous Integration has been achieved. Because Continuous Delivery requires that the whole development team is highly focused on prioritizing a codebase that can be pushed to production at any time, it requires very high standards of automation and testing. In a Continuous Delivery environment, code is compiled and packaged by a build server every time a change is committed to the source control repository. It is automatically tested before it can be marked as releasable. With Continuous Delivery, there is potentially no delay between enhancements and bug fixes being implemented and being made available to users. In a Continous Delivery environment, developers are aware that any code commit may be released to customers at any point. Code branching might still be necessary in a complex system, but only for limited periods of time. One advantage of an automated Continuous Delivery system, is that changes to the system can be released on a highly granular basis. This means that it becomes possible to release features to a subset of users, and seamlessly roll back to a previous version if needed. This is enormously beneficial if a company wishes to test a new subset of features to a limited audience to get feedback before releasing it to the entire customer base. Make no mistake: Continuous Delivery is not the exclusive domain of unicorn companies and tech darlings. Every team from the humblest startup to the stodgiest enterprise can and should practice continuous delivery. ATLASSIAN, THE BUSINESS CASE FOR CONTINOUS DELIVERY 10 Daitan Group 2016 Accelerating. TM

10 Conclusion At Daitan Group, we focus our expertise and our teams on delivering high quality software that meets all the latest best practices for Agile software development. We are frequently called upon to migrate existing team structures to a more Agile process, and to model Agile best practices. Our client needs vary, from quite immature software development processes, to advanced levels of Agile. But regardless of stage, it s a journey. New tools and techniques move to the forefront. It s important to remain completely up-to-date on these changes. We understand it s mandatory to know how to advise our clients on the choices that have positive impact on time to market. We welcome you to contact us if you have more detailed questions, or if you want to talk over how best we can help your software development teams become more Agile. About Daitan Group Daitan Group provides high quality software development services to significantly accelerate time to market for global technology companies. The company s expert agile teams deliver full lifecycle software product development, maintenance and quality assurance services across today s leading technologies, including: cloud computing and virtualization; communications, collaboration and messaging; and big data/analytics. For more information: DAITAN GROUP HEADQUARTERS 2410 CAMINO RAMON, SUITE 285 SAN RAMON, CALIFORNIA, USA PHONE: +1 (925) FAX: +1 (925) Daitan Group 2016 Accelerating. TM