Life between Iterations

Life between Iterations Challenges in Agile Testing Padmasankar Jadu, Senior Delivery Manager, Global Business Services, IBM India Pvt Ltd. STC 2013, Bangalore

Biography of the Author Padmasankar Jadu, Senior Delivery Manager at IBM Global Business Solutions brings in around 15 years of testing experience to the table through various development models including Agile, various domains, product development and complex program delivery. In his current role, he manages a testing engagement with one of the largest Telecom services provider in US serving projects in both Agile as well as Waterfall model. He has proven experience and research work in the area of Six Sigma for Software, Design of Experiments etc. Abstract A widespread adoption of agile development model, though a big boost for the consumers of the software industry but it brought in additional challenges for the practitioners especiallyfor the testing professional in the team. In this paper the author illustrates few of the evident challenges faced by the testing practitioners in agile projects. A mapping of these problems along with basic principles of agile methodology is illustrated by the Author. Most of these problems are faced by practitioners on the floor especially on projects with lower level of agile maturity or for the early adopter of Agile. Few solutions are given to address few of the challenges faced by the testing practitioner as well testing organizations practicing testing for agile projects. Though not discussed in detail this paper, a mindset change from Testing in Agile Projects to Agile Testing is the key for this transformation. The Author will take the audience through a case study to bring in more clarity. Life between Iterations, Challenges in Agile Testing Page 1 of 12

Background With the current macro-economic scenario which is persistent for more than a decade time to market and lean principles in practice are the key business drivers for any organization. The traditional software development model influenced by the earlier successful mammoth projects had a cycle time in years or months. Those projects were characterized by - Long term planning scale - Project teams distanced from the Customer - Time gap between specification and implementation - Longer time to discover problems - High risk project schedules - Inability to respond to change quickly In spite of following the best practices of the project implementation or using cutting edge technology many of these projects did not succeed with respect to time to market, customer expectations and cost. A decade ago Agile Software Development Methodologies came in with bang giving a hope to solve the deficiencies of the traditional development practices. Agile visions a ways of creating software in a lighter, faster and more people-centric way. The base of this is the Agile Manifesto, widely regarded as the canonical definition of agile development and accompanying agile principles. The Agile Manifesto states that 1 - Individuals and interactions over processes and tools - Working software over comprehensive documentation - Customer collaboration over contract negotiation - Responding to change over following a plan As a methodology, agile development tries to push the development process to the user intended direction, providing many opportunities to assess the direction of a project throughout its lifecycle. This is achieved through iterations of work and at whom end teams must present a finished piece of code, usually a piece of shippable code. Because of this repetition of small chunks of work, as well as the 1 http://agilemanifesto.org/ Life between Iterations, Challenges in Agile Testing Page 2 of 12

functional product they yield, agile methodology can be described as iterative and incremental. As opposite, in the traditional approach, development teams have only one chance to get right each aspect of a project. Because of the short periods between reevaluations, usually couple of weeks, when the team stops and re-evaluates the direction of a project, there s always time to redirect it in the correct direction. Testing in Agile Projects The wide spread acceptance of the Agile Methodology has a transformational impact on the way testing function is organized in software projects. In the plan driven traditional development models, testing was incorporated through various well defined phases viz., test requirements, test planning, test design, test execution, test reporting and closure with clearly defined roles like Test Manager, Test Architect, Test Engineers etc. Whereas in Agile, these phases no longer exist or exist with overlapped boundaries. This resulted in a paradigm shift in the way the separation of duties was defined in projects especially between testers and developers as well how the testing organization were structured. This figure depicts a typical SCRUM cycle. Taking an example of SCRUM 2, as agile development model, a higher degree of collaboration is required between the testers and the other roles (developer, product owner, scrum master and the project manager) as a day to day practice for the success of the project. Requirements in Agile projects are 2 http://en.wikipedia.org/wiki/scrum_%28software_development%29 Life between Iterations, Challenges in Agile Testing Page 3 of 12

mostly defined in form of user stories. Testers get involved as soon as a user story is defined. In some agile models like Test Driven development tests are written even before a piece of application code is written and hence putting a strong focuses on early validation. In other models like Pair programming, the testers & developers work together to develop and iteratively test a requirement. Thus a tester has to take varied roles in agile projects, unlike in traditional models where his roles & responsibilities are confined to writing test cases, executing tests and reporting defects. In agile, tester has to not only write test cases, but also ensure maximum coverage through automation, not only report defects, but also help developers debug and fix defects as required. Thus, throw over the wall concept does not work in agile projects, where development & testing effort is highly integrated. The change in testing practice is enormous and has put forth many challenges for testing organizations and testing professionals, few of the evident ones are list below Challenges faced by testing professionals The very construct of the Agile Principles brings in challenges for the Testing Team in an agile engagement; the specifics of those challenges are grouped as per the primary principles of Agile as defined by the manifesto. Various other researches are available on the technical, process and cultural challenges in agile projects and that is not in the scope of this write-up. - Individuals and interactions over processes and tools: In agile development, self-organization and motivation are important, as are interactions like co-location and pair programming.» Distributed Agile:A highly collaborative approach is required because the Traditional supplier-client relationship is often difficult and a common result is that either the client or the supplier will drive the project to a waterfall or even compressed waterfall approach. Face to face interaction is always required to build the needed mutual trust between the Developer, Product Owner and the tester. The absence of this puts the testing team in pressure of establishing themselves. The need for working on other time zone also puts pressure on the testing teams in agile projects» Too many meetings : Agile projects makes in mandatory for the testing teams to attend various meetings throughout the day Iteration Planning Meetings, UserStory Estimation Meetings, UserStory per Iteration Review Meetings,Daily scrum meetings and Weekly Life between Iterations, Challenges in Agile Testing Page 4 of 12

management meeting. etc. which takes away the precious time required for test planning or testing» Too many approvals: The closed involvement of the user or the user s representative in conjunction with the process (however light it may be ) brings in additional overheads to the testing team in getting approval for the test strategy or the test results.» Collaboration with the Developer / Product Owner:Especially in a distributed agile testing scenario it becomes tough for the testing resource to collaborate with the other stakeholders. With the product owner at a customer location testing team has to spend additional effort in connecting with them for reviews / approvals.» Demand for more Domain and Technical Knowledge: The demand for knowing the technology as well domain is high for a testing professional in an agile team as the time available to make decision is comparatively low. Since each of the testing professionals works independently, technical and domain competency plays a key role in the success of testing function in a project» New to Agile, New to the Project, New Technologies, New Vendors: In the beginning any agile project, the team faces many new entities; including new people, new roles, new process, new workflow tools etc. With many new entities around it takes a while for the test specialist in the agile team to optimize his space, control and establish his or her identity resulting in productivity loss during the early phase of the project.» Many unknowns in the early iterations: Estimating the defect arrival rate, test velocity as well the knowing the product itself is challenging during the first few iterations in any new agile project. - Working software over comprehensive documentation» Lack of Adequate documentation especially on Enhancement Projects: With the stated principle around working software over comprehensive documentation, in the absence of intuitive tools, agile projects ends up very poor project / system documentation. This will result in lack of documentation around the architecture of the application under Life between Iterations, Challenges in Agile Testing Page 5 of 12

test and lack historical documentation which is important for the testing and troubleshooting in complex projects» Poorly defined User Stories: User stories are the crux of requirements in any agile project. Agile methodology mandates the user stories to be Negotiable, Valuable, Estimable, Small and Testable. Any compromise in any of these qualities makes testing difficult.» Test Automation, Testability: Putting these two factors together, success of testing in any agile depends on the testability of the user stories as well as the system. If the system is not developed considering testability. Testing become ineffective it the test specialist is not able to find an effective means of defining a pass criteria for the user story. Test Automation plays a key role in agile project to meet the pace required for the agile project. While you move from one iteration to another, depending on manual testing becomes suicidal for regression Testing. Also, there is a need to considering the testability factor while designing the application ( from example testable component interfaces which can be addressed through white-box testing)» Poor Unit and Integration Testing: Dumping untested code to the testing resource will not yield the anticipated benefit from Agile. Most of the time, the development team is tempted to hand-over the code to testing team within an iteration due to time constraints or due to the availably (perceived) unit test engineer in the team. This will result in testing / defect backlog beyond the acceptable limit. - Customer collaboration over contract negotiation» Frequency of changes: Even though, there is nothing called scope creep in Agile poorly managed change control in Agile teams will makes testing less effective, especially when the changes arrive at the later iterations in the cycle. This will bring additional overhead of redefining the test strategy, re-work on the automated test scripts as well additional regression testing. Life between Iterations, Challenges in Agile Testing Page 6 of 12

» FrequentFeature Demos and review meetings: Along with the pressure for completing iterations, testing team (and the other functions in an Agile project ) is always under pressure for review meetings as well as feature / product with the product owner / customer. This becomes further difficult on distributed agile teams. - Responding to change over following a plan» Over commitment: Even though we have standard (agilespecific) methods in place, inexperienced and enthusiastic agile practitioners tend to over commit / under estimate especially during the early iterations putting the whole team in pressure. This will increase product backlog, test backlog as well as defect backlog.» Poor change management: Even though changes in customer needs are accepted throughout in an agile project, but unexpected changes which are made once the scope the iteration is defined makes it difficult for the tester to complete the committed work within a iteration. Also, too many changes being made to the application under test through iterations also makes it difficult to maintain the automated regression suite or do a manual coverage.» Crunch times in every iteration: In every iteration, the test specialist need to complete various task (mostly 2 weeks sprints), this would involve design of test cases (with our without a test case management tool ), get the approvals from the product owner, execute them, report fixes, test the defect fixes, review the readiness, participate in the product demos, attend daily review meetings, Automate test cases etc. Most of the time, poor quality of the defect fixes create havoc for the test specialist. - Other» Loss of identity: In an agile project, testers do not perceive an owner of quality identity. Test specialists moved from centralized testing organizations always feel a loss of identity in Agile Development teams. The traditional roles of Test Lead, Test Architect and Test Manager may not be present in many of the Agile Teams.» Lack of training: Crunch in the cycle time and the ever growing pressure of completing all the planned activities for every iteration, the on-going training need is ignored these would be for the application under development, domain, test automation tools, agile collaborations tools etc. Life between Iterations, Challenges in Agile Testing Page 7 of 12

» Ever growing test backlog: With poorly planned sprint cycles, the number of pending tests to be executed or test which are moved back to the backlog ( to be picked in the next cycle or the last test only cycle grows to an unmanageable number puts the test specialist into pressure to spend long working hours during the later sprints in the overall agile cycle.» Ever growing defect backlog: Poor quality of the sprint planning and the poor quality of the application under development / test would increase the number open defects or defects which are pushed across iterations will grow putting the test specialist in pressure to cover more during the later iterations. Challenges faced by Test Organizations» From centralized QA to Project centric organizations: In Agile Software Development, the test specialists are part of the agile projects and they no longer report to a centralized testing organization giving a shared vision to the testing team. This builds enormous pressure and disorganization for any supplier (or a self-sustained organization) to develop workforce with testing skills.» Employee Morale: With ongoing need of completing the work for every iteration, in most of the cases especially with lower level of agile maturity, the test specialist in the team has to spent longer hours at work and / or work during evening hours to collaborate with the developer or the product specialist. The situation is even worse in distributed agile teams where the customer and/or the product owners as well as the developers work on a different time zone. As stated earlier, effective collaboration is essential to the success of any Agile project, which results in wide-spread work-life balance issues in the work-force resulting in frequent rotations and attrition.» Increasing non-labor cost, including overtime allowances, cost of telecommuting: Agile projects, especially those distributed agile projects increase the project cost for both labor and non-labor. This is primarily from long and extended hours the testing resources need to put for a project as well as the additional cost for commuting / Life between Iterations, Challenges in Agile Testing Page 8 of 12

telecommuting to manage the time-zone difference as well as the crunch in various iterations.» Need of new generation agile software development tools and training: Various new tools are being introduced by the Agile Projects which the testers needs to be familiar with these would include Agile specific software engineering tools, On-line Collaboration tools, Test Automation tools etc.. These additional learning requirements bring new challenges for the testing organizations.» Managing career path for the professionals: With centralized testing organizations being dissolved, organizations are finding it challenging to provide new career paths and grooming options for the test specialist. As a result of agile adoptions many of the existing roles in the testing organizations like Test Lead, Test Manager etc has vanished. These were the few challenges faced by testing professionals and testing organization with the wide spread adoption of agile teams within their organizations or customers. Though some of these challenges would exist in the current scenario, few them could be addressed through a holistic approach towards Agile testing and differentiate Agile Testing from Testing in an Agile Project.» Managed Communication: Communication between all the stakeholder in the team plays a key role in the success of any Agile project especially face-to-face communication. The very need of more communication in agile projects helps team in building clarity of requirements, clarity of the implementation as well as clarity on how / what is tested with a clear outcome, but this also brings in additional pressure on the testing practitioner in the team in communicating with various stakeholders during various phases in the project. A definite communication plan needs to be in place with a clear definition of what needs to be communicated, the medium of communication, duration / length of the communication, frequency of the communication as well as the distribution. For distributed agile projects, representation from testers at all the location would be advantageous. Agile teams which do close interaction with projects / software applications which are being developed in a traditional manner would need additional Life between Iterations, Challenges in Agile Testing Page 9 of 12

communication channels and people performing additional roles to bridge with work between those teams.» Leverage of right tools and techniques. Like the traditional project management / software engineering tools, there are plenty of the tools available in the market to enable teams in Agile Development / Testing which can be effectively used.these tools can help the agile teams in having documented /shared iteration planning, track the user stories, track backlog, create lean documentation, raise defects, have online-collaborative-meetings etc. Online collaborative tools (for example RallyDev 3 ) can host online virtual meetings as well. Agile without test automation would a first degree catastrophe. Agile testing team should always have a white-box or gray-box approach and should use in-process test automation either UI based to back-end component based to improve the test coverage from iteration to iteration. In all the cases, regression testing should be done in an automated way using appropriate light weight tools. Design for testability is also another factor which will dictate the easiness of test automation in any application under agile development.» Focused Enablement reskilling of the talent: Agile Team member always face challenges in new teams with the new process, new tools, new set roles (like product owner, customer, scrum master). Transitioning an experienced testing professional from traditional development models to Agile is a challenge for most of the organizations as it involves some amount of un-learning in the context of from the traditional model. Organizations need to have a centralized mechanism for training the testing team (and resources from the other streams) in agile principles, the organization s vision on adopting agile and tools/processes used in agile development / testing. Working on simulations also will help team in getting accustomed with agile development methodology before jumping into a project. The role of an Agile Coach is very much important for any organization adopting agile.» Employee Morale: Over-run estimates and working long hours definitely have an impact on the employee morale in agile testing teams. As mentioned above in this write-up, a loss of identity 3 www.rallydev.com Life between Iterations, Challenges in Agile Testing Page 10 of 12

and unclear career path when testing professional are moved from Centralized Testing Organizations to Agile Projects. Good Customer relationship as well a sustainable pace of development is also a key factor in agile model. Training on agile principles and tools also plays a key role in employee morale and issues in change management (from traditional to agile).» Change control, In-process Metrics: Even though changes are welcome in Agile development, in sprint (in iteration ) changes can be difficult for the testers in agile team. Adequate change control mechanism has to be put in to control the changes within a sprint. Also, metrics should be in place to constantly monitor the test velocity, defect arrival rate, defect backlog, test backlog etc. and put in control mechanism or / and improve the accuracy of Estimations. Summary Though Agile development brings in improvement in time to market, reduction in wastage as well value for money for the customer, it brings in additional challenges for the testing teams. A careful evaluation of the challenges would help Testing Organizations overcome those and perform. Though these solutions of the model mentioned above may not be directly applicable to all the scenarios, a context driven approach can be taken to make the life of a tester in an agile project better by having change in mindset from Testing in Agile Project to Agile Testing. 10 principles for Agile testers as depicted by Lisa Crispin; Janet Gregory in their book Agile Testing: A Practical Guide for Testers and Agile Teams. 4 4 http://www.amazon.com/agile-testing-practical-guide-testers/dp/0321534468 Life between Iterations, Challenges in Agile Testing Page 11 of 12