7 Tips for Raising The Quality Bar With Visual Studio 2012

Transcription

1 Visit: 7 Tips for Raising The Quality Bar With Visual Studio 2012 Tip 1: Exploratory Testing I have to admit that when I first found out that enhanced exploratory testing was the major new feature in the 2012 release of Microsoft Test Manager, I was a little disappointed. After listening to the Microsoft team promote the feature and trying it out for myself I am sold. The Exploratory Testing feature really works well in an agile work flow. The idea behind this form of testing is that we spend more time on testing and less time on writing up bugs and setup. I believe Anu, the Product Owner for Visual Studio ALM tools focused on Test and Lab Management, likes to call this more T and less BS. Exploratory testing in MTM 2010 was possible but very clunky. We had to start with a test case with maybe a single step of explore, or something similar, associate the test case to the work item, and execute a test. When we were done with our scenario we could open the action recording, copy the actions, and paste them into the steps grid of the test case. Clearly this was a hack rather than the good use of a feature as designed. In 2012 the workflow is vastly improved. The Testing Center > Test > Do Exploratory Testing view was specifically designed to support a workflow where the feature was Explored rather than scripting the test steps in advance. Within this view we can start an exploratory session with a single click or we can search for a work item and click the Explore work item button which will automatically associate our generated test case(s) to the Work Item.

2 The Exploratory testing feature also works well with a help desk scenario: User calls help desk to report a problem. Help Desk operator launches an exploratory test. User describes how to duplicate the issue as the operator executes the steps. If things go well, operator duplicates the problem and can immediately create a bug and even a test case. To create a Bug and a Test Case with just a few clicks: Do Exploratory Testing > Recreate the problem > Click Create bug > Enter required fields > optionally edit steps to reproduce > Click Save and create test > Enter required fields > optionally edit steps > Save and close > resume testing. 2

3 Multiple bugs and test cases may be created for each Exploratory test session. 3

4 Use the comment field on the exploratory test runner to enter rich comments and capture screenshots as you explore the application under test. These comments and images will be inserted into any bugs we create as observations that are mixed in with the actions we performed. 4

5 If we are doing exploratory testing against a requirement, we can View the acceptance criteria by clicking the link containing the PBI Id and title 5

6 6 Consultants Who Teach

7 How many times does a feature sketched out on a white board match the final implementation? Not often!. By using the new enhanced Exploratory Testing feature, we can spend more time focusing on the quality of our software and less time on authoring and refining our test cases. Strength Vastly simplified work flow, reduced typing, less work item relationship management, reduced test case refinement. Weakness None really, we can always use the more traditional work flow of authoring test cases before we start testing. Caution -If you are running MTM 2012 against TFS 2010, you will not be able to use the exploratory testing feature. If the development team is not using good control naming conventions you are going to waste a bunch of time translating the steps generated from the action recording to something meaningful the team can execute next time the test case is run. 7

8 If you have Video recording enabled, do not perform private tasks on screen 1. Interactions with an or IM client, for example, will be visible in the recording if they are on screen 1. If you have audio recording enabled, be aware that conversations (or other sounds intended to be private) may be recorded as well. You don t want your developer teammate to know what you really think about their programming skills do you? 8

9 Tip 2: Pass Tests without Launching Test Runner Most companies I work with have a release process that involves a number of tasks that are executed right before a release. These tasks are necessary for each general release but they aren t traditionally considered tests. The tasks might include things like notifying the release manager of the release date, assembling documentation, compiling release notes, and any other tasks you must do 1 time before you can release. If you want to track the state of these items along with the state of the rest of your test plan, create a test case for each of these tasks (you don t even need to bother with steps). Once the task is complete you can rightclick the test case (multi-select supported) and mark the test pass/fail without having to launch test runner. Now your quality reports include the state of these tasks along with the rest of your tests. Testing Center > Test Tab > Run Tests activity. Select Suite > Highlight one or many test cases > Right-Click > Pass Test (or click the pass test button on the toolbar) 9

10 Tip 3: Configure Multiple Test Settings Executing a test run will generate and store diagnostic data in the TFS Database. This can quickly become a storage problem for the TFS Administrator. To help mitigate this problem, we should configure multiple test settings for use in our test plan. The default test setting should be a lighter weight configuration in that only the data collectors we find really useful day to day should be enabled. This light weight configuration should NOT include capturing of video but should probably include an action recording and test impact information. We should also configure a fuller diagnostic test setting that collects any information that might be useful to a developer, and this configuration may include the the video recording. 10

11 In the Test Plan Properties page, we should configure the plan to default to the lightweight diagnostics Test settings. If we are using MTM to execute automated tests, we would also want to configure lightweight and full diagnostic test settings for our automated runs as well. When we execute a test with the Run menu option, the default (Lightweight) test settings will automatically be used. If we start playing ping pong with the developer or they are having trouble reproducing a bug, we can use the Run with options feature to select the Full Diagnostics version of the test settings. 11

12 If we do ever run into problems with old test runs taking up to much space in our database we should consider using the TFS 2012 Power Tools Test Attachment Cleaner. 12

13 Tip 4: Pause and Resume Manual Testing I do a fair number of talks about and trainings on Microsoft Test Manager and one of the more frequently asked questions is how do I pause and resume a manual test?. When I show the feature in Microsoft Test Manger 2010, everyone is happy until I caution the group that the test run will be lost if you close MTM. This is not an ideal implementation but the good news is that with MTM 2012 Update 1 this feature has been improved. Multiple test cases may be paused at any given time and the paused tests can survive a restart of MTM. 13

14 To Pause a manual test, click the Pause button then the Save and Close button. Notice we have have a new Paused grouping in our test suite and a new Paused test(s) state indicator To resume the test execution either Right-click the test case and choose Resume test run or highlight the test case and click the resume button. 14

15 The attachments and state of the in-flight test run is stored in TFS so you WILL need to be running against TFS

16 Tip 5: Troubleshoot Coded UI Issues If you are new to coded UI tests, troubleshooting issues can be a challenge. My favorite tool to use when I am having problems with a Coded UI Test is the HTML Logger. To demonstrate a couple issues, I recorded a test method with the following Actions: 1. Launch a new instance of Internet Explorer. 2. Navigate to Bing 3. Enter a search string. 4. Hit the Back button 5. Click on the Search History link 6. Validate that my search string is in the history. 16

17 I then commented out the line of code responsible for clicking the Search History link, my Test Method looks like this: [TestMethod] public void CodedUIBrokenTestMethod() { this.uimap.launchbrowser(); this.uimap.searchcodeplexforcuiguidance(); this.uimap.clickback(); //this.uimap.clicksearchhistory(); this.uimap.validatesearchhistory(); } If we run the test, not surprisingly, we get a failure as the search history link is not on the Bing search result screen. In visual studio we get the information below. An experienced Coded UI Tester probably won t have any trouble figuring out what is wrong. What is Visual Studio telling us? 1. Test explorer shows us we have a failed test method. 2. Test Explorer also gives us some more troubleshooting information such as a link to the code file and the line number of the failing code, a link to the output window, exception details, the call stack, and execution time. 3. The output window gives us the test outcome and exception 4. The output window also gives us a link to a screen grab of what the desktop looked like at the time of failure. Not bad really, all the information we need is here if you know what to look for but for someone new to coded UI testing, it is intimidating. 17

18 Now we will rerun the failing test with the HTML logger enabled. Open C:\Program Files (x86)\microsoft Visual Studio 11.0\Common7\IDE\QTAgent32.exe.config with your favorite text or xml editor (be sure to launch the editor as admin). Locate the system.diagnostics switches EqtTraceLevel node and set the value to 4: <system.diagnostics> < switches> <! You must use integral values for value. Use 0 for off, 1 for error, 2 for warn, 3 for info, and 4 for verbose. > < add name= EqtTraceLevel value= 4 /> < /switches> < /system.diagnostics> Save the file and rerun the test. Notice at the bottom of the Test Output window rather than an image we have a link to an HTML file named UITestActionLog.htrml. 18

19 19 Consultants Who Teach

20 Click it (and select your browser of choice if this is your first time viewing the log). 1. We get a very nice visual representation of our test method run. 2. We get the total method execution time 3. We get timings for each action the test engine is executing 4. Each of these actions is expandable and by default all passing steps are collapsed. Also each action has a corresponding screen grab of the desktop. 5. We get a user friendly error message right after the failed Action 6. We get an image of the failure (click it and it toggles to the size of the browser window). We also get all the gnarly details A novice should be able to look through the actions in this log and determine that there is a navigation issue. Even if the problem is not obvious, the more granular actions being reported should give us something specific to search for out on the MSDN forums for example the friendly error or UITestControlNotFoundException. If you post a tough question to the forum, one of the first things the community will ask for is trace information this is it! 20

21 As an aside, by looking at all of the actions another thing we might notice is that we are spending 1.5 seconds waiting for a hover (hovers can add up when you are testing a more complicated interface). If you don t want to record and test hovers or if you want to fine tune other performance issues see page 34 of the Coded UI Testing guidance out on codeplex published by the Visual Studio ALM Rangers. 21

22 Tip 6: Tuning Coded UI Tests For our first tuning tip. let s say we have all of our Coded UI tests passing but execution time is taking too long, what do we do? If we find that we are working with a lot of controls that don t have a name or id associated to them, we need to remind the developers that it is a best practice to set the name/id property of all controls that will be used as a part of a coded UI test. Yes it will bloat our html a little bit and add some (usually fairly trivial) logic to the UI code but for testability it is well worth it. We could also use the HTML Logger I mentioned in my last post but inspecting the HTML log for a bunch of tests that are passing would be time consuming. Rather than that, I suggest we modify some playback settings and see what breaks and go from there. 22

23 We ll start by setting the Playback.PlaybackSettings.MatchExactHierarchy to true. We can do this in a Class Initialize method so we don t need to do this for each test method. [ClassInitialize()] public virtual void MyClassInit() { Playback.Initialize(); Playback.PlaybackSettings.MatchExactHierarchy = true; } By default, the playback engine will be configured to run more resiliently which can negatively impact performance. By setting the MatchExactHierarchy to true we are telling the playback engine to look for controls in the exact place they were recorded and fail if they are not found. When the value is false, the playback engine will first look for an exact match, then look for the next closest match which can be a time 23

24 consuming process. Now that the playback engine is configured to identify only controls matching an exact hierarchy we will rerun all of our tests. If none of them fail, we cannot improve performance with this technique. If some of the tests do fail, we will inspect the tests and see if we can modify them so that an exact match search does work. This optimization should reduce the execution time of that test. Once we have completed this analysis and optimization on each test we can turn the MatchExactHierarchy property back to false if we want. If you do leave this property false, I suggest repeating this exercise every couple weeks or months to keep the coded UI test execution times down. If we want exact match enabled for all but a couple of our tests we can simply override the setting in the test method: [TestMethod] public void MyClickyMethod() { //For this method don't use exact match Playback.PlaybackSettings.MatchExactHierarchy = false; this.uimap.anotherclick(); this.uimap.clickweblink(); this.uimap.searchsomething(); Playback.PlaybackSettings.MatchExactHierarchy = true } In conclusion, we set the Playback.PlaybackSettings.MatchExactHierarchy to true and reran our tests. The tests that fail are the ones that are not using the exact match algorithm and therefore the playback engine is taking longer to locate the controls. We attempt to modify the search properties for these controls so that the playback engine can use the exact match algorithm and speedup total execution time. 24

25 Tip 7: Test Suites Logically organizing test cases with test suites will make managing our test plan a lot easier. The first question we ask ourselves is do we want the suite to be static or dynamic? We have two choices for creating dynamic suites. The first choice is requirement based suites. To add a requirement based suite to a test plan, select the Add requirements button. We are presented with an editable query that defaults to the team project associated with the test plan, a list of work items in the requirement category group under the area path associated to the test plan. Run the query to return the requirement (Product Backlog Item / User Story / Requirement depending on the process template in use). 25

26 From out list of requirements we can select one or many requirements and add them to the test plan. Requirement based suites have the following icon: The cool thing about a requirement suites is that they are dynamic in nature which means that if another tester adds a test case associated to the requirement it will automatically show up in the requirement suite (there may be a delay so use the refresh button if you don t want to wait for the UI to catch up ). The other nice thing about a requirement suite is that if we create a new test case from within the requirement based suite, a link from the test case to the requirement will be automatically added for us. Any time we can automate traceability or eliminate manual steps that is a good thing. The down-side to requirement based suites is they are not hierarchical (cannot contain sub-suites) and the dynamic nature of the suites can be a problem. If we require absolute historical accuracy of our test plans, requirement based suites should be avoided. For example, assume we have a requirement based suite in test plan 1 with 5 test cases. Now 3 sprints later we discover we missed a test case for this requirement so we add a new test case in test plan 4. If we navigate back to test plan 1 and look at the list of test cases, this new test case will be included in the requirement based suite. This is generally not a problem but in highly regulated industries this may be a show stopper. 26

27 The second dynamic suite is a query based suite. This is similar to the requirement based suite only we can make the query about any thing we want. To add a query based suite use the New dropdown and select Query-based suite: Again we will be directed to a query wizard. In this example I created a suite that contains all test cases in this team project with a priority of 1 27

28 The icon for the resulting suite is as follows: This suite has similar pros and cons as those of a requirement based suite in that test cases are added to the suite automatically when other testers add test cases to TFS that satisfy all of the query criteria but again, historical accuracy may be an issue and query based suites cannot have sub-suites. The final type of suite is a static suite. The static suite is the most robust. To add a static suite, use the New dropdown and select Suite: Static suites may have multiple levels of sub-suites and these sub-suites may be static, requirement,or query based: 28

29 Static suites will have the following icon: We can use static suites to slice and dice our test plans however we want. The down-side is that we are responsible for adding test cases to the suite and creating test case to requirement links manually. The up-side is that Static suites will maintain their historical accuracy. 29