Automatic Test Suite for Physics Simulation System

Transcription

1 Examensarbete LITH-ITN-MT-EX--06/042--SE Automatic Test Suite for Physics Simulation System Anders-Petter Mannerfelt Alexander Schrab Department of Science and Technology Linköpings Universitet SE Norrköping, Sweden Institutionen för teknik och naturvetenskap Linköpings Universitet Norrköping

2 LITH-ITN-MT-EX--06/042--SE Automatic Test Suite for Physics Simulation System Examensarbete utfört i medieteknik vid Linköpings Tekniska Högskola, Campus Norrköping Anders-Petter Mannerfelt Alexander Schrab Handledare Marcus Lysén Examinator Mark Ollila Norrköping

3 Avdelning, Institution Division, Department Institutionen för teknik och naturvetenskap Datum Date Department of Science and Technology Språk Language x Svenska/Swedish Engelska/English Rapporttyp Report category Examensarbete B-uppsats C-uppsats x D-uppsats ISBN ISRN LITH-ITN-MT-EX--06/042--SE Serietitel och serienummer ISSN Title of series, numbering URL för elektronisk version Titel Title Automatic Test Suite for Physics Simulation System Författare Author Anders-Petter Mannerfelt, Alexander Schrab Sammanfattning Abstract Testing software is an important but time consuming task. Meqon AB as well as their product has grown in complexity and the need for a good testing system is greater than ever. This paper gives an overview of software testing and the needs for testing at Meqon as well as an introduction to the system we have designed and implemented for Meqon. The system which has requirements like ease of use and flexibility must still be powerful enough to be useful in the testing process. We discuss how this system was designed and what can be improved upon in the future to make the testing process at Meqon even better. Nyckelord Keyword Software testing, Automated testing, Test case framework, Rigid body simulation

4 Upphovsrätt Detta dokument hålls tillgängligt på Internet eller dess framtida ersättare under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: Anders-Petter Mannerfelt, Alexander Schrab

5 Abstract Testing software is an important but time consuming task. Meqon AB as well as their product has grown in complexity and the need for a good testing system is greater than ever. This paper gives an overview of software testing and the needs for testing at Meqon as well as an introduction to the system we have designed and implemented for Meqon. The system which has requirements like ease of use and flexibility must still be powerful enough to be useful in the testing process. We discuss how this system was designed and what can be improved upon in the future to make the testing process at Meqon even better. Keywords: Software testing, Automated testing, Test case framework, Rigid body simulation

6

7 Contents Abstract 1 Introduction Background Meqon The Meqon Middleware Purpose and goals Methodology Thesis overview Software testing Testability Reproducibility Model simulation testing General testing techniques Positive/Negative testing Black box vs. White box Test partitioning Regression testing Test phases Unit testing Integration testing System testing Acceptance test Design Testing frameworks Suitable testing for the Meqon SDK Meqon Research AB Existing tools at Meqon Visual debugger Demo framework Bug tracker i

8 3.2 Testing at Meqon System overview Testing technique Overview Test cases and the test case framework Implementation Usage Problems Analysis tool Design Implementation Problems User guide Test Cases and results Stacking test case Spring stability test case Vehicle test case Tunneling test case Contact point and normal direction Proximity detection Activation and deactivation test case Raycasting test case Detector speed test Vehicle test case up close Analysis and conclusions Analysis Problems Acceptance criteria Compilation issues Ageia s acquisition of Meqon Research AB Conclusion Questions revisited Future work The testing process at Meqon The automated test system Test Coverage Dealing with large test suites ii

9 Chapter 1 Introduction Software development is a complex task, more so today than ever. The idea of flawless programming is still just a dream, and no tools or methods exists today that can ensure perfect code. This issue becomes even more pressing in large scale projects where many people need to coordinate their efforts to create software that is infeasible to create single handed. Ever since the dawn of computers, programmers have been making mistakes in their programs. Mostly these mistakes are quite harmless and easy to detect and fix. However, some mistakes affect the quality of the software more than acceptable and depending on the system in question this might even be a matter of life or death. Bugs that directly lead to fatal accidents are luckily quite rare but bugs can have great economical impact on projects and might mean success or failure of an entire product, or even worse, the entire company. There is really just one protection against faulty systems, extensive testing. Testing is a key part of ensuring a quality product, and working with testing and quality in general is often referred to as QA (Quality Assurance). This final thesis project is an effort to create a flexible and powerful testing solution to be used at Meqon, the general idea for the project was presented to us by Meqon, but they left all the details to us. This report presents this project and is for anyone who is interested in automated testing and more specifically testing techniques applicable to a simulation SDK. It will also act as technical documentation of the system for Meqon as a future reference. 1

10 2 Chapter 1. Introduction 1.1 Background Meqon Meqon is a young Norrköping based company 1 whose main product is a physics simulation middleware for games and simulations. Typical customers include game developers that need to incorporate advanced physics into their games for a more realistic gaming experience, the demand for advanced physics in games is increasing since the growing power of gaming platforms and computers goes hand in hand with the customers demand for better graphics and a more immersed gaming experience. Instead of developing a physics system from scratch, a game developer can use the Meqon SDK which is a readily available physics system to simulate interactions in the physical environment like the behavior of cars, rag dolls (human like characters), boxes, hinges between objects etc and get an end result that will surpass anything that could be created with the resource and time limits of a regular game project. The Meqon SDK is known for it s nice design and good performance, but the development of an advanced system like this is always complex and this product suffers the same problems as any other advanced software systems. It is quite common that bugs in the system are discovered by the customers and reported back to the support team at Meqon. This is a quite common situation for software products and these customer discovered bugs can (and must, since they are impossible to avoid completely) be tolerated to some degree, but it must be kept in mind that for every bug the customer discovers and reports to the support team, there will always be a small decrease in confidence for the product. Since the company and the complexity of the product has grown over the years, the need for a good quality assurance process is greater than ever, and a testing system that could minimize these kinds of customer discovered bugs could prove very valuable in the long run. One problem with big systems like this product is that the correction of a fault in one part of the system might lead to errors in other parts since they interact in a complex way. The detection of these errors caused by side effects can be a time consuming task since it is very hard to predict if and where they might appear. Up to this day the detection of bugs and verification of system correctness has been a manual task using the visual debugger (this tool is described in chapter 3.1.1) and ad hoc tools such as small test programs written for the occasion, but a logical next step in the process of ensuring a quality product would be a software testing system that could detect certain types of errors before a release automatically. Such a system could test the system much more thoroughly than a human could ever do, both because it could run unattended for several hours as well as because it would never forget to test special cases that are known to easily cause trouble (this form of testing is usually called regression testing, see chapter 2.4.4). 1 see on page 39 for an update about Meqon, since this is based on information at the date we started our project at Meqon

11 1.1. Background The Meqon Middleware Games today are huge projects, which need large resources and a variety of expertise to complete. This is however only natural as the market for commercial games has grown the last years to before unseen levels. Demand for realism in games has grown as a reaction the growing capabilities of computers and gaming platforms and today the demand has gone beyond realistic 3d graphics into the field of dynamics; the realistic interaction and movement of objects. In a historic perspective objects in games have been quite static because of limitations in the computer resources and dynamics limited to very simple behavior of players and game critical objects. Mostly these dynamics systems created just enough realism for that particular game and didn t consider general physically correct behavior of objects. In an example game scenario the main character can freely move around a room, the programmer must ensure that the character seems to be walking on the floor without falling through it, as well as making sure it cannot walk through walls (collision detection). The character should also be able to jump, but not fly away which requires some form of gravitation. If this is all that is required of the game, an in house development of the physics system is a reasonable idea since it is quite simple to create. Games today however must go beyond these simple limitations and introduce objects like boxes, cars, doors etc that should all interact realistically and the dynamics system in the game must be able to handle all these interactions. It is still possible to write a dynamics system designed specifically for a particular game, but for the most part it pays off to buy an off the shelf solution from another company. Meqon fills this need with an advanced physics simulation system that is easy to integrate into games. The product can handle a range of scenarios from a simple character needing only collision detection and gravity, to very advanced physical simulations involving many objects with interconnections (cars and naturally acting hanging bridges are two examples). The game developers can concentrate on creating great games and don t need to worry about the advanced physical calculation required to handle these kind of simulations. The middleware has sprung from a final thesis conducted in 2001 and is based upon a rigid body simulation, which is a classic method of calculating a physically correct behavior of objects under the precondition that the objects are rigid and cannot be deformed. Rigid body dynamics is something many engineers come into contact with, and the main difference between this system and the calculations done in the classes teaching this technique is that the calculations are optimized for speed rather numerical exactness. The system can also handle a number of elements that bend the rules for what a rigid-body solver can achieve like cloth and ropes. Meqon also offers complete modules for simulation of cars, rag dolls etc.

12 4 Chapter 1. Introduction 1.2 Purpose and goals The purpose of this final thesis is to design and develop a suitable test system that can be used to test the Meqon middleware in an automated fashion. We aim to find a design for the system that is both flexible with portability and extendibility in mind as well as specifically designed to be suitable for testing the Meqon middleware. Standard patterns and file formats should be used where appropriate to minimize complications. The creation of test cases should be aided so that new test cases can be created quickly and with little effort, but the possibilities for what test cases can be created should not be limited by the test system. The implementation should be fully usable and easily integrated into the current development process at Meqon. To reach these goals, we need to answer the following questions: What kind of testing is suitable for the Meqon SDK? Where in the development process would a testing solution fit? How should a testing system be designed to maximize flexibility while maintaining usability? 1.3 Methodology This thesis consists of two phases, a research phase and a design/implementation phase. During the research phase we studied literature concerning automated software testing and the software testing process in general to understand the current state of software testing. The design process of the testing system started with a rudimentary system specification from Meqon. The description was then extended by qualitative interviews with the developers and the support team at Meqon, and a preliminary requirement specification was developed. During the implementation phase we used an extreme programminglike approach where design and implementation was closely integrated and a bottom up approach was taken to the design process. The reason behind a lazy design process in contrast to an eager one is that the actual requirements on this system were difficult to understand beforehand. We let the system design continuously evolve during the implementation phase with help of open discussions with Meqon and demonstrations of the current state of the system. It was very important that the system was developed to fit into the current development process at Meqon, and this is a major factor why we chose this iterative lazy design process. 1.4 Thesis overview This chapter is an introduction to the report. Chapter 2 gives an overview of the current state of software testing, we introduce theory and aspects of this

13 1.4. Thesis overview 5 field of science. Chapter 3 gives an overview of context where this system is going to be used, the attitudes and the current testing processes at Meqon. Chapter 4 describes the design of our system and some complications we met during implementation. An analysis of the results and conclusions (including future work) is given in chapter Introduction 2. Software testing 3. Needs and attitudes at Meqon 4. System overview 5. Analysis and conlusions

14 Chapter 2 Software testing This chapter introduces some concepts used in software testing as well as describing some testing techniques. 2.1 Testability In order to make a judgement of the quality of the test process one must be able to measure how much of the system has undergone testing. However the test coverage can be hard to determin for complex system, and this introduces the concept of testability. The IEEE Standard Glossary of Software Engineering Terminology [1] defines testability as: 1. the degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. 2. the degree to which a requirement as stated an terms that permit establishment of test criteria and performance of tests to determine whether those criteria have been met. In a paper [8] about testability Jeffrey Voas defines the testability of a software product as the prediction of the probability to find a certain fault with the help of testing. There are a number of standardized measurements of testability, among these is the McCabe score[8], which is calculated by counting certain program statements in the program code. The McCabe score is originally intended as a complexity measure, but is often used a testability predictor, since these properties have a clear correlation. 6

15 2.2. Reproducibility Reproducibility Testing software has the goal of uncovering previously unknown software errors, and by uncovering these faults the development team can improve the reliability of the product by removing them. But often it is not trivial to know if the change in the software solved the problem or not and the software needs to be retested to ensure an actual improvement. This is why reproducibility is important[7] when working with software testing. Reproducibility means that any fault found in the software can be reproduced by giving the program the same input once more, this is somewhat related to the concept of regression testing as well as determinism. Determinism in software is a way to ensure reproducibility of faults, in simple terms it means that for any given input the system will always produce the same output. At first it might seem that all software should be deterministic since all the involved calculations are deterministic, but this is often not the case since algorithms may depend on such unpredictable things like memory addresses or in the case with parallel algorithms the result might vary with the nature of the scheduler. Both these examples and many more problems might introduce great problems with determinism since they depend on factors outside the actual software system like the nature of the operating system or the structure of the free dynamic memory at the moment. It is therefore important to have this in mind when implementing a system that needs to be testable. 2.3 Model simulation testing In a paper by C. Michael Overstreet [6], the difference between testing of model simulations and regular software testing is discussed. He suggests that many of the aspects that make software testing difficult are especially common in model simulation testing, one important example is that it is difficult or impossible to define correctness of the simulation. A formalization of the simulation method might work as a correctness assessment aid, but this is often impractical since it can be both very hard and expensive to create such a formalization. Because of the difficulties to determine correctness manual techniques such as inspection can be applied on simulation testing. Test coverage is a way to measure the amount of the system that has been tested and can be used as a way to determine when to stop testing, however this technique can be misleading in model simulation testing since the behavior of the system is often very dependent on the order of which different parts of the program is executed and on the data used in the simulation [6].

16 8 Chapter 2. Software testing 2.4 General testing techniques This chapter describes a number of high level testing techniques, they are general approaches to the testing rather than a way to actually conducting the testing. They work as a basis when constructing test cases and they describe a general approach to the testing rather than methods Positive/Negative testing To test if a system is correct, you can either verify that the system lives up to all it s requirements (positive testing) or that it doesn t reach its expectations (negative testing) [9]. The two methods are basically the same since they booth fail if any faults are discovered, but a positive testing session can prove that the software is good enough whilst a negative test can only prove that a system is not good enough (hence if it does not discover any errors, the test does not prove anything). However, negative testing is a lot more common because of the simple fact that it is so much easier to do and generally provide a good enough confidence in the system. Test cases that use the positive technique are usually designed by analysis of the Requirement Specification document and can only be used if there is a complete specification for the system Black box vs. White box Depending on the knowledge of the system the test case can be designed with black box or white box technique. Once more the difference is not as great as it might seem. White box requires knowledge of the system and validates the internal aspects of the system and uses that knowledge to try to test weak points in the system. A system that uses Black box on the other hand tests the output of the system and without any knowledge of what the internal structure of the system looks like. Most testing uses a combination of black box and white box since both techniques have their strong points and these test are often called grey box tests. The difference between a white box test case and a black box test case can be hard to understand or detect when the test case is written and ready to use since the difference lies in how the test case is designed and what is tested rather than how the finished test case is working Test partitioning When testing a function it is not realistic to test all possible input values, it would simply take too long (and in some cases too long might even mean several thousands of years or more). Instead the test designer has to find a subset of values that covers up all subclasses of input values that would be treated the same by the system. If there is a fault in the function, the idea of test partitioning is that that fault will be general to the whole subclass of

17 2.5. Test phases 9 the function and thus will be discovered when the function is partitioned and all subclasses are tested. An example would be a function that consists of different subfunctions. The test case could then test a few values within each subfunction instead of randomly selecting input values. Test partitioning is a typical white box technique since it is generally impossible to know how the function is partitioned without knowledge of the implementation of it. This technique is very powerful, but in practice it is often very hard to do the partitioning of the functions. And studies have shown that a random approach to finding input values might even prove more useful [3] Regression testing In contrast to many other forms of testing, regression tests are not derived from specifications or customer demands. Instead, regression tests are conducted to prove that old bugs and errors don t find their way back into the system after they have been discovered and fixed. Normally the output from a regression test is compared with the output from a previous version. Regression testing can be conducted in many different ways, either through a similar mechanism to unit tests with small code snippets that can run automatically and be verified without human intervention. Another way is by gathering information about previous errors and manually testing the system with these as a basis manually before a release. 2.5 Test phases John Watkins presents a model for software design and testing called the V- model [9]. For each phase in the development process there is a corresponding test phase. The V-model makes it possible for the test team to give input on the testability of the design in an early stage. Figure 2.1 shows the main idea behind the v-model where each development phase is coupled with a testing phase Unit testing A unit test divides the program code into units (atoms) of functionality and tests each unit using both black box and white box testing techniques. A perfect unit would be a function or procedure that does not depend on any other functions in the system. If such a unit fails, the test leader knows that the functionality that has broken must be in that function. However, most functions depends on and uses other functions, and because of this all other units the tested unit interact with have to be either replaced by dummy units that behave correctly for the inputs that it will receive during testing or units that have already verified to be correct. The unit test is usually conducted and written by the developer who wrote the functionality contained within

18 10 Chapter 2. Software testing Figure 2.1: The v-model the unit. There are many different frameworks available for developers to use for unit testing, many of them are open source and can easily be adopted for use in a variety systems. Unit testing is a central concept in extreme programming, where the developer creates the unit test before creating the actual implementation, and has gained popularity because of it s simplicity. When the code passes all unit tests it is considered reliable (at least at a unit level of the system). Even if unit tests have gained popularity and they do not test everything and more comprehensive tests are usually needed to complement the unit tests Integration testing Integration testing is an extension to unit testing. An integration test unit is a test between two or more units. This is an important idea since even if all units are tested and working, interaction between them can be flawed. Integration testing is generally conducted with a black box approach, with the design specifications as a basis System testing The system test is based on the specifications of the system. It is a test conducted late in the development cycle to assure that the system will pass it s acceptance test. This high level test should be conducted using black box techniques and consideration should not be taken to the underlaying implementation details of the system.

19 2.6. Design Acceptance test Acceptance tests are based on the customer s requirements on the system. This is the only testing phase where interaction with user representatives is needed to create good tests. There is a lot of similarity with the system test phase, since this also is a high level test of the complete system. The difference is that the acceptance test uses a customer approach and should test aspects like usability and documentation. 2.6 Design Different design phases require different approaches to how the testing of that phase is conducted. Unit testing and integration testing are easily scripted and therefore often composed of multiple test cases in some sort of testing suite. Normally there is a test framework that the test case developer can use for the administrative test functionality. However, more high level testing phases are more likely to be conducted manually (some parts might be scripted with the help of tools to simulate user interaction) Testing frameworks The choice of testing framework depends on a number of things such as what computer language the system is written in, and what kind of system it is. There exist a number of solutions for handling test cases that can be adapted for different systems, but special requirements in the testing process might render them useless. In many test case frameworks (like the object oriented framework cppunit), test cases are classes extending a test case class. This way, the process of writing test cases is very straight forward and easy for developers to understand. However there are a number of issues to this approach as well, one is that multiple inheritance might be necessary and this creates since some languages like java don t allow multiple inheritance. 2.7 Suitable testing for the Meqon SDK There is a a jungle of ideas on how and when to test, some believe in a analytic approach (unit testing, positive testing, test partitioning) and try to prove reliability in one way or another while other (negative testing, integration testing) take a more relaxed approach and try to give a good enough confidence in the system. Since we can t know exactly how the system should behave we must use a negative testing approach. We see the urge for simplicity in our system and thus we will allow more complex test cases than simple unit tests (fewer test cases will be easier to maintain and develop), this leads us to choose a high level testing approach. This is also good since it is quite difficult to find true units in a a system like the Meqon SDK. Furthermore, we have found

20 12 Chapter 2. Software testing the regression testing model to be a nice model for the Meqon SDK, since it is difficult to know what is correct in a model simulation system, and thus a combination of manual examination of values and an automatic regression testing of future testing outcomes creates an easy way to inspire confidence in parts of the system. For the regression model to work nicely we will let the user interaction part of the testing system display output from the test cases so the user can set up constraints on these so that the system can analyze future test runs automatically based on those constraints. Since meqon is a quite small company with few resources to put into on testing we think that a model of running the tests after the automatic build process each night would be a good way of getting a quick idea of the condition of the system without much effort. This would be comparable to what you can do with an unit test system.

21 Chapter 3 Meqon Research AB 3.1 Existing tools at Meqon Visual debugger This tool gives a visual representation of a simulation in a running program using the Meqon SDK and offers the user the ability to apply forces to individual objects (drag them around) while the simulation is running. The tool can easily be used with existing programs that use the Meqon SDK by having the program connect to the visual debugger at start up, the rest of the communication with the visual debugger is done automatically. There is support for connecting to the visual debugger from all platforms that are supported by the Meqon SDK, which enables debugging on gaming platforms with the visual debugger conveniently running on a regular windows desktop. The visual debugger was developed as a part of a final thesis written by Anders Stenberg who is currently an employee at Meqon (the report was at writing date still a work in progress). The tool is available to developers using the Meqon SDK and is also frequently used internally at Meqon. It is a very important tool and has redefined the way developers could interact with the programs using the Meqon SDK. The debugger tool consists of a single window that show a 3d-view of objects in the inspected simulation as well as visually annotating interactions (forces, constraints, joints etc.) between the objects when desired. In addition to the 3d-view there is also a data view, which shows the values of all object properties and modules in the system for each step in the interaction. As a complement to the debugging capabilities of the visual debugger, it also gathers profiling information which can be used to optimize the performance of programs. See figure

22 14 Chapter 3. Meqon Research AB Figure 3.1: The visual debugger inspecting a simple physics simulation

23 3.2. Testing at Meqon Demo framework The demo framework is a set of demo programs that are shipped with the Meqon SDK with source code included. This framework is intended mainly as an example to developers and as a demo of what the system is capable of. Since it contains the basic codes for setting up the physics engine, it is often used as a basis for creating custom test cases quickly since it is often faster to modify existing code to fit your needs than write everything from scratch. It consists of a visualization system quite similar to the visual debugger (but without the debugging facilities) to visualize and interact with the simulations as well as a number of demos using the framework. The demos can act as a testing tool unmodified as well since they are since they offer a fairly complete set of scenarios where the developers can get a general idea if the system is in good health overall or if something is broken (in short: if the demo framework seem to work ok, games using the SDK will probably be ok) Bug tracker Whenever a bug is discovered that cannot be resolved straight away, it is entered into the bug tracking system. This is a guarantee that bugs don t get forgotten, as well as insurance that when a release is made all known bugs have been dealt with in one way or another. This is not a testing tool, but a very important component in the quality assurance process, it is also a way for developers to communicate the status of the system with the other developers so that everyone has an idea what parts area ready for shipping and what parts need more work. 3.2 Testing at Meqon The testing process at Meqon consists partly of individual developers testing the code that they are currently working on. This is a very natural way of testing code since the process of writing code mostly involves testing that what you write works as you intended, and this process will probable always be necessary and a source of newly discovered bugs since the code changed or added is the most likely place to find new bugs. Because of the complexity of the product it is necessary to complement this type of testing with a more intense and thorough testing before each release, which today is done by hand or with the help of ad hoc scripts and programs to test certain features that are known to be problematic. This is where the process is lacking, a tool that supports the team in this testing effort could prove very valuable as the complexity of the product growths. Our project aims to fill this void. Much of the testing today is conducted using the demo framework since it demonstrates many parts of the system with quite complex graphical demos. Since

24 16 Chapter 3. Meqon Research AB the demo framework is not designed for testing, and certainly not designed to try to break the system on weak points and it will never be as good as a specially designed testing tool could be. Another main reason to use a special testing system is that by automating the testing process, more rigorous testing than would ever be possible manually will be possible.

25 Chapter 4 System overview 4.1 Testing technique When Meqon Research AB gave us the assignment to make an automatic testing system they asked for something that would be easy to integrate with the current system developement process. After talking to the developement and support team we realized it also had to be time efficient for the developers to use, or it would end up outdated as no one is interested to continue the effort of updating it as the SDK evolves. We decided to create a framework that could handle multiple types of tests ranging from unit tests to more complex high level tests. Another idea we had was to let the tests behave unsupervised as long as they passed the acceptance criteria but when a constraint failed a developer would be able to manually debug the test in the visual debugger. One problem with unit testing an SDK like the meqon physics middleware is that the systems behavior is a result of a very complex interaction between various parts of the system. It is very hard to test one part without testing others. Therefore, good units would be very difficult to find if one tried to use strict unit testing, and interaction problems would not be discovered with a unit test approach. Instead we take a high level testing approach, system tests using a testing technique similar to regression testing. The test cases are created as scenarios, and measured once to get output that can be analyzed. Acceptance criteria is fitted manually to the output, and later tests are compared to these criteria. 4.2 Overview The base system is separated into two parts, an analysis tool and a framework for writing test cases. The test case framework is a set of classes in C++, which the developer of new test cases uses to create test cases. The analysis tool is used to set up a collection of test cases, tune their parameters and au- 17

26 18 Chapter 4. System overview tomatically compile and run them. Support for multiple platforms is included (only Win32 and XBOX is supported to date as a proof of this concept). See figure see Test cases and the test case framework Each test case is a small independent program, written with the help of the test case framework. The program generated from a test case written with the test case framework has two usages; the first one is to generate a specification file that is used to tell the analysis program about the parameters and properties of the test case. The second usage is the actual testing. The method of selecting which mode the program will run in is very simple, if the program can find a special input file (see 4.3) it will use the file and start a test case simulation, otherwise it will write a test case definition file and exit. The reason we chose this method instead of writing a separate definition alongside the program is that it dramatically increases the simplicity of writing test cases and the room for error of writing one thing in the test case and another in the definition file. The fact that the developer does not need to learn and understand the format of the definition file is also a great advantage. The test case part of the program is more interesting, it is basically a program that uses the Meqon SDK and tries to detect errors in the behavior of the scenario in the test case. At start up a set of objects are set up and manipulated with the help of the Meqon SDK and input parameters can control parameters in this scenario. The user defines a set of values for each parameter in the analysis tool. These values are imported into the test case at start up from the input file mentioned earlier and assigned to the parameters. The actual scenario is then instantiated one time for each unique way to assign the values to the parameters. If for instance a test case has one parameter controlling the time step of the simulation and one that controls the number of objects involved in the test scenario, and the parameters are assigned 2 values each from the analysis tool, the scenario will be instantiated 4 times (2*2). Every such instantiation will from now on be called an iteration. For all iterations in the test case, everything is set up according to the values of the parameters and the simulation is run until the test case is finished (how to determine if an iteration is finished differs from test case to test case). At the end of the iteration, one value for each output parameter is recorded. An output parameter is simply a measurement of the test case, usually it should measure something that has a great correlation to errors but can also be used to measure performance values. These values are the values that are used to determine if a test case has failed and to choose what to measure is therefore a very sensitive task in the test case development process. When all iterations of the test case are finished, the recorded values are written to file alongside the values of the input parameters for easy matching between the two.

27 4.3. Test cases and the test case framework Implementation The test case framework is an object-oriented framework written in C++, just like the Meqon SDK. This is to avoid limiting the developers of test cases to limitations in the language. C++ is also the industry de facto standard for writing programs for gaming platforms, and thus most tools and utilities are designed around this language. Negative aspects with this choice are the lack of advanced memory management like garbage collection and advanced standard cross platform libraries (C++ does have a cross platform library, but not a high level one like Java or C#/.Net). However, the Meqon SDK include networking and advanced storage classes that enabled us to write cross platform code that is guaranteed to work wherever the Meqon SDK is supported Usage The main class in the framework is the TestCase class, this class includes the basic structure for a test case including handling of iterations, input and output parameters etc. It enables very fast development of new test cases, which are classes that inherit from this class. The TestCase class, as well as the test cases themselves, uses a number of classes that simplifies the handling of parameters, static scenes etc. To develop a new test case, TestCase is extended and these members must be implemented: prepareparameters(), in this member, the strings containing information about the test case are set. 1. name, a short name for the test case 2. description, a longer description of the test case 3. uniqueid, a short and unique tag for the test case The input and output parameters are also created here by using the functions createvalue, and registermeasure. Values can be created entirely on the fly by simply using this function and later retrieving the values by using the function retrievevalue. Measurers are a little more complex, they require that a measure-object exists. Any object from a class extending the class Measurer will do, however a more simple approach is to use the class RealMeasure that is a simple and generic measure class that can store simple floating point values. simulateonetimestep(), in its most simple form, this member should only contain a call to the SDK s simulate-function. isfinished(), this member should return true when one iteration of the test case simulation is finished. Failure to do this will leave the test case in a never ending loop.

28 20 Chapter 4. System overview Besides these mandatory members, there are a number of members that can and will help the developer in the creation of a test case. These include: beforesimulation(), this member will always run before one iteration is started. A good place to reset certain values. betweensimulation(), this member is run after each simulation time step. A good place to update counters etc. aftersimulation(), this member is run after isfinished has returned true and the iteration is finished. This is a good place to do measurements etc. setscene(), here you can set a scene object to the scene field of the test case class. A scene object represents a static scene that can be common to many test cases. There are also a number of members in the TestCase class that are useful, these include functions like createbox and createsphere to simplify creation of standard objects. Exactly what the test case does is totally up to the test case developer. The system of using a native C++ class framework enables a wide variety of test cases since the only real restriction of what you can do is the restrictions of the SDK in it self. The test developer could create unit test like test cases by creating very simple test cases that test individual functions in the SDK as well as high level test scenarios where many objects interact in a way that is suitable for deetecting certain errors. Performance testing is also possible, even if it is not the main objective with the test system Problems The biggest problem we encountered was that the Meqon SDK was non deterministic in some situations. The non-deterministic behavior caused the simulation to behave differently even if the input parameters where exactly the same and caused problems when inspection of certain failed samples did not reveal the same result as recorded by the test run. This has to do with the fact that the layout of the memory space in the test case program is dependant of the number of iterations of the simulation that has run etc. To solve this, we created our own very simple memory allocator that was very deterministic in its behavior. This did solve the program, but the solution broke in version 1.5 of the SDK so we stopped using it at that point. The reason why we chose not to find a better solution was that the non-deterministic behavior stopped at that version upgrade as well, and this has not been an issue since then. We are not entirely confident that the problem is completely fixed, and this remains an issue for further inspection.

29 4.4. Analysis tool Analysis tool Figure 4.1: The analysis tool The analysis tool is designed to be as independent from the Meqon SDK as possible, while still tailor made to suite it. It is a stand-alone program designed to gather data from and to set acceptance criteria s for test cases. As the Meqon SDK evolves the API will probably change which is why we created a design that would allow us to switch physics SDK without changing the analysis part of the system more than necessary (some changes are still needed, mainly changes to the compilation modules). A screenshot of the program is provided in figure Design One of the requirements of the program was that it should be able to run both with a graphical user interface and from a command line. To satisfy this requirement we structured the program in two layers, one layer for functionality and the other for user interface. We solved the communication between the layers by making the user interface register event handlers in the functionality layer. By designing the program this way we could reuse all code with functionality in both the windowed and the command line application. The command line application is basically the same thing as the GUI program, except that it never instantiates the GUI parts. The command line program is

30 22 Chapter 4. System overview Figure 4.2: The main structure of the analysis tool. Figure 4.3: The bridge pattern. very simple and is meant to be used as apart of a build process to automatically test the current test suite, no user interaction is possible except from a few command line parameters. All target platforms require different compilers settings. We had to take that in consideration when we designed the part of the program that compiled the test cases. We settled with a solution where we used a bridge design pattern [2] to plug in functionality for compiling, running and analyzing data from different platforms. In reality this means that we only have to extend a super class with specific functionality and register the possible user variables in the option manager if we want to add support for another platform in the future. Figure 4.3 is a graph over how the bridge pattern is structured. Note that we have not followed this model blindly because of practical issues. Test cases can be imported for each available platform and saved in a test suite. Test suites are stored using the built in XML serialization in.net. The

31 4.4. Analysis tool 23 values for Input parameters are set through the analysis tool either as a single value or a range of desired values. When a test case is run and completed it saves all output parameters to an XML file, which is then moved to a directory containing all data for a test suite. Test data is only loaded into memory when it s used since the amount of test data will grow over time. By saving old test data the user can go back in time and compare test results to discover possible errors or improvements. The user has the ability to either visualize the data in a graph or a data grid and it s possible to filter the data in several ways. For each test case and platform the user can set an acceptance criteria to define accepted values and raise warnings if the test case is accepted but the results are worse than the previous test. In many other automatic testing systems the acceptance the criteria are placed in the test cases by default but since we re using the analysis tool as a way to understand how the test case behaves and what is indicates errors we felt that the functionality should be placed there so that the user can set different criteria on the same test on different platforms or in different situations. It is very hard to determine the criteria for what should be accepted and not before you can analyze the data from the test case, it is therefore very natural to have this functionality in the analysis tool. With a press of a button, the user can show only which samples have failed the acceptance criteria currently set, these samples can then be run once again and inspected more carefully with the help of the visual debugger. Results from test cases are stored into reports that can be viewed either in the analysis tool or an external XML reader like Microsoft Explorer. For each failed test case are stored exactly what constraints that have failed and they can be viewed and debugged through the analysis tool and the visual debugger Implementation The program is written in the.net framework using C#, the reason behind this choice is it s simplicity and that it is very fitting for a lightweight graphical program. We used multiple free open source.net components to save development time and to add functionality to the program that we wouldn t have had time to do ourselves. Most important of the components is a graph component which lets us visualize the data from the test cases Problems The first problem we encountered was that the analysis tool and our c++ framework calculated floating points differently, which is very unfortunate since it is important to be able to rerun tests for debugging purposes. We solved this by calculating all the values in the analysis tool and then sending them to the test cases. This solution forced us to implement at method of

32 24 Chapter 4. System overview sending data to and get data from a test case. The solution is a set of XML files that are passed between the analysis tool and the test case. Another problem concerning communication is the need for real time feedback. On the Windows platform we could easily use the standard out character stream from the test cases to give feed back to the analysis tool but that solution wasn t possible for XBOX (there is a way to send strings similar to how the standard out stream works is windows, but it turned out to give us a lot of problems) so we had to come up with another idea. In the end we created a small server in the analysis tool the test case connected to over a standard network socket, this gave us the possibility to give run time feedback from the test cases on the XBOX and on Windows in a similar fashion, which is desirable since it is cumbersome to handle this differently on different platforms. The serialization caused lots of problems since it required us to design the internal data representation according to the structure of our XML format. Even worse was that we couldn t use our xml files after we added new features since the serializer crashed when we tried to read the old xml files. Serialization looks like an easy and time efficient solution for saving application data at a first glance but as the project grew we realized that we should have made our own functionality that supported the addition of new xml tags from the start. When we first started to use external.net components we thought it would save us several hours of programming and even though it did save us from a lot of unnecessary programming we had to put in a lot of work to get them to work as intendd. A common problem with open source code is that it might not be production ready and if you don t use the code exactly as intended it might behave strangely. We had to debug most of the open source components we used since they caused a lot of problem with the graphical user interface and even caused crashes User guide This section describes how a new user may install and set up the analysis tool on a computer. To be able to use the analysis tool the user has to have Visual Studio installed on the computer since the analysis tool uses Visual Studio for compiling. The Meqon SDK has to be compiled for all the target platforms and if the user wants to be able to run test cases on the XBOX console the XBOX SDK from Microsoft has to be installed as well. When the user starts the program he or she will be asked to set the options. The options include paths to visual studio, meqon and xbox include files and library files. When all options are set it s time to import test cases. In the Solution Explorer floater on the right side of the screen the user can chose to import test cases for the Win32 or XBOX platform. All test cases can be imported to both platforms. Locate the cpp-file containing the test case and the analysis tool

33 4.5. Test Cases and results 25 will compile it, create a definition file and create an entry in the analysis tool for the test case. After the test case is imported the user has to define all input parameters before the test case can be run and tested. There are currently three different ways to set input parameters, value ranges, fixed values and enumerates. A test case is run by selecting it in the Solution Explorer and the pressing the Run button. The test case will run in the background and a progress bar will show it s current status. By double clicking the test case in the Solution Explorer a tab will open in the workspace window where the user can create graphs or show the test case data in a data grid. The analysis tool currently supports three different graphs, two dimensional graph, bar charts and finally a three dimensional graph where the z-axis is represented by a color. There are many ways to filter the test case data. In the data grid the user can filter the data by values and in the graphs it s possible to zoom and pane. In the graphs it s possible to compare multiple test runs to make it easier to compare data. With the help of the graphs and data grid the user can set an acceptance criteria for a test case. The criteria can either be set as a fixed value constraint where the test data has to be a subset of the criteria or as a regression test where the data is compared with older test data. It s possible for the user to only view failed samples in the graphs. By right clicking a test sample in a graph or data grid the user can debug a single sample in the Visual Debugger, which must be manually started. When all test cases have been set up in the test suite the user can create a report, all test cases are then run with their acceptance criteria s and the result is saved in a report that can be viewed from inside the program or using Microsoft Explorer, all failed samples are shown together with a description of the broken constraints. The analysis tool can be run from command line which makes it possible to use it as the last part of an automated build script. 4.5 Test Cases and results In this chapter we give an overview of the test cases we have created. This serves both as a documentation of these test cases and as way of giving an idea of the flexibility of the test case framework and what you can do with it Stacking test case Test scenario Stacking shapes in rigid body systems is a complicated task since many objects interact with each other at the same time and the interactions are propagated downward in the stack. It is however fairly common in games to stack objects, and therefore it makes sense to handle this gracefully. This test case

34 26 Chapter 4. System overview Figure 4.4: The stacking test case seen through the visual debugger. sets up a number of boxes and runs a certain amount of time and checks if the stack has fallen or not. This is one of the first test cases we developed and is a good example of a small and simple test case. See figure 4.4 Input parameters TimeStep, the time step of with the simulation is carried out. MaxIter, the maximum iteration setting of the engine. StackHeight, number of boxes stacked upon each other. Output parameters FallenMeasure, 1 if the stack has fallen, 0 otherwise. Acceptance Criteria The acceptance criteria for this testcase is somewhat complex. One could imagine a composite acceptance criteria where you check for fallen stacks under a certain threshold of MaxIter and StackHeight. This is a typical test case where the acceptance criteria is hard to determine beforehand.

35 4.5. Test Cases and results 27 Conclusion A natural assumption is that a smaller time step and a higher value of maximum iterations would yield better results, and tests with this test case confirms this. However, we had some problem with deactivation of objects (objects taken out of simulation because they are moving slowly). Turning this feature off easily solved this problem, but then we got another problem concerning how long the simulation should proceed. When deactivation was turned on, we simply ran the simulation until all objects were disabled, but now we have to either set a fixed time or manually detect when objects are in a static position. The latter will introduce similar problems as having deactivation on, so we decided to take the first approach. We ran some simple tests to decide upon a suitable time that allows most instable stacks to fall and used that, this gives nice and usable results Spring stability test case Test scenario Springs have always been a possible source of instability in simulation systems. In the Meqon SDK, care has been taken to avoid most of this instability through various measures, but there are still situations when instability can arise. This test case tries to provoke these situations by connecting multiple springs and see if the result is stable or not. Input parameters Numberofsprings, number of springs chained together. Springconstant, the spring constant of the springs. Springdampening, a dampening factor of the springs (increasing this value is a traditional way of avoiding instability) Timestep, the time step of with the simulation is carried out. Output parameters Stability, 1 if the system is stable, 0 otherwise. Acceptance Criteria This test case is quite similar to the stacking test case in many ways. And a similar approach can be taken when creating the acceptance criteria.

36 28 Chapter 4. System overview Conclusion It is fairly easy to provoke instability in this kind of set up with interconnected springs. When using the stable option for springs in the engine on the other hand we never saw any instability, but to the cost of a very aggressive dampening Vehicle test case Test scenario The vehicle module is a module that enables the creation and usage of vehicles with a lower performance penalty than one would get if a vehicle would be created with regular rigid body elements. This is a test to see if the car module is in good shape by creating a vehicle and letting it run a certain length of time. It shouldn t deviate too much from a straight line. Input parameters Timestep, the time step of with the simulation is carried out. Maxiter, the maximum iteration setting of the engine. Wheeltype, Type of wheel (0 simple wheel model, 1 advanced wheel model). Output parameters Carstraight, tells how many units the vehicle has deviated from a straight line. Carfar, tells how far the car has moved (this should not, but will change somewhat when time step changes). Carpen, this is a measurement of how much the wheels penetrate the ground, ideally this should always be 0. Acceptance criteria This test case can have several disjoint acceptance criteria. One based on carstraight, on on carfar and one on carpen. Again, it is important to make a test run of the test case to see what normal values are and create acceptance criteria based upon those values.

37 4.5. Test Cases and results 29 Conclusion The vehicle module is a complex module, there are many parameters that can be tweaked and one vehicle can look very different from another. Taking all parameters into account would yield a very complex test case and would not be practical; therefore we chose to limit the test case to a certain type of vehicle set up. This test gives pretty expected results, smaller time step gives a smaller error in the direction of the car, however it is hard to know how general this test case is or if it only tests the performance of the type of vehicle used in the test. We left the test case there for future improvements, knowing that at least it tests a number of parts of the vehicle module in one special case Tunneling test case Test scenario Two simple geometries represented as rigid bodies get thrown into each other from different angles and they should always collide and bounce away. When that doesn t happen something is wrong. Input parameters Timestep, the time step of with the simulation is carried out. Speed, the speed of the objects in m/s. Rotation, objects rotation. Objects, the objects colliding. box-box, sphere-sphere, box-sphere etc. Angle, the angle between the objects. Output parameters Direction diff, direction difference before collision and after collision. (radians) Speed diff, Speed difference before collision and after collision Acceptance Criteria Here we used a regression test technique to raise a warning if the result differs too much from the reference data. Both speed and direction have to differ after a collision, and speed should not be 0.

38 30 Chapter 4. System overview Conclusion How do you measure a good collision? Depending on the technique used for collision detection and handling the outcome will be different. Small adjustments in code can cause big differences on the screen and what we want for a computer game is a collision handling that looks realistic. We concluded that it was easier to spot errors by comparing tests with older verified tests like in regression testing than to look at the data and set constraints Contact point and normal direction Test scenario When two geometries collide, make sure all contact points are located exactly between the two bodies and make sure the normal is pointing in the same direction as closest distance between objects. Input parameters Objects, the objects to collide Output parameters Point, How far away from the middle the contact point is located Direction, how much the directions differ Acceptance criteria A simple acceptance criteria is to check if the point is located in the middle and that the direction does not differ too much. Conclusion All objects didn t pass the test but by debugging the failed samples in the visual debugger the support team was able to see exactly where the contact points where placed Proximity detection Test scenario Setup massive scene with thousands of objects. Compare output from DefaultProximityDetecter with output from AABBProximityDetector. Results must be the same.

39 4.5. Test Cases and results 31 Input parameters cycles, number of cycles to run the test Output parameters failed, number of objects that don t match. Acceptance criteria A simple acceptance criteria is to check for failed objects, ideally there should zero failed objects. Conclusion This test is very hard to debug since there are thousands of objects that make it hard to see the difference when comparing two test runs. The test succeeds after the first cycle but then it starts to differ, this might be because the objects are ordered in another way in the engine that makes the solver solve them in another order. We haven t been able to verify why this test fails Activation and deactivation test case Test Scenario Check that objects fall asleep correctly and wake up when forces are applied or when moved inside another object. Input parameters objects, enumerate with all objects Output parameters success, true or false depending on the success of the test case Acceptance criteria All tests should be a success. Conclusion This test case only shows success or failure which makes it less interesting than many other test cases. Visual debugger makes it easy to debug this test case and it s fun to look at.

40 32 Chapter 4. System overview Raycasting test case Test scenario Place objects and raytrance an image with the raycast function. This is an example of how flexible the system is and that the output parameters can be used in a number of creative ways. Input parameters x, pixel on the x-axis y, pixel on the y-axis Output parameters light, light intensity of a pixel (this is calculated with a traditional phong lighting model) Acceptance criteria We used a regression test technique and saved data from one verified test run to compare with. Conclusion The image isn t very well represented in the analysis tool and it wouldn t be possible to discover small errors by looking at it, but the regression functionality makes it very easy for the computer to verify an image. The debugger can t be used to debug this test case but by looking at the image it s possible to determine what geometry that is bugged for ray casts. The lighting model used takes many parameters into account, the location of the ray intersection, and the tilt of that surface relative to both the camera and the light. This enables us to have one parameter that can detect errors in many parameters at the same time Detector speed test Test scenario This test case is used to measure how long it takes to detect collision between different types of objects. Input parameters objects, enumerate of objects

41 4.5. Test Cases and results 33 Output parameters time, time it takes for the SDK to detect a collision Acceptance criteria We decided to make this a regression test and save data from a test run to compare with. Conclusion Collision detection is very fast but we can sometimes see spikes in the time it takes. We think this is caused by task switching and similar issues and not a bug in the SDK. This test is very interesting for the customers since it clearly shows the advantage of using basic geometries like spheres and boxes instead of complex convex geometries Vehicle test case up close To give a better understanding of how test cases fit into the analysis tool and how to get meaningful results from the test cases, this chapter will examine the vehicle test case in detail. Getting a grip on the interface Figure 4.5 shows the analysis tool as it looks when a test case has been imported into system. To the right in the window we see all currently imported test cases, in this case we can see vehicle test case is imported under the Win32 folder which means it has been imported for testing under the Win32 platform. For each test case there can be several variables, in the case of the vehicle test case there are three variables that can be altered: timestep, maxiter and wheeltype. To set these values we simply right click the variable name and choose an appropriate action. Lets start with the wheeltype, this variable is an enumeration variable which means it has preset values to choose from. These preset values are defined in the test case source code itself and cannot be changed from the analysis tool. Figure 4.6 shows the dialog box where we can choose what enumeration values the test case should use when testing. The two other variables are regular free value variables, for these variables we can either specify a single value to test on or a range of values 4.7. Moving along to the testing and analysis When all variables have defined values we have can start with the actual testing. When testing we will move through all possible combinations of values that the variables of the test case can take. And for every combination the test case will record a number of values that are measurements of the test

42 34 Chapter 4. System overview Figure 4.5: The analysis tool with one imported test case

43 4.5. Test Cases and results 35 Figure 4.6: Enumeration type variable wheeltype Figure 4.7: Variables can be defined in a couple of different ways

44 36 Chapter 4. System overview Figure 4.8: A graph from the vehicle test case case outcome. The test run is started from the analysis tool itself and can now run unattended until all combinations are complete When the test run is complete, the user can inspect the outcome by creating graphs from variables or recorded values within the test case. Figure 4.8 shows a graph that has the timestep as x-axis and carstraight as y-axis. The timestep determines how exactly the simulation is carried out and the carstraight is a measurement of how much the car that is simulated by the test case deviates from a straight line (it is supposed to run in a completely straight line). We can see that when the timestep goes up the car deviates more, which is expected. There are also a range of values that are very close to zero for all values on the x-axis, these come from the simple wheeltype which is behaves more stable even with a very high time step of the simulation. How to detect errors in the data differs from test case to test case, but in the case of the vehicle test case a good way is to find values of carstraight that deviates a lot from the neighboring values. These samples can then be examined by starting the vistual debugger and then right clicking the value and clicking examine this value.