Automated Testing Environment

Transcription

1 Automated Testing Environment Concepts required for testing embedded systems adopted in this course (quizzes, assignments and laboratories) 1 To be tackled today Why test, and what kinds of tests are there? Difference between TLD Test Last Development TDD Test Driven Development (Test First Development) How do you add tests? More details on the testing process E-TDD and the testing toll E-UNIT Other kinds of available tests time and access Design of custom TESTs and ASSERTS 2/ 28 Why test? Unit test I AM CURRENTLY BUILDING A NEW FUNCTION Does it work the way I expect? Testing my and my partner s understanding of the Design WE ARE DESIGNING A NEW PRODUCT -- If we can t explain (to ourselves via a test) the product ideas how do we know when product is working? Regression testing MY PARTNER ADDED A NEW FEATURE to a product how do I know it did not break something I wrote? System testing How do I PROVE TO THE CUSTOMER that the product (Lab. or assignment) works the way that was requested? What are the problems with this sort of Human intervention test? This TEST code would (confusingly) be placed in Assignments1 directory with the rest of the customer code 3/ 28 4/ 28

2 VisualDSP development environment (IDDE) Communication overhead VisualDSP++ program Debugging Interface HPUSB-ICE Debugging Interface JTAG boundary scan operation Processor Peripherals Lab. station (ENA301 / 305) BF533 Evaluation Board Each time we want to send a message (via printf( ) or cout) we must pause the processor from doing what we want it to do, and allow sending of messages over the interface Slow, plus the testing can impact on peripheral performance. E.g. no sound while doing (debug) printing in Assignment 2 and Lab. 1 10/1/2013 THIS PROBLEM Concept IS of Test COMMON Driven Development TO applied ALL to Embedded EMBEDDED Systems SYSTEM 5/ 28 DEVELOPMENT; NOT JUST THE BLACKFIN USED IN THE LAB. More issues with this sort of test! Tests are mixed up with production code, in many places Must remove tests before release to customer Unreliable when done manually may result in test messages at wrong time in released code (confusing to user) Difficult to add and remove tests from production code without introducing new errors Results must be checked manually therefore unlikely to have tests run often during development leads to unreliable product 6/ 28 E-TDD / E-UNIT Tool Test Environment E-TDD embedded test driven development Build-the-tests-first (during design) ideas for product development has been made popular by the Agile approach Requires an automated testing approach (test framework) In this class, we are going to use the E-Unit tool for its automated behaviour Will use in both test last development (what most people currently do) and in test first development (trying to encourage). I (the customer) will provide tests so that you can check that you have developed the right stuff at the high level You will build additional tests to check your own code at the low level 7/ 28 Can download LabTestEnvironment.zip file from the web (Check September webpages) LabX directory is where you keep your customer-specific code LabXLibrary is where you keep your utility s code. Re-use in other labs LabXTests is where you keep the tests for the customer specific code and for the Library code you will reuse Refactoring You may have to add additional tests, and move things between Library and Customer 8/ 28

3 Assign1 directory Customer code Assign1 code uses many library calls ReadSwitches( ) and WriteLED( ) Use Open-CCES-ame plug-in learned in Lab0 Use Open-CCES-ame plug-in to add in paths to Assign1Library.h and Assign1Library.dlb Assign1 code may just be a file with a small main( ) I developed code in Assign1 directory and then refactored into Assign1Library Build a new CCES project Assign1Tests and place in Assign1Tests directory I developed TESTs as I added new code to Assign1 and Assign1Library Certain standard files in a TEST project Same set of files for all TESTs in assignments and laboratory Use the automated E-UNIT HELPER features of the OPEN-CCES-ame plugin to convert project to TEST project 9/ / 28 Testing your (re-usable) code stored in the Assign1Library directory with E-UNIT HELPER Open-CESS-ame plug-in has automatically added the Tests_main.cpp Don t move or copy the files into the test directory End up with two different versions Add Path to Assign1Library.h header file by clicking on Assign1Library directory (NOT src or the specific.h file) Add Path to Assign1Library.dlb library (or archive) file in Assign1Library directory debug (Click on desired.dlb) 11 / / 28

4 Open-CCES-ame has automatically added a TestGroup file for you Modify Assign1Tests.cpp to perform the needed testing 13 / / 28 Tests and Test Groups have a standard format Tests then need to be customized Can be automated by Open-CCESS-ame TESTGROUP NAME TEST NAME Add further tests at this location TESTGROUP NAME, 15 / / 28

5 Test Components Remember format for exams and quizzes The ACTUAL Test CHECK(result1 == result2) If an error occurred what was result1 being used for? Error result1!= result2 We would get better test error messages if we self-documented the code int resultcppcode = ; int resultasmcode = CHECK(resultCPPCode == resultasmcode); Error resultcppcode!= resultasmcode Test CONTROL statements TEST_CONTROL( testgroup_name) TEST_FILE_RUN_NOTIFICATION( testgroup_name) 17 / 28 Add stub versions of functions into Assign1Library and Build library Note code DEFECT There is NO return value placed in R0 Assign1CPP.cpp Assign1ASM.asm 18 / 28 Now build Assign1Tests source files to make executable Note the format of the error messages with both C / C++ and name-mangled ASM name shown Fix and rebuild Library code. Review the code for syntax errors and logical errors. DON T IGNORE WARNINGS FIX THE PROBLEM, e.g. add.extern or fix type 19 / / 28

6 Now build and run the tests All the tests MUST fail (no ASM code written) otherwise using bad tests The test that passed has failed as a test Detailed test failure information Something wrong how did the ASM test pass with no code? 21 / 28 Clicking on the detailed test info automatically takes you to the test that failed 22 / 28 Better tests? Which one failed correctly and why (quiz question)? Test information modes are controlled in Test_main Show Failures and Expected Failures only Show Failures and Successes 23 / / 28

7 Use helper to add a separate test to show that result2 (from the ASM code) is equal to 42 Write the tests for self-documenting messages (What does result1 == result2 mean if 200 tests present?) HAVE What is a result1 and a result 2 Add further tests at WANT this location 25 / / 28 Write the CPP code to get to work but Write the ASM for speed (ISR) No need to measure times in Assignment1 or Labs unless doing Interrupt Service Routine FAIL SUCCESS SUCCESS 27 / 28 Available ASSERTS More can be customized CHECK(expected == actual) Success if passes XF_CHECK(expected == actual) Success if fails CHECK_EQUAL(expected, actual) XF_CHECK_EQUAL(expected, actual) Success if fails CHECK_CLOSE(expected, actual, tolerance) CHECK_ARRAY_EQUAL(expected, actual, length) CHECK_ARRAY_CLOSE(expected, actual, length, tolerance) REPORT( character message ); Changes to editor properties can make these ASSERTS into highlighted key-words 28 / 28

8 Handled today some easy nonprogramming quiz and midterm questions Why test, and what kinds of tests are there? Difference between TLD Test Last Development TDD Test Driven Development (Test First Development) How do you add tests for Assignment 1? More details on the testing process E-TDD and the testing tool E-UNIT Other kinds of available tests time and access Design of custom TESTs and ASSERTS 29 / 28