RealTime-UnitTest On Object Code Level

Transcription

1 RealTime-UnitTest On Object Code Level

2 The Blue Box

3 Motivation IEC ISO DO178B/C IEC EN 5012x More and more standards highly recommend intensive testing Model based development and testing explicitly mentioned Tool qualification to gain confidence in the use of software tools

4 RealTime-UnitTest - What s in it for you? Characteristics Object Code Level No test driver Integrated in the development environment Requirements For Code Coverage analysis a micro controller with trace support is a must The target application should already run on the target Some stack space is needed Compilers should be EABI (Embedded Application Binary Interface) compatible* * specifies standard conventions for file formats, data types, register usage, stack frame organization, and function parameter passing

5 RealTime-UnitTest - What will it do for you? Benefits Quick turn-around times No additional compile, link and download cycle while testing Flexibility in use Combine trace, performance analysis, code coverage and I/O stimuli with test runs Technology may be used for unit, integration and system test Test environment corresponds as closely as possible to the target environment

6 How does it work? Application is running on a target Initialization done Pieces of the application already implemented A test case is setup and will be executed on the target Run until main{} Jump to the function under test, push test parameters on the stack Execute the funtion under test with parameters Compare return values with expected values and jump back to main{} How to setup test cases and execute them on a target? Remote Control and Test API Architecture

7 Remote Control and Test API Architecture isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect test isystem.connect debug isystem.connect isystem IDE & Debugger winidea

8 C/C++ API isystem.connect isystem IDE & Debugger winidea

9 C/C++ API Verwendung isystem.connect debug debug.download(); debug.setbp("myfunction"); debug.run(); debug.waituntilstopped(0, 200); debug.deletebp("myfunction"); Use of isystem.connect m_piconnectdebug.runcontrol(iconnectdebug::rdownload, 0, 0); m_piconnectdebug.setbreakpoint(iconnectdebug::bsymbol IConnectDebug::bSet, 0, 0, "myfunction", 0); m_piconnectdebug.runcontrol(iconnectdebug::rrun, 0, 0); do { m_piconnectdebug.getstatus(iconnectdebug::tcpu, &status); Sleep(200); IConnectDebug::SStatus status; } while (status.m_bystatus!= IConnectDebug::stStopped); isystem IDE & Debugger winidea m_piconnectdebug.setbreakpoint(iconnectdebug::bsymbol IConnectDebug::bClear, 0, 0, "myfunction", 0);

10 C/C++ API Write your own C/C++ application and remote control winidea core functionality Use already built abstraction layers for National Instruments LabVIEW (VI library), The Mathworks Matlab (JAVA) Microsoft Automation (e.g., to read/write data from/in Excel, use C#, Jscript, ASP, VBScript,...)... Write your own abstraction layer To simplify the use of this API and to implement RealTime-UnitTest we added another abstraction layer isystem.connect debug Script language wrappers

11 Debug Abstraction and Script Language Layer isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect debug isystem.connect isystem IDE & Debugger winidea

12 Debug Abstraction and Script Language Layer Use of isystem.connect debug debug.download(); debug.setbp("myfunction"); debug.run(); debug.waituntilstopped(0, 200); debug.deletebp("myfunction"); Use of isystem.connect m_piconnectdebug.runcontrol(iconnectdebug::rdownload, 0, 0); m_piconnectdebug.setbreakpoint(iconnectdebug::bsymbol IConnectDebug::bSet, 0, 0, "myfunction", 0); m_piconnectdebug.runcontrol(iconnectdebug::rrun, 0, 0); IConnectDebug::SStatus status; do { m_piconnectdebug.getstatus(iconnectdebug::tcpu, &status); Sleep(200); } while (status.m_bystatus!= IConnectDebug::stStopped); isystem IDE & Debugger winidea m_piconnectdebug.setbreakpoint(iconnectdebug::bsymbol IConnectDebug::bClear, 0, 0, "myfunction", 0);

13 Read variable 'icounter' on the target and print its value until it reaches value of import isystem.connect from isystem import icutils from isystem.connect import IConnectDebug # First we obtain controller objects icfactory = icutils.getfactory() dbg = icfactory.getdebugfacade() ide = icfactory.getidectrl() import time # Initialize dbg.download() dbg.deleteall() dbg.setbp('main') dbg.run() dbg.waituntilstopped() # Set global variable 'isdebugtest' to 1 and run the target dbg.modify(iconnectdebug.fmonitor, 'isdebugtest', '1') dbg.run() py_icounter = 0 # Read value of 'icounter' until condition is met while py_icounter < : evalresult = dbg.evaluate(iconnectdebug.frealtime, 'icounter') py_icounter = evalresult.getint() print py_icounter time.sleep(1) dbg.stop() print "Target stopped, 'icounter' reached the limit:", py_icounter raw_input('press ENTER to exit')

14 Conclusion Generic API enables you to setup an automation scenario for Development Testing Effort doing it in C/C++ is a bit higher than using abstraction layers as well as script languages It still has not much to do with functional testing approach

15 Test API isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect test isystem.connect debug isystem.connect isystem IDE & Debugger winidea

16 Test API Use of isystem.connect test Use of isystem.connect debug debug.download(); Use of isystem.connect m_piconnectdebug.runcontrol(iconnectdebug::rdownload, 0, 0);

17 Test Specification Language YAML 1.1 yaml.org Similar to XML Data structures like in Perl, Python, C Human readable # Simple test of function: int functestint2(int a, int b, int c) func: [functestint2, [ , 2, 1], retval] expect: - retval == de.wikipedia.org/wiki/yaml isystem.com/downloads/isystem.connect/api/yaml_spec.html Use of isystem.connect test testcase.itestcpp("func: [functestglobals]\n" Use of isystem.connect debug debug.download(); Use of isystem.connect m_piconnectdebug.runcontrol(iconnectdebug::rdownload, 0, 0);

18 import isystem.connect as ic import isystem.itest as it def verifyresult(results): for result in results: if result.iserror(): out = ic.cstringstream() emitter = ic.emitterfactory.createyamlemitter(out) emitter.startstream() emitter.startdocument(false) result.serializeerrorsonly(emitter, None) emitter.enddocument(false) emitter.endstream() print out.getstring() else: print "OK" cmgr = ic.connectionmgr() cmgr.connectmru('') dbg = ic.cdebugfacade(cmgr) # initialize the target - this may differ in your environment dbg.download() dbg.rununtilfunction("main") dbg.waituntilstopped() testcase = it.ptestcase(cmgr) # Simple test of function: int functestint2(int a, int b, int c) testcase.itest( """ func: [functestint2, [ , 2, 1], retval] expect: - retval == """, None, True) print 'functestint2 test result:', verifyresult(testcase.gettestresults()) # Test with coverage testcase.itest( """ func: [functestglobals] coverage: runmode: start document: itestcoverage.ccv openmode: w issaveaftertest: yes exportformat: XML2 exportfile: coveragetest.xml statistics: - func: functestglobals """, None); print 'functestglobals test result:', verifyresult(testcase.gettestresults())

19 Remote Control and Test API Architecture isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect for Python wrapper isystem.connect test isystem.connect debug isystem.connect isystem IDE & Debugger winidea isystem Test GUI testidea

20 testidea GUI

21 testidea Standard Integrated in winidea Free of charge Online help and samples Descriptive tooltips TestID, tags, description Local test variables Stubs Trace, Profiler, Code Coverage support Derived/sub test case creation Debugging of test cases Report Generation in XML and HTML Stylesheet for HTML Report Generation

22 testidea Professional testidea standard features A license fee applies Python script generation by mouse click I/O module support Script execution before and after a test run Excel/CSV import and export Test filter based on tags Test specification can be saved and loaded into/from C/C++ source code Dry run precalculation of expected return values by test execution Automatic wrong parameter detection Tag for specifiying the sequence of functions called during test Breakpoints inside functions under test Real-time stubs (stubs executed in real-time) Support/update and upgrade service

23 Saved Test Specification from within testidea env: initbeforerun: false downloadoninit: true resetoninit: false runoninit: true stopfunction: main deletebreakpointsoninit: false reportconfig: reportcontents: full outformat: xml filename: 'D:\isystem.testIDEA\TestResults\PPC554 ITMPC554 GCC int RAM Test Report.xml' xsltfull: '<built-in> itestresult.blue.xslt' isincludetestspec: true usecustomtime: true testinfo: tester: 'Werner' date: ' ' time: '11:33:29' software: 'SW' hardware: 'HW' description: 'Desc' testspecificationfile: 'D:\isystem.testIDEA\My YAML Files\PPC5554 ITMPC5554 GCC Int RAM.iyaml' wiworkspacepath: 'D:\winIDEA\2011\Examples\OnChip\PowerPC\MPC55xx\ITM PC5554\GCC\IntRAM' tests: - id: 'StubTest' desc: 'Stub Test' init: icounter: 0 func: [Address_DifferentFunctionParameters] stubs: - func: - Func2 - retv assign: retv: 2 expect: - icounter == 1 - id: 'GlobalVariableTest' desc: 'Global Variable Test' init: icounter: 0 func: [Address_DifferentFunctionParameters] expect: - icounter == 1

24

25 emote *E*mbedded *Mo*del-based *Te*sting Funded by BMWi Germany Project partners: FZI Karlsruhe, sepp.med GmbH Project Number: KF SS9

26 Regression Test Tool Suite

27 Tool Qualification Safety Manual Test GUI Test Specification, Test Cases, Test Reports Reference-Target + Consulting/Integration Services isystem.connect for Python wrapper isystem.connect test isystem.connect debug isystem.connect isystem Development Environment & Debugger winidea testidea Reference Application Reference Target