Testing and Verification

Final Review Brussels, December 12th, 2008 Achievements and Perspectives : Testing and Verification Cluster leader : Kim Guldstrand Larsen CISS, Aalborg University, DENMARK

Core Partners of the Cluster CISS, Aalborg University (real-time verification and testing, controller synthesis, security) EPFL (models and tools for quantitative aspects of embedded systems) CFV / Centre Fédéré de Verification (model checking and robustness of hybrid and real-time systems) INRIA / Rennes (symbolic testing, security, controller synthesis) LSV / CNRS (model checking, security protocols and logics) OFFIS, Oldenborg (UML-based verification and testing) University of Twente (verification and testing of hybrid and stochastic systems, security) Uppsala University (real-time verification, testing and schedulability) Verimag (real-time verification and testing, security protocols analysis) Affiliated partners: 5 industrial 6 academic

Cluster Activities JPRA-Cluster Integration Quantitative Testing and Verification (Ed Brinksma) JPIA-Platform: Testing and Verification Platform (Kim G. Larsen) JPRA-Cluster Integration Verification of Security Properties

Vision & Long Term Goals 30-70% of production time is currently spend on elaborate, ad-hoc testing Gap between industrial practice and academic stateof-the-art Time-to-market may be shortened considerable by verification and performance analyses of early design models Models must deal with quantitative information (realtime, memory, bandwidth, energy). ARTIST2 Model Network of Excellence on Embedded Systems Design Year1 Review -- Grenoble, October 3rd-4th, 2005 Verification and Testing /* Wait for events */ void /* Wait OS_Wait(void); for events */ void OS_Wait(void); /* Operating system visualstate process. Mimics a OS process for a * /* visualstate Operating system system. visualstate In this implementation process. Mimics this a is OS the process mainloop for a * * interfacing visualstate to system. the visualstate In this implementation basic API. */ this is the mainloop void * interfacing OS_VS_Process(void); to the visualstate basic API. */ void OS_VS_Process(void); /* Define completion code variable. */ unsigned /* Define char completion cc; code variable. */ unsigned char cc; void HandleError(unsigned char ccarg) { void HandleError(unsigned char ccarg) { printf("error code %c detected, exiting application.\n", ccarg); exit(ccarg); printf("error code %c detected, exiting application.\n", ccarg); } exit(ccarg); } /* In d-241 we only use the OS_Wait call. It is used to simulate a * /* system. In d-241 It we purpose only use is to the generate OS_Wait events. call. It How is this used is to done simulate is up a to * * you. system. It purpose is to generate events. How this is done is up to */ * you. void */ OS_Wait(void) { void OS_Wait(void) { /* Ignore the parameters; just retrieve events from the keyboard and * /* put Ignore them the into parameters; the queue. just When retrieve EVENT_UNDEFINED events from is read the from keyboard the and * * keyboard, put them return into the to queue. the calling When EVENT_UNDEFINED process. */ is read from the SEM_EVENT_TYPE * keyboard, event; return to the calling process. */ int SEM_EVENT_TYPE num; event; int num; Code Φ ΦΦΦ Req Running System

Vision & Long Term Goals Improve current industrial practice for validating embedded systems applications by continuous dissemination and improvement of existing powerful testing and verification techniques and tools. Effort on making state-of-the-art verification and testing technology visible and easily accessible for industry with long term vision of integration in tool chains applied in industry.

Quantitative Testing and Verification: High Level Objectives Y4 controller synthesis, robustness and implementability property-preserving code generation, generic frameworks using abstraction and compositionality combinations of testing and verification techniques. optimal scheduling, monitoring and fault diagnosis, analysis of hybrid models, stochastic and timed models

Testing and Verification Platform High Level Objectives Y4 continued improvement related to the individual tools dissemination as well as application on industrial case studies (https://bugsy.grid.aau.dk/artist2) high performance tools server (64 bit architecture and distributed implementation, common web-interface)

State of Integration in Europe Extensive collaboration between partners of the cluster Extensive collaboration with leading research teams outside Europe. Extensive interaction with other communities National Centers and projects CISS, ESI,.. CREDO, DaNES, DOTS, Testec, SAVE++,.. New FP7/ARTEMIS/ESF Projects ARTIST Design (Modeling and Validation) QUASIMODO (STREP) MULTIFORM (STREP) COMBEST (STREP) GASICS CESAR ARTIST2 Network of Excellence on Embedded Systems Design Year1 Review -- Grenoble, October 3rd-4th, 2005 Testing and Verification in Europe Birmingham Uppsala Chalmers Royal Holloway CISS MicroSoft Research Aalborg Oxford Tampere CWI Twente Helsinki ESI Nijmegen Oldenborg Eindhoven AVACS Namur OFFIS Verisoft CFV Liege Brussels Aachen Bonn Mons Saarlandes LSV LIAFA Verimag INRIA Rhone-Alpes Brno Lausanne Trento IRCCyn 18 Exchange INRIA/Rennes 22 visits Visits Microsoft Research NASA Ames Kestrel Technologies Weizmann Haifa Rice, Texas Kansas Cadence Hebrew University Bangalore TCS

Building Excellence 98 publications Y4 (ARTIST2 total: 299) 19 joint publications Y4 (ARTIST2 total: 62) High level of dissemination through PhD schools and industrial seminars (>30 keynote presentations). 120 100 80 Strong impact on a number of important international conferences (CAV, TACAS, FORMATS, EMSOFT, CONCUR, ETAPS, HSCC,..) ARTIST2 PhD schools (Autrans, Shanghai). 60 40 20 0 Y1 Y2 Y3 Y4 Pub Joint Transfer to industry through long-term collaboration performed by individual partners. National centers and laboratories.

Building Excellence Workshops organized INFINITY08 TIME 08 PDMC 08 Dagstuhl Seminar on Distributed and GRID computing RTSS 08 Track on Design and Verification FIT 08 NWPT 08 MOVEP CAV 09 TURING AWARD 2007 Ed Clarke, Allen Emerson, Joseph Sifakis Grand Officier de l'ordre national du Merite

Overall Assessment at the end of the NoE Quantitative Testing and Verification Verification for new quantitative models (priced TA, probabilistic TA, priced HA, stochastic games,..) CEGAR for quantitative models (timed, hybrid, stochastic,..) Compositional Verification Frameworks Controller Synthesis (1-clock PTA, budget constraints, Part. Obs., ATL) Generation of predictable code not pursued

Overall Assessment at the end of the NoE Testing and Verification Platform Individual tools mature, with industrial applications (AMT, BIP, DeVINE,SPIN,TIMES, UPPAAL,..) High-performance Verification Server has been achieved. Joint infra-structure for European Verification Grid not pursued.

Overall Assessment at the end of the NoE Extensive list of publications, invited and keynote lectures, etc witnesses true excellence within the area. Substantial effort has been put by individual partners in dissemination to research and industry. A large number of new collaborative projects has been initiated.

Highlight 1: A Safety Critical System

Hardware JOP (Java Optimized Processor) Native execution of Java Bytecode Bytecode implemented in Microcode Avoid unpredictable data-cache Time predictable Developed new method and stack cache Implemented in FPGA

FPGA Java Optimizing Processor Martin Schöberl University of Tech., Vienna

SARTS Safety Critical Java public static void main(string[] args) { new SporadicPushMotor( new SporadicParameters(4, 4000, 60), 0); new SporadicPushMotor( private void handlebrick() { new SporadicParameters(2, Sensors.synchronizedReadSensors(); private void handlebrick() 4000, 60), { 1); input private = (Sensors.getBufferedSensor(0) void handlebrick() { + Sensors PeriodicMotorSpooler motorspooler intsensors.synchronizedreadsensors(); input private = (Sensors.getBufferedSensor(0) =.getbufferedsensor(1)) void handlebrick() { >> 1; + Sensors int input new PeriodicMotorSpooler( Sensors.synchronizedReadSensors(); = (Sensors.getBufferedSensor(0).getBufferedSensor(1)) >> 1; + Sensors if (awaitingbrick) int input private = {(Sensors.getBufferedSensor(0).getBufferedSensor(1)) void handlebrick() { >> 1; + Sensors new PeriodicParameters(4000)); if (awaitingbrick) Sensors.synchronizedReadSensors(); if (input {.getbufferedsensor(1)) > lastread) { >> 1; if (awaitingbrick) int input if private (input = {(Sensors.getBufferedSensor(0) > void lastread) handlebrick() { = input; { + Sensors new PeriodicReadSensor( if (awaitingbrick) } else Sensors.synchronizedReadSensors(); if if (input ((lastread {.getbufferedsensor(1)) > lastread) - input) { = >= input; TRESHOLD) >> 1; { new PeriodicParameters(2000), } int else input motorspooler); if (input ((lastread = (Sensors.getBufferedSensor(0) awaitingbrick > lastread) - input) = { >= input; false; TRESHOLD) + { Sensors if (awaitingbrick) } else if ((lastread if (lastread { awaitingbrick.getbufferedsensor(1)) lastread - input) > BRICK_DETECTED) >= = false; input; TRESHOLD) >> 1; { { brickfound(lastread); RealtimeSystem.start(); } else if if (input ((lastread if (lastread awaitingbrick > lastread) - input) > BRICK_DETECTED) { = >= false; TRESHOLD) { if (awaitingbrick) } if (lastread { awaitingbrick lastread > brickfound(lastread); BRICK_DETECTED) = input; false; { } } } else } if if (input ((lastread if (lastread > lastread) - input) brickfound(lastread); > BRICK_DETECTED) { >= TRESHOLD) { { } } awaitingbrick lastread brickfound(lastread); = input; false; } } else } if ((lastread if (lastread - input) > BRICK_DETECTED) >= TRESHOLD) { { } awaitingbrick brickfound(lastread); = false; TASKS } if (lastread > BRICK_DETECTED) { } brickfound(lastread); } } METHODS

Byte code Timed Automata protected boolean run() if i<5 { } else { } i = i + 4; i = i * 4; return true; Data abstracted } Timing = WCET from microcode

SARTS to Timed Automata Detection of Deadline Violation Integrated SARTS w ECLIPSE Visualize WCET in ECLIPSE 18 methods + 4 tasks = 76 components

Highlight 2: Plastic Injection Molding Machine Robust and optimal control Tool Chain Synthesis: UPPAAL TIGA Verification: PHAVer Performance: SIMULINK 40% improvement of existing solutions..

Oil Pump Control Problem R1: stay within safe interval [ 5, 25 ] R2: minimize average/overall oil volume

25 l Time: 0 Vol: 10 Rate: 0 5 l

25 l Time: 1 Vol: 10 Rate: 0 5 l

25 l Time: 2 Vol: 10 Rate: -1.2 5 l

25 l Time: 3 Vol: 8.8 Rate: +1.0 5 l

25 l Time: 4 Vol: 9.8 Rate: +2.2 5 l

25 l Time: 5 Vol: 12 Rate: 0 5 l

25 l Time: 6 Vol: 12 Rate: 0 5 l

25 l Time: 7 Vol: 12 Rate: 0 5 l

25 l Time: 8 Vol: 12 Rate: -1.2 5 l

25 l Time: 9 Vol: 10.8 Rate: -1.2 5 l

25 l Time: 10 Vol: 9.6 Rate: -0.3 5 l

25 l Time: 10 Vol: 9.6 Rate: -0.3 5 l

25 l Time: 11 Vol: 9.3 Rate: -0.3 5 l

25 l Time: 12 Vol: 9.3 Rate: +2.2 5 l

25 l Time: 13 Vol: 11.5 Rate: +2.2 5 l

25 l Time: 14 Vol: 13.7 Rate: +0.5 5 l

25 l Time: 15 Vol: 14.2 Rate: -1.7 5 l

25 l Time: 16 Vol: 12.5 Rate: -0.5 5 l

25 l Time: 17 Vol: 12.0 Rate: -0.5 5 l

25 l Time: 18 Vol: 11.5 Rate: 0 5 l

25 l Time: 19 Vol: 11.5 Rate: 0 5 l

25 l Time: 20 Vol: 11.5 Rate: 0 5 l

Tool Chain Synthesis TIGA Performance Evaluation SIMULINK Guaranteed Correctness Robustness with 40% improvement Verification PHAVER

To be continued in To be continued in ARTIST ARTIST DESIGN DESIGN (Modeling & Validation) (Modeling Validation) Quasimodo Quasimodo Multiform Multiform Gasics Gasics DaNES DaNES DOTS DOTS....!!

Verification Using PHAVER Bang-Bang safe and robust HyDAC optimized possibly unsafe under fluctuation

Uniform distribution in [-0.1,+0.1] Performance SIMULINK Cycle UPPAAL Tiga strategy in m-format

Results

HSCC09: Franck Cassez, Jan J Jessen, Kim G. Larsen, Jean-Francois Raskin, Pierre-Alain Reynier Results Guaranteed Correctness Robustness with 40% improvement in performance