Testing Tutorial. presented on in Prague slides updated on by Marco Kessler, Dassault Systèmes

Transcription

1 Testing Tutorial presented on in Prague slides updated on by Marco Kessler, Dassault Systèmes

2 Content (4) Testing Modelica Libraries (17) CSV Compare (22) ModelManagement Library (28) XogenyTest Library (32) buildingspy (40) PySimulator (48) Commercial Testing Solutions (52) Testing Library (70) Summary of available testing solutions for Dymola users (71) Feedback and more Information (72) Literature 2

3 Intention of this presentation This presentation gives an overview about the available testing solutions for Modelica libraries Every testing solution is described on a couple of slides, containing Facts who, what, how, where, up-to-date, documentation, highlights Getting started installation, test setup, running tests Additional info caveats, behavior in certain scenarios Regarding Modelica tools and tools and operating systems Everything was tested from the perspective of a Dymola user with Windows as operating system However all solutions also work on other operating systems and/or in other Modelica tools The getting started slides focus on Windows and Dymola, but they should also help to get the testing solution running with other tools and operating systems The information for all tools have been gathered from the corresponding websites, documentations and publications. Some features might have been overseen or are not described correctly e.g if they are undocumented or if I, the author of this slides, just oversaw it or didn t get it 3

4 Daily bread of a Library Developer Modelica Classes have many dependencies Tools and their versions Other libraries Modelica language specification Developer want s to ensure that All classes check All Examples translate and simulate Same results are produced with all configurations 4

5 Possibilities to run checks for a Dymola Library developer Within Dymola Check button Can check package content, but check aborts on error Functions and.mos scripts Tedious and not much fun (no debugging except print) Automatization.mos scripts Python interface Java interface 5

6 General Solution: In-house Implementation to run nightly tests Automatization of Dymola via.mos files Python interface Java interface Common test cases Check all classes Translate examples Simulate examples Periodic testing with CI solutions like Jenkins or Hudson TeamCity GitLab CI 6

7 Test dependencies: Regression tests To discover mistakes introduced later regression tests can be used: Reference simulation results are generated After any change simulation results are compared to reference Possible changes: Tool type, tool version, library version, changed classes More effort required to implement in-house tool but many Modelica implementations exist already 7

8 Comparing signals Common approach: definition of a tube How to set the tolerance? Relative? Absolute? of course there is no general valid answer. One possibility: start with 0 tolerance and increase if necessary. 8

9 Storing signals as references By sampling the signal we loose information Interpolation to recreate it Which interpolation method? 9

10 Comparing continuous with sampled signals Sampled signals are problematic for continuous comparison and must be interpolated 10

11 Comparing two sampled signals with different output interval Common scale needed Interpolation for intermediate points 11

12 Comparing signals with small time delays Large errors can occur if an event is triggered a little later Easy solution for real signals: Filter Better solution: Compute a tube around reference so a time delay is allowed 12

13 Testing Nomenclature Unit test test of a single component For a defined input a defined output is expected Integration test Test if combination of several components gives the expected result Test case Contains everything to check a component for one scenario Test suite Collection of test cases Test runner Tool that identifies tests, runs them and checks the result 13

14 Running tests in Modelica many options PySimulator Model Management Buildingspy XogenyTest Testing Library CSV compare MoUnit Model Testing Toolkit RegressionTest 14

15 Regression tests: some fundamental questions Test setup With config files? In separate tool? Test execution Local or automated on server Single models and batch mode How to identify tests How to specify tolerances Effort to maintain tests What happens on renames of class and variables? Reference files File size, possibility to view and maybe edit content 15

16 Brief presentation of freely available solutions for Dymola users 16

17 CSV Compare - facts who? ITI GmbH what? Compare two csv files in specific format Compare two directories with csv files Compare FMU simulation result to csv file how? Standalone command line application Using FMUComplianceChecker for FMU simulation 17

18 CSV Compare - facts where? binaries available under releases up-to-date? Last commit 10 months ago Last release 2015: v documentation? Online at or from the command line: compare.exe --help highlights Open Source (MIT) HTML reports with svg graphics cmd.exe C:\>Compare.exe --help Compatible with FMUchecker 18

19 CSV Compare getting started Download prebuild binary and extract from Create csv file from Dymola simulation results From loaded simulation results: Plot variable of interest Right click on the result in variable browser > Export Result > Only Plot Window > filetype.csv > Save From.mat file: Either Use alist.exe from C:\Program Files (x86)\dymola <version>\bin\alist.exe set argument e to extract single variables Or load result in Dymola and export as described above From Dymola script : Use DataFiles.writeCSVmatrix Call compare.exe with appropriate arguments If alist.exe or Dymola created the results the delimiter must be set with --delimiter=, or d, 19

20 CSV Compare comparing results cmd.exe compare C:\CSVcompare>compare.exe --mode=csvfilecompare --verbosity=2 --delimiter=, --override -t 1e-2 -- reportdir=".\result" ClassA.csv ClassA_ref.csv Z15:51:45 [ Debug ] Starting new result 15:51: Z15:51:45 [ Debug ] Parsing commandline options: * --mode=csvfilecompare --verbosity=2 -d, --override -t 2e-2 --reportdir=.\result ClassA.csv ClassA_ref.csv Z15:51:45 [ Debug ] Successfully parsed the following options: Z15:51:45 [ Debug ] Operation mode is CsvFileCompare Z15:51:45 [ Debug ] Tolerance is 2e Z15:51:45 [ Debug ] Delimiter to be used to parse csv has been explicitly set and is "," Z15:51:45 [ Debug ] Logfile is empty Z15:51:45 [ Debug ] Column "x" seems to be text, this is good Z15:51:45 [ Debug ] Column "x" seems to be text, this is good Z15:51:45 [ Debug ] Exiting with exit code "0" Z15:51:45 [ Information ] Comparing "C:\CSVcompare\ClassA.csv" to "C:\CSVcompare\ClassA_ref.csv" Z15:51:45 [ Debug ] The resolution of the base x-axis is good Z15:51:46 [ Warning ] x is invalid! 450 errors have been found during validation Z15:51:46 [ Information ] Report has been written to: C:\CSVcompare\result\ClassA_ref_report.html Z15:51:46 [ Information ] Writing meta report to C:\CSVcompare\result\ index.html Z15:51:46 [ Information ] Metareport has been written to: C:\CSVcompare\result\ index.html 20

21 CSV Compare behavior Refactoring Simulated class renamed Filename changes -> must adapt cmd call Variables or components renamed Behavior: Test is skipped, Warning in cmd, nothing in report Required action: ref csv must be updated manually with new name Comparison Tolerances Argument t to set tube size in x and y direction (time and abs. value) doc says its only for discontinuities in x direction, but tests showed that y direction is also affected Changed simulation settings Number of result points Test is ok, no info Simulation time: Act is shorter: Test is ok, no warning Act is longer: Test fails (seems like limits are hold) Result file Format, Size, View.csv files created by user from.mat files or with simulation tool 21

22 ModelManagement facts who? Dassault Systemes what? Regression Test Compare Statistics Style check Class coverage Condition coverage how? User calls function ModelManagement.Check.checkLibrary MM creates references in working directory.mat file with full simulation results.mo file with statistic.log file with some info Design.Validation is used to compare results 22

23 ModelManagement facts where? Shipped with Dymola up-to-date? Current v1.1.6 released with Dymola 2017 No major changes in recent Dymola versions documentation? Dymola User Manual Volume 2: Chapter "4.4 Model and library checking ModelManagement.Check.UsersGuide highlights Part of every Dymola installation Check of model statistics Coverage analysis HTML reports 23

24 ModelManagement getting started Open ModelManagement library in Dymola Define experiments Set annotation experiment(stoptime= ) in every model that shall be checked Load additional library with test cases if needed Create references Call ModelManagement.Check.checkLibrary() with generatereference=true Compare Call ModelManagement.Check.checkLibrary() with referencefiledirectory= path/to/references Dymola Commands Create References import ModelManagement.Check.checkLibrary = true checklibrary(classcov=false, condcov=false, stylecheck=false, name="mylib", generatereference=true, referencefiledirectory="c:/mm/ref", generatereferencetranslationstructure=true); Collecting classes in 'MyLib' Test cases collected from 'MyLib' Found 5 test case(s) in 'MyLib'... Done 24

25 ModelManagement compare results Dymola Commands Run test import ModelManagement.Check.checkLibrary = true checklibrary(name="mylib", referencefiledirectory="c:/mm/ref"); Collecting classes in 'MyLib' Found 148 classes Checking all classes Log of check generated. Path: 'C:/MM/wd/MyLib_StyleCheckLog.html' Done checking 'MyLib' Collecting classes in 'MyLib' Test cases collected from 'MyLib' Found 5 test case(s) in 'MyLib' = true = true = true Log generated. Path: 'file:///c:/mm/wd/mylib_html_log.html' Done MODEL VALIDATION Settings for validation Model Name: MyLib.Examples.LossyGearDemo Result couplings Variable: Inertia2.phi, Tolerance: Variable: Inertia2.w, Tolerance: Variable: gear.mode, Tolerance: Integrator Setup Integrator: Dassl Time interval -> Start time: 0, Stop time: 0.5 Tolerance: Advanced Setup Settings Validation file: C:/MM/ref/MyLib.Examples.LossyGearDemo_ref.mat Scalar Criteria check Variable: Inertia2.w has scalar criteria larger than tolerance

26 ModelManagement behavior Refactoring Renamed classes and variables Behavior Error: Reference is not found Required Actions: Create new reference with renamed model Old references stay in Reference directory -> delete manually Comparison Tolerances Absolute tolerance of 1e-3 (not changeable) Changed simulation settings Result points No warning Simulation time: Act shorter: Test is ok Act longer: Test is ok Result file Format, Size, View, Edit Dymola.mat result files All variables are stored > potentially large files Can be viewed with SDF Editor Other tool (like e.g. Matlab) needed to edit 26

27 ModelManagement caveats All state variables are compared Big reference files Errors for other variables are not detected! To test single components like a sine block a state must be introduced Test Identification All classes with experiment(stoptime= ) are treated as test cases Not possible to ignore an example e.g. for similar demo examples in a library No error if an experiment does not exist anymore old references must be removed by hand Storing default simulation settings with Dymola GUI not working as expected Default settings are not stored, but removed (nothing set = default) The annotation must be added by hand to get e.g. an experiment with StopTime=1 27

28 XogenyTest facts who? Xogeny / Michael Tiller what? Proof of concept only! No visual elements, no connectors, only models and functions Run unit tests with models and functions Contains functions to check single values Contains models to check trajectories how? By calling Modelica functions By instantiating models and setting parameters and inputs in text layer Expected trajectories are defined as 2D table 28

29 XogenyTest facts where? up-to-date? Last commit on GitHub was 4 years ago documentation? Inside Models and Functions highlights Open source 100% Modelica should work in every tool 29

30 XogenyTest getting started Clone or download from GitHub Create test model in Modelica tool Instantiate test component from library XogenyTest Set parameters and input for test component Optionally set TestCase annotation (has no effect yet) Simulate model / call function check simulation log Notepad CheckSuccess.mo model CheckSuccess Real x = time^2; XogenyTest.AssertTrajectory check_x( actual=x, expected=[0,0; 1,1; 2,4; 3,9]); annotation( TestCase( action="simulate", result="success"), experiment(stoptime=4)); end CheckSuccess; 30

31 XogenyTest behavior Refactoring Renamed classes and variables Tool can take care of it Renamed variables Tool can take care of it Comparison Tolerances Abs. tolerance is user choice Interpolation Not needed (tests run only for given values in trajectories record, not in between) Changed simulation settings Result points No influence (comparison during simulation with events at specified time) Simulation time: Depends on test (some not affected, asserts for others) Result file Format, Size, View user choice 31

32 buildingspy facts who? Lawrence Berkeley National Laboratory what? Run regression tests for whole libraries by comparing selected variables with references Simulate models Plot how? Calls dymola.exe with.mos scripts Test setup: one.mos-script for each model to test References as.txt 32

33 buildingspy facts where? (official website) (code) (install via pip) up-to-date? Last release: Version 1.6.0, March 29, 2017 documentation? Python functions are well documented Brief how-to on highlights Open Source (revised BSD) and hosted on GitHub Automatic reference file generation; user is asked to accept new results Supports Dymola and OpenModelica 33

34 buildingspy installation Install python 2.7 Either use a scientific python distribution Like anaconda: Or plain python installation from Install numpy, scipy and matplotlib Scipy on windows without scientific python distribution: see Make Dymola available to buildingspy Add dymola.exe to PATH environment variable Optional: Install htmltidy for HTML syntax validation From: Bit version must match with your python installation bit version Extract tidy.dll and add it to PATH environment variable cmd.exe install steps with anaconda C:\Anaconda2>conda create -n buildingspy python=2.7 C:\Anaconda2>activate buildingspy (buildingspy) C:\Anaconda2>conda install numpy... (buildingspy) C:\Anaconda2>conda install scipy... (buildingspy) C:\Anaconda2>conda install matplotlib... (buildingspy) C:\Anaconda2>pip install buildingspy... Successfully installed buildingspy future gitdb gitpython pytidylib smmap

35 buildingspy test setup Create test model in Modelica tool and simulate Create a.mos script containing simulatemodel with simulation settings createplot for each variable to compare Store the.mos in <library-dir>\resources\scripts\dymola Creating the.mos file with Dymola: Simulate model, plot variables of interest File > Generate Script > Plot setup > OK > Store file remove everything from file but createplot() Add simulatemodel() Notepad Sine.mos //.mos script for unit test of MCL.Sine simulatemodel( "MCL.Sine", resultfile="sine") createplot( id=1, position={15, 10, 584, 416}, y={"sine.y"}, range={0.0, 3.0, -1.5, 1.5}, grid=true, colors={{28,108,200}}); 35

36 buildingspy running tests Python C:\MyLib\run_tests.py import buildingspy.development.regressiontest as r import os def main(): rt = r.tester(executable='dymola', cleanup=true) mylib = r"c:\path\to\library" rt.setlibraryroot(mylib) # r.setsinglepackage('mylib.examples') rt.batchmode(false) rt.setnumberofthreads(4) rt.run() if name == ' main ': main() cmd.exe (buildingspy) C:\MyLib>python run_tets.py Using 4 of 8 processors to run unit tests. Number of models : 8 blocks : 0 functions: 1 Generated 4 regression tests. Script that runs unit tests had 0 warnings and 0 errors. See 'simulator.log' for details. Unit tests completed successfully. Execution time = s done. Process finished with exit code 0 36

37 buildingspy behavior Refactoring Renamed classes Either rename in tool and update.mos scripts manually Or use buildingspy.development.refactor.move_class Renamed variables.mos scripts must be updated manually Comparison Tolerances Fixed to 1e-3 for absolute and relative error No time delay Interpolation Simulation results are decimated to 101 points with linear interpolation Changed simulation settings Result points Warning that variable has fewer points than in reference Simulation time: Warning is given, plot is shown, results can be updated Result file Format, Size, View.txt files, only with variables user wants to compare Small, diff possible Up to 101 equally spaced data points 37

38 buildingspy caveats 1/2 Results trajectories are decimated to 101 values For comparison and for reference file generation Small reference files Be careful with fast changing signals Variable _npoi in buildingspy.development.regressiontest.tester e.g. for example on previous slide: rt._npoi = 201 For each test the complete library is copied to a temporary directory On Windows 7: C:\Users\<user-name>\AppData\Local\Temp 38

39 buildingspy caveats 2/2 Make sure that Dymola does not save the startup directory! Script will hang until Dymola.exe is killed timeout can be set for buildingspy.simulate.simulator, but apparently not for Tester Dymola 2018: Edit -> Options -> Settings -> Do not save Startup directory Multicore: add if name ==' main ': to your script as shown in the previous example Otherwise infinite processes are spawned, filling your hard disk with temporary files 39

40 PySimulator facts who? DLR Institute of System Dynamics and Control what? Simulation Environment with many plugins For different Modelica tools For analysis of models and simulation results Tester plugin included HTML report generated how? Simulation: Calling dymola.exe with.mos scripts Comparison: all variables in the result files of the selected folders are checked 40

41 PySimulator facts where? up-to-date? Last commit and latest release 5 months ago documentation? None in the tool 2 papers published See last slide of this presentation and highlights Support for many tools Can be easily extended GUI included, but also usage of python functions possible 41

42 PySimulator installation Get latest release from For Windows users with no Python installed Execute Install PySimulator.bat The.bat file installs Python(x,y) A python distribution like Anaconda Hosted No releases and commits since 2 years Not recommended to combine with other Python distributions First Installation Note on Python(x,y) Website: It is recommended to uninstall any other Python distribution before installing Python(x,y) If you already use e.g. Anaconda and want to run PySimulator in a virtual python environment Manual installation is described on See remarks on next slide For Linux users See 42

43 PySimulator remarks for manual installation on Windows Required packages for manual installation listed on You have to download all.whl files and install them one by with pip Easier alternative: Install Anaconda Download source of latest PySimulator release Extract source code to C:\pysim\src Create a virtual environment and follow these steps > Remarks: tested with PySimulator 0.62 on Windows 7 64bit with Anaconda2 Python setup.py install failed on the first execution, but on the 2 nd try it worked If you try to start PySimulator from src the wrong module will be started. Leave the directory first (hence the cd.. In the example on the right) Only the steps for the Dymola plugin are described here. Other plugins need more python packages. See the wiki on GitHub for details cmd.exe Install PySimulator C:\pysim>conda create n PySim python=2.7 C:\pysim>activate PySim (PySim) C:\pysim>conda install chaco (PySim) C:\pysim>conda install scipy (PySim) C:\pysim>conda install Matplotlib (PySim) C:\pysim>conda install tornado (PySim) C:\pysim>conda install PIL (PySim) C:\pysim>conda install H5py (PySim) C:\pysim>conda install pyside (PySim) C:\pysim>conda install kiwisolver (PySim) C:\pysim>cd pysim src (PySim) C:\pysim\src>python setup.py install (PySim) C:\pysim\src>cd.. (PySim) C:\pysim\>python -m PySimulator 43

44 PySimulator test setup Notepad setup.txt # Setup file for simulation of several models by PySimulator # # modelfile modelname subdir tstart tstop tol stepsize ninterv includeevents # "C:/MyLib/package.mo" MyLib.ClassA "" e true "C:/MyLib/package.mo" MyLib.ClassB "" e true "C:/MyLib/package.mo" MyLib.ClassC "" e false Create references Execute your setup.txt with Plugins > Testing > Simulate List of Models Choose the appropriate simulator (Dymola, FMUSimulator, etc.) Every model will be simulated and the result files (.mat for Dymola) are stored in the selected results directory 44

45 PySimulator running tests report.html Given Error Tolerance: 1e-3 Disk space of all used result files: 0.2 MB Total number of compared files: 2 against 2 baseline files Total number of Compared Variables: 68 (66 passed, 2 failed) Disk space of full report directory: 0.0 MB Generated: 12:21PM on May 11, 2017 by PySimulator Time Taken: 00h:00m:00s Result Files Status act Dymola 1 / 1 1 passed / 1 failed 50% 50% passed Baseline MCL.Sine 0 / / 9 / 2 [1.8e-1] MCL.SwitchRC 1 / / 59 [0.0e+0] 45

46 PySimulator behavior Refactoring Renamed classes Test fails, no info about missing signal Tests missing Only files which are found in both the ref and the act folders are compared, no info if something is missing Comparison Tolerances Deviation measure (abs. and rel. integrated error) For details: see the paper PySimulator A Simulation and Analysis Environment in Python with Plugin Infrastructure Changed simulation settings Result points Test is performed, no specific info Simulation time: Act shorter: Test fails Act longer: Test fails Result file Format, Size, View.mat,.mtsf 46

47 PySimulator caveats Settings are stored in PySimulator.ini On Windows 7: C:\User\<user-name>\.config\PySimulator\PySimulator.ini e.g. needed to change the path to Dymola.exe Pip version is not the most recent (used by pip) has PySimulator 0.61 listed, whereas 0.62 was released on GitHub Starting PySimulator if installed manually From cmd: python m PySimulator 47

48 Modelon Model Testing Toolkit 1 who? Modelon AB what? Regression Test with HTML reports Supports Several tools Dymola, OpenModelica and OPTIMICA Compiler Toolkit how? GUI and Python Scripts Local execution and on Jenkins Server where? Check the following blog posts to get an impression: : The tool was not tested for this tutorial. Instead all info and the picture are taken from 48

49 Claytex Regression Test Tool 1 who? Claytex Services Limited what? Check all classes in library Regression test (translation log, sim results) Verify encryption (regression test with.moe) how? Standalone tool that controls Dymola Experiments are simulated (models with simulation setting annotations) where? 1 : The tool was not tested for this tutorial. Instead all info and the picture are taken from 49

50 Regression tests in other Modelica Tools MoUnit Extension for OneModelica Samlaus, Roland, Mareike Strach, Claudio Hillmann, and Peter Fritzson. "MoUnit--A Framework for Automatic Modelica Model Testing." In Proceedings of the 10 th International Modelica Conference; March 10-12; 2014; Lund; Sweden, no. 96, pp Linköping University Electronic Press, Website down on and still on OpenModelica Wolfram System Modeler

51 The new Testing Library which will be included in Dymola 51

52 Why another implementation for Modelica Testing? User convenience: making testing easier All settings within Dymola Available for all Dymola users More control Set tolerances for every variable Define which signals to store and how to store Low effort when library is refactored 52

53 Testing Library - facts who? Dassault Systemes what? A Modelica Library which allows to create and run tests within Dymola how? Running tests during simulation by comparing signals Using features which ship with Dymola 95% pure Modelica 2.5% ModelManagement Library and DymolaCommands (for Test runners) 2.5% SDF Library (for reference results) 53

54 Testing Library - facts where? Not available yet, included in future Dymola versions Demo-version was handed out at the Testing tutorial at the Modelica Conference 2017 up-to-date? v1.0 still under active development documentation? User s guide Documented classes highlights Everything inside Dymola No config files, updated paths when refactoring Run tests within Dymola 54

55 Testing Library Includes models to generate and read references perform checks Provides functions to Create references Check library Run test models 55

56 Usual Workflow Develop Dymola model Build internal test models Plot results and check by hand Commit 56

57 Workflow with Testing Library Develop Dymola model Build test model with Testing Library Provide reference manually or generate reference as sdf Commit Run Test periodically 57

58 Creating a Test Model Build models as usual Add Setup block for global settings Add compare blocks Supply or generate references Simulate and check messages manually or let a test runner do the job 58

59 Creating References Build test model with Setup component Specify setup.filename extend Base.TestCase for Dymola selection annotation Connect signals to log block Call Testing.Functions.createRefs 59

60 Creating References createrefs() createrefs does the following for the selected model: Check if Testing.Base.TestCase is extended Get the value of setup.filename By simulating 0 seconds with setup(writefilename=true) and reading the file Testing.filename in the working dir Check if destination exists argument replace must be true to continue if file exists Read the experiment and output annotations Set the output setup equidistant time grid, doubleprecision Simulate the model With the experiment settings and setup.mode=createrefs Convert the result to sdf Copy the sdf file to setup.filename Dymola Commands Create References Testing.Functions.createRefs("MyLib.Example", replace=true); > result file copied to C:/MyLib/Resources/References/Example.sdf t=0 Testing.filename t=stoptime result <path-to-file> result In short: It simulates your model with the experiment settings stored inside the model and copies the result to the specified path. result <path-to-file> 60

61 Creating References Dymola selections are used to reduce the result file size TestCase Sets selection annotation Log Block Sets tag for Dymola selection Extend TestCase and using Log blocks Signals connected to the inputs will be in group refs in the SDF file Modelica Testing.Base.TestCase class TestCase "Base class to set Dymola selections" annotation ( Dymola_selections={ Selection( name="ref", match={ MatchVariable( tag="refs", name="*u[*] *u", newname="%name%")})},...); end TestCase; Modelica Testing.Refs.LogReal model LogReal "Set tag to store signal in group refs" extends Icons.AboveName;... Modelica.Blocks.Interfaces.RealInput u annotation (..., Dymola_tag={"refs"}); end LogReal; 61

62 Creating References reducing file size Unfortunately Dymola stores not only selections but also All parameters All states Remove unnecessary signals with SDF editor Open the file, make sure that all groups are collapsed Select everything with Ctrl+A Deselect the.sdf file itself at the top, refs and Time Delete and save 62

63 Compare Models Signal comparison for Boolean, Integer, Real Compare signal with reference trajectory Timed compare Specify in a table the expected value for a point in time Within Constant limits Signal must be inside specified limit 63

64 Compare Block RealSig Two Inputs actual and reference value Prints result to message window Optional: Writes JSON files on hard disk Contain expected and actual result Can be used for automatization e.g. by reading the results with a python script 64

65 Simulation results in Message Window Error message when signals are outside tolerance Summary at end Results stored in working directory If setup.writejson=true For every test block a JSON file is created in <working directory>\testinglib Output directory can be changed with setup.resultdir 65

66 Test runner checkpackage() does the following for the selected package: Get names of all classes Check every class and print the result Skip classes with the annotation Dymola_interactive=true Dymola Commands checkpackage Testing.Functions.checkPackage("MyLib"); > Checking all classes in MyLib (i) started at 17:15:34 #ok: MyLib.ClassA #fail: MyLib.ClassB #skip: MyLib.ClassC ( Dymola_interactive=true) #ok: MyLib.ClassD (i) check finished at 17:15:39 (i) Summary 4 classes 2 ok 1 fail 1 skipped = true 66

67 Test runner runtests() does the following for the selected package: Get names of all classes Simulate all classes with name starting with Test_ With the settings stored in experiment and output annotations Activate JSON output and set the output dir To <wd>\testinglib\<class-path> Get the check result from the JSON files in the dir Print the result Dymola Commands runtests Testing.Functions.runTests("MyLib.SubPackage"); > Running all tests in MyLib.SubPackage (i) started at 17:8:53 # ok: MyLib.SubPackage.Test_ClassA tests: 3 ok: 3 skip: 0 fail: 0 not_run: 0 #fail: MyLib.SubPackage.Test_ClassB tests: 4 ok: 2 skip: 0 fail: 1 not_run: 1 # ok: MyLib.SubPackage.Test_ClassC tests: 2 ok: 2 skip: 0 fail: 0 not_run: 0 (i) check finished at 17:9:22 (i) Summary 3 classes 2 ok 1 fail = true 67

68 Testing Library - behavior Refactoring Renamed classes Automatically updated by Dymola if test models are loaded Renamed variables Automatically updated by Dymola if test models are loaded Comparison Tolerances Absolute, relative and time delay can be set Changed simulation settings Result points Ref from file: Interpolation with SDF tables Simulation time: Tests are marked as not_run where applicable Ref from file: depending on chosen SDF extrapolation (hold, none, linear, etc.) Result file Format, Size, View sdf, editable with SDF Editor size can be controlled with simulation settings 68

69 Automatization calls writes reads runs junit.xml <check>.json report.html 69

70 Summary of available projects for Dymola users Lib Author Environment License Supports Test Identification Configuration buildingspy Berkeley Lab Python revised BSD Dymola, OM.mos files.mos files PySimulator DLR Python LGPL Dymola, OM, SimulationX, FMU Wolfram System Modeler Config files Config files XogenyTest Mike Tiller Modelica CC3 Modelica - Modelica models CSV compare ITI Command line BSD / MIT csv files - arguments Model Testing Toolkit Modelon Standalone, Python Proprietary Dymola, OM, OPTIMICA, FMU.mo and.py files In Tool Model Management Dassault Dymola Proprietary Dymola annotations Function args MoUnit OneWind Java Non-Commercial OneModelica / Java Config files Config files RegressionTest Claytex Standalone Proprietary Dymola annotations In Tool Testing Library Dassault Dymola Proprietary Dymola Model Name Modelica models 70

71 Feedback and more Information Please contact us for more information or if you want to give feedback about this slides or the Testing Library! 71

72 Literature on testing Modelica classes Tiller, Michael M., and Burit Kittirungsi. "UnitTesting: A Library for Modelica Unit Testing." In Proceedings of the 5th International Modelica Conference, Vienna Describes a commercial unit test library by Emmeskay, Inc (acquired by LMS in 2010), which does not seem to be available anymore. Samlaus, Roland, Mareike Strach, Claudio Hillmann, and Peter Fritzson. "MoUnit--A Framework for Automatic Modelica Model Testing." In Proceedings of the 10th International Modelica Conference, Lund Presents MoUnit, a test framework that can be used in OneModelica. Unfortunately the OneModelica Website is currently down. Tilly, Anders, Victor Johnsson, Jon Sten, Alexander Perlman, and Johan Åkesson. "Optimica Testing Toolkit: a Tool-Agnostic Testing Framework for Modelica Models." In Proceedings of the 11th International Modelica Conference, Versailles Optimica Testing Toolkit is now released as Modelon Model Testing Toolkit Asghar, Adeel, Andreas Pfeiffer, Arunkumar Palanisamy, Alachew Mengist, Martin Sjölund, Adrian Pop, and Peter Fritzson. "Automatic Regression Testing of Simulation Models and Concept for Simulation of Connected FMUs in PySimulator." In Proceedings of the 11th International Modelica Conference, Versailles Describes regression testing in PySimulator 72

73 73