Learning ctools and GammaLib development in an hour

Transcription

1 Learning ctools and GammaLib development in an hour Introduction to 6 th ctools coding sprint Jürgen Knödlseder (IRAP)

2 What I expect you know How to write C++ and/or Python code How to use Git Our GitLab development workflow! (see The content of our reference paper! (see 2

3 What I will cover in this presentation GammaLib and ctools overview Developing sustainable software Adding a new class to GammaLib Adding a new tool to ctools Adding a new script to ctools Adding tests 3

4 What is GammaLib? Using GammaLib! as C++ library 4

5 What is GammaLib? Using GammaLib! as Python module 5

6 What is in GammaLib? 6

7 What do I need as developer? ANSI C++ compiler (e.g. g++, clang) Git make, automake, autoconf, libtools cfitsio (survives without it, but not very useful) readline & ncurses (nice to have for tab completion) Python (including Python.h header file) SWIG Example installation on Mac OS X (after Xcode install): 7

8 What are ctools? Using ctools as! executables! from the! command line 8

9 What are ctools? Using ctools as! Python module 9

10 What is in ctools? 10

11 How do both relate? A ctool is a C++ executable that uses GammaLib as a! C++ library 11

12 How do both relate? A cscript is a Python script that uses GammaLib as a! Python module Example: cscaldb 12

13 Why should software be sustainable? Sustainability: meeting the needs of the present without compromising the ability of future generations to satisfy their own needs Short-term interest Long-term benefits Have a quick solution at the expense of software aging technical debt high lifecycle costs Have a long-lasting solution providing reliability adaptability maintainability 13

14 Why should software be sustainable? Sustainability: meeting the needs of the present without compromising the ability of future generations to satisfy their own needs Short-term interest Long-term benefits Have a quick solution at the expense of software aging technical debt high lifecycle costs Not sustainable Have a long-lasting solution providing reliability adaptability maintainability Sustainable 14

15 How to achieve sustainability? From Clear licensing Clear documentation Commonly adopted and modern programming language Modular design Clear revision management and change control Risk management Clearly established software testing regime and validated results Open and common standards Clear separation between data and code Clear understanding of dependencies 15

16 How we achieve sustainability? From Clear licensing GNU GPL3, Open Source Clear documentation Code documentation (Doxygen), User documentation (Sphinx) Commonly adopted and modern programming language C++ & Python Modular design implemented for GammaLib & ctools Clear revision management and change control Git workflow Risk management avoidance of OS, version, compiler and library dependencies (continuously tested) Clearly established software testing regime and validated results test suites are part of code (incl. Science Verification), Continuous Integration & Testing Open and common standards Strict application of coding standards Clear separation between data and code No data in code Clear understanding of dependencies Allowed dependencies established 16

17 What you should do Make sure you read and follow the coding conventions C++98 ISO (many servers still don t support C++11) Python (may evolve since few systems nowadays with Python 2.5-, but some with Python 2.6, and Python 2.7 supported at least until 2020) Indent with 4 spaces, no tabs Do not exceed 80 characters in one line Use coherent class and method names (always same method for same function, e.g. clear(), size(), is_empty(), append(), insert() ) Only use Python standard library modules Document your code (code documentation with Doxygen) Document the usage of your code (user documentation with Sphinx) Write unit tests (you may do this even before writing your code) If in doubt, ask for code review (can be nicely done using GitLab by everyone not just me :-) 17

18 Tutorial: adding a class to GammaLib Let s create a new spectral model (say an EBL model). Here are the steps: 1. Make sure that a Redmine issue exists for the new model 2. Find a good name for the class (e.g. GModelSpectralEBL) 3. Create a new feature branch (e.g add-ebl-model) 4. Start by copying an existing class 5. Adapt code 6. Add new files to build system 7. Update ChangeLog and NEWS file 8. Commit the code 9. Push the code into your GitLab fork 10. Ask for code review or code integration 18

19 .hpp code adaption (1) mass replacement your name and current year your name and purpose of file Adapt to class name (multiple! include protection) Things for which you need the interface definition Things that need to be known to exist 19

20 .hpp code adaption (2) use explicit to avoid automatic type conversion Implement all pure virtual methods from the base class add any other methods as needed Standard methods for initialisation, copying and freeing members Add members as needed (use mutable for pre-computed and cashed values) 20

21 .hpp code adaption (3) add one-liners as inline methods Every method has a single-line brief description Provide detailed description of method, including formulae, references, etc. (everything a user of this method may need) Document return value and units Document input parameters and units 21

22 Doxygen code documentation 22

23 .cpp code adaption (1) Make sure that package configuration is available (conditional compiling) Things that are used within the file Register the model (will handle detection in XML files) In case a method throws an exception 23

24 .cpp code adaption (2) Invoke base class constructor Initialise all class members Invoke base class constructor Initialise all class members Copy all class members Free all class members 24

25 .cpp code adaption (3) Don t copy identity Assign base class members Free all class members Initialise all class members Copy all class members Free all members, including all base classes (base classes last) Initialise all members, including all base classes (base classes first) 25

26 .cpp code adaption (4) The type of your model in the XML file One block for each parameter True if parameter has an analytical gradient, false otherwise Push all parameters into the parameter stack (allows iterating over parameters) 26

27 .cpp code adaption (5) The only function you really needed to adapt to the model (and possibly optimise) Formula in LaTeX Here the real magic happens And here even more magic happens. Hopefully you can work out the analytical parameter gradients (with respect to the parameter value, i.e. parameter gradients times parameter scaling) Compile option to catch NaNs / Infs 27

28 .i code adaption (1).i files are interface files processed by SWIG to create the gammalib Python module. SWIG generates a xxx_wrap.cpp and a xxx.py file for each module xxx. The GModelSpactralEBL.i file is part of the model module. Special SWIG directive 28

29 .i code adaption (2) Almost identical to the public class definition in the.hpp file No = operator No print() method Extensions can make the class more Pythonic (also extension written in Python possible) 29

30 Add new files to build system Update the following files and type automake &./configure include/gammalib.hpp include/makefile.am pyext/gammalib/model.i src/model/makefile.am 30

31 Tutorial: adding a new ctool Let s create a new tool to simulate pointings. Here are the steps: 1. Make sure that a Redmine issue exists for the new tool (#1632) 2. Find a good name for the tool (e.g. ctpntsim) 3. Create a new feature branch (e.g add-ctpntsim) 4. Start by copying an existing tool! $ cp -r src/ctobssim src/ctpntsim $ cp pyext/ctobssim.i pyext/ctpntsim.i $ cp doc/source/users/reference_manual/ctobssim.rst doc/source/ users/reference_manual/ctpntsim.rst 5. Rename files! $ mv src/ctpntsim/ctobssim.cpp src/ctpntsim/ctpntsim.cpp $ mv src/ctpntsim/ctobssim.hpp src/ctpntsim/ctpntsim.hpp $ mv src/ctpntsim/ctobssim.par src/ctpntsim/ctpntsim.par 6. Adapt code 7. Add new files to build system 8. Update ChangeLog and NEWS file 9. Commit the code 10. Push the code into your GitLab fork 11. Ask for code review or code integration 31

32 main.cpp code adaption The executable for a ctool allocates an instance of the tool, executes it and deletes the instance Each ctool is a C++ class Allocation Execution Deletion 32

33 ctpntsim.hpp code adaption Set tool name and version Choose appropriate base class 33

34 ctool base classes Tools dealing with observation containers Tools doing max. likelihood fitting 34

35 ctpntsim.cpp code adaption (1) Use tool name and version Allows passing an observation container upon construction Allows specification of command line parameters 35

36 ctpntsim.cpp code adaption (2) Every tool has a run() method Standard switch-on of debug mode Every tool first reads the user parameters Any saving is not done in the run() method but a dedicated save() method (allows in-memory handling) 36

37 ctpntsim.par file adaption Define parameters as needed. Use identical names than in other tools for identical meanings. Standard parameters 37

38 ctpntsim.rst file adaption Try to write text in block that fits in 80 characters (also used for text shown with help option). Describe meaning of each parameter Verbosity definition (try to follow this logic in your code): 0 : SILENT 1 : TERSE 2 : NORMAL 3 : VERBOSE 4 : EXPLICIT 38

39 ctpntsim.i code adaption Similar to the SWIG file for a GammaLib class 39

40 Add new files to build system Update the following files and type automake &./configure src/makefile.am src/ctpntsim/makefile.am pyext/ctools/tools.i configure.ac 40

41 Tutorial: adding a new cscript Let s create a new script to simulate pointings. Here are the steps: 1. Make sure that a Redmine issue exists for the new script (#1632) 2. Find a good name for the script (e.g. cspntsim) 3. Create a new feature branch (e.g add-cspntsim) 4. Start by copying an existing script! $ cp cscripts/csspec.py cscripts/cspntsim.py $ cp cscripts/csspec.par cscripts/cspntsim.par $ cp doc/source/users/reference_manual/csspec.rst doc/ source/users/reference_manual/cspntsim.rst 5. Adapt code 6. Add new files to build system 7. Update ChangeLog and NEWS file 8. Commit the code 9. Push the code into your GitLab fork 10. Ask for code review or code integration 41

42 cspntsim.py code adaption (1) Make Python script executable Derive class from ctools.cscript base class Use astropy-style docstrings (see coding conventions) Set name and version strings Call base class constructor 42

43 cspntsim.py code adaption (2) Code looks very similar to C++ example Execute method calls run() and save() 43

44 Add new files to build system Update the following files and type automake &./configure cscripts/ init.py.in cscripts/makefile.am 44

45 Fundamentals of software testing (from Unit testing: Validate that each unit (typically method of C++ or Python class) performs as designed. Integration testing: Combine individual units and test as a group. Expose faults in the interaction between the units. System testing: Test complete integrated software. Evaluate system s compliance with specified requirements (e.g. Can we generate a sky map? Can we fit a spectrum?) Acceptance testing: Test system for acceptability. Evaluate system s compliance for business requirements and assess whether acceptable for delivery (e.g. Can a Scientists perform a data analysis?) 45

46 Testing in GammaLib and ctools GammaLib includes classes for implementing tests GTestCase: implements one test GTestSuite: implements a collection of tests (for example all tests of a given module) GTestSuites: implements a container of test suites (for example all tests for a given module, all ctools tests, etc.) 46

47 Setting up a test suite in C++ Set all test methods One test method (for example one per class) Test assertions, values and exceptions Allocate and append test suite to container (TestGSupport class derives from GTestSuite base class) Run tests Write test report into XML file (JUnit format) 47

48 Setting up a test suite in Python Python test class derives from GPythonTestSuite Set all test methods Test method Test case 48

49 49