How to Write Tests for the AGL HW Test Infra. Jan-Simon Möller AGL Release Manager

Size: px

Start display at page:

Download "How to Write Tests for the AGL HW Test Infra. Jan-Simon Möller AGL Release Manager"

Charlotte Hodge
6 years ago
Views:

1 How to Write Tests for the AGL HW Test Infra AGL AMM Oct 2017 Jan-Simon Möller AGL Release Manager

2 Introduction, Topics

3 Introduction Dipl.-Ing. Jan-Simon Möller 'DL9PF' on #freenode AGL Release Manager, EG CIAT Lead

4 This talk is about: Intro AGL HW Test Infra overview Lava jobs Lava test-definitions AGL qa-testdefinitions repo AGL specifics Testing platform code (libraries/binaries) Testing single applications What about complex testsuites? A simple example & demo

5 AGL HW Test Infra overview

6 CI / Test Infra overview

7 From the AMM 2016 talk... e.g. AGL XDS WIP eval tooling Fuego WIP option / later

8 Lava jobs

9 Lava Job Definition sample-job.yaml metadata, notifier Is a yaml style file contains metadata for the job action/deploy section files to be used boot section test section action / deploy boot test

10 Metadata / Notifier Keeps track of metadata job metadata, links, labels any (text) data can be recorded Keeps track of notifiers used e.g. for KernelCI integration

11 Action section Determins what actions need to be taken to prepare for the boot of the board file downloads location of files method to share files

12 Boot section Configures boot method prompts (bootloader) bootloader commands prompts (login) login username test overlay transfer

13 Test section One or multiple inline from git repo uses yaml files lava-test-* are markers for visualizing in LAVA for visualizing in kernelci for cross-referencing

14 Test section details (inline/git) - test: [..] definitions: - repository: metadata: format: Lava-Test Test Definition 1.0 name: smoke-tests-basic description: "Basic test command for AGL images" run: INLINE METHOD steps: - agl-basic-test-shell-command from: inline name: agl-dut-inline-basic path: inline/agl-dut-inline-fake-filename.yaml - repository: git://git.automotivelinux.org/src/qa-testdefinitions.git from: git path: test-suites/short-smoke/smoke-tests-basic.yaml name: smoke-tests-basic - repository: from: git path: test-suites/short-smoke/service-check.yaml name: service-check git repo method

15 AGL tooling / wrapper AGL develops a templating tool leveraging jinja2 to simplify submitting test plans to the CIAT infrastructure URL: 'create-jobs.py' can generate the lava job yaml based on cmdline input for machine Lava instance KernelCI instance rootfs/kernel/dtb tests to be executed See talk from Baylibre and separate tutorial on friday!!!

16 Lava test-definitions

17 Lava test definitions Usually hosted in git repo yaml file describing the (set of) test(s) and scripts to execute the test(s) E.g.

19 busybox.yaml metadata execution parsing of output (deprecated!)

20 busybox.sh

21 lava-test-case (inline example) lava-test-case is a helper script lava-test-case records the results of a single test case. For example: steps: - "lava-test-case simpletestcase --result pass" - "lava-test-case fail-test --shell false"

22 lava-test-case (script example) for i in cynara.socket afm-system-daemon.service; do systemctl is-active ${i} >/dev/null 2>&1 if [ $? -eq 0 ] ; then RESULT="pass" else RESULT="fail" fi lava-test-case ${i} --result ${RESULT} done

23 lava-test-case --shell (inline) #Inline shortcut. Uses return value of command for pass/fail. run: steps: - lava-test-case test-ls-command --shell ls /usr/bin/sort - lava-test-case test-ls-fail --shell ls /user/nowhere/

24 lava-test-case (measurement/units) # inline example run: steps: - "lava-test-case simpletestcase --result pass" - "lava-test-case bottle-count --result pass \ --measurement 99 --units bottles"

25 lava-background-process-start This starts a process in the background The arguments are: The name that is used to identify the process later in lava-backgroundprocess-stop The command line for the process to be run in the background for example: steps: - lava-background-process-start MEM --cmd "free -m grep Mem >> /tmp/musage" - lava-background-process-start CPU --cmd "grep 'cpu ' /proc/stat"

26 lava-background-process-stop This stops a process previously started in the background using lava-background-process-start. The user can attach files to the test run if there is a need. The arguments are: The name that was specified in lava-background-process-start (optional) An indication that you want to attach file(s) to the test run with specified mime type. For example:

27 lava-background-process-stop steps: uname -a - lava-background-process-stop CPU - lava-background-process-stop MEM \ --attach /tmp/musage text/plain \ --attach /proc/meminfo application/octet-stream

28 AGL qa-testdefinitions git repo

29 qa-testdefinitions git repo src/qa-testdefinitions> tree is the repo holding test definitions for AGL and what we can use within the LAVA job template We host the scripts in common/scripts and the yaml file under test-suites/<name>/. common scripts busybox.sh lshw.sh service-check-gfx.sh service-check-nogfx.sh doc skeleton.sh skeleton.yaml LICENSE README.md staging kernel-version.yaml scripts kernel-version.sh test-suites short-smoke check the doc/ folder busybox.yaml submit your tests as usual through gerrit smoke-tests-basic.yaml service-check-nogfx.yaml service-check.yaml 7 directories, 14 files

30 Testing platform code

31 Platform components A 'platform component' in AGL is software that comes as part of the profile/packagegroups into the image. These are the essential tools, the standard libraries, drivers, the middleware. AGL Applications are no platform components (even if we have yocto recipes for them) E.g. glibc, basic userspace tools, application framework, support libraries (libxml2), wayland/weston, qt5

32 Testing platform components For platform components, there is an excellent mechanism available using the Yocto Project. It is called 'ptest'. A ptest is a runtime test that can be defined within the YP recipe. E.g. the 'make test' of the software package. It will produce a separate 'ptest' sub-package that can be installed and executed with 'ptest-runner' at runtime. Thus all of our platform components should ship such a ptest! No excuses.

33 Wrapping ptest for CIAT We work on a method to wrap the ptest's for use in the AGL CIAT infrastructure. This means that if your component provides a (self-)test and uses the YP ptest methods, we can easily/automagically add it to the CI Loop and execute it on all targets. The wrapping is mainly needed to facilitate the aggregation/evaluation/visualization of the results.

34 Example of ptest in bitbake recipe Example from poky/meta/recipes-support/attr/acl_ bb SRC_URI += "file://run-ptest" [ ] inherit ptest do_install_ptest() { tar -c --exclude=nfs test/ ( cd ${D}${PTEST_PATH} && tar -xf - ) mkdir ${D}${PTEST_PATH}/include cp ${S}/include/builddefs ${S}/include/buildmacros \ ${S}/include/buildrules ${D}${PTEST_PATH}/include/ } RDEPENDS_${PN}-ptest = "acl bash coreutils perl perl-module-filehandle \ perl-module-getopt-std perl-module-posix shadow"

35 Example of ptest in bitbake recipe poky/meta/recipes-core/kbd/kbd_2.0.4.bb SRC_URI += "file://run-ptest" inherit ptest do_compile_ptest() { oe_runmake -C ${B}/tests dumpkeys-fulltable alt-is-meta } do_install_ptest() { install -D ${B}/tests/Makefile ${D}${PTEST_PATH}/tests/Makefile sed -i -e '/Makefile:/,/^$/d' -e '/%: %.in/,/^$/d' \ -e '/libkeymap_.*_sources =/d' -e '/$(EXEEXT):/,/^$/d' ${D}${PTEST_PATH}/tests/Makefile find ${B}/tests -executable -exec install {} ${D}${PTEST_PATH}/tests \; find ${S}/tests $ -name \*.map -o -name \*.bin -o -name \*.output $ -exec install {} ${D}${PTEST_PATH}/tests \; install -D -m 755 ${S}/config/test-driver ${D}${PTEST_PATH}/config/test-driver }

36 run-ptest Is a simple wrapper script > cat poky/meta/recipes-core/kbd/kbd/run-ptest #!/bin/sh CURDIR=$(dirname `readlink -f $0`) make -k -C ${CURDIR}/tests check-tests > cat poky/meta/recipes-support/attr/files/run-ptest #!/bin/sh make -C test -k tests root-tests sed \ -e 's ^\[.*\] $.*$ -- ok$ PASS: \1 ' \ -e 's ^\[.*\] $.*$ -- failed FAIL: \1 '

37 Testing single applications

38 AGL application lifecycle The basic application lifecycle can be easily tested with a lava job: uninstall / install start / stop

39 Functional tests for AGL applications The more interesting case is to do actual functional tests of the applications. Will likely depend on the upcoming appfw introspection Applications will need to define these tests We need a way to ship these (logic choice would be also as wgt) TODO!!

40 What about complex testsuites?

41 Complex testsuites Multiple options available depends on requirements Fuego can e.g. run most of LTP can do sophisticated parsing and post-processing we want to capture the results also in the AGL KernelCI currently does not support multi-node test currently does not support 'distributed'/remote tests Lava has multi-node capabilities can request and drive tests across multiple DUTs (local/remote/distributed) can handle tests with different external HW attached (labels) result is limited to pass/fail or a value for measurements, but we can upload result files for later processing work to be done to support all sorts of use-cases anyway regardless of tool!

42 A simple example & demo

43 The skeleton example We'll create a hello-world testdefinition consisting out of: the yaml file (=skeleton.yaml) the script running the test(s) (=skeleton.sh)

44 skeleton.yaml (part 1) metadata: name: skeleton format: "Lava-Test-Shell Test Definition 1.0" description: "A test skeleton" os: - openembedded scope: - functional

45 skeleton.yaml (part 2) run: steps: - "cd doc/" - "./skeleton.sh"

46 skeleton.sh #!/bin/bash result=unkown get_result () { if [ $? -eq "0" ]; then echo "pass" else echo "fail" fi } # Reporting commands results as they get executed lava-test-set start print-to-log echo "Hello" result=$(get_result) lava-test-case echo-hello --result $result echo "World!" result=$(get_result) lava-test-case echo-world --result $result ls result=$(get_result) lava-test-case ls --result $result lava-test-set stop print-to-log exit 0

47 Demo

48 Questions? Go visit the related talks and the friday tutorial!

49 Links/References: AGL repos: Lava/Linaro repo with testdefinitions:

Testing your AGL, yocto ptest, lava and more

Testing your AGL, yocto ptest, lava and more ALS 2018 Jan-Simon Möller Release Manager, AGL, The Linux Foundation jsmoeller@linuxfoundation.org, DL9PF @IRC and elsewhere Image: public domain Dipl.-Ing.