Automation beyond Testing and Embedded System Validation Embedded Linux Conference Europe Jan Lübbe <j.luebbe@pengutronix.de> Slide 1 - http://www.pengutronix.de 2017-10-23
Some Background Embedded Linux integration and development for custom hardware Using Linux mainline, mesa, wayland, gstreamer, Qt, chromium, Everything changes all the time Updates break user-visible features Kernel and application level testing solved with Jenkins & LAVA Slide 2 - http://www.pengutronix.de 2017-10-23
A Short Survey Who has developed embedded Linux systems? rolled out a major base-system update? updates the base-system at least once a year? Who has automated tests for: The application? and the kernel (-drivers)? and the update installer? and the rollback mechanism? What do you use (in-house, Jenkins, LAVA, )? Slide 3 - http://www.pengutronix.de 2017-10-23
Current State Test automation: LAVA, Fuego, autotest, avocado, TI VATF, U-Boot test tool, CI-RT R4D, Baylibe Lab in a Box, Development automation: scripting via SSH expect Production automation: flash images via robot SoC-vendor-sepecific tools (running on windows) ad-hoc scripting Slide 4 - http://www.pengutronix.de 2017-10-23
Shortcomings Large overhead for running and writing a single test painful to use during iterative development Limited control over the target from the individual test no reboots no easy control over IO (buttons, dip switches, USB, ) Hard to reuse for other use-cases and one-off tools git bisect Slide 5 - http://www.pengutronix.de 2017-10-23
Goals Make automation useful during normal iterative development Upload bootloader via USB Control distributed equipment Easy test loops Support the same tests and tools from a CI environment Make it easy to extend and embed Connect/automate existing tools (LTP, ) Slide 6 - http://www.pengutronix.de 2017-10-23
Reduced Scope and Reusability no integrated build system (unlike Fuego) no integrated test runner (unlike LAVA, autotest, many others) use Jenkins instead or use from shell no fixed boot process (all? others) use pytest and/or custom scripts no scheduler (unlike LAVA, Fuego) use OE/PTXdist/buildroot instead full control from client code not only for testing Slide 7 - http://www.pengutronix.de 2017-10-23
HW/SW Control as a Library Embedded system testing feels like pure SW testing High-Level: Similar to what I would tell a colleague to do Slide 9 - http://www.pengutronix.de 2017-10-23
Architecture Target API Strategy Bootloader Shell Console Serial Slide 10 - http://www.pengutronix.de 2017-10-23 Power Fastboot Bootstrap PDU Fastboot USB-Loader
Architecture Target API Strategy Bootloader Shell Serial Power Fastboot Bootstrap PIO Serial PDU Fastboot USB-Loader 1W-PIO Slide 11 - http://www.pengutronix.de 2017-10-23
CI for Embedded Systeme - Problems? Du Server Entwickler Slide 18 - http://www.pengutronix.de 2017-10-23?? Entwicklerboard
Linaro Automated Validation - LAVA Distributed Kernel CI Boarst deticated to LAVA (except maintainance mode) Large overhead for interactive use Slide 20 - http://www.pengutronix.de 2017-10-23
Labgrid Pytest plugin Python - Pytest Fixtures pytest-entry point no integrated build system yaml configuration examples config.yaml prepared test suites Slide 21 - http://www.pengutronix.de 2017-10-23
Labgrid Remote Control Coordinator Exports Exporter Access Exports Client Exporter Access Direct resource access Exports Client Slide 22 - http://www.pengutronix.de 2017-10-23 Exporter
Demo Slide 31 - http://www.pengutronix.de 2017-10-23
Slide 32 - http://www.pengutronix.de 2017-10-23
Slide 33 - http://www.pengutronix.de 2017-10-23
Currently Working Remotly control boards in lab from CLI (console, power, BL upload, fastboot) Run pytest against local and remote boards Run tests from Jenkins and collect results via Junit-XML Ad-Hoc automation: git bisect, reproducing sporadic errors Automatic factory installation via USB directly from built BSP Slide 36 - http://www.pengutronix.de 2017-10-23
Next Steps Remote target reservation (for use with Jenkins CI) Improved logging and reports priorities (use ResetProtocol instead of PowerProtocol when available) preemption (handle unexpected state changes) Automatic integration tests for RAUC with QEmu in Jenkins Slide 37 - http://www.pengutronix.de 2017-10-23
Discussion @shoragan, +JanLübbe-jlu Slide 38 - http://www.pengutronix.de 2017-10-23