Jenkins and Load Sharing Facility (LSF) Enables Rapid Delivery of Device Driver Software. Brian Vandegriend. Jenkins World.

Transcription

1 Jenkins and Load Sharing Facility (LSF) Enables Rapid Delivery of Device Driver Software Brian Vandegriend

2 Jenkins and Load Sharing Facility (LSF) Enables Rapid Delivery of Device Driver Software Brian Vandegriend Product Verification Manager, Microsemi

3 Overview How to dynamically allocate hardware evaluation boards across software developers/testers and Jenkins in order to simplify resource sharing and increase automated regression throughput Agenda: Our Build/Test Environment Old/New Methods for Allocating Boards to Users/Jenkins LSF Concepts and Short Overview of the Tool Integrating LSF with Jenkins 4 Tips for Increasing Reliability and Throughput Benefits Realized by the Team

4 Our Build and Test Environment Our group develops, tests and supports device driver software/firmware for Optical/Ethernet networking SOCs with 100 Gbps ports Device driver written in C with 150Kloc Subversion used for revision control Cloudbees Jenkins Platform is used to continuously build and test the device driver 1 master with 2 slave nodes (VM/bare-metal) Production releases are shipped every 2 to 3 weeks Continuous Delivery: Release process is automated except for posting to web portal (manual approval) Driver testing is done on lab-based boards Over 500 automated system-level tests that have a runtime of 200+ board-hours SoC Evaluation / Test board Packets Intel-based COMExpress running Linux Packet Generator / Monitor FPGA

5 Old Method: Manual sharing of boards 20 4 sites 50 engineers Boards are manually assigned to engineers 10 SW Lab 25 (Vancouver) Jobs Regressions are sta2cally assigned Problems with approach: If an engineer is not using their board, it s hard for other engineers to use it. If an engineer s assigned board is used by someone else, they typically hunt around trying to find a free board. Software regressions can t take advantage of idle boards and run in parallel.

6 New Method: Use IBM s Load Sharing Facility (LSF) tool to dynamically allocate boards 1 User submits test to queue (FIF0) 2 Tool dispatches tests to free boards Jobs 4 LSF Jenkins uses lower priority queue to make use of idle boards SW Lab 3 25 Board becomes free when test finishes (Board can only run one test at a 2me) Advantages of using a queue-based solution, such as LSF: Enhanced productivity users do not have to find/reserve boards. The system will grant a free board to the user based upon their needs. Higher reliability "problematic" boards are taken offline and the system will direct jobs to the other boards. Individual users are not impacted. Scalability as users/boards are added, no re-adjustment of board assignments is necessary. Shorter automated testing cycles and higher efficiency of boards

7 Comparing 2 Solutions: Running LSF versus using Jenkins Slaves on Boards Running LSF on Boards Using boards as Jenkins Slaves 400 LSF LSF Hosts 25 Job 1 Job Slaves Jenkins Scheduler Board resources are treated as one large resource pool If a board crashes, only 1 test result is lost Test balancing across jobs is not required as tests are dynamically allocated to boards by LSF LSF can allocate all boards to automated tests when users are not using them Boards are divided into 2 groups: 1 for Jenkins slaves and 1 for users If a board crashes, all test results are lost Tests need to be equally partitioned across jobs to maximize throughput Jenkins can t take advantage of free boards that users are not using

8 Prerequisites for using LSF for Sharing Boards Boards have a version of Linux installed (CentOs, RedHat, Fedora, and so on) à LSF sees each board as a Linux Server Boards are fairly homogenous in their configuration/hardware Many different board types will lead to a fragmented pool o LSF can easily handle multiple resource types and allocate jobs based on resource requests For our project, we ensured we had chip fuse overrides that were controllable through SW Requires users to close their debug sessions when finished to allow the board to be allocated to the next user Timeouts are enforced by LSF to ensure boards are returned back to the pool Successful adoption by the team relies on individuals to use the system and not circumvent it by logging directly into boards 8

9 LSF Concepts short normal long Queues LSF daemons run on host (sbatchd/res/lim) Hosts Hosts Cluster Each queue can enforce user limits and run times Short queue typically has a job limit of 1, run time of 1 hour and highest priority Long queue typically allows multiple jobs per user and has the lowest priority LSF uses a priority-based, fair-share algorithm to dispatch jobs to hosts Each host has a number of attributes which can be requested Proj-1 Proj-2 Resource A2ributes - Proj-1 - atom_cpu - Num_devices=3 - Greenhills - Fedora_Linux - Proj-2 - i5_cpu - Num_devices=2 - Fedora_Linux

10 Submitting Jobs using LSF Running an interactive command on a board through LSF: Ø bsub -Ip -q short -R Proj1 echo "hello world! Job <623549> is submitted to queue <short>. <<Starting on board-105>> hello world! Request specific resource constraints: Ø bsub R Proj1 && i5_cpu R num_devices>=2 xterm Ø bsub q long m board_105 test_cmd ; # requests a specific board To view job status: > bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME vandegr RUN short swbuild01 board-105 xterm

11 Monitoring Jobs/Queues/Hosts using LSF To view the list of queues and their status: Ø bqueues QUEUE_NAME PRIO STATUS JL/U NJOBS PEND RUN short 60 Open:Active normal 25 Open:Active long 10 Open:Active To view the host status: Ø Bhosts HOST_NAME STATUS JL/U MAX NJOBS RUN swbuild01 ok board-105 closed board-106 unavail

12 Boards/Jobs Administration using LSF # Print out boards in LSF cluster: > lshosts HOST_NAME model cpuf ncpus maxmem maxswp RESOURCES Board-73 Atom G 3.9G (proj1 atom) Board-38 Intel_i G 3.9G (proj2 i5) # Take board offline: Ø badmin hclose C hardware issue board-38 Kill/suspend/resume a job running in LSF: Ø bkill/bstop/bresume <job_id>

13 Integrating LSF with Jenkins for Automated Testing 1 Jenkins job starts 2 Script dispatches tests to LSF as individual run_regression python jobs with resource constraints script that creates test list Job LSF SW Lab 25 3 Script scans LSF status and log files to print out real-2me status to console log 5 Script converts test log files into JUnit format which is summarized by Jenkins 4 Script waits un2l all tests are finished

14 Enabling User-based Automated Testing through Jenkins Regression script is also used in Jenkins for users to run regression tests against their SW/FW changes Users can submit their project workspaces to Jenkins for testing Helps ensure that the software trunk remains green Parameterized build is used to pass variables to the Python script LSF runs user regressions in parallel with the main (trunk) SW regressions Making QA regressions available to all developers!

15 Tip: Using LSF to prevent corrupted boards from killing your automated regression Problem: Boards in a bad state will causes tests to fail in rapid succession Solution: Automatically take boards offline that rapidly fail tests: LSF monitors the job exit rate for boards, and closes the board if the rate exceeds a configurable threshold For example, LSF takes a board offline if 5 tests exit abnormally in less than 30 sec Run a board initialization script prior to running the test LSF executes a pre-execution script that puts the board into a known good state If the board initialization script returns an error code, LSF takes the board offline and finds a new board for the test Re-queue tests that fail because of a board issue LSF will re-run a test if it exits with a special error code(such as 99) This technique has increased overall reliability of the system!

16 Tip: Automatically Reboot/Power-cycle Bad Boards Problem: Boards taken offline by LSF need to be quickly brought back online to maximize test throughput Solution: A board monitoring script, which is run periodically by Jenkins, will reset/ power-cycle offline boards and bring them back online If a soft reset fails, then an Ethernet-based remote power switch (from Digital Loggers) is used to power-cycle the board If power-cycling fails, the Jenkins job fails with an error and notification is sent to the administrator

17 Tip: Qualify Boards Prior to using Them in Automated Regression Testing Problem: A small percentage of our boards will have manufacturing defects (primarily due to the SoC devices being removed/re-soldered Boards with hardware faults will cause a small number of tests to fail intermittently/consistently, which takes a lot of time to track down Solution: As boards are added, run a battery of working tests with a stable SW release on each board with 5 to 10 iterations Weed out problematic/failing boards and use the rest for automated testing Use LSF groups to create a group of golden boards

18 Tip: Handling Flaky Tests Problem: Flaky tests (those that can fail or pass with the same software code) make it difficult for automated regressions to be green (no failures) Our system suffers from flaky tests, similar to what is experienced by Google Google Test blog article: We see a continual rate of about 1.5% of all test runs reporting a "flaky" result. There are many root causes why tests return flaky results, including concurrency, relying on non-deterministic or undefined behaviors, flaky third party code, infrastructure problems, etc. Workaround: Regression script re-runs test failures 3 times and if all re-runs have passed, then the test result is changed from failed to skipped Test result is only modified if the test has been tagged as a flaky test This method also helps to quickly identify new test failures as consistent or intermittent failures.

19 Tip: Handling Flaky Tests (continued) Flaky tests in Jenkins can be represented by using the Skip column It doesn t seem possible to have Jenkins recognize new values for the TEST_RESULT property: <property name="test_result" value="skip"/>

20 Benefits realized by adopting LSF Our solution using LSF and Jenkins for Automated Regression testing has: Eliminated manual sharing of boards à tedious and inefficient Simplifies a users experience of finding a free board Increased reliability of the system through automatic power-cycling of corrupted boards and running a board initialization script Improved quality of code commits as users run automated regression tests on their changes through Jenkins/LSF before committing Shortened release cycles as automated regressions complete faster by gaining access to more boards

21

22 Questions? Microsemi Corporation (MSCC) offers a comprehensive portfolio of semiconductor and system solutions for communications, defense and security, aerospace and industrial markets. Products include high-performance and radiation-hardened analog mixed-signal integrated circuits, FPGAs, SoCs and ASICs; power management products; timing and synchronization devices and precise time solutions, setting the world's standard for time; voice processing devices; RF solutions; discrete components; enterprise storage and communication solutions, security technologies and scalable anti-tamper products; Ethernet solutions; Power-over-Ethernet ICs and midspans; as well as custom design capabilities and services. Microsemi is headquartered in Aliso Viejo, Calif., and has approximately 4,800 employees globally. Learn more at Microsemi Corporate Headquarters One Enterprise, Aliso Viejo, CA USA Within the USA: +1 (800) Outside the USA: +1 (949) Sales: +1 (949) Fax: +1 (949) sales.support@microsemi.com Microsemi makes no warranty, representation, or guarantee regarding the information contained herein or the suitability of its products and services for any particular purpose, nor does Microsemi assume any liability whatsoever arising out of the application or use of any product or circuit. The products sold hereunder and any other products sold by Microsemi have been subject to limited testing and should not be used in conjunction with mission-critical equipment or applications. Any performance specifications are believed to be reliable but are not verified, and Buyer must conduct and complete all performance and other testing of the products, alone and together with, or installed in, any end-products. Buyer shall not rely on any data and performance specifications or parameters provided by Microsemi. It is the Buyer s responsibility to independently determine suitability of any products and to test and verify the same. The information provided by Microsemi hereunder is provided as is, where is and with all faults, and the entire risk associated with such information is entirely with the Buyer. Microsemi does not grant, explicitly or implicitly, to any party any patent rights, licenses, or any other IP rights, whether with regard to such information itself or anything described by such information. Information provided in this document is proprietary to Microsemi, and Microsemi reserves the right to make any changes to the information in this document or to any products and services at any time without notice Microsemi Corporation. All rights reserved. Microsemi and the Microsemi logo are registered trademarks of Microsemi Corporation. All other trademarks and service marks are the property of their respective owners.