LTSI kernel / Yocto Validation Proposal

Transcription

1 LTSI kernel / Yocto Validation Proposal August 2, 2013 LTSI Testing workshop COGENT EMBEDDED INC. 1

2 Introduction LTSI, Yocto becoming de-facto embedded Linux standards!!! Is Quality an issue? Can we make things better? Find and Fix bugs WORK HARD COGENT EMBEDDED INC. 2

3 LTSI, Yocto Quality? What is Quality? What Quality is about? Good Examples: Matching Product Requirements Compliance to Specifications Conformance to Standards Can we define LTSI/Yocto Requirements/ Specifications? What is important for LTSI/Yocto? What needs to be tested? COGENT EMBEDDED INC. 3

4 Two approaches. Approach A Run and see LTP, bonnie++... N Tests PASSED X tests Failed!!! Now What? Find and pick some open source tests/ test suites Run tests. Collect results Study failures/ anomalies, decide what is relevant COGENT EMBEDDED INC. 4

5 Two approaches. Approach B Define, Create, Run and Match Serial, Enet, SATA CPUFreq, PREEMPT_RT Create CPUFreq tests Use netperf and bonnie++ Set thresholds N Tests PASSED X tests Failed!!! Quality Metrics efine requirements: unckonal, Stability, etc. Create Test Coverage Run Tests. Collect Results Match results with Requirements COGENT EMBEDDED INC. 5

6 Formal Specifications & Open Source Success Stories Google Android Compatibility Definition (CDD) and Compatibility Test Suite (CTS) Hardware Certification Programs (by Linux Desktop/Server vendors). (think of additional examples)? Can we create one for LTSI? COGENT EMBEDDED INC. 6

7 5 Non-easy Steps to build LTSI QUALITY Step 1. Define what we test LTSI members provide their requirements/ expectations Examples: Vendor A: need tests for SOC B - serial, camera, storage, video Vendor C: need tests for SOC D power management Vendor: need tests for specific configuration options Outcome. Formal LTSI QUALITY spec: May have 3-4 categories: SHOULD/MUST/MAY, etc. Covers: list of SOCs/boards, peripherals, kernel features, expected performance thresholds COGENT EMBEDDED INC. 7

8 5 Non-easy Steps to build LTSI QUALITY Step 2. Define how we test Based on LTSI QUALITY spec we find/define test coverage Examples: Ethernet: may rely on popular tools (ping, ethtool, ifconfig, iperf/netperf), just need to identify specific parameters, Power Management: may need to create tests/test cases from the scratch Outcome. Formal LTSI TEST COVERAGE spec: List test cases vs requirements spec Narrows details of test suites/tests, including parameters, configurations Defines thresholds for benchmarks/performance tests COGENT EMBEDDED INC. 8

9 5 Non-easy Steps to build LTSI QUALITY Step 3. Automate We can not afford to run all tests manually need to use automation (if possible) Test automation frameworks: Autotest LAVA and many others (Jenkins, etc) We believe LAVA is best candidate now (mature, growing, deployed) Certain things are still not trivial with LAVA (Cogent Embedded will contribute UI enhancements, performance analysis graphs add-ons back to Linaro/LAVA) Outcome: Expect as many tests as possible to be integrated and ready for automated / unattended execution Define procedures that would allow multiple contributors to integrate their tests COGENT EMBEDDED INC. 9

10 5 Non-easy Steps to build LTSI QUALITY Step 4. Setup. Run. Track Things to set/deploy/configure: Servers (tests automation, user/web interface, bug tracker) Boards farm Tune things fix deployment/setup issues, set performance thresholds Track results, fill in anomalies COGENT EMBEDDED INC. 10

11 How it could look when finally deployed COGENT EMBEDDED INC. 11

12 5 Non-easy Steps to build LTSI QUALITY Step 5. Establish process Finalize roles, contributors and rules: LTSI QUALITY spec owner/gatekeeper LTSI TEST COVERAGE gatekeeper Contributors / LTSI members LTSI maintainers/gatekeepers Ensure process is repeatable, applied to all LTSI releases (and pre-releases) Improve, extend specs, requirements, test coverage COGENT EMBEDDED INC. 12