LTSI Project Update. LTSI Kernel, How We Can Help Automotive Industries. Hisao Munakata, Tsugikazu Shibata

Transcription

1 LTSI Project Update LTSI Kernel, How We Can Help Automotive Industries Hisao Munakata, Tsugikazu Shibata Linux Foundation Consumer Electronics working group July 1st / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

2 Who am I? (Munakata) From embedded SoC provider company Renesas Linux Foundation CE 1 working Gr Steering committee member, LF/CEWG Architecture Gr co-chair One of LF/CEWG LTSI 2 project initial proposer At my company, I had been encouraging my team developers to send a patches upstream Also I have supported various CE customers who develop digital-tv, Blu-ray recorder and Smart-phone 1 CE = consumer electronics 2 LTSI =Long Term Support Initiative 2 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

3 LTSI kernel February 24, 2014 LTSI > (EOL) LTSI > 3481 (update) 3 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

4 kernel release mechanism Messages from Greg KH Whoops, Is it too late? Why you should consider adopting LTS? 4 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

5 Upstream kernel release mechanism Messages from Greg KH Whoops, Is it too late? You can find 1)latest released, 2)under development (=mainline, next), and several stable kernels 5 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

6 kernel release mechanism Messages from Greg KH Whoops, Is it too late? The release record of 30 series upstream kernel version release date duration v days v days v days v days v days v days v days v days v days v days v days v days v days v days v days Ṛelease happened regularly at around every 70 days 6 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

7 kernel release mechanism Messages from Greg KH Whoops, Is it too late? However, not all kernels are maintained for longterm version maintenance status v31 maintained till 319, then now EOL v32 longterm (3255), kept maintained (by Debian) v33 maintained till 338, then now EOL v34 longterm (3495), kept maintained v35 maintained till 357, then now EOL v36 maintained till 3611, then now EOL v37 maintained till 3710, then now EOL v38 maintained till 3813, then now EOL v39 maintained till 3911, then now EOL v310 longterm stable (31045), kept maintained v311 maintained till 31110, then now EOL v312 longterm stable (31223), kept maintained (by???) v313 stable release (31311), till 315 released v314 next longterm stable version (3149) v315 latest release (3152), will be maintained as stable till 317 is out 7 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

8 kernel release mechanism Messages from Greg KH Whoops, Is it too late? Stable release include MUST APPLY essential fixes version fixes v30 -> v ,953 v31 -> v v32 -> v3260 5,001 v33 -> v v34 -> v3495 4,506 v35 -> v v36 -> v v37 -> v v38 -> v v39 -> v v310 -> v ,970 v311 -> v v312 -> v ,314 v313 -> v v314 -> v / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

9 kernel release mechanism Messages from Greg KH Whoops, Is it too late? Longterm stable (LTS) kernel release cadence Target kernel selection rules Maintainer will choose one LTS version per year Maintain it for 2 years from its original release LTSI-30 is moved to EOL when 310 became new LTS Then, we have 2 LTS kernels versions like 34 and / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

10 kernel release mechanism Messages from Greg KH Whoops, Is it too late? LTS/LTSI maintainer, Greg says 30 moves to EOL 10 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

11 kernel release mechanism Messages from Greg KH Whoops, Is it too late? Greg also announced longterm for 2014 is / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

12 kernel release mechanism Messages from Greg KH Whoops, Is it too late? Upstream kernel 310 development (is done) Whoops, we can not submit our latest device support code to 30 kernel now! Yes, that is true, because item date kernel 310 merge window open kernel 310 merge window close kernel 310 release As upstream 310 patch merge window is already closed, there is no chance to add your code to upstream kernel Thus a cutting-edge silicon release after development cycle can not be supported in longterm 310 kernel This might be problematic for embedded industry Linux adopter 12 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

13 LTSI 310 development result Governance 13 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

14 LTSI-310 development history LTSI 310 development result Governance item date kernel 310 merge window open kernel 310 merge window close kernel 310 release Announce of 2013 LTS kernel version LTSI-310 git tree open becomes LTS (=312 release) LTSI-310 merge window open patch collection period 75 days LTSI-310-rc1 (=merge window close) validation period 26 days LTSI-310 release / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

15 Major contributor for LTSI-310 LTSI 310 development result Governance Contributor Patch count Darren Hart (intel) 1,197 Simon Horman (for Renesas) 1,122 Daniel Sangorrin (Toshiba) 123 Patrik Jakobsson (for intel) 46 Mark Brown (linaroorg) 11 Greg Kroah-Hartman (Linuxfoundation) 11 Total 2, / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

16 Major achievement of LTSI 310 LTSI 310 development result Governance LTTng Power efficient workqueues Intel's BayTrail support Intel's Minnowboard support Renesas's R-Car H2/M2 series support backported from the latest mainline Xilinx Zynq board support 16 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

17 LTSI 310 development result Governance Hot news! Greg announced 2014 LTS will be 314 At the ELC2014 conference LTSI workshop, Greg stated next LTS (and LTSI) kernel version would be 314 item date kernel 314 merge window open kernel 314 merge window close kernel 314 release LTSI-314 merge window open (target) patch collection period 70 days LTSI-314-rc1 (=merge window close, target) validation period 50+ days LTSI-314 release (target) ? Please be ready for collecting patches to send LTSI-314 now! 17 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

18 LTSI 310 development result Governance Hot news 2! New LTS to LTSI update reflection cycle Every stable update will be ported to existing LTSI code 18 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

19 LTSI 310 development result Governance LTSI = community LTS(longterm) + industry extra 19 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

20 LTSI 310 development result Governance Yocto and LTSI project coordination is working now 20 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

21 LTSI 310 development result Governance Discipline of LTSI project management Community LTS + industry demanded extra patches Governed by LF/CEWG Focus on kernel code a, not aiming complete BSP Therefore, can be combined with existing platform b CPU architecture and platform neutral Comply with upstream rules c Industry friendly acceptance (flexible patch forms, etc) Help CE (and others) industry to utilize Linux a device drivers are part of kernel, of course b Android, Yocto, Tizen, AGL, WebOS and others c eg signed-off-by process 21 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

22 Concept of shared test trial LTSI-30 trial result 22 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

23 Concept of shared test trial LTSI-30 trial result Why LTSI kernel validation becomes important? Upstream LTS is managed to be completely safe LTSI can based on community LTS kernel, and LTSI is the place to add various NEW things Feature back port from latest mainline (relatively safe) Industry demanded not-mainlined (but commonly used) open source project code Privately maintained bug-fix code (may be valuable) Privately developed feature code We want to validate LTSI kernel does not include any bug or regression against the community LTS code 23 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

24 Concept of shared test trial LTSI-30 trial result Beyond the LTS(I) kernel use, share the test case! New value opportunity of sharing the kernel test case Now many industry start using LTS and LTSI kernel Each company may spend a lot of time for validation Some of fundamental kernel feature test might be duplicated common kernel function test (detail later) common kernel benchmark test (detail later) common compatibility conformance test Now we can consider sharing the (part of) kernel test case on top of LTS(I) kernel across the industry We need to assign appropriate OSS license to each test case itself so the we can share them 24 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

25 Concept of shared test trial LTSI-30 trial result Design target of shared LTSI test environment feature Fully automated execution (nightly run) Easy to manage operation (add/edit test case) Trend monitoring capability (to catch the regression) User friendly interface (web access, GUI front end) operation local text execution (can install to your computer) test case sharing mechanism test result sharing mechanism (future work) can penetrate to the upstream kernel development use 25 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

26 Current shape Why you should consider adopting LTS? Concept of shared test trial LTSI-30 trial result We did trial run during LTSI-310 development period Cogent Embedded / Renesas worked together We will donate environment to public so that anyone can execute pre-build test and write own test case Jenkins front-end (test automation) Customizations (UI/look&feel, representation) Open Source Test Suites (public, popular) are integrated Some private test suites (shell scripts) tested 26 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

27 Implementation Why you should consider adopting LTS? Concept of shared test trial LTSI-30 trial result Jenkins brings - mature, robust platform to manage & distribute jobs (could be test/tests suites distributed across various platforms) Tests/test suites wrapped into shell scripts Idea is to keep environment as simple/straightforward as possible: Every step is a script: build test, deploy on target, run, collect results, parse results, cleanup It should be possible to trigger scripts, run tests, collect results without complex Jenkins setup Targets are connected with server(s) via network (eg debug ethernet) and/or serial Test results, status, statistics, Target configuration, etc - visualized by Jenkins (accessible via web interface) 27 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

28 Concept of shared test trial LTSI-30 trial result Screen shot (Jenkins Web based test controller UI) 28 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

29 Concept of shared test trial LTSI-30 trial result Currently integrated 28 automated benchmarks 1 aim7 2 blobsallad 3 bonnie 4 Dhrystone 5 cyclictest 6 fio 7 GLMark 8 ebizzy 9 ffsb 10 hackbench 11 gtkperf 12 himeno 13 Interbench 14 IOzone 15 iperf 16 Java 17 linpack 18 lmbench2 19 nbench-byte 20 netperf 21 netpipe 22 OpenSSL 23 reboot 24 Stream 25 signaltest 26 tiobench 27 Whetstone 28 x11perf 29 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

30 Concept of shared test trial LTSI-30 trial result Currently integrated 33 automated tests 1 aiostress 2 bzip2 3 expat 4 cmt_renesas 5 crashme 6 ipv6connect 7 fontconfig 8 ft2demos 9 glib 10 jpeg 11 linus_stress 12 LTP-DDTDevices 13 LTP-DDTIpc 14 LTP-DDTMath 15 LTP-DDTMm 16 LTP-DDTNptl 17 LTP-DDTPipes 18 LTP-DDTSyscalls 19 LTP-DDTTimers 20 LTPDevices 21 LTPFilesystem 22 LTPOpen_Posix 23 netperf 24 OpenSSL 25 pi_tests 26 posixtestsuite 27 rmaptest 28 scifab_renesas 29 scrashme 30 sdhi_0_renesas 31 stress 32 synctest 33 zlib You can integrate your own test case (public/private) here 30 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

31 Target configuration Concept of shared test trial LTSI-30 trial result Target abstraction is just a set of environment variables Target Architecture (ARM, x86, MIPS) / toolchain path IP addr/login pair if target is connected/controlled via TCP/IP (SSH, FTP, telnet) or serial port parameters Target power-cycle settings Target Linux distro-specific settings (if any) temporary folder for test suites/logs command to grab system logs Target pre-setup required Bootcode + kernel (under test) + minimal distro 31 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

32 Result 1 : LTS 310 vs LTSI-310 Concept of shared test trial LTSI-30 trial result Use Intel Atom Minnow board Minnow is a good platform to compare LTS and LTSI (at least works without additional patch lifting, headaches, etc) - example how mainline support should be done No significant deviation in results observed, no major regressions Anomaly found: fio-158 fails when running on msd card using LTSI-310 (worked ok with LTS-310), on the other side, newer version of fio does not show similar anomaly 32 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

33 Result 2 : LTSI-34 vs LTSI-310 Concept of shared test trial LTSI-30 trial result Use Renesas R-CarM2 Koelsch board Renesas is using LTSI kernels as a baseline for product-quality BSPs delivered to customers R-Car M2 - good candidate to compare LTSI34 and LTSI310 code bases No significant deviation in results observed, no major regressions Anomaly found: cyclic test fails (will study further/deeper soon) 33 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

34 Concept of shared test trial LTSI-30 trial result Our struggles while we did LTSI test trial run We still don't have common understand/requirements What we want to test and how we want to do this (need to collect feedback), example: some members mentioned - they don't see sense in testing on real hardware, some - want one feature tested (eg IPv6, using TAHI tests), some - another, etc we need to have more formal approach to make testing useful A number of problems when comparing tests results Hardware support often behind, sometimes very different between various combination of linux kernels, etc difficult to find hardware platform that would be well-maintained in LTS/LTSI trees for a while Minnow - nice work, but now minnow-max is coming, etc Default configurations may get changes, behavior of some kernel features, etc especially when comparing results from older release with new release 34 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

35 Concept of shared test trial LTSI-30 trial result New/extended auto-test functionality 1/2 Report generation framework allows to generate readable/standalone reports) - completed Documentation Early version - completed Improvements - in progress Integration/tuning of new tests Renesas board-specific tests being added now A few more open source tests suites (eg dbench, etc) 35 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

36 Concept of shared test trial LTSI-30 trial result New/extended auto-test functionality 2/2 Following features will be integrated soon Serial port, ftp and telnet support ``board/target'' initialization/configuration/deployment Automated power cycle/reboot control integration Build everything from source target kernel bootcode minimal distro UI/Jenkins plugins improvements 36 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

37 conclusion Resources 37 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

38 Why you should consider adopting LTS? conclusion Resources Correct understanding of the longterm (LTS) cadence is important LF/CEWG develops LTSI version on top of community longterm kernel You can gain huge cost reduction if you can fully utilize LTS & LTSI scheme 30 longterm maintenance cycle has been moved to EOL and310 is the next longterm support target LTSI-310 was released in February 2014 And LTSI project maintainer Greg KHlately announced that next LTS(I) kernel will be 314 We have developed automated kernel test framework and tried with LTSI-310 release We are hoping to share the kernel test case on top of commonly used LTSI kernel and upstream kernel development 38 / 39 Hisao Munakata, Tsugikazu Shibata LTSI Project Update

39 project web = ltsilinuxfoundationorg LTSI process document (new) = ML ML subscription = ML archives = ML patchwork = git(each patch) = download (tar ball) = 39 / 39 twitter Hisao = Tsugikazu Shibata LTSI Project Update Resources Why you should consider adopting LTS? conclusion Resources