Ottimizzare i tempi di boot di Linux. Andrea Righi -

Transcription

1 Ottimizzare i tempi di boot di Linux

2 Agenda Overview Case study: Raspberry Pi Kernel optimizations rootfs optimizations Q/A

3 Why a faster boot? Consumer electronics products require very fast boot times (digital camera, TV, mobile phone, etc.) Fast recovery upon system failures (crashes) You can show to your friends your new awesome board booting in a couple of seconds ;-)

4 Overview of boot phases Firmware (boot-loader) Hardware probing Hardware initialization Kernel load and decompression Kernel execution Core init (start_kernel) Driver init (initcalls) User-space init /sbin/init RC scripts Application start (first user impression)

5 Generic concepts Identify boot time functionalities Measure boot time of each functionality Remove unnecessary functionalities Optimize required functions Defer loading of less-important features (modularization - LKM) Re-order initialization / parallelization Asynchronous initialization

6 Generic concepts Identify boot time functionalities Measure boot time of each functionality Remove unnecessary functionalities Optimize required functions Defer loading of less-important features (modularization - LKM) Re-order initialization / parallelization Asynchronous initialization Premature optimization is the root of all evil. -Donald Knuth

7 Case study Raspberry Pi Boot time functionality Reponsive ssh login Tools and techniques used to improve boot time Focusing on cold-boot optimizations

8 Boot steps RPi boot steps: 1st-stage boot loader from SoC BCM2835 ROM bootcode.bin: 2nd-stage boot loader from external SD card, starts the GPU start.elf: GPU firmware, starts the CPU kernel.img: Linux rootfs: external SD card application: ssh (dropbear)

9 Measure boot-time functionality (kernel) CONFIG_PRINTK_TIME Configure it in Kernel hacking section Adds timing informations to kernel messages (dmesg) # dmesg... [ ] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs [ ] calling 1 [ ] bcm_power: Broadcom power driver [ ] bcm_power_open() -> 0 [ ] bcm_power_request(0, 8) [ ] bcm_mailbox_read -> , 0 [ ] bcm_power_request -> 0 [ ] initcall bcm_power_init+0x0/0xa4 returned 0 after usecs

10 Kernel initcall tracer Introduced in Linux Add initcall_debug to the kernel boot options Allows to record the timings of each initcall in dmesg # dmesg grep " initcall " sed "s/ */ /g" sort -n -t' ' -k8 tail -5 [ ] initcall bcm2708_fb_init+0x0/0xc returned 0 after usecs [ ] initcall pty_init+0x0/0x3e0 returned 0 after usecs [ ] initcall init_kprobes+0x0/0x108 returned 0 after usecs [ ] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after usecs [ ] initcall bcm_power_init+0x0/0xac returned 0 after usecs

11 Kernel initcall tracer (example) dmesg > dmesg.log cat dmesg.log perl scripts/bootgraph.pl > /boot/dmesg.svg

12 Raspbian General-purpose distro based on Debian Optimized for the Raspberry Pi compilation settings adjusted to produce hard-float code wheezy-raspbian.img

13 Raspbian: boot time # dmesg... [ ] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ ] EXT4-fs (mmcblk0p2): recovery complete [ ] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) [ ] VFS: Mounted root (ext4 filesystem) on device 179:2. [ ] devtmpfs: mounted [ ] Freeing init memory: 128K [ ] init start [ ] udevd[154]: starting version 175 [ ] Registered led device: led0 [ ] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ ] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ ] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 [ ] Adding k swap on /var/swap. Priority:-1 extents:2 across:507900k SS [ ] init done - Shell after ~6.3sec - SSH after ~30.5sec

14 sysv-rc-conf Run-level configuration for SysV like init scripts disable npt disable plymouth disable rsync disable x11-common and lightdm disable alsa-utils disable swap (dphys-swapfile) disable checkfs disable ntp

15 Static IP vs DHCP Assign a static IP to the board (save the time to get an IP via DHCP) Disable CONFIG_IP_PNP in the kernel.config Used to mount rootfs via NFS

16 Raspbian: boot time after simple optimizations # dmesg... [ ] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ ] EXT4-fs (mmcblk0p2): recovery complete [ ] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null) [ ] VFS: Mounted root (ext4 filesystem) on device 179:2. [ ] devtmpfs: mounted [ ] Freeing init memory: 128K [ ] init start [ ] udevd[154]: starting version 175 [ ] calling leds_init+0x0/0x [ ] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs [ ] calling gpio_led_driver_init+0x0/0xc 229 [ ] Registered led device: led0 [ ] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs [ ] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ ] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ ] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 [ ] init done About 25 sec to login via ssh, instead of 30 sec (16% faster)

17 Build a minimal distro from scratch: requirements build toolchain (gcc, glibc, binutils) git://github.com/raspberrypi/tools.git HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-gnueabi) Linux (kernel) git://github.com/raspberrypi/linux.git busybox (rootfs) git://busybox.net/busybox.git dropbear (ssh server)

18 Kernel initcalls (before optimization) # dmesg grep " initcall " sed "s/ */ /g" sort -n -t' ' -k8... [ ] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs [ ] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs [ ] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs [ ] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs [ ] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs [ ] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs [ ] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs [ ] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs [ ] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs [ ] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs [ ] initcall chr_dev_init+0x0/0xd8 returned 0 after usecs [ ] initcall param_sysfs_init+0x0/0x1d4 returned 0 after usecs [ ] initcall bcm2708_fb_init+0x0/0xc returned 0 after usecs [ ] initcall pty_init+0x0/0x3e0 returned 0 after usecs [ ] initcall init_kprobes+0x0/0x108 returned 0 after usecs [ ] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after usecs [ ] initcall bcm_power_init+0x0/0xac returned 0 after usecs

19 Kernel optimizations (initcalls) disable kprobes (CONFIG_KPROBES): ~95ms reduce the number of PTYs: ~90ms CONFIG_LEGACY_PTY_COUNT=256 => CONFIG_LEGACY_PTY_COUNT=1 disable framebuffer: ~33ms remove loop device: ~5ms

20 Kernel initcalls (after optimization) # dmesg grep " initcall " sed "s/ */ /g" sort -n -t' ' -k8... [ ] initcall tun_init+0x0/0x8c returned 0 after 259 usecs [ ] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs [ ] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs [ ] initcall pty_init+0x0/0x194 returned 0 after 375 usecs [ ] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs [ ] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs [ ] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs [ ] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs [ ] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs [ ] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs [ ] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs [ ] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs [ ] initcall chr_dev_init+0x0/0xd8 returned 0 after usecs [ ] initcall param_sysfs_init+0x0/0x19c returned 0 after usecs [ ] initcall sdhci_drv_init+0x0/0xc returned 0 after usecs [ ] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after usecs [ ] initcall bcm_power_init+0x0/0xa4 returned 0 after usecs

21 Kernel optimizations (other optimizations) preset loops_per_jiffy At each boot the kernel calibrates a delay loop (used later by the udelay() function) 1 jiffy = time between 2 timer interrupts CONFIG_HZ=100 => 250ms!!! loop disable console output remove console=xxx and add quiet to the kernel boot parameters use LZO kernel decompression (CONFIG_KERNEL_LZO) LZO is a compression algorithm that is much faster than gzip, at the cost of a slightly degrade compression ratio (+10%) reduce kernel size... A smaller kernel is faster to load, less code also means smaller working set (good for caches)

22 Kernel image (before => after)

23 rootfs Generated using busybox and dropbear Hints about busybox (reduce fork()s): CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait external binaries CONFIG_FEATURE_SH_NOFORK: call <applet>_main() directly without spawning another task Build everything with -Os Use mklibs to strip system libraries (rootfs is mounted to /mnt) mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv lib2 lib After all these stops total rootfs size is ~3.2MB

24 filesystem optimizations Split the filesystem into read-only portion and read/write portion Read-only filesystem mounts faster Use Squashfs + tmpfs (or aufs)

25 root filesystem: boot time

26 Kernel image (before => after) 1.5GB 3.2MB

27 Demo: custom kernel+rootfs boot time... [ ] Freeing init memory: 92K [ ] init start [ ] waiting eth0 to come up [ ] usb 1-1: new high-speed USB device number 2 using dwc_otg [ ] Indeed it is in host mode hprt0 = [ ] hub 1-1:1.0: USB hub found [ ] hub 1-1:1.0: 3 ports detected [ ] usb 1-1.1: new high-speed USB device number 3 using dwc_otg [ ] smsc95xx v1.0.4 [ ] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24 [ ] init done [ ] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1 - Shell after ~1.2 sec - SSH after ~3.4 sec

28 References LPC: Booting Linux in 5 Seconds - Arjan van de Ven's talk: elinux wiki - Tim's fastboot tools: elinux wiki- RPi Kernel Compilation: Update on boot time reduction techniques - Michael Opdenacker Linux: Busybox: Dropbear: Raspberry Pi: ry-pi

29 Q/A You're very welcome! #bem2013