Building opensuse with link-time optimizations. Jan Hubička and Martin Liška SUSElabs

Transcription

1 Building opensuse with link-time optimizations Jan Hubička and Martin Liška SUSElabs

2 Outlilne What is link-time optimization? Link-time optimization and GCC Benchmarks Can we build opensuse with link-time optimization by default?

3 What is link-time optimization?

4 Per-file compilation File1.c GCC File1.o File2.c GCC File2.o File3.c GCC File3.o ld / gold a.out File4.c GCC File4.o

5 Link-time optimization File1.c GCC File1.o IL ld / gold a.out File2.c GCC File2.o IL File3.c GCC File3.o IL LTO plug-in link-time compiler File4.c GCC File4.o IL

6 Benefits of LTO Symbol promotion (from linker s resolution data most symbols become static ) Cross-module inlining, constant propagation Aggressive unreachable code removal Profile propagation EH optimization (propagation of nothrow ) Identical code folding Optimized code layout and more :)

7 Problems of LTO Whole toolchain has to be restructured Slow compile-edit cycle Harder bugreports (often whole program needed to reproduce issue) Not 100% transparent to user, but in most cases all one needs to do is to add -flto

8 Link-time optimization and GCC

9 Modernizing GCC (LTO perspective) 1999 (GCC 2.95): Function-at-a-time 2001 (GCC 3.0): New inliner (first high-level opt. in gcc) 2004 (GCC 3.4): Unit-at-a-time; intermodule compilation for C; Inter-procedural optimization framework 2005 (GCC 4.0): New SSA optimization framework 2006 (GCC 4.1): Inter-procedural optimizations: profile guided inlining, pure/const discovery, mod/ref, inter-procedural constant propagation, --fwhole-program combine 2008 (GCC 4.4): Inter-procedural optimization on SSA; early optimization and inlining (GCC 4.5): Basic LTO framework (5 years in development) 2011 (GCC 4.6): WHOPR (parallel link-time optimization); Firefox builds

10 Link-time optimization File1.c GCC File1.o IL ld / gold a.out File2.c GCC File2.o IL File3.c GCC File3.o IL LTO plug-in link-time compiler File4.c GCC File4.o IL

11 Parallelized Link-time optimization (WHOPR) File1.c GCC File1.o IL ld / gold a.out File2.c GCC File2.o IL File3.c GCC File3.o IL LTO plguin Local opt. File4.c GCC File4.o IL Whole Program Analysis Local opt. Local opt

12 Modernizing GCC (LTO perspective) 2012 (GCC 4.7.0): Memory use optimizations, new inliner heuristics, new inter-procedural constant propagation with clonning 2013 (GCC 4.8.0): symbol table; propagation of values passed through aggregates 2014 (GCC 4.9.0): slim LTO objects by default; on demand loading of functions; devirtualization pass; feedback directed code layout 2015 (GCC 5): Identical Code Folding; COMDAT optimization; One Definition Rule for C++; alignment propagation; correct command line options handling with LTO 2016 (GCC 6): Linker-plugin now detects type of output binary. C&Fortran type merging. Better alias anaysis 2017 (GCC 7): Inter-procedural value range propagaion; bitwise propagation 2018 (GCC 8): Early debug info. Profile representation rewrite. Function splitting now by default. Reworked runtime estimation; Malloc attribute propagation

13 GCC optimization pipeline Compile time Parser IL generation Early opts: Early Inliner Constant prop. Forwward prop. Jump threading Scalar repl. of aggr. Alias analysis Redundancy ellim. Dead store ellim. Dead code ellim. Tail recursion Switch conversion pure/const/nothrow EH optimization Profile guessing IP analysis streaming out Link-time serial Streaming Symbol in & symbols, type types streaming and declarations in + merging & link Inter-procedural (whole program) Opts: Dead symbol ellim. Symbol promotion profile analysis Identical code folding devirtualization Constant propagation const/destr merging Inlining pure/const/nothrow mod/ref comdat Partitioning streaming out Link-time parallel Stream in and apply transformations High level opts: Constant prop. Complette unroll Forward prop. Alias analysis Return slot opt. Redudancy ellim. Jump threading Dead code ellim. Conditional store ellim. Copy prop. If combine Tail recursion Copy loop headers Scalar repl. of aggr. Dead store ellim. Dead code ellim. Reassociation Sincos, bswap opt. Loop invariant motion Partial redundancy ellim. Loop splitting Unroll and jam Loop dsitribution Loop interchange... Low level opts: Common subexpression ellim. Forward propagation Copy propagation Partial Redundancy Ellim. Code hoisting Copy propagation Store motion If conversion Loop invariant motion Loop unrolling Doloop optimization Web construction Copy propagation Common subexpression ellim. Dead store ellim. Instruction combine Function partitioning Instruction splitting Live range shrinking Scheduling Register allocation Global common subexpr. Ellim. Shrink wrapping Stack adjustment opt. Register renaming Constant prop. Code reordering Scheduling X87 register stack Code/data alignment Machine dependent reorg. Code output

14 Benchmarks

15 SPECint 2006 performance (relative to GCC 6) generic native GCC 7 -O2 GCC 8 -O2

16 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast

17 SPECint 2006 performance (relative GCC -Ofast relative to to GCC 6 -O26) GCC 6 GCC 7 GCC mcf 458.sjeng 445.gobmk 403.gcc 462.libquantum 473.astar 464.h264ref 471.omnetpp 401.bzip2 483.xalancbmk 400.perlbench 456.hmmer Geomean generic native 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast

18 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast

19 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast

20 0 SPECint 2006 performance (relative to GCC 6) Clang & ICC -Ofast relative to GCC perlbench 401.bzip2 403.gcc 429.mcf 445.gobmk 456.hmmer 458.sjeng 462.libquantum 464.h264ref 471.omnetpp 473.astar 483.xalancbmk Geomean clang/flang 6 ICC generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast

21 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast

22 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto

23 SPECint 2006 performance (relative to GCC 6) GCC -Ofast -flto relative to -Ofast perlbench bzip gcc 429.mcf 445.gobmk 456.hmmer 458.sjeng 462.libquantum 464.h264ref 471.omnetpp 473.astar 483.xalancbmk Geomean generic native GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto

24 SPECint 2006 performance (relative to GCC 6) generic native 2 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto

25 SPECint 2006 performance (relative to GCC 6) generic native 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto clang/fllang 6 -Ofast -flto ICC 18 -Ofast -flto

26 SPECint 2006 performance (relative to GCC 6) Clang/flang 6 and ICC 18 relative to GCC 8 (-O2 -flto) clang/flang 6 ICC perlbench 401.bzip2 GCC 7 -O2 GCC 8 -O2 403.gcc 429.mcf 445.gobmk 456.hmmer GCC 6 -Ofast 464.h264ref GCC 8 -Ofast 458.sjeng 462.libquantum 471.omnetpp GCC 7 -Ofast 473.astar 483.xalancbmk Geomean clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto clang/fllang 6 -Ofast -flto ICC 18 -Ofast -flto generic native

27 SPECint 2006 performance (relative to GCC 6) generic native 0 GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast clang/flang 6 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto clang/fllang 6 -Ofast -flto ICC 18 -Ofast -flto

28 Profile feedback & LTO works well together 25 Performance relative to GCC 8 -Ofast LTO FDO FDO+LTO perlbench bzip2 403.gcc 429.mcf 445.gobmk 458.sjeng 462.libquantum 464.h264ref 471.omnetpp 473.astar 483.xalancbmk Geomean Hmmer benchmark has problems; was excluded. To build with profile feedback often you can use:./configure ; CFLAGS= -O2 -fprofile-generate make ; make check ; make clean ; CFLAGS= -O2 -fprofile-use make

29 SPECint 2006 code size Non-LTO LTO GCC 7 -O2 GCC 8 -O2 GCC 7 -Ofast GCC 8 -OfastGCC 8 -Ofast + FDO Clang 6 -O2 Clang 6 -Ofast Icc 18 -Ofast

30 SPECfp 2006 performance (relative to GCC 6) generic native GCC 7 -O2 GCC 8 -O2 GCC 6 -Ofast GCC 7 -Ofast GCC 8 -Ofast ICC 18 -Ofast GCC 8 -O2 -flto GCC 8 -Ofast lto ICC 18 -Ofast -flto

31 Firefox performance summary 35 Firefox performance relative to non-lto build static Profile feedback responsiveness tp5o dromaeo dom Displaylist mutate tap paint speedometer a11yr svgr_opacity tp5o ARES6 stylebench tart dromaeo css tsvgx startup time

32 Firefox performance dromaeo DOM

33 Firefox performance tp5o page responsiveness

34 Firefox binary size gcc6 -O3 -flto + FDO gcc7 -O3 -flto + FDO gcc8 -O3 -flto + FDO gcc8 -O3 gcc8 -O2 gcc8 -O3 + FDO gcc8 -Os gcc8 -O3 -flto gcc8 -O2 -flto gcc8 -O3 -flto + FDO gcc8 -Os -flto clang6 -O3 clang6 -O3 + FDO clang6 -O2 clang6 -Os clang6 -Oz EH data relocations text clang6 -O3 -flto=thin clang6 -O3 -flto clang6 O3 -flto + FDO clang6 -O3 -flto=thin + FDO clang6 -O2 -flto clang6 -Oz -flto=thin clang6 -Oz -flto

35 Firefox binary size gcc6 -O3 -flto + FDO gcc7 -O3 -flto + FDO gcc8 -O3 -flto + FDO gcc8 -O3 gcc8 -O2 gcc8 -O3 + FDO gcc8 -Os gcc8 -O3 -flto gcc8 -O2 -flto gcc8 -O3 -flto + FDO gcc8 -Os -flto clang6 -O3 clang6 -O3 + FDO clang6 -O2 clang6 -Os clang6 -Oz EH data relocations text clang6 -O3 -flto=thin clang6 -O3 -flto clang6 O3 -flto + FDO clang6 -O3 -flto=thin + FDO clang6 -O2 -flto clang6 -Oz -flto=thin clang6 -Oz -flto

36 Firefox binary size gcc6 -O3 -flto + FDO gcc7 -O3 -flto + FDO gcc8 -O3 -flto + FDO gcc8 -O3 gcc8 -O2 gcc8 -O3 + FDO gcc8 -Os gcc8 -O3 -flto gcc8 -O2 -flto gcc8 -O3 -flto + FDO gcc8 -Os -flto clang6 -O3 clang6 -O3 + FDO clang6 -O2 clang6 -Os clang6 -Oz EH data relocations text clang6 -O3 -flto=thin clang6 -O3 -flto clang6 O3 -flto + FDO clang6 -O3 -flto=thin + FDO clang6 -O2 -flto clang6 -Oz -flto=thin clang6 -Oz -flto

37 Firefox binary size gcc6 -O3 -flto + FDO gcc7 -O3 -flto + FDO gcc8 -O3 -flto + FDO gcc8 -O3 gcc8 -O2 gcc8 -O3 + FDO gcc8 -Os gcc8 -O3 -flto gcc8 -O2 -flto gcc8 -O3 -flto + FDO gcc8 -Os -flto clang6 -O3 clang6 -O3 + FDO clang6 -O2 clang6 -Os clang6 -Oz EH data relocations text clang6 -O3 -flto=thin clang6 -O3 -flto clang6 O3 -flto + FDO clang6 -O3 -flto=thin + FDO clang6 -O2 -flto clang6 -Oz -flto=thin clang6 -Oz -flto

38 Firefox binary size gcc6 -O3 -flto + FDO gcc7 -O3 -flto + FDO gcc8 -O3 -flto + FDO gcc8 -O3 gcc8 -O2 gcc8 -O3 + FDO gcc8 -Os gcc8 -O3 -flto gcc8 -O2 -flto gcc8 -O3 -flto + FDO gcc8 -Os -flto clang6 -O3 clang6 -O3 + FDO clang6 -O2 clang6 -Os clang6 -Oz EH data relocations text clang6 -O3 -flto=thin clang6 -O3 -flto clang6 O3 -flto + FDO clang6 -O3 -flto=thin + FDO clang6 -O2 -flto clang6 -Oz -flto=thin clang6 -Oz -flto

39 Summary LTO now works and scale for large applications (Firefox, Libreoffice, ) Important size optimization especially for large C++ programs Often important performance optimization especially when combined with profile feedback Space for future improvements (both on GCC side and re-optimization of programs to LTO model)

40 LTO in opensuse Factory

41 LTO in opensuse Factory A new staging project (opensuse:factory:staging:n) Setting: Optflags: * -flto ~80 packages failed (out of all ~2300) Branch all failing packages and disable LTO LTO Factory (w/ some disabled packages) can run openqa testsuite (with a small fallout)

42 Package statistics Total ELF files in distribution: ~6700 Reduction: 1839 MB to 1736MB (-5.6%) libmergedlo.so: reduction 10MB (-16%) Biggest reduction: mysql_upgrade (-92%) Innochecksum (-94%) mbstream (-94%)

43 Issues Argyllcms: lto1: fatal error: multiple prevailing defs for 'xcal_read_icc': LD PR: GCC LTO miscompilation: in shared libraries: asm (".symver New symver function attribute must be added (GCC 9.1.0)

44 Issues (cont.) static libraries (*.la): e2fsprogs, btrfsprogs, Fat LTO objects must be used (-ffat-lto-objects) LTO elf sections should be stripped OBS sanitizer should be extended LTO mode can combine both LTO objects and assembly objects

45 Issues (cont. 2) LTO warnings (-Wodr,...): ltrace: error: type of 'filter_matches_symbol' does not match original declaration [-Werror=lto-type-mismatch] gdb: error: type 'struct ipa_sym_addresses' violates the C++ One Definition Rule [-Werror=odr] mpir: weak configure script that scans *.o file as a blob weak support for LTO debug info in dwz tool Maybe higher memory constaints for selected packages

46 Issues (cont. 3) Usage of top-level assembler ( Example of syscall.cc in Chromium project: asm volatile(".text\n" ".align 16, 0x90\n" ".type "SyscallAsm:.cfi_startproc\n" Error:nacl_helper.ltrans1.ltrans.o: In function `playground2::sandboxsyscall(int, long, long, long, long, long, long)': nacl_helper.ltrans1.o:(.text+0x4503): undefined reference to `SyscallAsm'

47 Histogram of text segment sizes ,000,000 10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000

48 Conclusion LTO looks mature enough to be used by default Do we want it for opensuse Factory? For the future, can we offer it also for SLE?

49 License This slide deck is licensed under the Creative Commons Attribution-ShareAlike 4.0 International license. It can be shared and adapted for any purpose (even commercially) as long as Attribution is given and any derivative work is distributed under the same license. Details can be found at General Disclaimer This document is not to be construed as a promise by any participating organisation to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. opensuse makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for opensuse products remains at the sole discretion of opensuse. Further, opensuse reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All opensuse marks referenced in this presentation are trademarks or registered trademarks of SUSE LLC, in the United States and other countries. All third-party trademarks are the property of their respective owners. Credits Template Richard Brown rbrown@opensuse.org Design & Inspiration opensuse Design Team