ETOOMANYCATS. How we produce OpenStack

Transcription

1 ETOOMANYCATS How we produce OpenStack

2

3

4 OpenStack is large & growing 90+ code repositories.6+ MLOC (+30 %) Stats by

5 OpenStack is complex 9 integrated projects Each project is a framework

6 OpenStack is painful Difficult to deploy Difficult to test Difficult to keep up with changes

7 OpenStack is open innovation Anyone can propose a change Lots of different companies No traditional management structure Technical meritocracy (no DFL) Global community

8 And yet... We release every 6 months Master branches are always usable We keep on growing We retained (most of) our sanity

9 How?. Do code reviews & trunk gating. Reduce friction 3. Adopt time-based release cycles 4. Use a release manager

10 Code reviews Everyone's code should be reviewed At worse it's a learning opportunity Elite committers vs. Peer reviewers

11 Trunk gating Do not merge unless it passes tests «Automation for code review» Always-good trunk Testing is painful, will cats do it?

12 The OpenStack gate developer Your git branch github git.o.o reviewers review.o.o infrastructure PEP8 Hacking Unit tests Integration tests master git branch tarballs.o.o

13 Gating at scale 3 4 Serial gating is easy but limited

14 Serial gating

15 Serial gating

16 Serial gating 3

17 Serial gating 3 4

18 Gating at scale 3 4 Serial gating is easy but limited Parallel gating? Yay clouds But parallel gating may introduce fails

19 Parallel gating 3 4

20 Parallel gating 3 4

21 Gating at scale 3 4 Serial gating is easy but limited Parallel gating? Yay clouds But parallel gating may introduce fails Zuul : Speculative execution of tests

22 Zuul gating 3 3 4

23 Zuul gating 3 3 4

24 Zuul gating 3 3 4

25 Zuul gating 3 3 4

26 Zuul gating

27 . Reduce friction Automation & appropriate tools

28 A focused community Around a single language Around a single style (PEP8) Using regular face-to-face summits

29 Appropriate tools everywhere Code review on mailing-lists? Err... Align tools to your workflow Provide easy access to information Reduce interrupts Create the missing tools

30 Gerrit

31 Gerrit

32 Launchpad

33 Custom

34 Automation for the win

35 Infrastructure as code Puppet-driven infrastructure All project infrastructure is open Yes, everything is in a code repo Benefits from reviews, trunk gating Created tools where missing : jjb

36 3. Time-based release cycles

37 Release models CD (continuous delivery) Feature-based («when ready») Time-based

38 Time-based releases No way to tell when features will land Embrace benefits of knowing «when» Best length?

39 Good enough Accept it will never be perfect Release exposure will uncover bugs Manage risk as we get closer to release

40 Release candidates dance List release-critical bugs Publish RC when the list is empty Wash, rinse, repeat RC bug vs. Known bug Coordinated release of last RCs

41 RC dance example: Grizzly

42 Cycles > Releases Rhythm is essential Use cycles even with no formal release

43 Components of a cycle Design summits Milestones Freezes & Exceptions 3

44 4. The release manager

45 Role Manages cycles more than releases Communicates & enforces the rhythm Extracts & communicates roadmaps Documents shared understandings

46 Characteristics Obsession with detail Has duties, no rights Can be separate from the Tech Lead Needs to be a respected member

47 Summary. Do code reviews & trunk gating. Reduce friction 3. Adopt time-based release cycles 4. Use a release manager

48 Questions? on ttx on