Creating a World Class Build System

Transcription

1 Creating a World Class Build System Scott Estes Release Management and Ops Citrix Systems, Inc. 10 June 2011

2 AGENDA Getting Started Planning and Design Implementation and Execution Optimizing the System Maintenance and Evolution

3 GETTING STARTED Creating and maintaining a world class build system involves much more than a few well-written scripts, e.g. Know the Stakeholders Know the Stakeholders requirements Have a vision Engage, foster and grow support base at all levels Be a Center of Excellence

4 GETTING STARTED Evolution or Revolution? Pros Slow, stable change Can keep pace Cons Can be too slow Can lose momentum Pros Mitigates barriers to change Can facilitate other changes Cons Can be disruptive Unable to keep pace Unintended side-effects

5 GETTING STARTED Evolution or Revolution? Leverage both, Revolution in small doses Energy from revolution can fuel evolution across the organization

6 GETTING STARTED Build or Buy? Depends on, Resources Timeline/schedule

7 GETTING STARTED ONE LAST THING More than likely there will not be enough resources within the Build team to accomplish the Build Vision. Use others to help...

8 PLANNING AND DESIGN A great build system facilitates, Determinism Performance Parallel development Maintenance Business Continuity Self-service

9 PLANNING AND DESIGN WHERE ARE THE TEAMS? Knowing this will prevent support outages.

10 PLANNING AND DESIGN WHAT IS THE PRODUCT? Knowing the architecture ensures extensibility of the Vision. At Citrix: XenApp/XenDesktop use a Technology Componentbased architecture. Components treated as black boxes. Component visions are not tied to releases. Features are released to product when ready. Release process is much more efficient and dynamic.

11 PLANNING AND DESIGN DEVELOPER REQUIREMENTS Iterative feedback ensures success. Developers are pulled in different directions; don t add additional stress. Provide an Out of the box Build experience. Support as needed. Build confidence in the System and Vision.

12 IMPLEMENTATION AND EXECUTION At Citrix: Perforce Linked Automated Network Builder (PLANB) Determines when to build. Creates Build environment. Builds component binaries. Creates installers. Publishes product builds. Two types of Builds, SDK DVD

13 IMPLEMENTATION AND EXECUTION - PLANB Overview: Perforce Svr 1 Svr 2 Svr n Watch Create environment Build binaries Publish SDK Create DVD Publish/ Test Save Area

14 IMPLEMENTATION AND EXECUTION - BUILDING Most components can follow these steps: Input Pre-Build Steps Build Post-Build Steps Output With these attributes: Owner(s) Stakeholder(s) Input(s) Control(s) Output(s)

15 IMPLEMENTATION AND EXECUTION - INPUT Anything that causes a new build, e.g. code submission, SDK update, etc. At Citrix: A Watcher service is used to receive Perforce change notifications and start the build. Yes Update client specs Perforce Change received Project config changes? No Get changes Call VM Creator Cb

16 IMPLEMENTATION AND EXECUTION PRE-BUILD Configure the build environment. At Citrix: A VM creator service dynamically creates virtual build machines using XenServer technologies. Cb Yes Connect drives to VM Determine VM type VM available? Assign VM to host Start build script No Wait for available VM

17 IMPLEMENTATION AND EXECUTION BUILD Developer reviews are critical. At Citrix: An add-on to Visual Studio provides an out of the box experience while preserving company mandates. Conf. proj/ search No Done? Yes Get code ID what to build Check/ Build deps Compile No Built? Yes Sign, index symbols, etc. Create output Perforce Log error and continue

18 IMPLEMENTATION AND EXECUTION BUILD At Citrix: Non-Windows Builds are make-based. No Done? Yes Get code Search for makefiles Compile/ Link No Built? Yes Sign, index symbols, etc. Create output Perforce Log error and die

19 IMPLEMENTATION AND EXECUTION POST-BUILD Create a testable image and supporting build artifacts. At Citrix: A toolbox of PowerShell scripts create the DVD and releases the build for testing. Layout config Get layout info Poll for changes Update Symbol server Create reports Release to Test Symbol Server

20 IMPLEMENTATION AND EXECUTION OUTPUT Test the product and report. At Citrix: A test automation framework executes Build Verification Tests and other test suites. Test config Provision test env Execute BVT Pass? Yes Execute additional tests Report No Skynet Report

21 OPTIMIZING THE SYSTEM A great build system facilitates, Determinism Performance Parallel development Maintenance Business Continuity Self-service A World-Class build system optimizes, Determinism Performance Parallel development Maintenance Standards Business Continuity Self-service

22 OPTIMIZING THE SYSTEM WHAT S IN THE BUILD? Accurate, fast reporting, i.e. Determinism. At Citrix: Multiple reports based on audience, What binaries changed Test What code changed Dev/Release Mgmt/ Test

23 OPTIMIZING THE SYSTEM WHAT S IN THE BUILD? Accurate, fast reporting, i.e. Determinism. At Citrix: Multiple reports based on audience, What binaries changed Test What code changed Dev/Release Mgmt/ Test

24 OPTIMIZING THE SYSTEM LUDICROUS SPEED! Build faster than can be consumed. Don t, Compile code multiple times. Compile dead code. Ignore dependencies. Ignore the infrastructure. Ignore Business Continuity. Do, Facilitate Code Coverage efforts. Leverage incremental building and architecture. Use dedicated Perforce Proxies. Use as fast hardware as can afford. Isolate Build network. Review Business Continuity processes.

25 OPTIMIZING THE SYSTEM At Citrix: a topographical overview: US UK Business Continuity Center Cam Chf Perforce Svr 1 Perforce Svr 2 Perforce Svr1/ Proxy ports Perforce Svr1/ Proxy ports Builds NAS Builds NAS (+BC for Chf) Builds NAS (+BC for Cam) 1GB 1GB 1GB Build Farm 10GB NAS Servers Servers Servers P4 Proxy Data P4 Proxy Devs Devs Build machines (+BC for Chf) Devs Build machines (+BC for Cam) Build machines

26 MAINTENANCE AND EVOLUTION How to do it: Stage updates Automate, automate, automate! Test to trust Keep everyone updated When to do it: Align with project milestones and schedules Associate with a specific project

27 TAKEAWAYS Have a vision Enable instead of mandate. Talk to others and back it up. Provide a simple and maintainable infrastructure.

28