Orchestrating an OpenStack DevOps Cloud for R&D to Achieve Continuous Delivery. Ting Zou & Tanay Nagjee November 7, 2013

Transcription

1 Orchestrating an OpenStack DevOps Cloud for R&D to Achieve Continuous Delivery Ting Zou & Tanay Nagjee November 7, 2013

2 Introductions Ting Zou Director, Cloud Computing R&D Data Center Huawei USA Tanay Nagjee Solutions Electric Cloud Former ElectricCommander Engineer 2

3 This is Huawei A privately-owned Global Company An Innovative Industry Contributor An ICT Industry Leader

4 A Privately-owned Global Company A global company providing information and communications technology (ICT) solutions. Products and solutions have been deployed in 140+ countries, serving 1/3 of the world s population. A privately-owned company founded in 1987, Shenzhen

5 Sustainable Growth 2012 Revenue amounted to $35.4 billion, a YoY increase of 8%; net profit reached $2.47 billion 2012 USD 4.8 billion R&D investment 1H 2013, Revenue amounted to USD $18.5B, a YoY increase of 10.8%; Expected net profit margin of 7-8% in Enterprise 1.9 bn Consumer 7.8 bn By BG (2012) By Region (2012) 22% 5% 73% Americas 5.1 bn China 11.8 bn 35% 15% 33% 17% Currency: USD Carrier Network 25.7 bn Currency: USD EMEA 12.5bn Asia Pacific 6.0 bn 5

6 Continuous Innovation Investment 2012 R&D investment: USD 4.8 billion, 13% of 2012 revenues 70,000 R&D employees 16 R&D centers worldwide 28 joint innovation centers 41,948 patents in China, 12,453 international PCTs, 14,494 patents in outside of China. US$300 million in patent royalties each year standards organizations 30,000+ standards proposals Accumulated R&D investments: USD Bn 4.8Bn billion 1.5Bn 2Bn 2.7Bn

7 Problem overview

8 What are we solving? Large scale R&D environment with complicated tools requires multithousands of CPU cores available on demand Current Build Time Expected Build Time Actual Time Engineering Expectation 5 Million LOC Build (unit: minute) 10 Million LOC code coverage testing report (unit:minute) 8

9 What are we solving? Environment/tools provisioning is very time-consuming; lab asset utilization is low Current expected environment provision Current Provision time Expected Provision Time Compile/Build environment Provision (unit:minute) Software Testing environment provision (unit:hour) 9

10 What are we solving? R&D data grows rapidly; requires the flexibility of dynamically expanding PB storage capability G 500G 2011 年 2012 年 daily execution data generated by daily build Millions LoC build static analysis/ build coverage to generate PB data annually Data Generated by the Tools in Auto Testing/CI 10

11 What are we solving? Integration between development and operation; DevOps enabled continuous delivery required. Virtualization and cloud computing converged infrastructure Automatic provision and configuration management technology DevOps Cloud R & D requires multi-layer and multiplatform infrastructure Allow developers more control over the production environment Concise definition of R&D processes, automate as much as possible Bridging the gap between development and operations Globally distributed R&D teams 11

12 Solution overview

13 Virtualization/Cloud sounds like a good solution, but... To virtualize 3k 2-CPUs servers, commercial SW license + 1 year support is a big investment: Servers CPUs Standard Enterprise Enterprise Plus million US$ 20 million US$ 25 million US$ Shall we leverage low-cost(free) open source cloud OS to reduce R&D TCO? Source:

14 Choose OpenStack as the Provider of IAAS Cobbler Puppet Chef etc In-House Dev Module ilcm (Lab Configuration Manager) Swift Swift Swift-Proxy Swift-Proxy Controller Controller Controller Quantum(A) Quantum(A) Quantum(A) Storage Node Controller Controller Node Controller #1 Node Node #2 #3 Private Switch (Network) vswitch External Switch (Network) 8 Huawei RH2285(dual 6-cores CPUs, 96GB MEM, 8TB storage) servers used in PoC Openstack is provisioned and managed by Huawei ilcm(intelligent Lab Configuration Manager) with backend integration with cobbler/chef/puppet open source tools Leverage the OpenStack open source community as much as we can (Devstack, Mirantis, RPC, Dell crossbar etc) plus in house developed script modules, will contribute back to the community once it is ready.

15 Single Portal for Cloud Admin and Users Cloud Admin Portal for users with different level of privileges, integrated with LDAP at the backend Cloud resource usage monitoring (Roadmap) Cloud VM provision and software configuration management Cloud Network, Storage and Security configuration management (Roadmap)

16 Cloud is free now, what else can be free? Project Mgmt JIRA Redmine Trac Bugzilla Provision Monitor SCM Collaboratio n GIT Gerrit Subversion Reviewboard Mercurial OpenGrok Github SourceForge Chef Puppet CFEngine Nagios Design Development Testing Others Code analysis Hadoop Requirement Analysys Jenkins Cpplint PcLint CppCheck valgrind Func Testing Pef Testing Hive HBase Design modeling CI Travis-CI Unit Testing gunit,cppunit Automation Framework gmock,jmock Selenium IOMeter Jmeter Page Speed Mobitest memcached RabbitMQ ActiveMQ Cloud Platform OpenStack CloudStack

17 HUDOS (Huawei Unified DevOps System) Technologies

18 Single Portal for R&D engineering SaaS (instant creation of an environment with complete portfolio of common R&D tools) Lab topology creation with compute, network, storage etc (Roadmap) Project R&D data statistics monitoring Single dashboard portal for engineering to access all the needed resource/tools in R&D process

19 IaaS and PaaS enable DevOps-Engineering link IaaS integrated with engineering desktop dashboard R&D tools PaaS integrated with engineering desktop dashboard

20 OpenStack integration

21 Platform Apps Electric Cloud s Software Delivery System Continuous Delivery Manager Release Trains Feature Boards Pipelines Gates Dashboards Build Test Deploy Build Automation CI Test Automation CT Deploy AutomationC D Build Acceleration Test Acceleration Infrastructure Provisioning Software Delivery Platform Automation & Acceleration Services Workflow Resource management I Tools integration API Security Reporting

22 Before and After Electric Cloud Issue Before Electric Cloud After Electric Cloud Business Impact Develop to Deploy 90 Days 10 Minutes 99.93% Build to Release/Deploy 10+ errors/cycle ~0 errors/cycle 99+% Audit application changes (who, what, how, why, when) Days Minutes 90+% Time to troubleshoot problems 20 Days Minutes 99+%

23 OpenStack integration 23

24 OpenStack integration 24

25 OpenStack integration 25

26 OpenStack integration 26

27 OpenStack integration 27

28 Solution walk-through

29 Development Scenario Developer Joe HUDOS Reviewer Mike Eclipse Modify code launch preflight Subversion Check out sources overlay deltas Redmine Mark issue as build & unit test Eclipse Build + test failed; notify developer Build + test success? Jenkins Launch build + test w/ preflight source Time Savings Redmine Mark issue as code review Review Board Create review request Review Board Review modified code Eclipse Code Success rejected; auto-commit notify developer code Redmine Mark issue as resolved Review success? 29

30 Redmine: select a bug to fix 30

31 Eclipse: modify code & launch a preflight 31

32 Commander: SVN preflight & Jenkins launch 32

33 Redmine: auto-updated issue status 33

34 Jenkins: auto-launched build 34

35 Commander: build succeeds; wait for review 35

36 Review Board: auto-submitted review request 36

37 Review Board: accept the changes 37

38 Commander: the workflow succeeds 38

39 Eclipse: auto-commit the changes 39

40 Test Scenario Test Engineer Jill HUDOS Commander Pick issue to verify and tests cases Redmine Mark issue as verifying OpenStack Provision specified virtual machines Notification Tests failed; VMs ready to inspect Testing success? Commander Launch automated tests Time Savings OpenStack Teardown virtual machines Redmine Mark issue as closed Notification Tests passed; issue closed 40

41 OpenStack: no dynamically deployed VMs 41

42 Commander: launch test workflow 42

43 Commander: provision VMs 43

44 OpenStack: VMs are dynamically deployed 44

45 Commander: successful tests; auto-teardown 45

46 Summary

47 What a DevOps Cloud brings to us Design Develop Product Validation Solution Validation Days-> Hours Architect Evaluation Hours-> Minutes Code Analysis Weeks-> Days Full Functional Testing Months->Weeks Solution Testing Hours-> Minutes Days-> Hours Compile & Build Full Regression Testing(System) Days-> Hours Days-> Minutes Hardware Emulation Regression Testing(software) R&D Cloud Data Center facility to provide the capability of acceleration More than 2000 releases per year More than compile & builds per day More than 1million test cases run per day More than 30million LoC, product is complicated More than 480K code review/analysis per year More than 170k system integration testing per year 成都 ECTC 西安 ECTC 武汉 ECTC 北京 ECTC 苏州 ECTC 上海 ECTC 印度 ECTC 武汉 杭州 ECTC 深圳 ECTC

48 Did we address the stated concerns? Large scale R&D environment with complicated tools requires multi-thousands of CPU cores available on demand. Environment/tools provisioning is very time-consuming; lab asset utilization is low. R&D data grows rapidly; requires the flexibility of dynamically expanding PB storage capability. Integration between development and operation; DevOps enabled continuous delivery required. Achievements: Reduced cost of delivering software Increased resource utilization and productivity Shorter time to market with higher quality 48

49 What s next? R&D DATA CENTER server, 15k + core, 10K + virtual machine memory 100TB, storage 2PB, reduce costs 2M US$ Beijing: server Size: 800 core, memory 5TB, Storage 200TB 北京 成都 西安 武汉 Hangzhou: Server Size: core, memory 10TB, Storage 500TB 南京 杭州 上海 深圳 Shenzhen HQ: Server Size: 10k+ core, memory 50TB, 1-2PB storage 49

50 Thank you!