Introduction to Stacki. Greg Bruno, PhD VP Engineering, StackIQ

Transcription

1 Introduction to Stacki Greg Bruno, PhD VP Engineering, StackIQ

2 Open Source Stack Installer Stacki is a very fast and ultra reliable Linux server provisioning tool at scale. With zero prerequisites for taking systems from bare metal to a ping and prompt.

3 PayPal

4 PayPal DATACENTERS Data Science Infrastructure Footprint 3,000 nodes and growing 60+ initial server racks Heterogeneous HW across multiple DCs 24 nodes each rack Salt Lake City Bay Area 24 x 900GB 6G SAS 10K data drives Las Vegas 10GBE NIC 48 nodes each rack 48 nodes each rack 12 x 2TB SATA data drives 1GBE-10GBE NICs 8 x 4TB NR-SAS data drives 10 GBE NIC

5 PayPal Stacki + Ansible = Happiness. :D Stacki mailing list 8/11/15 Ambari HDP Ansible Health Detection Integration OS Disk Network DHCP / DNS / TFTP IPMI/iLO - Disk Array Controller Configuration - Disk Partitioning Configuration

6 Quick, Early Success And Now 14 Minutes* To Fully Provision 6 Racks of Bare Metal (288 Servers) Includes wiping all disks then fully partitioning & formatting ~3500 drives Upgrades all firmware automatically Executes Ansible scripts on all hosts Hadoop packages installed * Versus hours with other hyperscale management tools, or days to weeks with traditional tools and processes

7 How We Solve the Problem

8 History San Diego Supercomputer Center National Science Foundation Along with NCSA only two non-classified centers Mission: serve computational scientists Rocks First cluster group inside SDSC Version 1.0 released that November as open source 10k+ clusters world-wide StackIQ Commercial support for Rocks Venture Backed Focus on next generation clustered systems (Data, Cloud) Stacki June released as open source July first hyper-scale user

9 Must Haves Make it Automatic Think about it, test it. Deploy it. People don t scale, software does. Free your people allow ops guys to be ops/analysis guys, move them from single machine view to global machine view. Make it Repeatable State of the environment is guaranteed. Does not require homogeneity of hardware or functionality. Make compute environments homogenous on heterogeneous hardware and software. Really, nothing is homogenous. Environment maybe, behavior of that environment on different machines while predictable will not be the same across all hardware. Stacki gets you flexibility and predictability. Make it Reliable You always get what you want when you want it. You can make reasonable estimates of need because you ve made the environment predictable and repeatable. Just like science! Make it Comprehensive Manage application layer(s) down to kernels and device configuration with one tool. Never hit the network unconfigured. Provide turn-key deployment with reasonable default settings and ability to customize / re-wire as desired.

10 Stacki Positioning In-house developed deployment tools DevOps / Configuration Tool OS Disk Network DHCP / DNS / TFTP - Disk Array Controller Configuration - Disk Partitioning Configuration

11 Datacenter Architecture Frontend em1 Network em1 em1 em1 em1 Backend Backend Backend Backend

12 Download and Boot the ISO Go to and download the ISO It s 1.8 GB stacki pallet plus stripped down CentOS 6.6 Boot the ISO on the host that will be your frontend

13

14

15

16

17

18

19 Frontend Services Services to build backend nodes DHCP TFTP Named (optional) Services to access backend nodes SSH key management Parallel execution shell

20 Host Configuration Spreadsheet

21 Backend Installation Save your Host Configuration spreadsheet as a CSV Import CSV on frontend stack load hostfile file=hosts.csv Tell backend nodes to install on their next PXE boot stack set host boot backend action=install PXE boot all backend nodes Done! em1 em1 Frontend em1 Network em1 em1 Backend Backend Backend Backend

22 BitTorrent-Inspired Package Installation Stacki

23 Customizing Your Hosts

24 Advanced Networking Via Host Configuration spreadsheet, you can configure: Bonded interfaces VLANs Bridging Any combo of the above Manage hosts in multiple subnets Build a single cluster from hosts in multiple subnets Manage hosts in multiple datacenters

25 Host Configuration Spreadsheet

26 Disk Controller Configuration Spreadsheet

27 Disk Partition Configuration Spreadsheet

28 Multiple Distributions A frontend houses a default distribution Based on stripped down CentOS 6.6 or 7.1 Used to build backend nodes Can add any number of new distributions to a frontend E.g., RHEL 6.x based distro, CentOS 6.5, etc. Assign any backend node to any distro

29 Why is this hard and important?

30 The Step 0 Problem Check namenodes are empty Format/start HDFS Create all metastores Create all directories Deploy client configuration No bad kernel versions running (CDH) version consistency Consistent host timezones Start services (Hbase, Hive, Oozie, Sqoop, Impala, etc) Configure database Daemons versions consistency Mgmt Agents versions consistency Validate/resolve hostnames Setup/assign monitors (activity, services, and host) Test database connections Java version consistency Host specification/ssh ports MUCH MORE DHCP Server/Client setup Server OS installation Boot configuration Network node configuration Config Mgmt install and configuration SSH keys on all hosts TFTP/PXE configuration RAID configuration System/data disk partitioning OS upgrades/updates Site specific software and configuration User accounts added and synced Node OS Install Monitoring system setup and config Lights Out/IPMI setup Route configuration Host specification/ssh ports Security Cluster Mgmt utility Firewall setup Database install and config Package installation Multiple network config MUCH MORE

31 Clusters are Different Adding new servers does require coordination Newly added servers must: Have same software stack as original servers Have same configuration as original servers Know about original servers And, original servers must: Know about new servers Result: The management complexity added to the Operations staff is exponential

32 Exponential Complexity Management Complexity Clusters General Data Center Number of Servers

33 The Pain Curve Management Complexity PAIN Clusters General Data Center Number of Servers

34 The Pain Threshold The pain threshold differs for every organization Function of: cluster(s) size number of people in Operations Operations staff cluster expertise

35 Moore s Law 8 7 Density month doubling Time (Years)

36 Moore s Law and Infrastructure Value

37 What it Means for You Value (%) months 90% value 18 months 50% value Time (Years) 5

38 Time is Money The clock starts ticking when hosts land on your loading dock Without your applications online, you have an paper weight that consumes power, cooling, and management s attention

39 Try It Out

40 stacki.com Download - Source & Docs - github.com/stackiq/stacki/wiki Discuss - groups.google.com/forum/#!forum/stacki

41 PayPal s Options Bring what we used at former parent company ebay with us. Not Possible Build our own soups-to-nuts bespoke bare metal provisioning tool. Not Optimal Find the perfect open source tool that we can use and grow with. Not Likely

42 Quick, Early Success 2 Weeks Instead of 2 Years To Build a Scale-out Management Solution PayPal development with Stacki includes: 1. Installed Stacki Frontend (base management server) Ran test installations of backend servers 1. Single Server test 2. Full Rack test (48 nodes) 2. Updated distribution (CentOS 6.6) to install additional packages 3. Integrated IPMI information into Stacki 1. Can now ssh into all IPMI consoles from the Stacki frontend host using <hostname>.ipmi 4. Re-ran with PayPal kickstart changes/additions and was able to image 6 racks in 14 minutes, including: 1. Nuking disks/partitions and running a full format of all data drives 5. Updated the Stacki post-boot piece to do the following: 1. Upgrade firmware if host needs it 2. Runs PayPal Ansible playbook, which: 1. Installs additional packages 2. Creates user accounts 3. Disables unused services 4. Sets up resolver/ntp/syslog-ng/sudoers/limits. d/sysctl/etc. 5. Installs/configures Ambari agents 6. Checks data drive mounts, fstab 7. Prepares the rack to be added to a Hadoop cluster

43 DevOps Agnostic In-house developed deployment tools DevOps / Configuration Tool OS Disk Network DHCP / DNS / TFTP - Disk Array Controller Configuration - Disk Partitioning Configuration

44 The Step 0 Problem Check namenodes are empty Create all directories Deploy client configuration No bad kernel versions running Format/start HDFS Start services (Hbase, Hive, Oozie, Sqoop, Impala, etc) Configure database Daemons versions consistency Create all metastores Setup/assign monitors (activity, services, and host) Test database connections Java version consistency (CDH) version consistency Mgmt Agents versions consistency Host specification/ssh ports Hadoop Mgmt Tool Monitoring Tool Consistent host timezones Validate/resolve hostnames MUCH MORE Upgrades/Patching Configuration Tool App Config DHCP Server/Client setup TFTP/PXE configuration Node OS Install Bare Metal Installers System Performance Validation Site Config Server OS installation RAID configuration Monitoring system setup and config Systems Mgmt Tool Network/Site Config Tools HW Install Boot configuration Network node configuration Config Mgmt install and configuration SSH keys on all hosts System/data disk partitioning OS upgrades/updates Site specific software and configuration User accounts added and synced Lights Out/IPMI setup Route configuration Host specification/ssh ports Security Disk Configuration Others SEMI-AUTOMATED TOOLCHAIN (w/o StackIQ) w/stackiq FULLY AUTOMATED Cluster Mgmt utility Firewall setup Database install and config Package installation Multiple network config MUCH MORE MANUAL

45 StackIQ Boss

46 Configuration Database Server appliance types (e.g. data, namenode, tomcat, ) Number of CPUs Disk partitioning Hardware RAID config PCI bus information And other System Attributes

47 Attributes Global stack set attr Appliance stack set appliance attr OS stack set os attr Host stack set host attr

48 Kickstart Profiles

49 Zoom In

50 Starting from the Empty Set { }

51 { os } 2009 UC Regents

52 { os, core } 2009 UC Regents

53 { os, core, kernel } 2009 UC Regents

54 { os, core, kernel, mapr } 2009 UC Regents

55 Manage the Deltas {os, core, kernel, mapr} {os, core, kernel, horton} 2009 UC Regents

56