PRS. Richard Gass Intel

Transcription

1 Session E: PRS

2 PRS Richard Gass Intel 2

3 Agenda Sessions: (A) Introduction (B) Hadoop Break Hadoop :30 Lunch (C) Pig Break (D) Tashi Break (E) PRS I. Overview II. Plans/Status III. User View IV. Adminsitration V. Installation VI. Summary 3

4 Overview 4

5 Open Cirrus Stack Management and control subsystem Compute + network + storage resources Power + cooling Physical Resource set (PRS) service Credit: John Wilkes (HP) 5

6 Open Cirrus Stack PRS clients, each with their own physical data center Eucalyptus Tashi/HDFS NFS storage service Experiment PRS service 6

7 Open Cirrus Stack Virtual clusters Virtual cluster Virtual cluster Eucalyptus Tashi/HDFS NFS storage service Experiment PRS service 7

8 Open Cirrus Stack Web Service BigData App Hadoop 1. Application running 2. On Hadoop 3. On Tashi virtual cluster 4. On a PRS 5. On real hardware Virtual cluster Virtual cluster Eucalyptus Tashi/HDFS NFS storage service Experiment PRS service 8

9 Open Cirrus stack - PRS PRS service goals Provide mini-datacenters to users Isolate mini-datacenters from each other PRS service PRS service approach Allocate sets of physical co-located nodes, isolated inside VLANs. Initial PRS implementation from HP Re-write from Intel (in collaboration with HP) contributed to Apache Software Foundation 9

10 Further Motivation Enable innovation in virtualization Allow running without virtualization overhead Necessary for predictable QoS e.g. cache interference

11 Goals Reduce complexity in allocating physical resources Gain User Confidence Show users that we can efficiently allocate/deallocate resources Stop the squatting Incentives HP s tycoon (economic model) Simple points scheme for good behavior Early return 11

12 Responsibilities of PRS Isolate domains Provision system software Provide platform control On/Off Provide boot debug VLAN PXE IPMI IPMI 12

13 VLAN Virtual LAN technology allows a single physical network to appear as several isolated networks Ethernet packets are tagged with a VLAN id Switches and NICs enforce the policies associated with each VLAN By associating PRS domains with different VLANs, they can be isolated from each other The PRS system provides the interfaces necessary to abstract switch configuration programming across multiple switch vendors 13

14 PreeXecution Environment PXE Enables provisioning of OS image over the network On machine boot, the NIC firmware contacts a PXE server via the DHCP process for the appropriate kernel and initrd to load Once loaded, the init scripts in the initrd can pull the filesystem to the machine In our environment, we download the desired filesystem to a ramdisk from a NFS server enabling a very rapid provisioning (30 seconds or less) while leaving the host filesystem undisturbed 14

15 IPMI Intelligent Platform Defines a standardized, abstracted, messagebased interface to intelligent platform management hardware Defines standardized records for describing platform management devices and their characteristics Management Interface Enables cross-platform management software 15

16 Status/Plans 16

17 PRS Roadmap Stage 1 Manages all cluster hardware Handles resource provisioning Provides interfaces for VLAN definition/programming Administrator is still in the allocation decision-making loop Stage 2 Introduces a request queue and primitive scheduler Admin may still be in loop, definitely for special cases Enables provisioning of OS to local disk Enables virtual disk conversion to physical Stage 3 Incentives module added (Tycoon) Tashi integration 17

18 Some History Previous prototype developed at HP Labs Focus on economic model Nice web interface which will be available upon reconvergence of code 18

19

20

21

22

23 User View 23

24 PRS Roles Admin: root of all authority Controls the physical resources User: requests domains Controls the domain, once allocated 24

25 Domains ADomain is the unit of PRS isolation A simple domain is a set of compute nodes gathered into a single VLAN Nodes are allocated from pools of available resources 25

26 The PRS Interface Users and Admins currently interact with the PRS system through a command line interface This interface both: Queries and updates records in the PRS database Wraps the various commands that must be issued to effect changes in the cluster PRS is currently a centralized system; users log into the PRS manager to issue commands An RPC interface is planned for the near future 26

27 PRS Usage Usage: prs <options> Standard options: --help --version --verbose Common options: --nodename <name> --switchport <port> Common admin options: --username <name> --uid <UID> [show this help message and exit] [show program's version number and exit] [be verbose] [Specify node] [Specify switchport switchname:portnum] [Specify user name] [Specify user id] 27

28 Image Management Interface --addimage <img> --delimage <img> [Add image to PRS] [Delete image] 28

29 User Allocation Interface --createdomain <name> May fail if name already exists --submitdomainrequest <name> --destroydomain domain <name> --requestnodes --domain <name> [--count <N>] [--nodename <name>] [--cores <n> ] Add the requested nodes to the domain --assignimage <kernel> <image> Assign image to resource --associatenewvlan domain <name> Allocate an unused VLAN number to domain --createreservation <YYYYMMDD> <YYYYMMDD> Specify duration of node reservation where start time may be ASAP --reservationnotes notes --updatereservation 29

30 Admin Allocation Interface --allocatenode --releasenode allocation] [Assign node to a user] [Release node --vlanisolate <vlanid> isolation] [Specify vlan for 30

31 Hardware Control --hardware [Make hardware call] --powerstatus [Get power status] --rebootnode [Reboot node (Soft)] --powercycle [Power Cycle (Hard)] --poweroff [Power off node] --poweron [Power on node] 31

32 Query Interface --showreservations --showresources --procs <N> --clock <N> --memory <N> --cpuflags flags --cores <N> --showpxeimages --showpxeimagemap [Show current node reservations] [Show available resources to choose from] [Filter by number of processors] [Filter by processor clock] [Filter by amount of memory (Bytes)] [Filter by CPU flags] [Filter by number of cores] [Show available PXE images to choose from] [Show PXE images host mapping] 32

33 Administration Interface --admin Enter Admin mode --addpxeimage --enablehostport --disablehostport --removevlan <vlanid> --createvlan <vlanid> --addnodetovlan <vlanid> --removenodefromvlan <vlanid> --setnativevlan <vlanid> --restorenativevlan --removeallvlans --sendswitchcommand <command> --interactiveswitchconfig <switchname> --showswitchconfig <nodename> [Add PXE image to database] [Enable a switch port] [Disable a switch port] [Remove vlan from all switches] [Create a vlan on all switches] [Add node to a vlan] [Remove node from a vlan] [Configure native vlan] [Restore native vlan] [Removes all vlans from a switchport] [Send Raw Switch Command, BE CAREFUL] [Interactively configure a switch] [Show switch config for node] 33

34 Administration 34

35 Typical Workflow 1. Admin queries available systems 2. Admin requests systems with desired user configuration 1. i.e., cores, memory, image, duration, etc 3. Request goes in queue 4. PRS locates resources and provides a list to admin/tashi. 5. Admin/Tashi moves s to free resources 1. Add node to blacklist and tell hadoop to reload 6. PRS allocates resources 1. Provides estimated time to get resources 2. User can query 3. PRS sends notification when allocated 7. PRS reclaims resources and adds them back into respective pools 1. User may extend time period before expiration 35

36 After allocation A returned PRS node is typically untrusted update the system to default settings Clean physical node by PXE booting a reset image Restore all setting to defaults (address, IPMI passwords) Repartition and format disks (Option) Trust images from some users No re-format needed Clean network configuration (VLAN) 36

37 Example: Minicluster./prs addimage hardy-rgass-testing:hardy:8.03./prs assignimage hardy-rgass-testing nodename r1r1u25./prs allocatenode nodename r1r1u25 username rgass reservationduration 30 vlanisolate 300 notes Practice allocation./prs addnodetovlan 300 nodename r1r1u25./prs hardware rebootnode nodename r1r1u25 37

38 Example: CloudConnect 1 100Mb/s Switch 100Mb/s Switch VLAN #1: Electrical Rack A Rack B Rack C Rack D Rack A region Rack B region M Rack C region Rack D region M 1 Gb/s Switch M - server -switch - manager 4Gb/s Switch VLAN #2: Optical 4x1Gb trunk link Network isolate a rack of machines and PXE boot them with a user s kernel and initrd Create a that acts as a SSH gateway and a NAT for the private cluster Dynamically configure switches to support the networking experiment 1Gb/s Switch

39 Example: CloudConnect 2 for i in r1r1u12 r1r1u13 r1r1u14 r1r1u15;do./prs --admin --setnativevlan 300 -n ${i}./prs --admin --addnodetovlan 800 -n ${i}./prs --admin --addnodetovlan 801 -n ${i}./prs --admin --addnodetovlan 802 -n ${i} done./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface range ethernet g(25-28); spanning-tree disable"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g25;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g26;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g27;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g28;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g28;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1:25 --setnativevlan 802 -v./prs --admin --switchport sw0-r1r1:26 --setnativevlan 804 -v./prs --admin --switchport sw0-r1r1:27 --setnativevlan 806 -v./prs --admin --switchport sw0-r1r1:28 --setnativevlan 808 -v for i in $(seq 12 16);do./prs --hardware --rebootnode -n r1r1u${i} done 39

40 Example cluster: Networking Testbed Network isolate a rack of machines and PXE boot them with a user s kernel and initrd Create a that acts as a SSH gateway and a NAT for the private cluster Dynamically configure switches to support the networking experiment 40

41 Future Work Integration with Tashi Would enable free exchange of resources between the Tashi pool and the free pool 41

42 System Servers PRS client User chooses machine PRS attributes client and queries PRS server PRS submits Results for queries a available are request DB to for locate sent the resources resources back available to resources the for some client time period Management Servers DB PRS server Administrator or Cluster Manager Node 1 : 8 Core, 16G memory, 6TB disk,30day Node 2 : 8 Core, 16G memory, 6TB disk,30 day Node 3 : 8 Core, 16G memory, 6TB disk,90 day Node 4 : 8 Core, 16G memory, 6TB disk,1 day Node 5 : 8 Core, 8G memory, 2TB disk, 90 day Node 6 : 8 Core, 8G memory, 2TB disk,90 day Node 7 : 8 Core, 8G memory, 2TB disk,90 day Node 8 : 8 Core, 8G memory, 2TB disk,90 day Node 9 : 8 Core, 8G memory, 2TB disk,90 day Node 10: 8 Core, 8G memory, 2TB disk,30 day Tashi Cluster Manager PXE server 42

43 System Servers PRS client R1 Request Queue Management Servers DB PRS server Administrator or Cluster Manager Tashi Cluster Manager PXE server 43

44 System Servers PRS client PRS processes request and identifies physical machines that satify the user request Management Servers DB PRS server Administrator or Cluster Manager Tashi Cluster Manager PXE server 44

45 System Servers PRS client Administrator or Cluster Manager PRS sends request to Tashi to free selected nodes Tashi moves virtual machines off of selected nodes Management Servers DB PRS server Tashi Cluster Manager PXE server 45

46 System Servers PRS client Administrator or Cluster Manager Physical machines PRS PRS boot allocated reboots up with the PXE physical machines image and to the sets requested PXE image user to and isolates users them from the network Tashi using notifies VLANs PRS that migration of virutal machines has completed Virtual disk image is converted to PXE image Management Servers DB PRS server Tashi Cluster Manager PXE PXE server 46

47 System Servers PRS client PRS updates reservation database User connects to the machines and starts running experiments Management Servers DB PRS server PXE PXE PXE Administrator or Cluster Manager PRS client queries server for allocation Tashi Cluster Manager PXE server 47

48 System Servers PRS client Management Servers DB PRS server Administrator or Cluster Manager Tashi Cluster Manager PXE server 48

49 System Servers PRS client Management Servers DB PRS server Administrator or Cluster Manager Tashi Cluster Manager PXE server 49

50 Installation 50

51 Necessary Components DHCP Server PXE Server NFS Server DNS Server (optional) Configurable switches New switch types may require new PRS modules Hardware access method E.g. IPMI IP-addressable PDUs enable rescue if IPMI becomes compromised 51

52 Internals 52

53 Notes on Current Software PRS client code is Python 2.5 PRS database implemented in MySQL Reachable through python-mysqldb interface pexpect used for switch configuration User information currently obtained through LDAP 53

54 Summary 54

55 PRS PRS lays the foundation of the Open Cirrus software stack easing management of multiple projects in a single cluster PRS enables partitioning clusters into isolated domains of physical resources Current implementation allows rapid provisioning of system software PRS code base is open source software available through Tashi project in Apache Incubator Contributions welcome 55