Ganeti, the New and Arcane

Transcription

1

2 Ganeti, the New and Arcane ganeti's best kept secrets, and exciting new developments Ganeti Eng Team - Google LinuxCon Japan Feb 2014

3 Introduction to Ganeti A cluster virtualization manager, in one slide

4 What is Ganeti? Manage clusters of physical machines, divided in nodegroups Deploy Xen/KVM/LXC virtual machines on them - Live migration - Resiliency to failure (DRBD, Ceph, SAN/NAS,...) - Cluster balancing - Ease of repairs and hardware swaps Controlled via command line, REST, web interfaces 4/53

5 Newest features Development status

6 2.10 The very stable release Improved upgrade procedure "gnt-cluster upgrade" CPU Load in hail/hbal (GSOC project) Hotplug support (KVM) RBD storage direct access (KVM) Better Openvswitch support (GSOC project) 6/53

7 2.11 The latest stable release Faster instance moves GlusterFS support hsqueeze (achieve maximum cluster compaction) 7/53

8 2.12 and future The next stable release(s) Jobs as processes New install model More secure master candidates Better container support (GSOC) Resource reservation/extra parallelization Generic conversion between disk templates (GSOC) 8/53

9 Monitoring daemon What's going on in your cluster?

10 Monitoring a cluster The old school way Other Systems Cluster NICs Monitoring System Instance Master Node Storage 10/53

11 Monitoring a cluster Using the monitoring daemon Other Systems Cluster Monitoring System Monitoring Daemons 11/53

12 What is the monitoring daemon? Provides information: about the cluster state/health live read-only design doc: design-monitoring-agent.rst 12/53

13 More details HTTP daemon Replying to REST-like queries Actually, GET only Providing JSON replies Easy to parse in any language Already used in all the rest of Ganeti Running on every node (Not: only master-candidates, VM-enabled) Additionally: mon-collector: quick 'n dirty CLI tool 13/53

14 Data collectors provide data to the deamon one collector, one report one collector, one category: - storage, hypervisor, daemon, instance two kinds: performance reporting, status reporting new feature: stateful data collectors 14/53

15 Data collectors What data can be retrieved right now? Now: instance status (Xen only) (category: instance) diskstats information (storage) LVM logical volumes information (storage) DRBD status information (storage) Node OS CPU load average (no category, default) Soon(-ish): instance status for KVM (instance) Ganeti daemons status (daemon) Hypervisor resources (hypervisor) Node OS resources report (default) 15/53

16 The report format { "name":"thecollectoridentifier", "version":"1.2", "format_version": 1, "timestamp": , "category": null, "kind":0, "data":{"plugin_specific_data":"go_here"} } JSON name:the name of the plugin. Unique string. version:the version of the plugin. A string. format_version:the version of the dataformat of the plugin. Incremental integer. timestamp:when the report was produced. Nanoseconds. Can be zeropadded. 16/53

17 Status reporting collectors: report They introduce a mandatory part inside the datasection. "data":{... "status":{ "code": <value> "message:"some summary goes here" } } JSON <value>:by increasing criticality level 0: working as intended 1: temporarily wrong. Being auto-repaired 2: unknown. Potentially dangerous state 4: problems. External intervention required 17/53

18 How to use the daemon? Accepts HTTP connections on node.example.com:1815 Not authenticated: read only Just firewall, or bind on local address only GET requests to specific addresses Each address returns different info according to the API / (return the list of supported protocol version) /1/list/collectors /1/report/all /1/report/[category]/[collector_name] 18/53

19 Configuration Daemon (confd) How's your cluster supposed to look like?

20 Before confd Configuration only available on master candidates Few selected values replicated with ssconf Small pieces of config in text files on all the nodes Doesn't scale Need for a way to access config from other nodes Scalable No single point of failure (so, no RAPI) 20/53

21 What does confd do? Provides information from config.data Read-only Distributed Multiple daemons running on master candidates Accessible from all the nodes through confd protocol Resilient to failures Optional 21/53

22 What info does it provide? Replies to simple queries: Ping Master IP Node role Node primary IP Master candidates primary IPs Instance IPs Node primary IP from Instance primary IP Node DRBD minors Node instances 22/53

23 confd protocol General description UDP (port 1814) keyed-hash Message Authentication Code (HMAC) authentication Pre-shared, cluster wide key Generated at cluster-init Root-only readable Timestamp Checked (± 2.5 mins) to prevent replay attacks Used as HMAC salt Queries made to any subset of master candidates Timeout Maximum number of expected replies 23/53

24 Confd protocol Request/Reply request request request request request 24/53

25 Confd protocol Request/Reply reply (v: 56) reply (v: 57) timeout reply (v: 57) reply (v: 57) (enough replies) 25/53

26 confd protocol Request plj0{ "msg":"{\"type\":1, \"rsalt\":\"9aa6ce de-af38-001d093e835f\", \"protocol\":1, \"query\":\"node1.example.com\"}\n", "salt":" ", "hmac":"4a4139b2c3c5921f7e439469a0a45ad200aead0f" } CONFD plj0:fourcc detailing the message content (PLain Json 0) hmac:hmac signature of salt+msg with the cluster hmac key 26/53

27 confd protocol Request plj0{ "msg":"{\"type\":1, \"rsalt\":\"9aa6ce de-af38-001d093e835f\", \"protocol\":1, \"query\":\"node1.example.com\"}\n", "salt":" ", "hmac":"4a4139b2c3c5921f7e439469a0a45ad200aead0f" } CONFD msg:json-encoded query protocol:confd protocol version (=1) type:what to ask for (CONFD_REQ_*constants) query:additional parameters rsalt:response salt == UUID identifying the request 27/53

28 confd protocol Reply plj0{ "msg":"{\"status\":0, \"answer\":0, \"serial\":42, \"protocol\":1}\n", "salt":"9aa6ce de-af38-001d093e835f", "hmac":"aaeccc0dff9328fdf7967cb600b6a80a6a9332af" } CONFD salt:the rsalt of the query hmac:hmac signature of salt+msg 28/53

29 confd protocol Reply plj0{ "msg":"{\"status\":0, \"answer\":0, \"serial\":42, \"protocol\":1}\n", "salt":"9aa6ce de-af38-001d093e835f", "hmac":"aaeccc0dff9328fdf7967cb600b6a80a6a9332af" } CONFD msg:json-encoded answer protocol:protocol version (=1) status:0=ok; 1=error answer:query-specific reply serial:version of config.data 29/53

30 Ready-made clients The protocol is simple, but clients are simpler Ready to use confd clients Python lib/confd/client.py Haskell Since Ganeti 2.7 src/ganeti/confd/client.hs src/ganeti/confd/clientfunctions.hs 30/53

31 Expanding confd capabilities Currently not so many queries are supported Easy to add new ones Just add a new query type in the constants list...and extend the buildresponsefunction (src/ganeti/confd/server.hsto reply to it in the appropriate way 31/53

32 Ganeti and Networks How do your instances talk to the world? Some slides contributed by Dimitris Aragiorgis

33 current nics: MAC + IP + link + mode NIC configuration mode=bridgeduses brctl addif Hooks can deal with firewall rules, and more External systems needed for DHCP, IPv6, etc. Management Which VMs are on the same collision domain? Which IP is free for a new VM to use? 33/53

34 gnt-network overview manage collision domains for your instances easy way to assign IPs to instances - If resources are shared in multiple clusters, allocation must be done externally keep existing per-nic flexibility hide underlying infrastructure better networking overview 34/53

35 gnt-network: Who does what? masterd: config.dataintegrity abstract network infrastructure: network + netparams per nodegroup IP uniqueness inside network: IP pool management - bitarray, TemporaryReservationmanager, Locking encapsulate network information in NIC opjects: RPC external scripts and hooks: ping vm1.ganeti.example.com use exported environment provided by noded brctl, iptables, ebtables, ip rule, etc. update external dhcp/dns server entries let VM act unaware of the "situation" (dhclient, etc.) 35/53

36 gnt-network+ external scripts gnt-networkalone is nothing more than a nice config.data snf-network: node level scripts and hooks nfdhcpd: node level DHCP server based on NFQUEUE 36/53

37 snf-network node level scripts and hooks overrides Ganeti default scripts (kvm-ifup, vif-ganeti) looks for specific tag types in NIC's network applies corresponding rules created nfdhcpdbinding files provides hook to update DNS entries 37/53

38 nfdhcpd node level DHCP server based on NFQUEUE listens on specific NFQEUE updates its leases db - inotifyon specific directory for binding files mangles DHCP requests and replies based on it's db responds to RS and NS for IPv6 auto-configuration 38/53

39 gnt-network Examples Create and connect a new network gnt-network add--network /24--gateway tags nfdhcpd net1 gnt-network connect net1 bridged prv0 Create an instance inside this network gnt-instance add--net 0:ip=pool,network=net1...inst1 gnt-instanceinfoinst1 gnt-network info net1 39/53

40 gnt-network+ snf-* Examples Use snf-networkand nfdhcpd apt-get install snf-network nfdhcdpd iptables-t mangle-a PREROUTING-i prv+-p udp-m udp--dport 67\ -j NFQUEUE--queue-num 42 ip addr add /24 dev prv0 Test connectivity gnt-instancerebootinst1 ping /53

41 References snf-network: nfdhcpd: 41/53

42 Ganeti ExtStorage Interface More options for your data Some slides contributed by Constantinos Venetsanopoulos

43 State before the ExtStorage Interface Non-mirrored templates: plain, file Internally mirrored templates: drbd Externally mirrored templates: sharedfile, rbd, blockdev, diskless 43/53

44 Ganeti and external SAN/NAS applicances Instance disks residing inside an external SAN/NAS appliance visible by all Ganeti nodes (e.g. NetApp, EMC, IBM) Instances should be able to migrate/failover to any node that can access the appliance. Ganeti should integrate with external SAN/NAS appliances in a generic way, independent of the appliance itself in the easiest possible way from the admin's perspective. 44/53

45 Introducing the 'ExtStorage Interface' A simple interface inspired by the Ganeti OS interface To plug an appliance to Ganeti, we need a corresponding 'ExtStorage provider' which is a set of scripts residing under a directory. e.g. /usr/share/ganeti/extstorage/provider1/ 45/53

46 ExtStorage provider methods Every ExtStorage provider should provide the following methods: Create a disk on the appliance Remove a disk from the appliance Grow a disk on the appliance Attach a disk to a given Ganeti node Detach a disk from a given Ganeti node SetInfo on a disk (add metadata) Verify the provider's supported parameters 46/53

47 ExtStorage provider scripts The methods are implemented in the corresponding 7 executable scripts, using appliance-specific tools: #ls-l/usr/share/ganeti/extstorage/provider1 create remove grow attach detach setinfo verify attachreturns a block device path on success Input via environment variables, e.g. VOL_NAME, VOL_SIZE 47/53

48 The new 'ext' template Introduce a new externally mirrored disk template: ext Introduce a new disk option: provider 48/53

49 Using the interface Example Assuming two appliances visible by a Ganeti cluster and their two ExtStorage providers installed on all Ganeti nodes: /usr/share/ganeti/extstorage/emc/* /usr/share/ganeti/extstorage/ibm/* #gnt-instance add-t ext--disk=0:size=2g,provider=emc #gnt-instance add-t ext--disk=0:size=2g,provider=emc\ --disk=1:size=1g,provider=emc\ --disk=2:size=10g,provider=ibm #gnt-instance modify--disk 3:add,size=20G,provider=ibm #gnt-instance migrate testvm1 # gnt-instance migrate n- nodex.example.com testvm1 49/53

50 ExtStorage Interface dynamic parameters Support for dynamic passing of arbitrary parameters to ExtStorage providers during instance creation/modification per-disk: #gnt-instance add-t ext--disk=0:size=2g,provider=emc,param1=value1,param2=value2 --disk=1:size=10g,provider=ibm,param3=value3,param4=value4 #gnt-instance modify--disk 2:add,size=3G,provider=emc,param5=value5 The above parameters will be exported to the ExtStorage provider's scripts as environment variables: EXTP_PARAM1 = str(value1) EXTP_PARAM2 = str(value2)... 50/53

51 The new 'gnt-storage' client Inspired by gnt-os: #gnt-storagediagnose #gnt-storageinfo 51/53

52 d from Lance Albertson, Iustin Pop,