EGI-InSPIRE. cloud-init. Enol Fernández , EGI Webinar 1

Transcription

1 EGI-InSPIRE cloud-init Enol Fernández , EGI Webinar 1

2 Agenda Brief introduction to FedCloud Contextualization? Cloud-init Basic usage Cloud-config More use cases Preparing your own images Windows contextualization Q & A , EGI Webinar 2

3 EGI FedCloud What is the EGI Federated Cloud The EGI Federated Cloud is a federation of institutional private Clouds, offering Cloud Infrastructure as a Service to scientists in Europe and worldwide. EGI Federated Cloud is based on: Standards and validation: federation is based on common Open-Standards OCCI, CDMI, OVF, GLUE, etc... Heterogeneous implementation: no mandate on the cloud technology, the only condition is to expose the chosen interfaces and services , EGI Webinar 3

4 EGI FedCloud Cloud Infrastructure Platform , EGI Webinar 4

5 EGI FedCloud User workflows , EGI Webinar 5

6 Contextualization What Why? What? Contextualization is the process of installing, configuring and preparing software upon boot time on a pre-defined virtual machine image (e.g. hostname, IP address, ssh keys,... ) Why? Configuration not known until instantiation (e.g. data location). Private Information (e.g. host certs) Software that changes frequently or under development. Not practical to create a new VM image for every possible configuration , EGI Webinar 6

7 Contextualization How? How? Contextualization requires passing some data to the VMs on instantiation Standard OCCI API lacks such feature New mixins proposed and implemented in rocci-server, OCCI-OS and Synnefo (i.e. all current RPs of EGI s Fedcloud!) , EGI Webinar 7

8 Contextualization OCCI extensions User data: Category: user_data; scheme=" class="mixin X-OCCI-Attribute: org.openstack.compute.user_data="<base64 encoded data>" Public SSH keys: Category: public_key; scheme=" class="mixin" X-OCCI-Attribute: org.openstack.credentials.publickey.name="<key name>" X-OCCI-Attribute: org.openstack.credentials.publickey.data="<the public key>" , EGI Webinar 8

9 Contextualization rocciclient Use -T option with rocci-cli: ~$ occi -e $ENDPOINT \ -n x509 -X -x $X509_USER_PROXY \ -a create -r compute \ -M $OS_TPL -M $RESOURCE_TPL \ -t occi.core.title="mytutorialcontextvm_$(date +%s)" \ -T public_key="<public key>" \ -T user_data="<user data>" , EGI Webinar 9

10 Contextualization Using the context data OCCI extensions specifies how to pass data to the VM BUT not how the data will be available! Each RP has different mechanisms to provide the data: metadata server at a fixed location iso filesystem file injection... Need a way to abstract this mess! , EGI Webinar 10

11 cloud-init What is cloud-init? Cloud-init abstracts these mechanisms and defines a format for the data It can: configure network, users, ssh keys, filesystems,... install packages, execute arbitrary commands, execute user provied scripts, configure VM with puppet or chef,... It also provides mechanisms for extension , EGI Webinar 11

12 cloud-init Supported platforms Supports: EC2 metadata server (used by OpenStack) NoCloud data format as used by Synnefo. OpenNebula context.sh format included in v0.7.3 Since v0.7.5 also supports base64 encoded data as provided in FedCloud. Packages available for most Linux distributions: ubuntu/debian, SL5/SL6 (in EPEL), SUSE, , EGI Webinar 12

13 Data available at the VM user data vs meta data meta data Basic information on the VM VM Identifier Hostname, IP User Public Keys user data User data is treated as opaque data: what you give is what you get back. It is up to the instance (cloud-init) to be able to interpret it. ~$ occi -e $ENDPOINT \ -n x509 -X -x $X509_USER_PROXY \ -a create -r compute \ -M $OS_TPL -M $RESOURCE_TPL \ -t occi.core.title="mytutorialcontextvm_$(date +%s)" \ -T public_key="<public key>" \ -T user_data="<user data>" , EGI Webinar 13

14 cloud-init Handling the meta-data Create key (if you don t have one): ~$ ssh-keygen -f test.key Launch VM with rocci-cli: ~$ occi -e $ENDPOINT \ -n x509 -X -x $X509_USER_PROXY \ -a create -r compute \ -M $OS_TPL -M $RESOURCE_TPL \ -t occi.core.title="mytutorialcontextvm_$(date +%s)" \ -T public_key="file://$pwd/test.key.pub" Check results: ~$ ssh -i test.key ubuntu@ ubuntu@ip :~$ , EGI Webinar 14

15 cloud-init user-data cloud-init inspects the user-data Some of the supported formats: user script: execute it. (begins with #!) Cloud Config Data: cloud-config is the simplest way to accomplish some things via user-data. Using cloud-config syntax, the user can specify certain things in a human friendly format. (begins with #cloud-config) include file: contains a list of urls, one per line. Each of the URLs will be read, and their content will be passed through this same set of rules. (begins with #include) gzipped content uncompress and use as it were not compressed , EGI Webinar 15

16 cloud-init execute script Create simple script: #!/bin/sh echo "Hello World." > /root/context.txt echo "The time is now $(date -R)!" >> /root/context.txt Launch VM with rocci-cli: ~$ occi -e $ENDPOINT -n x509 -X -x $X509_USER_PROXY \ -a create -r compute \ [...] -T public_key="file://$pwd/test.key.pub" \ -T user_data="file://$pwd/script.sh" Check results: ~$ ssh -i test.key ubuntu@ "sudo cat /root/context.txt" Hello World The time is now Sat, 13 Dec :01: , EGI Webinar 16

17 cloud-config What is cloud-config? Cloud-config is cloud-init own configuration format. It uses a YAML (invalid syntax will make the contextualization fail!) Examples: apt upgrade should be run on first boot a different apt mirror should be used additional apt sources should be added certain ssh keys should be imported and many more , EGI Webinar 17

18 Groups and Users # Add groups to the system # The following example adds the ubuntu group with members foo and bar and # the group cloud-users. groups: - ubuntu: [foo,bar] - cloud-users # Add users to the system. Users are added after groups are added. users: - default - name: enol sudo: ALL=(ALL) NOPASSWD:ALL lock-passwd: true ssh-import-id: enol shell: /bin/bash groups: cloud-users ssh-authorized-keys: - ssh-rsa [...here goes the complete key...] , EGI Webinar 18

19 Package Management Upgrade, install, repos # run yum update package_upgrade: true # Install extra packages packages: - pwgen - pastebinit - [libpython2.7, ubuntu3.1] # Add you repositories yum_repos: EGI-trusanchors: name: EGI-trustanchors baseurl: enabled: true gpgcheck: true gpgkey: , EGI Webinar 19

20 Package Management apt sources # configure apt sources apt_sources: - source: "deb egi-igtf core" key: -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.2.1 (GNU/Linux) mqgibeltiyyrbad8gop2vwdf46e/stzvzgkbgjiftmkhqzoplqlcktrgf4vhuash hdaktdtpx44fvo4e3zmugc7fp6xz/hj3sqrukt98vzf1emb3i4umcobif+jm6vfs N5N3gDEukNpP2h46LkNPbRPgAEeUmUZy4kTyB9xC/VA7d1sFx6sJZpCHiwCg7DNX bj4wuk5b+fyycog9++xabokeajwt4+iydx3uyzrkzh9hoxgrbbiygroraz3jopqm 4L9+OKs5q9UsBwVXs5Zjei/irgxNjHNZCPo/V4f7o2CHxa88rn4GvstftSK6Oeey 8PaV3vdb5C5SRSbRgvxoUOo6eGVBpv8bVpKm//tNkTboHVsEAKQ1rYzx/m89aCZj VCw5A/0c3E0rH4ZCeNg7yvta9ur3U7n/aFhzbU3wFLhcIndrPaufz5Sy/SYhOaS9 RgH36GbsmOq6JskdtSpBLq0768BUmrjcosgWl3REpMAZc4vvtb55WRYsrNSrqmXZ /jhljqkfhfdobiecvxl+yiiwuxybmkvdxpzxnpgjf2gg6aop7rq5rvvhcmlkue1b IERpc3RyaWJ1dGlvbiBTaWduaW5nIEtleSAzIDxpbmZvQGV1Z3JpZHBtYS5vcmc+ ifkeexecabkfakltiyyecwcdagmvagmdfgibah4baheaaaojemmtmcg827xx5pqa oon2eh0dqfwnjgr1glgyt1o5bwkzaj0y4qopwacijfabolux5nifjkwv9w== =qxx END PGP PUBLIC KEY BLOCK , EGI Webinar 20

21 Write Files write_files: - encoding: b64 content: CiMgVGhpcyBmaWxlIGNvbnRyb2xzIHRoZSBzdGF0ZSBvZiBTRUxpbnV4... owner: root:root path: /etc/sysconfig/selinux permissions: content: # My new /etc/sysconfig/samba file SMBDOPTIONS="-D" path: /etc/sysconfig/samba - content:!!binary f0vmrgibaqaaaaaaaaaaaaiapgabaaaawaraaaaaaabaaaaaaaaaajavaaaaaaaaaaaaaeaaoaai AEAAHgAdAAYAAAAFAAAAQAAAAAAAAABAAEAAAAAAAEAAQAAAAAAAwAEAAAAAAADAAQAAAAAAAAgA AAAAAAAAAwAAAAQAAAAAAgAAAAAAAAACQAAAAAAAAAJAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAA... path: /bin/arch permissions: encoding: gzip content:!!binary H4sIAIDb/U8C/1NW1E/KzNMvzuBKTc7IV8hIzcnJVyjPL8pJ4QIA6N+MVxsAAAA= path: /usr/bin/hello permissions: , EGI Webinar 21

22 Run Commands # run commands # runcmd contains a list of either lists or a string # each item will be executed in order at rc.local like level with # output to the console runcmd: - [ ls, -l, / ] - [ sh, -xc, "echo $(date) : hello world! " ] - [ sh, -c, echo "=========hello world =========" ] - ls -l /root - [ wget, " -O, /tmp/index.html ] # boot commands # this is very similar to runcmd, but commands run very early # in the boot process bootcmd: - echo us.archive.ubuntu.com > /etc/hosts - [ cloud-init-per, once, mymkfs, mkfs, /dev/vdb ] , EGI Webinar 22

23 Create a FedCloud client VM Create user enol with my ssh-key Add EGI trust anchors and rocci yum repos Update packages Install ca-policy-egi-core, occi-cli & voms-clients Write fedcloud.egi.eu VO configuration:.lsc files at /etc/grid-security/vomsdir/fedcloud.egi.eu vo config files at /etc/vomses/ Put my grid certificate at /home/enol/.globus chmod -R /home/enol Make sure cloud-init runs all the needed modules , EGI Webinar 23

24 Create a FedCloud client VM demo _ /\ /\ \ \/ / \ /\ /\ \ / \ / \ / \/ /,. \ /,. \ \/ _ \/ / _ _ \ / \/ _ \/ , EGI Webinar 24

25 More complex cloud-init Mixing user data types If user data is a MIME multi part archive: cloud-init will consider each of the parts For example, both a script and a cloud-config type could be specified. Supported content-types: text/x-include-once-url text/x-include-url text/cloud-config-archive text/upstart-job text/cloud-config text/part-handler text/x-shellscript text/cloud-boothook , EGI Webinar 25

26 Mixing content types cloud-config + script Content-Type: multipart/mixed; boundary="=============== ==" MIME-Version: =============== == Content-Type: text/cloud-config; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="userdata.txt" #cloud-config users: - name: cloudy sudo: ALL=(ALL) NOPASSWD:ALL lock-passwd: true ssh-import-id: cloudy ssh-authorized-keys: - <your SSH public key> --=============== == Content-Type: text/x-shellscript; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="script.sh" #!/bin/bash echo "Hello" > /root/context.txt , EGI Webinar 26

27 Part Handlers Extending cloud-init Some python code that is executed for a given mime-type Two functions: list_types returns the mime-types accepted handle_type receives any user-data metching the accepted types , EGI Webinar 27

28 Part Handler demo _ /\ /\ \ \/ / \ /\ /\ \ / \ / \ / \/ /,. \ /,. \ \/ _ \/ / _ _ \ / \/ _ \/ , EGI Webinar 28

29 Some remarks FYI Keep in mind: You have root access to your virtual machines Your virtual machines are often visible from the Internet It is up to you to keep your virtual machines updated and secure DO NOT USE password-based authentication for remote access You should terminate your virtual machine as soon as it is not needed anymore cloud-init is NOT configuration management! , EGI Webinar 29

30 Prepare your own images Install image Create your image Install your OS: try to keep image size small, no need to store 0s any partition schema should work, I personally use a single / partition, no swap, nor /boot. Install cloud-init (v0.7.5 fully supports all FedCloud RP) Check cloud-init config (at /etc/cloud/) , EGI Webinar 30

31 Prepare your own images Convert to OVA Clean up Disable root password: passwd -d root Clean-up hardware details (e.g. network devices): Manually: remove /etc/udev/rules.d/70-persistent-net.rules With virt-sysprep Convert to OVA FedCloud promotes the use of OVA packages. VirtualBox does support exporting to OVA directly. See if you are using KVM/Xen , EGI Webinar 31

32 Prepare your own images Upload to AppDB Upload to AppDB A cloud-init script (user-data) can be attached to a VM in AppDB , EGI Webinar 32

33 Windows Windows can also be contextualizated! cloud-init is linux-only, but cloudbase-init can help! Features: setting hostname user creation group membership static networking SSH user s public keys user_data custom scripts running in various shells (CMD.exe / Powershell / bash) , EGI Webinar 33

34 Windows using cloudbase-init Installation Installer available at https: //github.com/stackforge/cloudbase-init#binaries Installer can also execute sysprep if needed. Contextualization user-data can only be a script but cloudbase-init will use all the meta-data , EGI Webinar 34

35 Useful pointers FedCloud & Contextualization FedCloud Wiki site User support User support Cloud Marketplace Contextualization FedCloud cloud-init cloudbase-init , EGI Webinar 35

36 Q & A? , EGI Webinar 36