Cloud Computing mit mathematischen Anwendungen

Transcription

1 Cloud Computing mit mathematischen Anwendungen Dr. habil. Marcel Kunze Engineering Mathematics and Computing Lab (EMCL) Institut für Angewandte und Numerische Mathematik IV Karlsruhe Institute of Technology (KIT) 3.Amazon Web Services (AWS) KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association

2 Lectures VL Intro VL Hadoop 1 VL Applications 1 VL Basics No Lecture! VL Applications 2 VL IaaS, PaaS VL Hadoop 2 VL Security VL SaaS, Cloud Architecture VL Parallel Computing 1 VL Summary, Q&A VL Programming Models VL Parallel Computing 2 2

3 Amazon Simple Storage Service (S3) 3

4 Amazon S3! Write, read, and delete objects containing up to 1 terabyte of data each. The number of objects you can store is unlimited.! Each object is stored in a bucket and retrieved via a unique, developer-assigned key. 4

5 S3! A bucket can be located in the United States, in Europe or in Asia. All objects within the bucket will be stored in the bucket s location, but the objects can be accessed from anywhere.! Authentication mechanisms are provided to ensure that data is kept secure from unauthorized access. Objects can be made private or public, and rights can be granted to specific users.! Uses standards-based REST and SOAP interfaces designed to work with any Internet-development toolkit.! Built to be flexible so that protocol or functional layers can easily be added. Default download protocol is HTTP. A BitTorrent protocol interface is provided to lower costs for high-scale distribution. Additional interfaces will be added in the future.! Reliability backed with the Amazon S3 Service Level Agreement. Source: aws.amazon.com 5

6 Amazon S3 Namespace Amazon S3 bucket bucket object object object object bucket object object 6

7 Amazon S3 Namespace Amazon S3 mculverimg.s3.amazonaws.com bucket234.s3.amazonaws. com Beach.jpg 2005/party/hat.jpg img1.jpg img2.jpg eorg.de -> eorg.de.s3.amazonaws.com index.html img/pic1.jpg 7

8 S3 Pricing Source: aws.amazon.com,

9 S3 Pricing Source: aws.amazon.com,

10 Google Cloud Storage! API is compatible to S3 (S3 may be regarded a de-facto standard)! 10

11 Google Cloud Storage! Pricing is almost compatible S3 (a little bit cheaper than S3) 11

12 AWS: Elastic Block Store (EBS)! Persistent data storage! Works like an unformatted storage device (like an USB disk drive)! Can be dynamically mapped to EC2 instance! Can take snapshots to conserve a specific status (backup)! Pricing is the same as for S3 12

13 Elastic IP Adresses! Provides public IP addresses (Visible on the web)! Dynamic mapping of public IP address to EC2 instance 13

14 Amazon Simple Queue Service (SQS) 14

15 Amazon Simple Queue Service (SQS)! Message queuing in the Cloud! Basic message queuing model, except: queues are hosted by Amazon, and queues are accessed using Web service protocols! Simple API! Platform agnostic! Basic support for access control and message locking! Reliability! Runs within Amazon's high-availability data centers! Messages stored redundantly across multiple servers and locations! Scalable to millions of messages a day 15

16 SQS Functionality! Developers can create an unlimited number of Amazon SQS queues, each of which can send and receive an unlimited number of messages.! New messages can be added to a queue at any time. The message body can contain up to 8 KB of text in any format.! A computer can check a queue at any time for messages waiting to be read.! A message is locked while a computer is processing it, keeping other computers from trying to process it simultaneously. If processing fails, the lock will expire and the message will again be available.! Messages can be retained in queues for up to 4 days.! Developers can access Amazon SQS through standards-based SOAP and Query interfaces designed to work with any Internetdevelopment toolkit. Source: aws.amazon.com 16

17 Amazon Management Console (1)! Management of virtual infrastructure! 17

18 Amazon Management Console (2)! Management of EC2 instances 18

19 Amazon Management Console (3)! Management of Amazon Machine Images (AMI) 19

20 Amazon Management Console (4)! Storage management (Elastic Block Store) 20

21 Amazon Management Console (5)! Management of public IP adresses: Elastic IP! Dynamic allocation to EC2 instances 21

22 Amazon Management Console (6)! Management of key pairs (Privileged access via SSH/RDP) 22

23 Amazon Management Console (6)! Security Groups: Administration of firewall rules 23

24 Browser Plugin: ElasticFox! ElasticFox FireFox Plugin! Download at 24

25 Browser Plugin: S3Fox! S3Fox FireFox Plugin! Download at 25

26 Access Control Lists (ACL)! Define access rights: Read, write fullcontrol! Add specific users with or user ID 26

27 Management! AWS command line tools are useful for developers and admins! Difficult to use for average customer! Download from 27

28 Amazon Elastic Compute Cloud (Version ) Quick Reference Card (Page 1) Revised:(3/25/2011 Instance Tools ec2$run$instances,(ec2run) Launches(one(or(more(instances(of(the(specified(AMI. ec2-run-instances ami_id [-n instance_count] [-g group [-g group...]] [-k keypair] [-d user_data -f user_data_file] [-t instance_type] [-z availability_zone] [-- kernel kernel_id] [--ramdisk ramdisk_id] [-b block-device-mapping] [--monitor] [--disableapi-termination] [--instance-initiatedshutdown-behavior behavior] [--placement-group placement_group] [--tenancy tenancy] [-s subnet] [--private-ip-address ip_address] ec2$describe$instances,(ec2din) Lists(the(specified(instances.(If(no(instance(is( specified,(all(your(instances(are(listed. ec2-describe-instances [instance_id...] [[--filter name=value]...] ec2$create$volume,(ec2addvol) Creates(an(Amazon(EBS(volume(from(a(snapshot(or(an( empty(volume(in(the(size(you(specify. ec2-create-volume -z availability_zone [-s size --snapshot snapshot] ec2$describe$volumes,(ec2dvol) Lists(the(specified(Amazon(EBS(volumes.(If(no(volume( is(specified,(all(your(volumes(are(listed. ec2-describe-volumes [volume_id...] [[--filter name=value]...] ec2$delete$volume,(ec2delvol) Deletes(the(specified(Amazon(EBS(volume. ec2-delete-volume volume_id ec2$attach$volume,(ec2attvol) Attaches(an(Amazon(EBS(volume(to(a(running(instance( and(exposes(it(as(the(specified(device.(the(volume( and(instance(must(be(in(the(same(availability(zone. ec2-attach-volume volume_id -i instance_id -d device Amazon EBS Tools ec2$terminate$instances,(ec2kill) Terminates(the(specified(instance. ec2-terminate-instances instance_id [instance_id...] ec2$start$instances,(ec2start) Starts(a(stopped(instance((applies(only(to(Amazon(EBSH backed(instances). ec2-start-instances instance_id [instance_id...] ec2$describe$instance$attribute, (ec2dinatt) Describes(an(attribute(for(the(specified(instance.( ec2-describe-instance-attribute instance_id { --block-device-mapping --disable-apitermination --instance-initiated-shutdownbehavior --instance-type --kernel -- ramdisk --root-device-name --user-data } ec2$create$snapshot,(ec2addsnap) Creates(a(snapshot(of(an(Amazon(EBS(volume(and( stores(it(in(amazon(s3. ec2-create-snapshot volume_id ec2$describe$snapshots,(ec2dsnap) Lists(the(specified(snapshots.(If(no(snapshot(is( specified,(all(your(snapshots(are(listed. ec2-describe-snapshots [snapshot_id...] [[--filter name=value]...] ec2$delete$snapshot,(ec2delsnap) Deletes(the(specified(Amazon(EBS(snapshot. ec2-delete-snapshot snapshot_id ec2$detach$volume,(ec2detvol) Detaches(an(Amazon(EBS(volume(from(an(instance. ec2-detach-volume volume_id [-i instance_id [- d device]] [--force] ec2$stop$instances,(ec2stop) Stops(an(instance((applies(only(to(Amazon(EBSHbacked( instances). ec2-stop-instances instance_id [instance_id...]! ec2$monitor$instances,(ec2min) Enables(monitoring(for(the(specified(instance.( ec2-monitor-instances instance-id [instance_id...] ec2$unmonitor$instances,(ec2umin) Disables(monitoring(for(the(specified(instance(s).( ec2-monitor-instances instance-id [instance_id...] EC2 Elastic IP Address Tools ec2$allocate$address,(ec2allocaddr) Acquires(an(EC2(Elastic(IP(address(for(use(with(your( account. ec2-allocate-address! ec2$describe$addresses,(ec2daddr) Lists(both(EC2(and(VPC(Elastic(IP(addresses(assigned(to( your(account.( ec2-describe-addresses [ip_address...]!! ec2$release$address,(ec2reladdr) Releases(an(EC2(Elastic(IP(address(associated(with(your( account.( ec2-release-address ip_address ec2$associate$address,(ec2assocaddr) Associates(an(EC2(Elastic(IP(address(with(an(instance.(If( the(ip(address(is(currently(assigned(to(another(instance,( the(ip(address(is(reassigned(to(the(specified(instance.( ec2-associate-address ip_address -i instance_id ec2$disassociate$address,(ec2disaddr) Disassociates(the(specified(EC2(Elastic(IP(address(from( the(instance(to(which(it(is(assigned.( ec2-disassociate-address ip_address!!

29 AWS Quick Reference Card (c td) Image Tools ec2$describe$images*(ec2dim) Returns(information(about(AMIs,(AKIs,(and(ARIs.(If(no( parameter(is(specified,(information(about(all(images(for( which(you(have(launch(permission(is(returned. ec2-describe-images [ami_id...] [--all] [-o owner...] [-x user_id] [[--filter name=value]...] ec2$create$image*(ec2cim) Creates(an(AMI(that(uses(an(Amazon(EBS(root(device( from(a(running(or(stopped(amazon(ebsjbacked( instance. ec2-create-image instance_id --name name [ --description description] [ --no-reboot] ec2$describe$image$attribute*(ec2dimatt) Describes(an(attribute(for(the(specified(AMI.( ec2-describe-image-attribute ami_id { --launchpermission --product-code --block-devicemapping --kernel --ramdisk } ec2$register*(ec2reg) Registers(the(AMI(specified(in(the(manifest(file(and( generates(a(new(ami(id. ec2-register manifest Availability Zone Tools ec2$describe$availability$zones*(ec2daz) Lists(Availability(Zones(that(are(currently(available(to( your(account.( ec2-describe-availability-zones [zone...]! Windows Tools ec2$get$password*(ec2gpass) Retrieves(and(decrypts(the(administrator(password(for( the(specified(windows(instance.( ec2-get-password instance_id -k key_pair ec2$bundle$instance*(ec2bundle) Key Pair Tools ec2$create$keypair*(ec2addkey) Creates(a(new(2048Jbit(RSA(key(pair(with(the(specified( name.( ec2-create-keypair key_pair ec2$describe$keypairs*(ec2dkey) Lists(the(specified(key(pairs.(If(no(key(pair(is(specified,( all(your(key(pairs(are(listed.( ec2-describe-keypairs [key_pair...] ec2$delete$keypair*(ec2delkey) Deletes(the(specified(key(pair(by(removing(the(public( key(from(amazon(ec2.( ec2-delete-keypair key_pair ec2$import$keypair*(ec2ikey) Imports(the(public(key(for(a(key(pair.(You(keep(the( private(key.(the(key(pair(works(in(all(ec2(regions.(( ec2-import-keypair key_pair --public-key-file file EC2 Security Group Tools ec2$create$group*(ec2addgrp) Creates(a(new(EC2(security(group.(Group(names(must( be(unique(per(account. ec2-create-group group_name -d description ec2$delete$group*(ec2delgrp) Deletes(the(specified(EC2(security(group. ec2-delete-group ec2_group_name_or_id!! ec2$describe$group*(ec2dgrp) Lists(your(EC2(and(VPC(security(groups.(If(no(security( group(is(specified,(all(your(security(groups(are(listed.( ec2-describe-group [ec2_group_name_or_id... vpc_group_id...] [[--filter name=value]...] ec2$authorize*(ec2auth) Adds(a(rule(to(an(EC2(security(group. ec2-authorize ec2_group_name_or_id Tagging Tools ec2$create$tags*(ec2addtag) Adds(or(overwrites(one(or(more(tags(for(the(specified( resource(or(resources.(each(tag(consists(of(a(key(and( an(optional(value.(tag(keys(must(be(unique(per( resource. ec2-create-tags resource_id [resource_id...] --tag key[=value] [--tag key[=value]...]! ec2$delete$tags*(ec2deltag) Removes(a(set(of(tags(from(a(set(of(resources.(The( tag(value(is(not(required. ec2-delete-tags resource_id [resource_id...] --tag key[=value] [--tag key[=value]...]! ec2$describe$tags*(ec2dtag) Lists(your(tags.(You(can(filter(the(list(to(return(only( tags(you(specify. ec2-describe-tags [[--filter name=value]...] Other Tools ec2$get$console$output*(ec2gcons) Retrieves(console(output(for(the(specified(instance.(( ec2-get-console-output instance_id [--raw-console-output]!! This(Amazon'Elastic'Compute'Cloud'Quick'Reference' Card(contains(commonly(used(commands(and( options.(for(complete(reference(information,(see(the( Amazon(EC2(Command(Line(Reference(at( aws.amazon.com/documentation/ec2/. 29

30 Hausaufgabe 2: API-Tools! Anlegen eines.ec2 Ordners im Home-Verzeichnis! Bei AWS anmelden, Zertifikat generieren und pk pem und cert pem Dateien nach.ec2 kopieren (x.509)! Herunterladen der API-Tools nach C:\Programme\ec2-api-tools ! Herunterladen und Installieren von Java! Setzen von Umgebungsvariablen (Systemsteuerung) EC2_HOME=C:\Programme\ec2-api-tools EC2_CERT=%HOME%\.ec2\cert...pem EC2_PRIVATE_KEY=%HOME%\.ec2\pk...pem JAVA_HOME=C:\Programme\Java\jdk6! Falls man in einer speziellen Zone arbeiten will (default: us-east-1) EC2_URL= Funktionstest: Zeige verfügbare Zonen mit ec2-describe-availability-zones AVAILABILITYZONE us-east-1a available us-east-1 AVAILABILITYZONE us-east-1b available us-east-1 AVAILABILITYZONE AVAILABILITYZONE us-east-1c us-east-1d available available us-east-1 us-east-1 30

31 Hausaufgabe 2 (c td)! Erzeugen eines Schlüsselpaars (Name z.b. mykey) ec2-add-keypair mykey Der zurück gelieferte private Schlüssel muss in einem File gespeichert werden (z.b. mykey.pem) und die Privilegien sollten auf readonly für den Besitzer gesetzt werden! Auflisten der verfügbaren Amazon Machine Images (AMI) ec2-describe-images -owner amazon IMAGE ami-42a2532b amazon/ec2 CentOS 5.5 GPU HVM AMI (Driver ) amazon available public x86_64 machine ebs hvm xen IMAGE ami-e565ba8c amazon/amzn-ami-pv x86_64-ebs amazon available public x86_64 machine aki-88aa75e1 ebs paravirtual xen! Starten einer virtuellen Maschine (Amazon Linux Micro-Instanz) ec2-run-instances ami-e565ba8c k mykey t t1.micro 31

32 Hausaufgabe 2 (c td)! Status der virtuellen Maschine anzeigen ec2-describe-instances i-2cb98d43 RESERVATION r default INSTANCE i-ba8685dd ami-e565ba8c ec compute-1.amazonaws.com ip ec2.internal running vorlesung-us 0 t1.micro T16:56: us-east-1d aki-88aa75e1 monitoring-disabled ebs paravirtual xen sg-c1c82da8 default BLOCKDEVICE /dev/sda1 vol-7ac4b T16:56:31.000Z TAG instance i-ba8685dd Name test! Firewall für SSH öffnen (Port 22) in der default security group ec2-authorize default -p 22 PERMISSION default ALLOWS tcp FROM CIDR /0! Mit SSH einloggen ssh -i mykey.pem root@ec compute-1.amazonaws.com! Stoppen der virtuellen Maschine ec2-terminate-instances i-ba8685dd INSTANCE i-10a64379 running shutting-down 32

33 33