Advanced Usage of the AWS CLI. Brian Wagner, Solutions Architect

Transcription

1 Berlin

2 Advanced Usage of the AWS CLI Brian Wagner, Solutions Architect

3 Crash Course Intro to the AWS CLI Foundation Exploring Key Functionality Advanced Scenarios Looking at Advanced CLI Features

4 Crash Course Introduction to the AWS CLI Novice Proficient Advanced

5 AWS Command Line Interface Unified tool to manage your AWS services

6 MSI (Windows) Bundled (cross platform) pip (cross platform)

7 aws configure

8 $ aws ec2 describe-instances service (command) operation (subcommand)

9 $ aws iam list-access-keys service (command) operation (subcommand)

10 --output json { Places : [ { City : Berlin, Country : Germany }, { City : Dublin, Country : Ireland } ]

11 --output text PLACES Berlin Germany PLACES Dublin Ireland

12 --output table SomeOperationName Places City Country Berlin Germany Dublin Ireland

13 All Outputs JSON Text Table { } Places : [ { City : Berlin, Country : Germany }, { City : Dublin, Country : Ireland } ] PLACES Berlin Germany PLACES Dublin Ireland SomeOperationName Places City Country Berlin Germany Dublin Ireland

14 Basic AWS CLI Usage Demo

15 Foundation Exploring Key Functionality Novice Proficient Advanced

16 Configuration

17 aws configure

18 aws configure AWS Access Key ID [**ABCD]: AWS Secret Access Key [****************EFGH]: Default region name [us-west-2]: Default output format [None]:

19 aws configure <subcommand> list - list common configuration sources get - Get the value of a single config var set - Set the value of a single config var

20 aws configure get region

21 aws configure set profile.prod.region eu-west-1

22 A profile is a group of configuration values

23 aws configure --profile prod

24 Configuration Files ~/.aws/credentials used only by the CLI Can contain credentials (but not the default behavior) supported by all AWS SDKs only contains credentials ~/.aws/config

25 aws configure set profile.prod.aws_access_key_id foo ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo

26 aws configure set profile.prod.aws_secret_access_key bar ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo

27 aws configure set profile.prod.aws_secret_access_key bar ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar

28 aws configure set profile.prod.region eu-west-1 ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar

29 aws configure set profile.prod.region eu-west-1 ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar [profile prod] region = eu-west-1

30 aws configure set profile.prod.output text ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar [profile prod] region = eu-west-1

31 aws configure set profile.prod.output text ~/.aws/credentials ~/.aws/config [prod] aws_access_key_id = foo aws_secret_access_key = bar [profile prod] region = eu-west-1 output = text

32 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

33 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

34 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

35 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

36 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

37 create-new-user.sh #!/bin/bash # Create a new user and create a new profile. aws iam create-user --user-name summit-user credentials=$(aws iam create-access-key --user-name summit-user \ --query 'AccessKey.[AccessKeyId,SecretAccessKey]' \ --output text) access_key_id=$(echo $credentials cut -d' ' -f 1) secret_access_key=$(echo $credentials cut -d' ' -f 2) aws configure set profile.summit.aws_access_key_id "$access_key_id" aws configure set profile.summit.secret_access_key "$secret_access_key"

38 Use the aws configure suite of subcommands

39 Query

40 --query (string) A JMESPath query to use in filtering the response data.

41 Implementation Details --query Processing aws ec2 parse params HTTP request parse response Retry Pagination format response display response

42 Implementation Details --query Processing aws ec2 parse params HTTP request parse response format response Retry parse response body Pagination apply --query display response

43 <ListUsersResponse xmlns="..."> <ListUsersResult> <Users> <member> <UserId>userid</UserId> <Path>/</Path> <UserName>james</UserName> <Arn>arn:aws:iam::..:user/james</Arn> <CreateDate> T23:36:32Z</CreateDate> </member> <Users> </ListUsersResult> <ListUsersResponse> Implementation Details --query Processing

44 Implementation Details --query Processing { } "Users": [ { "Arn": "arn:aws:iam::...:user/james", "UserId": userid", "CreateDate": " T23:36:32Z", "Path": "/", "UserName": "james" } ]

45 Implementation Details --query Processing --query Users[0].[UserName,Path,UserId] { } "Users": [ { "Arn": "arn:aws:iam::...:user/james", "UserId": userid", "CreateDate": " T23:36:32Z", "Path": "/", "UserName": "james" } ]

46 Implementation Details --query Processing --query Users[0].[UserName,Path,UserId] [ ] [ ], james, /, id

47 Implementation Details --query Processing ListUsers james / userid

48 Implementation Details --query Processing aws ec2 parse params HTTP request parse response Retry Pagination format response display response

49 A Query Language for JSON

50 Waiters

51 Amazon EC2 Instance State Transitions rebooting pending Start rebooting Reboot running Stop stopping stopped Terminate shutting-down terminated Terminate

52 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') while [ "$instance_state"!= "running" ] do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') done

53 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') while [ "$instance_state"!= "running" ] do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') done

54 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') while [ "$instance_state"!= "running" ] do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') done

55 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') while [ "$instance_state"!= "running" ] do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') done

56 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') while [ "$instance_state"!= "running" ] do sleep 1 instance_state=$(aws ec2 describe-instances --instance-ids $instance_id \ --query 'Reservations[].Instances[].State.Name') done

57 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) aws ec2 wait instance-running --instance-ids $instance_id

58 ec2-instance-running.sh #!/bin/bash instance_id=$(aws ec2 run-instances --image-id ami \ --query Reservations[].Instances[].InstanceId \ --output text) aws ec2 wait instance-running --instance-ids $instance_id subcommand waiter name describe-instances options

59 Advanced Scenarios Looking at advanced AWS CLI features Novice Proficient Advanced

60 Templates

61 The AWS CLI is data driven

62 Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"securitygroupstringlist", "locationname":"securitygroup" },

63 Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"securitygroupstringlist", "locationname":"securitygroup" }, --image-id --min-count --max-count --key-name --security-groups

64 Implementation Details JSON Models "RunInstancesRequest":{ "type":"structure", "required":[ "ImageId", "MinCount", "MaxCount" ], "members":{ "ImageId":{"shape":"String"}, "MinCount":{"shape":"Integer"}, "MaxCount":{"shape":"Integer"}, "KeyName":{"shape":"String"}, "SecurityGroups":{ "shape":"securitygroupstringlist", "locationname":"securitygroup" }, { } arguments.json "ImageId": "ami-12345", "MinCount": 1, "MaxCount": 1, "KeyName": "id_rsa", "SecurityGroups": ["SSH", "web"],

65 aws ec2 run-instances --cli-input-json file://arguments.json

66 What else can we do?

67 aws ec2 run-instances --generate-cli-skeleton

68 Creating and using JSON templates Demo

69 Credential Providers

70 export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config Amazon EC2 Instance Metadata

71 export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config Amazon EC2 Instance Metadata

72 export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config Amazon EC2 Instance Metadata

73 export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config Amazon EC2 Instance Metadata

74 export AWS_ACCESS_KEY_ID=... ~/.aws/credentials CredentialLocator ~/.aws/config AssumeRole Amazon EC2 Instance Metadata

75 Delegate access to AWS resources using AWS Identity and Access Management (IAM) Roles

76 Production Development Role Policy Trust Policy

77 Production Development Assume Role

78 aws configure set profile.prodrole.role_arn arn:aws:iam... aws configure set profile.prodrole.source_profile dev

79 ~/.aws/credentials ~/.aws/config [dev] aws_access_key_id = foo aws_secret_access_key = bar [profile prodrole] role_arn = arn:aws:iam source_profile = dev

80 Using Roles with the AWS CLI Demo

81 Amazon S3 Streaming

82 aws s3 cp

83 We want to avoid disk

84 aws s3 cp - s3://bucket/key

85 aws s3 cp s3://bucket/key -

86 Compress

87 aws s3 cp s3://bucket/key - \ bzip2 --best \ aws s3 cp - s3://buckey/key.bz2

88 Amazon S3 Streaming Demo

89 botocore

90 Implementation Details Architecture AWS CLI botocore Providers Serializers Parsers HTTP

91 Implementation Details Architecture AWS CLI AWS CLI botocore Providers Serializers Parsers HTTP

92 list_buckets.py import botocore s = botocore.session.get_session() s3 = s.create_client('s3', region_name='us-west-2') for bucket in s3.list_buckets()['buckets']: print("bucket: {bucket}.format(bucket=bucket['name']))

93 Using botocore in the python REPL Demo

94 Wrap-Up

95 Review Configuration Waiters Query Templates Credential Providers Amazon S3 Streaming

96 More Information Boto3 Talk

97 Brian Wagner, Solutions Architect