Forseti Documentation

Forseti Documentation Release 0.1.1 Ticketea dev team Sep 20, 2017

Contents 1 Introduction 3 1.1 Terminology and basic concepts..................................... 3 1.2 What Forseti can do for you?...................................... 4 1.3 What Forseti can t do for you?...................................... 4 2 Quickstart and first steps 5 2.1 Configuring AWS access......................................... 5 2.2 Setting up a Forseti application..................................... 6 3 Configuring Forseti 7 3.1 Application section............................................ 9 3.2 Autoscale section............................................. 10 4 Deployers 13 4.1 Deploy and snapshot........................................... 13 4.2 Golden instance............................................. 13 5 List of available oficial commands 15 6 Q&A 19 6.1 Why did we call it Forseti?........................................ 19 6.2 Why did you create Forseti instead of using other utilities or AWS official tools such as CloudFormation or CodeDeploy?........................................... 19 7 License 21 8 Internals documentation 23 8.1 Internal documentation.......................................... 23 9 Indices and tables 25 i

ii

Forseti is a two-in-one utility: A CLI tool to manage your AWS autoscaling groups, policies, etc. It allows you to easily deploy your code in AWS using your preferred strategy defined as a _deployer_. A _deployer_ is a class that, using previous models, defines a deployment strategy. More on this later. A set of classes wrapping boto, provinding friendly high level operations that allow you to easily do common administration operations. Forseti is devops agnostic in the sense that it all their commands can be plugged with any orquestration tool you use, in ticketea we ve used Chef, Puppet and Ansible in conjuction with it. Contents: Contents 1

2 Contents

CHAPTER 1 Introduction Forseti development began in 2011. We needed a tool to manage Amazon Web Services (AWS) EC2 1 auto-scale groups and at that time the web interface was lacking of support. The only way to manage autoscaling was using their API, which was very complete and well documented. We looked for third party tools but all of them were too much, with a lot of requirements, a complicated UI and a lot of effort in order to get started. We only wanted an easy CLI tool, one that did one task and did it right. And Foserti was born. Terminology and basic concepts It s not the objective of this guideline to explain all concepts regarding AWS but considering it s a tool to manage a very specific part of it, you need to be familiar with some of its concepts and how they work. EC2 instance: Virtual machine running inside AWS system. EBS root instance: It s an EC2 instance in which the root device is inside an EBS volume. Autoscaling group: A collection of homogeneous EC2 instances which can scale up or down. Launch configuration: Associated to an autoscaling group, there are launch configurations, which define which AMI will be used when an instance is launched, including its size, SSH key and others. Autoscaling policy: A policy defines how an autoscaling group will scale up or down. AWS offers three different types: increasing/decreasing the capacity using a number, change it to a specified number of instances or increasing/decreasing in percentual ranges. Alarms: Using AWS Cloudwatch you can define some alarms to trigger the autoscaling policies to scale in or out the number of instances in a group. This is the key part because it allows you to manage the capacity automatically with no human intervetion, usually named autoscaling. 1 Elastic Compute Cloud 3

What Forseti can do for you? Forseti is built on top of boto, a great Python library to interact with AWS apis, offering a complete access to all the services Amazon offers. On top of boto, we ve built a programatic system to manage all the autoscale items, from autoscaling groups, configurations and policies to CloudWatch alarms and SNS messages. Forseti is able to deploy your application code (using any external tool available) into the instances you want and build an autoscaling group, allowing you to scale up or down the number of instances to fit the load you require. Using autoscaling is a must nowadays if you want to offer a stable service, but setting up AWS autoscaling is a complicated process in which Forseti can help you. After deploying your application code into all the instances you want, Forseti will select one randomly to create an AMI 2 from it and setting up the autoscale group the way you want. We built forseti as an abstraction of boto classes, providing high level operations and introducing a new concept, the application. What Forseti can t do for you? Forseti was not thought to orchestrate machines, it s not a deploy utility, it can t run operations in your machines. You have better tools for that: ansbile, puppet, chef, capistrano... name yours. Forseti is a tool to manage AWS EC2 autoscaling, nothing more, nothing less. 2 Amazon Machine Images. An autoscale group uses AMIs to launch new instances. 4 Chapter 1. Introduction

CHAPTER 2 Quickstart and first steps Configuring AWS access The first thing you need to do is configure your AWS credentials and the region you want to use. Forseti depends on boto, so you can read their getting started guideling to get all the information you need. The minimum setup that Forseti requires is creating a file in ~/.boto with this content: [Boto] autoscale_region_name = eu-west-1 autoscale_endpoint = autoscaling.eu-west-1.amazonaws.com ec2_region_name = eu-west-1 ec2_region_endpoint = ec2.eu-west-1.amazonaws.com elb_region_name = eu-west-1 elb_region_endpoint = elasticloadbalancing.eu-west-1.amazonaws.com cloudwatch_region_name = eu-west-1 cloudwatch_region_endpoint = monitoring.eu-west-1.amazonaws.com sns_region_name = eu-west-1 sns_region_endpoint = sns.eu-west-1.amazonaws.com [Credentials] aws_access_key_id = <YOUR_AWS_KEY> aws_secret_access_key = <YOUR_AWS_SECRET> Note: In this example, we ve chosen eu-west-1 region, change it to use other region if that s your case 5

Setting up a Forseti application If you re already running instances in AWS, the best way to start with Forseti is using the init command. This command will help you creating an application using an instance (it could be running or stopped): forseti init <application_name> i-xxxxxxxx --deployer=deploy_and_snapshot This will create for you an autoscaling group, using an AMI created from the selected instance. That information will be added to the Forseti configuration file, located in ~/.forseti/config.json. Move to Configuring Forseti If you want more information about this command, please refer to the List of available oficial commands. 6 Chapter 2. Quickstart and first steps

CHAPTER 3 Configuring Forseti In this section you will find a deeper description on the Forseti s configuration. It has different sections very well defined but interrelated. Forseti looks for the configuration file in a very specific path inside the ~/.forseti/ config.json. First, let s see a basic example of a Forseti configuration file. { "applications": { "backend": { "deploy": { "working_directory": "/path/to/capistrano", "command": "cap production deploy -S servers={dns_name", "deployment_strategy": "deploy_and_snapshot", "autoscale_group": "BACKEND", "scaling_policies": [ "scale_down_policy", "scale_up_policy" ], "autoscale_notifications": [ { "topic": "arn:aws:sns:eu-west-1:1234567890ab:notifications", "type": "ALL" ], "sns_notification_arn": "arn:aws:sns:eu-west-1:1234567890ab:notifications ", "sns_extra_attributes": { "ParametersForYourSNSApplication": { "Priority": "Info", "Channel": "#notifications", "autoscale": { "groups": { 7

"BACKEND": { "availability_zones": [ "eu-west-1a", "eu-west-1b" ], "default_cooldown": 300, "termination_policies": [ "ClosestToNextInstanceHour", "OldestInstance" ], "min_size": 2, "desired_capacity": 2, "load_balancers": [ "backend" ], "max_size": 10, "configs": { "backend": { "key_name": "private-keys", "instance_type": "m1.small", "security_groups": [ "security-group" ], "instance_monitoring": true, "policies": { "scale_down_policy": { "cooldown": 180, "scaling_adjustment": -1, "adjustment_type": "ChangeInCapacity", "scale_up_policy": { "cooldown": 180, "scaling_adjustment": 3, "adjustment_type": "ChangeInCapacity", "alarms": { "scale_down_on_cpu": { "comparison": "<", "alarm_actions": "scale_down_policy", "evaluation_periods": 2, "metric": "CPUUtilization", "namespace": "AWS/EC2", "period": 300, "statistic": "Average", "threshold": 40, "scale_up_on_cpu": { "comparison": ">", "alarm_actions": "scale_up_policy", "evaluation_periods": 1, "metric": "CPUUtilization", "namespace": "AWS/EC2", "period": 60, 8 Chapter 3. Configuring Forseti

"statistic": "Average", "threshold": 60 As you can see, Forseti configuration defines two sections: applications and autoscale. In the first one, we define some aspects relevant to an application: how to deploy it, what ELB does it use... On the other section, we describe the autoscaling parts such as groups, policies, alarms... In this example, we ve defined a backend application and the minimum aspects to make it useful. Let s take a deeper look. Application section The application section is a dictionary which can hold different applications. The key of each application is the name it receives for the Forseti commands, so the application is named backend here. The first interesting part we have inside the application configuration is the strategy. In this case, we re using a deploy and snapshot strategy. "deployment_strategy": "deploy_and_snapshot", This strategy requires a deploy setting in which we define a command which will be the program that will be running inside working_directory. This command can have the special token {dns_name which will be replaced by a comma separated list of the EC2 instances public DNS. "deploy": { "working_directory": "/path/to/capistrano", "command": "cap production deploy -S servers={dns_name", From here, we have specific parts regarding autoscaling. We define the autoscaling group name and the policies it will have. We only list them because the configuration will be in other sections. "autoscale_group": "BACKEND", "scaling_policies": [ "scale_down_policy", "scale_up_policy" ], The next one (being optional) is relative to autoscaling notifications. We can setup the AWS autoscaling to publish messages through SNS whenever an action occurs in the group (launching or terminating an instance). Multiple actions can be defined, just by adding them to the list: "autoscale_notifications": [ { "topic": "arn:aws:sns:eu-west-1:1234567890ab:notifications", "type": "ALL" ], The notifications can be setup to attend only some actions. To do so, you may use one or more of the following values within type : 3.1. Application section 9

autoscaling:ec2_instance_launch autoscaling:ec2_instance_launch_error autoscaling:ec2_instance_terminate autoscaling:ec2_instance_terminate_error autoscaling:test_notification And the last one is relative to Forseti s notifications. It can push messages to a topic in SNS whenever when a deploy is being done. It will send a message when the deploy begins and ends, also when the AMI is being created and the last one when the autoscaling group is finished. To set it up, you have the following options. The section sns_extra_attributes can be used to attach different options to the message published to the SNS topic specified in sns_notification_arn. "sns_notification_arn": "arn:aws:sns:eu-west-1:1234567890ab:notifications", "sns_extra_attributes": { "ParametersForYourSNSApplication": { "Priority": "Info", "Channel": "#notifications" Autoscale section The other part of the configuration file defines all the autoscaling elements. It s divided in four groups. Groups In this section, you ll define the autoscaling groups, ideally one per application. Keep in mind that any application you define, must have an autoscale_group key and it must reference one group inside this section. All the parameters available for an autoscaling group are the same one that boto defines in its API. We ve defined some in this example and you can find the meaning of each one in the boto documentation. "groups": { "BACKEND": { "availability_zones": [ "eu-west-1a", "eu-west-1b" ], "default_cooldown": 300, "termination_policies": [ "ClosestToNextInstanceHour", "OldestInstance" ], "min_size": 2, "desired_capacity": 2, "load_balancers": [ "backend" ], "max_size": 10, 10 Chapter 3. Configuring Forseti

Configurations Every autoscaling group has one or more launch configurations and again all the parameters Forseti accepts are the same ones documented in boto. "configs": { "backend": { "key_name": "private-keys", "instance_type": "m1.small", "security_groups": [ "security-group" ], "instance_monitoring": true, Policies Policies defines how to scale in or out a group. You can do it in absolute numbers and chaning the group capacity in percentual ranges, whatever fits your application needs. All the policies listed for an application must be defined here. The parameters to define a policy are once again defined in the boto documentation regarding autoscaling policies. "policies": { "scale_down_policy": { "cooldown": 180, "scaling_adjustment": -1, "adjustment_type": "ChangeInCapacity", "scale_up_policy": { "cooldown": 180, "scaling_adjustment": 3, "adjustment_type": "ChangeInCapacity", Alarms The last section defines what CloudWatch alarms will trigger the autoscaling policies. In this example, we have two alarms, one that will trigger the increase of instances and another one to decrease them. The important token when defining this alarms is the alarm_actions, which should refer to the autocaling policy to trigger in case the alarm fails. Creating an alarm is easy and all the parameters are defined in the boto documentation as well. "alarms": { "scale_down_on_cpu": { "comparison": "<", "alarm_actions": "scale_down_policy", "evaluation_periods": 2, "metric": "CPUUtilization", "namespace": "AWS/EC2", "period": 300, "statistic": "Average", "threshold": 40, "scale_up_on_cpu": { 3.2. Autoscale section 11

"comparison": ">", "alarm_actions": "scale_up_policy", "evaluation_periods": 1, "metric": "CPUUtilization", "namespace": "AWS/EC2", "period": 60, "statistic": "Average", "threshold": 60 12 Chapter 3. Configuring Forseti

CHAPTER 4 Deployers Forseti has a flexible deployment system, which can be expanded with different deployment strategies and operations. Forseti already comes geared with two different deployers that you can use out of the box, also you can create yours it they don t fit you and submit a PR if you want. Deploy and snapshot The deploy and snapshot deployer is the easiest way of deployment. The process goes as follows: Deploy the application code on the instances belonging to an autoscale group. Select a random instance of the group Remove that instance from the load balancers of the autoscale group. Create an AMI from it (This will reboot the instance). Create a new autoscaling configuration with that new AMI. Update the autoscaling group to use the new configuration. The biggest drawback is that it works better for already existing autoscale groups. Also, it requires a minimum of two running instances in the group in order to use this deployer (remember that one of the instances will be rebooted). If your number of instances vs load is tight your service could be anavailable meanwhile. You can skip any specific instance to be selected for AMI creation by adding a tag forseti:avoid_ami_creation with value True to that instance. This can be useful if the instances are not simetric. For example imagine you have some specific cron tasks in only one of the instances belonging to the autoscale group. Golden instance The golden instance deployer process is similar to: 13

Launch an instance with the gold AMI (defined later). This is called golden instance. Deploy the application code on this new instance. Generate a new AMI from that golden instance. This is called golden AMI. Create a new autoscaling configuration with the new golden AMI. Create or update an autoscaling group to use the new configuration. Create or update the autoscaling policies to specify how AWS will scale up or down. Create or update the CloudWatch alarms which will trigger the autoscaling policies. Wait until the autoscaling group has the new instances with the golden AMI. Deregister the old instances. Before using this deployer Forseti s golden instance deployer uses a gold AMI, which is an AMI packed with all the software you need, except your application specific code. Unfortunately, forseti currently is not able to generate a gold AMI for you, so you ll need to create one manually. This is actually a very easy operation, so don t panic. All you need to do is create a new EC2 Instance from an AMI with EBS root and provision it with all the software you need (apache, nginx, nodejs...) and its configurations (virtualhosts or similars). Once you ve installed everything, create an AMI from the instance by right clicking on it on the AWS Console and select "Create Image (EBS AMI)". There you go, you now have a gold AMI! 14 Chapter 4. Deployers

CHAPTER 5 List of available oficial commands Forseti command system is an extensible system so anyone can create their own commands and operations. We ve created our own set of commands which are bundled by default. Those commands are common operations like creating, regenerating, deploying and showing the status of an application and its autoscaling group. init Initialize application in Forseti using an instance in AWS. Parameters: Options: application: Application name to be deployed instance_id: Id of an instance in AWS to create an AMI from. The id is in the format i-xxxxxxxx --deployer: This parameter is used to tell Forseti how to manage future deploys. For more information, please read our section about deployers. --no-reboot-instance: By default, the instance will be rebooted to create the AMI. If you want to override this behaviour, use this flag. It s not recommended because it doesn t guarantee the filesystem integrity. Examples: forseti init backend i-1a23b4567 --deployer=deploy_and_snapshot Forseti will create an AMI using the given instance and setup a new autoscaling group. This new group won t have any effect because Forseti doesn t create any alarm to scale in or out the instances in the group. forseti init backend i-1a23b4567 --deployer=deploy_and_snapshot --no-reboot- instance The same process but the instance will not be rebooted before creating the AMI. 15

deploy Deploy an application and create or update an autoscaling group. Parameters: Options: application: Application name to be deployed --ami=ami-id: Use this specific AMI to create or update the autoscaling group -- extra_arguments: Extra arguments to be passed to the deploy command as parameters. Note: Please notice the -- before passing the extra_arguments Examples: forseti deploy backend Deploy the backend application. If it was deployed successfuly (that is, the deploy command returned 0) update or create the autoscaling group, configurations, policies and alarms. forseti deploy backend -- --no-verify-ssh Deploy the backend application and pass --no-verify-ssh to the deploy command. forseti deploy backend --ami=ami-xxxxxx Create or update the autoscaling group, configurations, policies and alarms of the application using a specific AMI. regenerate Rebuild the AMI of the instances belonging to an application and regenerate the autoscaling group, configuration, alarms and policies. Notice this doesn t deploy your application code. Parameters: application: Application name to be regenerated status Show the status of the application. This includes information about the instances belonging to the autoscaling group (including the status in the load balancers) and the latest actions which happened in it. Parameters: Options: application: Application name to get the status from. --daemon: Run the status in a loop process. By default, status command will print the status and finish, if you want to monitor the status of an application, you can use this option to do it. --activities=<amount>: Number of autoscaling activities to show. --format=<format>: Output format. By default, Forseti will print the status using terminal colors in a structured format. You can use other formats: tree: Default formatter. json: JSON formatter. plain: Plain format. list_configurations This command will print all the autoscaling launch configurations. You can also get only the configurations of a specific application. 16 Chapter 5. List of available oficial commands

Options: application: Application name to get the autoscaling launch configurations. cleanup_configurations Delete the autoscaling launch configurations to clean up AWS and avoid reaching their limit. By default it will remove launch configurations from all the applications, you can do it in only one application too. AWS has some soft limitations with the number of autoscaling elements you can create. The default limit for autoscaling launch configurations is 50, and considering that each deploy will create a new one, you may reach that limit sooner or later. This command will help you removing old launch configurations and freeing some space and resources. Options: application: Application name to delete the autoscaling launch configurations. --desired_configurations=<desired>: Number of launch configurations to leave. 17

18 Chapter 5. List of available oficial commands

CHAPTER 6 Q&A Why did we call it Forseti? We like to use god s names for our internal projects. We began using only Norse gods, but currently we have also Greek and Roman gods. In the Norse mythology, Forseti is the presiding one and we thought it has some coincidences with the purpose of this application. Forseti is the president of our applications, the utility which rules them all. Why did you create Forseti instead of using other utilities or AWS official tools such as CloudFormation or CodeDeploy? We began building Forseti in 2013 and at that time there were no AWS official tools nor an interface to EC2 Autoscale. There was a good API to create images, alarms and autoscale groups but we missed an easy tool to mix everything. We did some research with [Netflix asgard](https://github.com/netflix/asgard) but it was a bit too much for us. So we started building Forseti to fit our needs and make it quick. 19

20 Chapter 6. Q&A

CHAPTER 7 License Copyright (c) 2015, ticketea dev team <dev@ticketea.com> Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH RE- GARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FIT- NESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CON- SEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 21

22 Chapter 7. License

CHAPTER 8 Internals documentation If you are looking for information on a specific function, class or method, this part of the documentation is for you. Internal documentation 23

24 Chapter 8. Internals documentation

CHAPTER 9 Indices and tables genindex modindex search 25

26 Chapter 9. Indices and tables

Index C cleanup_configurations forseti-cleanup_configurations command line option, 17 D deploy forseti-deploy command line option, 15 F forseti-cleanup_configurations command line option cleanup_configurations, 17 forseti-deploy command line option deploy, 15 regenerate, 16 forseti-init command line option init, 15 forseti-list_configurations command line option list_configurations, 16 forseti-status command line option status, 16 I init forseti-init command line option, 15 L list_configurations forseti-list_configurations command line option, 16 R regenerate forseti-deploy command line option, 16 S status forseti-status command line option, 16 27