Orchestrator ver

Transcription

1 EPAM Cloud Infrastructure Orchestrator ver What s New March 2015 CI2WN-S37-39 Version 1.0

2 Content Content Overview EPAM Private Cloud Billing Changes CLI Workspaces Rebranding AWS Improvements... 7 Frankfurt AWS Region Activation... 7 AWS Login Names Changed... 7 Shapes Mapping Changed... 9 IAM Users Management and Audit... 9 AWS-Specific Reporting External Account Management Using Native Auto Configuration on Amazon CloudFormation Support Improvements Connection to Instances Networking Control Commands SSH Keys Manipulations Update EPC Services Improvements GrayLog Server Update Docker Service Ambari Service Cloud Support Updates Flexible Billing Cloud Consulting Events Knowledge Base Improvements New Webinar Documentation Updates Version History EPAM Systems 2

3 1 Overview EPAM Orchestration Framework version was released on March 21 st, We are glad to introduce the following new features and improvements: EPAM Private Cloud Costs Cutting Multiple workspaces for CLI are introduced EPC rebranding started New AWS region - Frankfurt - is now available for EPC users A set of networking commands is introduced to allow assigning public IPs to your AWS-based instances via Maestro CLI and Maestro Stacks Shapes mapping is changed for running instances in AWS SSH Keys manipulation process is restructured. IAM Users management is improved Native Auto Configuration for AWS instances is supported External AWS account management is supported Gray Log, Docker and Ambari services are improved Billing strategies flexibility is improved All these improvements are described in details further in this document. Our users often address us with the questions for EPAM Private Cloud SLA details and clarifications. For you to be more acquainted with the subject and to prevent misunderstandings and unexpected situations, we strongly recommend to review EPAM Private Cloud Service Offering document. 3 EPAM Systems

4 2 EPAM Private Cloud Costs Cutting Those who have been with us for already several months, noticed, that we apply efforts to optimizing EPC billing strategies to make it more transparent and friendly to our users. This time, we introduced a number of changes that are aimed to unify prices for EPC regions and to cut the prices due to reduction of some of the prices. Thus, starting from April 1, the following region coefficients will be active: Location Service Region Coefficient Europe (Budapest, Hungary) EPAM-HU1 1 CIS (Minsk, Belarus) EPAM-BY1 1 CIS (Saratov, Russia) EPAM-RU1 1 Ukraine (Kiyiv) EPAM-UA1 1 Ukraine (Kiyiv) EPAM-UA1 1 USA (Princeton, NJ) EPAM-US1-PROD 1.4 USA (Princeton, NJ) EPAM-US 0.9 CIS (Minsk, Belarus) EPAM-MSQ3 0.8 Therewith, we reduce cut the following components of the VM prices: Component $/CPU old $/CPU new CPU 8 7 Soft&Labor These changes reduce the monthly price of VMs with one CPU by about $5. For more detail on EPAM Private Cloud billing, please, see our Billing Guide. EPAM Systems 4

5 3 CLI Workspaces When working with Maestro CLI, you often need to input similar parameters time after time. Maestro CLI allows to set up default parameter values that are fetched automatically if they are not given explicitly. These parameters are set in %Maestro_CLI%lib/default.properties file according to property_name=property_value syntax. For example, default project and region specification would look like: project=demopro region=epam-by1 However, our users often work with several environments in different locations and projects, having different typical parameter values. With the current update, you can set up different workspaces and different defaults collections in order to simplify your work with Maestro CLI. Workspaces setup is performed into several steps: 1. Add the %Maestro_CLI%bin path to the PATH environment variable 2. On your workstation, create a new folder for a new workspace. We would recommend to name it after project and region in which you are going to use this workspace. 3. Create a default.properties file in the new folder and specify the properties for the new workspace in it. After that, when working with Maestro CLI, just switch to the necessary workspace and run a command. The default properties will be automatically taken from the currently active folder. If the current folder includes no properties file, those specified in %Maestro_CLI%lib/default.properties will be used. EPAM Private Cloud team would like to thank Attilla Kisko for suggesting the Workspaces idea. 5 EPAM Systems

6 4 Rebranding EPAM is currently going through total rebranding of all company resources and services. This is also true for EAPM Private Cloud. Thus, we are glad to say that Orchestration v comes with the new Cloud Management Console design (desktop version). We updated the color scheme and elements layout, and introduced the Dashboardand introduced the Toolbar that gives access to the main Management Console monitoring and audit facilities, and we also updated our web-site to make it fit the new Cloud Style: However, the rebranding process does not end on this, and our nearest plans include the following changes: Reports design updates; Mobile Console design updates; Documentation design updates. EPAM Systems 6

7 5 AWS Improvements In current update, we go on introducing improvements in EPC integration with AWS. These changes are aimed to make your work with Amazon more effective, as well as this is another step in unification of procedures of EPC and AWS-based resources manipulation. EPAM Private Cloud team is grateful to Alexander Belousov, who provided detailed and sound feedback on AWS integration with EPC and suggested a number of improvements. Thanks to this feedback, we implemented a number of important changes that make AWS usage more convenient and user-friendly. Frankfurt AWS Region Activation Several months ago, AWS introduced a new Europe region, AWS Germany (Frankfurt). The region is based on the new hardware and provides fast and low-latency access to AWS facilities, and we are glad to announce, that with the current release, EPAM Orchestration supports working with this region, referenced as AWS-EUCENTRAL. AWS Services The new region supports most of AWS services, including EC2, Virtual Private Cloud, Auto Scaling, Elastic Load Balancing, and others. You can find the full list of the services available for AWS-EUCENTRAL region on Regional Product Services page. Pricing You can find the pricing for AWS Services in Frankfurt region in the detailed descriptions of each service on AWS Products and Services page. AWS Login Names Changed Please be informed that with the new update, the login names to AWS instances run with EPC were changed to those typically used in Amazon: Image OS Ubuntu CentOS Debian Default Login ubuntu ec2-user admin To get the login details for a Windows instance, login to AWS management console by following the link returned by the or2-aws-management-console Maestro CLI command (or use your IAM user credentials, if you have them). After that, go to the Instances section, rightclick on the instance to which you want to get access and select Get Windows Password in the dropdown menu. 7 EPAM Systems

8 EPAM Systems 8

9 Shapes Mapping Changed As it has been mentioned above, a new AWS Region, AWS-EU-CENTRAL (Frankfurt) is now available for EPAM Private Cloud users. This region works with resources of new generation, both hardware and virtual. This means, that the new region operates new types of instances and images. The other AWS regions have also been under update recently. The changes include not only the new elements, but also the ways they are processed. In different regions, each machine image has its own set of image types that can be used. As you know, in order to unify working with EPC and AWS instances via EPAM Orchestration, we introduced a collection of images and shapes aliases. The same alias are used for running instances in EPC and AWS regions, but the actual shape or image is invoked depending on the region you use at or2run command call. The recent changes in AWS images/types treatment lead to the necessity to introduce dynamic shape discovery and to review the existing alias mapping. Thus, some of image/shape combinations are not applicable in EPC. If you happen to call the or2run command with such combination, you will get an error message and the list of the available shapes for the selected image: IAM Users Management and Audit With the previous release, we introduced the two ways to get access to AWS management console via EPAM Private Cloud: Retrieving temporary access by calling the or2-aws-management-console command and following the link returned in the command response. The access is closed after an hour idle. Getting a constant access by requesting an IAM user. In this case, the access to AWS management console is unlimited in time. The access retrieved by following the second option grants Power User permission. In order to provide control and monitoring over Power User actions in AWS, we introduce the following functionality: IAM users creation audit. You can find the story of IAM users creation by calling the or2audit command with -g project parameter in Maestro CLI: or2-audit -p sample -r sample -g project Among other events, IAM users manipulations will be shown: Automatic IAM users management. Although IAM users are created manually, Orchestration constantly tracks the status of EPAMers who have IAM users for the following changes: o Whether a user status has changed to FIRED o o Whether the user has no Cloud Role (neither zcloudadmin nor zclouduser) Whether the user was removed from the project for which they had an IAM user activated. If at least one of these checks returns true result for a user, Orchestrator sends a request to AWS for their IAM user to be removed. 9 EPAM Systems

10 Cloud Support groups gets a weekly report on IAM users creation and deletion, as well as on general statistics on IAM users registered in EPC. AWS-Specific Reporting To run a VM in AWS via EPAM Orchestration, you use an alias name to specify which image and shape is to be used. The alias reference is transformed to a real AWS shape/image reference, that are used at the instance start. It is them that will be the real parameters of the new instance. Starting with this release, we use not alias, but the real Amazon image names in our reports to provide better visibility of resource usage. We also added Private and Public IP and DNS addresses info: External Account Management Sometimes, projects are provided with AWS accounts from outside EPC (for example, they can be granted by the customer). From now on, EPAM Private Cloud provides the ability to put these account under Orchestration control in order to establish infrastructure monitoring and quotas management. To put an account under EO control, please, submit a Request to HelpDesk. An external Account can be put under EO control only if the related project was not activated in Amazon before that. If it was, one can request creating a new PMC project and assign the account to it. Using Native Auto Configuration on Amazon During the previous history of AWS and EPC integration, using auto configuration on AWSbased instances needed special preparations. Auto configuration was available only on images that were specially prepared by EPC Support Group and included scripts that installed Chef and basic auto configuration. With the current release, we moved to a new Auto Configuration system, that allows you to use Auto Configuration on Amazon without any previous explicit preparation. The creation of EPAM Systems 10

11 new instances now followed by starting a script that automatically downloads, installs and runs Chef client on AWS instances. CloudFormation Support Improvements EPC supports working with both Maestro Stacks and Amazon Cloud Formation stacks to set up automatic infrastructure deployment in AWS. Our developers have been introducing this integration step by step during the latest updates, and now we are proud to announce that EPAM Orchestrator allows to take all the advantages of CloudFormation stacks. Please also be informed that the following changes were introduced to Amazon CloudFormation templates processing: Commands or2-upload-template, or2-describe-templates, or2-read-template were deactivated. Amazon Cloud Formation templates are now referred as cf-template files in the following commands: or2-upload-files; or2-describe-files; or2-delete-files. Only file owner can delete the template file previously uploaded to Orchestration All files are completely deleted from DB and cannot be restored. After the deletion, the user gets an with the content of the deleted file. You can also use Dashboard to manipulate cf-template files. Use the Run wizard to run a Cloudformation stack template, or Stack Builder to duplicate or delete an existing cftemplate: 11 EPAM Systems

12 6 Connection to Instances In Orchestration v , we paid special attention on the connection to instances. There are two main improvements we are glad to introduce: Supporting a set of CLI commands that allow to assign static IP addresses to your instances SSH keys manipulation flow changes Networking Control Commands There are situations in which the users need to make their Cloud VMs available externally via Internet. Currently, the described functionality can be used only with AWS-type regions. For long time, to get such access, one had to submit a Support request to HelpDesk and wait till the request is fulfilled. In order to make the process more user friendly and make it fit the self service concept, we implemented the possibility for you to grant external access to your VMs by a simple set of Maestro CLI commands: Command or2-allocate-static-ip (or2allsip) or2-associate-static-ip (or2assip) or2-describe-static-ips (or2dsip) or2-disassociate-static-ip (or2dissip) or2-release-static-ip (or2relsip) Description Allocates a static IP for a project Associates a static IP with an existing instance Describes the static IPs allocated for the project Disassociates the static IP from its VM Removes the specified IP from the project pull The general flow for getting a static IP for your VM is quite simple: first, you allocate a static IP to your project, and then you associate one of the allocated and free Static IPs with the VMs on this project Static IP 1 Static IP 2 Allocate Project Associate VM3 Static IP 2 VM1 VM2 When needed, you can disassociate the static IP from the VM and release it from the project. Below, you can find the usage examples of static IP - related commands: Allocates a static IP for a project: or2allsip -p sample -r aws-useast Assigns a static IP to a VM: EPAM Systems 12

13 or2assip -p sample -r aws-useast -i instance -a <address> Describes static IPs available for the project: or2dsip -p sample -r aws-useast Disassociates a static IP from a VM: or2dissip -p sample -r aws-useast -a <address> Removes the specified IP from the project pool: or2relsip -p sample -r aws-useast -a <address> For more details on these commands usage and parameters, please, see our Maestro CLI Reference Guide, Section 7: Security and Connection. Please note that retrieving static IPs for AWS-based instances is based on AWS Elastic IP Addresses service. The commands used in Maestro CLI functionally reproduce those used in AWS for Elastic IPs manipulations (ec2-allocate-address, ec2-describe-addresses, ec2- associate-address, ec2-disassociate-address, ec2-release-address). Thus, the specifics and limitations of Amazon EC2 Elastic IP Addresses are delegated to the static IPs you can retrieve via EPAM Orchestration: The number of IPs available at once for one project, is 5 An IP that is allocated to a project but not associated to a VM, is charged for $0.1/hour. You can find more details on Amazon EC2 Elastic IP Addresses service on the respective page of AWS web-site. SSH Keys Manipulations Update Before this update, the users who needed to use SSH keys for logging to instances in Cloud, had to create or load a single key for a project, and this key was available on any region. Starting from Orchestration v , a key should be loaded to each region explicitly. To make the process faster and easier, we introduce a number of key manipulation updates: The or2-create-keypair command now includes -r/--region parameter as a required one. The or2-import-keypair (or2key) command is implemented for importing the created key to a new region. If the key was created in EPC, its public part is stored in Orchestration database, and all you have to do is specifying the region in which the key is already stored and the region where it should be imported to: or2ikey -p sample -k keyname -s source_region -d destination_region If the key is created in AWS and you need to import it to a new AWS region, you will also have to specify the path to the Public key. The detailed guidelines on retrieving public AWS keys you can find by the following links: Linux Windows A key created with AWS tools will be referenced in EPC with the same name that was assigned to it during the creation. The keys created in EPC can be imported to AWS regions, whereas AWS-originating key can not be imported to EPC regions, as their data is stored in AWS database and is not shared outside. 13 EPAM Systems

14 The diagram below illustrates the most simple way to manipulate SSH keys in EPC: Maestro CLI Tools AWS Tools or2addkey Create AWS key Any Region AWS Region or2ikey or2ikey Get public key EPC Region AWS Region or2ikey AWS Region Please note that the keys which previously existed in Cloud were automatically imported to all EPC regions in which the respective project were activated. As to the AWS regions, please, import the keys there by yourselves using the or2-import-key command. EPAM Systems 14

15 7 EPC Services Improvements EPAM Private Cloud traditionally includes EPC services improvements to a new build. This time, we concentrated on improving the existing services, that arouse interest among our users: Ambari and GrayLog Services. GrayLog Server Update Several months ago, we introduced the Log Aggregation Service, based on GrayLog With the current release, we update the Gray Log Service and put GrayLog 1.0 behind it. The previous version of the service was deprecated. To get all the advantages of the new Log Aggregation service and to avoid issues with the older version, please, stop the existing Graylog Service with the following command: And then restart the service with or2ms -p project -r region -s log --deactivate or2ms p demopro r demoreg s log -activate Log Service deactivation will result in data loss. Please, make sure you have saved all the data that can be useful for you in future before you deactivate the service. In case you need assistance with this, feel free to address our Consulting Team. Among the main improvements of GrayLog 1.0 are the significantly increased performance, enhanced visualization, editable inputs and extractors, Grok support, stable and documented plugin API, and others. You can find the detailed descriptions of these improvements on GrayLog 1.0 Announcement page. The login and password that are to be used during the login to the server are returned at or2-describe-services command call: 15 EPAM Systems

16 Docker Service Orchestration v comes with the updated Docker Service that acquired an important feature - the ability for the user to select the image that will be used to run a container. This feature is implemented as another Docker command, available in CLI: Command or2-manage-service (or2ms) or2-docker-container (or2dc) or2-describe-docker (or2dd) or2-docker-image (or2di) Description When used with -s docker and --activate flags, starts the Docker service on the specified project and region Used to create and manipulate Docker containers Retrieves the list of Docker elements and their details Gives the list of available images to be used on Docker containers Each Docker hypervisor now goes with a set of four images. The list of these images can be retrieved by calling the or2-docker-image (or2di) command and providing the project and region in which the Docker service is run: The Name identifies the name of the image and the operating system, the Tag identifies the version of the system that will be installed. By default, ubuntu_epc image is used on Docker containers run. To use one of the other images, add the -i/--image parameter to the or2-docker-container (or2dc) command. As the parameter value, you can use the image ID or name:tag structure. Both examples below will run s container with debian_epc image: or2dc -p DEMOPRO -r DEMOREG -h hypervisor_id -c 8c3ab2e97b3f -a run or2dc -p DEMOPRO -r DEMOREG -h hypervisor_id -c debian_epc:7 -a run Ambari Service A significant part of any improvement is not only introducing new features and capabilities, but also paying careful attention to making the product more stable. Your feedback is very important for us in this respect. With this release, we stabilized the Ambari service by fixing a number of bugs and issues. We would like to give special thanks to our colleagues from the Big Data Competency Center and, especially, Igor Rudyak who provided their detailed feedback on the Ambari Service and shared a number of valuable suggestions that were scheduled for implementation. Please note that connection login changes for Ambari Servers hosted in Amazon. Starting from March 21, to connect to Ambari server run in AWS, use the ec2-user login and the SHH key you specified at service start. EPAM Systems 16

17 8 Cloud Support Updates With the current update, we also introduce a set of updates aimed to help Cloud Support teams in maintaining our service. Please note that the features, described in this section, are available for Cloud Support teams only. Flexible Billing EPAM Orchestration gathers the billing information by means of the Billing Engine. A part of this engine is a Quotas control mechanism that allows to control project expenses. Account manager and Project Coordinator can use UPSA to set quotas and specify the actions that should be taken when quotas limits gets exceeded - either to stop all the resources or not. But this is the control from user s side. For better Cloud usability and adaptability to various conditions and requests, it is very important to deliver a flexible billing system that would allow changing existing billing strategies in minutes. With Orchestration v , we moved to a new billing processing system that allows to introduce flexible billing strategies. This functionality is available only to Cloud Support Team 17 EPAM Systems

18 9 Cloud Consulting Events February was the Cloud Month in EPAM. During this month, Cloud experts delivered a number of meetings, webinars and lectures aimed to introduce the latest innovations in Cloud and to delegate Cloud-related knowledge to their colleagues in EPAM. The following presentations and webinars were delivered in February: Webinar About Presenter Lang Round Table. Cloud Solution Amazon Cloud for Architects Deployment and Management Services in AWS Three Cloud Competency Center experts: (Head), (Delivery Support), and (Consulting) telling about cloud trends and market, Cloud Competency Center activities, useful features of EPAM Private Cloud, and answer the questions from the Audience. The webinar delivers the general information on main Amazon services, gives examples of architecture best practices and solutions (hosting, high availability, disaster recovery, media encoding, and others) The webinar by reveals the details on Elastic BeansTalk, OpsWork, CloudFormation, and CodeDeploy services provided by Amazon Web Services. Eli Feldman Aaron Dodd Olena Matokhina Oleksandr Bukhanko Oleksandr Bukhanko EN EN EN Microsoft Azure The webinar by makes you acquainted with Azure as a Cloud platform, available services, supported databases, best practices and pricing. Pavlo Revenkov EN Cloud Computing from A to Z The webinar closes the lectures from Month of Cloud Architecture cycle, recalls the main Cloud concepts in simple, not highly technical way, and provides suggestions on the future of Cloud technologies. Andrii Gusiev RU EPAM Systems 18

19 10 Knowledge Base Improvements In current release, our documentation team concentrated on updating the existing documentation. We updated a bunch of important documents, published an article and a blog post telling a bit about EPAM Private Cloud team. New Webinar This month, EPAM Private Cloud Consulting team developed a new webinar, aimed to explain the main concepts of EPC integration with AWS. This covers the following topics: What is Cloud Computing? AWS Usage Roadmap AWS and EPC Management Consoles Resources unification Running a VM in AWS The training presentation is available by this link. We can also arrange a webinar for your team, and deliver it at a convenient time. The webinar is 1.5 hours long and can be delivered in English and Russian languages. To request a webinar, please contact your local CDP department or Svitlana Ostnek directly. We have also prepared an introductory Cloud Toon that will tell you about EPC integration with AWS. Documentation Updates In current release, we introduce two major documentation updates, a new training and a set of other changes. The following documents were updated: Quick Reference Card was updated according to the recent changes Maestro CLI Reference Guide - new commands and parameters added Cloud Services Guide - New commands and service details are added Regions information was updated throughout all documentation. Please note that all the documents are in constant update and development. Feel free to address EPAM Cloud Consulting or Svitlana Ostnek directly if you have any questions or remarks. 19 EPAM Systems

20 Version History Version Date Summary 1.0 March 21, 2014 Initial version is published EPAM Systems 20

21 Global 41 University Drive Suite 202, Newtown (PA), 18940, USA Phone: Fax: EU Corvin Offices I. Futó st Budapest, H-1082, Hungary Phone: Fax: CIS 9th Radialnaya Street, Building 2 Moscow, , Russia Phone: Fax: EPAM Systems. All Rights Reserved.