Infosys Information Platform How-to Launch on AWS Marketplace Version 1.2.2
Copyright Notice 2016 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the prior permission of Infosys Limited and/or any named intellectual property rights holders under this document. Infosys Information Platform, Release Version 1.2.2 Document Release Version 1.0 June, 2016 Infosys Limited Electronic City, Hosur Road, Bangalore 560 100, India. Website: http://www.infosys.com 2 2016, Infosys Limited
Table of Contents Launching Infosys Information Platform (IIP) on AWS Marketplace... 4 How to Access IIP... 10 Steps to verify IIP Installation... 12 3 2016, Infosys Limited
Launching Infosys Information Platform (IIP) on AWS Marketplace Infosys Information Platform (IIP) leverages the power of open source to address Big Data adoption challenges such as, inadequate accessibility of easy-to-use development tools; fragmented approach to building data pipelines; and lack of an enterprise-ready version of open source big data analytics platform that can support all forms of data: structured, semi-structured, and unstructured. Infosys Information Platform (IIP) has AMI s (Amazon Machine Image) that enable usage of the Platform over AWS. These AMIs, are pre-loaded and pre-configured with IIP and are launched on EC2 instances alongside the required supporting operating system (RHEL 6.5) provided by AWS. Please follow the instructions given below to launch AMIs using Cloud Formation Templates (CFT). Cloud Formation Templates automate the provisioning of the resources required to form an IIP cluster, and ensures that the IIP setup is complete. Pre-requisites Valid Amazon AWS Account for Marketplace Usage Amazon Key Pair for security. Don t have a Key Pair defined? Create Key Pair Security Group for security. Don t have a Security Group defined? Create Security Group Subnet Id for security. Don t have a Subnet Id defined? Reference details Placement Group for better network performance. Don t have a Placement Group defined? Create Placement Group IIP Calculator This calculator listed here will help with the no. of nodes and EBS Volumes required to be added in addition to the local disk. This would guide one to plan AWS Infrastructure for IIP Usage. Note These calculations are to the best available detail, user may contact our Support team for further help. The preference for Data Store is on EBS Volumes over local disk space mounted to EC2 instances, to avoid any data loss in case of any unforeseen machine restart. Refer to the details below for Launching Infosys Information Platform on AWS: A Cloud Formation Template has been created to guide user through the process of getting the initial IIP cluster setup, with CFT launching Infosys Information Platform in the AWS Marketplace is made simple and easy. During this process of setup, the CFT shall create a new S3 bucket, IAM Role, IAM Policy, ports to the existing security group (given at the end of the document), Auto Scaling Group and launch Amazon EC2 instances. 4 2016, Infosys Limited
To enable IIP in the AWS Marketplace, Click Continue. (Note: User will not be charged until the instance is launched in later steps.) Before launching IIP, please make sure the AWS account and user has the required permissions to create ec2-instances, create EBS volumes, create stack, update security group, create IAM role, IAM policies, S3 bucket (Example Users admin account will have full access for the same). Go to the AWS Cloud Formation Console here Select Create Stack or Create New Stack 5 2016, Infosys Limited
Select the option, Specify an Amazon S3 template URL and copy the below S3 link https://s3.amazonaws.com/iipmarketplace-122/iipmarketplacecft.template of Infosys Information Platform Cloud Formation Template into the textbox and click on Next Complete the required information for IIP Cloud formation template 6 2016, Infosys Limited
7 2016, Infosys Limited
Stack Name specify a name for the stack, so that user can differentiate between the IIP cluster and other clusters. InstanceType - specify the type of EC2 instances which user wants the cluster to contain. Keyname - name of an existing EC2 key pair. Key pairs help users to keep the cluster secure. MasterTagName- specify the name of the MasterNode of the cluster. MaxSize specify the number of slave machines (i.e. DataNodes) for the cluster. By default it is 2 and maximum is 20. If the nodes has to be increased beyond 20, please drop an email to IIPAWS_Support@infosys.com for further assistance. PlacementGroup specify the placement group. Using placement groups, enables applications to participate in a low-latency, 10 Gbps network. SecurityGroupIds- specify name of an existing Security Group, it acts as a virtual firewall that controls the traffic for one or more instances. SubnetId- specify name of an existing subnet id, this enables to communicate with other instances in the VPC VolumeSize specify the size of the EBS volume of type magnetic to attach to the instances. WhitelistIP specify the IP address and the CIDR netmask for all the IP addresses that can connect to the newly launched EC2 instances. Specify the tag value with any text and Click on Next 8 2016, Infosys Limited
9 2016, Infosys Limited
Review the information entered, select the checkbox that says: I acknowledge that this template might cause AWS Cloud Formation to create IAM resources. Click Create to start provisioning the required infrastructure and installing IIP. The IIP installation generally takes 90 minutes to complete on an average after which the user will be able to access IIP by referring to the How to Access IIP section defined below. If there is any issue or IIP URL is not accessible then kindly, see the Troubleshooting section for more information. For any queries or assistance please send an email to IIPAWS_Support@infosys.com How to Access IIP To access the cluster, user will need to use SSH (Secure Shell) to open a connection to the cluster and use.ppk file of the key name that was used to create the cluster. Login as an ec2- user to the instance. Select Outputs section of the Stack, IIP Cluster details are mentioned in the S3 bucket and the reference link is provided in IIPClusterDetails property value. On click of the URL, it navigates to S3 bucket, IIP_Cluster_Details.txt file contains IIP login and credentials details. iipadmin user and the password provided in the file will be created across all the nodes of the cluster. 10 2016, Infosys Limited
11 2016, Infosys Limited
Steps to verify IIP Installation SSH to the NameNode EC2 instance as ec2-user Execute this command # tail -100f /home/ec2-user/log.txt If the user gets an output like the one specified below which contains the line IIP Installation successfully completed. NameNodeIP: <NameNodeIP>. Total number of instances= This states that you are good to go and start using Infosys Information Platform. Press Ctrl-C to exit from log.txt and open the IIP URL in Mozilla Firefox or Google Chrome with the credentials provided in the IIP_Cluster_Details.txt file in your S3 storage. In case, public IP of the instances in not reachable from the network, please replace the public IP with the respective private IP of the instance and try accessing the instances. If the message is IIP Installation Failed. NameNodeIP: <Name Node IP>. Total number of instances=. Please check the log for errors. then execute the below mentioned command: # tail -1000f /home/ec2-user/oneclickprovision_1.2.2/logs/* The point where error has occurred will be specified in the logs. Copy both log files (i.e. - /home/ec2-user/log.txt and the log_ddmmyyyy_hh:mm:ss.log) file from /home/ec2-user/oneclickprovision_1.2.2/logs and send an email to IIPAWS_Support@infosys.com 12 2016, Infosys Limited
Failed instances can be terminated from Cloud Formation console. But, before you delete the stack: Go to S3 console and you will find a bucket (named in the format of StackName-mybucket-*), empty that bucket. Go to Cloud Formation console and select the stack which was created. Click on actions and delete the stack. All the AWS resources that was created will be deleted automatically when the stack is deleted. Execute the below command to check if the services are up in Name Node. # jps If you get an output similar to the one shown below in Name Node, it states that all IIP related services are active and running. 13 2016, Infosys Limited
SSH to any one of the DataNode EC2 instance as ec2-user Execute this command # jps If you get an output similar to the one shown below, it states that all IIP related services are active and running. 14 2016, Infosys Limited
Notes: Below Ports will be added to the Security Group specified in the CFT to access IIP platform: Port Number Process name Remarks 9094 YARN MANAGER Administrative Purpose 10000 hiveserver2 Administrative Purpose 10001 hive thrift server Administrative Purpose 3306 Mysql Administrative Purpose 8089 Azkaban Web UI Administrative Purpose 4040 spark server Administrative Purpose 8787 R URL Mandatory for IIP UI 8080 IIP/tomcat Mandatory for IIP UI 21,22 ssh(ftp, SFTP) Mandatory for few functions 50070 HDFS UI Administrative Purpose 449 Chef secure connection Administrative Purpose 89 Chef connection Administrative Purpose 1024-65535 Inter-communication between the Data Nodes Will be enabled only for the Security Group 15 2016, Infosys Limited