Hitachi Hyper Scale-Out Platform (HSP) MK-95HSP013-03 14 October 2016
2016 Hitachi, Ltd. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or stored in a database or retrieval system for commercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data Systems Corporation (collectively, Hitachi ). Licensee may make copies of the Materials provided that any such copy is: (i) created as an essential step in utilization of the Software as licensed and is used in no other manner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials. "Materials" mean text, data, photographs, graphics, audio, video and documents. Hitachi reserves the right to make changes to this Material at any time without notice and assumes no responsibility for its use. The Materials contain the most current information available at the time of publication. Some of the features described in the Materials might not be currently available. Refer to the most recent product announcement for information about feature and product availability, or contact Hitachi Data Systems Corporation at https://support.hds.com/en_us/contact-us.html. Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicable Hitachi agreements. The use of Hitachi products is governed by the terms of your agreements with Hitachi Data Systems Corporation. By using this software, you agree that you are responsible for: 1) Acquiring the relevant consents as may be required under local privacy laws or otherwise from authorized employees and other individuals to access relevant data; and 2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance with relevant laws. Notice on Export Controls. The technical data and technology inherent in this Document may be subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations, and may be subject to export or import regulations in other countries. Reader agrees to comply strictly with all such regulations and acknowledges that Reader has the responsibility to obtain licenses to export, re-export, or import the Document and any Compliant Products. Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries. AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eserver, FICON, FlashCopy, IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/os, z9, z10, z13, z/vm, and z/vse are registered trademarks or trademarks of International Business Machines Corporation. Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft, the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQL Server, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo, Windows Azure, Windows PowerShell, Windows Server, the Windows start button, and Windows Vista are registered trademarks or trademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission from Microsoft Corporation. All other trademarks, service marks, and company names in this document or web site are properties of their respective owners.
Contents Preface Intended audience Product version Document revision level Accessing product documentation Getting help Comments v vi vi vi vi vii vii Quick Reference 1 KVM image details 2 CentOS VM software bundle 3 Repository VM software bundle 3 Bundled components 4 Licenses used 5 Contents iii
iv Contents
Preface This document provides a quick reference for Ambari 2.2.2 and Hortonworks Hadoop Data Platform (HDP) 2.4 VM for Hitachi Hyper Scale- Out Platform (HSP). This Preface includes the following information: Intended audience Product version Document revision level Accessing product documentation Getting help Comments Preface v
Intended audience Intended audience This document is intended for system administrators, Hitachi Data Systems representatives, and authorized service providers who require quick reference information for Hortonworks Hadoop Data Platform (HDP) 2.4 VM for Hitachi Hyper Scale-Out Platform (HSP). Readers of this document should be familiar with the following: KVM based Virtual Machine images Ambari administration Linux operating system and working in a restricted shell environment Site-specific network information Product version This document applies to Hyper Scale-Out Platform release 1.2.037 or later. Document revision level Revision Date Description MK-95HSP013-00 April 2016 Initial release MK-95HSP013-01 April 2016 01, supersedes and replaces MK-95HSP013-00 MK-95HSP013-02 June 2016 02, supersedes and replaces MK-95HSP013-01 MK-95HSP013-03 14 October 2016 03, supersedes and replaces MK-95HSP013-02 Accessing product documentation Product documentation is available on Hitachi Data Systems Support Connect: https://knowledge.hds.com/documents. Check this site for the most current documentation, including important updates that may have been made after the release of the product. vi Preface
Getting help Getting help Hitachi Data Systems Support Portal is the destination for technical support of products and solutions sold by Hitachi Data Systems. To contact technical support, log on to Hitachi Data Systems Support Connect for contact information: https://support.hds.com/en_us/contact-us.html. Hitachi Data Systems Community is a global online community for HDS customers, partners, independent software vendors, employees, and prospects. It is the destination to get answers, discover insights, and make connections. Join the conversation today! Go to community.hds.com, register, and complete your profile. Comments Please send us your comments on this document: hsp.documentation.comments@hds.com Include the document title and part number, including the revision (for example, -01), and refer to specific sections and paragraphs whenever possible. All comments become the property of Hitachi Data Systems. Thank you! Preface vii
viii Preface
Quick Reference This chapter describes the Hortonworks Hadoop Data Platform (HDP) KVM based Virtual Machine image for HSP. KVM image details Bundled components Licenses used Quick Reference 1
KVM image details KVM image details Hortonworks Ambari deployment begins with a fresh CentOS 7 KVM based VM image and a repository VM image. The package consists of two VM images: CentOS-7.2 HSP-1.2 Vanilla HSP VM template Repo-with_HDP-2.4_Ambari-2.2.2 Repository VM The following describes the basic steps involved in deploying the VMs, see the Hortonworks Ambari VM deployment guide for full instructions. 1. Add the CentOS template and repository template to the HSP cluster and deploy a single instance of the repository VM to the cluster. 2. Deploy the Ambari VMs. One or more Ambari VMs can be deployed to form the HDP cluster. 3. The install script is packaged as an RPM on the repository VM. The Ambari-prep-2.2.2-1.noarch.rpm script sets up the /etc/hosts file so that all of the nodes can find each other. SSH into the Ambari VM you will use as your Ambari server and use wget to fetch the script. ssh <Ambari Server VM IP Address> wget <repository VM IP Address>/rpm/ambari-2.2.2/AmbariPrep-2.2.2-1.noarch.rpm 4. Install the script package using yum: yum -y install AmbariPrep-2.2.2-1.noarch.rpm 5. Run the script. /opt/escale/bin/ambariprep.py --repo-host <Repository VM IP address> When complete Ambari will have been installed, configured and started. The Ambari server GUI can take up to three minutes to start responding on port 8080. 6. Open a browser, log into the Ambari server instance and configure the new cluster from the browser http://<ambari server IP address>:8080 2 Quick Reference
KVM image details CentOS VM software bundle CentOS 7.2 x64 - Linux Operating system Installed Software Oracle JDK 1.8 jdk-8u60-linux-x64 Java virtual machine Python 2.7.5 Python interpreter (for scripting language) PostgreSQL 9.2.15-1.el7_2.x86_64 Repository VM software bundle HDP VM with Ambari Server/local HDP 2.4 repositories. Packaged Software Ambari Server 2.2.2 HDP 2.4 - Hortonworks Hadoop software HDFS compatible plugin for HSP Quick Reference 3
Bundled components Bundled components The following Hortonworks HDP components have been bundled and are pending certification to be compatible with the Hitachi Scale-Out Platform version 1.2.037 Apache DataFu 1.3.0 Apache Hadoop 2.7.1 Apache HBase 1.1.2 Apache Hive 1.2.1 Apache Pig 0.15.0 Apache Spark 1.5.2 Apache Sqoop 1.4.6 Apache Tez 0.7.0 Apache ZooKeeper 3.4.6 4 Quick Reference
Licenses used Licenses used This media kit includes software covered under the following licenses: Apache License, Version 2.0 (the "License") The Platform Components comprising the Hortonworks Data Platform (HDP) are released under the Apache 2.0 License. Unless required by applicable law, software distributed under the Apache License is distributed on an AS IS BASIS. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the Apache License at: http://www.apache.org/licenses/license-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Optional Add-Ons and 3rd-Party Components While HDP is provided under the terms of the Apache 2.0 License, it is also commonly used with 3rd-Party Components and Optional Add-Ons (example Hive ODBC Driver). When you choose to use those components, it is recommended you read and understand the licensing terms specific to each of those components. Also note that Add-Ons can be downloaded separately from HDP and each comes with instructions on how to install and leverage the features alongside HDP. Trademarks: Apache and the Apache feather logo are trademarks of The Apache Software Foundation CentOS is a trademark of Red Hat, Inc. Hortonworks Data Platform (HDP) is trademark of Hortonworks Inc. Python is a registered trademark of the Python Software Foundation Quick Reference 5
6 Quick Reference
Hitachi Data Systems Corporate Headquarters 2845 Lafayette Street Santa Clara, California 95050-2627 U.S.A. www.hds.com Regional Contact Information Americas +1 408 970 1000 info@hds.com Europe, Middle East, and Africa +44 (0) 1753 618000 info.emea@hds.com Asia Pacific +852 3189 7900 hds.marketing.apac@hds.com MK-95HSP013-03