Beta. VMware vsphere Big Data Extensions Administrator's and User's Guide. vsphere Big Data Extensions 1.0 EN

Transcription

1 VMware vsphere Big Data Extensions Administrator's and User's Guide vsphere Big Data Extensions 1.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see EN

2 VMware vsphere Big Data Extensions Administrator's and User's Guide You can find the most up-to-date technical documentation on the VMware Web site at: The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: Copyright 2013 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at VMware is a registered trademark or trademark of VMware, Inc. in the United States and other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. VMware, Inc Hillview Ave. Palo Alto, CA VMware, Inc.

3 Contents 1 About VMware Big Data Extensions 5 2 Getting Started with Big Data Extensions 7 for Big Data Extensions 8 Deploy the Serengeti vapp 9 Install the Big Data Extensions Plug-in 10 Connect to a Serengeti Server 12 Install the Command-Line Interface Console 12 Create a New Password for the Management Server 13 Create a New Username and Password for the CLI Console 14 About Tarball- and Yum-Deployed Hadoop Distributions 14 Configure a Tarball-Deployed Hadoop Distribution 15 Configure a Yum-Deployed Hadoop Distribution 16 3 Managing Resources for Hadoop and HBase Clusters 19 Add a Datastore to Big Data Extensions 19 Remove a Datastore 20 Add a Network to Big Data Extensions 20 Remove a Network 21 4 Create Hadoop and HBase Clusters 23 About Hadoop and HBase Clusters 23 Create a Hadoop or HBase Cluster 23 5 Managing Hadoop Clusters 27 Start and Stop a Hadoop Cluster 27 Scale Out a Hadoop Cluster 28 Delete a Hadoop Cluster 28 About Resource Usage and Elastic Scaling 28 Optimize Cluster Resource Usage with Elastic Scaling 29 Use Disk Shares to Prioritize Cluster Virtual Machines 30 6 Monitoring Hadoop Clusters 31 View Provisioned Hadoop Clusters 31 View Cluster Information 32 Monitor the Hadoop Distributed File System Status 33 Monitor MapReduce Status 33 Monitor HBase Status 33 Index 35 VMware, Inc. 3

4 VMware vsphere Big Data Extensions Administrator's and User's Guide 4 VMware, Inc.

5 About VMware Big Data Extensions 1 VMware vsphere Big Data Extensions lets you deploy and centrally operate Hadoop and HBase clusters running on VMware vsphere. Big Data Extensions simplifies the deployment and provisioning process, gives you a real time view of the running services and the status of their virtual hosts, provides a central place from which to manage and monitor your Hadoop cluster, and incorporates a full range of tools to help you optimize cluster performance and utilization. VMware vsphere Big Data Extensions Administrator's and User's Guide explains how to deploy Big Data Extensions, add a Hadoop distribution to the deployment, and create, manage, and monitor Hadoop clusters. Intended Audience This guide is for system administrators who want to use Big Data Extensions in a vsphere environment to deploy and manage Hadoop clusters. To successfully work with Big Data Extensions you should be familiar with vsphere and Hadoop. Related Publications See the VMware vsphere Big Data Extensions Command-Line Interface Guide for information about deploying, creating, and managing Hadoop clusters using the command-line interface (CLI) console. VMware, Inc. 5

6 VMware vsphere Big Data Extensions Administrator's and User's Guide 6 VMware, Inc.

7 Getting Started with Big Data 2 Extensions VMware vsphere Big Data Extensions allows you to quickly and easily deploy Hadoop and HBase clusters. The tasks in this chapter instruct you on how to set up vsphere for use with Big Data Extensions, deploy the Big Data Extensions vapp, access the vcenter and command line (CLI) administrative consoles, and configure a Hadoop distribution for use with Big Data Extensions. 1 for Big Data Extensions on page 8 Before you begin the Big Data Extensions deployment tasks, make sure that your system meets all of the prerequisites. 2 Deploy the Serengeti vapp on page 9 Deploying the Serengeti vapp is the first step in getting your Hadoop cluster up and running on vsphere Big Data Extensions. 3 Install the Big Data Extensions Plug-in on page 10 To enable Big Data Extensions for use with a vcenter Server system, you need to register it with the vsphere Web Client. 4 Connect to a Serengeti Server on page 12 You must connect the Big Data Extensions plug-in to the Serengeti server you want use. Connecting to the Serengeti server lets you manage and monitor Hadoop and HBase distributions deployed within the server instance. 5 Install the Command-Line Interface Console on page 12 While the vsphere Web console supports basic resource and cluster management tasks, you can perform a greater number of the management tasks using the command-line interface (CLI) console. 6 Create a New Password for the Management Server on page 13 Starting the Management Server the first time generates a random root user password. You can change that password and other operating system passwords using the virtual machine console. 7 Create a New Username and Password for the CLI Console on page 14 The CLI console uses the vcenter login credentials. If you enabled vcenter Single Sign-On (SSO) when you deployed the vapp, the CLI console uses the SSO credentials. Otherwise, the CLI console uses only the default vcenter administrator credentials. 8 About Tarball- and Yum-Deployed Hadoop Distributions on page 14 You can choose which Hadoop distribution to use when you deploy a cluster. The type of distribution you choose determines how you configure it for use with Big Data Extensions. VMware, Inc. 7

8 VMware vsphere Big Data Extensions Administrator's and User's Guide 9 Configure a Tarball-Deployed Hadoop Distribution on page 15 When you deploy the Big Data Extensions vapp, the Apache 1.2 Hadoop distribution is included in the OVA that you download and deploy. You can add and configure other Hadoop distributions using the command line. You can configure multiple Hadoop distributions from different vendors. 10 Configure a Yum-Deployed Hadoop Distribution on page 16 You can install Hadoop distributions that use Yum repositories (as opposed to a tar-balls) for use with Big Data Extensions. When you create a cluster for a Yum-deployed Hadoop distribution, the Hadoop nodes download RPM (Red Hat Package Manager) packages from the distribution's official Yum repositories. for Big Data Extensions Before you begin the Big Data Extensions deployment tasks, make sure that your system meets all of the prerequisites. Big Data Extensions requires that you install and configure vsphere, and that your environment meets minimum resource requirements. You must also make sure that you have licenses for the VMware components of your deployment. vsphere Requirements Before you can install Big Data Extensions, you must have set up the following VMware products. Install vsphere 5.0 (or later) Enterprise or Enterprise Plus. NOTE The Big Data Extensions graphical user interface is only supported when using vsphere Web Client 5.1 and later. If you install Big Data Extensions on vsphere 5.0, you must perform all administrative tasks using the command-line interface. When installing Big Data Extensions on vsphere 5.1 or later, you must use vcenter Single Sign-On (SSO) to provide user authentication. When logging in to vsphere 5.1 or later you pass authentication to the vcenter Single Sign-On server, which you can configure with multiple identity sources such as Active Directory and OpenLDAP. On successful authentication, your username and password is exchanged for a security token which is used to access vsphere components such as Big Data Extensions. Enable the vsphere Network Time Protocol on the ESXi hosts. The Network Time Protocol (NTP) daemon ensures that time-dependent processes occur in sync across hosts. Resource Requirements for the vsphere Management Server and Templates Resource pool with at least 27.5GB RAM. Port group (or dvportgroup) with at least 6 uplink ports that has connectivity with the dvportgroups used to deploy your Hadoop clusters. 40GB or more (recommended) disk space for the management server and Hadoop template virtual disks. Resource Requirements for the Hadoop Cluster By default, when you deploy the Big Data Extensions (Serengeti) OVA, the deployment process allocates resources for the Hadoop cluster from the resource pool you create in vsphere.. Datastore free space is not less than the total size needed by the Hadoop cluster, plus swap disks for each Hadoop node that is equal to the memory size requested. 8 VMware, Inc.

9 Chapter 2 Getting Started with Big Data Extensions Network configured across all relevant ESX hosts, and has connectivity with the network in use by the management server. HA is enabled for the master node if HA protection is needed.if you use HA or FT to protect the Hadoop master node, you must use shared storage. Licensing vsphere enterprise license because VMware High Availability (HA) and VMware Distributed Resources Scheduler (DRS) must be enabled. Deploy the Serengeti vapp Deploying the Serengeti vapp is the first step in getting your Hadoop cluster up and running on vsphere Big Data Extensions. Install and configure vsphere. Verify that you have the required licenses. You manage these licenses from vsphere. vsphere license. In a production environment, enterprise license is required to protect the master node with vsphere High Availability (HA) and Fault Tolerance (FT). One license for each Hadoop worker node virtual machine. For example, if a cluster contains 1 master node, 10 worker nodes and 1 client node, 10 licenses are required. Download the Big Data Extensions OVA from the VMware download site. Verify that you have at least 40GB available for the OVA. You need additional resources for the Hadoop cluster. Ensure that you know the vcenter Single Sign-On Look-up Service URL for your vcenter Single Sign- On service. If you are installing Big Data Extensions on vsphere 5.1 or later, ensure that your environment includes vcenter Single Sign-On (SSO). You must use SSO to provide user authentication on vsphere 5.1 or later. See for Big Data Extensions, on page 8 for a complete list. 1 In the vsphere Client, select File > Deploy OVF Template 2 Click Browse and select the location to which you downloaded the OVA. 3 View the OVF Template Details page and click Next. 4 Accept the license agreement and click Next. 5 Specify a name for the vapp, select a target datacenter for the OVA, and click Next. 6 Select a vsphere resource pool for the OVA and click Next. You must select a top-level resource pool. Child resource pools are not supported. 7 Select shared storage for the OVA if possible and click Next. If shared storage is not available, local storage is acceptable. NOTE For master nodes, shared storage is the best choice. Using shared storage, you can protect the master node's virtual machine using vmotion, HA, and FT. For worker nodes use local storage. With local storage, the throughout is scalable and the cost of storage is lower. VMware, Inc. 9

10 VMware vsphere Big Data Extensions Administrator's and User's Guide 8 Leave DHCP, the default network setting, or select static IP and provide the network settings and click Next. 9 Make sure the Initialize Resources check box is checked and click Next. If the check box is checked, the resource pool, datastore, and network assigned to the vapp are added to the Big Data Extensions server for use by the Hadoop cluster you create. If the check box is unchecked, the resource pool, data store, and network connection assigned to the vapp will not be added to Big Data Extensions for use by Hadoop clusters. If you choose not to automatically add the resource pool, datastore, and network when deploying the vapp, you must use the Big Data Extensions administrative console or the CLI console to specify resource pool, datastore, and network information before creating a Hadoop cluster. 10 If you are deploying Big data Extensions on vsphere 5.1 or later, provide the vcenter Single Sign-On Look-up Service URL for your vcenter Single Sign-On service. The Serengeti Management Server, vsphere Web client, and Serengeti Command-Line Interface Console must use the same SSO service. The default vcenter Single Sign-On Look-up Service URL is: /lookupservice/sdk where: SSO_FQDN_or_IP The Fully Qualified Domain Name (FQDN) or IP address of the SSO service The default vcenter Single Sign-On HTTPS port number. If you used a different port number when you installed vcenter Single Sign-On, use that port number. The vcenter Server starts deploying the Serengeti server. This process may take several minutes. When deployment completes, 2 virtual machines are in the vapp, which have the name: The Management Server virtual machine, which is started as part of the OVA deployment. The Hadoop Template virtual machine, which is not started. Big Data Extensions clones Hadoop nodes from this template when provisioning a cluster. You do not need to start or stop this virtual machine. No Hadoop distributions is included in the template. What to do next Install the Big Data Extensions vcenter plug-in. See Install the Big Data Extensions Plug-in, on page 10. If you did not leave the Initialize Resources check box checked, you must add resources to the Big Data Extensions server prior to creating a Hadoop cluster. Install the Big Data Extensions Plug-in To enable Big Data Extensions for use with a vcenter Server system, you need to register it with the vsphere Web Client. Log in to the system on which the vsphere 5.1 Web Client is installed. The Big Data Extensions graphical user interface is only supported when using vsphere Web Client 5.1 and later. If you install Big Data Extensions on vsphere 5.0, you must perform all administrative tasks using the command-line interface. Ensure that you have login credentials with administrator privileges for the vcenter Server system with which you are registering Big Data Extensions. 10 VMware, Inc.

11 Chapter 2 Getting Started with Big Data Extensions If you want to use the server IP address to access the vsphere Web Client and your browser uses a proxy, add the server IP address to the list of proxy exceptions. 1 Open a Web browser and go to the URL of the vsphere Web Client: The client-hostname can be either the DNS hostname or IP address of the vcenter server. By default the port is 9443, but this may have been changed during installation of the vsphere Web Client. The vsphere Web Client login screen displays. 2 Type the username and password that has permissions on vcenter Server, and click Login. 3 Using the vsphere Web Client Navigator panel, locate the Serengeti vapp that you want to register with the plug-in. You can find the Serengeti vapp under the datacenter and resource pool into which you deployed it in the previous task. See Deploy the Serengeti vapp, on page 9. 4 Select management-server in the inventory tree to display information about the object in the center pane, and click the Summary tab in the center pane to access additional information. 5 Record the IP address of the management-server virtual machine. 6 Open a Web browser and go to the URL of the management-server virtual machine: The management-server-ip-address is the IP address you recorded in step 5. The Register Big Data Extensions Plug-in Web page displays. 7 Specify the information to register the plug-in. Option Register or Unregister Radio Button vcenter Server host name or IP address User Name and Password Big Data Extension Package URL Description Select the Register radio button to install the plug-in. Select Unregister to un-install the plug-in. Type the server host name or IP address of the vcenter Server system. NOTE Do not include or when you type the host name or IP address. Type the user name and password that you use to connect to the vcenter Server system. Replace the local host address ( ) with the IP address of the management-server virtual machine where the Big Data Extension plug-in is installed: 8 Click Submit. The Big Data Extension plug-in registers with the vcenter server and the vsphere Web Client. 9 Log out of the vsphere Web Client, and log back in using the user name and password you use to connect to the vcenter Server system. The Big Data Extensions icon now appears in the list of objects in the inventory. 10 From the Inventory pane, click Big Data Extensions. The Big Data Extensions home page appears. VMware, Inc. 11

12 VMware vsphere Big Data Extensions Administrator's and User's Guide What to do next You can now connect the Big Data Extensions plug-in to the Big Data Extensions server you want to manage. Connect to a Serengeti Server You must connect the Big Data Extensions plug-in to the Serengeti server you want use. Connecting to the Serengeti server lets you manage and monitor Hadoop and HBase distributions deployed within the server instance. Verify that the Serengeti vapp deployment was successful and that the Management Server virtual machine is running. See Deploy the Serengeti vapp, on page 9. Install the Big Data Extensions plug-in. See Install the Big Data Extensions Plug-in, on page Log in to the vcenter Web console. 2 Select Big Data Extensions Home. 3 Click the Summary tab. 4 Click the Connect Server link in the Connected Server dialog. The Connect to Serengeti Server dialog box is displayed. 5 Navigate to the Management Server virtual machine within the Big Data Extensions vapp that you want to connect to, select it, and click OK to confirm your selection. The Serengeti server instance appears in the list of connected servers viewable in the Summary tab of the Big Data Extensions Home. What to do next You can add additonal datastore and network resources to your Big Data Extensions deployment, and then create Hadoop and HBase clusters that you can then provison for use. Install the Command-Line Interface Console While the vsphere Web console supports basic resource and cluster management tasks, you can perform a greater number of the management tasks using the command-line interface (CLI) console. NOTE If you are using Cloudera CDH3 or CDH4, some Hadoop operations cannot be run from the Serengeti CLI console due to incompatible protocols between Cloudera Impala and CDH3 and CDH4 distributions. If you wish to run Hadoop administrative commands using the command line (such as fs, mr, pig, and hive), use a Hadoop client node to issue these commands. Verify that the Big Data Extensions vapp deployment was successful and that the Management Server is running. Verify that you have the correct user name and password to log into the CLI. If you are deploying on vsphere 5.1 or later, the Serengeti CLI uses your vcenter Single Sign-On (SSO) credentials. If you are deploying on vsphere 5.0, the Serengeti CLI uses the default vcenter administrator credentials. 12 VMware, Inc.

13 Chapter 2 Getting Started with Big Data Extensions Verify that the Java Runtime Environment (JRE) is installed in your environment, and that its location is in your PATH environment variable. 1 Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 In the Getting Started tab, click the Download Big Data Extensions CLI Console link. A ZIP file containing the CLI console downloads to your computer. 4 Unzip and examine the download, which includes the following components in the cli directory. The serengeti-cli-version JAR file, which includes the CLI console. The samples directory, which includes sample cluster configurations. See the VMware vsphere Big Data Extensions Command-line Interface Guide for more information. Libraries in the lib directory. 5 Open a command shell, and navigate to the directory where you unzipped the CLI download package. Change to the cli directory, and run the following command to start the CLI: java -jar serengeti-cli-version.jar What to do next To learn more about using the CLI console, see the VMware vsphere Big Data Extensions Command-line Interface Guide. Create a New Password for the Management Server Starting the Management Server the first time generates a random root user password. You can change that password and other operating system passwords using the virtual machine console. When you log in to the Management Server service uaing a Web browser, you use the username serengeti and the password for the Serengeti user. You can change the password for that user and perform other virtual machine management tasks from the Management Server console. When you log in to the Management Server console as the root user for the first time, you use a randomly generated password. You can change the operating password. NOTE You can change the password for any node's virtual machine using this procedure. Deploy the Serengeti vapp. See Deploy the Serengeti vapp, on page 9. Log in to a vsphere Web Client connected to the vcenter Server and verify that the Management Server virtual machine is running. 1 Right-click the Management Server virtual machine and select Open Console. The current password for the Management Server displays. 2 Use PuTTY or another SSH client to log in to the Management Server as root, using the IP address displayed in the Summary tab and the current password. VMware, Inc. 13

14 VMware vsphere Big Data Extensions Administrator's and User's Guide 3 Use the /opt/serengeti/sbin/set-password command to change the password for the root user and the serengeti user. /opt/serengeti/sbin/set-password -u root You are prompted to type a new password and asked to retype it. The next time you access the Management Server using a Web browser, use the new password. What to do next You can change the username and password of the the CLI console. Create a New Username and Password for the CLI Console The CLI console uses the vcenter login credentials. If you enabled vcenter Single Sign-On (SSO) when you deployed the vapp, the CLI console uses the SSO credentials. Otherwise, the CLI console uses only the default vcenter administrator credentials. Deploy the Serengeti vapp. See Deploy the Serengeti vapp, on page 9. Install the CLI console. See Install the Command-Line Interface Console, on page Open a Web browser and go to the URL of the vsphere Web Client: The client-hostname can be either the DNS hostname or IP address of the vcenter server. By default the port is 9443, but this may have been changed during installation of the vsphere Web Client. The vsphere Web Client login screen displays. 2 Type the username and password that has permissions on vcenter Server, and click Login. 3 Click Administration > SSO Users and Groups in the the vsphere Web Client Navigator panel. 4 Change the login credentials. The login credentials are updated. The next time you access the CLI console, you use the new login credentials. About Tarball- and Yum-Deployed Hadoop Distributions You can choose which Hadoop distribution to use when you deploy a cluster. The type of distribution you choose determines how you configure it for use with Big Data Extensions. Depending on which Hadoop distribution you want to configure for use with Big Data Extensions, you will use either a tarball or Yum repository to install your chosen distribution. The table below lists all supported Hadoop distributions, and their deployment type. Table 2 1. Hadoop Deployment Types Vendor Version Deployment Type HVE Support? Apache 1.2 Tarball Yes Greenplum HD 1.2 Tarball Yes Pivotal PHD 1.0 Yum Yes Hortonworks HDP-1.3 Tarball No Cloudera CDH3 Update 6 Tarball No 14 VMware, Inc.

15 Chapter 2 Getting Started with Big Data Extensions Table 2 1. Hadoop Deployment Types (Continued) Vendor Version Deployment Type HVE Support? Cloudera CDH4.2 MapReduce v1 Yum No Cloudera CDH 4.2 MapReduce v2 Yum No MapR Yum No About Hadoop Virtualization Extensions VMware s Hadoop Virtualization Extensions (HVE), improves Hadoop performance in virtual environments by enhancing Hadoop s topology awareness mechanism to account for the virtualization layer. Configure a Tarball-Deployed Hadoop Distribution When you deploy the Big Data Extensions vapp, the Apache 1.2 Hadoop distribution is included in the OVA that you download and deploy. You can add and configure other Hadoop distributions using the command line. You can configure multiple Hadoop distributions from different vendors. In most cases, you download Hadoop distributions from the Internet. If you are behind a firewall you may need to modify your proxy settings to allow the download. If you have a local Hadoop distribution and your server does not have access to the Internet, you can manually upload the distribution. See VMware vsphere Big Data Extensions Command-Line Interface Guide. Deploy the Big Data Extensions vapp. See Deploy the Serengeti vapp, on page 9. (Optional) Set the password for the Management Server. 1 Log in to the Management Server virtual machine using PuTTY or another SSH client. 2 Run the /opt/serengeti/config-distro.rb Ruby script, as follows. cd sbin config-distro.rb --name distro_name --hadoop hadoop_package_url --pig pig_package_url --hive hive_package_url The script downloads the files. 3 When download completes, explore the newly created /opt/serengeti/www/distros directory, which includes the following directories and files. Item name manifest manifest.example Description Directory that is named after the distribution, for example, apache. manifest file generated by config-distro.rb that is used to download the Hadoop distribution. Example manifest file. This file is available before you perform the download. The manifest file is a JSON file with three sections: name, version, and packages. The VMware vsphere Big Data Extensions Command- Line Interface Guide includes information about the manifest. 4 Return to the vsphere Web Console and click Hadoop Distributions to verify that a distribution is configured. The distribution and the corresponding role are displayed. VMware, Inc. 15

16 VMware vsphere Big Data Extensions Administrator's and User's Guide The distribution is added to the Management Server, but is not installed in the Template virtual machine. The agent that is preinstalled on each virtual machine copies the distribution components that the user specifies from the Management Server to the nodes during the Hadoop cluster creation process. What to do next You can create and deploy Hadoop or HBase clusters using your chosen Hadoop distribution. See Create a Hadoop or HBase Cluster, on page 23. You can add additonal datastore and network resources for use by the Hadoop clusters you intend to create. See Chapter 3, Managing Resources for Hadoop and HBase Clusters, on page 19. Configure a Yum-Deployed Hadoop Distribution You can install Hadoop distributions that use Yum repositories (as opposed to a tar-balls) for use with Big Data Extensions. When you create a cluster for a Yum-deployed Hadoop distribution, the Hadoop nodes download RPM (Red Hat Package Manager) packages from the distribution's official Yum repositories. High-speed internet access from the Management Server in order to download RPM (Red Hat Package Manager) packages from your chosen Hadoop distribution's official Yum repository. If you do not have adequate internet access to download the RPM packages from your Hadoop vendor's Yum repository, you can create a local Yum repository for your Hadoop distribution. For information on creating a local Yum repository, see Appendix D, "Creating a Local Yum Repository" in the VMware vsphere Big Data Extensions Command-Line Interface Guide. 1 Log in to the Management Server virtual machine using PuTTY or another SSH client. 2 Navigate to the directory /opt/serengeti/www/distros, and open the file manifest.sample. The manifest.sample file is a JSON file with three sections: name, version, and packages. The VMware vsphere Big Data Extensions Command-Line Interface Guide includes additional information about the manifest. 3 Note the configuration for the MapR and CDH4 Hadoop distributions. { "name" : "mapr", "vendor" : "MAPR", "version" : "2.1.1", "packages" : [ { "roles" : ["mapr_zookeeper", "mapr_cldb", "mapr_jobtracker", "mapr_tasktracker", "mapr_fileserver", "mapr_nfs", "mapr_webserver", "mapr_metrics", "mapr_client", "mapr_pig", "mapr_hive", "mapr_hive_server", "mapr_mysql_server"], }, { ] } "package_repos" : [" "name" : "cdh4", "vendor" : "CDH", "version" : "4.1.2", "packages" : [ { "roles" : ["hadoop_namenode", "hadoop_jobtracker", "hadoop_tasktracker", "hadoop_datanode", "hadoop_journalnode", "hadoop_client", "hive", "hive_server", "pig", "hbase_master", "hbase_regionserver", "hbase_client", "zookeeper"], 16 VMware, Inc.

17 Chapter 2 Getting Started with Big Data Extensions } ] } "package_repos" : [" The two Yum repository files (mapr-m5.repo and cloudera-cdh4.repo) point to the official MapR and CDH4 Yum repositories, respectively. 4 Copy the relevant distribution's content from the sample manifest file (/opt/serengeti/www/distros/manifest.sample) to the actual manifest file (/opt/serengeti/www/distros/manifest). Tarball-deployed and Yum repository-deployed distributions can exist simultaneously within Big Data Extensions. 5 If you created a local Yum repository for the Hadoop distribution, change the manifest file's package_repos value from the distribution's official Yum repository location to your local Yum repository location. Open the manifest file (/opt/serengeti/www/distros/manifest) for editing, and add the distribution's configuration information. Follow the example in the sample manifest file, and change the package_repos value to the location of your local Yum repository. 6 To enable Big Data Extensions to use the added distribution, open a command shell on the Management Server and (from the Serengeti server, not the Serengeti CLI), and restart the Tomcat service, at which time the Management Server reads the revised manifest file. sudo service tomcat restart 7 Return to the vsphere Web Console and click Hadoop Distributions to verify that the distribution is configured. The distribution and the corresponding role are displayed. What to do next You can now create Hadoop and HBase clusters. SeeChapter 4, Create Hadoop and HBase Clusters, on page 23. VMware, Inc. 17

18 VMware vsphere Big Data Extensions Administrator's and User's Guide 18 VMware, Inc.

19 Managing Resources for Hadoop and 3 HBase Clusters Big Data Extensions allows you to manage the datastores and networks that you use to create Hadoop and HBase clusters. This chapter includes the following topics: Add a Datastore to Big Data Extensions, on page 19 Remove a Datastore, on page 20 Add a Network to Big Data Extensions, on page 20 Remove a Network, on page 21 Add a Datastore to Big Data Extensions You can add datastores to Big Data Extensions to make them available to Hadoop and HBase clusters. Big Data Extensions supports both shared datastores and local datastores. Deploy Big Data Extensions, and log in to the vsphere Web console. 1 Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Resources. 4 Expand Inventory List, and select Datastores. 5 Click the plus (+) icon. The Add Datastore dialog box displays. 6 Type a name with which to identify the datastore in Big Data Extensions. 7 Type the name of a datastore as it is labeled in vsphere. You can choose multiple vsphere datastores. You can use the * and? pattern matching operators (wildcards) to specify multiple datastores. For example, to specify all datastores whose name begins with data-, type data-*. VMware, Inc. 19

20 VMware vsphere Big Data Extensions Administrator's and User's Guide 8 Select the datastore type in vsphere. Type Shared Local Description Recommended for master nodes. Enables you to leverage vmotion, HA, and Fault Tolerance. Recommended for worker nodes. Throughput is scalable and the cost of storage is lower. 9 Click OK to save your changes and close the dialog box. The vsphere datastore is made available for use by Hadoop and HBase clusters deployed within Big Data Extensions. Remove a Datastore You remove a datastore from Big Data Extensions when you no longer want the Hadoop clusters you create to use that datastore. Remove all Hadoop clusters associated with the datastore. See Delete a Hadoop Cluster, on page Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Resources. 4 Expand Resources > Inventory List, and select Datastores. 5 Click the remove (x) icon. 6 Click Yes to confirm that you want to remove the datastore. If you have not removed the cluster that uses the datastore, you will receive an error message indicating the datastore cannot be removed because it is currently in use. The datastore is removed from Big Data Extensions. Add a Network to Big Data Extensions You add networks to Big Data Extensions to make the IP addresses contained by those networks available to Hadoop and HBase clusters. Deploy Big Data Extensions. See Chapter 2, Getting Started with Big Data Extensions, on page 7 1 Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Resources. 4 Expand Resources > Inventory List, and select Networks. 5 Click the plus (+) icon. The Add Networks dialog box displays. 20 VMware, Inc.

21 Chapter 3 Managing Resources for Hadoop and HBase Clusters 6 Type a name with which to identify the network resource in Big Data Extensions. 7 Type the name of the port group that you want to add as it is labeled in vsphere. 8 Select DCHP, or specify IP address information. 9 Click OK to save your changes and close the dialog box. The IP addresses of the network are available to Hadoop and HBase clusters you create within Big Data Extensions. Remove a Network You can remove an existing network from Big Data Extensions when you no longer need it. Removing an unused network frees the IP addresses for use by other services. Remove clusters assigned to the network. See Delete a Hadoop Cluster, on page Log in to the vcenter Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Resources. 4 Expand Resources > Inventory List, and select Networks. 5 Click the remove (x) icon. 6 Click Yes to confirm that you want to remove the network. If you have not removed the cluster that uses the network, you will receive an error message indicating the network cannot be removed because it is currently in use. The network is removed and the IP addresses become available for use. VMware, Inc. 21

22 VMware vsphere Big Data Extensions Administrator's and User's Guide 22 VMware, Inc.

23 Create Hadoop and HBase Clusters 4 Big Data Extensions allows you to quickly and easliy create and deploy Hadoop and HBase clusters. This chapter includes the following topics: About Hadoop and HBase Clusters, on page 23 Create a Hadoop or HBase Cluster, on page 23 About Hadoop and HBase Clusters Hadoop and HBase clusters are composed of three different node types: master nodes, worker nodes, and client nodes. Understanding the different node types will help you plan your cluster, and configure the appropriate number and type of nodes when creating a cluster. The three types of node groups in a Hadoop deployment are master nodes, worker nodes, and client nodes. Master nodes oversee the following key operations that comprise Hadoop: storing data in the Hadoop Distributed File System (HDFS) and running parallel computations on that data using MapReduce. The NameNode coordinates the data storage function (with the HDFS), while the JobTracker oversees and coordinates the parallel processing of data using MapReduce. Worker nodes make up the majority of virtual machines and perform the job of storing the data and running computations. Each worker node runs both a DataNode and TaskTracker service that communicates with, and receives instructions from their master nodes. The TaskTracker service is subordinate to the JobTracker, and the DataNode service is subordinate to the NameNode. Client nodes have Hadoop installed with all the cluster settings, but are neither master or worker nodes. Instead, the client node loads data into the cluster, submits MapReduce jobs describing how that data should be processed, and then retrieves or views the results of the job when processing is finished. Create a Hadoop or HBase Cluster After you complete deployment of the Hadoop distribution, you can create a Hadoop and HBase cluster to process data. You can create multiple clusters in your Big Data Extensions environment, but your environment must meet all prerequisites. Deploy the Big Data Extensions vapp. Install the Big Data Extensions plugin. Ensure that you have adequate resources allocated to run the Hadoop cluster. Configure one or more Hadoop distributions. VMware, Inc. 23

24 VMware vsphere Big Data Extensions Administrator's and User's Guide 1 Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 Click Create a New Hadoop Cluster from the Basic Tasks list in the Getting Started tab. The Create New Hadoop Cluster dialog displays. 4 Specify the information for the cluster that you want to create. Option Hadoop cluster name Hadoop distro Deployment type DataMaster Node Group ComputeMaster Node Group HBaseMaster Node Group (HBase cluster only) Worker Node Group Client Node Group Description Type a name by which to identify the cluster. Select the Hadoop distribution. Select the type of cluster you want to create. You can choose to create: Basic Hadoop Cluster HBase Cluster Data/Compute Separation Hadoop Cluster Compute-only Hadoop Cluster The type of cluster you create determines the available node group selections. The DataMaster node is a virtual machine that runs the Hadoop NameNode and TaskTracker service. This node manages HDFS data and assigns tasks to Hadoop JobTracker services deployed in the worker node group. Select a resource template from the drop-down menu, or select Customize to customize a resource template. For the master node, use shared storage so that you protect this virtual machine with VMware HA and FT. The ComputeMaster node is a virtual machine that runs the Hadoop compute nodes. This node assigns tasks to Hadoop JobTracker services deployed in the worker node group. Select a resource template from the drop-down menu, or select Customize to customize a resource template. For the master node, use shared storage so that you protect this virtual machine with VMware HA and FT. The HBaseMaster node is a virtual machine that runs the HBase master node. This node orchestrates a cluster of one or more regionserver slave nodes. Select a resource template from the drop-down menu, or select Customize to customize a resource template. For the master node, use shared storage so that you protect this virtual machine with VMware HA and FT. Worker nodes are virtual machines that run the Hadoop DataNodes and TaskTracker service. These nodes store HDFS data and execute tasks. Select the number of nodes and the resource template from the drop-down menu, or select Customize to customize a resource template. For worker nodes, use local storage. NOTE You can add nodes to the worker node group by using Scale Out Cluster. You cannot reduce the number of nodes. A client node is a virtual machine that contains Hadoop client components. From this virtual machine you can access HDFS, submit MapReduce jobs, run Pig scripts, or run Hive queries. Select the number of nodes and a resource template from the drop-down menu, or select Customize to customize a resource template. NOTE You can add nodes to the client node group by using Scale Out Cluster. You cannot reduce the number of nodes. 24 VMware, Inc.

25 Chapter 4 Create Hadoop and HBase Clusters Option Network Resource Pools Description Select the network that you want the cluster to use. Select the resource pool that you want the cluster to use. The management server clones the template virtual machine to create the nodes in the cluster. When each virtual machine starts, the agent on that virtual machine pulls the appropriate Big Data Extensions software components to that node and deploys the software. VMware, Inc. 25

26 VMware vsphere Big Data Extensions Administrator's and User's Guide 26 VMware, Inc.

27 Managing Hadoop Clusters 5 You can use the vcenter Web console to start and stop your Hadoop cluster and to modify cluster configuraton. You can also manage a cluster using the CLI. This chapter includes the following topics: Start and Stop a Hadoop Cluster, on page 27 Scale Out a Hadoop Cluster, on page 28 Delete a Hadoop Cluster, on page 28 About Resource Usage and Elastic Scaling, on page 28 Optimize Cluster Resource Usage with Elastic Scaling, on page 29 Use Disk Shares to Prioritize Cluster Virtual Machines, on page 30 Start and Stop a Hadoop Cluster You can stop a currently running Hadoop cluster and start a stopped Hadoop cluster from the vcenter Web console. Verify that the cluster is provisioned. Verify that enough resources, especially CPU and memory resources, are available to start the virtual machines in the Hadoop cluster. 1 Log in to the vcenter Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Hadoop Clusters. 4 Select the cluster that you want to stop or start from the Hadoop Cluster Name column, and right-click to display the Actions menu. 5 Select Shut Down Hadoop Cluster to stop a running cluster, or select Start Hadoop Cluster to start a cluster. VMware, Inc. 27

28 VMware vsphere Big Data Extensions Administrator's and User's Guide Scale Out a Hadoop Cluster You specify the number of nodes to use when you create Hadoop clusters. You can later scale out the cluster by increasing the number of worker nodes and client nodes. You can scale the cluster using the vcenter Web console or the CLI. The CLI provides more configuration options than the vcenter Web console. See Install the Command-Line Interface Console, on page 12 for installation instructions. See the Serengeti User's Guide for CLI documentation. You cannot decrease the number of worker and client nodes from the vcenter Web console. Start the cluster if it is not running. 1 Log in to the vcenter Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Hadoop Clusters. 4 Select the cluster that you want to scale out from the Hadoop Cluster Name column. 5 Click the All Actions icon, and select Scale Out. 6 Select the worker or client node group to scale out from the Node group drop-down menu. 7 Specify the number of node instances to add in the Instance number text box and click OK. Because you cannot decrease the number of nodes, a Scale Out Failed error occurs if you specify an instance number that is less than or equal to the current number of instances. The cluster is updated to include the specified number of nodes. Delete a Hadoop Cluster You can delete a cluster using the vcenter Web console. When you delete a cluster, all virtual machines in the cluster are removed Stop the cluster if it is running. See Start and Stop a Hadoop Cluster, on page Log in to the vcenter Web console. 2 In the object navigator select Big Data Extensions. 3 Under Inventory Lists click Hadoop Clusters. A list of the available clusters appears. 4 Select the cluster that you want to delete from the Hadoop Cluster Name column. 5 Click the All Actions icon, and select Delete Hadoop Cluster. The cluster and all the virtual machines it contains are removed from Big Data Extensions. About Resource Usage and Elastic Scaling Elasticity enables you to optimize cluster performance and utilization of compute nodes. 28 VMware, Inc.

29 Chapter 5 Managing Hadoop Clusters Elasticity (elastic scaling) lets you optimize cluster performance and utilization of Tasktracker and Nodemanager compute nodes. Elasticity-enabled clusters start and stop virtual machines automatically and dynamically to optimize resource consumption. Elasticity is ideal in a mixed workload environment to ensure that high priority jobs are assigned sufficient resources. Elasticity functions are performed by the Virtual Hadoop Runtime Manager (VHM), which sets the adjusts the of active compute virtual machines based on configuration settings that you specify. To control the number of running virtual machines between the desired minimum and maximum, you set reservations, shares, and limits on the resource pool in which the Hadoop cluster is deployed. Additionally, you can manage the resource settings of other resource pools and virtual machines running on the same vsphere deployment. vsphere resource management takes appropriate actions to satisfy the specified resource settings, and VHM powers the compute virtual machines on or off based on how the vsphere resource management treats the Hadoop VMs. If vsphere imposes significant contention on the Hadoop virtual machines (for example, if the Hadoop virtual machines have lower resource entitlements), VHM reduces the number of powered on virtual machines. If vsphere does not place any contention and Big Data Extensions finds that Hadoop virtual machines are actively using the resources they have been given, VHM increases the number of powered on virtual machines. Optimize Cluster Resource Usage with Elastic Scaling You can specify the elasticity of the cluster. Elasticity lets you specify the number of nodes the cluster can use, and whether it automatically adds nodes, or uses nodes within a targeted range. There are two elasticity modes. Automatic Manual 1 Log in to the vsphere Web console. 2 In the object navigator select Big Data Extensions. VHM estimates resource consumption and dynamically adjusts the number of active nodes within a range from the specified minimum (per the minimum number of compute nodes you specify) to the actual maximum (currently provisioned) number of available virtual machines. VHM adjusts the number of running compute nodes to settings you specify. 3 Under Inventory Lists click Hadoop Clusters. A list of the available clusters appears. 4 Select the cluster whose elasticity mode you want to set from the Hadoop Cluster Name column. 5 Click the All Actions icon, and select Set Elasticity Mode. 6 Specify the Elasticity settings for the cluster that you want to modify. Option Elasticity mode Target compute nodes Min compute nodes Description Select the type of elasticity mode you want to use. You can choose manual or automatic. Specify the desired number of compute nodes the cluster should target for use. This is applicable only when using manual elasticity mode. Specify the minimum number (the lower limit) of active virtual machines to maintain in the cluster. This is applicable only when using automatic elasticity mode. VMware, Inc. 29

30 VMware vsphere Big Data Extensions Administrator's and User's Guide What to do next Specify the cluster's access priority for datastores. See Use Disk Shares to Prioritize Cluster Virtual Machines, on page 30. Use Disk Shares to Prioritize Cluster Virtual Machines You can set the disk IO shares for the virtual machines running a cluster. Disk shares distinguish highpriority from low-priority virtual machines. You can change the disk resources for a virtual machine. If multiple virtual machines access the same VMFS datastore and the same logical unit number (LUN), use disk shares to prioritize the disk accesses from the virtual machines. Disk shares distinguish high-priority from low-priority virtual machines. You can allocate the host disk's I/O bandwidth to the virtual hard disks of a virtual machine. Disk I/O is a host-centric resource so you cannot pool it across a cluster. Shares is a value that represents the relative metric for controlling disk bandwidth to all virtual machines. The values are compared to the sum of all shares of all virtual machines on the server and, on an ESX host, the service console. For more information on using disk shares to prioritize virtual machines, see the VMware vsphere ESXi and vcenter Server documentation. 1 Log in to the vsphere Web console. 2 In the object navigator select Big Data Extensions. 3 Under Inventory Lists click Hadoop Clusters. A list of the available clusters appears. 4 Select the cluster whose disk IO shares you want to set from the Hadoop Cluster Name column. 5 Click the Actions icon, and select Set Disk IO Share. 6 Specify a value to allocate a number of shares of disk bandwidth to the virtual machine running the cluster. Clusters configured for HIGH IO Shares receive higher priority access than those with NORMAL and LOW priorities, which provides better disk I/O performance. 7 Click OK to save your changes and close the dialog box. 30 VMware, Inc.

31 Monitoring Hadoop Clusters 6 You can monitor Hadoop clusters using the Big Data Extensions CLI console. Limited monitoring capabilities are also available using the vcenter Web console. This chapter includes the following topics: View Provisioned Hadoop Clusters, on page 31 View Cluster Information, on page 32 Monitor the Hadoop Distributed File System Status, on page 33 Monitor MapReduce Status, on page 33 Monitor HBase Status, on page 33 View Provisioned Hadoop Clusters You can view the clusters deployed within Big Data Extensions, including information about whether the cluster is running, the type of Hadoop distribution used by a cluster, and the number and type of nodes in the cluster. Create one or more Hadoop clusters whose information you can view. 1 Log in to the vsphere Web console. 2 Select Big Data Extensions. 3 From the Inventory Lists, click Big Data Clusters. 4 Select Hadoop Clusters. Information about all provisioned clusters appears in the right pane. Table 6 1. Cluster Information Option Cluster Name Status Distro Description Name of the cluster. Status of the cluster, which can be: Running, Staring, Started, Stopping, Stopped, Provisioning, or Provision Error. Hadoop distribution in use by the cluster. The supported distributions are Apache, Cloudera (CDH), Greenplum (GPHD), Pivotal HD (PHD), Hortonworks (HW), and MapR. VMware, Inc. 31