DriveScale-CLOUDERA Reference Architecture

Transcription

1 DriveScale-CLOUDERA Reference Architecture

2 Table of Contents 1. Executive Summary Audience and Scope DriveScale Cloudera Enterprise Solution Overview DriveScale Components Overview Hardware: DriveScale Adapter Chassis with DriveScale Adapters Software Reference Architecture Details Physical Cluster Component List Logical Cluster Topology Physical Cluster Topology Cluster Management Enabling Hadoop Virtualization Extensions Disk and Filesystem Layout OS Supportability/Compatibility Matrix JBOD Supportability/Compatibility Matrix Rack Scalability of 55

3 7. References Bill of Materials Conclusion Appendix A: Glossary of Terms Appendix B: DriveScale Cluster Install Configure Your Domains with DriveScale Central (DSC) Set up DMS nodes Set up DriveScale Adapter (DSA) Start the DMS and setup login to DMS Set up Servers/DataNodes/MasterNodes Tagging JBOD and drives Creating Server Nodes and Clusters from templates Appendix B: Cloudera Manager Install Cloudera Manager Installation Procedure for Reference Architecture of 55

4 DRIVESCALE-CLOUDERA 1. Executive Summary This document is a high-level design reference architecture guide for implementing Cloudera Enterprise on a DriveScale solution with industry standard servers and JBOD. The reference architecture introduces all the high-level hardware, and software that are included in the stack. Each high-level component is then described individually. This reference architecture does not describe the Cloudera data components or their applications. DriveScale Technology Overview DriveScale is leading the charge in bringing hyperscale computing capabilities to mainstream enterprises. Its compose-able data center architecture transforms rigid data centers into flexible and responsive scale-out deployments. Using DriveScale, data center administrators can deploy independent pools of commodity compute and storage resources, automatically discover available assets, and combine and recombine these resources as needed. The solution is provided through a set of on-premises and SaaS tools that coordinate between multiple levels of infrastructure. With DriveScale, Hadoop architects can more easily support Hadoop deployments of any size as well as other modern application workloads. DriveScale provides hardware and software technology that allows separate deployment of compute and storage using commodity servers with minimal drives for Operating System and JBODs (Just a Bunch of Disks), with flexible binding of storage-to-compute resources in any ratio required by an application. As needs change, these bindings can be dissolved and reconfigured on demand, all under software control. DriveScale technology acquires a deep understanding of the physical infrastructure and dynamics of a data center, which it uses to provide an integrated set of intelligence and automation tools to scaleout data center infrastructure to greatly simplify and optimize the data center s operations. 2. Audience and Scope This reference architecture guide is for Hadoop and IT architects who are responsible for the design and deployment of Cloudera Enterprise solutions on premises, as well as for Apache Hadoop administrators and architects and data center architects/engineers who collaborate with specialists in that space. 3. DriveScale Cloudera Enterprise Solution Overview Apache Hadoop is designed to address the ever so changing hardware requirements from customers for a more flexible and dynamic hardware infrastructure that provides significant cost and operational benefits. It is designed with composability as the primary goal, saving money, improving utilization and greatly simplifying the deployment of Hadoop clusters. Hadoop is an Apache project being developed in the Java programming language by a global community of contributors. Yahoo!, has been the largest contributor to this project, and uses Apache Hadoop extensively across its businesses. Core committers on the Hadoop project include employees from Cloudera, ebay, Facebook, Getopt, Hortonworks, Huawei, IBM, InMobi, INRIA, LinkedIn, MapR, 4 of 55

5 Microsoft, Pivotal, Twitter, UC Berkeley, VMware, WANdisco, and Yahoo!, with contributions from many more individuals and organizations. Although Hadoop is popular and widely used, installing, configuring, and running a production Hadoop cluster involves many concerns, including: Choosing the appropriate Hadoop software distribution and extensions Installing monitoring and management software Allocation of Hadoop services to physical nodes Selection of appropriate server hardware Rightsizing the storage configuration Implementing data locality Design of the network fabric Sizing and system scalability Overall performance These concerns are complicated by the need to understand the workloads that will be running on the cluster, the fast-moving pace of the core Hadoop project, and the challenges to managing a system designed to scale to thousands of nodes in a single cluster. The DriveScale Cloudera Solution was designed by DriveScale in collaboration with Cloudera, and embodies all the hardware, software, resources and services needed to run Hadoop in a production environment. This end-to-end solution approach means that you can be in production with Hadoop in a shorter time than is typically possible with homegrown solutions. The solution is based on Cloudera Enterprise Data Hub 5.x (including Cloudera Distributed Hadoop), DriveScale hardware and software, industry standard servers, network switches and JBODs. This solution includes components that span the entire solution stack: Reference architecture and best practices Optimized storage configurations Optimized network infrastructure Cloudera Enterprise Data Hub including Cloudera Distributed Hadoop This solution is designed to address the clear majority of Apache Hadoop use cases including, but not limited to: Big data analytics ETL Offload Data Warehouse Optimization Batch processing of unstructured data Big data visualization Search and predictive analysis 5 of 55

6 4. DriveScale Components Overview DriveScale system is composed of one hardware component and four software components which are described below: 4.1 Hardware: DriveScale Adapter Chassis with DriveScale Adapters This is a 1U appliance with adapters that connect to servers via 10Gb Ethernet interfaces and to JBOD s via SAS interfaces. 4.2 Software There are four principal components of the DriveScale software: a) DriveScale Management Server (DMS) The server running the DMS software bundle is called the DMS node. A typical deployment consists of three DMS s in a clustered configuration for high availability (HA). The software manages and configure resources and contains the inventory/configuration information repository and database: 3 Inventory: DMS s, DS Adapters, switches, JBOD chassis, disks, server nodes 3 Configuration: node templates, cluster templates, configured clusters 3 DMS Database: used as a message bus to communicate with the end points. b) DriveScale Server Agent DriveScale Server Agent discovery action provides inventory for hardware and servers, and creates mappings between server nodes and the disks they consume. c) DriveScale Central (DSC) Cloud-based software management portal that acts as the: o o software distribution repositories for subscribers DriveScale keys repository 6 of 55

7 o o o centralized log file repository user documentation repository license manager d) DriveScale Adapter Firmware DriveScale Adapter firmware enables the JBODs to be mapped to the servers and over the network to be used as local drives. 5. Reference Architecture Details 5.1 Physical Cluster Component List The following table lists the physical components for the cluster. Component Configuration Description Quantity DriveScale Adapter Chassis DHCP, Jumbo frame enabled 1U appliance with adapters that connect to servers via Ethernet, and to JBOD s via SAS. 2 DriveScale Adapter DHCP, Jumbo frame enabled Provides the data network. 4 for each chassis DriveScale Management Server (DMS) DMS running as a VM Manages and configures the nodes and cluster and also stores the inventory/ configuration repository of every hardware in the cluster. Min 1, for HA 3 DMS s should be configured as master and slave Servers 2 socket CPU and memory according to the individual Hadoop cluster requirements Commodity x86 servers that house all the Node Manager, compute instances and DriveScale agents. Min 3 Master nodes + 5 Data nodes HDD for Servers 2 drives configured in RAID 1 The internal drives are used for OS install. 2 for each server NICs Dual port 10 Gbps Ethernet SFP+ NICs. Provides the data network 1 for each server JBOD Chassis Default configuration Houses the drive with dual IO controllers. Min 2, Recommended 3 for production environment by Cloudera 7 of 55

8 Component Configuration Description Quantity HDD for JBOD Default configuration Drives to house the data for the cluster. Depending on the cluster requirements ToR 10G switch LLDP, MLAG, 9K Jumbo Frame configured Provides data network connectivity. 2 for each rack ToR 1G switch Default configuration Provides management network connectivity. 1 for each rack 5.2 Logical Cluster Topology The minimum requirements to build out the cluster are: 3 Master Nodes 5 Data Nodes 1 DriveScale Adapter Chassis 1 DriveScale Management Server 2 10G Switches 1 1G Switch 2 JBOD s chassis with drives This reference architecture is built on 3 master nodes and 5 data nodes with 2 JBOD chassis and 126 drives of 1or 2 or 3 TB HDD. The following table lists the configurations of the servers and number of drives used. For clusters that require the maximum read bandwidth out of each attached drive concurrently, it is recommended that the nodes in such a cluster be configured with a maximum of 8 drives each, assuming 2 x 10Gbps Ethernet bandwidth per node. However, this is an extreme case. A general rule of thumb for calculating the number of drives to allocate to each node in a cluster is dependent on the application but it is safe to allocate up to 16 drives per node, again assuming 2 x 10Gbps Ethernet bandwidth per node. With the availability of quad-port 10Gbps Ethernet adapters, one can add significantly higher I/O per node and therefore greater numbers of drives as well. Component Configuration Description Quantity Master nodes 2 sockets 8 core CPU, 64GB RAM, 10GbE Intel NIC with 2 internal HDD for OS and 4 high capacity HDD mounted from the JBOD. Master nodes hosts the Cloudera master services and DriveScale agents. 3 8 of 55

9 Component Configuration Description Quantity Worker nodes 2 sockets 8 core CPU, 64GB RAM, 10GbE Intel NIC with 2 internal HDD for OS and 16 high capacity HDD mounted from the JBOD. For Impala nodes, the minimum RAM should be 128GB. Data nodes house the HDFS Data Nodes and YARN Node managers, any additional required services and DriveScale agents. 5 Notes: - Customers with higher (or lower) compute needs can acquire bigger (or smaller) data nodes configured with CPU and memory that fits the specific requirements of their applications. - Similarly, depending on the data requirements, customers can add or remove disk drives to match the specific needs of their applications. The following table identifies service roles for different node types. Master Node Master Node Master Node Worker Node ZooKeeper ZooKeeper ZooKeeper ZooKeeper YARN Resource Manager Resource Manager History Server Node Manager Hive MetaStore, WebHCat, HiveServer2 Management (misc) Navigator HUE Cloudera Agent Cloudera Agent Cloudera Agent, Oozie, Cloudera Manager, Management Services Navigator, Key Management Services HUE Cloudera Agent HBASE HMaster HMaster HMaster Region Server 9 of 55

10 Master Node Master Node Master Node Worker Node Impala Search Kafka StateStore, Catalog Impala Daemon Solr Broker Spark History Server Runs on YARN HDFS NameNode, QJN NameNode, QJN QJN DataNode 5.3 Physical Cluster Topology Diagram 1: DriveScale lab Architecture with 2xDSA Chassis (8x Adapters in use), 2x JBOD, 3 Master Nodes and 5 Data Nodes 10 of 55

11 Diagram 2: DriveScale lab Architecture with 3xDSA Chassis (12x Adapters in use), 2x JBOD, 3 Master Nodes and 5 Data Nodes Notes: - The 1GbE management connections were made to 10GbE switches, DSA chassis s and JBOD s. The connections are omitted in the diagram to ease readability. - 1GbE connection is used only for server management purpose with BMC IDRAC (Dell) or ilo (HPE). It is not a part of the Hadoop network. Please note that multi-homed clusters are not supported by Cloudera. - SAS connections from DSA chassis 2 to JBOD 2 was also replicated as DSA chassis 1 to JBOD 1. The connections are omitted in the diagram to make it look less congested. - The drives to the master nodes and data nodes were distributed across the two JBOD s chassis. 11 of 55

12 5.4 Cluster Management This section details the steps for setting up a DriveScale enabled Hadoop cluster using Cloudera manager. Setting up DriveScale cluster Before installing Cloudera Manager or using an existing install of Cloudera Manager, you must complete the following tasks for setting up the DriveScale solution: 1. Rack and install the DriveScale Adapter chassis and controllers (DSAs) using the documentation provided by DriveScale. 2. Rack and install the JBOD s using the documentation provided by the vendor. 3. Rack and install the server using the documentation provided by the vendor. 4. Create RAID 1 for the internal HDD on the server and install the OS on all the servers. 5. Install and configure DriveScale Management Server (DMS) either as a VM or on a standalone server. 6. Set up DSA configuration from DMS. 7. Install and configure DriveScale agents on the master and data nodes. 8. Create master/data node and cluster template with required drives using DMS. 9. Create the cluster from the template using DMS. 10. Ensure that DriveScale cluster is up and running before proceeding ahead. 12 of 55

13 Setting up Cloudera cluster 1. After the successful completion of the steps mentioned above, install Cloudera Manager using the Cloudera CDH Installation guide. 2. Ensure that Cloudera HDFS cluster is set up in a high availability mode. 3. The following services were set up for this reference architecture. HDFS HBase Hive Hue Impala Kafka Oozie Solr Spark YARN ZooKeeper 13 of 55

14 4. Ensure that the master and data nodes are up and running with the right assigned roles and storage. 5.5 Enabling Hadoop Virtualization Extensions With DriveScale solution, we enable configuration of a highly available Hadoop cluster including rack awareness. Hadoop Virtualization Extensions (HVE), enables customers to get additional capabilities for failure mitigation and rack awareness thereby enabling the cluster to survive the worst-casescenario of total power or hardware failure of any component including JBOD failures for an extended period. HVE can be enabled in Cloudera Manager. To enable HVE, follow the documentation on HVE from Cloudera. Also, below are the steps we followed for this reference architecture. For this reference architecture, below are the name details of the master and data nodes: Node Types Master Nodes Data Nodes Server Names u32.data1.r3.hq.drivescale.com u33.data1.r3.hq.drivescale.com u34.data1.r3.hq.drivescale.com u27.data1.r3.hq.drivescale.com u28.data1.r3.hq.drivescale.com u29.data1.r3.hq.drivescale.com u30.data1.r3.hq.drivescale.com u31.data1.r3.hq.drivescale.com 14 of 55

15 1. Go to the Cloudera Manager. a) Configure the following safety valves based on your environment: o HDFS hdfs coresite.xml: <property> <name>net.topology.impl</name> <value>org.apache.hadoop.net.networktopologywithnodeg roup</value> </property> <property> <name>net.topology.nodegroup.aware</name> <value>true</value> </property> <property> <name>dfs.block.replicator.classname</name> <value>org.apache.hadoop.hdfs.server.blockmanagement. BlockPlacementPolicyWithNodeGroup</value> </property> o YARN YARN Service MapReduce Advanced Configuration Snippet (Safety Valve), add the following properties and values: <property> <name>mapred.jobtracker.nodegroup.aware</name> <value>true</value> </property> <property> <name>mapred.task.cache.levels </name> <value>3</value> </property> 15 of 55

16 b) Based on the number of JBOD s and data nodes, create a minimum of 3 zones with at least 1 or 2 data nodes in each zone. Notes: - If the replication factor required in your environment is 3, then a minimum of 3 zones are required while setting up HVE. - This is because only one copy of the data is saved in each HVE zone. 16 of 55

17 c) For this reference architecture, we created 4 zones with a minimum of 1 or 2 data nodes in each zone. Refer to the notes section below for detailed reasoning. Notes: - For this reference architecture, each of the JBODs have two drawers of drives. There are two expanders in each of the four drawers in the 2 JBODs. All the drives in each drawer were tagged using the DMS UI. - Each of the nodes belonging to the same zone were created from drives of the same drawer of the JBOD. - The table below lists all the nodes with the JBOD and drawer ID along with the zone ID. Node Types Server Names JBOD/Drawer ID Zone ID Master Nodes u32.data1.r3.hq.drivescale.com J1D1 1 u33.data1.r3.hq.drivescale.com J1D2 2 u34.data1.r3.hq.drivescale.com J2D1 3 Data Nodes u27.data1.r3.hq.drivescale.com J1D1 1 u28.data1.r3.hq.drivescale.com J1D2 2 u29.data1.r3.hq.drivescale.com J2D1 3 u30.data1.r3.hq.drivescale.com J2D2 4 u31.data1.r3.hq.drivescale.com J2D2 4 d) Select Hosts -> All hosts. e) Select hosts u27 and u32. f) Click on Action: Assign Rack 17 of 55

18 g) Assign rack name /default/zone1 h) Select hosts u28 and u33 i) Click on Action: Assign Rack j) Assign rack name /default/zone2 k) Select hosts u29 and u34 l) Click on Action: Assign Rack m) Assign rack name /default/zone3 n) Select hosts u30 and u31 18 of 55

19 o) Click on Action: Assign Rack p) Assign rack name /default/zone4 2. Go back to Cloudera Manager and Hosts to see the new HVE configuration changes. 5.6 Disk and Filesystem Layout Node/Role Disk and Filesystem Layout Description Management/Master Ext4 1/2/3TB drives are mounted from the JBOD s YARN Node Manager nodes Ext4 1/2/3TB drives are mounted from the JBOD s 5.7 OS Supportability/Compatibility Matrix DMS Server Nodes CentOS/RHEL 6.x X X CentOS/RHEL 7.x X X Ubuntu X X 19 of 55

20 5.8 JBOD Supportability/Compatibility Matrix With DriveScale solution, we recommend customers should use high capacity JBODs with dual hot-pluggable IO controllers (Expanders) and enough upstream bandwidth. The JBODs should also have dual hot-pluggable redundant power supplies. DriveScale has evaluated and tested a few of the vendor offerings for redundancy, management functionality and performance. The table listed below has the JBOD vendor name and the model numbers which are certified by DriveScale. JBOD Vendor Dell Hewlett Packard Enterprise Model Number PowerVault MD3060e and 2.5, 60 bays, 4U, redundant expanders, 2 x 3 x mini-sas 6G D , 70 bays, 5U, quad expanders, 4 x 2 x mini-sas 12G D , 70 bays, 5U, quad expanders, 4 x 2 x mini-sas 6G RAID Inc./Newisys NDS-4600/ , 60 bays, 4U, redundant expanders, 2 x 4 x mini- SAS 6G NDS , 24 bays, 2U, redundant expanders, 2 x 3 x mini-sas 6G NDS , 90/96 bays, 4U, redundant expanders, 2 x 6 x mini- SAS-HD 12G NDS , 84 bays, 4U, redundant expanders, 2 x 5 x mini-sas-hd 12G Qunta (QCT) M6400H - 3.5, 60 bays, 4U, redundant expanders, 2 x 4 x mini-sas 6G JB , 60 bays, 4U, redundant expanders, 2 x 4 x mini-sas 12G Promise Inc. J5300s - 3.5, 12 bays,2u, redundant expanders, 2 x 2 x mini-sas-hd 12G J5320s - 2.5,24 bays, 2U, redundant expanders, 2 x 2 x mini-sas-hd 12G J , 16 bays, 3U, redundant expanders, 2 x 2 x mini-sas-hd 12G J , 24 bays, 4U, redundant expanders, 2 x 2 x mini-sas-hd 12G 20 of 55

21 6. Rack Scalability Customers can scale beyond one rack in a straightforward manner, in order to expand their compute and storage resources, as application needs grow. Customers can change or maintain the computeto-storage ratio for the new racks or an existing rack. For every new JBOD addition, a new DriveScale Adapter with four controllers must be added as well. Since drives are assigned from within the rack to servers in the rack, scaling is achieved by simply adding more racks with Servers, DriveScale Adapters, Switches and JBODs. Diagram 3: DriveScale Rack Scalability 7. References 1. Cloudera Manager Installation Guide 2. Cloudera High Availability documentation 3. High Availability for Other CDH components ha.html#xd_583c10bfdbd326ba--6eed2fb d04bee--7d18 4. Cloudera Multihoming support documentation html#cdh_cm_network_security 21 of 55

22 8. Bill of Materials Server Components Intel Xeon Processsor based servers with dual or quad port 10GbE SFP+ NICs. The exact CPU models, number of sockets, and memory are based on customer application needs Quantity Depends on customer application needs JBOD Components DriveScale certified JBODs NL-SAS HDDs Quantity Depends on customer application needs Depends on customer application needs Switch DriveScale certified 10GbE SFP+ switches 1GBaseT switch Quantity An even number of switches for redundant switch fabric Based on the number of Servers and JBODs in configuration DriveScale components DriveScale Adapter Chassis DriveScale Adapter Quantity One for each JBOD Four for each DSA Chassis Software CentOS Version Please refer to 6.7 section DriveScale Adapter 1.3 CDH of 55

23 9. Conclusion The DriveScale-Cloudera reference architecture guide is designed to provide an overview of the combined solution, the key components that are employed and details on how to install and setup clusters using these technologies. 10. Appendix A: Glossary of Terms Term Data Node DSA DSC DMS HBA HDD HDFS High Availability JBOD Description Worker nodes of the cluster to which the HDFS data is written. DriveScale Adapter. DriveScale Adapter is a 1RU Ethernet to SAS adapter serving as a bridge between 10 Gbps Ethernet connecting compute resources to JBODs full of commodity disks. DriveScale Central. DriveScale Central is a web-based user interface to the DriveScale cloud that performs DriveScale account management. DSC is where you download the keys to enable installation of the DriveScale software, and then set up your DriveScale Management Domain(s) (DMDs). This is where you create your domain, select and configure the DMS nodes for the domain, and select a chassis (with its associated DriveScale Adapters, DSAs) for the domain. DriveScale Management Server. DriveScale Management Server is the server that runs the bundle of software (service) that manages a set of Physical Resources to enable the DriveScale services. DriveScale Manager is the web-based user interface to the DMS. Host bus adapter. An I/O controller that is used to interface a host with storage devices. Hard disk drive. Hadoop Distributed File System. Configuration that addresses availability issues in a cluster. In a standard configuration, the Name Node is a single point of failure (SPOF). Each cluster has a single Name Node, and if that machine or process became unavailable, the cluster is unavailable until the Name Node is either restarted or brought up on a new host. The secondary Name Node does not provide failover capability. High availability enables running two Name Nodes in the same cluster: the active Name Node and the standby Name Node. The standby Name Node allows a fast failover to a new Name Node in case of machine crash or planned maintenance. Just a bunch of disks. A JBOD chassis hosts many HDDs and two redundant SAS switches (also called controllers). The SAS switches provide dual path access to each of the HDDs in the chassis through multiple Mini- SAS HD interface connectors 23 of 55

24 Term Job History Server MLAG Name Node NIC Node Manager NUMA PDU QJM QJN Description Process that archives job metrics and metadata. One per cluster. Multi-chassis Link Aggregation. MLAG is the ability of two or more switches to act like a single switch when forming link bundles. The metadata master of HDFS essential for the integrity and proper functioning of the distributed filesystem. Network interface card. The process that starts application processes and manages resources on the Data Nodes. Non-uniform memory access. Addresses variable memory access latency in multi-socket servers. This is typical of SMP (symmetric multiprocessing) systems, and there are several strategies to optimize applications and operating systems. vsphere ESXi can be optimized for NUMA. It can also present the NUMA architecture to the virtualized guest OS, which can then leverage it to optimize memory access. This is called vnuma. Power distribution unit. Quorum Journal Manager. Provides a fencing mechanism for high availability in a Hadoop cluster. This service is used to distribute HDFS edit logs to multiple hosts (at least three are required) from the active Name Node. The standby Name Node reads the edits from the Journal Nodes and constantly applies them to its own namespace. In case of a failover, the standby Name Node applies all the edits from the Journal Nodes before promoting itself to the active state. Quorum Journal Nodes. Nodes on which the journal services are installed. RM ToR ZK Resource Manager. The resource management component of YARN. This initiates application startup and controls scheduling on the Data Nodes of the cluster (one instance per cluster). Top of rack. ZooKeeper. A centralized service for maintaining configuration information, naming, and providing distributed synchronization and group services. 24 of 55

25 11. Appendix B: DriveScale Cluster Install Notes: - You must complete the racking and cabling of the Servers, DSA, JBOD s and switches per the details in the installation guide. - You must obtain the credentials of DSC which is shipped with hardware. - You must decide whether to set up your DMS as one standalone server or as a high-availability cluster with three servers. When three are used, the Management Domain can survive the failure of any one of the DMS machines. - The DMS servers should be configured with at least GB of memory. - DHCP server in the 10G/1G network - Access to the DHCP administrator/server to get the IP address(es) of the DSA(s) based on the MAC address(es) of the DSA(s). - The network address(es) of your DMS server(s). - The network address(es) of your DSA(s). - The network addresses of your compute servers Configure Your Domains with DriveScale Central (DSC) 1. Log in to DSC with the credentials obtained from DS. a) Go to b) Log in using the credentials provided to you by DriveScale. c) A checklist of the tasks that need to be accomplished appears on the main DSC page. 25 of 55

26 2. Go to the Domains link in the left navigation panel and click on Create Domain. 3. Fill the name, FQDN name and any notes for the domain. Click on Create. 4. Go to the Downloads link in the left navigation panel and download the config-training,ds-dmskeys-xxx.rpm and ds-repo-xxx.rpm files on your local machine. 26 of 55

27 11.2 Set up DMS nodes 1. On each of the DMS machines, copy and then install the repo and keys RPM package after downloading the file using WinScp tool or using scp command if you are using a linux machine. scp ds-* root@x.x.x.x:/tmp rpm -ivh ds-repo-* rpm -ivh ds-dms* 2. On each of the DMS machines, copy the config.training file into a file named /etc/drivescale/ conf after downloading the file using WinScp tool or using scp command if you are using a linux machine. scp config.training root@x.x.x.x:/tmp cp /tmp/config.training /etc/drivescale/conf 3. Install the dms server on all the DMS machines using the yum install command. Yum will automatically install the dms server from the repo. yum -y install dms-ds 11.3 Set up DriveScale Adapter (DSA) 1. Log into one of the DMS machines using ssh. 2. Set up the DriveScale management Domain configuration for each DSA using the same config file (config.training in this example) as was used in the DMSes. 3. This is done via the /opt/drivescale/bin/dsa command installed on each DMS. The default DSA username and password is admin/admin. 4. Run the command listed below to check the current configuration and settings of DSA. /opt/drivescale/bin/dsa --username admin --password admin --adapter <management IP of DSA adapter> service showconf 5. Verify the 10Gbps interface IPs address run the following command: /opt/drivescale/bin/dsa --username admin --password admin --adapter <management IP of the DSA> net show --interface 10gBond 6. To verify the management IP/gateway etc of the DSA in case we have the FQDN only. /opt/drivescale/bin/dsa --username admin --password admin --adapter <DSA Adapter Management FQDN > net show --interface mgmt. 7. Push the config from the DMS to the DSA /opt/drivescale/bin/dsa --username admin --password admin --adapter <management IP of the DSA> service config --file /tmp/conf.training 8. Restart DSA /opt/drivescale/bin/dsa --username admin --password admin --adapter <management IP of the DSA> service restart 27 of 55

28 11.4 Start the DMS and setup login to DMS 1. Log into one of the DMS machines using ssh. 2. Start the drivescale service by running the following command: service drivescale start 3. Set up the DMS in SET_UP_MODE by entering the following command: /opt/drivescale/bin/setup-mode 4. Login to the DMS UI using the FQDN name or IP address of the DMS. 5. Create a username and password for the first-time users. 6. To use internal authentication, you just need to provide the username for the initial admin (superuser), and enter and confirm the password. The first name and last name is optional. 7. Click Configure DMS to create the Admin user on the DMS, and to configure the authentication method 11.5 Set up Servers/DataNodes/MasterNodes Notes: The two 10GbE ports are bonded into a single port for the DSA. Users need to bond the two 10GbE ports on the server nodes as well. Create the ifcfg file for creating the bond on the server. auto bond0 iface bond0 inet static slaves eth2 eth3 bond_miimon 100 bond_mode 802.3ad bond_xmit_hash_policy layer2 pre-up ifconfig eth2 mtu 9000 && ifconfig eth3 mtu 9000 mtu 9000 auto vmbr1 iface vmbr1 inet static address <IP for the 20G Bond interface> netmask bridge_ports bond0 bridge_stp off bridge_fd 0 pre-up ifconfig eth2 mtu 9000 pre-up ifconfig eth3 mtu 9000 pre-up ifconfig bond0 mtu 9000 mtu of 55

29 1. On each of the server machines, copy and then install the repo RPM package after downloading the file using WinScp tool or using scp command if you are using a linux machine. scp ds-* root@x.x.x.x:/tmp rpm -ivh ds-repo-* 2. Install the server software on all the server machines using the yum install command. Yum will automatically install the dms server from the repo. yum -y install ds-server 3. Start the drivescale service. service drivescale start 4. Remove the temporary ds-repo*, ds-dms* and /tmp/config.mydomain files from each of the server and DMS machines 11.6 Tagging JBOD and drives 1. Use the API documentation to build the JBOD tagging script. 2. Create a script to tag drives from each JBOD with a different tag. 3. Below is an example of the script. You can run the script from the DMS server. #!/bin/bash list= 5000c500560df c ef7 5000c ff 5000c50040b201db 5000c dc2f 5000c50040b c50034ee c500035d7eb3 5000c500033aa c500033aaf9b 5000c500350a41bb 5000c50055a592db 5000c500033ae8f3 5000c50040e9fcbf 5000c50040ab7d c a of 55

30 5000c50040b230fb 5000c50040a6cf3b 5000c50040aa27f3 5000c50040b2aa77 for api in $list do curl -u -k -X PATCH --header Content-Type: application/vnd.drivescale. v2+json --header Accept: application/json -d { tags : [ 6000_1 ] } $api; sleep 1; done 11.7 Creating Server Nodes and Clusters from templates Creating Node and Cluster Template 1. Connect to the DMS UI Composer and from the left-hand panel navigate to the Composer section and click on the Node template. 2. From the top right corner select the Create template and fill in the details for the template name and click on Save. 3. Create 3 templates for MasterNode and 5 for DataNode Notes: Create a DataNode and MasterNode Template for each of the data node with the following minimum requirements: Data Nodes d1 d2 d3 Minimum requirements Drives: 16 Use drives with all these tags: 6000_1 Exclude drives with any of these tags: 6000_2 6020_1 6020_2 Drives: 16 Use drives with all these tags: 6000_2 Exclude drives with any of these tags: 6000_1 6020_1 6020_2 Drives: 16 Use drives with all these tags: 6020_1 Exclude drives with any of these tags: 6000_1 6000_1 6020_2 30 of 55

31 Data Nodes d4/d5 m1 m2 m3 Minimum requirements Drives: 16 Use drives with all these tags: 6020_2 Exclude drives with any of these tags: 6000_2 6000_1 6020_1 Drives: 16 Use drives with all these tags: 6000_1 Exclude drives with any of these tags: 6000_2 6020_1 6020_2 Drives: 16 Use drives with all these tags: 6000_2 Drives: 16 Use drives with all these tags: 6020_1 Customers can change the disk, RPM, CPU or RAM according to their cluster requirements. 31 of 55

32 32 of 55

33 4. From the left-hand panel navigate to the Composer section and click on the Cluster template. 5. From the top right corner select the Create template and fill in the details for the template name as CDH_CERT_TEMPLATE. Click on Add new node type(s) and select all the data and master node templates: 6. For the newly created cluster template, click on Edit and choose Min/Max instances as 1 for all the previously created data node and master node templates: 33 of 55

34 7. Select Save after editing all the Min/Max instances for all the node template Creating Cluster Template 1. From the left-hand panel navigate to the Composer section and click on the Cluster template. 2. From the top right corner select the Create template and fill in the details for the template name as CDH_CERT_TEMPLATE with Data Nodes (mounted with 16 Disks) and 3 Master Nodes (mounted with 4 Disks) based on Cluster Template CD. 3. Select Create. 34 of 55

35 DriveScale Cluster Verification 1. From the left-hand panel navigate to the Explorer section and click on the Logical section. Verify the cluster status. 2. Click on the Details tab on top right corner. 3. Select the Cluster_CD cluster and check for the details. 35 of 55

36 12. Appendix B: Cloudera Manager Install Notes: - Cloudera Manager install was performed on Master Node 3 (u34.data1.r3.hq.drivescale.com). - Make sure NTP is installed, configured and time is sync with all nodes. - Make sure IPTABLES is stopped and SELINUX is disabled. - Make sure the OS has access to internet to download Cloudera packages. - Make sure all nodes have their correct FQDN HOSTNAME in /etc/sysconfig/network. - Make sure on Cloudera manager host that /etc/hosts has an entry as follows: u34.data1.r3.hq.drivescale.com 12.1 Cloudera Manager Installation Procedure for Reference Architecture 1. ssh to the Master node 3 and start the cloudera manager installation. # ssh to u34: [root@u34 ~] yum install wget [root@u34 ~]# wget [root@u34~]# chmod u+x cloudera-manager-installer.bin [root@u34~]# sudo./cloudera-manager-installer.bin ê Accept install and Licensing. 2. Open Web Browser: Login with default login/pass: admin/admin 36 of 55

37 3. Accept the license agreement. 4. Select Cloudera Enterprise (60 days trial) or upload the Cloudera Enterprise license. 37 of 55

38 5. Click on Continue. 6. Discover all nodes via FQDN or IP addresses by adding the names or IP and clicking on Search. 38 of 55

39 7. Select all the hosts for the cluster installation and click on Continue. 8. Select the preferred method of repository installation, version of CDH, any additional parcels etc and click on Continue. 39 of 55

40 9. Select the Install Oracle Java Developer Kit option and click on Continue. 10. For this cluster, Single User Mode was not enabled. Click on Continue. 40 of 55

41 11. Enter the correct root credentials to connect the Nodes and click on Continue. 12. Wait for the cluster installation to complete. 41 of 55

42 Notes: In case of packages install failure, remove following packages on the nodes: rpm -e --nodeps --justdb glibc-common el6.x86_64 --allmatches ; rpm -e --nodeps --justdb glibc el6.x86_64 --allmatches ; rpm -e --nodeps --justdb gdbm el6.x86_64 allmatches In case Java version running on Nodes is older: move JAVA from version 1.6 to 1.7: [root@dn~]# java -version [root@dn~]# cd /usr/java ; rm -f latest ; ln -s /usr/java/jdk1.7.0_67-cloudera latest Transparent Huge Page Compaction can cause significant performance problems on all nodes. To disable this, run the following (add the same command to an init script such as /etc/rc.local so it will be set on system reboot): [root@dn~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag ; echo never > /sys/kernel/mm/transparent_hugepage/enabled 13. Wait for the Parcels installation to complete. 42 of 55

43 14. Run the host inspection agent to verify the correctness for packages. Click on Continue after the agent is run successfully. 43 of 55

44 15. Select the services you would like to install on the nodes. For this setup, we have selected custom services and the details are 44 of 55

45 16. Select the hosts which would have the various HBase, HDFS, Hue, Hive and other services running. 45 of 55

46 17. For this reference architecture, we are using the Embedded Database. Copy the username and passwords for all the different databases. Click on Test Connection and ensure that it is successful. Click on Continue. Please be advised that Cloudera recommends using external database for production environment. 46 of 55

47 18. Review all the changes for the setup and ensure all the details are correct. Click on Continue. 47 of 55

48 48 of 55

49 49 of 55

50 19. After the installation completes, go to the main page and enable the services. The steps are mentioned below. All the actions are performed in the Cloudera Manager (CM)>Cluster1 a) Enable ZooKeeper HA (select the 3 Masters Node) and start roles. b) Enable HDFS HA and select 2 Master nodes for NameNode and JournalNode: 50 of 55

51 Notes: Open HDFS permission for HBASE on all MASTER nodes: ~]# sudo -u hdfs /opt/cloudera/parcels/cdh cdh p0.41/bin/hadoop fs -chmod 777 / Initialize Postgres DB [root@u34 ~]# service postgresql initdb [root@u34 ~]# /etc/init.d/postgresql start 3 PGSQL Policy: [root@u34 ~]# cat /var/lib/pgsql/data/postgresql.conf grep -e listen -e [root@u34 ~]# vi /var/lib/pgsql/data/postgresql.conf listen_addresses = * standard_conforming_strings = off 51 of 55

52 ~]# vi /var/lib/pgsql/data/pg_hba.conf host all all md5 3 Install JDBC [root@u34 ~]# yum install postgresql-jdbc [root@u34 ~]# mkdir /usr/lib/hive/lib/ [root@u34 ~]# ln -s /usr/share/java/postgresql-jdbc.jar /usr/lib/hive/lib/ postgresql-jdbc.jar [root@u34 ~]# /etc/init.d/postgresql restart c) HIVE - Create PGSQL user for HIVE (user: hive, pass: hive) Connect to PostGresql DB: [root@u34 ~]# sudo -u postgres psql postgres=# CREATE USER hive WITH PASSWORD hive ; postgres=# CREATE DATABASE hive; postgres=# GRANT ALL PRIVILEGES ON DATABASE hive TO hive; Test connection: [root@u34 ~]# psql -h u34.data1.r3.hq.drivescale.com -U hive -d hive - Go to CM -> Cluster1 ->HIVE->Configuration-> Metastore DB Change Default port from 7423 to 5432 DB user: hive DB pass: hive 52 of 55

53 - Go to CM -> Cluster1-> Hive Action: Create MetaStore Database Action: Start all Hive services d) OOZIE: - Create PGSQL user for OOZIE (user: oozie, pass: oozie) Connect to PostGresql DB: [root@u34 ~]# sudo -u postgres psql postgres=# CREATE USER oozie WITH PASSWORD oozie ; postgres=# CREATE DATABASE oozie; postgres=# GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie; Test connection: [root@u34 ~]# psql -h u34.data1.r3.hq.drivescale.com -U oozie -d oozie - Go to CM -> Cluster1 ->OOZIE->Configuration-> OOZIE DB Change Default port from 7423 to 5432 DB user: oozie DB pass: oozie 53 of 55

54 - Go to CM -> Cluster1-> OOZIE Action: Create OOZIE Database Action: Start all OOZIE services e) HUE - Create PGSQL user for HUE (user: hue, pass: hue) Connect to PostGresql DB: [root@u34 ~]# sudo -u postgres psql postgres=# CREATE USER hue WITH PASSWORD hue ; postgres=# CREATE DATABASE hue; postgres=# GRANT ALL PRIVILEGES ON DATABASE hue TO hue; Test connection: [root@u34 ~]# psql -h u34.data1.r3.hq.drivescale.com -U hue -d hue - Go to CM -> Cluster1 ->OOZIE->Configuration-> HUE DB Change Default port from 7423 to 5432 DB user: hue DB pass: hue - Go to CM -> Cluster1-> HUE Action: Sync Database Action: Start all HUE services 54 of 55

55 f) SOLR - Go to CM -> Cluster1-> SOLR Action: Initialize SORL Action: Start all SOLR services g) SPARK - Go to CM -> Cluster1-> SPARK Action: Install SPARK Jar Action: Create SPARK History Log Directory Action: Start all SPARK services h) Ensure all the services are up and running for the cluster. - Go to CM -> Cluster1 -> Services and Hosts Status DriveScale, Inc 1230 Midas Way, Suite 210 Sunnyvale, CA Main: +1(408) www. drivescale.com WP of 55