Oracle BDA: Working With Mammoth - 1

Transcription

1 Hello and welcome to this online, self-paced course titled Administering and Managing the Oracle Big Data Appliance (BDA). This course contains several lessons. This lesson is titled Working With Mammoth. My name is Lauran Serhal. I am a curriculum developer at Oracle and I have helped educate customers on Oracle products since I'll be guiding you through this course, which consists of lectures, demos, and review sessions. The goal of this lesson is to get you familiar with how to download and install the Oracle Big Data Appliance Mammoth Software Deployment Bundle from My Oracle Support. You also learn how to install a CDH or NoSQL cluster based on your specifications using the mammoth utility. The result of this lesson is that you'll have a well-designed Hadoop cluster that is ready to start big data workloads. Oracle BDA: Working With Mammoth - 1

2 So, you know the title of the course, but you may be asking yourself, Is this the right course for me? Click the bars to learn about the course objectives, target audience, and prerequisites. Oracle BDA: Working With Mammoth - 2

3 What can you expect to get out of this course? Here are the core learning objectives. After completing this course, you should be able to define the Hadoop ecosystem and its components. You will also learn how to complete the BDA Site Checklists, run the Oracle BDA Configuration Utility, and install the Oracle BDA Mammoth software on the Oracle BDA. Finally, you will learn how to use, administer, and manage the Oracle Big Data Appliance.. Oracle BDA: Working With Mammoth - 3

4 Who is this course for? Here is the intended audience. Application Developers Database Administrators Hadoop/BigData Cluster Administrators Hadoop Programmers Oracle BDA: Working With Mammoth - 4

5 Before taking this course, you should have basic knowledge of Hadoop, exposure to Big Data, and optionally some basic database knowledge. Oracle BDA: Working With Mammoth - 5

6 In this course, we'll talk about the following lessons: In the Introduction to the Hadoop Ecosystem, you define the Hadoop ecosystem and describe the Hadoop core components and the other related projects in the ecosystem. You will also learn how to choose a Hadoop distribution and version. You will also learn about the architectural components of HDFS, use the FS shell command-line interface (CLI) to interact with data stored in HDFS. Finally, you review MapReduce, Spark, and YARN. In the Introduction to the Oracle BDA, you identify Oracle Big Data Appliance (BDA) and its hardware and software components. In the Oracle BDA Pre-Installation Steps lesson, you learn how to download and complete the BDA Site Checklists. You also learn how to download and run the Oracle BDA Configuration Utility and then review the generated configuration files. Finally, you identify the next steps in installing the Oracle BDA. In the Working With Mammoth lesson, you learn how to download the Oracle BDA Mammoth Software Deployment Bundle from My Oracle Support. You also learn how to install a CDH or NoSQL cluster based on your specifications. You then learn how to install the Oracle BDA Mammoth Software Deployment Bundle using the Mammoth utility. Oracle BDA: Working With Mammoth - 6

7 Now that you ve learned about some of the other lessons, let s take a look at the Working With Mammoth lesson. After completing this lesson, you should be able to: Download the Oracle Big Data Appliance Mammoth Software Deployment Bundle from My Oracle Support. Install a CDH or NoSQL cluster based on your specifications. Install the Oracle BDA Mammoth Software Deployment Bundle using the Mammoth utility. Oracle BDA: Working With Mammoth - 7

8 Let us look at the roadmap from an earlier lesson on how to configure and install the Oracle BDA at a high level. For detailed information on the phases and tasks, please refer to the Oracle BDA Owner's Guide. 1. The first phase in the process is the Pre-delivery of BDA procedures which was covered in an earlier lesson. This included completing the pre-installation procedures before receiving shipment of the Oracle BDA, completing the Oracle BDA Site Checklists to ensure that you are ready for delivery of the Oracle BDA, and then running the Oracle BDA Configuration Generation Utility to generate the configuration files. This phase also includes configuring the Network. 2. The second phase in the process is the Hardware Installation and Base Configuration which is covered in detail in the Oracle BDA Owner's Guide. This is not covered in this course. 3. The third phase in the process is the Software Installation and Configuration. In this step, you install, reinstall, and reconfigure the software on the Oracle BDA. Installing the software is covered in this lesson. 4. The fourth and last phase in the process is the Upgrade and Expansions step. This is covered in detail in the Oracle BDA Owner's Guide. This is not covered in this course. Oracle BDA: Working With Mammoth - 8

9 If you have completed the Oracle BDA Pre-Installation Steps lesson, you can skip this page and the next page. If you have not completed that lesson, let's go over some key definitions that you will need to understand in this tutorial. Oracle Big Data Appliance (BDA) is a highperformance, secure platform for running diverse workloads on Hadoop and NoSQL systems. With Oracle Big Data SQL, Oracle Big Data Appliance extends Oracle's industry-leading implementation of SQL to Hadoop and NoSQL systems. By combining the newest technologies from the Hadoop ecosystem and powerful Oracle SQL capabilities together on a single pre-configured platform, Oracle Big Data Appliance is uniquely able to support rapid development of new Big Data applications and tight integration with existing relational data. Oracle BDA is pre-configured for secure environments leveraging Apache Sentry, Kerberos, both network encryption and encryption at rest as well as Oracle Audit Vault and Database Firewall. The Oracle BDA Site Checklists provides site checklists that the customer must complete before the installation of Oracle BDA. The Oracle BDA Configuration Generation Utility enables the customers to provide their specific information such as IP addresses, software preferences, that are required for deploying the Oracle BDA. After guiding you through a series of pages, the utility generates a set of configuration files which you will see at the end of this lesson. These files help automate the deployment process and ensure that Oracle BDA is configured to your exact specifications. Oracle BDA: Working With Mammoth - 9

10 The Oracle BDA base image includes the operating system, drivers, firmware, and so on. The Mammoth Software Deployment Bundle contains the installation files and the Base Image. Before you install the software, you must run Oracle BDA Configuration Generation Utility to generate the configuration files. The Mammoth Utility is a command-line utility for installing and configuring the Oracle Big Data Appliance software. Using Mammoth, you can: Set up a cluster for either CDH or Oracle NoSQL Database. Create a cluster on one or more racks. Create multiple clusters on an Oracle Big Data Appliance rack. Extend a cluster to new servers on the same rack or a new rack. Update a cluster with new software. Oracle BDA: Working With Mammoth - 10

11 An Oracle Advanced Customer Support representative (ACS) typically configures the network configuration and installs and configures the software. The Mammoth bundle contains the installation files and the base image. Before you install the software, you must use Oracle Big Data Appliance Configuration Generation Utility to generate the configuration files. This was covered in the Oracle BDA Pre-Installation Steps lesson. In this lesson, you will learn how to install the Oracle BDA Mammoth Software Deployment Bundle on the Oracle BDA. Oracle BDA: Working With Mammoth - 11

12 Regardless of the rack size, and whether you are creating CDH or Oracle NoSQL Database clusters, or upgrading existing clusters, you use the same Oracle BDA Mammoth Software Deployment Bundle to do the following: Install the software on a new Rack. Add servers to a cluster. Upgrade the software on Oracle BDA. Change the configuration of optional software. Reinstall the base image. Install a one-off patch. Oracle BDA: Working With Mammoth - 12

13 Mammoth installs and configures the software on Oracle Big Data Appliance using the files generated by Oracle Big Data Appliance Configuration Generation Utility. A cluster can be dedicated either to CDH (Hadoop) or Oracle NoSQL Database. For a CDH cluster, Mammoth minimally installs and configures Cloudera's Distribution including Apache Hadoop. This includes all the Hadoop software and Cloudera Manager, which is the tool for administering your Hadoop cluster. If you have a license, Mammoth optionally installs and configures all components of Oracle Big Data Connectors. For a NoSQL cluster, Mammoth installs Oracle NoSQL Database. CDH and Oracle NoSQL Database do not share a cluster, beginning with Oracle Big Data Appliance 2.2. Oracle BDA: Working With Mammoth - 13

14 In addition to installing the software across all servers in the rack, the Mammoth Utility: Creates the required user accounts Starts the correct services Sets the appropriate configuration parameters. You must run the Mammoth Utility once for each rack. When you are done running the Mammoth Utility, you have a fully functional, highly tuned, up and running Hadoop cluster. Note: For additional information on installing the software, refer to Chapter 10 titled Installing the Oracle Big Data Appliance Software in the Oracle BDA Owner's Guide. Oracle BDA: Working With Mammoth - 14

15 Before we get into the details on how to download and install the required software in this lesson, let's review the steps at a high level. 1. Login to My Oracle Support (MOS) at support.oracle.com. 2. Review the appropriate installation documentation. 3. Download and install the appropriate Mammoth Software Deployment Bundle software patch. In this example, you review how to download Mammoth Bundle software version for new installs. This was the latest available version at the time this lesson was developed. Oracle BDA: Working With Mammoth - 15

16 In this lesson, you will learn how to access My Oracle Support (MOS) and how to download the Mammoth Bundle Patch and the Mammoth Bundle Installation Instructions. Let's start: 1. Login to My Oracle Support (MOS) at support.oracle.com. If you don't have a My Oracle Support account, click Register here and follow the prompts to create a new account. 2. Click Sign In. 3. Enter your SSO username and password in the Sign In page, and then click Sign In. Oracle BDA: Working With Mammoth - 16

17 Once you login to MOS, the main page is displayed as shown in the screen capture in the slide. You will explore the various components of MOS in another lesson. Oracle BDA: Working With Mammoth - 17

18 In this lesson you will focus on exploring the (3) processes on how to download the Oracle BDA Mammoth Software Deployment Bundle and the installation documents. First, you will access and bookmark the Install/Upgrade/Configure Oracle Big Data Appliance (Doc ID ) page from MOS. Next, you will access and bookmark the Oracle Big Data Appliance Patch Set Master Note (Doc ID ) from MOS. Finally, you will download Mammoth bundle patch # which you will use to install the software on the BDA. You also download the installation document # Oracle BDA: Working With Mammoth - 18

19 First, you will access the Install/Upgrade/Configure Oracle Big Data Appliance (Doc ID ) document. You can search for any document id or patch using the following procedure: In the Knowledge Base section of the page, type in the Enter search terms text box (highlighted in blue), and then click Search (also highlighted in blue). The Recommended Links section displays the url for the document. Click the link. Oracle BDA: Working With Mammoth - 19

20 The Information Center: Install/Upgrade/Configure Oracle Big Data Appliance (document ID ) page is displayed. This is an important page that you should bookmark in your Web browser. This page provides access to many of the tasks that you will need to configure and deploy the Oracle BDA which we will cover on the next page. Among other things, you can use this page to access the following: The latest Oracle Big Data Software Installation Documents. The Oracle Big Data Appliance Patch Set Master Note (Doc ID ) which contains all the versions of the OS Base Image and Mammoth released. Installation Frequently Asked Questions. Oracle BDA latest Site Checklists and Configuration Generation Utility. Let's access the Oracle BDA Patch Set Master Note (Doc ID ) page. Click on the Oracle Big Data Appliance Patch Set Master Note link. Oracle BDA: Working With Mammoth - 20

21 The Oracle BDA Patch Set Master Note (Doc ID ) document is displayed. This is another important page that you should bookmark in your Web browser. You can find all released versions for the software patches and installation instructions on this page. In this lesson, you will learn how to perform a new install using the latest version available to us at the time of developing this tutorial which is If you click the Mammoth Bundle software version for new installs (Oracle Linux 6), and upgrades (Oracle Linux 5) link, the first row in Table 1 shows the following: OS Base Image Patch Number and installation instructions, Mammoth Bundle Patch Number and installation or upgrade Instructions, Site Checklists, Configuration Generation Utility, and the Documentation link. This is where you can download the Mammoth Bundle Patch version and the associated installation instructions. This is covered in this lesson. If you look closely at the table in the slide, you will notice that the installation has two parts: 1. Installing the OS base image. This is not covered in this self-study. 2. Installing the Mammoth Bundle Software. This is covered in this lesson. Click the Mammoth Bundle software version for new installs (Oracle Linux 6), and upgrades (Oracle Linux 5) link. Table 1 is displayed which is explained on the next page. Oracle BDA: Working With Mammoth - 21

22 If you click the Patch link, the patch details window is displayed. This window also shows the two.zip files that are contained in this patch. The highlighted file in the second screen capture in the slide is used to upgrade the 4.1 OS base image to version 4.2. Mammoth can be installed on the 4.1 OL6 base image. There will not be any BDA Base Image released. The second Mammoth zip file cannot be used for reimaging BDA servers or racks; instead, the latest 4.1 BDA Base Image should be used. To download the two.zip files, click Download next to each file. Oracle BDA: Working With Mammoth - 22

23 Before you install the software, you should access, save, and read the Oracle BDA Mammoth Software Deployment Bundle Installation Document (Doc ID ). This document contains all the details about what you will need during the installation. To access this document from the Information Center page (Doc ID ), click the Install and Configure tab, and then click the Installation Support Documents as shown in the slide. The Installing Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release V4.2 for New CDH Installations with OL6 Base Image (Doc ID ) document is displayed. It is very important that you go through the Known Issues section of this document for important information. Oracle BDA: Working With Mammoth - 23

24 In the example in an earlier lesson, you already learned about using the Oracle BDA Configuration Generation Utility to generate the configuration files. In this lesson, we will download and install the Mammoth Software Deployment Bundle on a New Rack. Typically, a BDA arrives at the customer's site with a fresh OS base image installation. A customer seldom has to install a new base image because the Mammoth upgrade patches now update the base image as a part of their process (in steps 1 3 as we will see later in this lesson) prior to the reboot. Mammoth bundle patch # contains the following two files: 1. p _420_linux-x86-64_1of2.zip. This file contains the following: README.txt: This describes the files provided in the patch BDAMammoth-ol : A directory containing BDAMammoth-ol run file that you will run. This is a self-extracting Mammoth bundle to be installed on the BDA 2. p _420_linux-x86-64_2of2.zip. This file contains the OS Base Image version 4.2. You must unzip this file before executing the.run file; otherwise, you will get the error message shown in the slide. Oracle BDA: Working With Mammoth - 24

25 Before you install the software, you will need to validate the two.zip files. First, check the file sizes using the ls ltr command. Compare the two file sizes in your results with the sizes shown the first screen capture in the slide or with the results shown in the installation document. Second, verify the cksum for the two.zip files. Compare your results with the second screen capture in the slide or with the results shown in the installation document. Next, you verify the contents of the first file using the unzip command with the t option. Oracle BDA: Working With Mammoth - 25

26 Next, you verify the contents of the second file using the unzip command with the t option. Oracle BDA: Working With Mammoth - 26

27 Follow the steps in Oracle BDA Base Image Version for New Installations on OL6 (Doc ID ) to install Base Image version on a fresh BDA install or a new rack. You can install mammoth on OL6 base image; however, if the cluster has any release of mammoth software installed, then you need to perform an upgrade to instead of an install If mammoth software is installed, then you can perform an upgrade to only from the or higher version of mammoth Oracle BDA: Working With Mammoth - 27

28 Use the imageinfo utility on any server to get the image version. You can install the software on the new rack only if all racks of a single Hadoop cluster have the same image version. As we mentioned earlier, before you can proceed with the install, you need to run the Oracle BDA Configuration Generation Utility 4.2 to generate the required configuration files that will be used by Mammoth. This was covered in another lesson. Oracle BDA: Working With Mammoth - 28

29 Let's continue with the mammoth bundle install instructions. First, copy the two.zip files to any directory on node01 such as /tmp. Log in to node01 as root and decompress the downloaded.zip files using the unzip command as shown in the code example in the slide. Oracle BDA: Working With Mammoth - 29

30 Unzipping the two files on the previous page extracts Patch in the /tmp directory as shown in the slide using the ls command. The BDAMammoth-ol subdirectory contains the BDAMammoth-ol run file. This will run the mammoth utility. The BDABaseImage-ol _RELEASE directory contains the BDABaseImage-ol _RELEASE.iso file. This will update the OS base image in steps 1 through 3 when you run the mammoth utility. Oracle BDA: Working With Mammoth - 30

31 Mammoth accepts as input a JSON-formatted configuration file that should be placed in the directory /opt/oracle/bdamammoth. The name of this file will be of the form <cluster-name>-config.json where <cluster-name> is the name of the cluster. In our example, our file name is bda1h1-config.json that was generated using the Oracle BDA Configuration Utility in the EXAMPLE/bda1h1 directory. This was covered in an earlier lesson. Important: Before we cd to the /opt/oracle/bdamammoth directory, we need to copy the is bda1h1-config.json file from the EXAMPLE/bda1h1 directory to the /opt/oracle/bdamammoth directory. Oracle BDA: Working With Mammoth - 31

32 Extract all files from BDAMammoth-version.run, and then execute BDAMammoth-ol run as root on node01 of the primary rack. Change the directory to BDAMammothol , and then execute BDAMammoth-ol run as shown in the first code example in the slide. List the files created by BDAMammoth-ol run as shown in the second code example in the slide. One of the extracted files is the mammoth utility. The new version of the Mammoth software is installed in /opt/oracle/bdamammoth, and the previous version (if you are upgrading) is saved in /opt/oracle/bdamammoth/previous-bdamammoth. Oracle BDA: Working With Mammoth - 32

33 The two screen captures in the slide show a sample output after you run the BDAMammothol run script. The two screens provide very useful information about the mammoth utility which is covered in this lesson. Oracle BDA: Working With Mammoth - 33

34 Steps 1-16 are mandatory whereas step 17 is optional. Step 16 performs the cluster validation tests to ensure that the installed cluster is healthy. Oracle BDA: Working With Mammoth - 34

35 Mammoth is the command-line utility that deploys software on Oracle's Big Data Appliance. You can use Mammoth to: Setup the cluster using the generated configuration files. Create a cluster on one or more racks. Create multiple clusters on an Oracle BDA rack. Extend a cluster to new servers. Update a cluster with new software. Oracle BDA: Working With Mammoth - 35

36 You must run the mammoth utility from the /opt/oracle/bdamammoth directory which is created when you run BDAMammoth-ol run script. To run the utility, use the mammoth command followed by the arguments and the cluster name if required. For example, to run all of the required installation steps on our bda1h1 cluster, run the mammoth command with the i argument. The -i option or argument runs all the steps on the cluster. This is equivalent to running the mammoth script with the -r 1-18 argument. You can also run one step at a time. For example, the following command will run only step 1:./mammoth -s 1 When Mammoth completes step 3 of the installation, it prompts you to reboot, if it upgraded the base image. Oracle BDA: Working With Mammoth - 36

37 The table in the slide lists all of the arguments that you can use with the mammoth command. The l argument lists all the steps of the mammoth utility. The s <step #> argument runs the specified step on the primary rack. The -r <start step>-<end step> argument runs steps from <start step> through <end step>. The -i [cluster_name] argument runs all steps on the cluster, equivalent to -r Use this option when configuring a new rack or adding a group of servers to a cluster. The -e newnode1 newnode2 argument adds new nodes to an existing cluster and installs the needed software on the newly added nodes in one step. The -p argument upgrades the cluster to the latest patch level. The -p patchnumber argument installs a one-off patch. The -u patchnumber argument uninstalls a one-off patch. The c argument runs the BDA cluster check tests. The h argument displays the command help including the usage and a list of steps. The v argument displays the version number. Oracle BDA: Working With Mammoth - 37

38 Let's go over some examples using different arguments with the mammoth command. If you run the mammoth command with the -l argument, it will list all of the mammoth steps. If you run the mammoth command with the s 3 bda1h1 argument, it will run only step 3 on the bda1h1 cluster that is currently being setup. If you run the mammoth command with the r 2-6 bda1h1 argument, it will run only steps 2 through 6 on the cluster that is currently being set up. If you run the mammoth command with the -i bda1h1 argument, it performs a complete install on cluster bda1h1. If you run the mammoth command with the -p argument, it upgrades a cluster to the latest patch level. If you run the mammoth command with the -c argument, it runs the BDA cluster check tests. These are run automatically after Step 15 as well. Oracle BDA: Working With Mammoth - 38

39 This is a good place to go over few notes about steps 9, 10, and 11 of the installation. Step 9: InstallHadoop: Installs all packages in Cloudera's Distribution including Apache Hadoop (CDH) and Cloudera Manager. It then starts the Cloudera Manager server on node03 and configures the cluster. Mammoth does not install CDH or Cloudera Manager on Oracle NoSQL Database clusters. Step 10: StartHadoopServices: Starts the agents on all nodes and starts all CDH services. After this step, you have a fully functional Hadoop installation. Step 11: InstallBDASoftware: Installs the server-side components of Oracle Big Data Connectors, if this option was selected in Oracle Big Data Appliance Configuration Generation Utility. Oracle Big Data Connectors must be licensed separately. This step optionally installs Oracle NoSQL Database on clusters allocated to its use. Enterprise Edition requires a separate license. Oracle BDA: Working With Mammoth - 39

40 Cloudera Manager provides a single administrative interface to all Oracle Big Data Appliance servers that are configured as part of the Hadoop cluster. You can use Cloudera Manager to perform the following administrative tasks: Monitor jobs and services. Start and stop services. Manage security and Kerberos credentials. Monitor user activity. Monitor the health of the system. Monitor performance metrics. Track hardware use (disk, CPU, and RAM). Refer to the following links for further information about Cloudera Manager. latest/cloudera-manager-introduction/cloudera-manager-introduction.html Oracle BDA: Working With Mammoth - 40

41 To use Cloudera Manager, enter a URL as shown in the slide in your Web browser. In this example, bda1 is the name of the appliance, node03 is the name of the server, example.com is the domain name, and 7180 is the default port number for Cloudera Manager. Note: Only a user with administrative privileges can change the settings. The default user name and password is admin, which is changed in Step 17 when you run the Mammoth utility. Mammoth does not install or start CDH services on Oracle NoSQL Database clusters. Oracle BDA: Working With Mammoth - 41

42 In Cloudera Manager, you can choose any of the following pages from the menu bar at the top of the display: Home: Provides a graphic overview of activities and links to all services controlled by Cloudera Manager. Clusters: Accesses the services on multiple clusters. Hosts: Monitors the health, disk usage, load, physical memory, swap space, and other statistics for all servers in the cluster. Diagnostics: Accesses events and logs. Audits: Displays the audit history log for a selected time range. Charts: Enables you to view metrics from the Cloudera Manager time-series data store in a variety of chart types, such as line and bar. Backup: Accesses snapshot policies and scheduled replications. Administration: Provides a variety of administrative options, including Settings, Alerts, Users, and Kerberos. Oracle BDA: Working With Mammoth - 42

43 Oracle recommends that you execute steps 1, 2, and 3 individually using the s argument with the mammoth command. Next, you can run the rest of the steps together using the i argument or individually or the s argument. As mentioned earlier, at the end of step3, Mammoth reboots the BDA. To view a sample output of each step after running the mammoth command, see the following MOS document: Installing Oracle Big Data Appliance Mammoth (Software Deployment Bundle) Release V4.2 for New CDH Installations with OL6 Base Image (Doc ID ) You can find the log files for each step in the /opt/oracle/bdamammoth/bdaconfig/tmp/bdanode01-<timestamp>.log directory. Oracle BDA: Working With Mammoth - 43

44 In this lesson, you should have learned how to: Download the Oracle Big Data Appliance Mammoth Software Deployment Bundle from My Oracle Support. Install a CDH or NoSQL cluster based on your specifications. Install the Oracle BDA Mammoth Software Deployment Bundle using the Mammoth utility. The result of this lesson is that you'll have a well-designed Hadoop cluster that is ready to start big data workloads. Oracle BDA: Working With Mammoth - 44

45 In this course, we discussed Introduction to the Hadoop Ecosystem, Introduction to the Oracle BDA, Oracle BDA Pre-Installation Steps, and Working With Mammoth. You should now be able to: Define the Hadoop ecosystem and its components Describe Apache Hadoop Distributed File System (HDFS), MapReduce, and YARN Complete the BDA Site Checklists Run the Oracle BDA Configuration Utility Install the Oracle BDA Mammoth software on the Oracle BDA Use, administer, and manage the Oracle Big Data Appliance (BDA) Oracle BDA: Working With Mammoth - 45

46 The Oracle Learning Library offers other self-paced courses about Oracle Big Data Appliance. Visit the Oracle Learning Library to learn about the courses. Oracle BDA: Working With Mammoth - 46

47 Oracle University offers In-Class courses about Oracle Big Data and Oracle NoSQL database. Visit Oracle University to learn about the courses. Oracle BDA: Working With Mammoth - 47

48 The Oracle Learning Library offers many free demonstrations and tutorials. And, of course, the Oracle Big Data Appliance documentation and online help embedded within the product are also valuable resources. Oracle BDA: Working With Mammoth - 48

49 Oracle BDA: Working With Mammoth - 49

50 Oracle BDA: Working With Mammoth - 50