Oracle Cloud Using Oracle Big Data Cloud. Release 18.1

Transcription

1 Oracle Cloud Using Oracle Big Data Cloud Release 18.1 E March 2018

2 Oracle Cloud Using Oracle Big Data Cloud, Release 18.1 E Copyright 2017, 2018, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agencyspecific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle.

3 Contents Preface Audience Related Resources Conventions vii vii vii 1 Getting Started with Big Data Cloud About Big Data Cloud 1-1 Before You Begin with Big Data Cloud 1-1 How to Begin with Big Data Cloud Subscriptions 1-2 About Big Data Cloud Roles and Users 1-2 Typical Workflow for Big Data Cloud 1-3 About Big Data Cloud Clusters in Oracle Cloud Infrastructure Accessing Big Data Cloud Accessing the Service Console for Big Data Cloud 2-1 Accessing the Big Data Cloud Console 2-1 Accessing Big Data Cloud Using the REST API 2-3 Accessing Big Data Cloud Using the CLI 2-3 Accessing Big Data Cloud Using Ambari 2-3 About Accessing Thrift 2-4 Connecting to a Cluster Node Through Secure Shell (SSH) 2-8 Connecting to a Node by Using SSH on UNIX 2-9 Connecting to a Node by Using PuTTY on Windows Managing the Life Cycle of Big Data Cloud About Cluster Topology 3-1 Cluster Components 3-2 Cluster Extensions 3-3 Creating a Cluster 3-3 Creating a Cluster with Stack Manager 3-9 iii

4 Viewing All Clusters 3-10 Viewing Details for a Cluster 3-10 Viewing Activities for Clusters 3-11 Viewing Cluster Component Status 3-11 Monitoring the Health of a Cluster 3-11 Scaling a Cluster Out 3-12 Scaling a Cluster In 3-12 Stopping, Starting, and Restarting a Cluster 3-13 Deleting a Cluster 3-13 Stopping, Starting, and Restarting a Node Using Identity Cloud Service for Cluster Authentication About Cluster Authentication 4-1 Connecting to Identity Cloud Service from the Service Console 4-1 Adding Identity Cloud Service Users for Clusters 4-2 Making REST API Calls to Clusters That Use Identity Cloud Service 4-3 Updating the Identity Cloud Service Password for Big Data Cloud Managing Network Access About Network Access 5-1 Enabling Access Rules 5-2 Creating Access Rules 5-2 Generating a Secure Shell (SSH) Public/Private Key Pair 5-4 Generating an SSH Key Pair on UNIX and UNIX-Like Platforms Using the sshkeygen Utility 5-4 Generating an SSH Key Pair on Windows Using the PuTTYgen Program 5-5 System Properties of Big Data Cloud Patching Big Data Cloud About Operating System Patching 6-1 Viewing Available Patches 6-1 Checking Patch Prerequisites 6-2 Applying a Patch 6-3 Rolling Back a Patch or Failed Patch Managing Credentials Changing the Cluster Password 7-1 Replacing the SSH Keys for a Cluster 7-2 iv

5 Updating Cloud Storage Credentials 7-2 Using the Cluster Credential Store 7-3 Managing Certificates Used for the Cluster Console 7-3 Updating the Security Key for Big Data Cloud on Oracle Cloud Infrastructure Managing Data Loading Data Into Cloud Storage 8-1 Uploading Files Into HDFS 8-2 Browsing Data 8-3 About the Big Data File System (BDFS) Working with Jobs Creating a Job 9-1 Running a Job 9-3 About MapReduce Jobs 9-3 Stopping a Job 9-5 Viewing Jobs and Job Details 9-5 Viewing Job Logs 9-5 Monitoring and Troubleshooting Jobs 9-6 Managing Work Queue Capacity 9-6 Creating Work Queues Working with Notebook Creating a Note in a Notebook 10-1 Running a Note 10-2 Viewing and Editing a Note 10-2 Importing a Note 10-2 Exporting a Note 10-3 Deleting a Note 10-3 Organizing Notes 10-3 Managing Notebook Settings 10-4 Interpreters Available for Big Data Cloud Troubleshooting Big Data Cloud Problems with Patching and Rollback 11-1 I can t apply a patch 11-1 Services aren t being restarted properly after life cycle operations 11-1 How do I restart ambari-agent? 11-1 v

6 How do I view the status of running services? 11-2 Problems with Administering Clusters 11-2 I get a warning that the object store credentials are out of sync 11-2 A Oracle Cloud Pages for Big Data Cloud Service Console: Instances Page A-2 Service Console Create Instance: Instance Page A-4 Service Console Create Instance: Service Details Page A-6 Service Console Create Instance: Confirmation Page A-12 Service Console: Activity Page A-13 Service Console: SSH Access Page A-15 Service Console: Instance Overview Page A-16 Service Console: Access Rules Page A-21 Big Data Cloud Console: Overview Page A-23 Big Data Cloud Console: Jobs Page A-25 Big Data Cloud Console New Job: Details Page A-27 Big Data Cloud Console New Job: Configuration Page A-27 Big Data Cloud Console New Job: Driver File Page A-29 Big Data Cloud Console New Job: Confirmation Page A-30 Big Data Cloud Console: Notebook Page A-30 Big Data Cloud Console: Data Stores Page A-31 Big Data Cloud Console: Status Page A-34 Big Data Cloud Console: Settings Page A-35 B Customize Clusters About the Cluster Bootstrap Script B-1 Bootstrap Script Execution and Logging B-2 Sample Bootstrap Script B-3 Big Data Cloud Convenience Functions B-4 vi

7 Preface This document describes how to administer and use Oracle Big Data Cloud and provides references to related documentation. Topics: Audience Audience Related Resources Conventions Related Resources Conventions This document is intended for users who want to quickly spin up elastic Apache Spark or Apache Hadoop clusters and use the clusters to analyze data. For related information, see these Oracle resources: About Oracle Cloud in Getting Started with Oracle Cloud Getting Started with Oracle Platform Services in the Oracle Cloud Infrastructure documentation Getting Started with Oracle Cloud Infrastructure Object Storage Classic in Using Oracle Cloud Infrastructure Object Storage Classic REST API for Oracle Big Data Cloud REST API for Managing Oracle Big Data Cloud PSM CLI Reference Big Data Cloud on the Oracle Cloud website The following text conventions are used in this document: Convention boldface italic Meaning Boldface type indicates graphical user interface elements associated with an action, or terms defined in text or the glossary. Italic type indicates book titles, emphasis, or placeholder variables for which you supply particular values. vii

8 Preface Convention monospace Meaning Monospace type indicates commands within a paragraph, URLs, code in examples, text that appears on the screen, or text that you enter. viii

9 1 Getting Started with Big Data Cloud This section describes how to get started with Oracle Big Data Cloud. Topics About Big Data Cloud Before You Begin with Big Data Cloud How to Begin with Big Data Cloud Subscriptions About Big Data Cloud Roles and Users Typical Workflow for Big Data Cloud About Big Data Cloud Clusters in Oracle Cloud Infrastructure About Big Data Cloud Oracle Big Data Cloud combines open source technologies such as Apache Spark and Apache Hadoop with unique innovations from Oracle to deliver a complete Big Data platform for running and managing Big Data Analytics applications. Big Data Cloud leverages Oracle s Infrastructure Cloud Services to deliver a secure, elastic, integrated platform for all Big Data workloads. You can: Spin up multiple Hadoop or Spark clusters in minutes Use built-in tools such as Apache Zeppelin to understand your data, or use the jobs API to run non-interactive jobs Use open interfaces to integrate third-party tools to analyze your data Launch multiple clusters against a centralized data lake to achieve data sharing without compromising on job isolation Create small clusters or extremely large ones based on workload and use-cases Elastically scale the compute and storage tiers independently of one another, either manually or in an automated fashion Pause a cluster when not in use Use REST APIs to monitor, manage, and utilize the service For information about the open source components used in Big Data Cloud, see Cluster Components. Before You Begin with Big Data Cloud Before you start using Oracle Big Data Cloud, you should be familiar with the following technologies: The Apache Hadoop ecosystem 1-1

10 Chapter 1 How to Begin with Big Data Cloud Subscriptions Apache Spark OpenStack Swift Object Storage Before you create a cluster: Subscribe to Oracle Cloud Infrastructure Object Storage Classic, the persistent data lake for Big Data Cloud Subscribe to Oracle Big Data Cloud (Optional) Create an Oracle Cloud Infrastructure Object Storage Classic container for your data (Optional) Create a Secure Shell (SSH) public/private key pair to provide when you create a cluster How to Begin with Big Data Cloud Subscriptions To get started with Oracle Big Data Cloud subscriptions: 1. Sign up for a free credit promotion or purchase a subscription. See Requesting and Managing Free Oracle Cloud Promotions and Buying an Oracle Cloud Subscription in Getting Started with Oracle Cloud. 2. Access Oracle Big Data Cloud. See Accessing Big Data Cloud. Note: Be sure to review Before You Begin with Big Data Cloud before you create your first cluster. If you want to grant others access to Big Data Cloud, start by reviewing About Big Data Cloud Roles and Users. Then, create accounts for users and assign them appropriate privileges and roles. For instructions, see Adding Users and Assigning Roles in Getting Started with Oracle Cloud. About Big Data Cloud Roles and Users Oracle Big Data Cloud uses roles to control access to tasks and resources. A role assigned to a user gives certain privileges to that user. In addition to the roles and privileges described in Oracle Cloud User Roles and Privileges in Getting Started with Oracle Cloud, the following role is created for Big Data Cloud: BDCSCE_Administrator. When the Big Data Cloud account is first set up, the service administrator is given the BDCSCE_Administrator role. User accounts with this role must be added before anyone else can access and use the service. A user with the BDCSCE_Administrator role has complete administrative control over the service. This user can create and terminate clusters, add and delete nodes, monitor cluster health, stop and start clusters, and manage other life cycle events. In a 1-2

11 Chapter 1 Typical Workflow for Big Data Cloud typical workflow, the administrator spins up a cluster that users can use to do their work. When the cluster is no longer needed, the administrator terminates it. The identity domain administrator can create more Big Data Cloud administrators by creating user accounts and assigning the role to the user. Only the identity domain administrator is allowed to create user accounts and assign roles. See Adding Users and Assigning Roles in Getting Started with Oracle Cloud. Typical Workflow for Big Data Cloud To start using Oracle Big Data Cloud, refer to the following tasks as a guide. Some of these tasks are performed only by administrators. Task More Information Sign up for a free credit promotion or purchase a subscription Add and manage users and roles Create an SSH key pair Create a cluster Enable network access Load data Create and manage jobs Create and manage notes Monitor clusters Monitor the service Provide your information, and sign up for a free credit promotion or purchase a subscription to Oracle Big Data Cloud. Create accounts for your users and assign them appropriate privileges. Assign the necessary Oracle Big Data Cloud roles. Create SSH public/private key pairs to facilitate secure access to all virtual machines in your service. Use a wizard to create a cluster. Permit access to network services associated with your clusters. Load the data you ll be using for your analysis. Use jobs to analyze data. Use notes to analyze data. Check on the health and performance of individual clusters. Check on the day-to-day operation of your service, monitor performance, and review important notifications. How to Begin with Big Data Cloud Subscriptions Adding Users and Assigning Roles in Getting Started with Oracle Cloud, and About Big Data Cloud Roles and Users Generating a Secure Shell (SSH) Public/Private Key Pair Creating a Cluster About Network Access Managing Data Working with Jobs Working with Notebook About Big Data Cloud Clusters in Oracle Cloud Infrastructure Monitoring the Health of a Cluster Using My Services to Manage and Monitor Your Oracle Cloud Account in Managing and Monitoring Oracle Cloud You can create Oracle Big Data Cloud clusters in Oracle Cloud Infrastructure and in Oracle Cloud Infrastructure Classic. 1-3

12 Chapter 1 About Big Data Cloud Clusters in Oracle Cloud Infrastructure The infrastructure a cluster gets created in depends on the region you select when you create the cluster. If you see the Availability Domain and Subnet fields when you select a region for the cluster you're creating, that means the cluster will be created in Oracle Cloud Infrastructure. Otherwise, the cluster is created in Oracle Cloud Infrastructure Classic. The cluster environment in either type of infrastructure is substantially the same. A few differences exist in the underlying infrastructure components and in the supported capabilities. Awareness of these differences will help you choose an appropriate infrastructure when creating a cluster. Prerequisite steps before creating your first cluster: Oracle Big Data Cloud clusters in Oracle Cloud Infrastructure require certain networking and storage resources that you must create in Oracle Cloud Infrastructure before you create your first cluster. To learn about these resources, see Prerequisites for Oracle Platform Services in the Oracle Cloud Infrastructure documentation. For step-by-step instructions to create these resources, see Creating the Infrastructure Resources Required for Oracle Platform Services. Note: Oracle Big Data Cloud uses the native Oracle Cloud Infrastructure object storage API rather than the Swift API. As such, an API signing key is required for authentication to Oracle Cloud Infrastructure Object Storage, not a Swift user name and password as described in the Prerequisites documentation above. Regions and availability domains: While creating a cluster, you select a region in Oracle Cloud Infrastructure Classic or Oracle Cloud Infrastructure. If you select a region in Oracle Cloud Infrastructure, then you also select an Availability Domain. A region in Oracle Cloud Infrastructure has multiple isolated availability domains, each with separate power and cooling. The availability domains within a region are interconnected using a low-latency network. Subnets and IP networks: In Oracle Cloud Infrastructure, you can optionally attach a cluster to an IP network that you define beforehand. In Oracle Cloud Infrastructure, you must attach each cluster to a subnet, which is a part of a virtual cloud network that you create in Oracle Cloud Infrastructure. Compute shapes: The range of compute shapes that you can select from when creating a cluster is different for Oracle Cloud Infrastructure and Oracle Cloud Infrastructure Classic. IP reservations: Currently, Oracle Cloud Infrastructure does not support IP reservations. Network access to clusters: Regardless of the infrastructure that you create your cluster in, the rules to provide network access to the cluster are preconfigured for you. The interfaces that you use to manage these rules depend on the infrastructure that the cluster is created in: For clusters in Oracle Cloud Infrastructure, you configure the rules, called security rules, in the Oracle Cloud Infrastructure interfaces. 1-4

13 Chapter 1 About Big Data Cloud Clusters in Oracle Cloud Infrastructure For clusters in Oracle Cloud Infrastructure Classic, you configure the rules, called access rules, in the Big Data Cloud interfaces. Note that these access rules prohibit access by default (with the exception of SSH access on port 22), and you must enable them to provide access to other ports. Scaling clusters: In Oracle Cloud Infrastructure, you cannot scale the shape of a cluster s compute nodes; you can scale only the storage. The minimum size of a new storage volume in Oracle Cloud Infrastructure is 50 GB. 1-5

14 2 Accessing Big Data Cloud This section describes how to access the consoles and interfaces available for Oracle Big Data Cloud. Topics Accessing the Service Console for Big Data Cloud Accessing the Big Data Cloud Console Accessing Big Data Cloud Using the REST API Accessing Big Data Cloud Using the CLI Accessing Big Data Cloud Using Ambari About Accessing Thrift Connecting to a Cluster Node Through Secure Shell (SSH) Accessing the Service Console for Big Data Cloud Oracle Big Data Cloud can be accessed through the web-based service console. Access to this console is limited to administrators. To access the service console for Big Data Cloud: 1. Sign in to your Cloud Account and navigate to the My Services Dashboard. See Signing In to Your Cloud Account in Getting Started with Oracle Cloud. 2. Click the navigation menu in the top corner of the My Services dashboard and then click the entry for Big Data Cloud. The service console opens on the Instances page. For information about the details on the page, see Service Console: Instances Page. If this is the first time Oracle Big Data Cloud has been accessed for the account, a Welcome page is displayed. Accessing the Big Data Cloud Console Clusters in Oracle Big Data Cloud can be accessed through a web-based console. The Big Data Cloud Console (also referred to as the cluster console in this document), is used to create, terminate, monitor, and manage Apache Spark jobs; create and manage notes and notebooks; browse Hadoop Distributed File System (HDFS) and Cloud storage; and manage work queue configurations. After administrators create a cluster, they give users the information they need to connect to the cluster console. Administrators also provide information about the Oracle Cloud Infrastructure Object Storage Classic container associated with the cluster when the cluster was created. Oracle Cloud Infrastructure Object Storage 2-1

15 Chapter 2 Accessing the Big Data Cloud Console Classic is the persistent data lake for Big Data Cloud and is typically where the data used for analysis is stored. Job logs are also stored there. The cluster console can be accessed in several different ways, depending on whether you have administrator privileges, and whether the cluster uses Basic authentication or uses Oracle Identity Cloud Service (IDCS) for authentication. Accessing the Big Data Cloud Console Administrators To access the cluster as an administrator: 1. Open the service console. See Accessing the Service Console for Big Data Cloud. 2. From the menu for the cluster you want to access, select Big Data Cloud Console and log in with the appropriate credentials: For clusters that use HTTP Basic authentication, log in with the administrative user name and password specified for the cluster when the cluster was created. For clusters that use IDCS for authentication, log in with your existing IDCS user name and password. When IDCS is enabled as the authentication mechanism for a cluster, anyone who can authenticate to IDCS can log in and access all cluster services. After the cluster console opens, make note of the URL. This is the URL you ll provide to users who need to access the cluster. The URL and connection information differ depending on whether the cluster uses Basic authentication or IDCS for authentication. For clusters that use Basic authentication, provide users with the cluster URL and with the credentials specified for the cluster when the cluster was created. The URL is in the form of where address is the public IP address of the MASTER-1 node on the cluster. Note that the console can be accessed on port 1080 on all master nodes in a cluster. If you can't access the console on the MASTER-1 node, try accessing it on another master node. For clusters that use IDCS for authentication, provide users with just the cluster URL. The URL is in the form of where cluster_name is the name of the cluster, and load_balancing_server_uri is the URI assigned to the cluster by the load balancing service. Because authentication is managed by IDCS, you won t send a user name and password. Users will log in to the cluster using their own IDCS credentials, which should have already been provisioned before you send the cluster URL. For information about adding users, see Adding Identity Cloud Service Users for Clusters. For both cluster types, also provide users with the URL and credentials for the Oracle Cloud Infrastructure Object Storage Classic container that was associated with the cluster when the cluster was created. Accessing the Big Data Cloud Console Cluster Users To access the cluster if you are not an administrator: 1. Obtain the information you need from your administrator: For clusters that use Basic authentication, the administrator will give you the cluster URL and the user name and password for the cluster. 2-2

16 Chapter 2 Accessing Big Data Cloud Using the REST API For clusters that use IDCS for authentication, the administrator will give you just the cluster URL. An administrator should have already added you as a user in IDCS, and you should have received an with your IDCS login information. You ll log in with your IDCS user name and password. 2. Access the cluster URL in your browser and log in when prompted: For clusters that use Basic authentication, you re presented with a basic login dialog. Log in with the user name and password provided by your administrator. For clusters that use IDCS for authentication, you re presented with the Identity Cloud Service login screen. Log in with your IDCS user name and password. All that s required to access an IDCS-enabled cluster is a valid IDCS account. The Big Data Cloud Console opens. Administrators should also give you the URL and credentials for the Oracle Cloud Infrastructure Object Storage Classic container associated with the cluster when the cluster was created. Accessing Big Data Cloud Using the REST API You can use the REST API to create and manage Oracle Big Data Cloud clusters and perform many other tasks you can perform using the web-based consoles. See: REST API for Oracle Big Data Cloud REST API for Managing Oracle Big Data Cloud You can also access the API Catalog for Big Data Cloud from the user name menu in the Big Data Cloud Console. See Accessing the Big Data Cloud Console. Accessing Big Data Cloud Using the CLI You can use a command line interface (CLI) to create and manage Oracle Big Data Cloud clusters and perform many other tasks you can perform using the web-based consoles. The Oracle PaaS Service Manager (PSM) CLI enables you to manage the lifecycle of various services in Oracle Public Cloud, including Big Data Cloud. See PSM CLI Reference. Accessing Big Data Cloud Using Ambari You can use Apache Ambari to access and manage Oracle Big Data Cloud clusters. While Ambari isn't needed for normal operations with the cluster, it's useful to open Ambari access to help with troubleshooting and certain administrative actions. To access a cluster using Ambari, you enable an access rule to open the port for Ambari, and then use the Ambari URL: 1. Open the service console. See Accessing the Service Console for Big Data Cloud. 2-3

17 Chapter 2 About Accessing Thrift 2. From the menu for the cluster you want to access using Ambari, select Access Rules. Access rules control which ports can be accessed on the VMs that are part of a cluster. 3. In the list of access rules, find the Ambari REST rule, which is associated with port 8080, the port that needs to be open. 4. From the menu for the Ambari REST rule, select Enable. The Enable Access Rule window is displayed. 5. Select Enable. The Enable Access Rule window closes and the rule is displayed as enabled in the list of rules. The given port on the cluster is opened to the public internet. 6. After the rule is enabled, click the link for the cluster at the top of the page to return to the cluster overview page. 7. On the cluster overview page, under Resources, find the MASTER-1 host, copy the Public IP address, and paste it into your browser address bar, adding port 8080 if necessary. For example, You must use https or you won t be able to connect. 8. If you re prompted for credentials, enter the user name and password specified for the cluster when the cluster was created. You should now be connected to the Ambari management console on the cluster. For information about using Ambari to upload files into HDFS, see Uploading Files Into HDFS. For general information about using Ambari, see the Ambari 2.4 documentation. About Accessing Thrift Oracle Big Data Cloud deploys two Thrift servers to provide JDBC connectivity to Hive and Spark: Spark Thrift Server and Hive Thrift Server. JDBC clients can connect to Hive or Spark servers and execute SQL. Spark Thrift Server provides a way to submit Spark jobs via SQL, and Hive Thrift Server provides a way to submit Hadoop jobs via SQL. A common use for this capability is to allow business intelligence (BI) tools to leverage the power of Apache Spark and Apache Hive. Thrift servers are automatically started when a cluster is provisioned in Big Data Cloud and are made available by default for the Full deployment profile. Thrift servers are not available with the Basic deployment profile. Creating a Keystore and Certificate Note: This section about creating a keystore and certificate does not apply to clusters that use Oracle Identity Cloud Service (IDCS) for authentication. Certificates associated with the load balancing service are typically signed by a certificate authority (are not self-signed), which means the following steps generally aren't necessary for IDCS-enabled clusters. 2-4

18 Chapter 2 About Accessing Thrift Before you can access a Thrift server, a keystore must be created with the appropriate certificate: 1. Download the certificate locally (on *nix environments): echo \ openssl s_client -connect ip_address:1080 2>/dev/null \ openssl x509 >nginx.crt where ip_address is the IP address of the Big Data Cloud Console (cluster console) or any of the master nodes in the cluster. 2. Create a TrustStore: /usr/java/default/bin/keytool -import -trustcacerts \ -keystore /tmp/bdcsce.jks \ -storepass truststore_password -noprompt \ -alias bdcsce-certs \ -file nginx.crt; where truststore_password is a password of your choosing. 3. (Optional) Verify the certificate is properly added: /usr/java/default/bin/keytool \ -keystore /tmp/bdcsce.jks \ -storepass truststore_password \ -list -v Accessing Spark or Hive Thrift Servers Most JDBC clients can access the Spark and Hive Thrift Servers. The examples in this section use the Beeline client to show how to connect. The Spark Thrift Server can be accessed using the Beeline client both inside and outside of the cluster, as well as programmatically. About the JDBC URL If inside the cluster: Spark and MapReduce jobs can read the Hive URL as a system property. Applications can access the URL and the user name from the /etc/bdcsce/datasources.properties file inside the cluster. For external clients (external to the cluster), the URL must be manually constructed and use one of the following formats. Note that the URLs are almost identical and vary only by the value of the hive.server2.thrift.http.path attribute. Note: Thrift URLs are listed on the JDBC URLs tab on the Settings page in the Big Data Cloud Console and can be copied from there. See Accessing the Big Data Cloud Console. The URLs differ depending on whether a cluster uses Basic authentication or uses IDCS for authentication. For IDCS-enabled clusters, interactions are routed through the load balancing server instead of going directly to the cluster, and that difference is reflected in the URL. 2-5

19 Chapter 2 About Accessing Thrift Basic authentication cluster URL for Spark Thrift Server: jdbc:hive2://ip_address:1080/ default;ssl=true;ssltruststore=path_to_truststore;truststorepassword=truststore_passw ord?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice URL for Hive Thrift Server: jdbc:hive2://ip_address:1080/ default;ssl=true;ssltruststore=path_to_truststore;truststorepassword=truststore_passw ord?hive.server2.transport.mode=http;hive.server2.thrift.http.path=hs2service where: ip_address is the IP address of the desired endpoint path_to_truststore is the absolute path to the Java Trust Store that holds the certificate truststore_password is the password used with the trust store IDCS-enabled cluster URL for Spark Thrift Server: jdbc:hive2://cluster_name-load_balancing_server_uri/default;ssl=true? hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice URL for Hive Thrift Server: jdbc:hive2://cluster_name-load_balancing_server_uri/default;ssl=true? hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice where: cluster_name is the name of the cluster load_balancing_server_uri is the URI assigned to the cluster by the load balancing service Accessing Using the Beeline CLI The following examples show how to access the Thrift servers using Beeline. Note: The URLs shown in the examples are for clusters that use Basic authentication. For IDCS-enabled clusters, substitute the URLs listed above, and use IDCS credentials (user name and password). Access Spark Thrift Server (example): beeline u \ 'jdbc:hive2://ip_address:1080/default;ssl=true;ssltruststore=/tmp/ bdcsce.jks;truststorepassword=truststore_password? hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice' \ -n user_name \ -p password 2-6

20 Chapter 2 About Accessing Thrift Access Hive Thrift Server (example): beeline u \ 'jdbc:hive2://ip_address:1080/default;ssl=true;ssltruststore=/tmp/ bdcsce.jks;truststorepassword=truststore_password? hive.server2.transport.mode=http;hive.server2.thrift.http.path=hs2service' \ -n user_name \ -p password where: ip_address is the IP address of the desired endpoint truststore_password is the password used with the trust store user_name is the name of the user that was specified when the cluster was created password is the password specified for the cluster when the cluster was created Accessing Thrift Programmatically Thrift can easily be accessed programmatically. The following code snippet illustrates how Thrift can be accessed within the cluster using the available system properties: String url = System.getProperty("bdcsce.hivethrift.default_connect"); Properties prop = new Properties (); prop.put ("user", System.getProperty("bdcsce.hivethrift.default_user ")); prop.put ("password", password); System.out.println("Connecting to url: " +url); Connection conn = DriverManager.getConnection(url, prop); System.out.println("connected"); Note that the Hive Thrift Server system properties used in the snippet: bdcsce.hivethrift.default_connect bdcsce.hivethrift.default_user can be replaced with the Spark Thrift Server equivalents to connect to the Spark Thrift Server instead of the Hive Thrift Server: bdcsce.sparkthrift.default_connect bdcsce.sparkthrift.default_user password can be an empty string if the client is run within the cluster. If the client is run outside of the cluster over https, password should be the Big Data Cloud Console password. CLASSPATH is used to specify any additional jars required by the job. When running an application outside the cluster, CLASSPATH should include all libraries under $ {spark_home}/jars, where spark_home points to the Spark2 install directory. System Properties Related to Thrift The following table summarizes system properties that can be used by applications or in Zeppelin to facilitate simpler connection to Thrift. 2-7

21 Chapter 2 Connecting to a Cluster Node Through Secure Shell (SSH) Property Example Value bdcsce.hivethrift.default_u ser bdcsce.hivethrift.default_c onnect bdcsce.sparkthrift.default_ user bdcsce.sparkthrift.default_ connect oscs.default.container bdcsce_admin jdbc:hive2://host:10002/ default;transportmode=http; httppath=hs2service bdcsce_admin jdbc:hive2://host:10001/ default;transportmode=http; httppath=cliservice storage.oraclecorp.com/v1/ Storage-tenant/container User name that should be used for connecting to the Hive Thrift Server. URL to connect to the Hive Thrift Server within the cluster. This can be used by jobs to execute queries against the Hive Thrift Server. User name that should be used for connecting to the Spark Thrift Server. URL to connect to the Spark Thrift Server within the cluster. This can be used by jobs to execute queries against the Spark Thrift Server. REST URL to connect to the object store. Applications running inside the cluster can query this system property to get the URL. Connecting to a Cluster Node Through Secure Shell (SSH) To gain local access to the tools, utilities, and other resources on a cluster node associated with Oracle Big Data Cloud, you use Secure Shell (SSH) client software to establish a secure connection and log in as the user opc. By default, network access to cluster nodes associated with Oracle Big Data Cloud is provided by Secure Shell (SSH) connections on port 22. Port 22 is the standard TCP/IP port that is assigned to SSH servers. Several SSH clients are freely available. The following sections describe how to use SSH clients on UNIX, UNIX-like, and Windows platforms to connect to a cluster node associated with Oracle Big Data Cloud. Note: The ora_p2bdcsce_ssh access rule controls SSH access to a cluster. The rule is created automatically when a cluster is created and is disabled by default. Before you can connect to a cluster node through SSH, you must enable the ora_p2bdcsce_ssh access rule. See Enabling Access Rules. 2-8

22 Chapter 2 Connecting to a Cluster Node Through Secure Shell (SSH) Connecting to a Node by Using SSH on UNIX UNIX and UNIX-like platforms (including Solaris and Linux) include the ssh utility, an SSH client. Before You Begin Before you use the ssh utility to connect to a node, you need the following: The IP address of the node The IP address of the node is listed on the details page of the cluster that contains the node. To display this page, see Viewing Details for a Cluster. The SSH private key file that pairs with the public key associated with the cluster The public key was associated with your cluster when it was created. If you don t have the private key that s paired with the public key, contact your administrator. Procedure To connect to a node using the ssh utility on UNIX and UNIX-like platforms: 1. In a command shell, set the file permissions of the private key file so that only you have access to it: $ chmod 600 private-key-file private-key-file is the path to the SSH private key file that matches the public key that is associated with the cluster. 2. Run the ssh utility: $ ssh -i private-key-file opc@node-ip-address where: private-key-file is the path to the SSH private key file. opc is the opc operating system user. As opc, you can use the sudo command to gain root access to the node, as described in the next step. node-ip-address is the IP address of the node in x.x.x.x format. If this is the first time you are connecting to the node, the ssh utility prompts you to confirm the public key. In response to the prompt, enter yes. 3. To perform operations that require root access to the node such as issuing ambari-server commands open a root command shell. Enter sudo -s at the command prompt: $ sudo -s # whoami # root 2-9

23 Chapter 2 Connecting to a Cluster Node Through Secure Shell (SSH) Connecting to a Node by Using PuTTY on Windows PuTTY is a freely available SSH client program for Windows. Before You Begin Before you use the PuTTY program to connect to a node, you need the following: The IP address of the node The IP address of the node is listed on the details page of the cluster that contains the node. To display this page, see Viewing Details for a Cluster. The SSH private key file that pairs with the public key associated with the cluster The public key was associated with your cluster when it was created. If you don t have the private key that s paired with the public key, contact your administrator. The private key file must be of the PuTTY.ppk format. If the private key file was originally created on the Linux platform, you can use the PuTTYgen program to convert it to the.ppk format. Procedure 1. Download and install PuTTY. To download PuTTY, go to and click the You can download PuTTY here link. 2. Run the PuTTY program. The PuTTY Configuration window is displayed, showing the Session panel. 3. In Host Name (or IP address) box, enter the IP address of the node. 4. Confirm that the Connection type option is set to SSH. 5. In the Category tree, expand Connection if necessary and then click Data. The Data panel is displayed. 6. In the Auto-login username box, enter opc. As the opc user, you can use the sudo command to gain root access to the node, as described in the last step, below. 7. Confirm that the When username is not specified option is set to Prompt. 8. In the Category tree, expand SSH and then click Auth. The Auth panel is displayed. 9. Click the Browse button next to the Private key file for authentication box. Then, in the Select private key file window, navigate to and open the private key file that matches the public key that is associated with the cluster. 10. In the Category tree, click Session. The Session panel is displayed. 11. In the Saved Sessions box, enter a name for this connection configuration. Then, click Save. 12. Click Open to open the connection. The PuTTY Configuration window is closed and the PuTTY window is displayed. 2-10

24 Chapter 2 Connecting to a Cluster Node Through Secure Shell (SSH) If this is the first time you are connecting to the VM, the PuTTY Security Alert window is displayed, prompting you to confirm the public key. Click Yes to continue connecting. 13. To perform operations that require root access to the node such as issuing ambari-server commands open a root command shell. Enter sudo -s at the command prompt: $ sudo -s # whoami # root 2-11

25 3 Managing the Life Cycle of Big Data Cloud This section describes tasks to manage the life cycle of Oracle Big Data Cloud. Topics About Cluster Topology Cluster Components Cluster Extensions Creating a Cluster Creating a Cluster with Stack Manager Viewing All Clusters Viewing Details for a Cluster Viewing Activities for Clusters Viewing Cluster Component Status Monitoring the Health of a Cluster Scaling a Cluster Out Scaling a Cluster In Stopping, Starting, and Restarting a Cluster Deleting a Cluster Stopping, Starting, and Restarting a Node About Cluster Topology The cluster topology in Oracle Big Data Cloud is based on the initial size of the cluster when it was first created. While a cluster can be scaled up or down later, the underlying cluster topology that defines master services remains unchanged. Therefore, when you re creating a cluster, it s important to consider the maximum anticipated size of the cluster and start with a master topology that can scale to meet the expected demands. Big Data Cloud provides three different cluster topologies based on the initial size of the cluster when it was created. These topologies are described in the following table. 3-1

26 Chapter 3 Cluster Components Initial Cluster Size Cluster Components 1 or 2 nodes A cluster initially created with 1 or 2 nodes has a single master node that hosts all master services. This topology is well suited for smaller clusters of less than 5 nodes. A cluster initially created with 1 or 2 nodes is not expected to scale well beyond several nodes. This type of cluster is not highly available. All services run on the same node in non-ha mode. This cluster has: 1 master node that hosts all master services N+ compute nodes N+ compute and storage nodes 3 nodes A cluster initially created with 3 nodes has 3 master nodes that host all master services. These nodes also act as storage and compute nodes. This type of cluster is expected to scale to 10 nodes and is highly available. This cluster has: 3 master nodes that host all master services N+ compute nodes N+ compute and storage nodes 4+ nodes A cluster initially created with 4 nodes has 4 master nodes, 2 of which provide NameNode services and 2 others that host the other master services. Larger clusters should initially be created with 4 nodes. This type of cluster is highly available. This cluster has: 2 master nodes that host redundant NameNodes. The NameNodes are of shape OC2m regardless of the shape of the other nodes. DataNode storage is not mounted on the NameNodes. 2 master nodes that host other master services N+ compute nodes N+ compute and storage nodes The following table lists the open source components deployed in Oracle Big Data Cloud clusters. Note: Administrators can view component status. See Viewing Cluster Component Status. Component Version More Information Alluxio Alluxio documentation Apache Ambari Ambari documentation Apache Hadoop Hadoop documentation Apache Hive Hive documentation Apache Pig Pig documentation Apache Spark 1.6 Spark 1.6 documentation 3-2

27 Chapter 3 Cluster Extensions Cluster Extensions Creating a Cluster Component Version More Information Apache Spark 2.1 Spark 2.1 documentation Apache Tez Tez documentation Apache Zeppelin 0.7 Zeppelin documentation Oracle R Oracle R documentation Oracle Big Data Cloud bundles Oracle R with all newly provisioned clusters. For information about Oracle R, see details about the Oracle R Distribution. To create a cluster, use the Oracle Big Data Cloud wizard as described in the following procedure. Before You Begin When you create a cluster, you may need to provide information about other resources, such as the following: An SSH public/private key pair An SSH public key is used for authentication when you use an SSH client to connect to a node associated with the cluster. When you connect, you must provide the private key that matches the public key. You can have the wizard create a public/private key pair for you, or you can create one beforehand and upload or paste its private key value. If you want to create a key pair beforehand, you can use a standard SSH key generation tool. See Generating a Secure Shell (SSH) Public/Private Key Pair for instructions. A cloud storage location (Optional on Oracle Cloud Infrastructure Classic) The type of location you specify depends on the infrastructure the cluster is built on: Oracle Cloud Infrastructure: Data consumed and generated by Big Data Cloud is stored in an Oracle Cloud Infrastructure Object Storage bucket. You must create a storage bucket before you create a cluster. See Prerequisites for Oracle Platform Services in the Oracle Cloud Infrastructure documentation. Oracle Cloud Infrastructure Classic: Data consumed and generated by Big Data Cloud is stored in the Oracle Cloud Infrastructure Object Storage Classic container associated with a cluster when the cluster is created. Job logs are also stored in Oracle Cloud Infrastructure Object Storage Classic. You can create the container beforehand and provide the wizard with information about it, or you can have the wizard create the container for you. If you want to create the container beforehand, see Creating Containers in Using Oracle Cloud Infrastructure Object Storage Classic for instructions. Also, before you create a cluster, review the information in About Cluster Topology. The size of a cluster when it's first created determines the cluster's topology, and even though the cluster can be scaled up or down later, the underlying cluster topology that defines master services remains unchanged. 3-3

28 Chapter 3 Creating a Cluster Procedure To create a cluster: 1. Open the service console. See Accessing the Service Console for Big Data Cloud. 2. Click Create Instance. The Create Instance wizard starts and the Instance page is displayed. For information about the details on this page, see Service Console Create Instance: Instance Page. 3. On the Instance, provide cluster information, then click Next to advance to the Service Details page. Instance Name Notification Region Availability Domain Subnet IP Network Name for the new cluster. The name: Must not exceed 30 characters. Must start with a letter. For IDCS-enabled clusters: Must contain only letters and numbers. For non-idcs-enabled clusters: Can contain hyphens. Hyphens are the only special characters you can use. Must be unique within the identity domain. (Optional) for the new cluster. (Optional) address that provisioning status updates should be sent to. (Available only if your identity domain is enabled for regions.) The region for the cluster. If you choose a region that supports Oracle Cloud Infrastructure, the Availability Domain and Subnet fields are displayed and populated, and the cluster will be created on Oracle Cloud Infrastructure. Otherwise, those fields are not displayed and the cluster will be created on Oracle Cloud Infrastructure Classic. Choose No Preference to let Big Data Cloud choose an Oracle Cloud Infrastructure Classic region for you. (Displayed only on Oracle Cloud Infrastructure) The availability domain (within the region) where the cluster will be placed. (Displayed only on Oracle Cloud Infrastructure) The subnet (within the availability domain) that will determine network access to the cluster. (Not available on Oracle Cloud Infrastructure) (Available only if you have selected a region and you have defined one or more IP networks created in that region using Oracle Cloud Infrastructure Compute Classic.) Select the IP network where you want the cluster placed. Choose No Preference to use the default shared network provided by Oracle Cloud Infrastructure Compute Classic. For more information about IP networks, see About IP Networks and Creating an IP Network in Using Oracle Cloud Infrastructure Compute Classic. 3-4

29 Chapter 3 Creating a Cluster Metering Frequency Tags Metering frequency used to determine the billing for resources used by the cluster. Tags are not supported for Oracle Big Data Cloud. 4. On the Service Details page, complete the Cluster Configuration section. For information about the details on this page, see Service Console Create Instance: Service Details Page. Deployment Profile Number of Nodes Compute Shape Queue Profile Type of cluster you want to create based on its intended use. Deployment profiles are predefined sets of services optimized for specific uses. The deployment profile can t be changed after the cluster is created. Choices are: Full: (default) Provisions the cluster with Spark, Spark Thrift, Zeppelin, MapReduce, Hive, Alluxio, and Ambari Metrics. Use this profile if you want all of the features of Big Data Cloud. Basic: Subset of the Full profile. Provisions the cluster with Spark, Zeppelin, MapReduce, and Ambari Metrics. Use this profile if you don t need all of the features of Big Data Cloud and just want to run Spark or MapReduce jobs and use Notebooks. This profile does not include Alluxio (the in-memory cache), or Hive or JDBC connectivity for BI tools. Number of nodes to be allocated to the cluster. Specify three or more nodes to provide high availability (HA), with multiple master nodes. If fewer than three nodes are specified, one node will be the master node with all critical services running on the same node in non-ha mode. Number of Oracle Compute Units (OCPUs) and amount of memory (RAM) for each node of the new cluster. Big Data Cloud offers many OCPU/RAM combinations. YARN capacity scheduler queue profile. Defines how queues and workloads are managed. Also determines which queues are created and available by default when the cluster is created. See Managing Work Queue Capacity. Note: The preemption setting can t be changed after the cluster is created. Preemption Off: Jobs can't consume more resources than a specific queue allows. Preemption On: Jobs can consume more resources than a queue allows, but could lose those resources when another job comes in that has priority for those resources. If preemption is on, higher-priority applications don t have to wait because lower priority applications have taken up the available capacity. Spark Version Spark version to be deployed on the cluster, Spark 1.6 or 2.1. Note: Oracle R Advanced Analytics for Hadoop (ORAAH) is installed for Spark 1.6 clusters only. 5. On the Service Details page, complete the Credentials section. The user name and password credentials are used to log in to the cluster and run jobs. 3-5