Gemini Enterprise Administration Guide V2.6

Transcription

1 Gemini Enterprise Administration Guide V2.6

2 CONTENTS Prerequisites Note on Network Access Control System Initialization Localization License Activation Provisioning Bulk Provisioning Bulk Provisioning - Appliance Discovery Bulk Provisioning - Network Settings Bulk Provisioning - Hostname Bulk Provisioning - Change Admin Password Bulk Provisioning - Connect to LDAP Bulk Provisioning - SSH Authentication Bulk Provisioning - Summary Stand-Alone Provisioning - Join Cluster Stand-Alone Provisioning - Change Admin Password Success Initial Login Adjust Storage Plan Navigation Featured Platforms Integration Center Splunk Installation Splunk Clustered Environment Installation Stand-Alone Installation Cloudera CDH Installation Node System Time Timezone Name Hostname Local Hosts Network 1

3 Bonding Port Redirect OS Users FTP FTP Service FTP User SSH SNMP SNMP Service SNMP Agent SNMP Agent version 1 SNMP Agent version 2c SNMP Agent version 3 SNMP Trap Thresholds SNMP Trap Destinations Failover Creating a Failover Group Joining a Failover Group Log Receiver Storage Storage Devices Mount disk and mount points Encryption and Decryption Create Software RAID Disk Create Logical Volume Merge Disk Add NFS Mount Add CIFS Mount Add S3 Mount Add iscsi Target Manage Swap space Monitoring Diagnostics Rsync Backup Benchmark Cluster Manage Nodes 2

4 Manage Groups Execute Jobs Membership Settings Bulk Provision License License Status Remote Licenses Inventory License Server Splunk Daemon Web Interface Hunk Apps Splunk Diag Optimizer Config Editor Splunk Versioning Command Splunk Environments Add Node Build Environment Create Environment Create Cluster Organize Nodes Locate Nodes Deploy Independent Stream Forwarder Create Environment Organize Nodes Locate Nodes Environment Administrations Upgrade Splunk Cloudera Initial Cloudera Deployment Accessing and Using Cloudera Manager Add Node to Cloudera 3

5 Remove Node from Cloudera Settings System Admin Custom SSL Certificate Information Authentication Manager Users User Permissions LDAP Single Sign-on (SSO) Password Policy Proxy Login Banner Reboot Shutdown Account Profile Logout CLI Commands Command Reference Commands for initial setup Network Settings Provisioning Gemini Cluster Membership Manager Web Administrations Commands for information gathering Manager Version Model Service Tag Installed Packages Service Status Exposed Network Ports System Information Commands for troubleshooting Password Reset Password Change Generate SSL Key 4

6 Cloudera Installation Reset Gemini Cluster Reset Network Reset Service Restart Remove Splunk Instance Reset Splunk Cluster Manager Apply Patch Independent Stream Forwarder Operations Commands for system operations System Reboot System Power Off Default Passwords for CLI Operations 5

7 Prerequisites Cabling - An ethernet cable and available ethernet switch connection. Networking - One IP address for Gemini appliance s ethernet port 1 (address, netmask and gateway are required for manual configuration. DHCP is supported) Client - A client PC with access the Gemini appliance s IP address defined above. Browser - Google Chrome browser is recommended. (optional) Access to the Splunk Enterprise installation tarball (splunk-*.*.*-*-linux-x86_64.tgz). (optional) Internet access is required for Cloudera CDH installation. Note on Network Access Control To administrate and run the Gemini Appliance and services, certain communication channels between clients and nodes are required. As a minimum, ports tcp/443 (HTTPS) and tcp/22 (SSH) are required for basic system configuration and operations. As the Web Interface and SSH console offer low-level system access, make sure to not expose these ports towards public access (Anywhere, /0). Depending on the Deployment, add inbound/outbound rules as needed. For a complete list of used network ports, see the respective article on our Support Portal ( ). 6

8 System Initialization The appliance configuration is performed using the Gemini Enterprise: Manager web console. Using a supported web browser, navigate to: <IP address from prior step> A security warning or message may be displayed. This is expected and does not indicate a problem. Depending on your browser you may have to choose proceed anyway or continue for the page to load. Upon the first visit to Manager, the End User Software License Agreement is shown. After reading the terms, choose Accept to advance to the next screen. 7

9 Localization Manager supports multiple languages. Setting a preferred language adjusts the entire user experience accordingly. If the hostname and timezone settings were skipped in previous basic configurations, this step provides an opportunity to configure them. 8

10 License Activation This step allows you to activate the appropriate license for your intended use. You can choose to activate the full Enterprise Edition which requires a pre-purchased license or the Free Edition, which requires no license. You may also opt for the third option of a 30-day trial of the Enterprise Edition which does not require a licence. Note In Free Edition: The following premium features are restricted: No Failover group No LDAP Authentication No support of external storages, including NFS, CIFS, and S3. No remote license server. Limited Splunk configurations versioning, restricted to roll back to last 3 versions. Gemini Cluster features are restricted. Up to 4 nodes in a cluster in maximum. No scheduled jobs. Jobs for Splunk upgrade are restricted. Jobs for Gemini appliance boot control are restricted. Jobs for Splunk service control are restricted. 9

11 If you chose to activate a pre-purchased Enterprise license, the next step will walk through the application process. You may either choose to retrieve the license from an existing license server or apply a new license from a file. Applying a license from a file involves a three step process, outlined in this screen. It involves generating a request, submitting the file to Gemini Support and finally uploading the received license file to the appliance. 10

12 Alternatively, if a license server is used to manage all available licenses, selecting the Connect to a License Server will allow specification of license server information - IP Address and token - in order to perform the validation. 11

13 12 Gemini Enterprise: Manage Administration Guide 2018 Gemini

14 Provisioning If you have multiple appliances to be configured, select Bulk Provisioning to launch a wizard that allows configuration of multiple appliances at one time. The configuration includes all aspects of naming, network configuration, LDAP access and software installation. To operate as a standalone appliance, select Stand-Alone (single) provisioning. Bulk Provisioning The Bulk Provisioning Wizard is a step-by-step workflow that allows configuration of multiple appliances in one simple process. Before proceeding further, all other appliances that are to be configured, will need to be powered up and network-accessible. No other configuration steps are needed on these other appliances. 13

15 Bulk Provisioning - Appliance Discovery This steps discovers and confirms the entire set of appliances that are to be configured. The discovery process can be done in one of two ways: If the IP addresses of all appliances are known, and available in a text file, it can be uploaded to launch the discovery process. The file will need to contain one IP address per line. Alternatively, the appliances can be discovered on the network by performing an IP subnet scan, using a CIDR notation to specify the subnet. E.g /24. Please note that the scope of the subnet will determine the length of time taken for the scan to complete. 14

16 Bulk Provisioning - Network Settings This step configures the mechanism used for assigning IP addresses of the appliances being configured. The default option Network settings - Static assigned uses dynamic assignment using a DHCP Server. Alternatively, choosing Network settings - Static assigned will utilize DHCP for the initial assignment only, but then use that IP as the static setting from that point onwards. This is useful when building a DHCP server for deployment temporarily, and the DHCP server is not needed after that step. 15

17 Bulk Provisioning - Hostname This step provides multiple options for automatically assigning a hostname to each appliance being provisioned. If DNS records have been assigned for each appliance, the Use Reverse DNS Lookup option uses the hostname discovered through the DNS server. Alternatively, the Specify Custom Pattern option allows the specification of a custom string pattern composed of text and certain allowed tokens, to automatically compose a hostname. The following tokens are allowed: $service_tag$ - is replaced with the service tag of the appliance, as indicated on the box or available within Gemini support. $increment$ - is an automatically incrementing number 16

18 Bulk Provisioning - Change Admin Password This step is used to specify the password for the admin account on each appliance being configured. It is recommended that you use a strong password or if applicable, follow appropriate password security policy as required for your enterprise. Please note all the appliances will be updated with the same admin password. 17

19 Bulk Provisioning - Connect to LDAP If an LDAP account is to be required for access to each appliance, this step allows the specification of LDAP server authentication information. This will be used to authenticate users upon login to the appliance. Read LDAP Authentication in Settings chapter for more details. Please note that LDAP authentication is optional, and by default the appliance resorts to local account-based authentication. 18

20 Bulk Provisioning - SSH Authentication This step allows configuration of SSH access on each appliance being configured. It allows the specification of a password for SSH access, or the upload of an SSH key to complete the key exchange and bypass the password. Please note this step is optional and can be skipped. 19

21 Bulk Provisioning - Summary This summary screen lists all the appliances about to be provisioned along with configuration settings. It should be used as a final review and confirmation step before starting the automated configuration step. it is also strongly recommended that the a CSV file be downloaded for future reference. Clicking on Start initiates the automated provisioning process, which can take several minutes or longer depending on the number of appliances being provisioned. The status of each appliance is updated in real-time. After all appliances have been provisioned, clicking on Finished completes the process. 20

22 Stand-Alone Provisioning - Join Cluster To include this node into an existing Manager Cluster, select Join an existing appliance cluster and provide the IP Address and the Token String that were assigned to the Master Node. To operate as a standalone appliance, or to configure a cluster at a later time, select Operate as a standalone appliance. 21

23 Stand-Alone Provisioning - Change Admin Password Update the password for the account admin in Manager here. It is recommended that you use a strong password or if applicable, follow appropriate password security policy as required for your enterprise. 22

24 Success Congratulations! The Completed screen lets you know that this appliance has been configured. Click Get Started to launch Manager. Return visits to this page will proceed directly to login. Configured settings may still be changed within the corresponding areas within Manager. 23

25 24 Gemini Enterprise: Manage Administration Guide 2018 Gemini

26 Initial Login Upon completion of the setup process and clicking on Get Started in the previous section, you will be presented with the login screen. Log in to Manager with the username admin and the newly changed password from the previous step. 25

27 Adjust Storage Plan Beware of the disk partitions and mount points. Not every disk are mounted on the system default partitions especially the Gemini Appliance models. There re different storage configurations on each appliance model, such as onboard flash disks, all hard disk drive(hdd), all solid state disk(ssd), and hybrid configurations. If this appliance is planned to store large amount of data, such as Splunk indexer and Hadoop worker node, you should do the following checks before you start to deploy any applications: 1. Understand the storage devices and mount points on the appliance. Go to NODE -> Storage -> Storage and you will see a list view of storage devices and mount points. 2. HDD and SSD are mounted on the following mount points by default: a. HDD disk will be mounted on /opt/mnt/hdd01. b. SSD disk will be mounted on /opt/mnt/ssd01. c. Not applicable to the following models: G1000, IB-1050D. 3. Design your storage plan and adjust the logical volumes and mount points. The default storage plan might not satisfy your needs. You may adjust them to the new logical volumes and new mount points. For example: a. Unmount SSD and remove it from logical volume, and merge it into /opt to extend it s capacity. b. Unmount HDD and mount it on /opt/splunk so that the whole splunk including binary, configurations, and data are stored in the same disk. 4. Remember the final storage plan. This is useful when deploying applications and configuring the data store path. Note The mix use of variant storage types and speed, e.g. SSD, HDD and iscsi connected disks in one RAID disk or in one logical volume, are not recommended. It will slow down the disk performance and make it unpredictable. 26

28 Navigation The Home Screen is the first and default screen shown upon successful login. The navigation bar on the left provides easy access to different areas of management, as described in the rest of this guide. The entire Manager experience is organized into different areas accessed by the vertical navigation bar on the left, and are as follows. HOME - This is the default home page upon login that provides access to setup and manage different platforms such as Splunk, Hadoop and other applications. NODE - Provides configuration settings for such as networking, identity, and preferences for a specific appliance CLUSTER - Allows management of appliances as part of a multi-node cluster. LICENSE - Allows centralized management of Manager licenses. SPLUNK - Allows for the configuration of Splunk operating parameters. This section is only available once the Splunk configuration has been completed. It appears when Splunk Enterprise is installed. 27

29 HADOOP - Allows for the configuration of Cloudera cluster management. This section is only available once the Cloudera configuration has been completed. It appears when Cloudera is installed. SETTINGS - This section provides configuration settings for Manager itself such as system update, authentication, security settings, and backup/restore. ACCOUNT - This section allows other configuration of user information and account level functions such as password reset. Featured Platforms The featured platforms are data processing platforms which are available for activation and usage on the appliance. Clicking on Activate will allow easy deployment of a platform with a few clicks and an easy process. Please note that each platform consumes resources. Installed and running more than one platforms in a Gemini appliance may result in resource competition. Ensure that your appliance storage, CPU and configuration are suited to the workload imposed by each platform. Please consult with Gemini Support on sizing considerations. Integration Center The Integration Center is a repository of applications that provide insights, management or connectivity to data sources. Some apps mirror those provided within other platform ecosystems such as SplunkBase, while others are specific to Gemini. Regardless of origin, full support is provided for all solutions. 28

30 Each solution or application is represented by a descriptive card that allows easy installation, or if already installed, easy configuration or removal. The option buttons and configuration screens pertaining to each application are specific to each application. 29

31 Splunk Installation This process ensures proper installation and configuration of Splunk. Note that the appliance does not ship with the Splunk binaries, and a download from the Splunk site is required, as part of this process. Before proceeding, ensure that you have the following ready: An active Splunk account to download the Splunk installation file The current Splunk Enterprise for Linux binary (.tgz) Splunk Enterprise Licence If you don t have a Splunk account or haven t downloaded the latest installation file, instructions and links are provided within the Manager Install Splunk Enterprise installation page. Note Apple/Safari users, you will want to ensure you have disabled Open Safe Files after downloading to retain the.tgz extension. From the Manager Home Screen, start the Splunk installation process by clicking the Install button located under the Splunk Enterprise banner. Alternatively, select Splunk from the toolbar on the left side of the page. 30

32 Splunk Clustered Environment Installation If there are multiple Gemini Appliances provisioned and planned to build one or more Splunk clusters, you may leverage Splunk Environments to complete the Splunk installation on all the nodes at once. By leveraging Splunk Environments, you can create one or more Splunk clusters, including index clusters and search head clusters, in one or more locations within an environment. When you activate Splunk on the home page, you will see the Environments page shown below. If there re nodes are provisioned by the Bulk Provisioning mentioned earlier, they will be listed in the Unassigned Nodes section. You may also add nodes manually by clicking Add Node. Make sure there re nodes in Unassigned Nodes and meet the requirements for Splunk clustering. Click the Build Environments to start the wizard. It will guide you to complete the Splunk Environments configurations. Please refer to Splunk Environments in Splunk Chapter for more details. 31

33 Stand-Alone Installation If you want to complete the Splunk installation only on this appliance, click the Daemon on the left menu, you will see the Install Splunk Enterprise page shown below. From the Upload Splunk Tar File panel, click the Upload & Install button to begin the installation process. In the Upload and Install subpanel, click the...choose the tarball button then select your previously downloaded binary of Splunk Enterprise for Linux. You will see the name of the selected file appear in the installation subpanel. Click the Upload button. 32

34 When the file has been uploaded, its status is displayed in the subpanel. Click the Install button and accept the Splunk Software License Agreement. The Splunk Enterprise software will now be installed on your Gemini Appliance. 33

35 When completed, the Manager Home page displays the currently installed version of Splunk and its status. Note If you re installing Splunk 7.1 and above, the default Splunk account and password would be admin and changeme. For security considerations, you should change your password after the first time logged into Splunk. 34

36 Cloudera CDH Installation The Cloudera CDH is Cloudera s software distribution including Apache Hadoop and related additional components. A full list of available pacakges can be found here: Gemini Enterprise Manager installs prerequisites and the required Cloudera software, such as the Cloudera Manager and Cloudera Manager Agents, across a deployment of Gemini Appliances. More information about Cloudera Manager can be found at Note To run a production-ready Cloudera Cluster, the Cloudera Manager and some of the Apache Hadoop components require an external database. For more details and system requirements, please cosult the Cloudera Documentation at s/cm_ig_installing_configuring_dbs.html#cmig_topic_5_1. Cloudera Setup on Manager requires at least two Gemini Appliance instances, one Cloudera Manager and up to 80 Agents. During installation, the appliance hosting the Cloudera Manager will download required software packages from the official Cloudera Mirror ( ) and share them to connected Manager nodes which will be used as Cloudera Agent. 35

37 To begin the Cloudera CDH Installation on Gemini Appliance, log in to Manager on the appliance node which is the designated Cloudera Manager and activate the Cloudera platform from the HOME screen. Once activated, follow the detailed instructions in the Cloudera chapter in this documentation. 36

38 Node The Node tab is the starting point for the configuration of the host and server functions related to the Gemini appliance. System Time Accurate timekeeping is vital to ensure correct event order. Distributed environments may become out of sync and transactional searches may return inaccurate information if time is not accurately set and frequently updated. 37

39 Manager uses pool.ntp.org as a time source by default. Additional network time sources, either external or internal may be added by clicking Add NTP Server Setting the NTP Sync toggle to the OFF position will halt further network time updates and allow for manual editing of the system time. This may be required under special circumstances, but is not advised for general operations. Click Set Time to set the datetime manually, or click Sync with Browser to update the datetime settings with client PC. 38

40 39 Gemini Enterprise: Manage Administration Guide 2018 Gemini

41 Timezone Accurate timezone configuration is essential in maintaining event-order integrity, particularly in geographically distributed environments. 40

42 Name Hostname To prevent conflicts in distributed environments as well as declare the source path of received events, Manager requires that each device has a unique hostname. Splunk uses this hostname as a default value to populate both server.conf and inputs.conf when started for the first time. Local Hosts While not required in normal operation, manually configuring local hosts can ensure connectivity between hosts in the absence or failure of a DNS server. In the case of high latency DNS servers manually configuring hosts may improve performance. Manually configuring local hosts is not considered best practice and should only be used in exceptional cases as multiple static configurations can be complicated to manage and easily become out of date. Note DNS settings should be configured separately in each network interface in the Network tab. 41

43 To add a static host, click Add New Record and specify the host s IP address and hostname. 42

44 Network The Manager Network Interfaces configurations may be reviewed and edited here. Manager supports multiple network interface cards (NICs) and Gemini appliances contain four or six NICs. Additionally Bonding and Port Redirects may also be configured here. 43

45 Each NIC may be configured with an IP address either manually or via DHCP. Advanced configurations like MTU and TX Queue Length can be configured especially for network performances. Note Set MTU to a value larger than 1,500 to enable Jumbo Frame if this ethernet interface is used for iscsi connection. Consult your NAS vendor for more details. Static route rules may be added to a specific network interface to communicate with networks which do not directly connect to the Gemini appliance. 44

46 Bonding Manager provides support for Link Aggregation. Here you can bind multiple physical NICs into one Virtual interface to increase throughput beyond what a single connection can sustain while providing redundancy in the event of a single NIC failure. Select the Physical Network Interfaces to be included in the Bonding group. Select the Mode for load balancing and fault tolerance from the drop-down menu. Options include: Round Robin Active-Backup XOR Broadcast IEEE 802.3ad Dynamic Link Aggregation Adaptive Transmit Load Balancing Adaptive Load Balancing. MII is used to verify the status of the NIC. Specify the frequency of MII link monitoring by entering a value in milliseconds in Millisecond Monitor. The default value is 100. Then click Add. 45

47 46 Gemini Enterprise: Manage Administration Guide 2018 Gemini

48 Once created, the Virtual Interface is shown in the list You can edit the configuration of the newly created bond interface: 47

49 Port Redirect As with any other application running as a non- root user, Splunk will be unable to bind to and listen on the privileged ports, those which are less than Port Redirect allows you to define rules for Manager to redirect incoming connections on privileged ports to a port above Enter the Source port and the Destination port. OS Users To access the Gemini Appliance using SSH, the password for OS users can be changed from this screen. Note that OS Users don t have access to the Manager web interface. You may also see if an OS user is locked or forbidden to login. 48

50 To unlock an OS user, select Yes in Allow Login and save the changes. If you forgot the password, it also gives you a chance to reset it. Note For security considerations, you should lock out unused OS user accounts here. 49

51 In some cases you might need a dedicated OS user account to run scripts or applications. You may create a new OS account here, and assign it to desired groups so that you can share access permissions to other accounts. FTP Enabling the FTP service allows for the uploading of data to the /opt directory. Generally this is performed when hosts are not able to have a Splunk Forwarder installed locally, allowing them to move data to a host such as a Gemini appliance that can ingest the data correctly. The FTP service requires the configuration of an FTP user (see below) as well as an entry in inputs.conf instructing Splunk to ingest any new data found. Note the FTP protocol is not natively encrypted and should only be used when security practices allow for it. FTP Service Use the FTP Service toggle and select the desired port (Manager defaults :2121) to enable FTP. 50

52 51 Gemini Enterprise: Manage Administration Guide 2018 Gemini

53 FTP User The FTP protocol requires both user credentials and a directory to store received files. Manager installs with a default FTP user named splunk with the home directory of /opt/sbox. Click Add FTP User and provide the desired username, password and root folder to add subsequent accounts. Click on an existing username to change it s root folder or password. 52

54 SSH The SSH service provides remote command line access to Manage. SSH is enabled by default for the users sbox and splunk. The SSH protocol is natively encrypted. Toggle SSH Service to start or stop the service. Port: Configure the listening port of SSH service. Session Timeout: the session timeout interval in minutes. Forward SSHD Log: when you enabled Forward SSHD Log, Manage will forward a copy of SSHD logs to /var/log/sshd/sshd.log for further use. Allowed Authentication Method: SSH login with password or authorized keys. Note that only authorized keys is allowed when Manage is running on AWS. Rekey Limit: it will renegotiate a new key when the traffic reached 1GB. This will prevent against the key being cracked and traffics being decrypted by attackers. Fail to Ban: Once enabled, it will ban the client IP within one Note Please refer to CLI commands chapter for default passwords. It requires you to change password when the 1st time login to Manage. 53

55 SNMP SNMP Service The SNMP service allows a remote Simple Network Management Protocol enabled host to interrogate the Gemini appliance for monitoring and alerting data. The SNMP service requires the configuration of an SNMP Agent (below). To enable SNMP toggle SNMP Service. SNMP Agent The SNMP Agent requires that the SNMP Service (above) be enabled to function. Select CREATE SNMP AGENT to create a new SNMP Agent entry. Multiple SNMP Agents can be configured. Select a unique name for the SNMP Agent entry and choose an SNMP Agent version (see below). Note Only Alphanumeric, dot, hyphen, and underscore characters are allowed in the input fileds. 54

56 SNMP Agent version 1 SNMP Version 1 is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure versions are not possible. SNMP v1 supports a maximum of 32 bits per counter. Input the NETWORK and MASK BITS (Subnet Mask) of the host network. Enter a COMMUNITY STRING for SNMP authentication. The default is public - changing this is strongly recommended. 55

57 SNMP Agent version 2c SNMP Version 2c is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure versions are not possible. SNMP v2c supports a maximum of 64 bits per counter. Input the NETWORK and MASK BITS (Subnet Mask) of the host network. Enter a COMMUNITY STRING for SNMP authentication. The default is public - changing this is strongly recommended. 56

58 SNMP Agent version 3 SNMP Version 3 supports authentication, encryption and 64 bit counters. Select which password hashing method to use under AUTHORIZATION ALGORITHM. Manager supports either MD5 or SHA. Enter the authentication password. Select an encryption method for SNMP communication using ENCRYPTION ALGORITHM. Manager supports DES and AES128. AES128 is considered to be the more robust of the two. Enter the encryption password and select ADD to add the agent. 57

59 SNMP Trap Thresholds Enable SNMP Traps for Gemini appliance s performance data and specify the frequency and threshold. SNMP Traps may be enabled for: CPU Disk Link Memory Process - ftp, splunk, ssh, syslog-ng. 58

60 SNMP Trap Destinations Provide information for the SNMP manager: Enter the IP address of your SNMP Host Select a protocol: use trapsink to send SNMPv1 traps use trap2sink to send SNMPv2 traps use informsink to send inform notifications Enter a Community String Enter a Port over which to send the information. Note Only Alphanumeric, dot, hyphen, and underscore characters are allowed in the input fileds. 59

61 Failover Gemini appliance allows users to create Failover groups. Manager nodes configured in a Failover group provide for a high level of availability in the event one of the nodes become unavailable due to a maintenance window, network outage, or other event. Each group has one active master peer to hold a virtual IP and several standby slave peers that are ready to take over for a failed master. Each Gemini appliance can be part of different Failover Group and each group should be provisioned using a different port number. Creating a Failover Group Under the Node Manager menu, click on Failover and then click Create New Failover Group to create a new group. Virtual NIC - IP Address - Assign a IP to this virtual group that is reachable from existing network interfaces. Please note this IP must be reachable from every member within the group. Monitor - Determine what event will trigger a hand over to another group member. For example, if Detect Splunk was selected, when Splunk running on the master node went down, one slave node will step in and take over the master role. Add node to this group - You may directly assign group members here and member nodes will join into this group automatically. 60

62 Joining a Failover Group In the Failover page, click Join Existing Group and enter the IP of Virtual NIC assigned on the master node within the group: Once joined, the Failover group will display all members and their role within the system (Master or Slave): If the Master node becomes unavailable or detected the monitoring events, the Slave node will step in and serve the virtual IP which used to be managed by the Master node. 61

63 Log Receiver Enabling the Syslog service allows Gemini appliance to receive events from remote devices using the commonly used Syslog protocol. Source filtering rules can be specified to direct how to parse, split and direct different components into different destinations. Configuring a rule involves the following steps. 1. Create Rule - Creates a syslog receiver rule. You can create multiple rules. 2. Source - Data source indicates the source location of the data. a. Specify the protocol and port to receive syslogs. b. Each rule can only utilize a single data source. c. Logs will be stored at the designated destination. d. Multiple destinations are allowed. 62

64 3. Add Filter - Specify a filter for this rule. Select it in the drop down menu from the associated data source. a. This step is optional. b. Filter is used to parse and split log entries with defined types. Only matched logs will be stored into defined destination. 63

65 4. Add Destination - Specify a destination for this filter. Select it in the drop down menu from the associated filter. a. Destination is used to indicate where to store the received log data. b. There must be at least one destination for each filter. 5. Add Destination - If there are no filters created, specify a destination for this source. Select it in the drop down menu from created data source. a. There must be at least one destination in each rule. b. Multiple destinations are allowed for each data source. c. User custom path is allowed to be inserted in the full path. 64

66 6. Save Rule - Once configuration of source, filter and destination are complete, configured, click on Save Rule to save the rule settings. If there are more than one data sources, create rules as needed for each data source, and then specify filters and destinations. 65

67 If you re willing to apply the same rules to other appliances which play the log receiver role as well, you may replicate the settings to these nodes by clicking Cluster and then Replicate Syslog Settings. 66

68 Storage The Storage section allows administrators to manage local storages and attached storages, including direct attached and network attached, to extend disk capacity for data applications such as Splunk and Cloudera. This allows the volume of an existing system to be extended and the mount point for Splunk indexes may also be defined. This also gives capability that allows for reading files from network storage. Storage Devices All the detected attached storages are listed here for further actions: you may create a RAID disk from multiple storage devices, create a new logical volume for grouping storage devices as one, merge storage devices with the existing logical volume to extend it s disk capacity, or mount it to a designated mount point. Plan your storage use by considering the data growth and expansion plan. Some actions are not revertible so plan it before doing actions. 67

69 Mount disk and mount points You may mount a new storage device to a user custom mount point under /opt/mnt/. The owner of this mount point is sbox and permission is open to all. You may maintain owners and permissions of files and folders under this mount point by your own. 68

70 If this storage device is entirely planned for Splunk, you may mount it to /opt/splunk directly. Note The custom path /opt/sbox/mount is deprecated and has been removed from selections. Existing mounts will continue without any impacts until unmounted. Encryption and Decryption Manage supports disk encryptions and this has been simplified and implemented as an option while mounting disks. This is optional and it s disabled to all disks by default. You may encrypt a disk and mount it with a new key, or mount it with an existing key. Create New Key File: This will encrypt the disk with a new key file. All the data on this disk will be erased. Use Existing Key File: If this disk was encrypted from this machine, this will allow the disk mounted again with existing key. Upload Key File: If this disk was encrypted from somewhere else, this will allow the disk mounted again with the uploaded key file. 69

71 Once mounted, you may backup the key by download the encryption key file. This is strong recommended for preventing from key loss situation happened. Note Encrypted disk can t be used for creating RAID disk or merging into logical volume. Decrypt it before new allocation. Encrypt a logical volume is not supported. 70

72 Encryption with a new key and decryption will erase all data. Backup the key file is recommended. Create Software RAID Disk With RAID, you can group more than one storage devices as a disk array with redundancy or acceleration, depends on the RAID level. You may refer to this page to understand more about the RAID and RAID level: You may select the most proper RAID level for your use cases: RAID 0(Striping): When disk redundancy doesn t matter, and cares about disk performance. RAID 1(Mirroring): When there re only 2 disks, and cares about the data integrity and availability. RAID 5: When there re more than 3 disks, cares about data integrity and availability as well as the performance. This is balanced in performance, capacity, and availability. Note This is specifically benefit to software instances without having hardware RAID controller, e.g. Vmware, Hyper-V, and AWS. Disk drives on Gemini Appliance are supported and managed by RAID controller already. Merge a RAID disk into a logical volume is not supported. The size of each storage device can be different when selecting RAID 5, but there might have disk space waste. 71

73 The mix use of variant storage types and speed, e.g. SSD, HDD and iscsi connected disks in one RAID disk, are not recommended. It will slow down the RAID disk performance and increase latency. Create Logical Volume The major advantage of a logical volume is disk space extensibility for growing data. You may extend it s disk capacity by merging more storage devices into an existing logical volume any time. Note A logical volume can be created with one or more storage devices. The size of each storage device can be different. There s no way to split storage devices from an existing logical volume but remove the logical volume entirely. Plan the storage devices carefully. The default logical volume rootvg-lv01 can t be removed. The mix use of variant storage types and speed, e.g. SSD, HDD and iscsi connected disks in one logical volume, are not recommended. It will make the disk performance unpredictable. 72

74 Merge Disk Merge storage device into a logical volume - You may select a target logical volume if there re more than one logical volumes existed. Note Once a storage device has been merged into the default logical volume rootvg-lv01, this action will not be able to be reverted. When a device has been merged into a logical volume, it must keep attached unless the partition might be corrupted and data will be lost. Merge a RAID disk into a logical volume is not supported. Merge an encrypted disk into a logical volume is not supported. Add NFS Mount To define an NFS Mount Point: Enter the local mount point (located at /opt/sbox/data folder), Enter the IP address of the remote server Enter the remote folder (starting with a leading /). Select the mount type. Hard mount is recommended by Splunk when the mount point is used for cold buckets. Select NFS version. This must match to the version of NFS server. Click the Add button to add the new NFS mount. 73

75 Note A mount point will not be detected and validated until you enabled the configuration by clicking the mount button. Once enabled, Manager will automatically mount the NFS Mount Point upon boot. 74

76 Add CIFS Mount To define a CIFS Mount Point: Enter the local mount point (located at /opt/sbox/data folder) Enter the IP address of the remote server Enter the remote folder (starting with a leading /) Enter the Username Enter the Password Click the Add button to add the new CIFS mount. Please note that a mount point will not be detected and validated until you enable the configuration. When enabled, Manager will automatically mount the CIFS Mount Point upon boot. 75

77 Add S3 Mount To define an Amazon S3 Mount Point: Enter the S3 bucket name you want to mount and the local mount point will locate at /opt/sbox/data/s3/<bucket name> folder. Enter the IAM Access Key ID Enter the IAM Secret Access Key If you want all the data stored in the S3 bucket are encrypted, enable Server-Side Encryption(SSE) and select a proper key option. To get your S3 Access credentials, log in to the AWS Console, open the Users section in the IAM Service and click the desired user. Create an Access key in the Security credentials tab. Please note that access to S3 storage requires a connection to the public internet from the node. Note S3 is designed for data archival and not applicable to Splunk indexing. Specify hot/warm/cold buckets to S3 mount mounts will cause Splunk malfunctions. 76

78 Add iscsi Target To add an iscsi target: Modify the Initiator Settings and specify the Login CHAP and Discovery CHAP. This must match to the settings on iscsi target. In Target Discovery field, input the iscsi target IP address and port, e.g :3260. Note the default discovery port is 3260/tcp. Once there re iscsi targets found, it will be listed in below. Select Login to connect to this iscsi target. Once connected, there is a new block device detected and listed in Undefined Storage tab. Go to Undefined Storage tab and mount it. 77

79 Advise your NAS administrator to get the iscsi target information and CHAP credentials. Please note that connected iscsi target only means there are new block devices available. Don t forget to mount them in Undefined Storage. Note Set MTU to a value larger than 1,500 to enable Jumbo Frame for the ethernet interface used for iscsi connection. This will improve iscsi performance. Consult your NAS vendor for more details. Manage Swap space Swap space is disabled by default on Manage for better performances. It s only enabled on several Gemini Appliance models. However you will still need it on an appliance with heavy loading or for applications that might consume a lot of memory. Once it is enabled, it will allocate disk spaces as swap space by following the formula: 1. Swap space size is equal to RAM size. 2. If RAM size is larger than 64GB, swap space size will be set to 64GB. 78

80 Monitoring Enabling Monitoring allows Manager to send the output of its own Admin and System logs to the destination of your choosing, either local storage on the Gemini appliance or a central syslog server. To create a file locally, enter the name of the file to be output (e.g., admin_file.log ). To send a file to a syslog server, select the Destination Protocol, UDP or TCP, and enter the IP address of your syslog server. The service defaults to port 514, which you may also customize. 79

81 80 Gemini Enterprise: Manage Administration Guide 2018 Gemini

82 Diagnostics The Diagnostics Panel provides access to useful network tools without the need to access the command line interface. The following commands can be executed and and the resulting output shown in the window. Ping 81

83 TCP Connect NS Lookup 82

84 Traceroute 83

85 TCP Dump IOSTAT 84

86 85 Gemini Enterprise: Manage Administration Guide 2018 Gemini

87 Rsync Backup There are many modern network attached storages now supports backup through rsync. With this feature you can backup Splunk configurations and data in the /opt/sbox folder to the remote storage regularly. To enable rsync backup, you need to do the following: 1. Complete the SSH key exchange process between Manager and remote server, and make this remote server allow SSH login from Manager with public key. a. Click Download SSH Public Key to download the SSH public key from Manager. The default file name should be id_rsa.pub. b. Add this public key into authorized list in the remote server. The authorized list usually located at ~/.ssh/authorized_keys in the remote server. In a manual way, you may use the following command to add it into the list on the remote server: cat id_rsa.pub >> ~/.ssh/authorized_keys 2. Configure remote server information. There re 4 fields need to be filled: a. Remote Hostname/IP. b. Remote Port. This is the port SSH protocol listened on the remote server. Default value is 22/tcp. c. Destination Path. The folder name the data will backup to. d. User Name. This is exactly the same with the one the SSH public key added to. 86

88 3. Determine backup scope. There re 2 options you may select: Splunk configuration and folders in /opt/sbox. in /opt/sbox/, you can specify which folders would you like to backup. 4. Configure the backup plan. In this section you need to determine the backup strategy, including the policy and schedule: a. If you select Always create a new full copy, the disk space may consume very fast. Watch the free disk space of the remote server regularly. b. If you select Keep a single copy up-to-date, then there ll be only one copy existed. You will not be able to restore data from older copies. 5. Click Save in the bottom to save all the configurations above. 6. Click the Backup Configuration Through Rsync toggle button to enable rsync backup. This will test if you successfully exchanged the public key and make it added into the authorized key list in the remote server. 87

89 Benchmark Sometime you need to evaluate if the hardware specs are qualified taking the role and running disk I/O intensive tasks, e.g. Splunk indexers. In this page, you can run disk benchmark on specific devices, monitor the disk IOPS(Input and Output Operations Per Second) in real time, and download the result. The detailed benchmark methodology is available in the Gemini Support Portal. How to start a disk benchmark: Click Run Benchmark and select the target device to benchmark from side panel. Read the note carefully before you click Run Benchmark. When clicked, the disk benchmark will start. During the benchmark process there s no way to cancel or stop. During the benchmark process, it will monitor the operating system and display the IOPS in real time. It will also record the max IOPS on screen. 88

90 When benchmark is completed, you can see the benchmark result in average. You can download the complete result in below for deeper analysis. 89

91 Cluster The Gemini Cluster tab is the starting point for managing distributed configuration options. Here you can view your distributed topology, connect the current node to a parent or manage child nodes belonging to the current node. Manage Nodes Manage Nodes provides a tabular view of all Gemini appliances in your deployment. Here you can view key information on each host including: IP Address Parent number of joined groups number of dispatched jobs. You can also drill into each host and assign it to a Node Group. 90

92 91 Gemini Enterprise: Manage Administration Guide 2018 Gemini

93 Manage Groups Manage Groups allow you to create logical categories of your Manager nodes. Node Groups may be used to view a subset of your Manager nodes as well as assign jobs to specific Manager nodes. Child Nodes may be assigned to multiple Node Groups. 92

94 Execute Jobs Execute Jobs allows you to easily execute predefined tasks including starting/stopping/enabling/disabling services and retrieving and updating system parameters to multiple Manager instances and execute them at a specific time. Jobs can be assigned to all nodes or previously defined Node Groups. 93

95 94 Gemini Enterprise: Manage Administration Guide 2018 Gemini

96 Membership Settings Membership Settings allows you to enable and configure registration of your Manager nodes to this Manager. Manager Nodes which are registered may be monitored and managed from this device. Here you may also configure the Token String to be used by the remote node for registration. You may restrict the nodes allowed to register by creating a Whitelist using either IP addresses or Hostnames. By default, all nodes are allowed to register. Multiple entries must be separated by a comma. Bulk Provision The same the option provided during initial configuration. This is a step-by-step wizard to guide you complete the initial setup configurations. Refer to Bulk Provisioning in System Initialization for detailed configuration steps. 95

97 License The Manage License tab allows you to configure your Manager License Servers and License Agents. License Status License Status presents you the current active license, including license type, volume, and expiration date. By clicking the current license in Product field, system will display detailed license status. If you have a pre-purchased license, you can request and then attach your license by following the steps in this page. Remote Licenses Remote Licenses allows you to manage the license sources, including remote license servers and from local. 96

98 Inventory Inventory lists all the license attached and allows you to manage Gemini licenses, and for convenience, groups volumes in total and licenses used. 97

99 License Server License Server allows you to enable and configure your Gemini appliance as a Manager license server. Manager license server manages licenses and grant permissions to other nodes who registered and connected to this license server. Here you may also configure the Token String to be used by the remote nodes for registration. You may restrict the nodes allowed to register by creating a Whitelist using either IP addresses or Hostnames. By default, all nodes are allowed to register. Multiple entries must be separated by a comma. 98

100 Splunk The Splunk tab contains various areas of management related to your Splunk installation. It allows you to perform common tasks, edit configuration files, and manage Splunk applications. Daemon Allows you to review and modify settings related to Splunk Enterprise s splunkd process without using the command-line. Here you may stop or restart Splunk; upgrade Splunk; reset the Splunk Admin password; enable or disable boot-start; and review and modify advanced configurations. 99

101 Web Interface Allows you to review and modify settings related to Splunk Enterprise s Web Interface, Splunk Web. Here you may disable or enable Splunk Web; launch Splunk Web in your browser; and review and modify advanced configurations such as enabling encryption and configuring the default port. 100

102 Hunk Splunk Analytics for Hadoop, formerly known as Hunk, provides seamless search and report functionalities on data stored in HDFS. As a minimum, Hunk requires HDFS with one NameNode and at least one DataNode, as well as MapReduce or Yarn (recommended). The Hunk section in Manager allows you to configure the required, so-called Hadoop Providers to connect to HDFS and verify the required permissions (optional). Specify a Provider Name to identify the configuration later back in Splunk. The HDFS NameNode FQN requires a URL in the format hdfs://<name_or_ip_of_namenode>:<port>/ The port is optional and defaults to To operate, Hunk stores a handful files on HDFS in an intermediate directory. First, create the path on HDFS per Search Head to be connected to HDFS manually and define read/write permissions for the splunk user (Splunk Enterprise on Gemini Appliance runs as non-privileged user splunk, so as all HDFS operations performed by Hunk). Hunk in Manager will also automatically create additional parameters to the Virtual Index Provider in order to run Hunk. Review the configuration manually in /opt/splunk/etc/system/local/indexes.conf. After successfully adding the provider, go to Splunk Enterprise administration, make sure that the Splunk Analytics for Hadoop license is installed and click Settings Virtual Indexes to add a new Virtual Index. Use the provider created in the wizard above. 101

103 Apps Apps provides a list of all Splunk Apps currently installed on your Gemini Appliance. Each App may be downloaded to your desktop as a tarball file. Drilling down on the App directs you to the App s directory listing in the Splunk Conf Editor. 102

104 Splunk Diag Splunk diagnostic allows you to quickly create a Splunk Diag file [./splunk diag] on-demand. Multiple copies of Splunk Diag files may be stored for later retrieval and download. 103

105 Optimizer Allows you to select a predefined Splunk role for your Gemini appliance automatically updating all the conf_files for you. Options include: Splunk Default Indexer Heavy Forwarder Search Head All In One. 104

106 Config Editor The Conf Editor allows you to edit, create, or upload Splunk configuration files within the $SPLUNK_HOME/etc/ file path from the convenience of the Manager web console. The editor provides file path navigation links and complete versioning of all file revisions. 105

107 Splunk Versioning Enabling the Splunk Configuration Repository allows you to manage changes, and retain multiple versions of configuration files. This provides for roll-back capability after making changes. 106

108 Command Splunk Command allows you to issue Splunk commands directly from your browser. Additionally, the Splunk Command Helper provides for easy, interactive building of complex Splunk commands which may then be issued within the browser. 107

109 Splunk Environments This Splunk Environments will allow you to manage a multiple sites, full-clustered Splunk environment with ease. You can manage to build an environment with Splunk Indexer Clusters and Splunk Search Head Clusters created in one or more locations. When there re new Splunk versions released, you can upgrade the whole environment with few easy clicks. Prerequisites: Splunk must not installed on all the nodes which are waiting for assignment by Splunk Environments. If you want to create a Splunk Indexer Cluster, the following conditions must be met: At least 3 nodes needed for a indexer cluster, one for cluster master and others for peer nodes. If multi-site clustering is enabled, there must be at least 2 indexers in each site. There must have enough nodes in Nodes tab. If you want to create a Search Head Cluster, the following conditions must be met: A Splunk Indexer Cluster must be created. It will be used when creating a Search Head Cluster. At least 4 nodes needed for a Search Head Cluster, one for deployer and others for peer nodes. 108

110 A standard configuration procedure would be the following: 1. Add Nodes and make sure all the nodes are waiting for assignment. 2. Build an Environment by following the 4-steps instructions. 3. In step 1, specify the environment name, sites, and upload Splunk binary. 4. In step 2, create one or more Splunk clusters for this environment. 5. In step 3, organize the nodes to the relevant clusters. 6. In step 4, specify the sites for each cluster. 109

111 Add Node Nodes can be added in 2 ways: Bulk Provisioning. All the bulk provisioned nodes will be added automatically. Manually add. You may add other Manager nodes manually. Note that the Manager version must be the same with the host running cluster management. 110

112 Build Environment This is a 4-steps wizard to guide users to build a Splunk environment and complete the all necessary settings. Create Environment In this step there re several attributes need to be determined: Deployment Type - Select Deploy Multi-Use Environment to continue the rest steps. Environment Name Available Sites - Refer to Splunk documentation and understand the sites meaning. Splunk Software - The running Splunk version in this environment. There s only one Splunk version used in an environment. 111

113 Create Cluster There are 2 types of Splunk clusters that can be created: Splunk Indexer Cluster Splunk Search Head Cluster Refer to Splunk documentation and architecture Best Practices to understand how clustering works in Splunk. 112

114 Organize Nodes In this step you need to search and select unassigned nodes and add them into the created clusters. Please note each cluster has it s minimum requirements. If the condition can t be met, there ll be a warning in the status field. In Your Clusters, you can specify the desired Master Node for indexer cluster and Deployer for search head cluster. Locate Nodes In the last step, we need to specify where the nodes located especially if there are multiple sites available in this environment. 113

115 In the end, click Deploy and it will start the deployment. This may take minutes, depends on the amount of nodes and the size of each cluster in this environment. Deploy Independent Stream Forwarder You may also deploy Splunk Independent Stream Forwarder onto multiple Manage nodes located in multiple sites by leveraging Splunk Environments. There s also a 114

116 wizard to guide users to deploy independent stream forwarders and complete the all necessary settings. Create Environment Similar to Splunk cluster deployment, but select Deploy Independent Stream Forwarder Only when determine the deployment type. Organize Nodes Select the nodes in Available Nodes and assign the Independent Stream Forwarder role to them by using Assign Standalone Role button. Configure the settings to specify how to acquire the binary and where the program reported to when it s running. You may obtain required information from Distributed Forwarder Management in Stream Forwarder App. 115

117 Locate Nodes In the last step, we need to specify where the nodes located especially if there are multiple sites available in this environment. Environment Administrations Once the environment has built and deployed, we can do the following administrations to this environment: Redeploy - If there re any error occurred during deployment, use this to redeploy again to fix the issue and complete the deployment. Upgrade - Upgrade the Splunk version to all the nodes in this environment. Delete - Delete the whole environment. Note: all the installed Splunk instances in this environment will be removed. 116

118 Remove cluster from existing cluster. Note: all the installed Splunk instances in this cluster will be removed. Add nodes into a cluster Remove nodes from existing cluster. Note: all the installed Splunk instances in this node will be removed. Upgrade Splunk 117

119 This will upgrade all the Splunk instances running in this environment. Before it starts, there re something you should know: All the running Splunk version in an environment are managed and should keep them consistent. Splunk running on these nodes should not be upgraded individually. To keep zero down time, the upgrade process is taking rolling upgrade by following Splunk recommended upgrade procedure. Only one node will go down for upgrade at a time. This may take long time when the environment is large. Only upgrade Splunk to a newer minor version is accepted. Upgrade Splunk to a next major version is forbidden. For example, upgrade Splunk from version to is accepted, however upgrade from version to is forbidden. 118

120 Cloudera Initial Cloudera Deployment Before deploying Cloudera to Gemini Appliance, please activate the Cloudera platform from the Integration Center as described in section 5 Cloudera CDH Installation of this document. It also helps to have a general understanding of Cloudera capabilities and usage. To continue, add the Full Qualified Domain Names (FQDN) of the Gemini Appliances where Cloudera Manager and Cloudera Manager Agents should get installed to: Add as many agents as required by clicking the Add Agent button. Everytime another FQDN is added to the table, Manager will confirm proper DNS resolution and connectivity to the host. A green check indicates success on both fronts, and a red icon indicates a failure in either or both. Additionally, the setup verifies if the entered FQDN for the Cloudera Manager matches the appliance where the administrator is currently logged in to. Also, at least one additional Cloudera Agent is required to continue the setup. If Manager cannot resolve the FQDN using a configured Name Server, a red icon indicates that the IP Address needs to be added manually. When entering the IP Address manually, Manager will again check the connectivity and update the status icon. 119

121 Example 1: Manager cannot resolved mycustomfqdn, admin is required to enter IP Address manually. Example 2: Manually entered IP Address is incorrect. Example 3: Manually entered IP Address is correct and a valid Manager installation is detected. When all prerequisites are met, the Deploy button is activated and the installation can start. Once deployment has started, the horizontal bar indicates the progress and the current step of the installation. This stage of the installation runs in the background, allowing the administrator to work on other areas while it is in progress. Returning to the Hadoop page will show the current status and allow continuation when the installation is complete. 120

122 Successful installation may take up to 20 minutes. The page reloads automatically once completed. Open the Cloudera Manager web interface with the equivalent button or using the URLs respectively and follow the instructions to perform a Cloudera Cluster setup. If any unexpected error occurs during the installation, the setup will stop and show an error message. To restart the setup, reset the Cloudera installation on each appliance separately using the CLI command sbox cloudera --undo : 121

123 Accessing and Using Cloudera Manager The Cloudera Manager web interface is available at respectively for SSL secured access. The default username and password for Cloudera Manager are both admin. Make sure to accept the Cloudera End User License Terms and Conditions before proceeding: 122

124 After choosing the installation type (Cloudera Express, Cloudera Enterprise or Cloudera Enterprise Trial), select the hosts where the Cloudera Agents have been deployed to by Manager in the following screen: 123

125 Choose Use Parcels (Recommended) as method and the latest supported CDH version (Currently: CDH 5.10). If required, choose any additional parcels: The Cloudera Cluster Installation will now distribute all required parcels from the Manager to the Agents using a temporary mirror of the Cloudera Archive on the Cloudera Manager appliance. Please wait until all tasks are finished. 124

126 After passing the Host Inspector, finish the Cluster Installation and proceed to the Cluster Setup. Choose which Hadoop Services that you wish to use on your cluster from the list. To use Cloudera in conjunction with Splunk Analytics for Hadoop (see chapter Hunk in this guide), choose Custom Services with Services Types HDFS and YARN : Customize the the Role Assignments based on your requirements before proceeding to step 3: 125

127 In step 3, select Use Custom Databases for production deployments and enter the connection details: After the Database Connection test was successful, proceed to step 4 and update the Cloudera Setup paths according to the scheme in the table below: 126

128 Original Path /opt/dfs/... /var/lib/... /opt/yarn/... /tmp/... /var/log/... /var/run/... Path on Manager /opt/cloudera/dfs/... /opt/cloudera/lib/... /opt/cloudera/yarn/... /opt/cloudera/tmp/... /opt/cloudera/log/... /opt/cloudera/run/... The wizard will now run the Cloudera setup commands: 127

129 Once finished, the Cloudera Cluster has been set up and can be administered using the Cloudera Manager. Add Node to Cloudera The Deployment of Cloudera CDH on Gemini Appliance can be extended with additional Cloudera Agent installations to new appliance nodes. To perform this action, open the Hadoop section from the Manager web console and click on Add Node for every additionally required node. Manager will again perform validation checks, such as DNS Resolution and Connectivity, and indicate the result with green check marks or red icons. If all checks are successful, start the installation by clicking Deploy. After the installation, the new node will be propagated automatically to Cloudera Manager where it can be added to existing Cloudera Clusters. For that, login to the Cloudera Manager, go to Hosts All Hosts and click Add New Hosts to Cluster. If asked, click Classic Wizard in the next step and continue. Switch to the Currently Managed Hosts tab, where the newly added Gemini Appliance appears: 128

130 Check all hosts which you like to add to the Cluster and click Continue. The Cloudera Manager will now distributed required parcels to the new hosts. After passing the Host Inspector, optionally choose a Host Template to be applied and finish the wizard. You might now redistributed the Roles to the new host from the Cluster configuration. Remove Node from Cloudera To remove an Agent from a Cloudera deployment, first ensure that the Agent has no more Cloudera Services assigned, and is removed from any Cloudera Cluster. Go to Hosts All Hosts in Cloudera Manager, select the node which should be removed and click Actions for Selected (1) Delete. 129

131 After removing the node from the Cloudera Configuration, uninstall all Cloudera artefacts from that node using the Manager web console by clicking the red remove icon: 130

132 Settings The Platform Settings tab allows you to configure general Manager settings, authentication options and perform reboot and shutdown operations. System Admin System Admin allows you to: review the currently installed Manager version install Manager upgrade packs list system update history. All the applied updates after Manager 2.3 will be listed here. configure the Manager web service listening port configure the Manager web service with custom SSL key 131

133 download and restore Manager system configuration backup files Collect system information and generate system diagnostic file Collect hardware information and generate hardware diagnostic file 132

134 Custom SSL Certificate Custom SSL certificates can be used to comply with enterprise security policies. To use a SSL certificate from an external PKI, click Upload SSL Key in the Admin Web section. 133

135 Paste the Private Key in PEM (Base64 encoded DER certificate) format to the SSL Key field, and the certificate in the field below, again PEM formatted. The certificate supports Root and Intermediate Certificates of the related Certificate Authorities. In that case, pase the whole chain to the field with the correct order as shown below: 1. Root Certificate 2. Intermediate Certificate 3. Server Certificate Note Make sure that the passphrase is removed from the private key. Click Apply to install the certificates. The Manager web console will restart immediately and the new certificate will be presented. In some cases, it is necessary to refresh the browser window. For security reason, the following principles are recommended if you re willing to generate key pairs for this Gemini appliance: 2048 bit at least, 4096 bit would be great. Key pairs generated with AES256. Signed with SHA-2(SHA-256 or SHA-384), no SHA-1, no MD5 134

136 Information Information displays detailed software and hardware information of your Gemini appliance. Here you may review the currently installed software version of: Gemini Enterprise Manager(Appliance) Linux Kernel Java Detailed hardware information on your Gemini appliance includes: CPU Memory NIC Chassis 135

137 The Listen Port tab will list all the currently listening ports. This information are usually requested by the Gemini Support Team to assist with your support case. 136

138 The Audit Report tab allows you to create downloadable audit reports which included a list of all the libraries that Manager used with the version info and licenses. The current listening ports are also included in the reports. Authentication Gemini Appliance offers administration access either with the Gemini Enterprise: Manager web console or by running CLI commands using SSH. Manager Users To configure access to the Manager web console, configure as many local Admin Users as required. The password of Manager Admin Users need to comply with the Password Policy configured at a later stage. 137

139 User Permissions A User Role is a template of individual user permissions that control behavior and access across different areas. Several roles are provides by default. New roles may be created as needed with customized permissions as desired. Please note: If your Manager is upgraded from versions prior to Manager 2.4, all the existing users will be granted to Supervisor after upgrade, which has full permissions to all functions. The roles need be applied to each Manager user. In LDAP authentication, you also need to specify a default permission role for each LDAP resource. The role Supervisor and Manager User are default roles and cannot be removed. A role may be deleted only if there are no users assigned to that role. Once only read permission of one function is granted to a user, this use can read the status and settings in this function page, but it is forbidden to do any actions including input form and click buttons. Granting a Write permission also implies Read permission. 138

140 LDAP You may configure LDAP resources here to support LDAP authentication for the Manager web console. When LDAP resources are configured successfully and correctly, a user will be able to login to Manager with their LDAP account. Note that Manage supports simple BIND request and Search/Bind request to connect with LDAP server, and LDAP server access is only used for authentication not for accessing roles or permissions. How to configure a LDAP resource with Simple BIND: 1. Use the toggle button to enable LDAP Authentication. 2. Click Add LDAP Resource to create a new LDAP resource. 3. Configure the Host and Port, use FQDN but not IP address. Enable SSL if needed. 4. Select Simple BIND. 5. Configure the User BaseDN. The BaseDN should be able to locate the users who should access to Manage only. 6. Configure the Login Attribute. The login attribute should be a real attribute existed in the LDAP directory and can be used for Manager account user name, e.g. uid, CN or name. 7. Configure Role. Select a default permission role for new Manager user created during LDAP authentication. 139

141 Note Please ensure that your BaseDN includes only those users who should have access to the appliance administration screens. How to configure a LDAP resource with Search/BIND: 1. Use the toggle button to enable LDAP Authentication. 2. Click Add LDAP Resource to create a new LDAP resource. 3. Configure the Host and Port, use FQDN but not IP address. Enable SSL if needed. 4. Select Search/BIND. 5. Configure the Lookup DN and Lookup Password. This is used to login to LDAP server and fetch the LDAP trees. The whole LDAP trees will be cached on the system for further use. 6. Configure the User BaseDN. The BaseDN should be able to locate the users who should access to Manage only. 7. Configure the User Search Filter. 8. Configure the Login Attribute. The login attribute should be a real attribute existed in the LDAP directory and can be used for Manager account user name, e.g. uid, CN or name. 140

142 9. Configure Role. Select a default permission role for new Manager user created during LDAP authentication. Single Sign-on (SSO) Gemini Appliance Single Sign-on (SSO) provides the ability to use an HTTP Reverse Proxy Server to handle Manager authentication. Once a user successfully logged in to the proxy, they can seamlessly access the Manager web console without having to login again. 141

143 Manager expects a specific HTTP Request Header from the Reverse Proxy. The name of the HTTP Header field can be configured in the Single Sign-On configuration screen. Select the Automatically Create User option when the username from an authenticated request through the Reverse Proxy does not exist as a local Manager admin user. If this option is not selected and the username from the request doesn t exist in Manager, the request will fail and the Manager login prompt will be shown. For added security, authentication requests can be restricted to a specific set of IP addresses, and only requests having the Username Field in the HTTP Header. Once SSO is authenticated, it will bypass any other authentication methods such as LDAP. 142

144 Password Policy Password Policy allows you to enforce password requirements to meet your security needs, including password complexity and password duration. Note Password Complexity applies to both Web admins and OS users. Password Duration only applies to OS users. Proxy Proxy Settings allows you to configure a proxy server for specific services, e.g. download Cloudera artifacts. 143

145 Login Banner Enable and edit the banner message to present to users when accessing the appliance via console, SSH, or browser. 144

146 Reboot Allows you to reboot your Gemini appliance immediately. 145

147 Shutdown Allows you to shutdown and power off your Gemini appliance immediately. Note that Splunk services will be stopped prior to shutdown in order to prevent unexpected errors. Account Displays information on the session s currently logged in user. Profile 146

148 Update the current user s name, password, avatar, and preferred languages on the user interface. Manager supports multiple languages including English, German, Japanese, and Traditional-Chinese. Logout Immediately logs the current user out of the session. 147

149 CLI Commands Gemini Appliance supports a series of shell commands that can be executed locally from console (tty0) or remotely logged in using the sbox account over SSH. Type sbox for more details. Please note that the first time a command is executed, authentication using the sbox OS user and it s password is required: Command Reference To display all available commands, options and usage details for reference, use sbox help 148

150 149 Gemini Enterprise: Manage Administration Guide 2018 Gemini

151 Commands for initial setup Network Settings The sbox network command allows you to complete the basic network settings, including DHCP and static network settings. With this command you re able to build the basic network connection capabilities on your Gemini appliance. If you wish to configure the network with DHCP, use the following command: sbox network -nic <Network interface name> --dhcp 150

152 If you wish to configure the static network settings, use the following command: sbox network -nic <Network interface name> -ip <IP address> -netmask <Netmask> -gateway <Gateway IP> For example: sbox network -nic nic0 -ip netmask gateway This can configure the static network settings for network interface nic0. The Network interface name can be found by using ip addr command. Provisioning Beyond basic network connectivity, additional settings such as accepting the EULA, appliance hostname and timezone can be configured using sbox config alternatively to the web-based setup wizard: 151

153 Note that the license file has to be uploaded to the appliance first, before it can be applied. Gemini Cluster Membership To join an existing Gemini Cluster or to reset the membership, run sbox cluster with the required parameters. For example, sbox cluster --token displays the token string of this cluster node which is used by children who want to setup a membership to this node: Manager Web Administrations The command group sbox admin 152

154 provides manipulation of the Manager Web GUI, for example to reset a custom installed SSL certificate or to disable the web-based setup wizard when all settings have been applied using the CLI. Use these commands with caution as some of the restart the web GUI. 153

155 Commands for information gathering Manager Version To display the currently installed version of Gemini Manager, run sbox --version Model To acquire which model this appliance runs on, use sbox --model to display the string. On virtualized environments or on public clouds, the returned string represents the Hypervisor type. For example on Amazon EC2, HVM domu will be returned. Service Tag When reaching out to Gemini Customer support, it s useful to submit the unique service tag of this appliance which can be retrieved which the command sbox --service-tag Please make sure to include these details, such as Manager Version, Service Tag and as well as the Model number when opening a customer support request. Note that this information is automatically included in the Diagnostic Report created using the Manager web GUI. Installed Packages For support and audit purpose, a list of installed packages and their versions can be printed. 154

156 The command sbox admin --installed-packages lists installed Gemini packages. Service Status The status of Gemini components are displayed with sbox service --status Exposed Network Ports Some of the administrative components such as the Web GUI use a TCP port exposed to the connected network. To get a list of open ports and their exposure, run sbox service --listen-port 155

157 Note that the wildcard * character means that the related network port is open on all active network interfaces. If is shown in the Host column, it means that the port is not exposed to any external connected network and allows only Host-only communication. System Information This command will display hardware and software information. This is good for collecting system information for further use. sbox system --info 156

158 Commands for troubleshooting Password Reset To reset the Manager Web GUI admin users password, use sbox admin --reset-password This will unlock the account and set a random generated password. Password Change If you want to set a new password instead of using a random generated string, there s the option to use sbox admin -set-password <new_password> where <new_password> has to replaced with the desired token. It s recommended to change the admin password using the Web GUI in general, and to change it again after resetting it using CLI. Generate SSL Key When the Web GUI is unavailable to any certificate issues (e.g. expired or invalid certificate), reset the SSL certificate by running the command below: sbox admin --gen-ssl 157

159 Important: This will overwrite any custom private key and certificate installed using the Manager Web GUI. It is recommended to backup private keys and certificates using Manager s backup capabilities in the Web GUI before performing this operation. Cloudera Installation Reset To remove an existing Cloudera installation on a Gemini appliance completely, run sbox cloudera --undo Important: This will erase the Cloudera installation on this node completely. Gemini recommends to backup the Cloudera configuration and manually remove and redistributed any assigned Hadoop services in Cloudera Clusters from this node using Cloudera Manager. Gemini Cluster Reset Resetting Cluster settings will disconnect and remove related membership settings from this particular appliance. Additionally, the token used to connect children as well as any whitelist settings will be set to default. To perform this action, run sbox cluster --reset 158

160 Network Reset The sbox network --reset command allows you to reset network settings and remove IP bondings. After running this command, basic network settings are set to default and have to be configured again (see above for reference): Service Restart In some cases, Gemini Customer Support might ask to restart the system administration services. Additionally, this action can be performed if the Web GUI is irresponsive. To perform, run sbox service --restart which will immediately restart necessary services: Remove Splunk Instance The sbox splunk --kill command allows you to remove the installed Splunk instance entirely, including binary, configurations, and ingested data. All the configurations and data will be deleted. This is not a recovable action. Run it with caution. 159

161 Reset Splunk Cluster Manager In case if there re unintended situations happened to the Splunk Cluster Management in Manager, the sbox splunk --undo_manager command allows you to reset the Splunk Cluster Manager database. All the data and settings in the database will be removed, including node information and Splunk cluster configurations. All the existing Splunk configurations will be kept. Apply Patch In case if the Manage Web UI is unaccessible for applying patches, the sbox system -patch <patch_file> command allows you to apply a patch without relying on the web interface. Upload the patch file to the target instance, login with SSH, and run this command to apply the patch. 160

162 Independent Stream Forwarder Operations When this instance has been deployed as an Independent Stream Forwarder(ISF) in Splunk Environments, you may control the ISF services by leveraging the following commands. Run sbox isf --restart to restart the ISF service. run sbox isf --stop to stop the ISF service. And run sbox isf --log to read the application log. 161

163 Commands for system operations There are additional commands available, which are limited to interactive shells using the console (tty0). Note: The commands listed below related to System Operations are restricted from being used through SSH sessions! System Reboot To reboot the Gemini Appliance, just type reboot System Power Off The shut down the Gemini Appliance, just type poweroff 162

164 Default Passwords for CLI Operations Only three accounts are provisioned for command line login to the appliance. They are listed here along with their default passwords: OS account default password Description sbox facing jet function drive Used for Manager administration splunk think adventure kitchen chest Used for Splunk administration hadoop popular fully apple hello Used for Cloudera administration These users will be required to change their password upon logging in for the first time. 163