Gemini Enterprise: Manager Administration Guide V2.3

Transcription

1 Gemini Enterprise: Manager Administration Guide V2.3

2 CONTENTS Prerequisites System Initialization Localization License Activation Provisioning Bulk Provisioning Bulk Provisioning - Appliance Discovery Bulk Provisioning - Network Settings Bulk Provisioning - Hostname Bulk Provisioning - Change Admin Password Bulk Provisioning - Connect to LDAP Bulk Provisioning - SSH Authentication Bulk Provisioning - Summary Stand-Alone Provisioning - Join Cluster Stand-Alone Provisioning - Change Admin Password Success Initial Login Navigation Featured Platforms Integration Center Splunk Installation Cloudera CDH Installation Node System Time Timezone Name Hostname Local Hosts Network Bonding Port Redirect FTP FTP Service 1

3 FTP User SSH SNMP SNMP Service SNMP Agent SNMP Agent version 1 SNMP Agent version 2c SNMP Agent version 3 SNMP Trap Thresholds SNMP Trap Destinations Failover Creating a Failover Group Joining a Failover Group Log Receiver Storage Add NFS Mount Add CIFS Mount Add S3 Mount Monitoring Diagnostic Cluster Manage Nodes Manage Groups Execute Jobs Membership Settings Bulk Provision License License Status Remote Licenses Inventory License Server Splunk Daemon Web Interface Hunk Apps 2

4 Splunk Diag Optimizer Config Editor Splunk Versioning Command Cluster Management Manage Nodes Manage Clusters Cloudera Initial Cloudera Deployment Accessing and Using Cloudera Manager Add Node to Cloudera Remove Node from Cloudera Settings System Admin Custom SSL Certificate Information Authentication Admin Users OS Users LDAP Single Sign-on (SSO) Password Policy Proxy Login Banner Reboot Shutdown Account Profile Logout CLI Commands Commands for initial setup Commands for information gathering Commands for troubleshooting Commands for system operations Default Passwords for CLI Operations 3

5 4

6 Prerequisites Cabling - An ethernet cable and available ethernet switch connection. Networking - One IP address for Gemini appliance s ethernet port 1 (address, netmask and gateway are required for manual configuration. DHCP is supported) Client - A client PC with access the Gemini appliance s IP address defined above. Browser - Google Chrome browser is recommended. (optional) Access to the Splunk Enterprise installation tarball (splunk-*.*.*-*-linux-x86_64.tgz). (optional) Internet access is required for Cloudera CDH installation. 5

7 System Initialization The appliance configuration is performed using the Gemini Enterprise: Manager web console. Using a supported web browser, navigate to: <IP address from prior step> A security warning or message may be displayed. This is expected and does not indicate a problem. Depending on your browser you may have to choose proceed anyway or continue for the page to load. Upon the first visit to Manager, the End User Software License Agreement is shown. After reading the terms, choose Accept to advance to the next screen. 6

8 Localization Manager supports multiple languages. Setting a preferred language adjusts the entire user experience accordingly. If the hostname and timezone settings were skipped in previous basic configurations, this step provides an opportunity to configure them. 7

9 License Activation This step allows you to activate the appropriate license for your intended use. You can choose to activate the full Enterprise Edition which requires a pre-purchased license or the Free Edition, which requires no license. You may also opt for the third option of a 30-day trial of the Enterprise Edition which does not require a licence. If you chose to activate a pre-purchased Enterprise license, the next step will walk through the application process. You may either choose to retrieve the license from an existing license server or apply a new license from a file. 8

10 Applying a license from a file involves a three step process, outlined in this screen. It involves generating a request, submitting the file to Gemini Support and finally uploading the received license file to the appliance. 9

11 Alternatively, if a license server is used to manage all available licenses, selecting the Connect to a License Server will allow specification of license server information - IP Address and token - in order to perform the validation. 10

12 Provisioning If you have multiple appliances to be configured, select Bulk Provisioning to launch a wizard that allows configuration of multiple appliances at one time. The configuration includes all aspects of naming, network configuration, LDAP access and software installation. To operate as a standalone appliance, select Stand-Alone (single) provisioning. Bulk Provisioning The Bulk Provisioning Wizard is a step-by-step workflow that allows configuration of multiple appliances in one simple process. Before proceeding further, all other appliances that are to be configured, will need to be powered up and network-accessible. No other configuration steps are needed on these other appliances. 11

13 Bulk Provisioning - Appliance Discovery This steps discovers and confirms the entire set of appliances that are to be configured. The discovery process can be done in one of two ways: If the IP addresses of all appliances are known, and available in a text file, it can be uploaded to launch the discovery process. The file will need to contain one IP address per line. Alternatively, the appliances can be discovered on the network by performing an IP subnet scan, using a CIDR notation to specify the subnet. E.g /24. Please note that the scope of the subnet will determine the length of time taken for the scan to complete. 12

14 Bulk Provisioning - Network Settings This step configures the mechanism used for assigning IP addresses of the appliances being configured. The default option Network settings - Static assigned uses dynamic assignment using a DHCP Server. Alternatively, choosing Network settings - Static assigned will utilize DHCP for the initial assignment only, but then use that IP as the static setting from that point onwards. This is useful when building a DHCP server for deployment temporarily, and the DHCP server is not needed after that step. 13

15 Bulk Provisioning - Hostname This step provides multiple options for automatically assigning a hostname to each appliance being provisioned. If DNS records have been assigned for each appliance, the Use Reverse DNS Lookup option uses the hostname discovered through the DNS server. Alternatively, the Specify Custom Pattern option allows the specification of a custom string pattern composed of text and certain allowed tokens, to automatically compose a hostname. The following tokens are allowed: $service_tag$ - is replaced with the service tag of the appliance, as indicated on the box or available within Gemini support. $increment$ - is an automatically incrementing number 14

16 Bulk Provisioning - Change Admin Password This step is used to specify the password for the admin account on each appliance being configured. It is recommended that you use a strong password or if applicable, follow appropriate password security policy as required for your enterprise. Please note all the appliances will be updated with the same admin password. 15

17 Bulk Provisioning - Connect to LDAP If an LDAP account is to be required for access to each appliance, this step allows the specification of LDAP server authentication information. This will be used to authenticate users upon login to the appliance. Read LDAP Authentication in Settings chapter for more details. Please note that LDAP authentication is optional, and by default the appliance resorts to local account-based authentication. 16

18 Bulk Provisioning - SSH Authentication This step allows configuration of SSH access on each appliance being configured. It allows the specification of a password for SSH access, or the upload of an SSH key to complete the key exchange and bypass the password. Please note this step is optional and can be skipped. 17

19 Bulk Provisioning - Summary This summary screen lists all the appliances about to be provisioned along with configuration settings. It should be used as a final review and confirmation step before starting the automated configuration step. it is also strongly recommended that the a CSV file be downloaded for future reference. Clicking on Start initiates the automated provisioning process, which can take several minutes or longer depending on the number of appliances being provisioned. The status of each appliance is updated in real-time. After all appliances have been provisioned, clicking on Finished completes the process. 18

20 Stand-Alone Provisioning - Join Cluster To include this node into an existing Manager Cluster, select Join an existing appliance cluster and provide the IP Address and the Token String that were assigned to the Master Node. To operate as a standalone appliance, or to configure a cluster at a later time, select Operate as a standalone appliance. 19

21 Stand-Alone Provisioning - Change Admin Password Update the password for the account admin in Manager here. It is recommended that you use a strong password or if applicable, follow appropriate password security policy as required for your enterprise. 20

22 Success Congratulations! The Completed screen lets you know that this appliance has been configured. Click Get Started to launch Manager. Return visits to this page will proceed directly to login. Configured settings may still be changed within the corresponding areas within Manager. 21

23 22

24 Initial Login Upon completion of the setup process and clicking on Get Started in the previous section, you will be presented with the login screen. Log in to Manager with the username admin and the newly changed password from the previous step. 23

25 Navigation The Home Screen is the first and default screen shown upon successful login. The navigation bar on the left provides easy access to different areas of management, as described in the rest of this guide. The entire Manager experience is organized into different areas accessed by the vertical navigation bar on the left, and are as follows. HOME - This is the default home page upon login that provides access to setup and manage different platforms such as Splunk, Hadoop and other applications. NODE - Provides configuration settings for such as networking, identity, and preferences for a specific appliance CLUSTER - Allows management of appliances as part of a multi-node cluster. LICENSE - Allows centralized management of Manager licenses. SPLUNK - Allows for the configuration of Splunk operating parameters. This section is only available once the Splunk configuration has been completed. It appears when Splunk Enterprise is installed. 24

26 HADOOP - Allows for the configuration of Cloudera cluster management. This section is only available once the Cloudera configuration has been completed. It appears when Cloudera is installed. SETTINGS - This section provides configuration settings for Manager itself such as system update, authentication, security settings, and backup/restore. ACCOUNT - This section allows other configuration of user information and account level functions such as password reset. Featured Platforms The featured platforms are data processing platforms which are available for activation and usage on the appliance. Clicking on Activate will allow easy deployment of a platform with a few clicks and an easy process. Please note that each platform consumes resources. Installed and running more than one platforms in a Gemini appliance may result in resource competition. Ensure that your appliance storage, CPU and configuration are suited to the workload imposed by each platform. Please consult with Gemini Support on sizing considerations. Integration Center The Integration Center is a repository of applications that provide insights, management or connectivity to data sources. Some apps mirror those provided within other platform ecosystems such as SplunkBase, while others are specific to Gemini. Regardless of origin, full support is provided for all solutions. 25

27 Each solution or application is represented by a descriptive card that allows easy installation, or if already installed, easy configuration or removal. The option buttons and configuration screens pertaining to each application are specific to each application. 26

28 Splunk Installation This process ensures proper installation and configuration of Splunk. Note that the appliance does not ship with the Splunk binaries, and a download from the Splunk site is required, as part of this process. Before proceeding, ensure that you have the following ready: An active Splunk account to download the Splunk installation file The current Splunk Enterprise for Linux binary (.tgz) Splunk Enterprise Licence If you don t have a Splunk account or haven t downloaded the latest installation file, instructions and links are provided within the Manager Install Splunk Enterprise installation page. Note Apple/Safari users, you will want to ensure you have disabled Open Safe Files after downloading to retain the.tgz extension. From the Manager Home Screen, start the Splunk installation process by clicking the Install button located under the Splunk Enterprise banner. Alternatively, select Splunk from the toolbar on the left side of the page. From the Upload Splunk Tar File panel, click the Upload & Install button to begin the installation process. In the Upload and Install subpanel, click the...choose the tarball 27

29 button then select your previously downloaded binary of Splunk Enterprise for Linux. You will see the name of the selected file appear in the installation subpanel. Click the Upload button. When the file has been uploaded, its status is displayed in the subpanel. Click the Install button and accept the Splunk Software License Agreement. The Splunk Enterprise software will now be installed on your Gemini Appliance. 28

30 When completed, the Manager Home page displays the currently installed version of Splunk and its status. 29

31 Cloudera CDH Installation The Cloudera CDH is Cloudera s software distribution including Apache Hadoop and related additional components. A full list of available pacakges can be found here: Gemini Enterprise Manager installs prerequisites and the required Cloudera software, such as the Cloudera Manager and Cloudera Manager Agents, across a deployment of Gemini Appliances. More information about Cloudera Manager can be found at Note To run a production-ready Cloudera Cluster, the Cloudera Manager and some of the Apache Hadoop components require an external database. For more details and system requirements, please cosult the Cloudera Documentation at s/cm_ig_installing_configuring_dbs.html#cmig_topic_5_1. Cloudera Setup on Manager requires at least two Gemini Appliance instances, one Cloudera Manager and up to 80 Agents. During installation, the appliance hosting the Cloudera Manager will download required software packages from the official Cloudera Mirror ( ) and share them to connected Manager nodes which will be used as Cloudera Agent. 30

32 To begin the Cloudera CDH Installation on Gemini Appliance, log in to Manager on the appliance node which is the designated Cloudera Manager and activate the Cloudera platform from the HOME screen. Once activated, follow the detailed instructions in the Cloudera chapter in this documentation. 31

33 Node The Node tab is the starting point for the configuration of the host and server functions related to the Gemini appliance. System Time Accurate timekeeping is vital to ensure correct event order. Distributed environments may become out of sync and transactional searches may return inaccurate information if time is not accurately set and frequently updated. 32

34 Manager uses pool.ntp.org as a time source by default. Additional network time sources, either external or internal may be added by clicking Add NTP Server Setting the NTP Sync toggle to the OFF position will halt further network time updates and allow for manual editing of the system time. This may be required under special circumstances, but is not advised for general operations. Click Set Time to set the datetime manually, or click Sync with Browser to update the datetime settings with client PC. 33

35 34

36 Timezone Accurate timezone configuration is essential in maintaining event-order integrity, particularly in geographically distributed environments. 35

37 Name Hostname To prevent conflicts in distributed environments as well as declare the source path of received events, Manager requires that each device has a unique hostname. Splunk uses this hostname as a default value to populate both server.conf and inputs.conf when started for the first time. Local Hosts While not required in normal operation, manually configuring local hosts can ensure connectivity between hosts in the absence or failure of a DNS server. In the case of high latency DNS servers manually configuring hosts may improve performance. Manually configuring local hosts is not considered best practice and should only be used in exceptional cases as multiple static configurations can be complicated to manage and easily become out of date. Note DNS settings should be configured separately in each network interface in the Network tab. 36

38 To add a static host, click Add New Record and specify the host s IP address and hostname. 37

39 Network The Manager Network Interfaces configurations may be reviewed and edited here. Manager supports multiple network interface cards (NICs) and Gemini appliances contain four or six NICs. Additionally Bonding and Port Redirects may also be configured here. 38

40 Each NIC may be configured with an IP address either manually or via DHCP. Static route rules may be added to a specific network interface to communicate with networks which do not directly connect to the Gemini appliance. 39

41 Bonding Manager provides support for Link Aggregation. Here you can bind multiple physical NICs into one Virtual interface to increase throughput beyond what a single connection can sustain while providing redundancy in the event of a single NIC failure. Select the Physical Network Interfaces to be included in the Bonding group. Select the Mode for load balancing and fault tolerance from the drop-down menu. Options include: Round Robin Active-Backup XOR Broadcast IEEE 802.3ad Dynamic Link Aggregation Adaptive Transmit Load Balancing Adaptive Load Balancing. MII is used to verify the status of the NIC. Specify the frequency of MII link monitoring by entering a value in milliseconds in Millisecond Monitor. The default value is 100. Then click Add. 40

42 Once created, the Virtual Interface is shown in the list You can edit the configuration of the newly created bond interface: 41

43 Port Redirect As with any other application running as a non- root user, Splunk will be unable to bind to and listen on the privileged ports, those which are less than Port Redirect allows you to define rules for Manager to redirect incoming connections on privileged ports to a port above Enter the Source port and the Destination port. FTP Enabling the FTP service allows for the uploading of data to the /opt directory. Generally this is performed when hosts are not able to have a Splunk Forwarder installed locally, allowing them to move data to a host such as a Gemini appliance that can ingest the data correctly. The FTP service requires the configuration of an FTP user (see below) as well as an entry in inputs.conf instructing Splunk to ingest any new data found. Note the FTP protocol is not natively encrypted and should only be used when security practices allow for it. 42

44 FTP Service Use the FTP Service toggle and select the desired port (Manager defaults :2121) to enable FTP. 43

45 FTP User The FTP protocol requires both user credentials and a directory to store received files. Manager installs with a default FTP user named splunk with the home directory of /opt/sbox. Click Add FTP User and provide the desired username, password and root folder to add subsequent accounts. Click on an existing username to change it s root folder or password. 44

46 SSH The SSH service provides remote command line access to Manager. SSH is enabled by default for the users sbox and splunk. The SSH protocol is natively encrypted. Toggle SSH Service to start or stop the service. When you enabled Forward SSHD Log, Manager will forward a copy of SSHD logs to /var/log/sshd/sshd.log for further use. Note Please refer to CLI commands chapter for default passwords. It requires you to change password when the 1st time login to Manager. 45

47 SNMP SNMP Service The SNMP service allows a remote Simple Network Management Protocol enabled host to interrogate the Gemini appliance for monitoring and alerting data. The SNMP service requires the configuration of an SNMP Agent (below). To enable SNMP toggle SNMP Service. SNMP Agent The SNMP Agent requires that the SNMP Service (above) be enabled to function. Select CREATE SNMP AGENT to create a new SNMP Agent entry. Multiple SNMP Agents can be configured. Select a unique name for the SNMP Agent entry and choose an SNMP Agent version (see below). 46

48 SNMP Agent version 1 SNMP Version 1 is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure versions are not possible. SNMP v1 supports a maximum of 32 bits per counter. Input the NETWORK and MASK BITS (Subnet Mask) of the host network. Enter a COMMUNITY STRING for SNMP authentication. The default is public - changing this is strongly recommended. 47

49 SNMP Agent version 2c SNMP Version 2c is not encrypted and authentication happens in plain text. As such, it should only be used when other, more secure versions are not possible. SNMP v2c supports a maximum of 64 bits per counter. Input the NETWORK and MASK BITS (Subnet Mask) of the host network. Enter a COMMUNITY STRING for SNMP authentication. The default is public - changing this is strongly recommended. 48

50 SNMP Agent version 3 SNMP Version 3 supports authentication, encryption and 64 bit counters. Select which password hashing method to use under AUTHORIZATION ALGORITHM. Manager supports either MD5 or SHA. Enter the authentication password. Select an encryption method for SNMP communication using ENCRYPTION ALGORITHM. Manager supports DES and AES128. AES128 is considered to be the more robust of the two. Enter the encryption password and select ADD to add the agent. 49

51 SNMP Trap Thresholds Enable SNMP Traps for Gemini appliance s performance data and specify the frequency and threshold. SNMP Traps may be enabled for: CPU Disk Link Memory Process - ftp, splunk, ssh, syslog-ng. 50

52 SNMP Trap Destinations Provide information for the SNMP manager: Enter the IP address of your SNMP Host Select a protocol: use trapsink to send SNMPv1 traps use trap2sink to send SNMPv2 traps use informsink to send inform notifications Enter a Community String Enter a Port over which to send the information. 51

53 Failover Gemini appliance allows users to create Failover groups. Manager nodes configured in a Failover group provide for a high level of availability in the event one of the nodes become unavailable due to a maintenance window, network outage, or other event. Each group has one active master peer to hold a virtual IP and several standby slave peers that are ready to take over for a failed master. Each Gemini appliance can be part of different Failover Group and each group should be provisioned using a different port number. Creating a Failover Group Under the Node Manager menu, click on Failover and then click Create New Failover Group to create a new group. Virtual NIC - IP Address - Assign a IP to this virtual group that is reachable from existing network interfaces. Please note this IP must be reachable from every member within the group. Monitor - Determine what event will trigger a hand over to another group member. For example, if Detect Splunk was selected, when Splunk running on the master node went down, one slave node will step in and take over the master role. Add node to this group - You may directly assign group members here and member nodes will join into this group automatically. 52

54 Joining a Failover Group In the Failover page, click Join Existing Group and enter the IP of Virtual NIC assigned on the master node within the group: Once joined, the Failover group will display all members and their role within the system (Master or Slave): If the Master node becomes unavailable or detected the monitoring events, the Slave node will step in and serve the virtual IP which used to be managed by the Master node. 53

55 Log Receiver Enabling the Syslog service allows Gemini appliance to receive events from remote devices using the commonly used Syslog protocol. Source filtering rules can be specified to direct how to parse, split and direct different components into different destinations. Configuring a rule involves the following steps. 1. Create Rule - Creates a syslog receiver rule. You can create multiple rules. 2. Source - Data source indicates the source location of the data. a. Specify the protocol and port to receive syslogs. b. Each rule can only utilize a single data source. c. Logs will be stored at the designated destination. d. Multiple destinations are allowed. 54

56 3. Add Filter - Specify a filter for this rule. Select it in the drop down menu from the associated data source. a. This step is optional. b. Filter is used to parse and split log entries with defined types. Only matched logs will be stored into defined destination. 55

57 4. Add Destination - Specify a destination for this filter. Select it in the drop down menu from the associated filter. a. Destination is used to indicate where to store the received log data. b. There must be at least one destination for each filter. 5. Add Destination - If there are no filters created, specify a destination for this source. Select it in the drop down menu from created data source. a. There must be at least one destination in each rule. b. Multiple destinations are allowed for each data source. 56

58 6. Save Rule - Once configuration of source, filter and destination are complete, configured, click on Save Rule to save the rule settings. If there are more than one data sources, create rules as needed for each data source, and then specify filters and destinations. 57

59 Storage The Storage section allows administrators to monitor the physical hard drives and mount NFS, CIFS, S3, and DAS (Direct Attached Storage) volumes to extend the capacity of Splunk storage and allow for reading files from network storage. This allows the volume of an existing system to be extended and the mount point for Splunk indexes may also be defined. 58

60 59

61 Add NFS Mount To define an NFS Mount Point: Enter the local mount point (located at /opt/sbox/data folder), Enter the IP address of the remote server Enter the remote folder (starting with a leading /). Select the mount type. Hard mount is recommended by Splunk when the mount point is used for cold buckets. Click the Add button to add the new NFS mount. Note A mount point will not be detected and validated until you enabled the configuration by clicking the mount button. Once enabled, Manager will automatically mount the NFS Mount Point upon boot. 60

62 Add CIFS Mount To define a CIFS Mount Point: Enter the local mount point (located at /opt/sbox/data folder) Enter the IP address of the remote server Enter the remote folder (starting with a leading /) Enter the Username Enter the Password Click the Add button to add the new CIFS mount. Please note that a mount point will not be detected and validated until you enable the configuration. When enabled, Manager will automatically mount the CIFS Mount Point upon boot. 61

63 Add S3 Mount To define an Amazon S3 Mount Point: Enter the S3 bucket name you want to mount and the local mount point will locate at /opt/sbox/data/s3/<bucket name> folder. Enter the IAM Access Key ID Enter the IAM Secret Access Key If you want all the data stored in the S3 bucket are encrypted, enable Server-Side Encryption(SSE) and select a proper key option. To get your S3 Access credentials, log in to the AWS Console, open the Users section in the IAM Service and click the desired user. Create an Access key in the Security credentials tab. Please note that access to S3 storage requires a connection to the public internet from the node. Note S3 is designed for data archival and not applicable to Splunk indexing. Specify hot/warm/cold buckets to S3 mount mounts will cause Splunk malfunctions. 62

64 Monitoring Enabling Monitoring allows Manager to send the output of its own Admin and System logs to the destination of your choosing, either local storage on the Gemini appliance or a central syslog server. To create a file locally, enter the name of the file to be output (e.g., admin_file.log ). To send a file to a syslog server, select the Destination Protocol, UDP or TCP, and enter the IP address of your syslog server. The service defaults to port 514, which you may also customize. 63

65 64

66 Diagnostics The Diagnostics Panel provides access to useful network tools without the need to access the command line interface. The following commands can be executed and and the resulting output shown in the window. Ping TCP Connect NS Lookup Traceroute TCP Dump 65

67 66

68 67

69 68

70 Cluster The Gemini Cluster tab is the starting point for managing distributed configuration options. Here you can view your distributed topology, connect the current node to a parent or manage child nodes belonging to the current node. Manage Nodes Manage Nodes provides a tabular view of all Gemini appliances in your deployment. Here you can view key information on each host including: IP Address Parent number of joined groups number of dispatched jobs. You can also drill into each host and assign it to a Node Group. 69

71 70

72 Manage Groups Manage Groups allow you to create logical categories of your Manager nodes. Node Groups may be used to view a subset of your Manager nodes as well as assign jobs to specific Manager nodes. Child Nodes may be assigned to multiple Node Groups. 71

73 Execute Jobs Execute Jobs allows you to easily execute predefined tasks including starting/stopping/enabling/disabling services and retrieving and updating system parameters to multiple Manager instances and execute them at a specific time. Jobs can be assigned to all nodes or previously defined Node Groups. 72

74 73

75 Membership Settings Membership Settings allows you to enable and configure registration of your Manager nodes to this Manager. Manager Nodes which are registered may be monitored and managed from this device. Here you may also configure the Token String to be used by the remote node for registration. You may restrict the nodes allowed to register by creating a Whitelist using either IP addresses or Hostnames. By default, all nodes are allowed to register. Multiple entries must be separated by a comma. Bulk Provision The same the option provided during initial configuration. This is a step-by-step wizard to guide you complete the initial setup configurations. Refer to Bulk Provisioning in System Initialization for detailed configuration steps. 74

76 License The Manage License tab allows you to configure your Manager License Servers and License Agents. License Status License Status presents you the current active license, including license type, volume, and expiration date. By clicking the current license in Product field, system will display detailed license status. If you have a pre-purchased license, you can request and then attach your license by following the steps in this page. Remote Licenses Remote Licenses allows you to manage the license sources, including remote license servers and from local. 75

77 Inventory Inventory lists all the license attached and allows you to manage Gemini licenses, and for convenience, groups volumes in total and licenses used. 76

78 License Server License Server allows you to enable and configure your Gemini appliance as a Manager license server. Manager license server manages licenses and grant permissions to other nodes who registered and connected to this license server. Here you may also configure the Token String to be used by the remote nodes for registration. You may restrict the nodes allowed to register by creating a Whitelist using either IP addresses or Hostnames. By default, all nodes are allowed to register. Multiple entries must be separated by a comma. 77

79 Splunk The Splunk tab contains various areas of management related to your Splunk installation. It allows you to perform common tasks, edit configuration files, and manage Splunk applications. Daemon Allows you to review and modify settings related to Splunk Enterprise s splunkd process without using the command-line. Here you may stop or restart Splunk; upgrade Splunk; reset the Splunk Admin password; enable or disable boot-start; and review and modify advanced configurations. 78

80 Web Interface Allows you to review and modify settings related to Splunk Enterprise s Web Interface, Splunk Web. Here you may disable or enable Splunk Web; launch Splunk Web in your browser; and review and modify advanced configurations such as enabling encryption and configuring the default port. 79

81 Hunk Splunk Analytics for Hadoop, formerly known as Hunk, provides seamless search and report functionalities on data stored in HDFS. As a minimum, Hunk requires HDFS with one NameNode and at least one DataNode, as well as MapReduce or Yarn (recommended). The Hunk section in Manager allows you to configure the required, so-called Hadoop Providers to connect to HDFS and verify the required permissions (optional). Specify a Provider Name to identify the configuration later back in Splunk. The HDFS NameNode FQN requires a URL in the format hdfs://<name_or_ip_of_namenode>:<port>/ The port is optional and defaults to To operate, Hunk stores a handful files on HDFS in an intermediate directory. First, create the path on HDFS per Search Head to be connected to HDFS manually and define read/write permissions for the splunk user (Splunk Enterprise on Gemini Appliance runs as non-privileged user splunk, so as all HDFS operations performed by Hunk). Hunk in Manager will also automatically create additional parameters to the Virtual Index Provider in order to run Hunk. Review the configuration manually in /opt/splunk/etc/system/local/indexes.conf. After successfully adding the provider, go to Splunk Enterprise administration, make sure that the Splunk Analytics for Hadoop license is installed and click Settings Virtual Indexes to add a new Virtual Index. Use the provider created in the wizard above. 80

82 Apps Apps provides a list of all Splunk Apps currently installed on your Gemini Appliance. Each App may be downloaded to your desktop as a tarball file. Drilling down on the App directs you to the App s directory listing in the Splunk Conf Editor. 81

83 Splunk Diag Splunk diagnostic allows you to quickly create a Splunk Diag file [./splunk diag] on-demand. Multiple copies of Splunk Diag files may be stored for later retrieval and download. 82

84 Optimizer Allows you to select a predefined Splunk role for your Gemini appliance automatically updating all the conf_files for you. Options include: Splunk Default Indexer Heavy Forwarder Search Head All In One. 83

85 Config Editor The Conf Editor allows you to edit, create, or upload Splunk configuration files within the $SPLUNK_HOME/etc/ file path from the convenience of the Manager web console. The editor provides file path navigation links and complete versioning of all file revisions. 84

86 Splunk Versioning Enabling the Splunk Configuration Repository allows you to manage changes, and retain multiple versions of configuration files. This provides for roll-back capability after making changes. 85

87 Command Splunk Command allows you to issue Splunk commands directly from your browser. Additionally, the Splunk Command Helper provides for easy, interactive building of complex Splunk commands which may then be issued within the browser. 86

88 Cluster Management This Splunk Cluster Manager will allow you to create a Splunk cluster with ease. In Nodes tab, you can manage all the nodes for Splunk clusters. In Clusters tab, you can manage to create Splunk Indexer Cluster and Splunk Search Head Cluster. Prerequisites: If you want to create a Splunk Indexer Cluster, the following conditions must be met: A Splunk Master Node must be assigned in Node tab. At least 2 indexer must be assigned in Node tab. If multi-site clustering is enabled, there must be at least 2 indexers in each site. There must have enough nodes in Nodes tab. If you want to create Search Head Cluster, the following conditions must be met: A Splunk Indexer Cluster must be created. It will be used when creating a Search Head Cluster. A Splunk Deployer must be assigned in Node tab. At least 3 Search Head must be assigned in Node tab. A standard configuration procedure would be the following: 1. Add Nodes. 2. Specify role and site for each node. 87

89 3. Create a Splunk Indexer Cluster with defined roles and sites. 4. Create a Splunk Search Head Cluster with defined roles and sites. 88

90 Manage Nodes Nodes can be added in 2 ways: Bulk Provisioning. All the bulk provisioned nodes will be added automatically. Manually add. You may add other Manager nodes manually. Note that the Manager version must be the same with the host running cluster management. Before creating clusters, the following information in each node must be defined: Role. There are 4 Splunk roles can be defined: Master Node Indexer Search Head Deployer Site. Define where your nodes are located. Select Site 1 for each node if only one site exists. A node without a Splunk role defined can not be used in creating Splunk clusters. 89

91 90

92 Manage Clusters There are 2 types of Splunk clusters that can be created: Splunk Indexer Cluster Splunk Search Head Cluster Refer to Splunk documentation and architecture Best Practices to understand how clustering works in Splunk and the significance of the parameters used in configuration. Steps to Create a Splunk Indexer Cluster: Create Cluster - Select Indexer Cluster. Specify parameters for this cluster: Name - The name of this cluster, space and special chararcters are not allowed. Secret - Used in Splunk cluster communications to identify if this node is a cluster member. Splunk Admin User Password - The Splunk admin password configured for each node in this cluster. The Splunk default password changeme cannot used here. Replication Factor Origin. Specify how many indexed raw data copies in a cluster in each site. Set to 2 or higher for redundancy. Replication Factor Total. This value must be equal or larger than the Replication Factor Origin. Search Factor Origin. Specify how many metadata copies in a cluster in each site. Set to 2 or higher for redundancy. Search Factor Total. This value must be equal or larger than the Search Factor Origin. 91

93 Upload Splunk Enterprise Software. The uploaded Splunk binary will be deployed into each cluster member. Click SAVE to create this cluster. Add nodes for this cluster. Click Add Node to select which node should be added. Only nodes defined with Master Node or Indexer can be selected. Only one Master Node can be selected in each cluster. Deployment. When all the nodes are set, click Deploy to start the deployment. Splunk will be installed on all the nodes and configured with pre-defined settings. 92

94 Once each node in the cluster is deployed, the status is updated to a lock icon indicating success. Re-initiating this process is not allowed. Scale this cluster - Scaling and expanding the cluster is done from this screen simply by adding additional nodes to the cluster and selecting Deploy. Remove nodes from cluster - Similarly, nodes can be removed from a cluster individually. Removed nodes can be added into another cluster. Delete Cluster - A cluster can be deleted entirely, and will free up the nodes, allowing their inclusion into another cluster. Steps create a Splunk Search Head Cluster: Create Cluster - Select Search Head Cluster. Specify parameters for this cluster: Name - The name of this cluster, space and special chararcters are not allowed. Select Master Node - Select an existing Master Node from a deployed Indexer Cluster. Secret - Automatically populated when a Master Node is selected Splunk Admin User Password - Automatically populated when a Master Node is selected Replication Factor - Specify the number of data copies in this cluster. Set to 2 or higher for redundancy. Upload Splunk Enterprise Software. The uploaded Splunk binary that will be deployed on each cluster member. Click SAVE to create this cluster. 93

95 Add nodes for this cluster. Click Add Node to select which node should be added. Only nodes defined with Search Head or Deployer(SHD) can be selected. Only one Deployer can be selected in each cluster. Once all the nodes are defined and listed, selecting Deploy starts the process of implementing the cluster on each member node. The specified version of Splunk will be installed on all the nodes and configured with the specified settings. 94

96 Once each node in the cluster is deployed, the status is updated to a lock icon indicating success. Re-initiating this process is not allowed. Scale this cluster - Scaling and expanding the cluster is done from this screen simply by adding additional nodes to the cluster and selecting Deploy. Remove nodes from cluster - Similarly, nodes can be removed from a cluster individually. Removed nodes can be added into another cluster. Delete Cluster - A cluster can be deleted entirely, and will free up the nodes, allowing their inclusion into another cluster. 95

97 Cloudera Initial Cloudera Deployment Before deploying Cloudera to Gemini Appliance, please activate the Cloudera platform from the Integration Center as described in section 5 Cloudera CDH Installation of this document. It also helps to have a general understanding of Cloudera capabilities and usage. To continue, add the Full Qualified Domain Names (FQDN) of the Gemini Appliances where Cloudera Manager and Cloudera Manager Agents should get installed to: Add as many agents as required by clicking the Add Agent button. Everytime another FQDN is added to the table, Manager will confirm proper DNS resolution and connectivity to the host. A green check indicates success on both fronts, and a red icon indicates a failure in either or both. Additionally, the setup verifies if the entered FQDN for the Cloudera Manager matches the appliance where the administrator is currently logged in to. Also, at least one additional Cloudera Agent is required to continue the setup. If Manager cannot resolve the FQDN using a configured Name Server, a red icon indicates that the IP Address needs to be added manually. When entering the IP Address manually, Manager will again check the connectivity and update the status icon. 96

98 Example 1: Manager cannot resolved mycustomfqdn, admin is required to enter IP Address manually. Example 2: Manually entered IP Address is incorrect. Example 3: Manually entered IP Address is correct and a valid Manager installation is detected. When all prerequisites are met, the Deploy button is activated and the installation can start. Once deployment has started, the horizontal bar indicates the progress and the current step of the installation. This stage of the installation runs in the background, allowing the administrator to work on other areas while it is in progress. Returning to the Hadoop page will show the current status and allow continuation when the installation is complete. 97

99 Successful installation may take up to 20 minutes. The page reloads automatically once completed. Open the Cloudera Manager web interface with the equivalent button or using the URLs respectively and follow the instructions to perform a Cloudera Cluster setup. If any unexpected error occurs during the installation, the setup will stop and show an error message. To restart the setup, reset the Cloudera installation on each appliance separately using the CLI command sbox cloudera --undo : 98

100 Accessing and Using Cloudera Manager The Cloudera Manager web interface is available at respectively for SSL secured access. The default username and password for Cloudera Manager are both admin. Make sure to accept the Cloudera End User License Terms and Conditions before proceeding: 99

101 After choosing the installation type (Cloudera Express, Cloudera Enterprise or Cloudera Enterprise Trial), select the hosts where the Cloudera Agents have been deployed to by Manager in the following screen: 100

102 Choose Use Parcels (Recommended) as method and the latest supported CDH version (Currently: CDH 5.10). If required, choose any additional parcels: The Cloudera Cluster Installation will now distribute all required parcels from the Manager to the Agents using a temporary mirror of the Cloudera Archive on the Cloudera Manager appliance. Please wait until all tasks are finished. 101

103 After passing the Host Inspector, finish the Cluster Installation and proceed to the Cluster Setup. Choose which Hadoop Services that you wish to use on your cluster from the list. To use Cloudera in conjunction with Splunk Analytics for Hadoop (see chapter Hunk in this guide), choose Custom Services with Services Types HDFS and YARN : Customize the the Role Assignments based on your requirements before proceeding to step 3: 102

104 In step 3, select Use Custom Databases for production deployments and enter the connection details: After the Database Connection test was successful, proceed to step 4 and update the Cloudera Setup paths according to the scheme in the table below: 103

105 Original Path /opt/dfs/... /var/lib/... /opt/yarn/... /tmp/... /var/log/... /var/run/... Path on Manager /opt/cloudera/dfs/... /opt/cloudera/lib/... /opt/cloudera/yarn/... /opt/cloudera/tmp/... /opt/cloudera/log/... /opt/cloudera/run/... The wizard will now run the Cloudera setup commands: 104

106 Once finished, the Cloudera Cluster has been set up and can be administered using the Cloudera Manager. Add Node to Cloudera The Deployment of Cloudera CDH on Gemini Appliance can be extended with additional Cloudera Agent installations to new appliance nodes. To perform this action, open the Hadoop section from the Manager web console and click on Add Node for every additionally required node. Manager will again perform validation checks, such as DNS Resolution and Connectivity, and indicate the result with green check marks or red icons. If all checks are successful, start the installation by clicking Deploy. After the installation, the new node will be propagated automatically to Cloudera Manager where it can be added to existing Cloudera Clusters. For that, login to the Cloudera Manager, go to Hosts All Hosts and click Add New Hosts to Cluster. If asked, click Classic Wizard in the next step and continue. Switch to the Currently Managed Hosts tab, where the newly added Gemini Appliance appears: 105

107 Check all hosts which you like to add to the Cluster and click Continue. The Cloudera Manager will now distributed required parcels to the new hosts. After passing the Host Inspector, optionally choose a Host Template to be applied and finish the wizard. You might now redistributed the Roles to the new host from the Cluster configuration. Remove Node from Cloudera To remove an Agent from a Cloudera deployment, first ensure that the Agent has no more Cloudera Services assigned, and is removed from any Cloudera Cluster. Go to Hosts All Hosts in Cloudera Manager, select the node which should be removed and click Actions for Selected (1) Delete. 106

108 After removing the node from the Cloudera Configuration, uninstall all Cloudera artefacts from that node using the Manager web console by clicking the red remove icon: 107

109 Settings The Platform Settings tab allows you to configure general Manager settings, authentication options and perform reboot and shutdown operations. System Admin System Admin allows you to: review the currently installed Manager version install Manager upgrade packs list system update history. All the applied updates after Manager 2.3 will be listed here. configure the Manager web service listening port configure the Manager web service with custom SSL key 108

110 download and restore Manager system configuration backup files Collect system information and generate system diagnostic file Collect hardware information and generate hardware diagnostic file 109

111 Custom SSL Certificate Custom SSL certificates can be used to comply with enterprise security policies. To use a SSL certificate from an external PKI, click Upload SSL Key in the Admin Web section. 110

112 Paste the Private Key in PEM (Base64 encoded DER certificate) format to the SSL Key field, and the certificate in the field below, again PEM formatted. The certificate supports Root and Intermediate Certificates of the related Certificate Authorities. In that case, pase the whole chain to the field with the correct order as shown below: 1. Root Certificate 2. Intermediate Certificate 3. Server Certificate Note Make sure that the passphrase is removed from the private key. Click Apply to install the certificates. The Manager web console will restart immediately and the new certificate will be presented. In some cases, it is necessary to refresh the browser window. For security reason, the following principles are recommended if you re willing to generate key pairs for this Gemini appliance: 2048 bit at least, 4096 bit would be great. Key pairs generated with AES256. Signed with SHA-2(SHA-256 or SHA-384), no SHA-1, no MD5 111

113 Information Information displays detailed software and hardware information of your Gemini appliance. Here you may review the currently installed software version of: Gemini Enterprise Manager(Appliance) Linux Kernel Java Detailed hardware information on your Gemini appliance includes: CPU Memory NIC Chassis 112

114 The Listen Port tab will list all the currently listening ports. This information are usually requested by the Gemini Support Team to assist with your support case. 113

115 The Audit Report tab allows you to create downloadable audit reports which included a list of all the libraries that Manager used with the version info and licenses. The current listening ports are also included in the reports. Authentication Gemini Appliance offers administration access either with the Gemini Enterprise: Manager web console or by running CLI commands using SSH. Admin Users To configure access to the Manager web console, configure as many local Admin Users as required. The password of Manager Admin Users need to comply with the Password Policy configured at a later stage. 114

116 OS Users To access the Gemini Appliance using SSH, the password for OS users can be changed from this screen. Note that OS Users don t have access to the Manager web interface. You may also see if an OS user is locked or forbidden to login. 115

117 To unlock an OS user, enter the user detail and switch the toggle button to green to allow login. If you forgot password, it also gives you a chance to reset it. LDAP You may configure LDAP resources here to support LDAP authentication for the Manager web console. When LDAP resources are configured successfully and correctly, a user will be able to login to Manager with their LDAP account. Note that LDAP server access is only used for authentication not for accessing roles or permissions. Note Please ensure that your BaseDN includes only those users who should have access to the appliance administration screens. 116

118 Single Sign-on (SSO) Gemini Appliance Single Sign-on (SSO) provides the ability to use an HTTP Reverse Proxy Server to handle Manager authentication. Once a user successfully logged in to the proxy, they can seamlessly access the Manager web console without having to login again. Manager expects a specific HTTP Request Header from the Reverse Proxy. The name of the HTTP Header field can be configured in the Single Sign-On configuration screen. Select the Automatically Create User option when the username from an authenticated request through the Reverse Proxy does not exist as a local Manager admin user. If this option is not selected and the username from the request doesn t exist in Manager, the request will fail and the Manager login prompt will be shown. For added security, authentication requests can be restricted to a specific set of IP addresses, and only requests having the Username Field in the HTTP Header. 117

119 Once SSO is authenticated, it will bypass any other authentication methods such as LDAP. 118

120 Password Policy Password Policy allows you to enforce password requirements to meet your security needs, including password complexity and password duration. Note Password Complexity applies to both Web admins and OS users. Password Duration only applies to OS users. Proxy Proxy Settings allows you to configure a proxy server for specific services, e.g. download Cloudera artifacts. 119

121 Login Banner Enable and edit the banner message to present to users when accessing the appliance via console, SSH, or browser. 120

122 Reboot Allows you to reboot your Gemini appliance immediately. 121

123 Shutdown Allows you to shutdown and power off your Gemini appliance immediately. Note that Splunk services will be stopped prior to shutdown in order to prevent unexpected errors. Account Displays information on the session s currently logged in user. Profile 122

124 Update the current user s name, password, avatar, and preferred languages on the user interface. Manager supports multiple languages including English, German, Japanese, and Traditional-Chinese. Logout Immediately logs the current user out of the session. 123

125 CLI Commands There are a series of shell commands that can be executed locally from console(tty0) or remotely logged in by sbox account from SSH. Type sbox for more details. Please note when the first time to execute sbox series of commands it will prompt you to input the password of account sbox. 124

126 Commands for initial setup The sbox setup command allows you to complete the basic network settings, including IPv4 settings of eth0, hostname and timezone. With this command you re able to build the basic network connection capabilities to the first network interface on your Gemini appliance. The sbox config command allows you to complete the system initial setup thru CLI, including Accept EULA, Hostname, Timezone, and activate Manager with license. The sbox cluster command allows you to configure Gemini cluster settings, including join a cluster and reset the settings. The sbox cluster --token will display the token string of this cluster node which is used by other nodes who want to join into this node. 125

127 The sbox admin command allows you to configure Manager web administrations. The sbox admin -set-password command allows you to set the password of default admin user. Once if all the initial setup are done thru CLI, type sbox admin --skip-wizard to disable the setup wizard in Manager web console when the first login. 126

128 Commands for information gathering The sbox --version command will display Manager version. The sbox --model command will display model name of this appliance. The sbox --service-tag command will display service tag on the appliance. All these information listed above are important when you request a technical support from us. The sbox admin --installed-packages will display installed Manager packages. The sbox service --status will display the status of Manager services. The sbox service --listen-port will display all listening ports of the services. 127

129 128

130 Commands for troubleshooting The sbox admin --reset-password command allows you to reset the Manager web console s admin password. The sbox admin --gen-ssl command allows you to generate a new SSL key for Manager web service. The sbox cloudera --undo command allows you remove all cloudera deployment on this Gemini appliance. The sbox cluster --reset command allows you to reset all the cluster settings on this appliance. The sbox network --reset command allows you to reset network settings and remove IP bondings. After running this command, basic network settings will need to be configured again to access Manager. 129

131 The sbox service --restart commands allows you to restart services of Manager in case the Manager isn t working properly. Commands for system operations When you are logged into the appliance as the sbox account from the console(tty0), the following commands are allowed. The reboot command allows you to reboot Gemini appliance. The poweroff command allows you to shutdown Gemini appliance. 130

132 Default Passwords for CLI Operations Only three accounts are provisioned for command line login to the appliance. They are listed here along with their default passwords: OS account default password Description sbox facing jet function drive Used for Manager administration splunk think adventure kitchen chest Used for Splunk administration hadoop popular fully apple hello Used for Cloudera administration These users will be required to change their password upon logging in for the first time. 131