Vertica on Microsoft Azure HPE Vertica Analytic Database. Software Version: 7.2.x

Transcription

1 HPE Vertica Analytic Database Software Version: 7.2.x Document Release Date: 8/29/2016

2 Legal Notices Warranty The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HPE shall not be liable for technical or editorial errors or omissions contained herein. The information contained herein is subject to change without notice. Restricted Rights Legend Confidential computer software. Valid license from HPE required for possession, use or copying. Consistent with FAR and , Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. Copyright Notice Copyright Hewlett Packard Enterprise Development LP Trademark Notices Adobe is a trademark of Adobe Systems Incorporated. Apache Hadoop and Hadoop are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. UNIX is a registered trademark of The Open Group. This product includes an interface of the 'zlib' general purpose compression library, which is Copyright Jean-loup Gailly and Mark Adler. HPE Vertica Analytic Database (7.2.x) Page 2 of 25

3 Contents Overview of Vertica on Microsoft Azure 5 Recommended VM Types 6 More Information 6 Deploying Vertica from the Azure Marketplace 7 Installing and Running Vertica on Azure 9 Configuring and Launching a Virtual Machine (VM) 10 Configure Your VM 10 Add Storage 10 Configure Network Security Group 10 Configure Credentials 10 Assign a Public IP 10 Create Additional VMs 11 Network Security Group Configurations 12 Inbound Settings 12 Outbound Settings 12 More Information 13 Connecting to a Virtual Machine (VM) 14 Connect to Your VM 14 Connect to an Instance from Windows Using PuTTY 14 Connect to Other VMs 14 Preparing Virtual Machines (VMs) 15 Add Vertica License and Private Key 15 Install Dependencies 15 Configuring Storage 16 Attach Storage Volumes to Virtual Machines (VMs) 16 Combine Volumes for Storage 16 Create a Swap File 17 HPE Vertica Analytic Database (7.2.x) Page 3 of 25

4 More Information 17 Downloading Vertica 18 Forming a Cluster 19 Combine Virtual Machines (VMs) 19 Use the install_vertica or update_vertica Scripts 20 After Your Cluster Is Up and Running 21 Safe Shutdown 22 Using Management Console (MC) on Azure 23 Limitations 23 Version Compatibility 23 Learning More About MC 23 MC User Accounts 23 Send Documentation Feedback 25 HPE Vertica Analytic Database (7.2.x) Page 4 of 25

5 Overview of Vertica on Microsoft Azure Overview of Vertica on Microsoft Azure Vertica clusters on Microsoft Azure operate on Virtual Machines (VMs) within a Virtual Network. These instructions apply to VMs built with Vertica Version 7.2.x. HPE Vertica Analytic Database (7.2.x) Page 5 of 25

6 Recommended VM Types Recommended VM Types Vertica supports a range of Microsoft Azure virtual machine (VM) types, each optimized for different purposes. Choose the VM type that best matches your performance and price needs as a user. For the best performance in most common scenarios, Hewlett Packard Enterprise recommends the following VMs: Optimization General Purpose High Memory and I/O Throughput Type DS13_V2 DS14_V2 DS15_V2 GS3 GS4 GS5 Note: Data stored on the temporary drives of a VM exists only while that VM is powered on. After powering off a VM, data on temporary drives is lost. More Information For more information about Azure VMs and their limitations, see the Azure documentation. HPE Vertica Analytic Database (7.2.x) Page 6 of 25

7 Deploying Vertica from the Azure Marketplace Deploying Vertica from the Azure Marketplace You can deploy standard Vertica clusters from the Azure Marketplace. Azure Marketplace deployment allows you to: Create a 1 5 node Vertica cluster. Specify a VM type. Specify storage size. If you deploy a standard Vertica cluster from the Azure Marketplace, a template created by Hewlett Packard Enterprise automatically configures and creates your chosen cluster, based on your specifications. To obtain the one-click VM, search for Vertica in the Azure Marketplace. HPE Vertica Analytic Database (7.2.x) Page 7 of 25

8 Deploying Vertica from the Azure Marketplace HPE Vertica Analytic Database (7.2.x) Page 8 of 25

9 Installing and Running Vertica on Azure Installing and Running Vertica on Azure Start creating your cluster by creating your first VM. During the VM creation process, you can create and configure the various other resources required for your cluster, which will then be available for additional VMs you create. Follow these procedures to install and run Vertica on Azure: Configuring and Launching a VM Network Security Group Configuration Connecting to a VM Preparing VMs Configuring Storage Downloading Vertica Forming a Cluster After Your Cluster Is Up and Running Safe Shutdown Using Management Console (MC) on Azure HPE Vertica Analytic Database (7.2.x) Page 9 of 25

10 Configuring and Launching a Virtual Machine (VM) Configuring and Launching a Virtual Machine (VM) An Azure VM is similar to a traditional host. Just as with an on-premises cluster, you must prepare and configure your cluster and network at the hardware level before you can install Vertica. Configure Your VM 1. From the Azure marketplace, select an operating system that Vertica supports. 2. Select a VM type. 3. Choose a deployment model. Hewlett Packard Enterprise recommends the resource manager deployment model. Add Storage Create an Azure storage account, which will later contain your cluster storage volumes. For the best performance, Hewlett Packard Enterprise recommends Premium Storage. For more information about what a Storage Account is, and how to create one, refer to the Azure Documentation. Configure Network Security Group Vertica has specific network security group requirements. Create and name your own network security group with the basic security group settings outlined in the Network Security Group Configuration section. You may make additional modifications, based on your specific needs. Configure Credentials Create a password or assign a key pair you intend to use with Vertica. For information about what a key pair is, and how to create one, refer to the Azure documentation. Assign a Public IP Create a public IP address. A public IP is an IP address that you can use to connect to your cluster externally. Hewlett Packard Enterprise recommends you assign a single static public IP to a node in your cluster. You can then connect to other nodes in your cluster from your primary node using their internal IP addresses dictated by your Virtual Network settings. HPE Vertica Analytic Database (7.2.x) Page 10 of 25

11 Configuring and Launching a Virtual Machine (VM) For information about public IPs and how to create them, refer to the Azure documentation. Create Additional VMs Create additional VMs using the resources you configured in the steps above. HPE Vertica Analytic Database (7.2.x) Page 11 of 25

12 Network Security Group Configurations Network Security Group Configurations Inbound Settings Name Protocol Source Port Range Destination Port Range Source SSH TCP * 22 Any Any HTTP TCP * 80 Any Any HTTPS TCP * 443 Any Any DNS (UDP) UDP * 53 Any Any Spread UDP * Any Any Spread TCP * Any Any VSQL/SQL TCP * 5433 Any Any Destination Inter-node Communication TCP * 5434 Any Any TCP * 5444 Any Any MC TCP * 5450 Any Any TCP * 8080 Any Any TCP * Any Any rsync TCP * Any Any Outbound Settings Name Protocol Source Port Range Destination Port Range Source Destination All TCP TCP * Any Any All ICMP ICMP * Any Any All UDP UDP * Any Any HPE Vertica Analytic Database (7.2.x) Page 12 of 25

13 Network Security Group Configurations More Information For information about what a network security group is, and how to create one, refer to the Azure documentation. HPE Vertica Analytic Database (7.2.x) Page 13 of 25

14 Connecting to a Virtual Machine (VM) Connecting to a Virtual Machine (VM) Before you can connect to any of the VMs you created, you must first make your network externally accessible. To do so, you must attach the public IP address you created during network configuration to one of your VMs. Connect to Your VM To connect to your VM, complete the following tasks: 1. Attach your public IP address to one of your VMs. 2. Connect to your VM using SSH with the public IP address you created in the configuration steps. 3. Authenticate using the credentials and authentication method you specified during the VM creation process. Connect to an Instance from Windows Using PuTTY You can connect to the instance from the Windows operating system with PuTTY or WinSCP over the public IP. To do so, you must use your key. Connect to Other VMs Connect to other virtual machines in your virtual network by first using SSH to connect to your publicly connected VM. Then, use SSH again from that VM to connect through the private IP addresses of your other VMs. If you are using private key authentication, you may need to move your key file to the root directory of your publicly connected VM. Then, use PuTTY or WinSCP to connect to other VMs in your Virtual Network. HPE Vertica Analytic Database (7.2.x) Page 14 of 25

15 Preparing Virtual Machines (VMs) Preparing Virtual Machines (VMs) After you create your VMs, you need to prepare them for cluster formation. Add Vertica License and Private Key Prepare your instances by adding your private key, if you are using one, and your Vertica license. 1. As the dbadmin user, copy your private key file (from where you saved it locally) onto your primary instance. Depending upon the procedure you use to copy the file, the permissions on the file may change. If permissions change, the install_vertica script fails with a message similar to the following: Failed Login Validation , cannot resolve or connect to host as root. If you receive a failure message, enter the following command to correct permissions on your private key file: chmod 600 /<name-of-key>.pem 2. Copy your Vertica license over to your primary VM, placing it in your home directory or other known location. Install Dependencies In addition to the Vertica standard dependencies, you must install the following packages specific to Vertica on Azure: microsoft-hyper-v pstack mcelog sysstat HPE Vertica Analytic Database (7.2.x) Page 15 of 25

16 Configuring Storage Configuring Storage Use a dedicated Azure storage account for node storage. Caution: Hewlett Packard Enterprise recommends that you do not store your information on the root volume, especially your data and catalog directories. Storing information on the root volume may result in data loss. When configuring your storage, make sure to use a supported file system. Attach Storage Volumes to Virtual Machines (VMs) Using your previously created storage account, attach storage volumes to your VMs that are appropriate to your needs. For best performance, you can combine multiple storage volumes into RAID-0. Hewlett Packard Enterprise recommends 6 storage volumes per VM for most RAID-0 implementations. Combine Volumes for Storage If your are using RAID, follow these steps to create a RAID-0 drive on your VMs. The following examples show how you can create a RAID-0 volume named "md10" from 6 individual volumes, named "sdc", "sdd", "sde", "sdf", "sdg", and "sdh": 1. Form a RAID-0 volume using the mdadm utility. mdadm --create /dev/md10 --level 0 --raid-devices=6 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh 2. Format the file system to be one that Vertica supports. mkfs.ext4 /dev/md10 3. Find the UUID on the newly formed RAID volume. ls -l /dev/disk/by-uuid 4. Copy the UUID, and place it in the FSTAB file. sudo vi /etc/fstab UUID=<uuid here> /dev/md10 ext4 defaults, errors=remount-ro Create a directory for your Vertica data and catalog folders. mkdir vertica mkdir vertica/data HPE Vertica Analytic Database (7.2.x) Page 16 of 25

17 Configuring Storage 6. Mount the RAID volume to your data and catalog directory. mount /dev/md10 /home/dbadmin/vertica/data Create a Swap File In addition to storage volumes to store your data, Vertica requires a swap volume or swap file to operate. Create a swap file or swap volume of at least 2 GB. The following steps show how you can create a swap file within Vertica on Azure: 1. Install devnull and swapfile: install -o root -g root -m 0600 /dev/null /swapfile 2. Create the swap file: dd if=/dev/zero of=/swapfile bs=1024 count=2048k 3. Prepare the swap file using mkswap: mkswap /swapfile 4. Use swapon to instruct Linux to swap on the swapfile: swapon /swapfile 5. Persist the swapfile in FSTAB: echo "/swapfile swap swap auto 0 0" >> /etc/fstab Repeat the volume attachment, combination, and swap file creation procedures on each VM in your cluster. More Information For more information about Azure storage accounts, refer to the Azure documentation. For more information about Vertica storage configuration, refer to the Prepare Disk Storage Locations section of the Vertica documentation. HPE Vertica Analytic Database (7.2.x) Page 17 of 25

18 Downloading Vertica Downloading Vertica Prepare to install Vertica by downloading the vertica server file from my.vertica.com. Choose the Vertica server file appropriate for your operating system and license type. HPE Vertica Analytic Database (7.2.x) Page 18 of 25

19 Forming a Cluster Forming a Cluster Use the install_vertica script to combine two or more individual VMs and create a cluster. Check the Virutal Network page for a list of current VMs and their associated private IP addresses. You need these IP addresses when you run the install_vertica script. Combine Virtual Machines (VMs) The following example shows how you can combine VMs using the install_vertica script. 1. While connected to your primary instance, enter the following command to combine your instances into a cluster. a. Substitute the IP addresses for your VMs, and include your root key file name, if applicable. b. Include the --point-to-point parameter to configure spread to use direct point-to-point communication between all Vertica nodes, as required for clusters on Azure when installing or updating Vertica. If you do not use this parameter, you receive an error telling you that you must use point-to-point communication on Azure. sudo /opt/vertica/sbin/install_vertica --hosts , , dbauser-password-disabled --point-to-point --data-dir /home/dbadmin/vertica/data --sshidentity ~/<name-of-private-key>.pem --license <license.file> Note: If you are using Community Edition, which limits you to three instances, you can simply specify -L CE with no license file. After you combine your instances, Hewlett Packard Enterprise recommends that you reduce security risks by keeping your key file in a secure place, separate from your cluster and deleting your on-cluster key with the shred command. shred examplekey.pem Important: You will need your key file to perform future Vertica updates. 2. Reboot your cluster to complete cluster formation. After you form a cluster, you can create a database. HPE Vertica Analytic Database (7.2.x) Page 19 of 25

20 Forming a Cluster Use the install_vertica or update_vertica Scripts By default, the installer assumes that you have mounted your storage to /vertica/data. To specify another location, use the --data-dir argument. Caution: Hewlett Packard Enterprise recommends that you do not store your data on the root drive. For complete information on the install_vertica script and its parameters, see the Installation Guide, specifically the section, About the install_vertica Script. HPE Vertica Analytic Database (7.2.x) Page 20 of 25

21 After Your Cluster Is Up and Running After Your Cluster Is Up and Running After your cluster is configured and running, you can: 1. Create a database. When you installed Vertica, a database administrator user was created, dbadmin. You can use this pre-created dbadmin user to create and start a database. Refer to the Vertica Installation Guide for information on the dbadmin administrator role. 2. Configure a database. Refer to the Vertica Administrator s Guide for information on configuring a database. HPE Vertica Analytic Database (7.2.x) Page 21 of 25

22 Safe Shutdown Safe Shutdown Shut the database down first to make sure that Vertica is not in the process of writing to disk when you shutdown. To safely stop or reboot your cluster: 1. Stop the database. 2. Stop or reboot one or more VMs. Caution: If you stop or reboot a VM (or the cluster) without shutting the database down first, disk or database corruption could result. Refer to the Vertica Administrator s Guide for information on stopping a database. HPE Vertica Analytic Database (7.2.x) Page 22 of 25

23 Using Management Console (MC) on Azure Using Management Console (MC) on Azure Management Console (MC) is a database management tool that provides a way for you to view and manage aspects of your Vertica cluster. If you are running Vertica Release or later, you can install and run MC. Limitations This release of MC on Azure includes the following limitations: You cannot create or import a cluster on Azure using the MC. You cannot monitor an Azure cluster using MC on a node that is outside of your cluster. You must install MC on a VM within the Azure cluster environment. Version Compatibility Each version of Vertica Management Console (MC) is compatible only with the matching version of the Vertica server. Version numbers must match to three digits. For example, Vertica server is supported by Vertica MC only. This general MC requirement applies to all cases and is not specific to MC on Azure. Learning More About MC Use the following reading path for learning more about MC: For an overview of MC, where you can get it, and what you can do with it, refer to the Concepts Guide, specifically, Management Console. For information on installing and configuring MC, refer to the Installation Guide, specifically, Installing and Configuring Management Console (MC). For information on the differences between what you can do with MC in comparison with what you can do with admintools, refer to the Administration Guide, specifically the section, Administration Tools and Management Console. For information on creating a database using MC, refer to the Getting Started Guide, specifically the section, Create the Example Database Using Management Console. MC User Accounts When you first configure MC, during the configuration process you create an MC superuser (a Linux account). Issuing a Factory Reset on the MC does not create a new MC superuser, nor does it delete the existing MC superuser. When initializing after a Factory Reset, you must log on using the original MC superuser account. HPE Vertica Analytic Database (7.2.x) Page 23 of 25

24 Using Management Console (MC) on Azure For information on setting MC to its original state (Factory Reset), see the Administration Guide, specifically the section, Resetting MC to Pre-Configured state. After you configure MC, you can add users that are specific to MC. Users created through the MC interface are MC specific. When you subsequently change a password through the MC, you only change the password for the specific MC user. Passwords external to MC (for example, system Linux user and Vertica database passwords) remain unchanged. For information on MC users, refer to the Administration Guide, specifically the sections, Creating an MC User and MC configuration privileges. HPE Vertica Analytic Database (7.2.x) Page 24 of 25

25 Send Documentation Feedback If you have comments about this document, you can contact the documentation team by . If an client is configured on this system, click the link above and an window opens with the following information in the subject line: Feedback on Vertica on Microsoft Azure (Vertica Analytic Database 7.2.x) Just add your feedback to the and click send. If no client is available, copy the information above to a new message in a web mail client, and send your feedback to vertica-docfeedback@hpe.com. We appreciate your feedback! HPE Vertica Analytic Database (7.2.x) Page 25 of 25