A Sandbox environment for Hadoop

Transcription

1 A Sandbox environment for Hadoop Presentation 21 February 2017 Peter Smyth UK Data Service

2 Overview of this presentation Download and basic install of Hortonworks Sandbox Changing the size of the Sandbox VM Adding RStudio Server Problems using VMware player and how to overcome them Demonstrations HDFS & Hive Spark

3 Download and basic install Assuming for your desktop Two virtualisation systems VMware Player (Free for non-commercial use) Virtualbox (Free Open source, but owned by Oracle) If starting from scratch use Virtualbox The VMware VM has problems If already a VMware user, there are ways to overcome the problems covered later

4 Download and Install Download and install the latest version of Virtualbox from (5.0.32) From the Hortonworks site ( ) Download the virtualbox VM

5 Download and Install The downloaded Sandbox VM will have a.ova extension Right mouse click and open with Virtualbox manager You will see the import virtual appliance Change file extension to.vdi Change the RAM to Mb

6 Download and Install After the import is complete, start the VM using the start button

7 Download and Install Once the VM has finished loading (4-5 mins) you will be left with a Window looking very much like this

8 Virtualbox v Vmware Player Virtualbox always gives you an IP address of Vmware Player gives an IP address of xxx.yyy yyy starts at 128 and is then incremented for each VM you create xxx is random but remains the same for all subsequent VMs This different IP address allocation strategy is very relevant when dealing with VMware Sandbox problem

9 Accessing the Hadoop Sandbox Use :8888 in a web browser (not IE) to access the Web interface Use :2222 to access the command line using PuTTY or other tool You will need to use the command line to set the Ambari admin password When you know where you are going, you can use :8080 to access Ambari (for Hive and HDFS) :9995 to Access Zeppelin

10 Changing the hosts file located in C:\Windows\System32\drivers\etc

11 Setting the Ambari admin Password :8888 provides links you need to take the Advanced HDP option first

12 Setting the Ambari admin Password This has the link to setting the Ambari password

13 Setting the Ambari admin Password The link is to the Hortonworks site, so you need an Internet connection The instructions boil down to Use SSH (PuTTY) to logon to command line with user=root and pw=hadoop And run the command ambari-admin-password-reset

14 While at the commandline Try typing in: spark-shell pyspark sparkr

15 Extending the VM By default the provided VM is only ~50Gb Of which only about half is available to HDFS Extending the VM is a two step process Extend the VM size in Virtual box Using the Command line Extend the Linux partition size Using gpartd

16 Extending the VM (step 1) Navigate to the Virtualbox installation directory C:\Program Files\Oracle\VirtualBox in my case Open a command window And enter the command: vboxmanage modifyhd "C:\Users\pfsmy\VirtualBox VMs\Hortonworks Docker Sandbox\Hortonworks Docker Sandbox-disk1.vdi" --resize

17 Extending the Linux partition (step 2) To do this we will use gpartd You download it a an.iso image file From You then add this as a device in your VM (from Virtualbox) You then tell Virtual box to boot from this device

18 Extending the VM

19 Extending the VM Once the Virtualbox command line command has been run, the VM appears to have been extended But internally, Linux still has the same partition dimensions Gpartd is the simple visual tool to extend the Linux partition When you commit the changes, it effectively runs a script of all of the necessary Linux commands

20 Extending the VM (VMware) The internal structure of the Virtualbox and VMware VMs are different Virtualbox uses a normal Ext3 partition and VMware uses an LVM (Linux Virtual Machine) Gpartd cannot ( I don t think) extend a LVM but there are a good set of instructions available on the internet.

21 Installing RStudio Server Install R first From the commandline: yum install R yum install R-devel libcurl-devel openssl-devel And some R libraries (could be done after Rstudio install) R -e "install.packages('devtools', repos = ' R -e "install.packages('ggplot2', repos = '

22 Installing RStudio Server Installing the libraries, indicates the R has been successfully installed You can also run sparkr now from the command line The Linux VM ( or Docker container) has no gui interface installed, so the likes of ggplot2 are a bit pointless as it stands

23 Installing RStudio Server Find the latest version from the RStudio site wget sudo yum install --nogpgcheck rstudio-server-rhel x86_64.rpm Need to add a user to use with Rstudio server (requires a UID > 500) useradd sandbox passwd sandbox

24 Installing RStudio Server To test the installation run rstudio-server verify-installation It will fail with : initctl: Unable to connect to Upstart: This is a know problem and a solution is available ( r-rstudio-hdp-25.html)

25 Installing RStudio Server Use the following commands ln -s /usr/lib64/libssl.so.10 /usr/lib64/libssl.so.6 ln -s /usr/lib64/libcrypto.so.10 /usr/lib64/libcrypto.so.6 yum install dpkg dpkg-divert --local --rename --add /sbin/initctl ln -s /bin/true /sbin/initctl

26 Installing RStudio Server Changing the default port for Rstudio Server By default it is set to 8787, but this is not exported from the Docker VM So, create/edit /etc/rstudio/rserver.conf and add the single line www-port = 8090

27 Automating the starting of RStudio Server Create a script Rstudio in /etc/init.d #!/bin/sh /usr/lib/rstudio-server/extras/init.d/redhat/rstudio-server start Don t forget to run chmod +x /etc/init.d/rstudio

28 Automating the starting of RStudio Server In the Docker container, /etc/init.d files are not automatically run We need to ssh into the Docker Host VM using port 2122 Login with root/hadoop Edit the script ~/start_scripts/start_sandbox.sh script And append the line docker exec -d sandbox /etc/init.d/rstudio

29 Automating the starting of RStudio Server

30 Demos o o o o o o o Login to Ambari Hive view File (HDFS) view Zeppelin Run queries Rstudio Server SparkR code

31 Questions Peter Smyth ukdataservice.ac.uk/help/ Subscribe to the UK Data Service news list at Follow us on Twitter or Facebook