Enable Interactive R and Python Environments on Hadoop

Transcription

1 Enable Interactive R and Python Environments on Hadoop Dave Carlson Data Science Technical Lead Zurich North America Ming Yuan Architect, Big Data & Cloud Capital One June 27, 2017 Room #308

2 Disclaimer The views expressed in this presentation are solely those of Dave Carlson or Ming Yan and are not those Zurich American Insurance Company or Capital One. July 10,

3 Goals Enables organizations to take advantage of a ecosystem for Big Data Analytics Leverage storage and processing power of Hadoop and Analytical Power of R Build an interactive and easy-to-use interface for data scentists Leverage open source projects to minimum the cost Overcome technical challenges from Kerberos July 10,

4 Why R and Python? Analysts have been transitioning to open source languages like R and Python and away from proprietary languages like SAS and SPSS. This is true for both academic and commercial communities. July 10,

5 Python integration uses Anaconda and Jupyter July 10,

6 R integration uses RStudio Server and RHadoop July 10,

7 R on Hadoop Install Hadoop Client Node Installed Rhadoop/rJDBC Installed RStudio Server Integrated R libraries with RStudio server July 10,

8 RHadoop and RJDBC Packages rhdfs rjdbc Provides basic connectivity to HDFS so that R programmers can browse, read, write, and modify files stored in HDFS Installed on the node that will run the R code plyrmr Enables R users to perform common data manipulation operations on very large data sets stored on Hadoop Installed on every node in Hadoop cluster Provides basic connectivity to Hive with a JDBC driver Installed on the node that will run the R code Depends on rjava and DBI packages rmr2 Allows R developers to perform statistical analysis in R via Hadoop MapReduce functionality on a Hadoop cluster Installed on every node in Hadoop cluster ravro Enable the read and write to avro files from local and HDFS file system and adds an avro input format for rmr2 Installed only on the node that will run the R code rhbase Not use July 10,

9 R and Hadoop Integration July 10,

10 RStudio Server Installation and Configuration Install RStudio Server rpm User Authentication against LDAP sudo cp /etc/pam.d/login /etc/pam.d/rstudio Integrate with R and its libraries in rserver.conf Specify R version rsession-which-r=/usr/local/bin/r Locates shared libraries rsession-ld-library-path =/opt/someapp/lib:/opt/anotherapp/lib Resource Allocation July 10,

11 RStudio Server, R and Hadoop Integration July 10,

12 Python -- Anaconda Cluster Head Node A system configured to act as the intermediary between the cluster and the outside network Can also be referred to as the master or edge node Compute node The machines managed by the head node that all work together to complete a single task Compute Node Thin Client Head Node Compute Node Compute Node July 10,

13 System Architecture RStudio Server Hadoop libraries rhdfs rmr2 plyrmr rjdbc Sparkr Hadoop Management Node MANAGE Thin Client Hadoop Data Node Hadoop data node Anaconda Head Node Anaconda Enterprise Notebook Anaconda Compute Node rmr2 plyrmr July 10,

14 R Framework R framework requires dplyr The easiest way to verify that the correct set of dependencies are installed is to use a fixed snapshot of CRAN using a fixed MRAN (Microsoft) snapshot. Add the following lines to your Rprofile.site and Renviron.site files. If you have a fresh installation of R you will need to create the files. $R_HOME/etc/Rprofile.site options(repos=structure(c(cran=' $R_HOME/etc/Renviron.site HADOOP_CMD=/usr/bin/hadoop USE_KERBEROS=0 HIVE_SERVER_HOST=localhost HIVE_SERVER_PORT=10000 HIVE_JAR_FOLDERS=/usr/lib/hive/lib Now install the honeycomb package using the devtools library within R. devtools::install_github('zurichpa/rhdfs', subdir='pkg') devtools::install_github('zurichpa/orpheus') devtools::install_github('zurichpa/honeycomb') July 10,

15 DEMO July 10,

16 Additional References R RHadoop ( Zurich Predictive Analytics ( honeycomb ( Python Impyla (