Numascale Analytics Appliance in the Cloud

Transcription

1 Numascale Analytics Appliance in the Cloud Atle Vesterkjær, Senior Software Engineer Numascale, Version 1.0

2 Contents REVISION HISTORY... 3 INTRODUCTION... 4 WHAT IS IN IT FOR YOU... 4 HOW TO LOG IN TO THE SYSTEM... 5 LINUX... 5 WINDOWS... 6 ACCESSING RSTUDIO RUN SPK PREDICTIONS IN PARALLEL WORKSHOP TASKS IN RSTUDIO: ACCESSING SHINY RUNNING A SHINY APP FROM RSTUDIO READING A DATASET READING DATA FROM Numascale Analytics Appliance for Cloud

3 Revision History Rev Date Author Changes Atle Vesterkjær Initial version Atle Vesterkjær Added code from Qlik workshop 3 Numascale Analytics Appliance for Cloud

4 Introduction The Numascale R Appliance is a turnkey solution for doing R development and production in a Shared Memory Environment, see for details. It contains (among other things): - Numascale Optimized kernel - CentOS 7.1 (CentOS Linux release (Core)) - Numascale R: An optimized version of the Open Source R from CRAN, which comes with Numascales preloaded LAPACK module for this hardware architecture, NC-LAPACK) - RStudio - Shiny In addition to the software components Numascale has made available some public datasets for climate research that may be of interest to the participants. Please check out the Numascale R Appliance Startup Guide for more details. What is in it for you The Numascale R Appliance for Cloud provides a mature infrastructure for doing analytics based on the Numascale Anlaytics Stack The data scientists that are experts on finding patterns in data will find a well oiled engine for getting results faster. Numascale develops unique hardware and software 4 Numascale Analytics Appliance for Cloud

5 for the most challenging computing workloads. The technology has its legacy in decades of Norwegian CPU development and have since been delivering world-class products in a highly competitive global market. Numascale has delivered the enabling technology for the world s largest computer running under one instance of the operating system, with more than 5000 cores and 21 Terabytes of memory. This knowledge in making our infrastructure scalable enable data scientists to focus on prediction models, machine learning in small datasets. They can even use the same code when the data size increases. The Numascale R Appliance in the Cloud comes in 3 different offerings: Basic offering is a system with 32 cores, 128 GB of memory and 500 GB disk Standard offering is a system with 64 cores, 256 GB of memory and 1TB disk. Professional offering is a system with 128 cores, 512 GB of memory and 2 TB disk The datasets are located in the rcloud folder that all the users of the Numascale R Appliance have access too: > setwd("/srv/data/rcloud/luftkvalitet/csvfiles/") In addition, the climate model from the World Climate Bank in R can be used, as presented in Running the example on this site can be a nice exercise. How to log in to the system In order to get access to the Numascale R Appliance for Cloud, please check out the Putty setup file for windows clients and the Open SSH setup file for Linux and Mac clients. Linux The users can test login to a gateway server after the public key is in place: ssh -p rcloudn@l20a.c.gompute.com.numascale.com After you are logged in to the gateway you can log in to the system itself [rcloud25@l20a.c.gompute.com ~]$ ssh rcloud25@ rcloud25@ 's password: Last login: Wed Jan 6 15:40: from gateway 5 Numascale Analytics Appliance for Cloud

6 ~]$ or access its RStudio interface through a browser. Windows Setting up PuTTY Start PuTTY. Add a new session in PuTTY with the following settings: Host Name: l20a.c.gompute.com Port: Numascale Analytics Appliance for Cloud

7 Under Connection->Data, add: Auto-login username: <username> (you will receive the correct username on from Numascale) 7 Numascale Analytics Appliance for Cloud

8 Under Connection->SSH->Auth, select your private key 8 Numascale Analytics Appliance for Cloud

9 Under Connection->SSH->Tunnels, add hostname and port Source port: 3838, Destination: X:3838 Source port: 8787, Destination: X: Numascale Analytics Appliance for Cloud

10 Go back to Session, type r-cloud in the Saved Sessions textbox, then click save. 10 Numascale Analytics Appliance for Cloud

11 Connect to the Numascale R Appliance Start PuTTY. In Sessions, select r-cloud, then click load. Now click Open. The first time you connect, you will get a security alert. The reason for the alert is that your computer doesn t know the server you connect to, as it has never connected to it before. Just click Yes on this alert. Accessing RStudio If you have set up ssh tunneling correctly, you can simply start RStudio in a browser using: 11 Numascale Analytics Appliance for Cloud

12 Each user has its own session, but common data is always located at: > setwd("/srv/data/rcloud") More information on Rstudio is available at Run SPK predictions in parallel In cooperation with The Norwegian Government Public Pension Fund (SPK) Jørn I. Halvorsen Senior Advisor & Data Scientist The Norwegian Government Public Pension Fund (SPK) c Workshop tasks in RStudio: setwd("/srv/data/qlik") In shell login to server as shown above and run lscp and htop: [rcloud25@l20a.c.gompute.com ~]$ ssh rcloud25@ lscpu htop 12 Numascale Analytics Appliance for Cloud

13 What do you see? How many cores are in the system? Open /srv/data/qlik/estimering_serial.r in Rstudio. Mark all the text (Ctrl-A) and press the run button in the upper left corner (watch the progress in Rstudio and htop) How long time did it take? How many cases were solved? How many cores were used in the computation (look at htop)? Open /srv/data/qlik/estimering_par.r in Rstudio. Mark all the text and press the run button in the upper left corner (watch the progress in Rstudio and htop) How long time did it take? How many cases were solved? How many cores were used? Open /srv/data/qlik/estimering_par_128.r in Rstudio. Mark all the and press the run button in the upper left corner (watch the progress in Rstudio and htop) How long time did it take? How many cases were solved? How many cores were used? You can also try out other fun commands head(a) tail(a) nrow(tidseriedatadf) head(tidseriedatadf) tail(tidseriedatadf) Accessing Shiny If you have set up the ssh tunnel correctly, you should be able to create shiny apps in the /home/rcloudnn/shinyapps directory. These will post to: 13 Numascale Analytics Appliance for Cloud

14 Running a Shiny app from RStudio It is also possible to run a Shiny app from RStudio: > library(shiny) > runexample("01_hello") Reading a dataset The pre downloaded dataset is collected from: - Reading data from Downloading this data is not public available, but we have been given permission/access to it from It is Trondheim Municipality in cooperation with Statens Vegvesen that are responsible for the measurements we have downloaded, thank you very much for your efforts. The data has been stored in the most common format in R, csv. The MeanValuesAllDaily show measurements from all stations with granularity days. The MeanValuesAllMontly show measurements from all stations with granularity months. The MeanValuesAllYearly show measurements from all stations with granularity Years. The MeanValuesBakke13.csv show measurements from Bakke church stations with granularity hours for the year The same logic applies for the other stations. In order to open a file do e.g: 14 Numascale Analytics Appliance for Cloud

15 > setwd("/srv/data/rcloud/luftkvalitet/csvfiles/") > getwd() [1] "~/rcloud/luftkvalitet/csvfiles" > list.files() [1] "MeanValuesAllDaily.csv" "MeanValuesAllMonthly.csv" "MeanValuesAllYearly.csv" [4] "MeanValuesBakke08_16.csv" "MeanValuesBakke11.csv" "MeanValuesBakke12.csv" [7] "MeanValuesBakke13.csv" "MeanValuesBakke14.csv" "MeanValuesBakke15.csv" [10] "MeanValuesElgseter1_14.csv" "MeanValuesElgseter1_15.csv" "MeanValuesElgseter11.csv" [13] "MeanValuesElgseter12.csv" "MeanValuesElgseter13.csv" "MeanValuesElgseter14.csv" [16] "MeanValuesElgseter15.csv" "MeanValuesHeimdalsmyra11.csv" "MeanValuesHeimdalsmyra12.csv" [19] "MeanValuesHeimdalsmyra13.csv" "MeanValuesHeimdalsmyra14.csv" "MeanValuesTiller08_16.csv" [22] "MeanValuesTiller13.csv" "MeanValuesTiller14.csv" "MeanValuesTiller15.csv" [25] "MeanValuesTorvet11.csv" "MeanValuesTorvet12.csv" "MeanValuesTorvet13.csv" [28] "MeanValuesTorvet14.csv" "MeanValuesTorvet15.csv" x <- read.csv("meanvaluesbakke15.csv") > x <- subset(x, select=c(dato, Time..Bakke.kirke...NO..ug.m3, Time..Bakke.kirke...PM10..ug.m3, Time..Bakke.kirke...PM2.5..ug.m3)) > plot(x[,1], x[,2], xlab=names(x[,1]), ylab=names(x[,2])) If you would like to compare the data with weather data from Trondheim, then please take a look at the sample R scripts luftkvalitetdailytmp.r, or luftkvalitetmonthlytmp.r: source('/srv/data/rcloud/luftkvalitetmonthlytmp.r') source('/srv/data/rcloud/luftkvalitetdailytmp.r ) 15 Numascale Analytics Appliance for Cloud

16 16 Numascale Analytics Appliance for Cloud

17 Happy Testing! 17 Numascale Analytics Appliance for Cloud