Verteego VDS Documentation

Transcription

1 Verteego VDS Documentation Release 1.0 Verteego May 31, 2017

2

3 Installation 1 Getting started 3 2 Ansible Install Ansible Clone installation package Install Verteego DS Sign in Custom settings Docker Install Docker Clone installation package Install Verteego DS Sign in Dataflow 13 5 Data cleaning 15 6 Analytics & dashboarding 17 7 Notebooks 19 8 Prediction 21 9 Examples Contact us Community 27 i

4 ii

5 Verteego Data Science Suite was built to deliver a plug-and-play environment for data scientists enabling both fast prototyping and scalable production deployments. Verteego s purpose is to provide a bundle of tools covering all common requirements of data scientists. Verteego is not just another big data platform but a best-of-breed mash-up of the most powerful big data and prediction solutions currently available. Installation 1

6 2 Installation

7 CHAPTER 1 Getting started Verteego Data Science Suite is aiming to gather the best data science tools into a unique consistent and powerful stack. Installing of Verteego is fast and straight forward. No hassle, no complicated configuration. Follow the installation instructions here. 3

8 4 Chapter 1. Getting started

9 CHAPTER 2 Ansible Please note that in the following installation instructions we use 2 placeholders for local directories: VDS_ROOT and SSH_ROOT. Just replace them with the correct directories from your system. VDS_ROOT: local directory where the installation package will be cloned to SSH_ROOT: local.ssh directory 1. Install Ansible We ll use Ansible to deploy Verteego DS to your remote server or Virtualbox. If you don t have Ansible yet, please install it as we ll use Ansible to orchestrate the automatic installation process for you. Linux Mac OS (Not tested) Windows (Not tested) 2. Clone installation package Clone the following repository to your local machine (NOT to the remote server on which you want run Verteego DS). We ll call this repository VDS_ROOT in the following. git clone 5

10 3. Install Verteego DS 3.1 Installation on Amazon web service 1. Configure account and fetch needed authentication file Create a key-pair and call it vds: Change the right of the downloaded file (vds.pem) to 400 : chmod 0400 Downloads/vds.pem Copy it to cp Downloads/vds.pem VDS_ROOT/deployment/ansible/files/aws/vds.pem Configure account rights : Create Access/Secret keys : Copy the Access and secret keys into key.json file under VDS_ROOT/deployment/ansible/files/aws/keys.json 2. Launch installation ansible-playbook -i VDS_ROOT/deployment/ansible/hosts --private-key=vds_root/ deployment/ansible/files/aws/vds.pem -u admin VDS_ROOT/deployment/ansible/setup_on_ aws.yml 3.2 Installation on google cloud platform 1. Install Google Cloud SDK Before you start you should make sure that you have a running Google Cloud platform account and the GCloud SDK installed (to install GCloud SDK: Configure your account and project gcloud init Generate SSH key for GCloud gcloud compute config-ssh 2. Set up the VDS environment on Google Cloud Create a Google service account : Go to Select the project into which you want to create the VDS instance Create a service account with project editor role Check the Furnish a new private key option Chose JSON key type When you click the Create button, a key file will be the downloaded. Copy the downloaded key file to VDS_ROOT/deployment/ansible/files/gcp and rename it to ansible.json 6 Chapter 2. Ansible

11 cp Downloads/ORIGINAL_KEYFILE.json VDS_ROOT/deployment/ansible/files/gcp/ansible.json 3. Install libcloud sudo apt-get install python-pip sudo pip install apache-libcloud==1.5.0 # in case you encounter an ssl certificate validation issue ( readthedocs.io/en/latest/other/ssl-certificate-validation.html#ssl-certificate- validation-in-v2-0) sudo pip install --upgrade certifi 4. Launch installation This will launch the default installation of Verteego Data Suite. For custom settings such as instance calibration, read this. ansible-playbook -i VDS_ROOT/deployment/ansible/hosts --private-key=ssh_root/google_ compute_engine VDS_ROOT/deployment/ansible/setup_gc_instance.yml Be patient, the deployment of all files can take a while depending on the capacity of the instance you ve chosen. 5. Start playing When the installation process has finished, using a browser, navigate to the newly created instance external IP on port : You can find the external ip address on on your Google Cloud Compute Engine web page console ( cloud.google.com/compute/instances). 3.3 Installation on a local virtual server (virtualbox) from sources 1. Install Virtualbox and Vagrant Install Virtualbox: Install Vagrant: 2. Launch Vagrant Go to the Vagrant directory (VDS_ROOT/vagrant) and launch Vagrant (this may take a while as it will download a full Debian image to be installed on Virtualbox): cd VDS_ROOT/vagrant vagrant up 3. Installation Launch installation ansible-playbook -i VDS_ROOT/deployment/ansible/hosts --private-key=vds_root/vagrant/. vagrant/machines/vds/virtualbox/private_key VDS_ROOT/deployment/ansible/setup_on_ vbox.yml 4. Start playing Navigate to Install Verteego DS 7

12 3.4 Installation on a remote virtual private server (vps) Requirements : this playbook is designed to work on a debian 8 distribution, so we assume your VPS to be running a debian 8 you should be able to connect o you VPS using a private key without password you should know your VPS s public ip remote user should be part of group sudoer, because we need sudo privileges to run all commands your server should expose the port range to 33335, to enable external access to the verteego datasuite. 1. Install VDS # Pay attention to the comma after the VPS_PUBLIC_IP ansible-playbook \ -i 'VPS_PUBLIC_IP,' \ --private-key=path_to_vps_private_ssh_key \ -u REMOTE_USER \ VDS_ROOT/setup_on_vps.yml 2. Start playing Navigate to 3. Sign in For your first sign in you can use the following credentials. For security reasons, remember to change them or delete the default user after your first login. Username: vds-user Password: verteego 4. Custom settings Customize infrastructure settings Your installation can be easily customised using the different.yml files in the VDS_ROOT/deployment/ansible directory. Example: Use a high-memory instance on Google Cloud Open VDS_ROOT/deployment/ansible/setup_gc_instance.yml In the vars:machine_type variable replace n1-standard-1 with n1-highmem-16. (see You can also directly precise specific settings in the command line using the extra-vars parameter while running ansible-playbook. Example : Use a high-memory instance on Google Cloud and deploy instance in a different zone ansible-playbook \n -i VDS_ROOT/deployment/ansible/hosts \n 8 Chapter 2. Ansible

13 --private-key=ssh_root/google_compute_engine VDS_ROOT/deployment/ansible/setup_gc_ instance.yml --extra-vars "ginstance_type=n1-highmem-16 gzone=us-central1-f" Customize application settings Open VDS_ROOT/deployment/ansible/group_vars/all/vars_file.yml to change the default settings for the different applications composing Verteego Data Suite Custom settings 9

14 10 Chapter 2. Ansible

15 CHAPTER 3 Docker 1. Install Docker Head to and download and install docker for your specfic plateform, docker is supported by Windows, Mac Os, Linux. Similarly head to and install docker-compose. Once installed you ll have access to docker and docker-compose commands using the command line. docker --version output : Docker version ce, build c6d412e docker-compose --version output : docker-compose version , build b31ff33 2. Clone installation package git clone 3. Install Verteego DS 3.1 using docker-compose on a single machine Using the command line, go to the directory deployment/docker inside the cloned repository from step 2. docker-compose up 11

16 Navigate to 4. Sign in For your first sign in you can use the following credentials. For security reasons, remember to change them or delete the default user after your first login. Username: vds-user Password: verteego 12 Chapter 3. Docker

17 CHAPTER 4 Dataflow The dataflow represents the backbone of your data science projects. It describes the different steps your data will run through from its source to prediction and visualization. The dataflow module is based on the Apache Nifi technology, one of the most powerful data routing and transformation tools currently available. It offers you a production-ready and highly configurable platform that can be run with a broad panel of data sources, formatting standards (XML, JSON, Avro, CSV,...) and big data technologies (Hadoop, Kafka, HDFS, Flume, Elasticsearch, HBase, Couchbase, Mongo DB, Solr, Splunk, Lumberjack, Cassandra, Hive,...). To learn more about the dataflow technology read on here. 13

18 14 Chapter 4. Dataflow

19 CHAPTER 5 Data cleaning The data cleaning module provides a powerful user interface to define a stack of data cleaning tasks that can be applied to any file running through your dataflow. Technology The data cleaning module is powered by Google s Open Refine technology. You ll find a full user documentation here. General Refine Expression Language (GREL) The data cleaning module comes with a powerful expression language allowing to apply precise transformation and cleaning tasks to your data. Check out the GREL documentation Running cleaning tasks in dataflow The dataflow module offers special processors that can be used to apply a cleaning script to a data flow. You may find them amongst the other processors by typing openrefine in the processor search field. Currently there are processors for XLS, XLSX, CSV and TSV support. 15

20 16 Chapter 5. Data cleaning

21 CHAPTER 6 Analytics & dashboarding Verteego Data Science Suite offers comprehensive analytical and dashboarding capabilities allowing to slice, dice and visualize your data at any step of your dataflow. Technology The analytics and dashboarding module is powered by Superset, a data exploration platform designed to be visual, intuitive and interactive. To dive deeper into the technology please check out the Superset Github and the user group. Database connections The analytics and dashboarding module can be run with a variety of databases using the SQL alchemy libraries. 17

22 18 Chapter 6. Analytics & dashboarding

23 CHAPTER 7 Notebooks Notebooks enable you to write custom scripts that can be run within your dataflow using the ExecuteProcess and ExecuteStreamCommand processors. Verteego comes with a few pre-installed kernels to cover the most common languages used by data scientists: Python 2.7 R Bash Technology The integrated notebooks run on Jupyter. Install additional kernels If you need programming languages that are not pre-installed with your package of Verteego you can find more of them here: List of community supported language kernels 19

24 20 Chapter 7. Notebooks

25 CHAPTER 8 Prediction The prediction module offers a simple and powerful way to produce predictive models without writing a line of code. Supported algorithms are: Aggregator Deep Learning Distributed Random Forest Gradient Boosting Machine Generalized Linear Modeling Generalized Low Rank Modeling K-means Naive Bayes Principal Components Analysis Technology Verteego Data Science Suite s predictive technology runs on open source software H2O. You may find the full documentation and some tutorials here. Predictive modelling Models trained on H2O are packaged as Plain Old Java Objects (POJO) that can easily be deployed and executed within your dataflow. Check out some examples of production environments. 21

26 22 Chapter 8. Prediction

27 CHAPTER 9 Examples Here are a few examples of what we and our users have built with the Verteego Data Science Suite: Use Case HR Reduce employee attrition and make talents stay longer (Part 1: Data Analysis) Use Case HR Reduce employee attrition and make talents stay longer (Part 2: Prediction) Clean and verify s All scripts belonging to these examples can be downloaded on the Verteego Github. Prediction deployment examples The H2O doc offers some examples of prediction deployments. 23

28 24 Chapter 9. Examples

29 CHAPTER 10 Contact us Need support, detected an issue or have a question? Drop a message to our community support! 25

30 26 Chapter 10. Contact us

31 CHAPTER 11 Community Drop a message and we ll do our best to help you out as fast as we can. You can also text us on Twitter. 27