Get your own Galaxy within minutes

Transcription

1 Get your own Galaxy within minutes Enis Afgan, Nitesh Turaga, Nuwan Goonasekera GCC 2016 Bloomington, IN

2 Access slides from bit.ly/gcc2016_usecloud

3 Today s agenda Introduction Hands on, part 1 Launch your own instance CloudMan Access Galaxy and register as a new user Getting data into Galaxy, part 1: FTP, GenomeSpace importer Importing a Galaxy workflow Becoming a Galaxy Admin Installing Galaxy tools via the Tool Shed Break Hands on, part 2 Add/remove instances Create an instance share Publish the share string on the Galaxy wiki Walk through the CloudMan wiki Launch a shared instance ssh access Troubleshooting Advanced Topics Upgrading Galaxy Getting data into Galaxy, part 2: scp + Data libraries Adding a custom file system (NFS, Gluster) persistent_data.yaml: architecture & editing Reducing the size of galaxyindices FS

4 Three ways to get Galaxy Use one of 80+ public servers Use Galaxy on the Cloud bit.ly/gxyservers launch.usegalaxy.org Ease of use Install Galaxy on your own infrastructure getgalaxy.org Flexibility

5 Cloud? What cloud? Cloud computing: on demand access to virtual computers and storage. Highly available, flexible, scalable, shareable, accessible Increasingly becoming the default platform for managing compute infrastructure.

6 Galaxy on the Cloud

7 Beyond Galaxy on the Cloud Genomics Virtual Lab (GVL): a workbench on the cloud Started in Australia running Galaxy on a national research cloud. Now is a platform for deploying bioinformatics application on any cloud. Galaxy, ssh access to tools, RStudio, IPython Jupyter, SMRT portal, Multiple flavors: microbial GVL, tutorial, more coming

8 Regional availability Free Free Free Galaxy on the Cloud GVL African VL (future)

9 When to use the cloud? Don t have informatics expertise or the infrastructure to run and maintain Have variable or particular resource needs Cannot upload data to a shared resource Need for customization Have oscillating data volume Want to test or share a tool, quickly & safely Want to make your analysis readily available to others Want fast access to AWS public datasets

10 How to use the cloud? 1. Get an account on the supported cloud 2. Start a master instance via CloudLaunch 3. Use CloudMan s web interface on the master instance to manage the platform 4. Use or customize Galaxy / GVL

11 Launching Galaxy on the Cloud

12 AWS access credentials for the conference Access key: AKIAIJ5NLK2IQTRKCOGA Secret key: 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 AWS console: Username: GCC16_cloud Pwd: V}n#Zo9SbMCu ssh key GCC16_cloud - select it when launching instances

13 CloudLaunch A service for discovering services and launching cloud instances. Today, we will be using a brand new CloudLaunch! Access via IP address: and login via Github

14 Add your cloud credentials and launch an app Visit My Profile page and add cloud credentials: Cloud: Amazon US East Virginia Access key: AKIAIJ5NLK2IQTRKCOGA Secret key: talwdzzicimumsn7jxrc9tgz/qqucrhxytfhkc3m Head over to the Marketplace and launch Galaxy on the Cloud. Once launched, open a new browser tab and paste the instance IP address.

15 CloudMan as a cloud resource manager Login using username ubuntu and password you supplied at launch.

16 CloudMan admin page: service and cluster administration

17 Your own Galaxy instance: ready for use Dozens of installed tools (list of tools: bit.ly/gctoolslist) Many gigabytes of reference genome data.

18 Your own Galaxy instance! Independent of other instances: requires a new Galaxy account Data available on other (public) instances needs to be imported Data generated on your instance needs to be downloaded / exported Customizable No quotas Let s register:

19 Let s import a Galaxy workflow Click on Workflow Upload or Import Workflow and paste this URL: Hmm - a missing tool!?!

20 Becoming a Galaxy administrator Galaxy has an admin interface for application and resource management Admin can manage users, jobs, tools, data libraries, reference data, etc. A user needs to be listed as an admin in the configuration to have access Become a Galaxy admin by entering your Galaxy username on CloudMan Admin

21 The tool Genetrack is missing in this workflow. This is your Galaxy, so let s add it.

22 Galaxy Toolshed: a repository of installable Galaxy tools Thousands to tools have been made available in the Tool Shed Can install any of the tools into your Galaxy through the browser Tools can automatically install their dependencies (most of the time) Multiple versions of the tool can be installed and coexist Several Tool Sheds exist - you can even run your own, private one

23 Browse or seach for a tool Tool repository has tool and dependencies info

24 Install a tool into your Galaxy A few options exist but typically just need to choose the tool category

25 Back at the workflow: the tool is installed

26 Let s run the workflow now Workflow input files: download them from AWS S3 (Just paste the following list into Galaxy s upload/fetch data form, choosing fastqsanger as the datatype and SacCer3 as the genome)

27 The workflow runs, but slow! Let s scale

28 1. Instance Pending 2. Instance Booting 3. Instance Ready

29 Making your work accessible Each instance of Galaxy CloudMan can be shared: for collaboration or publication Shares include all of user data and even the customizations Publish a self-contained analysis Sharing happens via share-string

30 1. Choose share visibility 2. Wait for the configuration changes 3. Share your string

31 CloudMan wiki - Getting started guide - Published share string - FAQ - Troubleshooting info

32 Command line access via ssh You have full system access to your instance, including sudo; use it as any other computer system (e.g., campus cluster) galaxy system user exists on the system and should be used when manipulating Galaxy (sudo su galaxy) Slurm is used as a job manager so can submit any jobs via the standard sbatch command To access: from terminal, run: ssh ubuntu@<instance IP address> and supply the password supplied at launch (or use a private ssh key associated with the instance at launch)

33 Command line access via ssh - on Windows Download PuTTY client (putty.exe and puttygen.exe) from It is necessary to convert the downloaded.pem key to PuTTY s.ppk format 1. Click File Load private key 2. Click File Save private key as gcc2016.ppk

34 Command line access via ssh - on Windows (cont ) Start PuTTY.exe 1. From the Category menu, find Connection SSH Auth. Under Private key file for authentication choose the path to gcc2016.ppk file created earlier 2. In the Category menu, choose Session and, under the Host Name field, enter ubuntu@<instance IP address> Click Open.

35 Command line access via Secure Shell for Chrome Add the Secure Shell extension to Chrome from Specify ubunut as the username and supply your Instance IP address for the hostname Type yes when asked about server s fingerprint Supply the password you chose when launching this instance.

36 Once logged in Basic file system structure Galaxy application ~/galaxy-app Galaxy tools ~/galaxy-tools Reference data is at ~/galaxy-indices Change into galaxy system user with sudo su galaxy Submit a Slurm job job_script.sh Create a job script file Submit with sbatch job_script.sh Monitor job with squeue #!/bin/bash #SBATCH -n 1 hn=`hostname -i` of="/mnt/transient_nfs/stats_$hn.out" samtools flagstat [input_file].bam > $of sleep 10

37 Troubleshooting Things can occasionally go sideways on a running system The easiest thing to try is restarting broken services: via CloudMan admin Can also reboot the entire cluster (also via CloudMan admin page) Check wiki for up-to-date documentation

38 Troubleshooting: log files To find out what might have happened, check the logs: /var/log/cloudman/cm_autorun.log /var/log/cloudman/cm_boot.log /var/log/cloudman/cloudman.log /mnt/galaxy/galaxy-app/main.log Listed order is the order in which CloudMan and Galaxy start on the system so one log will feed into the next.

39 Backend access, via cloud console (AWS here) Launcher and CloudMan create cloud resources under your cloud account Can view (and manage) all resources from the cloud console Good for cleanup in case applications crashed or were not properly shut down Each persistent cluster gets a bucket, named cm-[hash] plus a data volume, so check for those.

40 Advanced topics

41 Updating Galaxy code Basic, manual Galaxy update Stop Galaxy via CloudMan admin interface ssh to the instance and change to galaxy user (sudo su galaxy) Navigate to the Galaxy app dir; issue git pull command Start Galaxy from CloudMan In general, Galaxy on the Cloud is built via an Ansible playbook a. b. Come to the Build It workshop tomorrow github.com/galaxyproject/galaxy-cloudman-playbook To update more of Galaxy, run a portion of the playbook against a running instance.

42 Adding data to a Galaxy Data Library Data Libraries are shared repositories of data on a Galaxy instance. Can be populated via upload, from a history, or system paths. Let s place some data on our instance and create a data library. Ssh to the instance; create a data destination dir (e.g., /tmp/sampledata) Option A: wget data from S3 wget Option B: copy some data from your local machine scp <local file> ubuntu@<instance IP>:/tmp/sampleData From Galaxy, create a DL and import data from system path

43 Adding an additional file system: an external NFS server

44 Additional file systems can be made part of your system CloudMan can work with S3 buckets, volumes, snapshots or NFS servers To add an external NFS server, enter the server address and give the FS a name

45 A file system is visible from CloudMan and can be accessed from the command line

46 Customizing your deployment: runtime architecture persistent_data.yaml

47 persistent_data.yaml cluster_name: 'GCCdemo cluster_storage_type: 'volume' cluster_type: 'Galaxy' deployment_version: 2 filesystems: - ids: - snap-c332f2b0 kind: snapshot mount_point: /mnt/galaxyindices name: galaxyindices roles: - galaxyindices - ids: - 'vol-5dc1298e' kind: volume mount_point: /mnt/galaxy name: galaxy roles: - galaxytools - galaxydata machine_image_id: ami-b45e59de persistent_data_version: 3 placement: us-east-1e services: - name: Supervisor roles: - Supervisor - name: Postgres roles: - Postgres - name: ProFTPd roles: - ProFTPd - name: Slurmd roles: - Slurmd - name: Nginx roles: - Nginx - name: NodeJSProxy roles: - NodeJSProxy - home: /mnt/galaxy/galaxy-app name: Galaxy roles: - Galaxy - name: Slurmctld roles: - Slurmctld - Job manager

48 Acknowledgments

49 launch.usegalaxy.org

50

51

52

53 Launching an instance from AWS console and setting it up a NFS server

54

55

56

57

58

59

60

61

62 $ ssh -i key_pair.pem # This is the server launched via AWS console $ sudo apt-get update && sudo apt-get install nfs-kernel-server $ sudo vim /etc/exports; add name of folder to be shared and hostname Eg: /home/ubuntu *(rw,sync,no_root_squash,no_subtree_check) $ sudo service nfs-kernel-server restart $ ps aux grep nfsd #check process if its running

63

64

65 Let s run the workflow now

66 Set admin user

67