Running Galaxy in an HPC environment requirements, challenges and some solutions : the LIFEPORTAL

Size: px
Start display at page:

Download "Running Galaxy in an HPC environment requirements, challenges and some solutions : the LIFEPORTAL"

Transcription

1 Running Galaxy in an HPC environment requirements, challenges and some solutions : the LIFEPORTAL Nikolay Vazov University Center for Information Technologies University of Oslo

2 Major requirements Authentication and authorization How do we manage users and their access to the cluster resources Resource management How do we control and allocate number of CPUs, memory, CPU hours Friendly redeployment of Galaxy code How can we easily customize and re-deploy a Galaxy instance and keep the updates in the original code

3 Authentication Federated National Identity provider (FEIDE) Additional IdP for non-feide users but covering as many Galaxy instances as possible Integration of EduGAIN (a super body of partner federations) Integration of OpenID type domain specific IdPs (e.g. CLARIN) Drop the local authentication

4 Authorization For all users : based on project (account) affiliation Visitor-users : browse but can't execute Per application basis : some applications are open to everybody Reviewers : anonymous users with a limited active period and application access (used to reproduce the results in published articles)

5 Resource management Users grouped in projects (accounts) Projects receive allocations : CPU time Every job on the cluster follows the procedure: Reservation of resources Execution Charging Refunding (if requested resources are bigger than really used) Allocation manager - GOLD

6 GOLD Allocation Manager open source L(A)PP technology Linux Apache (optional, needed for GUI only) Postgresql (MySQL) Perl easy to install (compilation needed) terminal and GUI (optional) easy to plug into Galaxy gold commands are called as subprocesses in python excellent manual v at present, project stopped, the resumed, now 3.x available

7 User registration and job execution galaxy user registered in Galaxy user DB 2. job user launches a job to be executed on the cluster 1. (simultaneously) 3. check for available resources in GOLD 4. confirmation for available resources from GOLD, resources reserved GOLD allocation manager a. user registered in GOLD user DB b. user associated to an account in GOLD c. user receives an allocation within the account 7. job data sent to GOLD, account charged* 6. completed job info sent to Galaxy results displayed in history panel 5. job sent to cluster, all job parameters included all job parametres sent to the cluster Job running -> Job completed cluster

8 Resource management (screenshot)

9 What do we send and what do we receive? To the cluster >>> DRMAA.PY slurm script #!/bin/sh #SBATCH -A nn9108k #SBATCH -t 300:00:00 #SBATCH -D /work/projects/galaxy/data/database_galaxy_prod01/job_working_directory/000/629 #SBATCH -J #SBATCH -n 1 #SBATCH --mem-per-cpu=4000 #SBATCH --ntasks-per-node=8 #SBATCH --partition=normal source /etc/profile source /cluster/bin/jobsetup cd /work/projects/galaxy/data/database_galaxy_prod01/job_working_directory/000/629 module load blast+; blastn -query "/work/projects/galaxy/data/database_galaxy_prod01/files/000/dataset_922.dat" -db "/work/databases/bio/ncbipre/est_others" -task blastn -evalue out /work/projects/galaxy/data/database_galaxy_prod01/files/000/dataset_924.dat -outfmt 6 -num_threads 8 -dust yes -strand both -max_target_seqs 100 echo $? > /work/projects/galaxy/data/database_galaxy_prod01/job_working_directory/000/629/galaxy_629.ec From the cluster {'charge_duration': '451587', 'galaxy_job_id': '629', 'lifeportal_project': 'lp2', 'machine': 'c17-6', 'processes': '8', 'slurm_account': 'nn9108k', 'slurm_job_id': ' ', 'user': 'a.k.krabberod@ibv.uio.no'} Successfully charged job 629 for credits MESSAGE FROM GOLD galaxy.jobs.runners.drmaa DEBUG :27:26,034 Successfully charged job 629 for credits galaxy.jobs.runners.drmaa DEBUG :27:26,034 (629/ ) state change: job finished normally

10 Redeployment Keep the Galaxy code as clean as possible from customizations : AAI outside Galaxy code Minimal changes in the code Repositories (git) to organize versions with different features

Authentication in Galaxy : let's use what is out there - (National) Identity Providers

Authentication in Galaxy : let's use what is out there - (National) Identity Providers Authentication in Galaxy : let's use what is out there - (National) Identity Providers Nikolay Vazov University Center for Information Technologies University of Oslo Authentication challenges Galaxy on

More information

Setting up a Galaxy instance as a service

Setting up a Galaxy instance as a service Setting up a Galaxy instance as a service (or... how can we make simple things more complicated) Nikolay Vazov University Center for Information Technologies (USIT) University of Oslo Norway Astonishment

More information

Introduction to GACRC Teaching Cluster

Introduction to GACRC Teaching Cluster Introduction to GACRC Teaching Cluster Georgia Advanced Computing Resource Center (GACRC) EITS/University of Georgia Zhuofei Hou zhuofei@uga.edu 1 Outline GACRC Overview Computing Resources Three Folders

More information

Slurm and Abel job scripts. Katerina Michalickova The Research Computing Services Group SUF/USIT October 23, 2012

Slurm and Abel job scripts. Katerina Michalickova The Research Computing Services Group SUF/USIT October 23, 2012 Slurm and Abel job scripts Katerina Michalickova The Research Computing Services Group SUF/USIT October 23, 2012 Abel in numbers Nodes - 600+ Cores - 10000+ (1 node->2 processors->16 cores) Total memory

More information

Slurm and Abel job scripts. Katerina Michalickova The Research Computing Services Group SUF/USIT November 13, 2013

Slurm and Abel job scripts. Katerina Michalickova The Research Computing Services Group SUF/USIT November 13, 2013 Slurm and Abel job scripts Katerina Michalickova The Research Computing Services Group SUF/USIT November 13, 2013 Abel in numbers Nodes - 600+ Cores - 10000+ (1 node->2 processors->16 cores) Total memory

More information

Working with Shell Scripting. Daniel Balagué

Working with Shell Scripting. Daniel Balagué Working with Shell Scripting Daniel Balagué Editing Text Files We offer many text editors in the HPC cluster. Command-Line Interface (CLI) editors: vi / vim nano (very intuitive and easy to use if you

More information

Introduction to Abel/Colossus and the queuing system

Introduction to Abel/Colossus and the queuing system Introduction to Abel/Colossus and the queuing system November 14, 2018 Sabry Razick Research Infrastructure Services Group, USIT Topics First 7 slides are about us and links The Research Computing Services

More information

Exercises: Abel/Colossus and SLURM

Exercises: Abel/Colossus and SLURM Exercises: Abel/Colossus and SLURM November 08, 2016 Sabry Razick The Research Computing Services Group, USIT Topics Get access Running a simple job Job script Running a simple job -- qlogin Customize

More information

Bright Cluster Manager

Bright Cluster Manager Bright Cluster Manager Using Slurm for Data Aware Scheduling in the Cloud Martijn de Vries CTO About Bright Computing Bright Computing 1. Develops and supports Bright Cluster Manager for HPC systems, server

More information

Overview of a virtual cluster

Overview of a virtual cluster Overview of a virtual cluster using OpenNebula and SLURM Ismael Farfán Estrada ifarfane0900@ipn.mx October 27, 2011 Ismael Farfán Estrada ifarfane0900@ipn.mx () Overview of a virtual cluster October 27,

More information

Introduction to Joker Cyber Infrastructure Architecture Team CIA.NMSU.EDU

Introduction to Joker Cyber Infrastructure Architecture Team CIA.NMSU.EDU Introduction to Joker Cyber Infrastructure Architecture Team CIA.NMSU.EDU What is Joker? NMSU s supercomputer. 238 core computer cluster. Intel E-5 Xeon CPUs and Nvidia K-40 GPUs. InfiniBand innerconnect.

More information

Introduction to SLURM & SLURM batch scripts

Introduction to SLURM & SLURM batch scripts Introduction to SLURM & SLURM batch scripts Anita Orendt Assistant Director Research Consulting & Faculty Engagement anita.orendt@utah.edu 6 February 2018 Overview of Talk Basic SLURM commands SLURM batch

More information

Introduction to SLURM & SLURM batch scripts

Introduction to SLURM & SLURM batch scripts Introduction to SLURM & SLURM batch scripts Anita Orendt Assistant Director Research Consulting & Faculty Engagement anita.orendt@utah.edu 16 Feb 2017 Overview of Talk Basic SLURM commands SLURM batch

More information

Notes for installing a local blast+ instance of NCBI BLAST F. J. Pineda 09/25/2017

Notes for installing a local blast+ instance of NCBI BLAST F. J. Pineda 09/25/2017 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 Notes for installing a local blast+ instance of NCBI BLAST F. J. Pineda 09/25/2017

More information

Using a Linux System 6

Using a Linux System 6 Canaan User Guide Connecting to the Cluster 1 SSH (Secure Shell) 1 Starting an ssh session from a Mac or Linux system 1 Starting an ssh session from a Windows PC 1 Once you're connected... 1 Ending an

More information

Introduction to High-Performance Computing (HPC)

Introduction to High-Performance Computing (HPC) Introduction to High-Performance Computing (HPC) Computer components CPU : Central Processing Unit cores : individual processing units within a CPU Storage : Disk drives HDD : Hard Disk Drive SSD : Solid

More information

Slurm basics. Summer Kickstart June slide 1 of 49

Slurm basics. Summer Kickstart June slide 1 of 49 Slurm basics Summer Kickstart 2017 June 2017 slide 1 of 49 Triton layers Triton is a powerful but complex machine. You have to consider: Connecting (ssh) Data storage (filesystems and Lustre) Resource

More information

Introduction to SLURM on the High Performance Cluster at the Center for Computational Research

Introduction to SLURM on the High Performance Cluster at the Center for Computational Research Introduction to SLURM on the High Performance Cluster at the Center for Computational Research Cynthia Cornelius Center for Computational Research University at Buffalo, SUNY 701 Ellicott St Buffalo, NY

More information

Batch Systems & Parallel Application Launchers Running your jobs on an HPC machine

Batch Systems & Parallel Application Launchers Running your jobs on an HPC machine Batch Systems & Parallel Application Launchers Running your jobs on an HPC machine Partners Funding Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike

More information

Introduction to SLURM & SLURM batch scripts

Introduction to SLURM & SLURM batch scripts Introduction to SLURM & SLURM batch scripts Anita Orendt Assistant Director Research Consulting & Faculty Engagement anita.orendt@utah.edu 23 June 2016 Overview of Talk Basic SLURM commands SLURM batch

More information

Heterogeneous Job Support

Heterogeneous Job Support Heterogeneous Job Support Tim Wickberg SchedMD SC17 Submitting Jobs Multiple independent job specifications identified in command line using : separator The job specifications are sent to slurmctld daemon

More information

High Performance Computing Cluster Basic course

High Performance Computing Cluster Basic course High Performance Computing Cluster Basic course Jeremie Vandenplas, Gwen Dawes 30 October 2017 Outline Introduction to the Agrogenomics HPC Connecting with Secure Shell to the HPC Introduction to the Unix/Linux

More information

Using Galaxy to provide a NGS Analysis Platform

Using Galaxy to provide a NGS Analysis Platform 11/15/11 Using Galaxy to provide a NGS Analysis Platform Friedrich Miescher Institute - part of the Novartis Research Foundation - affiliated institute of Basel University - member of Swiss Institute of

More information

High Performance Computing Cluster Advanced course

High Performance Computing Cluster Advanced course High Performance Computing Cluster Advanced course Jeremie Vandenplas, Gwen Dawes 9 November 2017 Outline Introduction to the Agrogenomics HPC Submitting and monitoring jobs on the HPC Parallel jobs on

More information

Helping Non-traditional HPC Users Using XSEDE Resources Efficiently

Helping Non-traditional HPC Users Using XSEDE Resources Efficiently Helping Non-traditional HPC Users Using XSEDE Resources Efficiently PIs: Robert Sean Norman (U South Carolina) Atsuko Tanaka and Chao Fu (U Wisconsin) ECSS staff: Shiquan Su National Institute of Computational

More information

An introduction to checkpointing. for scientific applications

An introduction to checkpointing. for scientific applications damien.francois@uclouvain.be UCL/CISM - FNRS/CÉCI An introduction to checkpointing for scientific applications November 2013 CISM/CÉCI training session What is checkpointing? Without checkpointing: $./count

More information

Blog site (cont.) theme, 202 view creations, 205 Browser tools, 196 Buytaert, Dries, 185

Blog site (cont.) theme, 202 view creations, 205 Browser tools, 196 Buytaert, Dries, 185 Index A Administration, 157 backups and restore (see Backups and restore website) file system, 161 log files, 162 tasks, 157 updates and security patches, 165 user accounts, 166 Aggregator module, 218

More information

INTRODUCTION TO NEXTFLOW

INTRODUCTION TO NEXTFLOW INTRODUCTION TO NEXTFLOW Paolo Di Tommaso, CRG NETTAB workshop - Roma October 25th, 2016 @PaoloDiTommaso Research software engineer Comparative Bioinformatics, Notredame Lab Center for Genomic Regulation

More information

Using Compute Canada. Masao Fujinaga Information Services and Technology University of Alberta

Using Compute Canada. Masao Fujinaga Information Services and Technology University of Alberta Using Compute Canada Masao Fujinaga Information Services and Technology University of Alberta Introduction to cedar batch system jobs are queued priority depends on allocation and past usage Cedar Nodes

More information

Cluster Upgrade Procedure with Job Queue Migration.

Cluster Upgrade Procedure with Job Queue Migration. Cluster Upgrade Procedure with Job Queue Migration. Zend Server 5.6 Overview Zend Server 5.6 introduces a new highly-reliable Job Queue architecture, based on a MySQL database storage backend. This document

More information

Bash for SLURM. Author: Wesley Schaal Pharmaceutical Bioinformatics, Uppsala University

Bash for SLURM. Author: Wesley Schaal Pharmaceutical Bioinformatics, Uppsala University Bash for SLURM Author: Wesley Schaal Pharmaceutical Bioinformatics, Uppsala University wesley.schaal@farmbio.uu.se Lab session: Pavlin Mitev (pavlin.mitev@kemi.uu.se) it i slides at http://uppmax.uu.se/support/courses

More information

Graham vs legacy systems

Graham vs legacy systems New User Seminar Graham vs legacy systems This webinar only covers topics pertaining to graham. For the introduction to our legacy systems (Orca etc.), please check the following recorded webinar: SHARCNet

More information

Applications Software Example

Applications Software Example Applications Software Example How to run an application on Cluster? Rooh Khurram Supercomputing Laboratory King Abdullah University of Science and Technology (KAUST), Saudi Arabia Cluster Training: Applications

More information

An introduction to checkpointing. for scientifc applications

An introduction to checkpointing. for scientifc applications damien.francois@uclouvain.be UCL/CISM An introduction to checkpointing for scientifc applications November 2016 CISM/CÉCI training session What is checkpointing? Without checkpointing: $./count 1 2 3^C

More information

Porting SLURM to the Cray XT and XE. Neil Stringfellow and Gerrit Renker

Porting SLURM to the Cray XT and XE. Neil Stringfellow and Gerrit Renker Porting SLURM to the Cray XT and XE Neil Stringfellow and Gerrit Renker Background Cray XT/XE basics Cray XT systems are among the largest in the world 9 out of the top 30 machines on the top500 list June

More information

DARIAH-AAI. DASISH AAI Meeting. Nijmegen, March 9th,

DARIAH-AAI. DASISH AAI Meeting. Nijmegen, March 9th, DARIAH-AAI DASISH AAI Meeting Nijmegen, March 9th, 2014 www.dariah.eu What is DARIAH? DARIAH: Digital Research Infrastructure for the Arts and Humanities One of the few ESFRI research infrastructures for

More information

OpenMP Example. $ ssh # You might have to type yes if this is the first time you log on

OpenMP Example. $ ssh # You might have to type yes if this is the first time you log on OpenMP Example Day 1, afternoon session 1: We examine a serial and parallel implementation of a code solving an N-body problem with a star, a planet and many small particles near the planet s radius. The

More information

Introduction to the NCAR HPC Systems. 25 May 2018 Consulting Services Group Brian Vanderwende

Introduction to the NCAR HPC Systems. 25 May 2018 Consulting Services Group Brian Vanderwende Introduction to the NCAR HPC Systems 25 May 2018 Consulting Services Group Brian Vanderwende Topics to cover Overview of the NCAR cluster resources Basic tasks in the HPC environment Accessing pre-built

More information

Selenium Testing Training

Selenium Testing Training About Intellipaat Intellipaat is a fast-growing professional training provider that is offering training in over 150 most sought-after tools and technologies. We have a learner base of 600,000 in over

More information

pbsacct: A Workload Analysis System for PBS-Based HPC Systems

pbsacct: A Workload Analysis System for PBS-Based HPC Systems pbsacct: A Workload Analysis System for PBS-Based HPC Systems Troy Baer Senior HPC System Administrator National Institute for Computational Sciences University of Tennessee Doug Johnson Chief Systems

More information

Resource Management at LLNL SLURM Version 1.2

Resource Management at LLNL SLURM Version 1.2 UCRL PRES 230170 Resource Management at LLNL SLURM Version 1.2 April 2007 Morris Jette (jette1@llnl.gov) Danny Auble (auble1@llnl.gov) Chris Morrone (morrone2@llnl.gov) Lawrence Livermore National Laboratory

More information

Introduction to High-Performance Computing (HPC)

Introduction to High-Performance Computing (HPC) Introduction to High-Performance Computing (HPC) Computer components CPU : Central Processing Unit cores : individual processing units within a CPU Storage : Disk drives HDD : Hard Disk Drive SSD : Solid

More information

Batch Systems. Running your jobs on an HPC machine

Batch Systems. Running your jobs on an HPC machine Batch Systems Running your jobs on an HPC machine Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_us

More information

Manual Backup Sql Server Express 2008 Schedule Database Sql Agent

Manual Backup Sql Server Express 2008 Schedule Database Sql Agent Manual Backup Sql Server Express 2008 Schedule Database Sql Agent Automate the Backup of Your Microsoft SQL Server Express Databases Server Express database server, only to discover that the handy job

More information

Introduction to GACRC Teaching Cluster

Introduction to GACRC Teaching Cluster Introduction to GACRC Teaching Cluster Georgia Advanced Computing Resource Center (GACRC) EITS/University of Georgia Zhuofei Hou zhuofei@uga.edu 1 Outline GACRC Overview Computing Resources Three Folders

More information

Using the Galaxy Local Bioinformatics Cloud at CARC

Using the Galaxy Local Bioinformatics Cloud at CARC Using the Galaxy Local Bioinformatics Cloud at CARC Lijing Bu Sr. Research Scientist Bioinformatics Specialist Center for Evolutionary and Theoretical Immunology (CETI) Department of Biology, University

More information

Introduction to High Performance Computing at Case Western Reserve University. KSL Data Center

Introduction to High Performance Computing at Case Western Reserve University. KSL Data Center Introduction to High Performance Computing at Case Western Reserve University Research Computing and CyberInfrastructure team KSL Data Center Presenters Emily Dragowsky Daniel Balagué Guardia Hadrian Djohari

More information

ADAC Federated Testbed Creating a Blueprint for Portable Ecosystems

ADAC Federated Testbed Creating a Blueprint for Portable Ecosystems ADAC Federated Testbed Creating a Blueprint for Portable Ecosystems Sadaf Alam, Jeffrey Vetter, Mark Klein, Maxime Martinasso, ExCL team @ ORNL,... ADAC Workshop February 15, 2018 January, 2016 June, 2016

More information

A Brief Overview of CIPRES Job Management

A Brief Overview of CIPRES Job Management A Brief Overview of CIPRES Job Management Mark A. Miller San Diego Supercomputer Center CIPRES is a classic browser-based gateway designed to submit jobs to multiple hosts. (REST services are now available

More information

Scheduling By Trackable Resources

Scheduling By Trackable Resources Scheduling By Trackable Resources Morris Jette and Dominik Bartkiewicz SchedMD Slurm User Group Meeting 2018 Thanks to NVIDIA for sponsoring this work Goals More flexible scheduling mechanism Especially

More information

g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.

g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac. g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse

More information

How to run a job on a Cluster?

How to run a job on a Cluster? How to run a job on a Cluster? Cluster Training Workshop Dr Samuel Kortas Computational Scientist KAUST Supercomputing Laboratory Samuel.kortas@kaust.edu.sa 17 October 2017 Outline 1. Resources available

More information

Brigham Young University

Brigham Young University Brigham Young University Fulton Supercomputing Lab Ryan Cox Slurm User Group September 16, 2015 Washington, D.C. Open Source Code I'll reference several codes we have open sourced http://github.com/byuhpc

More information

Submitting and running jobs on PlaFRIM2 Redouane Bouchouirbat

Submitting and running jobs on PlaFRIM2 Redouane Bouchouirbat Submitting and running jobs on PlaFRIM2 Redouane Bouchouirbat Summary 1. Submitting Jobs: Batch mode - Interactive mode 2. Partition 3. Jobs: Serial, Parallel 4. Using generic resources Gres : GPUs, MICs.

More information

STARTING THE DDT DEBUGGER ON MIO, AUN, & MC2. (Mouse over to the left to see thumbnails of all of the slides)

STARTING THE DDT DEBUGGER ON MIO, AUN, & MC2. (Mouse over to the left to see thumbnails of all of the slides) STARTING THE DDT DEBUGGER ON MIO, AUN, & MC2 (Mouse over to the left to see thumbnails of all of the slides) ALLINEA DDT Allinea DDT is a powerful, easy-to-use graphical debugger capable of debugging a

More information

An Introduction to Gauss. Paul D. Baines University of California, Davis November 20 th 2012

An Introduction to Gauss. Paul D. Baines University of California, Davis November 20 th 2012 An Introduction to Gauss Paul D. Baines University of California, Davis November 20 th 2012 What is Gauss? * http://wiki.cse.ucdavis.edu/support:systems:gauss * 12 node compute cluster (2 x 16 cores per

More information

Using and Developing with Azure. Joshua Drew

Using and Developing with Azure. Joshua Drew Using and Developing with Azure Joshua Drew Visual Studio Microsoft Azure X-Plat ASP.NET Visual Studio - Every App Our vision Every App Every Developer .NET and mobile development Desktop apps - WPF Universal

More information

Hands-on CUDA exercises

Hands-on CUDA exercises Hands-on CUDA exercises CUDA Exercises We have provided skeletons and solutions for 6 hands-on CUDA exercises In each exercise (except for #5), you have to implement the missing portions of the code Finished

More information

Using DC/OS for Continuous Delivery

Using DC/OS for Continuous Delivery Using DC/OS for Continuous Delivery DevPulseCon 2017 Elizabeth K. Joseph, @pleia2 Mesosphere 1 Elizabeth K. Joseph, Developer Advocate, Mesosphere 15+ years working in open source communities 10+ years

More information

Introduction to the Cluster

Introduction to the Cluster Follow us on Twitter for important news and updates: @ACCREVandy Introduction to the Cluster Advanced Computing Center for Research and Education http://www.accre.vanderbilt.edu The Cluster We will be

More information

CNAG Advanced User Training

CNAG Advanced User Training www.bsc.es CNAG Advanced User Training Aníbal Moreno, CNAG System Administrator Pablo Ródenas, BSC HPC Support Rubén Ramos Horta, CNAG HPC Support Barcelona,May the 5th Aim Understand CNAG s cluster design

More information

Federated access to e-infrastructures worldwide

Federated access to e-infrastructures worldwide Federated access to e-infrastructures worldwide Marco Fargetta, INFN Catania - Italy (marco.fargetta@ct.infn.it) DCIs developed in the last decade 2 Evolution Research organisations are moving to cloud

More information

Tutorial 4 Data Persistence in Java

Tutorial 4 Data Persistence in Java TCSS 360: Software Development Institute of Technology and Quality Assurance Techniques University of Washington Tacoma Winter 2017 http://faculty.washington.edu/wlloyd/courses/tcss360 Tutorial 4 Data

More information

Product Data Sheet: Ignition 8 Industrial Application Platform. A Whole New View

Product Data Sheet: Ignition 8 Industrial Application Platform. A Whole New View Product Data Sheet: Ignition 8 Industrial Application Platform A Whole New View With the powerful tools, web technologies, and next-generation visualization system we re building in Ignition 8, you ll

More information

Managing PKI Deployments. WhiteRabbitSecurity

Managing PKI Deployments. WhiteRabbitSecurity Managing PKI Deployments Overview Business requirements Automated CI/CD process overview Demo VM (with Vagrant) (directory layout, usage) Modifying the configuration Deployment in test environment Marking

More information

Introduction to GALILEO

Introduction to GALILEO Introduction to GALILEO Parallel & production environment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Domenico Guida d.guida@cineca.it Maurizio Cremonesi m.cremonesi@cineca.it

More information

Schema Management with Flyway

Schema Management with Flyway Schema Management with Flyway Scott Mead scottm@openscg.com Sr. Architect OpenSCG & BigSQL PGConf US 2017 1 Strategies Single File Application Managed (ORM) Migration 2 Single File 3 Single File Easiest

More information

Using the Computer Programming Environment

Using the Computer Programming Environment Information sheet EN064 Overview C2k has developed an environment to allow GCSE and A-Level students to undertake computer programming from within the C2k Managed Service. This environment will deliver

More information

Ing. José A. Mejía Villar M.Sc. Computing Center of the Alfred Wegener Institute for Polar and Marine Research

Ing. José A. Mejía Villar M.Sc. Computing Center of the Alfred Wegener Institute for Polar and Marine Research Ing. José A. Mejía Villar M.Sc. jmejia@awi.de Computing Center of the Alfred Wegener Institute for Polar and Marine Research 29. November 2011 Contents 1. Fedora Commons Repository 2. Federico 3. Federico's

More information

Case Study. CMS for Management of Monetization Training Resources

Case Study. CMS for Management of Monetization Training Resources Case Study CMS for Management of Monetization Training Resources Client Requirement The client is a digital marketing company providing efficient strategies for marketing and data monetization to their

More information

Andrej Filipčič

Andrej Filipčič Singularity@SiGNET Andrej Filipčič SiGNET 4.5k cores, 3PB storage, 4.8.17 kernel on WNs and Gentoo host OS 2 ARC-CEs with 700TB cephfs ARC cache and 3 data delivery nodes for input/output file staging

More information

Slurm Birds of a Feather

Slurm Birds of a Feather Slurm Birds of a Feather Tim Wickberg SchedMD SC17 Outline Welcome Roadmap Review of 17.02 release (Februrary 2017) Overview of upcoming 17.11 (November 2017) release Roadmap for 18.08 and beyond Time

More information

Automatic Dependency Management for Scientific Applications on Clusters. Ben Tovar*, Nicholas Hazekamp, Nathaniel Kremer-Herman, Douglas Thain

Automatic Dependency Management for Scientific Applications on Clusters. Ben Tovar*, Nicholas Hazekamp, Nathaniel Kremer-Herman, Douglas Thain Automatic Dependency Management for Scientific Applications on Clusters Ben Tovar*, Nicholas Hazekamp, Nathaniel Kremer-Herman, Douglas Thain Where users are Scientist says: "This demo task runs on my

More information

Tech Note 969 Port Configuration List for System Platform 2012 R2 Multi-Galaxy Environment

Tech Note 969 Port Configuration List for System Platform 2012 R2 Multi-Galaxy Environment Tech Note 969 Port Configuration List for System Platform 2012 R2 Multi-Galaxy Environment All Tech Notes, Tech Alerts and KBCD documents and software are provided "as is" without warranty of any kind.

More information

iems Interactive Experiment Management System Final Report

iems Interactive Experiment Management System Final Report iems Interactive Experiment Management System Final Report Pēteris Ņikiforovs Introduction Interactive Experiment Management System (Interactive EMS or iems) is an experiment management system with a graphical

More information

UL HPC Monitoring in practice: why, what, how, where to look

UL HPC Monitoring in practice: why, what, how, where to look C. Parisot UL HPC Monitoring in practice: why, what, how, where to look 1 / 22 What is HPC? Best Practices Getting Fast & Efficient UL HPC Monitoring in practice: why, what, how, where to look Clément

More information

HPC Workshop. Nov. 9, 2018 James Coyle, PhD Dir. Of High Perf. Computing

HPC Workshop. Nov. 9, 2018 James Coyle, PhD Dir. Of High Perf. Computing HPC Workshop Nov. 9, 2018 James Coyle, PhD Dir. Of High Perf. Computing NEEDED EQUIPMENT 1. Laptop with Secure Shell (ssh) for login A. Windows: download/install putty from https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

More information

The Why and How of HPC-Cloud Hybrids with OpenStack

The Why and How of HPC-Cloud Hybrids with OpenStack The Why and How of HPC-Cloud Hybrids with OpenStack OpenStack Australia Day Melbourne June, 2017 Lev Lafayette, HPC Support and Training Officer, University of Melbourne lev.lafayette@unimelb.edu.au 1.0

More information

June Workshop Series June 27th: All About SLURM University of Nebraska Lincoln Holland Computing Center. Carrie Brown, Adam Caprez

June Workshop Series June 27th: All About SLURM University of Nebraska Lincoln Holland Computing Center. Carrie Brown, Adam Caprez June Workshop Series June 27th: All About SLURM University of Nebraska Lincoln Holland Computing Center Carrie Brown, Adam Caprez Setup Instructions Please complete these steps before the lessons start

More information

Compilation and Parallel Start

Compilation and Parallel Start Compiling MPI Programs Programming with MPI Compiling and running MPI programs Type to enter text Jan Thorbecke Delft University of Technology 2 Challenge the future Compiling and Starting MPI Jobs Compiling:

More information

Sherlock for IBIIS. William Law Stanford Research Computing

Sherlock for IBIIS. William Law Stanford Research Computing Sherlock for IBIIS William Law Stanford Research Computing Overview How we can help System overview Tech specs Signing on Batch submission Software environment Interactive jobs Next steps We are here to

More information

Crystal Enterprise. Overview. Contents. Web Server Overview - Internet Information System (IIS)

Crystal Enterprise. Overview. Contents. Web Server Overview - Internet Information System (IIS) Overview Contents This document provides an overview to web server technology particularly Microsoft s Internet Information Server (IIS) and its relationship with. Although this article has been written

More information

Internet2 Meeting September 2005

Internet2 Meeting September 2005 End User Agents: extending the "intelligence" to the edge in Distributed Systems Internet2 Meeting California Institute of Technology 1 OUTLINE (Monitoring Agents using a Large, Integrated s Architecture)

More information

Real Life Web Development. Joseph Paul Cohen

Real Life Web Development. Joseph Paul Cohen Real Life Web Development Joseph Paul Cohen joecohen@cs.umb.edu Index 201 - The code 404 - How to run it? 500 - Your code is broken? 200 - Someone broke into your server? 400 - How are people using your

More information

Fusion Registry 9 SDMX Data and Metadata Management System

Fusion Registry 9 SDMX Data and Metadata Management System Registry 9 Data and Management System Registry 9 is a complete and fully integrated statistical data and metadata management system using. Whether you require a metadata repository supporting a highperformance

More information

Submitting batch jobs

Submitting batch jobs Submitting batch jobs SLURM on ECGATE Xavi Abellan Xavier.Abellan@ecmwf.int ECMWF February 20, 2017 Outline Interactive mode versus Batch mode Overview of the Slurm batch system on ecgate Batch basic concepts

More information

Database Replication in Tashkent. CSEP 545 Transaction Processing Sameh Elnikety

Database Replication in Tashkent. CSEP 545 Transaction Processing Sameh Elnikety Database Replication in Tashkent CSEP 545 Transaction Processing Sameh Elnikety Replication for Performance Expensive Limited scalability DB Replication is Challenging Single database system Large, persistent

More information

Mastering Linux. Paul S. Wang. CRC Press. Taylor & Francis Group. Taylor & Francis Croup an informa business. A CHAPMAN St HALL BOOK

Mastering Linux. Paul S. Wang. CRC Press. Taylor & Francis Group. Taylor & Francis Croup an informa business. A CHAPMAN St HALL BOOK Mastering Linux Paul S. Wang CRC Press Taylor & Francis Group Boca Raton London New York CRC Press is an Imprint of the Taylor & Francis Croup an informa business A CHAPMAN St HALL BOOK Contents Preface

More information

Jenkins User Conference Israel. #jenkinsconf. CI / Liveperson. Gidi Samuels. July 16, #jenkinsconf

Jenkins User Conference Israel. #jenkinsconf. CI / Liveperson. Gidi Samuels.  July 16, #jenkinsconf Jenkins User Conference Israel #jenkinsconf CI / CD @ Liveperson Gidi Samuels www.liveperson.com July 16, 2014 #jenkinsconf About Myself Lead the CI Team at Liveperson 3 years in Liveperson About Myself

More information

The LGI Pilot job portal. EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers

The LGI Pilot job portal. EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers The LGI Pilot job portal EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers Outline What? Why? How? Pro's and Cons What's next? Credits 2 What is LGI? LGI Project Server

More information

Using CSC Environment Efficiently,

Using CSC Environment Efficiently, Using CSC Environment Efficiently, 13.2.2017 1 Exercises a) Log in to Taito either with your training or CSC user account, either from a terminal (with X11 forwarding) or using NX client b) Go to working

More information

Sql 2008 Copy Table Structure And Database To

Sql 2008 Copy Table Structure And Database To Sql 2008 Copy Table Structure And Database To Another Table Different you can create a table with same schema in another database first and copy the data like Browse other questions tagged sql-server sql-server-2008r2-express.

More information

Oracle Enterprise Manager. 1 Introduction. System Monitoring Plug-in for Oracle Enterprise Manager Ops Center Guide 11g Release 1 (

Oracle Enterprise Manager. 1 Introduction. System Monitoring Plug-in for Oracle Enterprise Manager Ops Center Guide 11g Release 1 ( Oracle Enterprise Manager System Monitoring Plug-in for Oracle Enterprise Manager Ops Center Guide 11g Release 1 (11.1.3.0.0) E18950-03 November 2011 This document provides describes the System Monitoring

More information

DOCS

DOCS HOME DOWNLOAD COMMUNITY DEVELOP NEWS DOCS Docker Images Docker Images for Avatica Docker is a popular piece of software that enables other software to run anywhere. In the context of Avatica, we can use

More information

Managing and Deploying GPU Accelerators. ADAC17 - Resource Management Stephane Thiell and Kilian Cavalotti Stanford Research Computing Center

Managing and Deploying GPU Accelerators. ADAC17 - Resource Management Stephane Thiell and Kilian Cavalotti Stanford Research Computing Center Managing and Deploying GPU Accelerators ADAC17 - Resource Management Stephane Thiell and Kilian Cavalotti Stanford Research Computing Center OUTLINE GPU resources at the SRCC Slurm and GPUs Slurm and GPU

More information

Let's Play... Try to name the databases described on the following slides...

Let's Play... Try to name the databases described on the following slides... Database Software Let's Play... Try to name the databases described on the following slides... "World's most popular" Free relational database system (RDBMS) that... the "M" in "LAMP" and "XAMP" stacks

More information

Repository In A Box (RIB)

Repository In A Box (RIB) Repository In A Box (RIB) August 18, 2003 Yuanlei Zhang To Be Covered» Brief overview of RIB» The RIB patches» RIB v2.2 The last Perl version release» RIB v3.0 Java version release 1 RIB History & Overview»

More information

DevOps examples on NonStop Tools Overview. Cor Geboers, ATC Consultant

DevOps examples on NonStop Tools Overview. Cor Geboers, ATC Consultant DevOps examples on NonStop Tools Overview Cor Geboers, ATC Consultant About me Cor Geboers Senior Consultant in NonStop ATC, based in Belgium 35+ years in IT development and support 25+ years NonStop experience

More information

EGI Check-in service. Secure and user-friendly federated authentication and authorisation

EGI Check-in service. Secure and user-friendly federated authentication and authorisation EGI Check-in service Secure and user-friendly federated authentication and authorisation EGI Check-in Secure and user-friendly federated authentication and authorisation Check-in provides a reliable and

More information

Chapter 2. Operating-System Structures

Chapter 2. Operating-System Structures Chapter 2 Operating-System Structures 2.1 Chapter 2: Operating-System Structures Operating System Services User Operating System Interface System Calls Types of System Calls System Programs Operating System

More information