Deploy containers on your cluster - A proof of concept
|
|
- Gerald Harmon
- 6 years ago
- Views:
Transcription
1 Deploy containers on your cluster - A proof of concept
2 What is HPC cluster (in my world!) Where do I come from? Run and maintain a bioinformatics cluster at Bioinformatic Research Centre (BiRC), Aarhus University anders.dannesboe@birc.au.dk The setup cores 3.5PB parallel file system (henceforth known as /faststorage ) Use SLURM as our scheduler
3 What is HPC cluster (in my world!) A bunch of servers connected together with access to a shared file system Pipelines are spread into parallel pieces and run on multiple nodes at onces, to achieve accumulated speedup A multiuser system. Pipelines are run by unprivileged users (no root!) Everything is orchestrated by a scheduler. Takes care of resource sharing. E.g: Kill jobs that takes to long Enforces the limits of cores+memory of each job Packs multiple jobs from multiple users together on as few nodes as possible
4 What is HPC cluster (in my world!) What kinds of jobs do we run? Lots of data - Large input datasets, large shared reference dataset - Sensitive data Lots of different software by lots of different people - Versions keeps on changing Work-in-progress pipelines - Batches are seldom run twice. But a batch can have 50,000 of the same job-type Everything is in flux
5 Docker Docker: A Revolutionary Change in Cloud Computing
6 Docker Docker: A Revolutionary Change in Cloud Computing
7 Docker Dockers focus: Make software run the same anywhere Use containers to make software OS independent Take over networking, to make containers datacenter environment independent no static/fixed ip s One storage model, to make it backing independent image/container content is just fills in your filesystem Docker takes care of many of the nitty gritty details and lets you focus on package your software ones and for all
8 What are linux containers? Chroot on steroids Each container comes with its own OS Spawning a container runs a new init. Every running container on the host is a independent OS running on the system Uses features i Linux-kernel to achieve process isolation Cgroups for resource management Linux namespaces for process isolation Leverage OverlayFS in data/deployment model Spawn multiple containers from the same template without copying a thing
9 What are linux containers? Linux Namespaces PID namespace Network namespace UTS namespace(hostname) User namespace(uid/gid) Mount namespace Has been long underway. Full support under anything by Ubuntu/Debian can be tricky
10 What are linux containers? Why is this powerful? Container will work the same anywhere Each container is isolated Allow unprivileged users to run anything. Let them become root Utilize OverlayFS Spawn a new full OS in under a second Spawning multiple containers from the same template takes up no extra space No hypervisor, just native performance No need for syscall translation => No overhead Run 100+ containers on one host Back to Docker =>
11 Docker Docker is by far the most popular container implementation The design philosophy of docker has been adopted wholesale Creating docker images through recipes (Dockerfile) Running containers are ephemeral Make docker images reusable by others Images are be easy to publish and to download and use Split your software stack into smaller units by containerizing one service at a time
12 Docker Has gain serious traction amongst companies/developers working in the cloud. Here Docker and its philosophy helps: Plan, structure, develop and deploy the software stack Lots of effort has been but into containerizing existing software stack (also in academia) Restructure code under a better more scalable model Cloud ready Get in while the buzz it hot
13 Docker Some of the heavy hitters From academia Björn Grüning (bgruening) from University of Freiburg
14 Meanwhile in HPC...
15 Can we get Docker into our HPC clusters? How can we capitalize? A lot of software has already been dockerized. Projects like: Or easy to get into containerize: And the list of container resources gross every day How can we deploy all these containers with ready to use software inside our HPC cluster?
16 Merging containers into cluster computing Let's look at the pipeline Individual pieces of software strung together in a chain* Each link in the chain takes output from the previous link and uses it as input. Instead of the actual software being the link, how about using containers? To rephrase: Split your pipeline into smaller units by containerizing one link at a time Makes your pipelines cluster independent** Much of the development can be done off-cluster, on your own system Write your awesome software once, and everybody can use it. #citations Reuse others (a little bit less awesome) software in your pipeline *A lattice I guess, or else we wouldn't be doing stuff in parallel **well no. But a step in the right direction
17 Use case - The cluster user Missing a piece of software? Search the web for existing images: Or query from the cmd: $:> docker search bowtie2 * Find a link in a research paper *This does require mulled, biodocker etc. to be setup as repos
18 Use case - The cluster user No luck? Build your own container. $:> mkdir bowtie2 && cd bowtie2 $:> vim Dockerfile 1 FROM ubuntu 2 3 RUN apt-get update -qq --fix-missing 4 RUN apt-get install -qq -y wget unzip 5 RUN wget -q -O bowtie2.zip 6 RUN unzip bowtie2.zip -d /opt/ 7 RUN ln -s /opt/bowtie /opt/bowtie2 8 RUN rm bowtie2.zip 9 10 ENV PATH $PATH:/opt/bowtie2 $:> docker build -t bowtie $:> docker images REPOSITORY TAG IMAGE ID CREATED SIZE bowtie latest 49c23f71b287 9 seconds ago 289 MB ubuntu latest c73a085dc378 5 days ago 127 MB $:> docker run --rm -it bowtie bowtie2 -h Bowtie 2 version by Ben Langmead (langmea@cs.jhu.edu, Usage: bowtie2 [options]* -x <bt2-idx> {-1 <m1> -2 <m2> -U <r>} [-S <sam>]...
19 Use case - The cluster user Push our own work to dockerhub for others to re-use: $:> docker push bowtie Docker images can be pushed to repositories (dockerhub being one), and automatically pulled in if needed. Dockerhub can monitor git repositories and rebuild a new docker image on commits. Setup a (private) docker repository on your local network that pulls content from the most relevant global repos. Each docker daemon can stream in >1GB docker images within seconds.
20 What would we like to achieve? Make your lives as user easier by reusing existing and working docker images from papers, colleage, previous projects Make your lives as an administrator easier by not maintaining a plethora of software compiled to custom specifications from source Make our pipelines easier to rerun on a different cluster, by packaging the software into docker images that can run everywhere
21 What do we need? 1. Mapping of data Enable containers to work on the data (massive in size) on the HPC filesystem like any piece of software (within reason ;)) 2. Resource limiting A way for the docker daemon to run under the resource management of SLURM, so that the scheduler can do resource sharing. 3. Maintain security A cluster user should never be able to achieve priviledge escalation (of any sort) Alice should only be able to run as alice No one but Alice should be able to run as alice
22 Mapping of data Map data from host to container via mount-bind docker run -v /storage:/storage debian /bin/bash Idear: Make a 1-1 map of the shared storage into the container. File paths are the same outside and inside a container. Easy to work with. Example: #sbatch tool_a /storage/input -o /storage/output.a tool_b /storage/output.a -o /storage output.b cat /storage/output.b #sbatch docker run -v /storage:/storage tool_a /storage/input -o /storage/output.a docker run -v /storage:/storage tool_b /storage/output.a -o /storage/output.b docker run -v /storage:/storage cat /storage/output.b
23 Mapping of data Problem solved. Let crack on
24 Mapping of data Problem solved. Let crack on Major break of nr. 3: Maintain security Docker defaults Containers run as root Anyone in the docker group can spawn containers All are equal in eyes of the daemon Alice get to spawn just as much as root does
25 Mapping of data Evil Alice Mapping part of the host OS into a container, Alice can act like root in the mother OS. What about: docker run -v /storage/sensitive_data:/unsensitive_data debian /bin/bash And even worse: docker run -v /etc/shadow:/root/shadow debian /bin/bash Read-write access to our password file!
26 Mapping of data Unprivileged containers Any storage that is mapped inside a container retain the restrictions of the user spawning Filesystems doesn t have multiple and separate UID/GID ranges Utilize the size of this UID/GID space, and shift containers into unused UID/GID s to isolate them. UID/GID gets translated back and forth when Unprivileged containers has existed and been used in LXC for a while. Fairly new (and unknown) option Docker
27 Mapping of data Who does it work? Assign a isolated UID-space and GID-space to a user 2 new files /etc/subuid and /etc/subgid Use these UID/GID s inside the container $:> usermod --add-subuids alice $:> usermod --add-subgids alice $:> docker daemon --userns-remap alice:alice & $:> docker run --rm -it -v /etc/shadow:/root/shadow debian /bin/bash #:> touch /etc/shadow #:> touch /root/shadow touch: cannot touch '/root/shadow': Permission denied *Available in Ubuntu since But not in CentOS 7 yet.
28 Mapping of data That was a step too far! What about reference data, input data and output data? Soulution: Shift UID s and GID s into boring isolation but keep the UID of the user and GID on the project. cat /etc/subuid alice:100000:1000 alice:1000:1 alice:101001:64535 cat /etc/subgid plants:100000:10000 plants:10000:1 plants:110000:64535
29 Mapping of data Succes! $:> docker daemon --userns-remap alice:plant & $:> docker run --rm -it \ -v /etc/shadow:/root/shadow \ -v /storage:/storage debian /bin/bash #:> touch /root/shadow touch: cannot touch '/root/shadow': Permission denied #:> cd /storage #:> ls humans lost+found plants #:> ls humans/ ls: cannot open directory humans/: Permission denied #:> ls plants/ some_plant.gene
30 Mapping of data What did we need? Edit /etc/subuid and /etc/subgid to shift anything but the user uid and project gid into a isolated uid/gid range Multiple running docker daemons. One pr. <user>:<group> mapping Add --userns-remap to restrict container file access Add --group to restrict access to the docker daemon docker daemon \ --graph=/mnt/scratch/$user.$project/docker \ --pidfile=/mnt/scratch/$user.$project/docker.pid \ -H unix:///mnt/scratch/$user.$project/docker.sock \ --group=$user_id \ --userns-remap=$user_id:$group_id Your users are now able to run containers on your filesystem!
31 Resource limiting In any HPC cluster the scheduler must have total resource control. Jobs are run with the privileges of the use Processes are subprocesses of slurmd Docker daemon must be spawned by root Containers run as subprocesses of the docker daemon 1. Unprivileged user must be able to start the docker daemon 2. The scheduler must be able to monitor/control the resources of docker 3. When a job is killed, all containers spawned by that job must die
32 Resource limiting SLURM already uses cgroups. And that is all we need Write a setuid script start_docker that assert permissions and forks out a docker daemon locked to the <user>:<project> Run start_docker inside a job to use containers The cgroup stay with the daemon. Monitoring/limiting its resources Use SLURMs epilog-hook to cleanup afterwards Kills docker daemon and containers if still running Delete any container leftovers
33 Resource limiting Check the process tree slurmstepd bash pstree a sudo docker_daemon plants docker_daemon /usr/local/bin/docker_daemo... dockerd --graph=/mnt/scratch/alice.pl... docker-containe -l unix:///var/ru... 7*[{docker-containe}] 14*[{dockerd}] 5*[{slurmstepd}] And the cgroup cat /proc/self/cgroup 11:name=systemd:/user/0.user/6.session 10:hugetlb:/user/0.user/6.session... cat /proc/`pidof dockerd`/cgroup 11:name=systemd:/user/0.user/6.session 10:hugetlb:/user/0.user/6.session...
34 Limitations This is a proof of concept Docker locks /etc/passwd and /etc/group No way to inject user/project names. Only UID and GID available Dockers --userns-remap limits user to one project at a time Limitations in the kernel make this unlikely to change Limitations in the kernel allow no more than 5 lines in subgid(!?) * There is an (arbitrary) limit on the number of lines in the file. As at Linux 3.18, the limit is five lines. - user_namespaces manpage
35 Limitations How about network? How to communicate with containers on different nodes? How about RDMA? Docker is still in very active development Docker August 12, 2015 Docker November 3, 2015 Docker February 4, 2016 Docker April 13, 2016 Docker June 20, 2016 All saw major changes and introduction of concepts and features. Not all features are support in the major distribution Ubuntu/debian Archlinux CentOS
Introduction to Containers
Introduction to Containers Shawfeng Dong Principal Cyberinfrastructure Engineer University of California, Santa Cruz What are Containers? Containerization, aka operating-system-level virtualization, refers
More informationDocker A FRAMEWORK FOR DATA INTENSIVE COMPUTING
Docker A FRAMEWORK FOR DATA INTENSIVE COMPUTING Agenda Intro / Prep Environments Day 1: Docker Deep Dive Day 2: Kubernetes Deep Dive Day 3: Advanced Kubernetes: Concepts, Management, Middleware Day 4:
More informationAn introduction to Docker
An introduction to Docker Ing. Vincenzo Maffione Operating Systems Security Container technologies on Linux Several light virtualization technologies are available for Linux They build on cgroups, namespaces
More informationTravis Cardwell Technical Meeting
.. Introduction to Docker Travis Cardwell Tokyo Linux Users Group 2014-01-18 Technical Meeting Presentation Motivation OS-level virtualization is becoming accessible Docker makes it very easy to experiment
More informationSTATUS OF PLANS TO USE CONTAINERS IN THE WORLDWIDE LHC COMPUTING GRID
The WLCG Motivation and benefits Container engines Experiments status and plans Security considerations Summary and outlook STATUS OF PLANS TO USE CONTAINERS IN THE WORLDWIDE LHC COMPUTING GRID SWISS EXPERIENCE
More informationSingularity: Containers for High-Performance Computing. Grigory Shamov Nov 21, 2017
Singularity: Containers for High-Performance Computing Grigory Shamov Nov 21, 2017 Outline Software and High Performance Computing: Installation/Maintenance of the HPC Software stack Why containers and
More informationState of Containers. Convergence of Big Data, AI and HPC
State of Containers Convergence of Big Data, AI and HPC Technology ReCap Comparison of Hypervisor and Container Virtualization VM1 VM2 appa appb Userland Userland Kernel Kernel Operational Abstraction
More informationEngineering Robust Server Software
Engineering Robust Server Software Containers Isolation Isolation: keep different programs separate Good for security Might also consider performance isolation Also has security implications (side channel
More informationDocker und IBM Digital Experience in Docker Container
Docker und IBM Digital Experience in Docker Container 20. 21. Juni 2017 IBM Labor Böblingen 1 What is docker Introduction VMs vs. containers Terminology v Docker components 2 6/22/2017 What is docker?
More informationBioshadock. O. Sallou - IRISA Nettab 2016 CC BY-CA 3.0
Bioshadock O. Sallou - IRISA Nettab 2016 CC BY-CA 3.0 Containers 2 Docker, LXC, Rkt and Co Docker is the current leader in container ecosystem but not alone in ecosystem Rkt compatible with Docker images
More informationWho is Docker and how he can help us? Heino Talvik
Who is Docker and how he can help us? Heino Talvik heino.talvik@seb.ee heino.talvik@gmail.com What is Docker? Software guy view: Marriage of infrastucture and Source Code Management Hardware guy view:
More informationDocker & why we should use it
Docker & why we should use it Vicențiu Ciorbaru Software Engineer @ MariaDB Foundation * * Agenda What is Docker? What Docker brings to the table compared to KVM and Vagrant? Docker tutorial What is Docker
More informationBasic Linux Security. Roman Bohuk University of Virginia
Basic Linux Security Roman Bohuk University of Virginia What is Linux? An open source operating system Project started by Linus Torvalds kernel Kernel: core program that controls everything else (controls
More informationPresented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory CONTAINERS IN HPC WITH SINGULARITY
Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory gmkurtzer@lbl.gov CONTAINERS IN HPC WITH SINGULARITY A QUICK REVIEW OF THE LANDSCAPE Many types of virtualization
More informationSingularity in CMS. Over a million containers served
Singularity in CMS Over a million containers served Introduction The topic of containers is broad - and this is a 15 minute talk! I m filtering out a lot of relevant details, particularly why we are using
More informationCS-580K/480K Advanced Topics in Cloud Computing. Container III
CS-580/480 Advanced Topics in Cloud Computing Container III 1 Docker Container https://www.docker.com/ Docker is a platform for developers and sysadmins to develop, deploy, and run applications with containers.
More informationIntroduction to containers
Introduction to containers Nabil Abdennadher nabil.abdennadher@hesge.ch 1 Plan Introduction Details : chroot, control groups, namespaces My first container Deploying a distributed application using containers
More informationHigh Performance Containers. Convergence of Hyperscale, Big Data and Big Compute
High Performance Containers Convergence of Hyperscale, Big Data and Big Compute Christian Kniep Technical Account Manager, Docker Brief Recap of Container Technology Brief History of Container Technology
More informationAndrej Filipčič
Singularity@SiGNET Andrej Filipčič SiGNET 4.5k cores, 3PB storage, 4.8.17 kernel on WNs and Gentoo host OS 2 ARC-CEs with 700TB cephfs ARC cache and 3 data delivery nodes for input/output file staging
More informationDeployment Patterns using Docker and Chef
Deployment Patterns using Docker and Chef Sandeep Chellingi Sandeep.chellingi@prolifics.com Agenda + + Rapid Provisioning + Automated and Managed Deployment IT Challenges - Use-cases What is Docker? What
More informationdocker & HEP: containerization of applications for development, distribution and preservation
docker & HEP: containerization of applications for development, distribution and preservation Sébastien Binet LAL/IN2P3 2015-04-13 S. Binet (LAL) docker-hep 2015-04-13 1 / 16 Docker: what is it? http://www.docker.io/
More information4 Effective Tools for Docker Monitoring. By Ranvijay Jamwal
4 Effective Tools for Docker Monitoring By Ranvijay Jamwal CONTENT 1. The need for Container Technologies 2. Introduction to Docker 2.1. What is Docker? 2.2. Why is Docker popular? 2.3. How does a Docker
More informationContainers. Pablo F. Ordóñez. October 18, 2018
Containers Pablo F. Ordóñez October 18, 2018 1 Welcome Song: Sola vaya Interpreter: La Sonora Ponceña 2 Goals Containers!= ( Moby-Dick ) Containers are part of the Linux Kernel Make your own container
More informationIntroduction to Linux
Introduction to Linux University of Bristol - Advance Computing Research Centre 1 / 47 Operating Systems Program running all the time Interfaces between other programs and hardware Provides abstractions
More informationOS Virtualization. Linux Containers (LXC)
OS Virtualization Emulate OS-level interface with native interface Lightweight virtual machines No hypervisor, OS provides necessary support Referred to as containers Solaris containers, BSD jails, Linux
More informationRun containerized applications from pre-existing images stored in a centralized registry
Introduction This examination is based upon the most critical job activities a Docker Certified Associate performs. The skills and knowledge certified by this examination represent a level of expertise
More informationGuillimin HPC Users Meeting
Guillimin HPC Users Meeting July 16, 2015 guillimin@calculquebec.ca McGill University / Calcul Québec / Compute Canada Montréal, QC Canada Outline Compute Canada News Storage Updates Software Updates Training
More informationThink Small to Scale Big
Think Small to Scale Big Intro to Containers for the Datacenter Admin Pete Zerger Principal Program Manager, MVP pete.zerger@cireson.com Cireson Lee Berg Blog, e-mail address, title Company Pete Zerger
More informationInvestigating Containers for Future Services and User Application Support
Investigating Containers for Future Services and User Application Support JLAB CNI NLIT 2018 () Overview JLAB scope What is a container? Why are we interested? Platform-as-a-Service (PaaS) for orchestration
More informationGetting Started with Hadoop
Getting Started with Hadoop May 28, 2018 Michael Völske, Shahbaz Syed Web Technology & Information Systems Bauhaus-Universität Weimar 1 webis 2018 What is Hadoop Started in 2004 by Yahoo Open-Source implementation
More informationCentre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules. Singularity overview. Vanessa HAMAR
Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules Singularity overview Vanessa HAMAR Disclaimer } The information in this presentation was compiled from different
More informationGetting Started With Containers
DEVNET 2042 Getting Started With Containers Matt Johnson Developer Evangelist @mattdashj Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session
More informationSingularity CRI User Documentation
Singularity CRI User Documentation Release 1.0 Sylabs Apr 02, 2019 CONTENTS 1 Installation 1 1.1 Overview................................................. 1 1.2 Before you begin.............................................
More informationPortable, lightweight, & interoperable Docker containers across Red Hat solutions
Portable, lightweight, & interoperable Docker containers across Red Hat solutions Jérôme Petazzoni Tinkerer Extraordinaire Docker, Inc Alexander Larsson Principal Software engineer Red Hat, Inc What? Why?
More informationMidterm Presentation Schedule
Midterm Presentation Schedule October 18 th Aurora, Bash, Sangam October 20 th Flash, Omega, CodeRing October 25th Omni, Aviato, NPComplete Mid Term Presentation Format 25 minutes Be prepared to use the
More informationBright Cluster Manager: Using the NVIDIA NGC Deep Learning Containers
Bright Cluster Manager: Using the NVIDIA NGC Deep Learning Containers Technical White Paper Table of Contents Pre-requisites...1 Setup...2 Run PyTorch in Kubernetes...3 Run PyTorch in Singularity...4 Run
More informationOpportunities for container environments on Cray XC30 with GPU devices
Opportunities for container environments on Cray XC30 with GPU devices Cray User Group 2016, London Sadaf Alam, Lucas Benedicic, T. Schulthess, Miguel Gila May 12, 2016 Agenda Motivation Container technologies,
More informationDocker Security. Mika Vatanen
Docker Security Mika Vatanen 13.6.2017 About me Mika Vatanen, Solution Architect @ Digia 18 years at the industry, 6 months at Digia Established ii2 a Finnish MySpace, top-5 most used web service in Finland
More informationInfrastructure Security 2.0
Infrastructure Security 2.0 $ whoami Infrastructure Security Engineer @ Shopify Certified Kubernetes Administrator twitter.com/jonpulsifer github.com/jonpulsifer Previously Team Lead at CFNOC Network Defense
More informationTHE ROUTE TO ROOTLESS
THE ROUTE TO ROOTLESS THE ROUTE TO ROOTLESS BILL AND TED'S ROOTLESS ADVENTURE THE ROUTE TO ROOTLESS WHAT SECURITY PROBLEM IS GARDEN SOLVING IN CLOUD FOUNDRY? THE PROBLEM IN CLOUD FOUNDRY Public Multi-Tenant
More informationAutomatic Dependency Management for Scientific Applications on Clusters. Ben Tovar*, Nicholas Hazekamp, Nathaniel Kremer-Herman, Douglas Thain
Automatic Dependency Management for Scientific Applications on Clusters Ben Tovar*, Nicholas Hazekamp, Nathaniel Kremer-Herman, Douglas Thain Where users are Scientist says: "This demo task runs on my
More informationDOCKER 101 FOR JS AFFICIONADOS. Christian Ulbrich, Zalari UG
DOCKER 101 FOR JS AFFICIONADOS Christian Ulbrich, Zalari UG AGENDA Docker what is all the craze about? Docker is hard One-Liners Orchestration Outlook Links DOCKER WTF? DOCKER WTF? Docker is light-weight
More informationBEST PRACTICES FOR DOCKER
BEST PRACTICES FOR DOCKER DG-08863-001 _v001 December 2018 Best Practices TABLE OF CONTENTS Chapter 1. NVIDIA Container Best Practices...1 1.1. Hello World For Containers... 1 1.2. Logging Into Docker...
More informationCross platform enablement for the yocto project with containers. ELC 2017 Randy Witt Intel Open Source Technology Center
Cross platform enablement for the yocto project with containers ELC 2017 Randy Witt Intel Open Source Technology Center My personal problems Why d I even do this? THE multiple distro Problem Yocto Project
More informationThe new Docker networking put into action to spin up a SLURM cluster
Multi-host containerised ss HPC cluster The new Docker networking put into action to spin up a SLURM cluster The Bits and Pieces Docker Containers do not spin up their own kernel All containers share the
More informationLinux Essentials Objectives Topics:
Linux Essentials Linux Essentials is a professional development certificate program that covers basic knowledge for those working and studying Open Source and various distributions of Linux. Exam Objectives
More informationThe State of Rootless Containers
The State of Rootless Containers Aleksa Sarai / SUSE Akihiro Suda / NTT @lordcyphar @_AkihiroSuda_ Who are we? Aleksa Sarai Senior Software Engineer at SUSE. Maintainer of runc and several other Open Container
More informationContainerized Cloud Scheduling Environment
University of Victoria Engineering & Computer Science Co-op Work Term Report Fall 2017 Containerized Cloud Scheduling Environment Department of Physics University of Victoria Victoria, BC Tahya Weiss-Gibbons
More informationSingularity: container formats
Singularity Easy to install and configure Easy to run/use: no daemons no root works with scheduling systems User outside container == user inside container Access to host resources Mount (parts of) filesystems
More informationPerl and R Scripting for Biologists
Perl and R Scripting for Biologists Lukas Mueller PLBR 4092 Course overview Linux basics (today) Linux advanced (Aure, next week) Why Linux? Free open source operating system based on UNIX specifications
More informationInfoblox Kubernetes1.0.0 IPAM Plugin
2h DEPLOYMENT GUIDE Infoblox Kubernetes1.0.0 IPAM Plugin NIOS version 8.X August 2018 2018 Infoblox Inc. All rights reserved. Infoblox Kubernetes 1.0.0 IPAM Deployment Guide August 2018 Page 1 of 18 Overview...
More informationSeccomp, network and namespaces. Francesco Tornieri <francesco.tornieri AT kiratech.it>
Seccomp, network and namespaces Francesco Tornieri VM vs Container 2 Namespaces ecc 3 Namespaces ecc man namespaces: A namespaces wraps a global system resource in a
More informationINTRODUCTION TO LINUX
INTRODUCTION TO LINUX REALLY SHORT HISTORY Before GNU/Linux there were DOS, MAC and UNIX. All systems were proprietary. The GNU project started in the early 80s by Richard Stallman Goal to make a free
More informationDocker task in HPC Pack
Docker task in HPC Pack We introduced docker task in HPC Pack 2016 Update1. To use this feature, set the environment variable CCP_DOCKER_IMAGE of a task so that it could be run in a docker container on
More informationThe kernel is the low-level software that manages hardware, multitasks programs, etc.
November 2011 1 Why Use Linux? Save Money Initial purchase and maintenance Resume Linux is used by MANY organizations More choices Tons of Linux operating systems November 2011 2 What is Linux? 1. Contains
More informationDGX-1 DOCKER USER GUIDE Josh Park Senior Solutions Architect Contents created by Jack Han Solutions Architect
DGX-1 DOCKER USER GUIDE 17.08 Josh Park Senior Solutions Architect Contents created by Jack Han Solutions Architect AGENDA Introduction to Docker & DGX-1 SW Stack Docker basic & nvidia-docker Docker image
More informationDockerized Tizen Platform
Dockerized 1 Tizen Platform Copyright 2017 Samsung. All Rights Reserved. Abstract Tizen Pla.orm ECO System Container ECO System Build CI Management (Update) Cloud Monitoring Store Data (DB) Cloud 2 Cloud
More informationShifter at CSCS Docker Containers for HPC
Shifter at CSCS Docker Containers for HPC HPC Advisory Council Swiss Conference Alberto Madonna, Lucas Benedicic, Felipe A. Cruz, Kean Mariotti - CSCS April 9 th, 2018 Table of Contents 1. Introduction
More informationOS Security III: Sandbox and SFI
1 OS Security III: Sandbox and SFI Chengyu Song Slides modified from Dawn Song 2 Administrivia Lab2 VMs on lab machine Extension? 3 Users and processes FACT: although ACLs use users as subject, the OS
More informationDocker for Developers
Docker for Developers Chris Tankersley This book is for sale at http://leanpub.com/dockerfordevs This version was published on 2017-08-07 This is a Leanpub book. Leanpub empowers authors and publishers
More information[Docker] Containerization
[Docker] Containerization ABCD-LMA Working Group Will Kinard October 12, 2017 WILL Kinard Infrastructure Architect Software Developer Startup Venture IC Husband Father Clemson University That s me. 2 The
More informationThe failure of Operating Systems,
The failure of Operating Systems, and how we can fix it. Glauber Costa Lead Software Engineer August 30th, 2012 Linuxcon Opening Notes I'll be doing Hypervisors vs Containers here. But: 2 2 Opening Notes
More informationOS Containers. Michal Sekletár November 06, 2016
OS Containers Michal Sekletár msekleta@redhat.com November 06, 2016 whoami Senior Software Engineer @ Red Hat systemd and udev maintainer Free/Open Source Software contributor Michal Sekletár msekleta@redhat.com
More informationAndroid meets Docker. Jing Li
Android meets Docker Jing Li 1 2 > 50 cities in Europe 3 Developer Story 4 Pain in the Admin provision machines ( e.g. mobile CI ) 5 Containerization vs Virtualization 6 Why Docker? Docker Vagrant Resource
More informationContainer-based virtualization: Docker
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Container-based virtualization: Docker Corso di Sistemi Distribuiti e Cloud Computing A.A. 2018/19
More informationCS197U: A Hands on Introduction to Unix
CS197U: A Hands on Introduction to Unix Lecture 4: My First Linux System Tian Guo University of Massachusetts Amherst CICS 1 Reminders Assignment 2 was due before class Assignment 3 will be posted soon
More informationNetwork softwarization Lab session 2: OS Virtualization Networking
Network softwarization Lab session 2: OS Virtualization Networking Nicolas Herbaut David Bourasseau Daniel Negru December 16, 2015 1 Introduction 1.1 Discovering docker 1.1.1 Installation Please launch
More informationIntroduction To Linux. Rob Thomas - ACRC
Introduction To Linux Rob Thomas - ACRC What Is Linux A free Operating System based on UNIX (TM) An operating system originating at Bell Labs. circa 1969 in the USA More of this later... Why Linux? Free
More informationHTCondor: Virtualization (without Virtual Machines)
HTCondor: Virtualization (without Virtual Machines) Brian Bockelman HTCondor Week 2013 Dictionary Definition vir tu al ize [vur-choo-uh-lahyz] verb to create a virtual version of (a computer, operating
More informationSecuring Containers on the High Seas. Jack OWASP Belgium September 2018
Securing Containers on the High Seas Jack Mannino @ OWASP Belgium September 2018 Who Am I? Jack Mannino CEO at nvisium, since 2009 Former OWASP Northern Virginia chapter leader Hobbies: Scala, Go and Kubernetes
More informationFlatpak a technical walk-through. Alexander Larsson, Red Hat
Flatpak a technical walk-through Alexander Larsson, Red Hat What is Flatpak? apps for the Linux Desktop Distribute your app Run it anywhere Build in anywhere Run it sandboxed How is this different from
More informationBEST PRACTICES FOR DOCKER
BEST PRACTICES FOR DOCKER DG-08863-001 _v001 March 2018 Best Practices TABLE OF CONTENTS Chapter 1. Docker Best Practices with NVIDIA Containers... 1 1.1. Prerequisites... 1 1.1.1. Hello World For Containers...
More informationDockerfile Best Practices
Dockerfile Best Practices OpenRheinRuhr 2015 November 07th, 2015 1 Dockerfile Best Practices Outline About Dockerfile Best Practices Building Images This work is licensed under the Creative Commons Attribution-ShareAlike
More informationImproving User Accounting and Isolation with Linux Kernel Features. Brian Bockelman Condor Week 2011
Improving User Accounting and Isolation with Linux Kernel Features Brian Bockelman Condor Week 2011 Case Study: MPD The MPICH2 library is a common implementation of the MPI interface, a popular parallel
More informationAutomating the Build Pipeline for Docker Container
Automating the Build Pipeline for Docker Container Nikolai Reed, Jürgen Walter, and Samuel Kounev University of Würzburg November 9-10, 2017 SSP 2017 Karlsruhe, Germany 08.11.17 2 Reproducibility Problem
More informationIntroduction to Container Technology. Patrick Ladd Technical Account Manager April 13, 2016
Introduction to Container Technology Patrick Ladd Technical Account Manager April 13, 2016 Container Technology Containers 3 "Linux Containers" is a Linux kernel feature to contain a group of processes
More informationDocker. Master the execution environment of your applications. Aurélien Dumez. Inria Bordeaux - Sud-Ouest. Tuesday, March 24th 2015
Docker Master the execution environment of your applications Aurélien Dumez Inria Bordeaux - Sud-Ouest Tuesday, March 24th 2015 Aurélien Dumez Docker 1 / 34 Content 1 The bad parts 2 Overview 3 Internals
More informationIt s probably the most popular containerization technology on Linux these days
Docker & Perl What is docker? It s probably the most popular containerization technology on Linux these days It s somewhere between chroot jails and virtual machines. Effectively lightweight virtual machines
More informationSoftware containers are likely to become a very important tool over the
MARK LAMOURINE Mark Lamourine is a senior software developer at Red Hat. He s worked for the last few years on the OpenShift project. He s a coder by training, a sysadmin and toolsmith by trade, and an
More informationBuilding A Better Test Platform:
Building A Better Test Platform: A Case Study of Improving Apache HBase Testing with Docker Aleks Shulman, Dima Spivak Outline About Cloudera Apache HBase Overview API compatibility API compatibility testing
More informationRunning Docker applications on Linux on the Mainframe
Running Docker applications on Linux on the Mainframe Jay Brenneman - rjbrenn@us.ibm.com 10 August, 2015 Trademarks The following are trademarks of the International Business Machines Corporation in the
More information/ Cloud Computing. Recitation 5 September 27 th, 2016
15-319 / 15-619 Cloud Computing Recitation 5 September 27 th, 2016 1 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 2.1, OLI Unit 2 modules 5 and 6 This week
More informationDevOps in the Cloud A pipeline to heaven?! Robert Cowham BCS CMSG Vice Chair
DevOps in the Cloud A pipeline to heaven?! Robert Cowham BCS CMSG Vice Chair Agenda Definitions, History & Background Cloud intro DevOps Pipelines Docker containers Examples 2 Definitions DevOps Agile
More informationAGILE DEVELOPMENT AND PAAS USING THE MESOSPHERE DCOS
Sunil Shah AGILE DEVELOPMENT AND PAAS USING THE MESOSPHERE DCOS 1 THE DATACENTER OPERATING SYSTEM (DCOS) 2 DCOS INTRODUCTION The Mesosphere Datacenter Operating System (DCOS) is a distributed operating
More informationPBS Pro with Docker Integration
PBS Pro with Docker Integration Arun Grover Technical Manager 31 August 2015 Table of Contents 1. Why Integrate Docker and PBS?... 3 2. How PBS Runs Jobs Using Docker... 4 3. Steps to Integrate PBS and
More information/ Cloud Computing. Recitation 5 February 14th, 2017
15-319 / 15-619 Cloud Computing Recitation 5 February 14th, 2017 1 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 2.1, OLI Unit 2 modules 5 and 6 This week
More informationFEniCS Containers Documentation
FEniCS Containers Documentation Release 1.0 FEniCS Project Jan 29, 2018 Contents 1 Quickstart 3 2 Introduction 5 2.1 What is Docker?............................................. 5 2.2 Installing Docker.............................................
More informationA Hands on Introduction to Docker
A Hands on Introduction to Docker Len Bass A Hands on introduction Introduction to to Docker May 2017 1 4, Len 2017 Bass 2017 Len Bass 1 Setting expectations This is an introduction to Docker intended
More informationCONTAINERIZING JOBS ON THE ACCRE CLUSTER WITH SINGULARITY
CONTAINERIZING JOBS ON THE ACCRE CLUSTER WITH SINGULARITY VIRTUAL MACHINE (VM) Uses so&ware to emulate an en/re computer, including both hardware and so&ware. Host Computer Virtual Machine Host Resources:
More informationDocker and Security. September 28, 2017 VASCAN Michael Irwin
Docker and Security September 28, 2017 VASCAN Michael Irwin Quick Intro - Michael Irwin 2011 - Graduated (CS@VT); started full-time at VT Sept 2015 - Started using Docker for QA June 2016 - Attended first
More informationbistro Documentation Release dev Philippe Veber
bistro Documentation Release dev Philippe Veber Oct 10, 2018 Contents 1 Getting started 1 1.1 Installation................................................ 1 1.2 A simple example............................................
More informationAsterisk & the Docker revolution Some lessons from the trenches
Asterisk & the Docker revolution Some lessons from the trenches Asterisk Africa Johannesburg - March 14, 2018 Presented by: Lenz Emilitri Founder, Loway @lenz Today s presentation Docker Benefits How it
More informationIntroduction to the shell Part II
Introduction to the shell Part II Graham Markall http://www.doc.ic.ac.uk/~grm08 grm08@doc.ic.ac.uk Civil Engineering Tech Talks 16 th November, 1pm Last week Covered applications and Windows compatibility
More informationContainers and isolation as implemented in the Linux kernel
Containers and isolation as implemented in the Linux kernel Technical Deep Dive Session Hannes Frederic Sowa Senior Software Engineer 13. September 2016 Outline Containers and isolation
More informationGitLab-CI and Docker Registry
GitLab-CI and Docker Registry Oleg Fiksel Security Consultant @ CSPI GmbH oleg.fiksel@cspi.com oleg@fiksel.info Matrix: @oleg:fiksel.info FrOSCon 2017 AGENDA ABOUT INTRODUCTION GitLab 101 Deploying on-premise
More informationLSST software stack and deployment on other architectures. William O Mullane for Andy Connolly with material from Owen Boberg
LSST software stack and deployment on other architectures William O Mullane for Andy Connolly with material from Owen Boberg Containers and Docker Packaged piece of software with complete file system it
More informationSection 1: Tools. Contents CS162. January 19, Make More details about Make Git Commands to know... 3
CS162 January 19, 2017 Contents 1 Make 2 1.1 More details about Make.................................... 2 2 Git 3 2.1 Commands to know....................................... 3 3 GDB: The GNU Debugger
More informationIntroduction to Containers. Martin Čuma Center for High Performance Computing University of Utah
Introduction to Containers Martin Čuma Center for High Performance Computing University of Utah m.cuma@utah.edu Overview Why do we want to use containers? Containers basics Prepare your computer for containers
More informationArup Nanda VP, Data Services Priceline.com
Jumpstarting Docker Arup Nanda VP, Data Services Priceline.com My application worked in Dev but not in QA Will it work in production? I need an environment right now No, I can t wait for 2 weeks I just
More informationLENS Server Maintenance Guide JZ 2017/07/28
LENS Server Maintenance Guide JZ 2017/07/28 Duty Maintain LENS server with minimum downtime Patch critical vulnerabilities Assist LAB member for using the LENS services Evaluate for custom requirements
More information