Guillimin HPC Users Meeting

Transcription

1 Guillimin HPC Users Meeting July 16, 2015 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Outline Compute Canada News Storage Updates Software Updates Training News Special Topics: Docker 2

3 Compute Canada News Reminder: Research Platforms and Portals (RPP) 2016 Call for Proposals process of compute or storage resources required to support a platform or portal Application category examples: Resources for larger communities of researchers Applications that provide a public platform using CC computing or storage Groups with international agreements for multi-year computing or storage commitments Groups providing shared datasets accessible using noncompute Canada interfaces / portals Important Timelines Letter of Intent submission period: Aug 4-13 Inquiries: rac@computecanada.ca or ask us 3

4 Storage Updates Reminder: New modular storage hardware Base enclosure: 60x 8 TB 7200 rpm SAS drives Expansion enclosure(s): 84 x 8 TB SAS drives full integration within the GPFS Environment Installation in progress Target Use Case support for groups or individual researcher projects looking to add disk capacity to Guillimin through external funding grants (NSERC, CFI, ) Please contact us at guillimin@calculquebec.ca if you or your group would like to discuss acquisition of additional storage 4

5 Software Update New Installations blitz/ C++ class library for scientific computing ffmpeg/2.7 - multi-media framework opencv/ open source computer vision mesa/ d graphics library duplicity/ utility for local or remote backup 5

6 Training News See Training at for our calendar of training and workshops for 2015 and to register all materials from previous workshops are available online suggestions for training? Please let us know! Upcoming: August 6 - Visualization Recently Completed: July 9 - MapReduce with Hadoop for Big Data June 4 - Advanced OpenMP May Calcul Quebec Spring School 6

7 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 7

8 Guillimin HPC Users Meeting July 16, 2015 Docker Guillimin McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

9 Docker Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. Run on Mac, Windows, Linux 9

10 Public Repository of Docker Containers 10

11 Don t like CentOS6? docker run centos:7 bash 11

12 The NeuroImaging Analysis Kit (NIAK) docker run simexp/niak: (from Dr P. Bellec) user/application:version 12

13 The NeuroImaging Analysis Kit (NIAK) cont d 13

14 Dockerfile: how to build a Docker container FROM simexp/octave:3.8.1 MAINTAINER Pierre Bellec <pierre.bellec@criugm.qc.ca> # Install NIAK RUN cd /home/ && wget zip && unzip v zip && rm v zip RUN mv /home/niak /home/niak # Install PSOM RUN cd /home/niak/extensions && wget org/frs/download.php/7065/psom zip && unzip psom zip && rm psom zip # Install BCT RUN cd /home/niak/extensions && wget com/site/bctnet/home/functions/bct.zip && unzip BCT.zip && rm BCT.zip # Build octave configure file RUN echo addpath\(genpath\(\"/home/niak/\"\)\)\; >> /etc/octave. conf 14

15 Docker on Windows or Mac Run boot2docker (a VirtualBox VM with a minimum Linux kernel) 15

16 bbcp on Windows bbcp doesn t have a Windows version 16

17 bbcp on Windows 17

18 You need to know A container is seen by the host OS as one or several normal processes Application running in one container can not see or is not aware of any other container running on same host A container is ephemeral. All data written inside the container will be lost when the container exits: The container can write to a network file or cloud storage (or scp, rsync to somewhere else) The container can write to a file on the local host (e.g /sb/home/xxx) through a Docker volume mounted at the start 18

19 In Summary Docker is a software packaging and distribution technology Run on Mac, Windows, Linux, clouds Alternative to RPM, library modules, CVMFS repositories Promote micro-services architecture: Each container should only do one thing Don t even include sshd if not required Orchestration feature allow to define and launch hundred or thousands of containers doing different things But first and foremost it is a sysadmin tool to deploy software Requires root privilege 19

20 Our udocker Wrapper to docker run command: force to run in interactive mode (-it --rm) force $HOME to user home on /sb/home force docker to run under user s UID and GID erase /etc/sudoers inside container to prevent user to run sudo Docker is enabled on aw-4r12-n[01-40] and aw-4r13-n[0110] nodes login normally through login nodes qsub -I -X -l nodes=1:ppn=16:mics=2 -l walltime=1:00:00 20

21 Next Steps Help us evaluate the technology and identify use cases Contact us via regular support address This is still an experiment Docker evolves rapidly and might address root access issue in the future Todo: better parser allow project space container cleanup after crash how to deal with batch/background containers 21

22 For your reference Docker workflow 22

23 VMs vs Docker Container 23

24 Docker on Linux vs on Mac/Windows Note: Docker client is not the container 24

25 Native Docker (c) vs Others 25

26 Docker Architecture No longer depends on LXC 26

27 Moments in Container history

28 References KVM and Docker LXC Benchmarking with OpenStack com/presentation/d/1hgnknby8kylzojx2mr20ztl4nrswnflajj4xsg4c_ hu/edit#slide=id.ga38ef073e_5_51 ~jpetazzo/index Docker Containers Performance in VMware vsphere qid=927abc65-0e1d-4dda-9ef58144f00f01c6&v=default&b=&from_search=1 Docker Misconceptions 27