Introduction to Advanced Research Computing (ARC)

Transcription

1 Introduction to Advanced Research Computing (ARC) September 29, 2016 By: Pier-Luc St-Onge 1

2 Financial Partners 2

3 Setup for the workshop 1. Get a user ID and password paper (provided in class): ##: ********** 2. Access to local computer (replace ## and with appropriate values, is provided in class): a. User name: csuser## (ex.: csuser99) b. (ex.: sec@[s99) 3. Slides: 3

4 Outline Introduction to ARC Laptop vs HPC Cluster Moving Data With Globus Software Development Types of Parallelism Scientific Software and Modules Best Practices The Scheduler and Job Submission Compute Canada and Calcul Québec 4

5 Introduction to ARC 5

6 Introduction This workshop is built from the HPCS 2015 Introduction to ARC workshop: HPCS website: Etherpad with extra information: Presentation: Instructors and participants Name and research domain 6

7 Other research domains Genomics Molecular Dynamics (MD) Finite Element Analysis (FEA) Computational Fluid Dynamics (CFD) Astronomy and Astrophysics Geospatial Data Analysis 3D renderings, Image Analysis Digital Humanities (DH) Artificial Intelligence 7

8 Advanced Research Computing High Performance Computing (HPC) Parallel code Running code on accelerators Profiling code Management of massive data Complete pipelines for jobs submission Sharing data Web portals for special projects Cloud computing / storage 8

9 ARC is also about having enough Resources Problem: we want to compute a dense matrix multiplication C = A * B Each matrix is made of * floating point numbers in double precision How much memory do you need for this job? a) 3.2 GB b) 8.6 GB c) 12.9 GB d) 25.8 GB 9

10 Different Tools for Different Needs Portable computer HPC Compute node 2-4 cores 4-16 GB Local storage, USB 128GB to 2TB Wi-Fi and Ethernet General usage Cloud storage Gaming GPU 32 bits Direct access 8-32 cores GB Network storage 150TB to 5PB Ethernet and fast network (Infiniband) Multi-node tasks Accelerators 64 bits, ECC mem. Access by queued jobs 10

11 Zoom on a HPC Compute Node RAM Proc. Proc. RAM Infiniband RAM Local hard-drive (scratch) RAM Proc. Proc. Ethernet RAM Proc. Proc. RAM Accelerator RAM RAM Proc. Proc. 11

12 Why Infiniband? Some tasks need more than one compute node Split workload, memory usage and storage access High bandwidth Ethernet: 1Gbps (~100MB/s) Infiniband: Gbps (2-10GB/s) Best for I/O on large files Reduce latency Ethernet: ~30µs (33k iops = I/O operations per sec.) Infiniband: 2µs (500k iops) Best for several small communications 12

13 Storage Available on Compute Nodes Bandwidth RAMDISK Local scratch Project space Scratch space Capacity (Not to scale) Latency 13

14 Typical Cluster Job Login node Scheduler Job Nodes Ethernet & Infiniband Internet Storage home projet scratch ssh 14

15 Typical Cluster Usage 1. Connect to a login node a. Import data b. Compile code or install applications 2. Submit jobs to the scheduler a. Define needed resources: processors, accelerators, memory, walltime limit b. Wait in queue - monitor your position c. Automatic execution of your jobs on compute nodes 3. Get your results 15

16 Typical Environment on Cluster The operating system is usually Linux (CentOS, Scientific Linux, ) To connect to a cluster, we usually use an encrypted session through SSH (Secure SHell) To transfer data safely: SCP (Secure CoPy) and others tools Some web portals may also provide an access Development environment Multiple text editors Scientific software Version control software Tools for transferring large amount of data 10Gbps, 40Gbps or 100Gbps between sites 16

17 Moving Data With Globus 17

18 Moving Data Over SSH Secure CoPy with SCP or WinSCP: Most clusters only allow SSH connections SSH: Secure SHell (username and password) SCP uses the SSH protocol to copy files Simple tool for small amount of data (<1GB) On Windows: use WinSCP or FileZilla Two panels, where one side shows your files on the cluster, then drag-and-drop files or folders On Linux or Mac OS X: use the scp command: scp file.txt scp -pr 18

19 Globus - Why? Secure CoPy with SCP or WinSCP: One process and one thread Fails on dropped connections BBCP - parallel transfers Need a calibration for the round-trip time Command line with multiple arguments Globus - Web interface and CLI No need for calibration Convenient Web interface Confirmation after the transfer Possible to share data with other users 19

20 Go to Compute Canada Research Portal CC > Research Portal > National Services > Data Movement (Globus) Create an account if you do not have Or Log in to your account if you already have an account 20

21 Account Creation and Login Go to: Select Compute Canada if you have a CCDB account, or Globus ID if not. To create a Globus ID account: 21

22 Compute Canada Login and Globus ID Account Creation 22

23 Exercise: Sign-up! Globus ID credentials are stored on Globus: Can be a different username than the one on CCDB, or it can be the same. But both accounts (on CCDB and Globus ID) are different! For sure, use a different password! Put the green Post-it on the computer screen when done 23

24 Globus Home Interface To Logout Endpoint Viewports Transfer options Globus Endpoint on your computer 24

25 Choosing an Endpoint at Compute Canada 25

26 Authentication to an Endpoint The authentication is done on a server corresponding to the endpoint Globus will never know your credentials for a given cluster/endpoint The authentication server only grants access to Globus. 26

27 Files and Directories 27

28 Exercise: Import Pictures Connect to computecanada#guillimin Use the class account credentials (on your paper) class## ********** Connect to intro-arc_guillimin_plstonge Select both exercises and pictures folders Click on the blue button to initiate the transfer to your (class##) home directory Wait for the report When the transfer is completed, put the green Post-it on top of the computer screen 28

29 Confirmation TASK DETAILS Task ID: 5ef1f2c6-81f5-11e6-afd b92c261 Task Type: TRANSFER Status: SUCCEEDED Source: intro-arc_guillimin_plstonge (...) Destination: computecanada#guillimin (...) Label: n/a 29

30 Transfer Report 30

31 Managing Endpoints 31

32 Create a Local Endpoint 32

33 Create a Local Endpoint 1. Choose a unique Endpoint name a. Generate the Setup Key b. Copy the Setup Key in a text editor 2. Download and Install Globus Connect Personal a. Start it and paste the Setup Key 3. A Globus icon ( g) appears in the taskbar a. [Right-]click -> Preferences -> Access b. Select accessible directories you want to export 4. Use your Endpoint: user#uniquename 33

34 Sharing Option 34

35 Software Development Types of Parallelism 35

36 Vocabulary Serial tasks Any task that cannot be split in two simultaneous sequences of actions Examples: starting a process, reading a file, any communication between two processes Parallel tasks Data parallelism: same action applied on different data. Could be serial tasks done in parallel. Process parallelism: one action on one set of data. Action split in multiple processes or threads. Data partitioning: rectangles or blocks 36

37 Parallel tasks Shared memory Multiple threads share the same memory space in a single process: full read and write access. Distributed memory Each process has its own memory space Information is sent and received by messages Parallel efficiency (scaling) Amdahl s law: how long does it take to compute a task with an infinite number of processors? Gustafson's law: what size of problem can we solve in a given time with N processors? 37

38 Software Development Scientific Software and Modules 38

39 Connecting to Guillimin - SSH Client From Linux, open a Terminal window From Mac OS X, open a Terminal window From Windows, install PuTTY, MobaXterm or any other SSH (2.0 compliant) client /download.html putty.exe is a standalone executable Simply run putty.exe 39

40 Connecting to Guillimin Terminal Server name: guillimin.calculquebec.ca Username: class## ## = Two digits on the first line of the small piece of paper Password: ********** Sequence of 10 characters on small piece of paper Linux/Mac OS X example: ssh class99@guillimin.calculquebec.ca Accept the server s key fingerprint 40

41 Connecting to Guillimin PuTTY 41

42 Scientific Software You will use or create scientific software Applications Libraries Make sure you are using a code language widely used in your research domain Make sure your code is well documented and easy to maintain Keep in mind that future researchers are going to use your software Do not reinvent the wheel: use modules! 42

43 Modules Exercise: module avail Compilers: GNU, Intel, LLVM, PGI, Java Language interpreters: Python, R Libraries and other applications: OpenMPI, MVAPICH2, Boost MKL, FFTW GROMACS, NAMD, Quantum-Espresso, NetCDF Other tools: Paraview, GNU-Parallel, GDB, Valgrind 43

44 Exercises with Modules which gcc module add gcc/4.8.2 module list which gcc module rm gcc/4.8.2 module add ifort_icc openmpi echo $LD_LIBRARY_PATH echo $CC module purge 44

45 New Lmod/EasyBuild-based module system Default set of modules on Colosse Both legacy and Lmod modules are available on Guillimin Exercise on Guillimin: 1. Load the main toolchain Intel + OpenMPI + MKL: module load iomkl/2015b 2. List loaded modules: module list 3. You now have access to much more modules: module avail 45

46 Introduction to GNU-Parallel Automatic online process scheduler for serial tasks User provides a command line template and one or multiple sequences of arguments For N cores on a compute node, GNU-Parallel will start N processes As soon as one process is complete, it will start the next process according to values in sequences Example: module load parallel/ parallel echo {1} ::: $(seq 1 960) less 46

47 Introduction to Threads Threads are used for parallel tasks on a single node One process, one shared memory space, but multiple threads Synchronization issues, like race conditions Two threads reading or writing to the same variable Instead of POSIX threads: OpenMP Threading Building Blocks 47

48 Introduction to MPI Message Passing Interface (MPI) is a standard for passing messages between multiple processes on multiple nodes Each process can know its rank number Same set of functions, but different implementations MPICH2, MVAPICH2, OpenMPI, Intel MPI,... Example: module load iomkl/2015b mpiexec -n 4 hostname 48

49 Software Development Best Practices 49

50 Version Control & Revision Control Bad practice: Keeping multiple versions in sub-directories Sending a specific version by Trying to merge two versions manually Good practice: using a version control system Choose one that fits your project: Git (git) Mercurial (hg) Subversion (svn) Repository: shared on a remote server 50

51 Exercises with git mkdir -p ~/projecta; cd ~/projecta; module add git git init # Local repository touch script.sh; git status git add script.sh # Ready to commit git commit -m "intro-arc" # Do commit Add line env > env.log in script.sh git git git git diff # See differences add script.sh # Ready to commit commit -m "Cleaner job" # Do commit log # Show past commits 51

52 One Last Good Practice Using SSH Keys Having access with a local passphrase: ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (~/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Copy the public key to a remote site (id_rsa.pub): scp.ssh/id_rsa.pub <user>@<server>: ssh <user>@<server> \ "cat id_rsa.pub >>.ssh/authorized_keys" 52

53 Exercise - Code Compilation Go to the exercises folder cd ~/exercises Load the following modules: iomkl/2015b Boost/ Python ImageMagick/ Build the application: make 53

54 The Scheduler and Job Submission 54

55 The Scheduler Jobs are not running on login nodes You need to submit your jobs to the scheduler: Resources you need: n processors, memory, accelerators Job duration Account name The script to run on worker nodes Where you want your results 55

56 Schedulers in Calcul Québec Job component Torque + Maui/Moab Submission command Torque: qsub Maui/Moab: msub Account -A xyz-123-aa Processors (small L) Some clusters force full nodes -l nodes=n:ppn=p -l procs=pp Memory (small L) -l pmem=1700m -l mem=20g Job duration (small L) -l walltime=d:hh:mm:ss Job name -N name Where you want your results -o output_path -e error_path your_app > out.log 56

57 Environment Variables During Job Variable $PBS_JOBID $PBS_JOBNAME $PBS_NODEFILE Description Job ID Ex.: server.name Job name set by -N Ex.: introarc The name of the file that contains a list of assigned nodes for the job $PBS_O_WORKDIR The current working directory ($PWD) when the job has been submitted. $PBS_ARRAYID One of the values specified by the job array: -t 1-3,6,

58 Exercise - Manage your Jobs Torque - jobs and job array details qstat -u $USER qstat -t -u $USER Maui / Moab - jobs showq -u $USER (Optional) Torque - cancel job qdel jobid (Optional) Maui / Moab - cancel job mjobctl -c jobid canceljob jobid 58

59 Exercises: Edit and Submit a Job Scripts Go to the exercises folder Check the README.en file for general instructions Start with the exercise 1-base. All the instructions are in the README.en file The solution is in solution.sh Then try exercises 2, 3 and/or 6 For exercise 2, a description of & and wait is in the README.en file of the exercise You need to understand the solution of the exercise 3 to do exercise 4 59

60 Compute Canada and Calcul Québec 60

61 Compute Canada Four consortia (from West to East): WestGrid British Columbia, Alberta, Saskatchewan, Manitoba Compute Ontario (SciNet + SHARCNET + HPCVL) Ontario Calcul Québec Québec ACENET New Brunswick, Newfoundland, Nova Scotia, Prince Edward Island 61

62 Example - Calcul Québec Consortium Compute Canada Calcul Québec - Compute Resources Briarée & Hadès (U. Montréal) Colosse & Helios (U. Laval) Guillimin (McGill) MS2 & MP2 (U. Sherbrooke) Calcul Québec - Human Resources About 40 employees ARC analysts System administrators Scientific and operational directors 62

63 Biggest Clusters - GP1, GP2 and GP3 are coming soon... Name / Institution Cores Summary MP2 / U. Sherbrooke cores/node, 32+ GB/node, QDR IB GPC / U. Toronto cores/node, 16+ GB/node, D/QDR IB Guillimin / McGill U ,16 cores/node, 24+ GB/node, QDR IB Orcinus / U. British Columbia ,12 cores/node, 16+ GB/node, D/QDR IB Orca / SHARCNET ,24 cores/node, 32 GB/node, QDR IB Colosse / U. Laval cores/node, 24+ GB/node, QDR IB Briarée / U. Montréal cores/node, 24+ GB/node, QDR IB Parallel / U. Calgary cores/node, 24 GB/node, QDR IB Example of job requirements: 8 or 16 cores/node, 30 GB/node, QDR IB 63

64 Roles and Allocations CCRI : abc ABCD University Alloc.: abc-123-aa (default alloc.) CCRI : def HIJK University Alloc.: def-456-ab 50 core*years xyz Prof Prof CCRI : abc DEFG University Alloc.: abc-123-ac 200 core*years xyz Student 64

65 About Allocations Amount of compute cycles (only a target): 1 core provides 2 to 3 GHz (cycles per second) Core*seconds, core*hours or core*years (cy) Each site has a default allocation (up to 80 cy) Amount of storage space (hard limit): Private scratch space (default: up to 1TB) Group shared project space (default: up to 1TB) Special allocations Resource Allocation Competition (RAC 2017) ource-allocation-competitions/ 65

66 How to Access Resources The Complete Procedure CCDB: A professor creates an account in CCDB Request role of Principal Investigator (PI) - CCRI (Optional) Request a special allocation (RAC) Special allocation granted on a specific cluster A student creates an account in CCDB Request sponsored role - need to provide the PI s CCRI Apply for a consortium account (ex.: Calcul Québec) In the Calcul Québec portal, activate your access to any cluster Choose clusters based on special allocations 66

67 Usage Policy Each user has his own account; never share your private account! No heavy process (in time and space) on login nodes Resources must be used in an optimal way Large parallel jobs must use tools that scale well Cold data should not stay on disk Maximize the amount of research done in Canada Must acknowledge Compute Canada and any regional partner organization 67

68 Service and Support Training workshops: Future: Past: Documentation: Documentation Support: 68

69 Conclusion 69

70 Conclusion Need help? WestGrid: SciNet: SHARCNET: Calcul Québec: ACENET: Useful information to send us: User name Job ID (if applicable) Path to {job script, output/error files, code} or as attached files 70