Introduction Matlab Compiler Matlab & Condor Conclusion. From Compilation to Distribution. David J. Herzfeld
|
|
|
- Geoffrey Doyle
- 5 years ago
- Views:
Transcription
1 Matlab R and MUGrid From Compilation to Distribution David J. Herzfeld david.herzfeld@marquette.edu Marquette University Department of Biomedical Engineering July 15, 2010 David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
2 Why Matlab R? Matlab R is used extensively at MU. Computational & Financial Modeling Rapid Prototyping Comparatively Easy to Learn Matlab R problems can quickly become large (parameter exploration). Ideal candidate for high throughput computing! David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
3 Outline 1 Introduction 2 Matlab Compiler 3 Matlab & Condor David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
4 We won t be covering... In-depth Matlab R Programming Matlab R Parallel Computing Toolbox Matlab R Distributed Computing Server David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
5 Assumed Knowledge Basic understanding of shell scripting (Bash) See notes from Sessions 1 & 2 Basic understanding of Condor R HTC See notes from Session 3 Basic understanding of Matlab R David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
6 Environment & Examples Examples will be using Père Techniques can be generalized to the rest of MUGrid and beyond Login to Père From Linux/Unix/Mac: ssh Use an SSH client from Windows Copy Examples $ cp -r /group/hpc-bootcamp/2010summer/session5 \ ~/matlab_examples David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
7 Hello World! for Matlab R example1a.m disp([ Hello World! ]); disp([ My random number: num2str(rand())]); David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
8 Modules on Père Where s Matlab R? $ which matlab /usr/bin/which: no matlab in \ (/opt/pgi/linux86-64/10.3/bin:\ /usr/mpi/gcc/openmpi-1.2.8/bin:\ /usr/kerberos/bin:\ /opt/condor-7.4.1/bin:\ /opt/condor-7.4.1/sbin:\ /usr/local/bin:/bin:/usr/bin:\ /home/marqnet/8589herzfed/bin) David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
9 Modules on Père Use modules to set paths and environment variables See List available modules: module avail Notice the matlab module David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
10 example1a.m on Père (hpc-cn1) Running example1a.m $ module load matlab $ which matlab /group/hpc-share/matlab/bin/matlab $ matlab -nodesktop -nosplash -nojvm -nodisplay >> example1 Hello World! My random number: David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
11 example1a.m Required Steps 1 Set up the environment 2 Start Matlab R 3 Run the script David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
12 General Required Steps 1 Check for required input files & code 2 Set up the environment 3 Start Matlab R 4 Run the script 5 Tear down: 1 Stop Matlab R 2 Remove temporary/unneeded files David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
13 Creating a Bash Script example1a.sh #!/bin/bash if [! -f example1a.m ] then echo "ERROR: Unable to find example1a.m" >&2 exit 1 fi module load matlab matlab -nodesktop -nojvm -nodisplay -r example1a # Note: script exits with status of the matlab command # Note: We do not want the.m extension here David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
14 Running example1a.sh Shell Commands $ chmod +x example1a.sh $./example1a.sh David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
15 Issues Matlab R did not close after showing the random number If we could not see the output, it would look like the program hung Potential solutions: Add an exit at the end of the script Pipe the contents to stdin $ matlab -nodesktop -nojvm -nodisplay < example1a.m David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
16 example1a.m example1b.m example1b.m disp([ Hello World! ]); disp([ My random number: num2str(rand())]); exit; David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
17 Current Problem example1a.m and example1b.m require a working copy of Matlab R Concurrent use of Matlab R or toolbox licenses may be an issue on some systems Matlab R is expensive Due to the tightly-coupled nature of current clusters, Matlab R Distributed Computing Server would be required = $$$ David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
18 Potential Solutions Simple Matlab R Scripts: Try a Matlab R alternative octave ANSI C Python with NumPy & SciPy Advanced Matlab R Scripts: Matlab R Compiled Executable David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
19 Introduction to the Matlab R Compiler Matlab R Compiler Limited GUI Support David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
20 Guidelines for Deployable Code Dynamic code is NOT allowed NO calling of specific files via modified paths NO help NO loadlibrary unless you provide function prototypes Common pitfalls: Mismatched Matlab R runtime (MCR) and development environments Global variables whose names are used in other functions David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
21 Another Simple Example example2.m function m = example2(n) % Returns a random matrix of size n if ischar(n) n = str2num(n); end m = rand(n) David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
22 Makefile Makefile example2: example2.m mcc -m -R -nodisplay -R -nojvm example2.m clean: rm -f example2 rm -f *_main.c *_data.c *.prj readme.txt *.log run_*.sh David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
23 mcc Options -m: Generate a standalone executable (not a library) -R: Specify runtime options -nodisplay: No need to display anything. -nojvm: Since there is no display, there is little need for Java Additional Useful Flags: -a: Add a filename or folder to an archive -I: Include a path when searching for m files (include external toolboxes, for example) -l: Use instead of -m to generate a library rather than a standalone executable See mcc -help for more details David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
24 Running example2 On Père (hpc-hn1) $ make $./example2 5 On Other Nodes $ module load matlab $ make $./example2 5 David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
25 Bash Script for Compiled Matlab R Executable example2.sh #!/bin/bash if [! -f example2 ] then echo "ERROR: Unable to find example2" >&2 exit 1 fi if [! -x example2 ] then chmod +x example2 fi module load matlab./example2 $1 David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
26 When to Distribute Matlab R Code? Statistical analysis of simulation/model outcome ( I need to run this Matlab R model 500 times ) Problem is composed of independent tasks (parameter exploration) Datasets are very large (memory limitations), but some/all analyses can be performed independently Many others... David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
27 Why Condor R & Matlab R? Independent tasks are ideal for high throughput computing (HTC). Condor R is a great HTC scheduler Matlab R + Condor R = Identical Condor R techniques can be used across MUGrid Scales to national grids for really big problems. Topic of Session 7! David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
28 X = A i, B i, and C are square. i is very large (10, 000+) N Ai B i + C i=1 Loading a.mat file containing all A and B gives:??? Error using ==> load Out of memory. Type HELP MEMORY for your options. David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
29 Serial Matlab R Code example3.m function [] = example3() load( A.mat ); % Get A (3D matrix) load( B.mat ); % Get B (3D matrix) load( C.mat ); % Get C (matrix) M = size(a, 1); X = zeros(m); % Create MxM matrix for i = 1:size(A, 3) X = X + sqrt(a(:,:,i) * B(:,:,i) + C); end disp(x); save( X.mat, X ); exit; David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
30 Distributing example3 The calculation of each X i are independent We only need all of the X i s during a final summation step Distribution Basics: Distributing for and while loops are easy ways of increasing throughput! Especially if - my file.m for i = 1:N... % LOTS OF COMPUTATIONAL WORK... end David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
31 Distributing example3 Need A, B, and C for each job Distribute A i and B i with the job Distribute C to every job Could use.mat or flat text file Matlab R loads A i, B i, and C then computes X David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
32 Distributing example3 Performing the summation 1 Save X for each job and transfer the file back 2 Matlab R script that runs after distributed work is complete. 3 Condor R -ized Matlab R job. Stick around for Session 6 to learn how to set it and forget it David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
33 example3.m example4.m example4.m function [] = example4(i) if ischar(i) i = str2num(i); end load([ A_ num2str(i).mat ]); load([ B_ num2str(i).mat ]); load( C.mat ); X = sqrt(a * B + C); disp(x); save([ X_ num2str(i).mat ], X ); exit; David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
34 Condor R Submit File example4.sub Universe = vanilla Executable = example4.sh Arguments = $(PROCESS) Output = example4_$(process).out Error = example4_$(process).err Log = example4.log Requirements = ( OpSys == "LINUX" && Arch == "X86_64" ) transfer_input_files = example4, \ A_$(PROCESS).mat, B_$(PROCESS).mat, C.mat should_transfer_files = TRUE when_to_transfer_output = ON_EXIT Queue 50 David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
35 Performing the Summation example5.m function [] = example5(n) if ischar(n) N = str2num(n); end load( X_0.mat, X ); M = length(x); % Determine M x_sum = zeros(m); for i = 0:N-1 load([ X_ num2str(i).mat ], X ); % Load X x_sum = x_sum + X; end disp(x_sum); exit; David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
36 Performing the Summation example5.sub Universe = vanilla Executable = example5.sh Arguments = 50 Output = example5.out Error = example5.err Log = example5.log Requirements = ( OpSys == "LINUX" && Arch == "X86_64" ) transfer_input_files = example5, X.tar.gz should_transfer_files = TRUE when_to_transfer_output = ON_EXIT Queue 1 David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
37 Performing the Summation example5.sh #!/bin/bash source /etc/bashrc && source /etc/profile # Source the modules script source /cluster/modules/3.2.7/init/bash # Do Error Checking tar xzf X.tar.gz # More Error Checking chmod +x example5 module load matlab./example5 $1 # Cleanup rm -rf X_*.mat David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
38 Concluding Remarks Condor R can help solve many Matlab R problems quickly The Matlab R compiler (mcc) can be used to distribute code across MUGrid Management of Matlab R data is required for distribution David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
39 Stay tuned for... Session 6: Advanced High-Throughput Computing (1:30-2:45) Combine multiple Matlab R jobs into a workflow using a DAG David J. Herzfeld (Marquette University) Matlab & MUGrid July 15, / 39
Session 1: Accessing MUGrid and Command Line Basics
Session 1: Accessing MUGrid and Command Line Basics Craig A. Struble, Ph.D. July 14, 2010 1 Introduction The Marquette University Grid (MUGrid) is a collection of dedicated and opportunistic resources
Calcul intensif et Stockage de Masse. CÉCI/CISM HPC training sessions Use of Matlab on the clusters
Calcul intensif et Stockage de Masse CÉCI/ HPC training sessions Use of Matlab on the clusters Typical usage... Interactive Batch Type in and get an answer Submit job and fetch results Sequential Parallel
Calcul intensif et Stockage de Masse. CÉCI/CISM HPC training sessions
Calcul intensif et Stockage de Masse CÉCI/ HPC training sessions Calcul intensif et Stockage de Masse Parallel Matlab on the cluster /CÉCI Training session www.uclouvain.be/cism www.ceci-hpc.be November
HIGH-THROUGHPUT COMPUTING AND YOUR RESEARCH
HIGH-THROUGHPUT COMPUTING AND YOUR RESEARCH Christina Koch, Research Computing Facilitator Center for High Throughput Computing STAT679, October 29, 2018 1 About Me I work for the Center for High Throughput
Introduction to HTCondor
Introduction to HTCondor Kenyi Hurtado June 17, 2016 1 Covered in this presentation What is HTCondor? How to run a job (and multiple ones) Monitoring your queue 2 What is HTCondor? A specialized workload
Day 9: Introduction to CHTC
Day 9: Introduction to CHTC Suggested reading: Condor 7.7 Manual: http://www.cs.wisc.edu/condor/manual/v7.7/ Chapter 1: Overview Chapter 2: Users Manual (at most, 2.1 2.7) 1 Turn In Homework 2 Homework
UoW HPC Quick Start. Information Technology Services University of Wollongong. ( Last updated on October 10, 2011)
UoW HPC Quick Start Information Technology Services University of Wollongong ( Last updated on October 10, 2011) 1 Contents 1 Logging into the HPC Cluster 3 1.1 From within the UoW campus.......................
NOTE: From the engineering network, users may login directly to the desired computer server using the SSH connection.
Overview of MSU Compute Servers The DECS Linux-based compute servers are well suited for programs that are too slow to run on typical desktop computers but do not require the power of supercomputers. The
Center for Mathematical Modeling University of Chile HPC 101. HPC systems basics and concepts. Juan Carlos Maureira B.
Center for Mathematical Modeling University of Chile HPC 101 HPC systems basics and concepts By Juan Carlos Maureira B. BioMedicina II Calculo Masivo en Biomedicina CMM - FCFM - University
Introduction to Condor. Jari Varje
Introduction to Condor Jari Varje 25. 27.4.2016 Outline Basics Condor overview Submitting a job Monitoring jobs Parallel jobs Advanced topics Host requirements Running MATLAB jobs Checkpointing Case study:
Getting Started with OSG Connect ~ an Interactive Tutorial ~
Getting Started with OSG Connect ~ an Interactive Tutorial ~ Emelie Harstad , Mats Rynge , Lincoln Bryant , Suchandra Thapa ,
Introduction to MATLAB
Introduction to MATLAB Aapo Nummenmaa, PhD Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston Background Overview! What is MATLAB?! MATLAB=(MATrix
Tutorial 4: Condor. John Watt, National e-science Centre
Tutorial 4: Condor John Watt, National e-science Centre Tutorials Timetable Week Day/Time Topic Staff 3 Fri 11am Introduction to Globus J.W. 4 Fri 11am Globus Development J.W. 5 Fri 11am Globus Development
Exercise 1: Basic Tools
Exercise 1: Basic Tools This exercise is created so everybody can learn the basic tools we will use during this course. It is really more like a tutorial than an exercise and, you are not required to submit
History of SURAgrid Deployment
All Hands Meeting: May 20, 2013 History of SURAgrid Deployment Steve Johnson Texas A&M University Copyright 2013, Steve Johnson, All Rights Reserved. Original Deployment Each job would send entire R binary
Slurm basics. Summer Kickstart June slide 1 of 49
Slurm basics Summer Kickstart 2017 June 2017 slide 1 of 49 Triton layers Triton is a powerful but complex machine. You have to consider: Connecting (ssh) Data storage (filesystems and Lustre) Resource
ITNPBD7 Cluster Computing Spring Using Condor
The aim of this practical is to work through the Condor examples demonstrated in the lectures and adapt them to alternative tasks. Before we start, you will need to map a network drive to \\wsv.cs.stir.ac.uk\datasets
Introduction to Linux Part 2b: basic scripting. Brett Milash and Wim Cardoen CHPC User Services 18 January, 2018
Introduction to Linux Part 2b: basic scripting Brett Milash and Wim Cardoen CHPC User Services 18 January, 2018 Overview Scripting in Linux What is a script? Why scripting? Scripting languages + syntax
Introduction to Linux for BlueBEAR. January
Introduction to Linux for BlueBEAR January 2019 http://intranet.birmingham.ac.uk/bear Overview Understanding of the BlueBEAR workflow Logging in to BlueBEAR Introduction to basic Linux commands Basic file
NBIC TechTrack PBS Tutorial
NBIC TechTrack PBS Tutorial by Marcel Kempenaar, NBIC Bioinformatics Research Support group, University Medical Center Groningen Visit our webpage at: http://www.nbic.nl/support/brs 1 NBIC PBS Tutorial
Contents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
Movidius Neural Compute Stick
Movidius Neural Compute Stick You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to
Introduction To. Barry Grant
Introduction To Barry Grant bjgrant@umich.edu http://thegrantlab.org Working with Unix How do we actually use Unix? Inspecting text files less - visualize a text file: use arrow keys page down/page up
A Hands-On Tutorial: RNA Sequencing Using High-Performance Computing
A Hands-On Tutorial: RNA Sequencing Using Computing February 11th and 12th, 2016 1st session (Thursday) Preliminaries: Linux, HPC, command line interface Using HPC: modules, queuing system Presented by:
A Guide to Condor. Joe Antognini. October 25, Condor is on Our Network What is an Our Network?
A Guide to Condor Joe Antognini October 25, 2013 1 Condor is on Our Network What is an Our Network? The computers in the OSU astronomy department are all networked together. In fact, they re networked
Using ISMLL Cluster. Tutorial Lec 5. Mohsan Jameel, Information Systems and Machine Learning Lab, University of Hildesheim
Using ISMLL Cluster Tutorial Lec 5 1 Agenda Hardware Useful command Submitting job 2 Computing Cluster http://www.admin-magazine.com/hpc/articles/building-an-hpc-cluster Any problem or query regarding
Working with Basic Linux. Daniel Balagué
Working with Basic Linux Daniel Balagué How Linux Works? Everything in Linux is either a file or a process. A process is an executing program identified with a PID number. It runs in short or long duration
Mills HPC Tutorial Series. Linux Basics II
Mills HPC Tutorial Series Linux Basics II Objectives Bash Shell Script Basics Script Project This project is based on using the Gnuplot program which reads a command file, a data file and writes an image
Intro to Linux. this will open up a new terminal window for you is super convenient on the computers in the lab
Basic Terminal Intro to Linux ssh short for s ecure sh ell usage: ssh [host]@[computer].[otheripstuff] for lab computers: ssh [CSID]@[comp].cs.utexas.edu can get a list of active computers from the UTCS
Support for Vanilla Universe Checkpointing. Thomas Downes University of Wisconsin-Milwaukee (LIGO)
Support for Vanilla Universe Checkpointing Thomas Downes University of Wisconsin-Milwaukee (LIGO) Experimental feature! All features discussed are present in the official 8.5 releases. The Morgridge Institute
Introduction to Shell Scripting
Introduction to Shell Scripting Evan Bollig and Geoffrey Womeldorff Presenter Yusong Liu Before we begin... Everyone please visit this page for example scripts and grab a crib sheet from the front http://www.scs.fsu.edu/~bollig/techseries
Environment Variables
Environment Variables 1 A shell is simply a program that supplies certain services to users. As such, a shell may take parameters whose values modify or define certain behaviors. These parameters (or shell
Introduction to GALILEO
Introduction to GALILEO Parallel & production environment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Domenico Guida d.guida@cineca.it Maurizio Cremonesi m.cremonesi@cineca.it
Name Department/Research Area Have you used the Linux command line?
Please log in with HawkID (IOWA domain) Macs are available at stations as marked To switch between the Windows and the Mac systems, press scroll lock twice 9/27/2018 1 Ben Rogers ITS-Research Services
Working with Shell Scripting. Daniel Balagué
Working with Shell Scripting Daniel Balagué Editing Text Files We offer many text editors in the HPC cluster. Command-Line Interface (CLI) editors: vi / vim nano (very intuitive and easy to use if you
ICS-ACI System Basics
ICS-ACI System Basics Adam W. Lavely, Ph.D. Fall 2017 Slides available: goo.gl/ss9itf awl5173 ICS@PSU 1 Contents 1 Overview 2 HPC Overview 3 Getting Started on ACI 4 Moving On awl5173 ICS@PSU 2 Contents
GNU/Linux 101. Casey McLaughlin. Research Computing Center Spring Workshop Series 2018
GNU/Linux 101 Casey McLaughlin Research Computing Center Spring Workshop Series 2018 rccworkshop IC;3df4mu bash-2.1~# man workshop Linux101 RCC Workshop L101 OBJECTIVES - Operating system concepts - Linux
Oregon State University School of Electrical Engineering and Computer Science. CS 261 Recitation 1. Spring 2011
Oregon State University School of Electrical Engineering and Computer Science CS 261 Recitation 1 Spring 2011 Outline Using Secure Shell Clients GCC Some Examples Intro to C * * Windows File transfer client:
User Guide of High Performance Computing Cluster in School of Physics
User Guide of High Performance Computing Cluster in School of Physics Prepared by Sue Yang (xue.yang@sydney.edu.au) This document aims at helping users to quickly log into the cluster, set up the software
Introduction: What is Unix?
Introduction Introduction: What is Unix? An operating system Developed at AT&T Bell Labs in the 1960 s Command Line Interpreter GUIs (Window systems) are now available Introduction: Unix vs. Linux Unix
Unix Essentials. BaRC Hot Topics Bioinformatics and Research Computing Whitehead Institute October 12 th
Unix Essentials BaRC Hot Topics Bioinformatics and Research Computing Whitehead Institute October 12 th 2016 http://barc.wi.mit.edu/hot_topics/ 1 Outline Unix overview Logging in to tak Directory structure
CS246-Assign01 V1.1: Winter 2013
CS246-Assign01 V1.1: Winter 2013 http://plg.uwaterloo.ca/~holt/cs/246/2013/asgn01/asgn01.htm Submit. Instructions to submit assignments (using Marmoset): https://marmoset.student.cs.uwaterloo.ca/ Instructions
FACADE Financial Analysis Computing Architecture in Distributed Environment
FACADE Financial Analysis Computing Architecture in Distributed Environment V. Motoška, L. Slebodník, M. Jurečko, M. Zvada May 4, 2011 Outline Motivation CADE Middleware Future work 2 / 19 What? Motivation
Topic 2: More Shell Skills
Topic 2: More Shell Skills Sub-topics: simple shell scripts (no ifs or loops yet) sub-shells quoting shell variables aliases bash initialization files I/O redirection & pipes text file formats 1 Reading
CSCI2467: Systems Programming Concepts
CSCI2467: Systems Programming Concepts Class activity: bash shell literacy Instructor: Matthew Toups Fall 2017 Today 0 Shells History Usage Scripts vs. Programs 1 Bash shell: practical uses for your systems
Parallel Computing with Matlab and R
Parallel Computing with Matlab and R scsc@duke.edu https://wiki.duke.edu/display/scsc Tom Milledge tm103@duke.edu Overview Running Matlab and R interactively and in batch mode Introduction to Parallel
HPC Resources at Lehigh. Steve Anthony March 22, 2012
HPC Resources at Lehigh Steve Anthony March 22, 2012 HPC at Lehigh: Resources What's Available? Service Level Basic Service Level E-1 Service Level E-2 Leaf and Condor Pool Altair Trits, Cuda0, Inferno,
Introduction to Joker Cyber Infrastructure Architecture Team CIA.NMSU.EDU
Introduction to Joker Cyber Infrastructure Architecture Team CIA.NMSU.EDU What is Joker? NMSU s supercomputer. 238 core computer cluster. Intel E-5 Xeon CPUs and Nvidia K-40 GPUs. InfiniBand innerconnect.
Part One: The Files. C MPI Slurm Tutorial - Hello World. Introduction. Hello World! hello.tar. The files, summary. Output Files, summary
C MPI Slurm Tutorial - Hello World Introduction The example shown here demonstrates the use of the Slurm Scheduler for the purpose of running a C/MPI program. Knowledge of C is assumed. Having read the
High Performance Computing (HPC) Club Training Session. Xinsheng (Shawn) Qin
High Performance Computing (HPC) Club Training Session Xinsheng (Shawn) Qin Outline HPC Club The Hyak Supercomputer Logging in to Hyak Basic Linux Commands Transferring Files Between Your PC and Hyak Submitting
CSCI 211 UNIX Lab. Shell Programming. Dr. Jiang Li. Jiang Li, Ph.D. Department of Computer Science
CSCI 211 UNIX Lab Shell Programming Dr. Jiang Li Why Shell Scripting Saves a lot of typing A shell script can run many commands at once A shell script can repeatedly run commands Help avoid mistakes Once
More Linux Shell Scripts
nit 5 ore Linux hell cripts opics for this nit ore shell commands Control statements Constraints xpressions elect command Case command unning hell cripts shell script as a standalone is an executable program
The Unix Shell & Shell Scripts
The Unix Shell & Shell Scripts You should do steps 1 to 7 before going to the lab. Use the Linux system you installed in the previous lab. In the lab do step 8, the TA may give you additional exercises
High Performance Computing How-To Joseph Paul Cohen
High Performance Computing How-To Joseph Paul Cohen This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Abstract This talk discusses how HPC is used and how
Using the Yale HPC Clusters
Using the Yale HPC Clusters Stephen Weston Robert Bjornson Yale Center for Research Computing Yale University Oct 2015 To get help Send an email to: hpc@yale.edu Read documentation at: http://research.computing.yale.edu/hpc-support
COSC 6385 Computer Architecture. - Homework
COSC 6385 Computer Architecture - Homework Fall 2008 1 st Assignment Rules Each team should deliver Source code (.c,.h and Makefiles files) Please: no.o files and no executables! Documentation (.pdf,.doc,.tex
COMS 6100 Class Notes 3
COMS 6100 Class Notes 3 Daniel Solus September 1, 2016 1 General Remarks The class was split into two main sections. We finished our introduction to Linux commands by reviewing Linux commands I and II
SCALABLE HYBRID PROTOTYPE
SCALABLE HYBRID PROTOTYPE Scalable Hybrid Prototype Part of the PRACE Technology Evaluation Objectives Enabling key applications on new architectures Familiarizing users and providing a research platform
Introduction to Unix The Windows User perspective. Wes Frisby Kyle Horne Todd Johansen
Introduction to Unix The Windows User perspective Wes Frisby Kyle Horne Todd Johansen What is Unix? Portable, multi-tasking, and multi-user operating system Software development environment Hardware independent
Introduction to UNIX. SURF Research Boot Camp April Jeroen Engelberts Consultant Supercomputing
Introduction to UNIX SURF Research Boot Camp April 2018 Jeroen Engelberts jeroen.engelberts@surfsara.nl Consultant Supercomputing Outline Introduction to UNIX What is UNIX? (Short) history of UNIX Cartesius
HTCondor Essentials. Index
HTCondor Essentials 31.10.2017 Index Login How to submit a job in the HTCondor pool Why the -name option? Submitting a job Checking status of submitted jobs Getting id and other info about a job
More Raspian. An editor Configuration files Shell scripts Shell variables System admin
More Raspian An editor Configuration files Shell scripts Shell variables System admin Nano, a simple editor Nano does not require the mouse. You must use your keyboard to move around the file and make
HTCONDOR USER TUTORIAL. Greg Thain Center for High Throughput Computing University of Wisconsin Madison
HTCONDOR USER TUTORIAL Greg Thain Center for High Throughput Computing University of Wisconsin Madison gthain@cs.wisc.edu 2015 Internet2 HTCondor User Tutorial CONTENTS Overview Basic job submission How
Environment Variables
Environment Variables 1 A shell is simply a program that supplies certain services to users. As such, a shell may take parameters whose values modify or define certain behaviors. These parameters (or shell
Computing with the Moore Cluster
Computing with the Moore Cluster Edward Walter An overview of data management and job processing in the Moore compute cluster. Overview Getting access to the cluster Data management Submitting jobs (MPI
Dalhousie University CSCI 2132 Software Development Winter 2018 Lab 8, March 22
Dalhousie University CSCI 2132 Software Development Winter 2018 Lab 8, March 22 In this lab, you will first learn more about git. After that, you will get some practice on the make utility and learn more
Linux Systems Administration Shell Scripting Basics. Mike Jager Network Startup Resource Center
Linux Systems Administration Shell Scripting Basics Mike Jager Network Startup Resource Center mike.jager@synack.co.nz These materials are licensed under the Creative Commons Attribution-NonCommercial
Using bash. Administrative Shell Scripting COMP2101 Fall 2017
Using bash Administrative Shell Scripting COMP2101 Fall 2017 Bash Background Bash was written to replace the Bourne shell The Bourne shell (sh) was not a good candidate for rewrite, so bash was a completely
Introduction to Discovery.
Introduction to Discovery http://discovery.dartmouth.edu The Discovery Cluster 2 Agenda What is a cluster and why use it Overview of computer hardware in cluster Help Available to Discovery Users Logging
Carnegie Mellon. Linux Boot Camp. Jack, Matthew, Nishad, Stanley 6 Sep 2016
Linux Boot Camp Jack, Matthew, Nishad, Stanley 6 Sep 2016 1 Connecting SSH Windows users: MobaXterm, PuTTY, SSH Tectia Mac & Linux users: Terminal (Just type ssh) andrewid@shark.ics.cs.cmu.edu 2 Let s
DAVE LIDDAMENT INTRODUCTION TO BASH
DAVE LIDDAMENT INTRODUCTION TO BASH @daveliddament FORMAT Short lectures Practical exercises (help each other) Write scripts LEARNING OBJECTIVES What is Bash When should you use Bash Basic concepts of
Other useful tools. Eugeniy E. Mikhailov. Lecture 11. The College of William & Mary. Eugeniy Mikhailov (W&M) Practical Computing Lecture 11 1 / 9
Other useful tools Eugeniy E. Mikhailov The College of William & Mary Lecture 11 Eugeniy Mikhailov (W&M) Practical Computing Lecture 11 1 / 9 Specialization is... A human being should be able to change
Bash Tutorial. ASL Fall 2017 Week 2
Bash Tutorial ASL Fall 2017 Week 2 Comments from previous years I had to stay up all night to run experiments! I cannot work on this from home.. It took me more than 40hrs./week to work on this. Solution
Chapter 9. Shell and Kernel
Chapter 9 Linux Shell 1 Shell and Kernel Shell and desktop enviroment provide user interface 2 1 Shell Shell is a Unix term for the interactive user interface with an operating system A shell usually implies
XSEDE High Throughput Computing Use Cases
XSEDE High Throughput Computing Use Cases 31 May 2013 Version 0.3 XSEDE HTC Use Cases Page 1 XSEDE HTC Use Cases Page 2 Table of Contents A. Document History B. Document Scope C. High Throughput Computing
EECS2031 Winter Software Tools. Assignment 1 (15%): Shell Programming
EECS2031 Winter 2018 Software Tools Assignment 1 (15%): Shell Programming Due Date: 11:59 pm on Friday, Feb 9, 2018 Objective In this assignment, you will be writing four shell programs. The first program
Scaling MATLAB. for Your Organisation and Beyond. Rory Adams The MathWorks, Inc. 1
Scaling MATLAB for Your Organisation and Beyond Rory Adams 2015 The MathWorks, Inc. 1 MATLAB at Scale Front-end scaling Scale with increasing access requests Back-end scaling Scale with increasing computational
CMPS 12A Introduction to Programming Lab Assignment 7
CMPS 12A Introduction to Programming Lab Assignment 7 In this assignment you will write a bash script that interacts with the user and does some simple calculations, emulating the functionality of programming
GDB and Makefile. Bin Li Assistant Professor Dept. of Electrical, Computer and Biomedical Engineering University of Rhode Island
GDB and Makefile Bin Li Assistant Professor Dept. of Electrical, Computer and Biomedical Engineering University of Rhode Island GDB Debugging: An Example #include void main() { int i; int result
M2PGER FORTRAN programming. General introduction. Virginie DURAND and Jean VIRIEUX 10/13/2013 M2PGER - ALGORITHME SCIENTIFIQUE
M2PGER 2013-2014 FORTRAN programming General introduction Virginie DURAND and Jean VIRIEUX 1 Why learning programming??? 2 Why learning programming??? ACQUISITION numerical Recording on magnetic supports
Introduction to UNIX. Logging in. Basic System Architecture 10/7/10. most systems have graphical login on Linux machines
Introduction to UNIX Logging in Basic system architecture Getting help Intro to shell (tcsh) Basic UNIX File Maintenance Intro to emacs I/O Redirection Shell scripts Logging in most systems have graphical
Using echo command in shell script
Lab 4a Shell Script Lab 2 Using echo command in shell script Objective Upon completion of this lab, the student will be able to use echo command in the shell script. Scenario The student is the administrator
Linux Bash Shell Scripting
University of Chicago Initiative in Biomedical Informatics Computation Institute Linux Bash Shell Scripting Present by: Mohammad Reza Gerami gerami@ipm.ir Day 2 Outline Support Review of Day 1 exercise
Introduction in Unix. Linus Torvalds Ken Thompson & Dennis Ritchie
Introduction in Unix Linus Torvalds Ken Thompson & Dennis Ritchie My name: John Donners John.Donners@surfsara.nl Consultant at SURFsara And Cedric Nugteren Cedric.Nugteren@surfsara.nl Consultant at SURFsara
CSCI 2132: Software Development. Norbert Zeh. Faculty of Computer Science Dalhousie University. Shell Scripting. Winter 2019
CSCI 2132: Software Development Shell Scripting Norbert Zeh Faculty of Computer Science Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your Shell vs Your File Manager File manager
Bash scripting basics
Bash scripting basics prepared by Anatoliy Antonov for ESSReS community September 2012 1 Outline Definitions Foundations Flow control References and exercises 2 Definitions 3 Definitions Script - [small]
Introduction to PICO Parallel & Production Enviroment
Introduction to PICO Parallel & Production Enviroment Mirko Cestari m.cestari@cineca.it Alessandro Marani a.marani@cineca.it Domenico Guida d.guida@cineca.it Nicola Spallanzani n.spallanzani@cineca.it
The Linux Command Line & Shell Scripting
The Linux Command Line & Shell Scripting [web] [email] portal.biohpc.swmed.edu biohpc-help@utsouthwestern.edu 1 Updated for 2017-11-18 Study Resources : A Free Book 500+ pages * Some of the materials covered
Hands-on Keyboard: Cyber Experiments for Strategists and Policy Makers
Hands-on Keyboard: Cyber Experiments for Strategists and Policy Makers Review of the Linux File System and Linux Commands 1. Introduction Becoming adept at using the Linux OS requires gaining familiarity
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR
CISC 220 fall 2011, set 1: Linux basics
CISC 220: System-Level Programming instructor: Margaret Lamb e-mail: malamb@cs.queensu.ca office: Goodwin 554 office phone: 533-6059 (internal extension 36059) office hours: Tues/Wed/Thurs 2-3 (this week
Introduction to Discovery.
Introduction to Discovery http://discovery.dartmouth.edu The Discovery Cluster 2 Agenda What is a cluster and why use it Overview of computer hardware in cluster Help Available to Discovery Users Logging
CS CS Tutorial 2 2 Winter 2018
CS CS 230 - Tutorial 2 2 Winter 2018 Sections 1. Unix Basics and connecting to CS environment 2. MIPS Introduction & CS230 Interface 3. Connecting Remotely If you haven t set up a CS environment password,
Using HTCondor on the Biochemistry Computational Cluster v Jean-Yves Sgro
HTCondor@Biochem Using HTCondor on the Biochemistry Computational Cluster v1.2.0 Jean-Yves Sgro 2016 Jean-Yves Sgro Biochemistry Computational Research Facility Note: The HTCondor software was known as
Unix/Linux Basics. Cpt S 223, Fall 2007 Copyright: Washington State University
Unix/Linux Basics 1 Some basics to remember Everything is case sensitive Eg., you can have two different files of the same name but different case in the same folder Console-driven (same as terminal )
Installing Connector on Linux
CHAPTER 3 Revised: July 15, 2010 Overview This chapter provides a step-by-step guide to installing the Linux Connector on x86 and x86-64 servers running either Red Hat Enterprise Linux version 5 or Cent
Install Matlab on Linux (Ubuntu)
Install Matlab on Linux (Ubuntu) And execute scripts/commands using the command line Sigve Karolius September 21, 2015 Abstract Matlab is a domain-specific scripting language that is used in many areas
BIOINFORMATICS POST-DIPLOMA PROGRAM SUBJECT OUTLINE Subject Title: OPERATING SYSTEMS AND PROJECT MANAGEMENT Subject Code: BIF713 Subject Description:
BIOINFORMATICS POST-DIPLOMA PROGRAM SUBJECT OUTLINE Subject Title: OPERATING SYSTEMS AND PROJECT MANAGEMENT Subject Code: BIF713 Subject Description: This course provides Bioinformatics students with the
Snakemake overview. Thomas Cokelaer. Nov 9th 2017 Snakemake and Sequana overview. Institut Pasteur
Snakemake overview Thomas Cokelaer Institut Pasteur Nov 9th 2017 Snakemake and Sequana overview Many bioinformatic pipeline frameworks available A review of bioinformatic pipeline frameworks. Jeremy Leipzig
Who is Docker and how he can help us? Heino Talvik
Who is Docker and how he can help us? Heino Talvik heino.talvik@seb.ee heino.talvik@gmail.com What is Docker? Software guy view: Marriage of infrastucture and Source Code Management Hardware guy view: