An Introduction to Cluster Computing Using Newton

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "An Introduction to Cluster Computing Using Newton"

Transcription

1 An Introduction to Cluster Computing Using Newton Jason Harris and Dylan Storey March 25th, 2014 Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

2 Workshop design. Slides are going to be up here mostly for you to stare at and so that I remember what I m supposed to be talking about. Mostly you will be getting onto newton and submitting a few trivial jobs so that you ll be more comfortable in the future. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

3 What is a cluster? Simply put, a cluster is a number of computers that are connected together over a LAN that act like one large computer by being able to talk with each other. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

4 When should I use the cluster? You should look to using the cluster when computing becomes a bottle neck. You have many of the same processes to run serially. A few intensive processes to run. A process that needs to use more CPUs than you have on your current computer. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

5 Why should I be using the cluster? Free support Technical advising You don t need to buy your own hardware. You don t need to maintain your own hardware and software. Basically, you get a resource without having to pay for and maintain it yourself. None of this is free, but as a student you do get non priority access to the equipment and individuals can invest in the cluster to get more computing resources. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

6 Newton Specs In total Newton has: 4,200 cores 8TB of RAM 50TB of Lustre Storage 50TB of Network Attached Storage 50TB off site backup Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

7 Access to Newton All students and faculty have access to this resource. How much priority you have depends on money. To register : Go To: Register for an account. Wait If you remember everyone who didn t know what Newton was gave me their UT username several months ago. Your accounts are already made. Some of you didn t write legibly you ll need to do this yourself. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

8 Connecting to Newton Connecting to Newton is a trivial matter Linux/UNIX : type ssh into your terminal PuTTy : Hostname newlogin.newton.utk.edu Username NetID Password NetID password Remote port to connect to 22 A successful connection will open up a bash shell for you to use. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

9 Linux Refresher Linux Bash Shell Cheat Sheet Basic Commands Basic Terminal Shortcuts Basic file manipulation CTRL L = Clear the terminal cat <filename> = show content of file CTRL D = Logout (less, more) SHIFT Page Up/Down = Go up/down the terminal head = from the top CTRL A = Cursor to start of line -n <#oflines> <filename> CTRL E = Cursor the end of line CTRL U = Delete left of the cursor tail = from the bottom CTRL K = Delete right of the cursor -n <#oflines> <filename> CTRL W = Delete word on the left CTRL Y = Paste (after CTRL U,K or W) mkdir = create new folder TAB = auto completion of file or command mkdir mystuff.. CTRL R = reverse search history mkdir mystuff/pictures/..!! = repeat last command CTRL Z = stops the current command (resume with fg in foreground or bg in background) cp image.jpg newimage.jpg = copy and rename a file Basic Terminal Navigation cp image.jpg <foldername>/ = copy to folder cp image.jpg folder/sameimagenewname.jpg ls -a = list all files and folders cp -R stuff otherstuff = copy and rename a folder ls <foldername> = list files in folder cp *.txt stuff/ = copy all of *<file type> to folder ls -lh = Detailed list, Human readable ls -l *.jpg = list jpeg files only mv file.txt Documents/ = move file to a folder ls -lh <filename> = Result for file only mv <foldername> <foldername2> = move folder in folder mv filename.txt filename2.txt = rename file cd <foldername> = change directory mv <filename> stuff/newfilename if folder name has spaces use mv <foldername>/.. = move folder up in hierarchy cd / = go to root cd.. = go up one folder, tip:../../../ rm <filename>.. = delete file (s) rm -i <filename>.. = ask for confirmation each file du -h: Disk usage of folders, human readable rm -f <filename> = force deletion of a file du -ah: files & folders, Human readable rm -r <foldername>/ = delete folder du -sh: only show disc usage of folders touch <filename> = create or update a file pwd = print working directory ln file1 file2 = physical link man <command> = shows manual (RTFM) ln -s file1 file2 = symbolic link Courtesy of cli.learncodethehardway.org Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

10 Transfering Files Of course logging into Newton isn t enough. You ll need to get data to and from the cluster as well. There are a number of tools for this that exist. Command line: scp - Secure Copy. Its just like cp but over SSH. wget - gets content from web servers. (HTTP,HTTPS,FTP) rsync - Linux/Unix synchronization utility. Good for lots of files or if only a few changes are being made between files. GUI: FileZilla, Drag and drop file queuing and transfer program. Fugu, another popular file transfer interface for Mac OSX. Most Linux file managers have built in functionality for this as well. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

11 Screen Shots, Because it Happened Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

12 Screen Shots, Because it Happened Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

13 Screen Shots, Because it Happened Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

14 Screen Shots, Because it Happened Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

15 Screen Shots, Because it Happened Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

16 Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

17 The Sun Gride Engine Once you connect to the cluster you ll be able to do a few basic things like: edit files compile code run basic programs But keep in mind that you re resource limited and if you try to run any resource intensive jobs you ll anger the admin gods. In order to use the cluster effectively you ll need to submit jobs to the Grid Engine. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

18 Submitting a Job to SGE This process goes as follows: Create a file that tells the scheduler what you need and how longtable you ll need it for. Grid Engine finds the resources on the cluster for you to use and reserves them. Once all of the resources are collected the Grid Engine runs your job. While the job is running, the Grid Engine collects stderr and stdout and writes them to log files. Upon completion, resources are freed. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

19 Queues on the Newton Cluster The queue you request tells Newton how long you want a job to run for. If your job runs longer than the queue you re in, Newton will kill the job before it finishes so make sure to pick an appropriate queue. Queue selection also changes the number of resources you can request for a job as well. short- 256 cpus, 2 hours max run time medium cpus, 24 hours max run time long - priority groups only Don t just put everything in the medium queue, it will effect how fast your jobs are executed. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

20 A Basic Job, Single CPU! Create your first SGE file by opening a new basic text document on your computer Type the following lines into the file and save. #$ N Test #$ cwd #$ q s h o r t uname a s l e e p 30 Copy this file to your home directory on Newton (/home/username/) Submit your job by typing the following : qsub Basicjob.sge Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

21 Monitoring your Job(s) Type qstat to get a list of your currently running or submitted jobs. You can check the state column to find if your job is queued and waiting (qw), running(r), or in error (e). You can also check the direct status of running jobs by looking at output generated during the run. These file are named by the Name you submitted in your script, followd by an e or o for each stream. Pro Tip : Typing qstat repeatedly is annoying, try typing watch qstat and it will check the command regularly for you! Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

22 Requesting More Resources Obviously if you re using this resource you probably have software capable of using multiple CPUs. In order to do this you need to ask for those CPUs before you can use them. Pro Tip: It s also important to tell the program you re running how many CPUs it is allowed to use in order to make sure you re not using resources that aren t allocated to you. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

23 Multiple CPUs on a Single Node If you want to run a job in which you ll be using multiple CPUs on a single node your job file might look something like this: #$ pe t h r e a d s 4 #$ N Test #$ cwd #$ q s h o r t / l u s t r e /home/ d s t o r e y / W o r k s h o p F i l e s / t h r e a d s. p l You ll notice a new line requesting 2 threads. This tells the scheduler to find you two CPUs on the same node. The files and an example script for this available at : /lustre/home/dstorey/threads/ Remember, you can only ask for resources that exist. If no nodes exist with the number of threads you request your job can t run. Try out the qhost command and see what it says! Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

24 Trivially Parallel Jobs Sometimes you need to execute the same job repeatedly. The Grid Engine has a specific way of managing these tasks called array jobs. The files and an example script for this available at : /lustre/home/dstorey/array/ To use an array job: Break your data in to x number of files with the names having sequential integers. Submit your job file with the option #$-t 1-x The job then accesses an enviroment variable $SGE TASK ID to determine the file to use. Your Job File should look something like this : #$ N Test #$ cwd #$ q s h o r t #$ t / l u s t r e /home/ d s t o r e y / W o r k s h o p F i l e s / Array / a v e r a g e. p l $SGE TASK ID. dat Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

25 Trivially Parallel Jobs I m sure you can already see that this is not always a feasible method for submitting jobs. Thankfully you can use scripts to write and submit job files for you. I ve included and example script for to do this if you need any ideas(/lustre/home/dstorey/workshopfiles/generate and submit jobs.pl). Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

26 Other Methods There are other methods of requesting resources: MPI jobs request many CPUs from many Nodes. Hybridized jobs request specific number of CPUs on a node across a number of Nodes. This is all I m going to say about these. Know that they exist and you may want to use them one day. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

27 Modules on Newton The module system provides you with a way of managing all of the enviromental variables in your session. Each module corresponds to a specific application. Use the module system with the module command. For example if you wanted to submit a job that uses the signalp module and calls signalp your sun grid engine would look something like this: #$ N Test #$ cwd #$ q s h o r t module l o a d s i g n a l p s i g n a l p m y p e p t i d e. f a s You can get a complete list of modules available by typing module avail. Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, / 26

A Hands-On Tutorial: RNA Sequencing Using High-Performance Computing

A Hands-On Tutorial: RNA Sequencing Using High-Performance Computing A Hands-On Tutorial: RNA Sequencing Using Computing February 11th and 12th, 2016 1st session (Thursday) Preliminaries: Linux, HPC, command line interface Using HPC: modules, queuing system Presented by:

More information

Introduction to Unix The Windows User perspective. Wes Frisby Kyle Horne Todd Johansen

Introduction to Unix The Windows User perspective. Wes Frisby Kyle Horne Todd Johansen Introduction to Unix The Windows User perspective Wes Frisby Kyle Horne Todd Johansen What is Unix? Portable, multi-tasking, and multi-user operating system Software development environment Hardware independent

More information

Working with Basic Linux. Daniel Balagué

Working with Basic Linux. Daniel Balagué Working with Basic Linux Daniel Balagué How Linux Works? Everything in Linux is either a file or a process. A process is an executing program identified with a PID number. It runs in short or long duration

More information

Linux Command Line Interface. December 27, 2017

Linux Command Line Interface. December 27, 2017 Linux Command Line Interface December 27, 2017 Foreword It is supposed to be a refresher (?!) If you are familiar with UNIX/Linux/MacOS X CLI, this is going to be boring... I will not talk about editors

More information

Using ISMLL Cluster. Tutorial Lec 5. Mohsan Jameel, Information Systems and Machine Learning Lab, University of Hildesheim

Using ISMLL Cluster. Tutorial Lec 5. Mohsan Jameel, Information Systems and Machine Learning Lab, University of Hildesheim Using ISMLL Cluster Tutorial Lec 5 1 Agenda Hardware Useful command Submitting job 2 Computing Cluster http://www.admin-magazine.com/hpc/articles/building-an-hpc-cluster Any problem or query regarding

More information

COMS 6100 Class Notes 3

COMS 6100 Class Notes 3 COMS 6100 Class Notes 3 Daniel Solus September 1, 2016 1 General Remarks The class was split into two main sections. We finished our introduction to Linux commands by reviewing Linux commands I and II

More information

High Performance Computing (HPC) Club Training Session. Xinsheng (Shawn) Qin

High Performance Computing (HPC) Club Training Session. Xinsheng (Shawn) Qin High Performance Computing (HPC) Club Training Session Xinsheng (Shawn) Qin Outline HPC Club The Hyak Supercomputer Logging in to Hyak Basic Linux Commands Transferring Files Between Your PC and Hyak Submitting

More information

Introduction to Linux Part 1. Anita Orendt and Wim Cardoen Center for High Performance Computing 24 May 2017

Introduction to Linux Part 1. Anita Orendt and Wim Cardoen Center for High Performance Computing 24 May 2017 Introduction to Linux Part 1 Anita Orendt and Wim Cardoen Center for High Performance Computing 24 May 2017 ssh Login or Interactive Node kingspeak.chpc.utah.edu Batch queue system kp001 kp002. kpxxx FastX

More information

Joint High Performance Computing Exchange (JHPCE) Cluster Orientation.

Joint High Performance Computing Exchange (JHPCE) Cluster Orientation. Joint High Performance Computing Exchange (JHPCE) Cluster Orientation http://www.jhpce.jhu.edu/ Schedule - Introductions who are we, who are you? - Terminology - Logging in and account setup - Basics of

More information

Lab 2: Linux/Unix shell

Lab 2: Linux/Unix shell Lab 2: Linux/Unix shell Comp Sci 1585 Data Structures Lab: Tools for Computer Scientists Outline 1 2 3 4 5 6 7 What is a shell? What is a shell? login is a program that logs users in to a computer. When

More information

Useful Unix Commands Cheat Sheet

Useful Unix Commands Cheat Sheet Useful Unix Commands Cheat Sheet The Chinese University of Hong Kong SIGSC Training (Fall 2016) FILE AND DIRECTORY pwd Return path to current directory. ls List directories and files here. ls dir List

More information

Exercise 1: Connecting to BW using ssh: NOTE: $ = command starts here, =means one space between words/characters.

Exercise 1: Connecting to BW using ssh: NOTE: $ = command starts here, =means one space between words/characters. Exercise 1: Connecting to BW using ssh: NOTE: $ = command starts here, =means one space between words/characters. Before you login to the Blue Waters system, make sure you have the following information

More information

A Brief Introduction to the Linux Shell for Data Science

A Brief Introduction to the Linux Shell for Data Science A Brief Introduction to the Linux Shell for Data Science Aris Anagnostopoulos 1 Introduction Here we will see a brief introduction of the Linux command line or shell as it is called. Linux is a Unix-like

More information

Introduction to Linux Workshop 1

Introduction to Linux Workshop 1 Introduction to Linux Workshop 1 The George Washington University SEAS Computing Facility Created by Jason Hurlburt, Hadi Mohammadi, Marco Suarez hurlburj@gwu.edu Logging In The lab computers will authenticate

More information

Introduction to Unix: Fundamental Commands

Introduction to Unix: Fundamental Commands Introduction to Unix: Fundamental Commands Ricky Patterson UVA Library Based on slides from Turgut Yilmaz Istanbul Teknik University 1 What We Will Learn The fundamental commands of the Unix operating

More information

C++ Programming on Linux

C++ Programming on Linux C++ Programming on Linux What is Linux? CS 2308 Spring 2017 Jill Seaman Slides 14-end are for your information only, you will not be tested over that material. 1 l an operating system l Unix-like l Open

More information

Advanced Linux Commands & Shell Scripting

Advanced Linux Commands & Shell Scripting Advanced Linux Commands & Shell Scripting Advanced Genomics & Bioinformatics Workshop James Oguya Nairobi, Kenya August, 2016 Man pages Most Linux commands are shipped with their reference manuals To view

More information

Network Monitoring & Management. A few Linux basics

Network Monitoring & Management. A few Linux basics Network Monitoring & Management A few Linux basics Our chosen platform Ubuntu Linux 14.04.3 LTS 64-bit LTS = Long Term Support no GUI, we administer using ssh Ubuntu is Debian underneath There are other

More information

Lab 1 Introduction to UNIX and C

Lab 1 Introduction to UNIX and C Name: Lab 1 Introduction to UNIX and C This first lab is meant to be an introduction to computer environments we will be using this term. You must have a Pitt username to complete this lab. NOTE: Text

More information

Introduction to HPC Resources and Linux

Introduction to HPC Resources and Linux Introduction to HPC Resources and Linux Burak Himmetoglu Enterprise Technology Services & Center for Scientific Computing e-mail: bhimmetoglu@ucsb.edu Paul Weakliem California Nanosystems Institute & Center

More information

Chap2: Operating-System Structures

Chap2: Operating-System Structures Chap2: Operating-System Structures Objectives: services OS provides to users, processes, and other systems structuring an operating system how operating systems are designed and customized and how they

More information

bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs

bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR

More information

Unix/Linux Operating System. Introduction to Computational Statistics STAT 598G, Fall 2011

Unix/Linux Operating System. Introduction to Computational Statistics STAT 598G, Fall 2011 Unix/Linux Operating System Introduction to Computational Statistics STAT 598G, Fall 2011 Sergey Kirshner Department of Statistics, Purdue University September 7, 2011 Sergey Kirshner (Purdue University)

More information

Study Guide Processes & Job Control

Study Guide Processes & Job Control Study Guide Processes & Job Control Q1 - PID What does PID stand for? Q2 - Shell PID What shell command would I issue to display the PID of the shell I'm using? Q3 - Process vs. executable file Explain,

More information

The Command Line. Matthew Bender. September 10, CMSC Command Line Workshop. Matthew Bender (2015) The Command Line September 10, / 25

The Command Line. Matthew Bender. September 10, CMSC Command Line Workshop. Matthew Bender (2015) The Command Line September 10, / 25 The Command Line Matthew Bender CMSC Command Line Workshop September 10, 2015 Matthew Bender (2015) The Command Line September 10, 2015 1 / 25 Introduction Section 1 Introduction Matthew Bender (2015)

More information

CS 261 Recitation 1 Compiling C on UNIX

CS 261 Recitation 1 Compiling C on UNIX Oregon State University School of Electrical Engineering and Computer Science CS 261 Recitation 1 Compiling C on UNIX Winter 2017 Outline Secure Shell Basic UNIX commands Editing text The GNU Compiler

More information

27-Sep CSCI 2132 Software Development Lab 4: Exploring bash and C Compilation. Faculty of Computer Science, Dalhousie University

27-Sep CSCI 2132 Software Development Lab 4: Exploring bash and C Compilation. Faculty of Computer Science, Dalhousie University Lecture 4 p.1 Faculty of Computer Science, Dalhousie University CSCI 2132 Software Development Lab 4: Exploring bash and C Compilation 27-Sep-2017 Location: Goldberg CS Building Time: Wednesday, 16:05

More information

Physics REU Unix Tutorial

Physics REU Unix Tutorial Physics REU Unix Tutorial What is unix? Unix is an operating system. In simple terms, its the set of programs that makes a computer work. It can be broken down into three parts. (1) kernel: The component

More information

Crash Course in Unix. For more info check out the Unix man pages -orhttp://www.cs.rpi.edu/~hollingd/unix. -or- Unix in a Nutshell (an O Reilly book).

Crash Course in Unix. For more info check out the Unix man pages -orhttp://www.cs.rpi.edu/~hollingd/unix. -or- Unix in a Nutshell (an O Reilly book). Crash Course in Unix For more info check out the Unix man pages -orhttp://www.cs.rpi.edu/~hollingd/unix -or- Unix in a Nutshell (an O Reilly book). 1 Unix Accounts To access a Unix system you need to have

More information

CSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209

CSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209 CSC209 Software Tools and Systems Programming https://mcs.utm.utoronto.ca/~209 What is this Course About? Software Tools Using them Building them Systems Programming Quirks of C The file system System

More information

CpSc 1111 Lab 1 Introduction to Unix Systems, Editors, and C

CpSc 1111 Lab 1 Introduction to Unix Systems, Editors, and C CpSc 1111 Lab 1 Introduction to Unix Systems, Editors, and C Welcome! Welcome to your CpSc 111 lab! For each lab this semester, you will be provided a document like this to guide you. This material, as

More information

Programming Studio #1 ECE 190

Programming Studio #1 ECE 190 Programming Studio #1 ECE 190 Programming Studio #1 Announcements In Studio Assignment Introduction to Linux Command-Line Operations Recitation Floating Point Representation Binary & Hexadecimal 2 s Complement

More information

Command-line interpreters

Command-line interpreters Command-line interpreters shell Wiki: A command-line interface (CLI) is a means of interaction with a computer program where the user (or client) issues commands to the program in the form of successive

More information

Introduction to UNIX Command Line

Introduction to UNIX Command Line Introduction to UNIX Command Line Files and directories Some useful commands (echo, cat, grep, find, diff, tar) Redirection Pipes Variables Background processes Remote connections (e.g. ssh, curl) Scripts

More information

Helpful Tips for Labs. CS140, Spring 2015

Helpful Tips for Labs. CS140, Spring 2015 Helpful Tips for Labs CS140, Spring 2015 Linux/Unix Commands Creating, Entering, Changing Directories to Create a Directory (a Folder) on the command line type mkdir folder_name to Enter that Folder cd

More information

COSC UNIX. Textbook. Grading Scheme

COSC UNIX. Textbook. Grading Scheme COSC 2306 - UNIX Education has failed in a very serious way to convey the most important lesson science can teach: skepticism. - David Suzuki Fall 2008 Aaron Langille Textbook Linux for Programmers and

More information

Quick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing

Quick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Contents User access, logging in Linux/Unix

More information

Linux Introduction Martin Dahlö

Linux Introduction Martin Dahlö Linux Introduction 160418 Martin Dahlö martin.dahlo@scilifelab.uu.se Linux Introduction You will not learn this now. Google it or look at lecture slides when you need it. Practice makes perfect :) UPPMAX

More information

Unix basics exercise MBV-INFX410

Unix basics exercise MBV-INFX410 Unix basics exercise MBV-INFX410 In order to start this exercise, you need to be logged in on a UNIX computer with a terminal window open on your computer. It is best if you are logged in on freebee.abel.uio.no.

More information

Introduction to HPC Using zcluster at GACRC

Introduction to HPC Using zcluster at GACRC Introduction to HPC Using zcluster at GACRC On-class STAT8330 Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu Slides courtesy: Zhoufei Hou 1 Outline What

More information

Logging in to the CRAY

Logging in to the CRAY Logging in to the CRAY 1. Open Terminal Cray Hostname: cray2.colostate.edu Cray IP address: 129.82.103.183 On a Mac 2. type ssh username@cray2.colostate.edu where username is your account name 3. enter

More information

Introduction to the shell Part II

Introduction to the shell Part II Introduction to the shell Part II Graham Markall http://www.doc.ic.ac.uk/~grm08 grm08@doc.ic.ac.uk Civil Engineering Tech Talks 16 th November, 1pm Last week Covered applications and Windows compatibility

More information

Linux and Git Boot Camp

Linux and Git Boot Camp Linux and Git Boot Camp Roshan, Zack, Blair, Ian Jan. 21, 2018 1 Connecting Clients SSH Windows users: MobaXterm, PuTTY, SSH Tectia Mac & Linux users: Terminal (Just type ssh) ssh andrewid@shark.ics.cs.cmu.edu

More information

Introduction to Computing V - Linux and High-Performance Computing

Introduction to Computing V - Linux and High-Performance Computing Introduction to Computing V - Linux and High-Performance Computing Jonathan Mascie-Taylor (Slides originally by Quentin CAUDRON) Centre for Complexity Science, University of Warwick Outline 1 Program Arguments

More information

DATA 301 Introduction to Data Analytics Command Line. Dr. Ramon Lawrence University of British Columbia Okanagan

DATA 301 Introduction to Data Analytics Command Line. Dr. Ramon Lawrence University of British Columbia Okanagan DATA 301 Introduction to Data Analytics Command Line Dr. Ramon Lawrence University of British Columbia Okanagan ramon.lawrence@ubc.ca Why learn the Command Line? The command line is the text interface

More information

CS/IT 114 Introduction to Java, Part 1 FALL 2016 CLASS 2: SEP. 8TH INSTRUCTOR: JIAYIN WANG

CS/IT 114 Introduction to Java, Part 1 FALL 2016 CLASS 2: SEP. 8TH INSTRUCTOR: JIAYIN WANG CS/IT 114 Introduction to Java, Part 1 FALL 2016 CLASS 2: SEP. 8TH INSTRUCTOR: JIAYIN WANG 1 Notice Class Website http://www.cs.umb.edu/~jane/cs114/ Reading Assignment Chapter 1: Introduction to Java Programming

More information

You can use the WinSCP program to load or copy (FTP) files from your computer onto the Codd server.

You can use the WinSCP program to load or copy (FTP) files from your computer onto the Codd server. CODD SERVER ACCESS INSTRUCTIONS OVERVIEW Codd (codd.franklin.edu) is a server that is used for many Computer Science (COMP) courses. To access the Franklin University Linux Server called Codd, an SSH connection

More information

Using the Zoo Workstations

Using the Zoo Workstations Using the Zoo Workstations Version 1.86: January 16, 2014 If you ve used Linux before, you can probably skip many of these instructions, but skim just in case. Please direct corrections and suggestions

More information

Linux Survival Guide

Linux Survival Guide Linux Survival Guide Introduction: This guide is intended for use with classes at DACC that use a Linux operating system as the platform for students. It provides a quick summary and examples of how to

More information

GNU/Linux 101. Casey McLaughlin. Research Computing Center Spring Workshop Series 2018

GNU/Linux 101. Casey McLaughlin. Research Computing Center Spring Workshop Series 2018 GNU/Linux 101 Casey McLaughlin Research Computing Center Spring Workshop Series 2018 rccworkshop IC;3df4mu bash-2.1~# man workshop Linux101 RCC Workshop L101 OBJECTIVES - Operating system concepts - Linux

More information

BIOINFORMATICS POST-DIPLOMA PROGRAM SUBJECT OUTLINE Subject Title: OPERATING SYSTEMS AND PROJECT MANAGEMENT Subject Code: BIF713 Subject Description:

BIOINFORMATICS POST-DIPLOMA PROGRAM SUBJECT OUTLINE Subject Title: OPERATING SYSTEMS AND PROJECT MANAGEMENT Subject Code: BIF713 Subject Description: BIOINFORMATICS POST-DIPLOMA PROGRAM SUBJECT OUTLINE Subject Title: OPERATING SYSTEMS AND PROJECT MANAGEMENT Subject Code: BIF713 Subject Description: This course provides Bioinformatics students with the

More information

Introduction. File System. Note. Achtung!

Introduction. File System. Note. Achtung! 3 Unix Shell 1: Introduction Lab Objective: Explore the basics of the Unix Shell. Understand how to navigate and manipulate file directories. Introduce the Vim text editor for easy writing and editing

More information

Using the MaRC2 HPC Cluster

Using the MaRC2 HPC Cluster Using the MaRC2 HPC Cluster Manuel Haim, 06/2013 Using MaRC2??? 2 Using MaRC2 Overview Get access rights and permissions Starting a terminal session (Linux, Windows, Mac) Intro to the BASH Shell (and available

More information

Oregon State University School of Electrical Engineering and Computer Science. CS 261 Recitation 1. Spring 2011

Oregon State University School of Electrical Engineering and Computer Science. CS 261 Recitation 1. Spring 2011 Oregon State University School of Electrical Engineering and Computer Science CS 261 Recitation 1 Spring 2011 Outline Using Secure Shell Clients GCC Some Examples Intro to C * * Windows File transfer client:

More information

Linux Command Line Primer. By: Scott Marshall

Linux Command Line Primer. By: Scott Marshall Linux Command Line Primer By: Scott Marshall Draft: 10/21/2007 Table of Contents Topic Page(s) Preface 1 General Filesystem Background Information 2 General Filesystem Commands 2 Working with Files and

More information

Research. We make it happen. Unix Basics. User Support Group help-line: personal:

Research. We make it happen. Unix Basics. User Support Group help-line: personal: Research. We make it happen. Unix Basics Presented by: Patton Fast User Support Group help-line: help@msi.umn.edu 612-626-0802 personal: pfast@msi.umn.edu 612-625-6573 Outline I. Warnings! II. III. IV.

More information

Introduction to Linux. Fundamentals of Computer Science

Introduction to Linux. Fundamentals of Computer Science Introduction to Linux Fundamentals of Computer Science Outline Operating Systems Linux History Linux Architecture Logging in to Linux Command Format Linux Filesystem Directory and File Commands Wildcard

More information

Introduction to UNIX command-line II

Introduction to UNIX command-line II Introduction to UNIX command-line II Boyce Thompson Institute 2017 Prashant Hosmani Class Content Terminal file system navigation Wildcards, shortcuts and special characters File permissions Compression

More information

Linux File System and Basic Commands

Linux File System and Basic Commands Linux File System and Basic Commands 0.1 Files, directories, and pwd The GNU/Linux operating system is much different from your typical Microsoft Windows PC, and probably looks different from Apple OS

More information

CSC111 Computer Science II

CSC111 Computer Science II CSC111 Computer Science II Lab 1 Getting to know Linux Introduction The purpose of this lab is to introduce you to the command line interface in Linux. Getting started In our labs If you are in one of

More information

CHE3935. Lecture 1. Introduction to Linux

CHE3935. Lecture 1. Introduction to Linux CHE3935 Lecture 1 Introduction to Linux 1 Logging In PuTTY is a free telnet/ssh client that can be run without installing it within Windows. It will only give you a terminal interface, but used with a

More information

Introduction in Unix. Linus Torvalds Ken Thompson & Dennis Ritchie

Introduction in Unix. Linus Torvalds Ken Thompson & Dennis Ritchie Introduction in Unix Linus Torvalds Ken Thompson & Dennis Ritchie My name: John Donners John.Donners@surfsara.nl Consultant at SURFsara And Cedric Nugteren Cedric.Nugteren@surfsara.nl Consultant at SURFsara

More information

Command Line Interface The basics

Command Line Interface The basics Command Line Interface The basics Marco Berghoff, SCC, KIT Steinbuch Centre for Computing (SCC) Funding: www.bwhpc-c5.de Motivation In the Beginning was the Command Line by Neal Stephenson In contrast

More information

A Brief Introduction to The Center for Advanced Computing

A Brief Introduction to The Center for Advanced Computing A Brief Introduction to The Center for Advanced Computing November 10, 2009 Outline 1 Resources Hardware Software 2 Mechanics: Access Transferring files and data to and from the clusters Logging into the

More information

L.A.M.P. Stack Part I

L.A.M.P. Stack Part I L.A.M.P. Stack Part I By George Beatty and Matt Frantz This lab will cover the basic installation and some configuration of a LAMP stack on a Ubuntu virtual box. Students will download and install the

More information

Lab Working with Linux Command Line

Lab Working with Linux Command Line Introduction In this lab, you will use the Linux command line to manage files and folders and perform some basic administrative tasks. Recommended Equipment A computer with a Linux OS, either installed

More information

When you first log in, you will be placed in your home directory. To see what this directory is named, type:

When you first log in, you will be placed in your home directory. To see what this directory is named, type: Chem 7520 Unix Crash Course Throughout this page, the command prompt will be signified by > at the beginning of a line (you do not type this symbol, just everything after it). Navigation When you first

More information

EECS Software Tools. Lab 2 Tutorial: Introduction to UNIX/Linux. Tilemachos Pechlivanoglou

EECS Software Tools. Lab 2 Tutorial: Introduction to UNIX/Linux. Tilemachos Pechlivanoglou EECS 2031 - Software Tools Lab 2 Tutorial: Introduction to UNIX/Linux Tilemachos Pechlivanoglou (tipech@eecs.yorku.ca) Sep 22 & 25, 2017 Material marked with will be in your exams Sep 22 & 25, 2017 Introduction

More information

Mid Term from Feb-2005 to Nov 2012 CS604- Operating System

Mid Term from Feb-2005 to Nov 2012 CS604- Operating System Mid Term from Feb-2005 to Nov 2012 CS604- Operating System Latest Solved from Mid term Papers Resource Person Hina 1-The problem with priority scheduling algorithm is. Deadlock Starvation (Page# 84) Aging

More information

Introduction to Linux

Introduction to Linux p. 1/40 Introduction to Linux Xiaoxu Guan High Performance Computing, LSU January 31, 2018 p. 2/40 Outline What is an OS or Linux OS? Basic commands for files/directories Basic commands for text processing

More information

Basic Survival UNIX.

Basic Survival UNIX. Basic Survival UNIX Many Unix based operating systems make available a Graphical User Interface for the sake of providing an easy way for less experienced users to work with the system. Some examples are

More information

Perl and R Scripting for Biologists

Perl and R Scripting for Biologists Perl and R Scripting for Biologists Lukas Mueller PLBR 4092 Course overview Linux basics (today) Linux advanced (Aure, next week) Why Linux? Free open source operating system based on UNIX specifications

More information

Using WestGrid from the desktop Oct on Access Grid

Using WestGrid from the desktop Oct on Access Grid Using WestGrid from the desktop Oct 11 2007 on Access Grid Introduction Simon Sharpe, UCIT Client Services The best way to contact WestGrid support is to email support@westgrid.ca This seminar gives you

More information

Lecture 3. Unix. Question? b. The world s best restaurant. c. Being in the top three happiest countries in the world.

Lecture 3. Unix. Question? b. The world s best restaurant. c. Being in the top three happiest countries in the world. Lecture 3 Unix Question? Denmark is famous for? a. LEGO. b. The world s best restaurant. c. Being in the top three happiest countries in the world. d. Having the highest taxes in Europe (57%). e. All of

More information

Programming introduction part I:

Programming introduction part I: Programming introduction part I: Perl, Unix/Linux and using the BlueHive cluster Bio472- Spring 2014 Amanda Larracuente Text editor Syntax coloring Recognize several languages Line numbers Free! Mac/Windows

More information

Introduction to Supercomputing

Introduction to Supercomputing Introduction to Supercomputing TMA4280 Introduction to UNIX environment and tools 0.1 Getting started with the environment and the bash shell interpreter Desktop computers are usually operated from a graphical

More information

Unix L555. Dept. of Linguistics, Indiana University Fall Unix. Unix. Directories. Files. Useful Commands. Permissions. tar.

Unix L555. Dept. of Linguistics, Indiana University Fall Unix. Unix. Directories. Files. Useful Commands. Permissions. tar. L555 Dept. of Linguistics, Indiana University Fall 2010 1 / 21 What is? is an operating system, like DOS or Windows developed in 1969 by Bell Labs works well for single computers as well as for servers

More information

ICS-ACI System Basics

ICS-ACI System Basics ICS-ACI System Basics Adam W. Lavely, Ph.D. Fall 2017 Slides available: goo.gl/ss9itf awl5173 ICS@PSU 1 Contents 1 Overview 2 HPC Overview 3 Getting Started on ACI 4 Moving On awl5173 ICS@PSU 2 Contents

More information

Linux hep.wisc.edu

Linux hep.wisc.edu Linux Environment @ hep.wisc.edu 1 Your Account : Login Name and usage You are given a unique login name (e.g. john) A temporary password is given to you Use this to login name and password to enter the

More information

Firewalls can prevent access to the Unix Servers. Please make sure any firewall software or hardware allows access through Port 22.

Firewalls can prevent access to the Unix Servers. Please make sure any firewall software or hardware allows access through Port 22. EINSTEIN OVERVIEW Einstein (Einstein.franklin.edu) and Codd (codd.franklin.edu) are two servers that are used for many Computer Science (COMP) courses. Students will be directed to use either Einstein

More information

Lec 1 add-on: Linux Intro

Lec 1 add-on: Linux Intro Lec 1 add-on: Linux Intro Readings: - Unix Power Tools, Powers et al., O Reilly - Linux in a Nutshell, Siever et al., O Reilly Summary: - Linux File System - Users and Groups - Shell - Text Editors - Misc

More information

Introduction to Unix and Linux. Workshop 1: Directories and Files

Introduction to Unix and Linux. Workshop 1: Directories and Files Introduction to Unix and Linux Workshop 1: Directories and Files Genomics Core Lab TEXAS A&M UNIVERSITY CORPUS CHRISTI Anvesh Paidipala, Evan Krell, Kelly Pennoyer, Chris Bird Genomics Core Lab Informatics

More information

: the User (owner) for this file (your cruzid, when you do it) Position: directory flag. read Group.

: the User (owner) for this file (your cruzid, when you do it) Position: directory flag. read Group. CMPS 12L Introduction to Programming Lab Assignment 2 We have three goals in this assignment: to learn about file permissions in Unix, to get a basic introduction to the Andrew File System and it s directory

More information

Lab 2 Building on Linux

Lab 2 Building on Linux Lab 2 Building on Linux Assignment Details Assigned: January 28 th, 2013. Due: January 30 th, 2013 at midnight. Background This assignment should introduce the basic development tools on Linux. This assumes

More information

XSEDE Scholars Program Introduction to the Linux Environment. Tips and Tricks for Linux Users John Lockman III June 5 th, 2012

XSEDE Scholars Program Introduction to the Linux Environment. Tips and Tricks for Linux Users John Lockman III June 5 th, 2012 XSEDE Scholars Program Introduction to the Linux Environment Tips and Tricks for Linux Users John Lockman III June 5 th, 2012 Tools for this Course You will need a Linux terminal to do the examples as

More information

Operating Systems 3. Operating Systems. Content. What is an Operating System? What is an Operating System? Resource Abstraction and Sharing

Operating Systems 3. Operating Systems. Content. What is an Operating System? What is an Operating System? Resource Abstraction and Sharing Content 3 Operating Systems The concept of an operating system. The internal architecture of an operating system. The architecture of the Linux operating system in more detail. How to log into (and out

More information

Saint Louis University. Intro to Linux and C. CSCI 2400/ ECE 3217: Computer Architecture. Instructors: David Ferry

Saint Louis University. Intro to Linux and C. CSCI 2400/ ECE 3217: Computer Architecture. Instructors: David Ferry Intro to Linux and C CSCI 2400/ ECE 3217: Computer Architecture Instructors: David Ferry 1 Overview Linux C Hello program in C Compiling 2 History of Linux Way back in the day: Bell Labs Unix Widely available

More information

Grid Engine Users Guide. 7.0 Edition

Grid Engine Users Guide. 7.0 Edition Grid Engine Users Guide 7.0 Edition Grid Engine Users Guide : 7.0 Edition Published Dec 01 2017 Copyright 2017 University of California and Scalable Systems This document is subject to the Rocks License

More information

Lecture-4. Introduction to Unix: More Commands, Boot-up Actions and X Window

Lecture-4. Introduction to Unix: More Commands, Boot-up Actions and X Window Lecture-4 Introduction to Unix: More Commands, Boot-up Actions and X Window What You Will Learn We continue to give more information about the fundamental commands of the Unix operating system. We also

More information

Analytical Processing of Data of statistical genetics research in UNIX like Systems

Analytical Processing of Data of statistical genetics research in UNIX like Systems Survival Skills for Analytical Processing of Data of statistical genetics research in UNIX like Systems robert yu :: March 2011 anote UNIX like? Traditional/classical UNIX, e.g. System V (Solaris), BSD

More information

How To Upload Your Newsletter

How To Upload Your Newsletter How To Upload Your Newsletter Using The WS_FTP Client Copyright 2005, DPW Enterprises All Rights Reserved Welcome, Hi, my name is Donna Warren. I m a certified Webmaster and have been teaching web design

More information

Shell Programming Overview

Shell Programming Overview Overview Shell programming is a way of taking several command line instructions that you would use in a Unix command prompt and incorporating them into one program. There are many versions of Unix. Some

More information

Batch Systems. Running calculations on HPC resources

Batch Systems. Running calculations on HPC resources Batch Systems Running calculations on HPC resources Outline What is a batch system? How do I interact with the batch system Job submission scripts Interactive jobs Common batch systems Converting between

More information

Unix Workshop Aug 2014

Unix Workshop Aug 2014 Unix Workshop 2014 5 Aug 2014 What is Unix Multitasking, multiuser operating system Often the OS of choice for large servers, large clusters Unix Around You You re probably familiar with these: Linux Solaris

More information

CSCI 201 Lab 1 Environment Setup

CSCI 201 Lab 1 Environment Setup CSCI 201 Lab 1 Environment Setup "The journey of a thousand miles begins with one step." - Lao Tzu Introduction This lab document will go over the steps to install and set up Eclipse, which is a Java integrated

More information

Linux shell scripting Getting started *

Linux shell scripting Getting started * Linux shell scripting Getting started * David Morgan *based on chapter by the same name in Classic Shell Scripting by Robbins and Beebe What s s a script? text file containing commands executed as a unit

More information

Grid Engine Users Guide. 6.1 Edition

Grid Engine Users Guide. 6.1 Edition Grid Engine Users Guide 6.1 Edition Grid Engine Users Guide : 6.1 Edition Published Nov 27 2012 Copyright 2012 University of California and Scalable Systems This document is subject to the Rocks License

More information

Practical Session 0 Introduction to Linux

Practical Session 0 Introduction to Linux School of Computer Science and Software Engineering Clayton Campus, Monash University CSE2303 and CSE2304 Semester I, 2001 Practical Session 0 Introduction to Linux Novell accounts. Every Monash student

More information

bash, part 3 Chris GauthierDickey

bash, part 3 Chris GauthierDickey bash, part 3 Chris GauthierDickey More redirection As you know, by default we have 3 standard streams: input, output, error How do we redirect more than one stream? This requires an introduction to file

More information

Triton file systems - an introduction. slide 1 of 28

Triton file systems - an introduction. slide 1 of 28 Triton file systems - an introduction slide 1 of 28 File systems Motivation & basic concepts Storage locations Basic flow of IO Do's and Don'ts Exercises slide 2 of 28 File systems: Motivation Case #1:

More information