Data transfer at CINECA: how to enjoy it!
|
|
- Martha Charity Thornton
- 5 years ago
- Views:
Transcription
1 Data transfer at CINECA: how to enjoy it! Giacomo Mariani Bologna 03/05/2011
2 Table of contents Resources What we have Use Case What users need Tools How we support them
3 Resources
4 File system HPC machines at CINECA (PLX and SP6) have similar FS organization:
5 File system $HOME: permanent/backuped, local, 2GB; $CINECA_DATA: permanent/not backuped, shared, i.e. GPFS, quite slow; $CINECA_SCRATCH: temporary, local (20 days retention), very performing; cart(uccia) backup your data on tape.
6 Network One HPC machine is a cluster: lots of nodes connected through an HP-network (infiniband). Each cluster is connected to the others through a private network. The clusters are reachable from the public network through GARR (1Gb/s). DEISA2/PRACE projects have a dedicated private network which has 10Gb/s guaranteed bandwidth.
7 A-Move data from user workstations to CINECA file systems and vice-versa. Use Cases B-Move data across file systems on a single HPC machine. C-Move data across different machines. D-Archive data.
8 Tools
9 cp/mv/scp/sftp CP and MV are local commands. SFTP and SCP are secure methods for transferring data among computers on a network: SFTP stands for "Secure File Transfer Protocal"; SCP stands for "Secure CoPy". The data are encrypted before being sent across the Internet.
10 cp/mv/scp/sftp The SFTP and SCP commands allow you to connect from a local machine to a remote machine in order to transfer data. The machine on which you run the client software is the local system. The machine on the other end of your data transfer session (the machine running the server software) is the remote machine.
11 cp/mv/scp/sftp: use cases A: $HOME/path/to/file B: $HOME/path/to/file $CINECA_SCRATCH/path/to/ $CINECA_SCRATCH/path/to/file $CINECA_DATA/path/to/
12 rsync rsync is a software tool for Unix and Windows systems which synchronizes files and directories from one location to another while optimizing data transfer. rsync tries to optimize the exploitation of network performance. rsync can copy or show directory contents and copy files, even using compression and recursion.
13 rsync rsync was originally written as a replacement for rcp and scp. Like its predecessors, it still requires a source and a destination to be specified, one of these may be remote. rsync does not overwrite unmodified files.
14 rsync: use cases A: -va $HOME/path/to/dir B: load rsync --timeout=600 -avhs -r --numericids --bwlimit= block-size= progress $CINECA_SCRATCH/path/file $CINECA_DATA/path/ C: -va $CINECA_DATA/path/to/directory $HPC_DATA/path/to/
15 What is GridFTP GridFTP is a protocol which extends established technologies like FTP (File Transfer Protocol) and SCP (Secure CoPy) with the following improvements: Authentication via GSI Multiple parallel channels with streams and stripes Third-party transfers Tunability for network and I/O parameters
16 How do I use globus-url-copy? GridFTP main client is used like a normal FTP client. It may only be necessary to define some variables. Transferring: user$ globus-url-copy sshftp://<username>@sp.sp6.cineca.it/<remote_path/to/yourfile> file:///home/user/<local_path/to/yourfile> user$ globus-url-copy gsiftp://grid.cineca.it/<remote_path/to/yourfile> file:///home/user/<local_path/to/yourfile>
17 How do I authenticate with globus-url-copy? GridFTP main client is able to manage various authentication mechanisms. The instances at CINECA only support two of them: username-password authentication (eventually with RSA keys) GSI authentication
18 Useful options -p <number>: number of streams, i.e. parallel tcp channels. -stripe: enable striped tranfer. -tcp-bs <size>: dimension (in bytes) of the tcp buffer. -pp: allow pipelining. -list: used with one argument, lists the given directory. -v: verbose output... -dbg: very verbose output.
19 Third Party Transfer with Stripe GridFTP Client GridFTP Server1 FE PUT/SPUT GET/SGET GridFTP Server2 FE BEn BEn
20 Clients: CL UberFTP: is the most official client and improves globus-url-copy making it more interactive: Interactive Commands This listing is generated by typing 'help' at the command prompt.!? active ascii binary blksize bugs bye cat cd cdup chgrp chmod cksum close dcau debug dir family get glob hash help keepalive lcat lcd lcdup lchgrp lchmod lclose ldir lls lmkdir lopen lpwd lquote lrename lrm lrmdir ls lsize lstage mget mkdir mode mput open order parallel passive pbsz pget pput prot put pwd quit quote rename resume retry rm rmdir runique size stage sunique tcpbuf versions wait
21 Clients: Java Java CoG Kit Project Webstart: File Transfer GUI.
22 Clients: Java Java-Eclipse: GridFTP Client. \
23 GridFTP: use cases A: /path/to/file C: -pp -restart gsiftp://grid.cineca.it/path/to/dir -p 8 -restart gsiftp://grid.cineca.it/path/to/file sshftp://user@remote-hpc/path/to/
24 Usage example Examples: -list \ sshftp://user@sp.sp6.cineca.it/ user@bsd>globus-url-copy -list \ gsiftp://grid.cineca.it/ user@bsd>grid-proxy-init user@bsd>globus-url-copy -list \ gsiftp://grid.cineca.it/ user@bsd>uberftp UberFTP>help UberFTP>open grid.cineca.it UberFTP>ls; pwd
25 Usage example Examples: -Y -vb -len 50M \ /dev/zero gsiftp://grid.cineca.it/dev/null user@astrct:~>globus-url-copy -vb -p 6 \ -len 50M /dev/zero \ gsiftp://grid.cineca.it/dev/null
26 Ready available at CINECA A pubblic installation (without -stripe option), available for both CINECA and DEISA -users, which is reachable with: SSH authentication at sshftp://user@sp.sp6.cineca.it GSI authentication at gsiftp://grid.cineca.it A DEISA-user only installation (with -stripe option) reachable at: GSI only at gsiftp://grid-deisa.sp6.cineca.it All of them are on SP6 and let users read and write filesystems on the base of their permissions.
27 cart commands You need to be habilitated in order to use the cart commands. To obtain an habilitation send an to The default quota is 500GB, but it can be extended providing some good reasons. The official guide is available at:
28 cart commands cart_new <vol_name>: create a new volume <vol_name> cart_dir: show all the defined volumes from the user cart_dir <vol_name>: show the files stored on the volume <vol_name> cart_put <vol_name> <file>: save <file> in the volume <vol_name> cart_get <vol_name> <file>: retrieve the <file> from the volume <vol_name> cart_del <vol_name> <file>: delete the <file> from the volume <vol_name> cart_del <vol_name>: delete the volume <vol_name> (the volume is empty) cart_acl -u <user> <vol_name>: grant access to a given volume for a specific user cart_dir_access -u <owner> [<vol_name>]: show the volumes/files of a different owner (cart_acl is needed) cart_get_access -u <owner> <vol_name> <file>: retrieve <file> from a volume of a different owner (cart_acl is needed)
29 cart commands #!/bin/bash job_name = archive output = archive.out error = archive.err wall_clock_limit = 30:00 job_type = serial resources = ConsumableMemory(1500MB) class = archive account_no = myaccount! only for new style users queue export CART=cart_name export DIR=$CINECA_SCRATCH/xxx echo "storing my " $DIR " directory on cartridge " $CART cart_dir cart_dir $CART ### tar the directory cd $CINECA_SCRATCH tar -cvzf $DIR.tar $DIR ### copy my tar file on the cart cart_put $CART $DIR.tar
30 Use cases: comparative table A: Local CINECA B: CINECA CINECA C: CINECA HPC-machines D: CINECA Archive cp/mv VV scp/sftp VV V rsync VV V V GridFTP V VVV cart_* V
31 File dimension: comparative table remote<->remote local<->remote scp/sft (interactive) Not Available 20GB rsync (interactive) Not Available 20GB globus-url-copy (interactive) Unlimited 150GB scp/sft (batch) Not Available WCT-limited rsync (batch) Not Available WCT-limited globus-url-copy (batch) Unlimited WCT-limited WCT=Wall Clock Time
32 Data management page at CINECA. GridFTP page at Globus official site. GridFTP page at CINECA Official CINECA GridFTP guide UberFTP official web-page Globus GUI web-page OFFIS GUI web-page For further informations... Documentation
33 :wq Thanks
Mass Storage System User Guide
Mass Storage System User Guide NCSA Mass Storage System User Guide Table of Contents 1. 2. 3. 4. 5. 6. 7. Overview Managing Home and Project Space Access MSS Commands Using MSS Effectively Sample Sessions
More informationPart 5: FTP More about this course about Internet
*:96 Overheads 2a-1 Part 5: FTP More about this course about Internet application protocols can be found at URL: http://www.dsv.su.se/jpalme/internet-course/int-appprot-kurs.html Last update: 2005-09-29
More informationData Management at ARSC
Data Management at ARSC David Newman (From slides by Tom Logan) (from Slides from Don Bahls) Presentation Overview 1. ARSC storage 2. Data Management within ARSC 3. Additional Notes on Long Term Storage
More informationData Movement and Storage. 04/07/09 1
Data Movement and Storage 04/07/09 www.cac.cornell.edu 1 Data Location, Storage, Sharing and Movement Four of the seven main challenges of Data Intensive Computing, according to SC06. (Other three: viewing,
More informationSecure Shell Commands
This module describes the Cisco IOS XR software commands used to configure Secure Shell (SSH). For detailed information about SSH concepts, configuration tasks, and examples, see the Implementing Secure
More informationTable of Contents 1 FTP and SFTP Configuration TFTP Configuration 2-1
Table of Contents 1 FTP and SFTP Configuration 1-1 Introduction to FTP and SFTP 1-1 Introduction to FTP 1-1 Introduction to SFTP 1-1 FTP Configuration 1-2 FTP Configuration: A Switch Operating as an FTP
More informationLinux Training. for New Users of Cluster. Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala
Linux Training for New Users of Cluster Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu 1 Overview GACRC Linux Operating System Shell, Filesystem, and Common
More informationCENG 334 Computer Networks. Laboratory I Linux Tutorial
CENG 334 Computer Networks Laboratory I Linux Tutorial Contents 1. Logging In and Starting Session 2. Using Commands 1. Basic Commands 2. Working With Files and Directories 3. Permission Bits 3. Introduction
More informationContents. FTP commands 1
Contents FTP commands 1 FTP server commands 1 display ftp-server 1 display ftp-user 1 free ftp user 2 free ftp user-ip 3 free ftp user-ip ipv6 3 ftp server acl 4 ftp server dscp 5 ftp server enable 5 ftp
More informationContents. Configuring SSH 1
Contents Configuring SSH 1 Overview 1 How SSH works 1 SSH authentication methods 2 SSH support for Suite B 3 FIPS compliance 3 Configuring the device as an SSH server 4 SSH server configuration task list
More informationSecure Shell Commands
This module describes the Cisco IOS XR software commands used to configure Secure Shell (SSH). For detailed information about SSH concepts, configuration tasks, and examples, see the Implementing Secure
More informationWelcome to getting started with Ubuntu Server. This System Administrator Manual. guide to be simple to follow, with step by step instructions
Welcome to getting started with Ubuntu 12.04 Server. This System Administrator Manual guide to be simple to follow, with step by step instructions with screenshots INDEX 1.Installation of Ubuntu 12.04
More informationCOMPUTE CANADA GLOBUS PORTAL
COMPUTE CANADA GLOBUS PORTAL Fast, user-friendly data transfer and sharing Jason Hlady University of Saskatchewan WestGrid / Compute Canada February 4, 2015 Why Globus? I need to easily, quickly, and reliably
More informationUnix/Linux Operating System. Introduction to Computational Statistics STAT 598G, Fall 2011
Unix/Linux Operating System Introduction to Computational Statistics STAT 598G, Fall 2011 Sergey Kirshner Department of Statistics, Purdue University September 7, 2011 Sergey Kirshner (Purdue University)
More informationSession 1: Accessing MUGrid and Command Line Basics
Session 1: Accessing MUGrid and Command Line Basics Craig A. Struble, Ph.D. July 14, 2010 1 Introduction The Marquette University Grid (MUGrid) is a collection of dedicated and opportunistic resources
More informationThis lab exercise is to be submitted at the end of the lab session! passwd [That is the command to change your current password to a new one]
Data and Computer Security (CMPD414) Lab II Topics: secure login, moving into HOME-directory, navigation on Unix, basic commands for vi, Message Digest This lab exercise is to be submitted at the end of
More informationTable of Contents 1 FTP Configuration Commands TFTP Configuration Commands 2-1
Table of Contents 1 FTP Configuration Commands 1-1 FTP Server Configuration Commands 1-1 display ftp-server 1-1 display ftp-user 1-2 free ftp user 1-3 ftp server acl 1-3 ftp server enable 1-4 ftp timeout
More informationIT Services Security. The Dark Arts Of SSH. Author: John Curran Version: 0.1
IT Services Security The Dark Arts Of SSH Author: John Curran Version: 0.1 STATUS\ REVISION HISTORY Date Version Description 0.1-0.9 Review preparation 1.0 Presented to business & retained by Information
More informationIntroduction to the Linux Command Line
Introduction to the Linux Command Line May, 2015 How to Connect (securely) ssh sftp scp Basic Unix or Linux Commands Files & directories Environment variables Not necessarily in this order.? Getting Connected
More informationEMC DiskXtender for UNIX/Linux Storage Manager Release 2.11
EMC DiskXtender for UNIX/Linux Storage Manager Release 2.11 User Guide P/N 300-006-394 REV A01 EMC Corporation Corporate Headquarters: Hopkinton, MA 01748-9103 1-508-435-1000 www.emc.com Copyright 2002
More informationSIEMENS UserAdmin Workshop TELEPERM XP Version 4 Chapter 1
1 UNIX... 1-2 1.1 USER-ENVIRONMENT... 1-2 1.1.1 User-environment... 1-3 1.1.2 Basic commands... 1-3 1.1.3 SCO Unix specific commands... 1-4 1.1.4 HP Unix specific commands... 1-4 1.2 FILE SYSTEM... 1-5
More informationCS CS Tutorial 2 2 Winter 2018
CS CS 230 - Tutorial 2 2 Winter 2018 Sections 1. Unix Basics and connecting to CS environment 2. MIPS Introduction & CS230 Interface 3. Connecting Remotely If you haven t set up a CS environment password,
More informationbwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR
More informationBasic UNIX commands. HORT Lab 2 Instructor: Kranthi Varala
Basic UNIX commands HORT 59000 Lab 2 Instructor: Kranthi Varala Client/Server architecture User1 User2 User3 Server (UNIX/ Web/ Database etc..) User4 High Performance Compute (HPC) cluster User1 Compute
More informationIntroduction to UNIX. Logging in. Basic System Architecture 10/7/10. most systems have graphical login on Linux machines
Introduction to UNIX Logging in Basic system architecture Getting help Intro to shell (tcsh) Basic UNIX File Maintenance Intro to emacs I/O Redirection Shell scripts Logging in most systems have graphical
More informationTable of Contents 1 FTP Configuration Commands TFTP Configuration Commands 2-1
Table of Contents 1 FTP Configuration Commands 1-1 FTP Client Configuration Commands 1-1 ascii 1-1 binary 1-2 bye 1-2 cd 1-3 cdup 1-4 close 1-4 debugging 1-5 delete 1-6 dir 1-6 disconnect 1-8 display ftp
More informationIntroduction to the UNIX command line
Introduction to the UNIX command line Steven Abreu Introduction to Computer Science (ICS) Tutorial Jacobs University s.abreu@jacobs-university.de September 19, 2017 Overview What is UNIX? UNIX Shell Commands
More informationAn Introduction to Unix
An Introduction to Unix Sylvia Plöckinger March 3, 2011 Sylvia Plöckinger () An Introduction to Unix March 3, 2011 1 / 29 General Information Find this file on: http://homepage.univie.ac.at/nigel.mitchell/numprac/
More informationContents. TFTP configuration commands 29 display tftp client configuration 29 tftp-server acl 29 tftp 30 tftp client source 32 tftp ipv6 33
Contents FTP configuration commands 1 FTP server configuration commands 1 display ftp-server 1 display ftp-user 2 free ftp user 3 ftp server acl 3 ftp server enable 4 ftp timeout 5 ftp update 5 FTP client
More informationPerl and R Scripting for Biologists
Perl and R Scripting for Biologists Lukas Mueller PLBR 4092 Course overview Linux basics (today) Linux advanced (Aure, next week) Why Linux? Free open source operating system based on UNIX specifications
More informationCHAPTER 1 UNIX FOR NONPROGRAMMERS
CHAPTER 1 UNIX FOR NONPROGRAMMERS The man command is used to display the manual entry associated with word entered as argument. The -k option is used displays a list of manual entries that contain entered
More informationUNIX Essentials Featuring Solaris 10 Op System
A Active Window... 7:11 Application Development Tools... 7:7 Application Manager... 7:4 Architectures - Supported - UNIX... 1:13 Arithmetic Expansion... 9:10 B Background Processing... 3:14 Background
More informationSecure Shell Commands
Secure Shell Commands This module describes the Cisco IOS XR software commands used to configure Secure Shell (SSH). For detailed information about SSH concepts, configuration tasks, and examples, see
More informationLinux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades Q2
Linux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades 2017-2018 Q2 Facultat d Informàtica de Barcelona This first lab session is focused on getting experience in working
More informationBasic Unix Commands. CGS 3460, Lecture 6 Jan 23, 2006 Zhen Yang
Basic Unix Commands CGS 3460, Lecture 6 Jan 23, 2006 Zhen Yang For this class you need to work from your grove account to finish your homework Knowing basic UNIX commands is essential to finish your homework
More informationTFTP and FTP Basics BUPT/QMUL
TFTP and FTP Basics BUPT/QMUL 2017-04-24 Agenda File transfer and access TFTP (Trivial File Transfer Protocol) FTP (File Transfer Protocol) NFS (Network File System) 2 File Transfer And Access 3 File Transfer
More informationTIBCO ActiveMatrix BusinessWorks Plug-in for sftp User's Guide
TIBCO ActiveMatrix BusinessWorks Plug-in for sftp User's Guide Software Release 6.1 January 2016 Two-Second Advantage 2 Important Information SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE.
More informationContents. TFTP configuration commands 28 display tftp client configuration 28 tftp-server acl 28 tftp 29 tftp client source 30 tftp ipv6 31
Contents FTP configuration commands 1 FTP server configuration commands 1 display ftp-server 1 display ftp-user 2 free ftp user 3 ftp server acl 3 ftp server enable 4 ftp timeout 4 ftp update 5 FTP client
More informationLab 1 Introduction to UNIX and C
Name: Lab 1 Introduction to UNIX and C This first lab is meant to be an introduction to computer environments we will be using this term. You must have a Pitt username to complete this lab. NOTE: Text
More informationbwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs
bwunicluster Tutorial Access, Data Transfer, Compiling, Modulefiles, Batch Jobs Frauke Bösert, SCC, KIT 1 Material: Slides & Scripts https://indico.scc.kit.edu/indico/event/263/ @bwunicluster/forhlr I/ForHLR
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationQuick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing
Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Linux/Unix basic commands Basic command structure:
More informationFiles
http://www.cs.fsu.edu/~langley/cop3353-2013-1/reveal.js-2013-02-11/02.html?print-pdf 02/11/2013 10:55 AM Files A normal "flat" file is a collection of information. It's usually stored somewhere reasonably
More informationChapter Two. Lesson A. Objectives. Exploring the UNIX File System and File Security. Understanding Files and Directories
Chapter Two Exploring the UNIX File System and File Security Lesson A Understanding Files and Directories 2 Objectives Discuss and explain the UNIX file system Define a UNIX file system partition Use the
More informationRH033 Red Hat Linux Essentials
RH033 Red Hat Linux Essentials Version 3.5 QUESTION NO: 1 You work as a Network Administrator for McNeil Inc. The company has a Linux-based network. A printer is configured on the network. You want to
More informationComputer Systems and Architecture
Computer Systems and Architecture Introduction to UNIX Stephen Pauwels University of Antwerp October 2, 2015 Outline What is Unix? Getting started Streams Exercises UNIX Operating system Servers, desktops,
More informationData Staging: Moving large amounts of data around, and moving it close to compute resources
Data Staging: Moving large amounts of data around, and moving it close to compute resources PRACE advanced training course on Data Staging and Data Movement Helsinki, September 10 th 2013 Claudio Cacciari
More information$ftp = Net::FTP->new("some.host.name", Debug => 0) or die "Cannot connect to some.host.name:
NAME Net::FTP - FTP Client class SYNOPSIS use Net::FTP; $ftp = Net::FTP->new("some.host.name", Debug => 0) or die "Cannot connect to some.host.name: $@"; $ftp->login("anonymous",'-anonymous@') or die "Cannot
More informationIntroduction to UNIX Command Line
Introduction to UNIX Command Line Files and directories Some useful commands (echo, cat, grep, find, diff, tar) Redirection Pipes Variables Background processes Remote connections (e.g. ssh, curl) Scripts
More informationUnix Introduction to UNIX
Unix Introduction to UNIX Get Started Introduction The UNIX operating system Set of programs that act as a link between the computer and the user. Developed in 1969 by a group of AT&T employees Various
More informationPractical Knowledge Transfering, moving and exporting files Martin Dahlö
Practical Knowledge Transfering, moving and exporting files 160318 Martin Dahlö martin.dahlo@scilifelab.uu.se UPPMAX Moving files to and from UPPMAX Moving files within UPPMAX Compressing files Web export
More informationWeek 2 Lecture 3. Unix
Lecture 3 Unix Terminal and Shell 2 Terminal Prompt Command Argument Result 3 Shell Intro A system program that allows a user to execute: shell functions (e.g., ls -la) other programs (e.g., eclipse) shell
More informationComputer Systems and Architecture
Computer Systems and Architecture Stephen Pauwels Computer Systems Academic Year 2018-2019 Overview of the Semester UNIX Introductie Regular Expressions Scripting Data Representation Integers, Fixed point,
More informationUnix/Linux Basics. Cpt S 223, Fall 2007 Copyright: Washington State University
Unix/Linux Basics 1 Some basics to remember Everything is case sensitive Eg., you can have two different files of the same name but different case in the same folder Console-driven (same as terminal )
More informationUFTP STANDALONE CLIENT
UFTP Standalone Client UFTP STANDALONE CLIENT UNICORE Team Document Version: 1.0.0 Component Version: 0.7.0 Date: 19 07 2017 UFTP Standalone Client Contents 1 Prerequisites 1 2 Installation 1 3 Basic Usage
More informationParallel Programming Pre-Assignment. Setting up the Software Environment
Parallel Programming Pre-Assignment Setting up the Software Environment Authors: B. Wilkinson and C. Ferner. Modification date: Aug 21, 2014 (Minor correction Aug 27, 2014.) Software The purpose of this
More informationUNIX Quick Reference
UNIX Quick Reference This card represents a brief summary of some of the more frequently used UNIX commands that all users should be at least somewhat familiar with. Some commands listed have much more
More informationData Staging: Moving large amounts of data around, and moving it close to compute resources
Data Staging: Moving large amounts of data around, and moving it close to compute resources Digital Preserva-on Advanced Prac--oner Course Glasgow, July 19 th 2013 c.cacciari@cineca.it Definition Starting
More informationIT 341: Introduction to System Administration. Notes for Project #8: Backing Up Files with rsync
IT 341: Introduction to System Administration Notes for Project #8: Backing Up Files with rsync These notes explain some of the concepts you will encounter in Project #08: Backing Up Files with rsync Topics
More informationCommon UNIX Commands. Unix. User Interfaces. Unix Commands Winter COMP 1270 Computer Usage II 9-1. Using UNIX. Unix has a command line interface
Common UNIX Commands Using UNIX Unix Unix has a command line interface Unix commands must be typed Similar to the DOS operating system for PC s Compare to the Graphical User Interface (GUI) used by Windows,
More informationIntroduction to the SHARCNET Environment May-25 Pre-(summer)school webinar Speaker: Alex Razoumov University of Ontario Institute of Technology
Introduction to the SHARCNET Environment 2010-May-25 Pre-(summer)school webinar Speaker: Alex Razoumov University of Ontario Institute of Technology available hardware and software resources our web portal
More informationMTU Computer Structure
1 MTU Computer Structure Home directory: h drive Same for Suns & PC s Location to store files Organize files by creating appropriate directories (folders) Subdirectory - any directory within in another
More informationUnix Filesystem. January 26 th, 2004 Class Meeting 2
Unix Filesystem January 26 th, 2004 Class Meeting 2 * Notes adapted by Christian Allgood from previous work by other members of the CS faculty at Virginia Tech Unix Filesystem! The filesystem is your interface
More informationUoW HPC Quick Start. Information Technology Services University of Wollongong. ( Last updated on October 10, 2011)
UoW HPC Quick Start Information Technology Services University of Wollongong ( Last updated on October 10, 2011) 1 Contents 1 Logging into the HPC Cluster 3 1.1 From within the UoW campus.......................
More informationCarnegie Mellon. Linux Boot Camp. Jack, Matthew, Nishad, Stanley 6 Sep 2016
Linux Boot Camp Jack, Matthew, Nishad, Stanley 6 Sep 2016 1 Connecting SSH Windows users: MobaXterm, PuTTY, SSH Tectia Mac & Linux users: Terminal (Just type ssh) andrewid@shark.ics.cs.cmu.edu 2 Let s
More informationOutline. Structure of a UNIX command
Outline Structure of Unix Commands Command help (man) Log on (terminal vs. graphical) System information (utility) File and directory structure (path) Permission (owner, group, rwx) File and directory
More informationIntroduction of Linux
Introduction of Linux 阳 oslab2018_class1@163.com 寅 oslab2018_class2@163.com PART I Brief Introduction Basic Conceptions & Environment Install & Configure a Virtual Machine Basic Commands PART II Shell
More informationThe cluster system. Introduction 22th February Jan Saalbach Scientific Computing Group
The cluster system Introduction 22th February 2018 Jan Saalbach Scientific Computing Group cluster-help@luis.uni-hannover.de Contents 1 General information about the compute cluster 2 Available computing
More informationCS Fundamentals of Programming II Fall Very Basic UNIX
CS 215 - Fundamentals of Programming II Fall 2012 - Very Basic UNIX This handout very briefly describes how to use Unix and how to use the Linux server and client machines in the CS (Project) Lab (KC-265)
More informationUsing UNIX. -rwxr--r-- 1 root sys Sep 5 14:15 good_program
Using UNIX. UNIX is mainly a command line interface. This means that you write the commands you want executed. In the beginning that will seem inferior to windows point-and-click, but in the long run the
More informationThe JANUS Computing Environment
Research Computing UNIVERSITY OF COLORADO The JANUS Computing Environment Monte Lunacek monte.lunacek@colorado.edu rc-help@colorado.edu What is JANUS? November, 2011 1,368 Compute nodes 16,416 processors
More informationUFTP STANDALONE CLIENT
UFTP Standalone Client UFTP STANDALONE CLIENT UNICORE Team Document Version: 1.0.0 Component Version: 0.5.0 Date: 14 06 2016 UFTP Standalone Client Contents 1 Prerequisites 1 2 Installation 1 3 Basic Usage
More informationPhysics REU Unix Tutorial
Physics REU Unix Tutorial What is unix? Unix is an operating system. In simple terms, its the set of programs that makes a computer work. It can be broken down into three parts. (1) kernel: The component
More information1.1 Target group. 1.2 Summary of contents
1 Preface The product internet Services supplements the TCP/IP functionality of opennet Server with the following standards: DNS Resolver and Server DHCP Server BOOTP Server TFTP Server NTP Client and
More informationFILE MAINTENANCE COMMANDS
Birla Institute of Technology & Science, Pilani Computer Programming (CS F111) Lab-2 ----------------------------------------------------------------------------------------------------------------------
More informationIT341 Introduction to System Administration. Project 4 - Backup Strategies with rsync and crontab
IT341 Introduction to System Administration Project 4 - Backup Strategies with rsync and crontab Backup is one of the most important things a system administrator does. It is important to decide what data
More informationGetting Started With UNIX Lab Exercises
Getting Started With UNIX Lab Exercises This is the lab exercise handout for the Getting Started with UNIX tutorial. The exercises provide hands-on experience with the topics discussed in the tutorial.
More informationXSEDE New User Training. Ritu Arora November 14, 2014
XSEDE New User Training Ritu Arora Email: rauta@tacc.utexas.edu November 14, 2014 1 Objectives Provide a brief overview of XSEDE Computational, Visualization and Storage Resources Extended Collaborative
More informationUnix/Linux Primer. Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois
Unix/Linux Primer Taras V. Pogorelov and Mike Hallock School of Chemical Sciences, University of Illinois August 25, 2017 This primer is designed to introduce basic UNIX/Linux concepts and commands. No
More informationA Hands-On Tutorial: RNA Sequencing Using High-Performance Computing
A Hands-On Tutorial: RNA Sequencing Using Computing February 11th and 12th, 2016 1st session (Thursday) Preliminaries: Linux, HPC, command line interface Using HPC: modules, queuing system Presented by:
More informationIntroduction to remote command line Linux. Research Computing Team University of Birmingham
Introduction to remote command line Linux Research Computing Team University of Birmingham Linux/UNIX/BSD/OSX/what? v All different v UNIX is the oldest, mostly now commercial only in large environments
More informationShort Read Sequencing Analysis Workshop
Short Read Sequencing Analysis Workshop Day 2 Learning the Linux Compute Environment In-class Slides Matt Hynes-Grace Manager of IT Operations, BioFrontiers Institute Review of Day 2 Videos Video 1 Introduction
More informationChange and Configuration Management Administration
CHAPTER 7 Change and Configuration Management Administration These topics provide administrative information on Change and Configuration Management: Configuring Global Settings for Configuration Management,
More informationQuick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing
Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Contents User access, logging in Linux/Unix
More informationNew User Tutorial. OSU High Performance Computing Center
New User Tutorial OSU High Performance Computing Center TABLE OF CONTENTS Logging In... 3-5 Windows... 3-4 Linux... 4 Mac... 4-5 Changing Password... 5 Using Linux Commands... 6 File Systems... 7 File
More informationPart I. Introduction to Linux
Part I Introduction to Linux 7 Chapter 1 Linux operating system Goal-of-the-Day Familiarisation with basic Linux commands and creation of data plots. 1.1 What is Linux? All astronomical data processing
More informationGetting Started. Running Utilities. Shells. Special Characters. Special Characters. Chapter 2 Unix Utilities for non-programmers
Chapter 2 Unix Utilities for non-programmers Graham Glass and King Ables, UNIX for Programmers and Users, Third Edition, Pearson Prentice Hall, 2003. Original Notes by Raj Sunderraman Converted to presentation
More informationResearch. We make it happen. Unix Basics. User Support Group help-line: personal:
Research. We make it happen. Unix Basics Presented by: Patton Fast User Support Group help-line: help@msi.umn.edu 612-626-0802 personal: pfast@msi.umn.edu 612-625-6573 Outline I. Warnings! II. III. IV.
More informationRead the relevant material in Sobell! If you want to follow along with the examples that follow, and you do, open a Linux terminal.
Warnings 1 First of all, these notes will cover only a small subset of the available commands and utilities, and will cover most of those in a shallow fashion. Read the relevant material in Sobell! If
More informationFirst of all, these notes will cover only a small subset of the available commands and utilities, and will cover most of those in a shallow fashion.
Warnings 1 First of all, these notes will cover only a small subset of the available commands and utilities, and will cover most of those in a shallow fashion. Read the relevant material in Sobell! If
More informationUsing the computational resources at the GACRC
An introduction to zcluster Georgia Advanced Computing Resource Center (GACRC) University of Georgia Dr. Landau s PHYS4601/6601 course - Spring 2017 What is GACRC? Georgia Advanced Computing Resource Center
More informationAdditional laboratory
Additional laboratory This is addicional laboratory session where you will get familiar with the working environment. Firstly, you will learn about the different servers present in the lab and how desktops
More informationFREEENGINEER.ORG. 1 of 6 11/5/15 8:31 PM. Learn UNIX in 10 minutes. Version 1.3. Preface
FREEENGINEER.ORG Learn UNIX in 10 minutes. Version 1.3 Preface This is something that I had given out to students (CAD user training) in years past. The purpose was to have on one page the basics commands
More informationIntroduction to Linux Workshop 2. The George Washington University SEAS Computing Facility
Introduction to Linux Workshop 2 The George Washington University SEAS Computing Facility Course Goals SSH and communicating with other machines Public/Private key generation,.ssh directory, and the config
More informationProject #6: Using ssh, scp and sftp with Key-Based Authentication
Project #6: Using ssh, scp and sftp with Key-Based Authentication ssh, scp and sftp Going beyond Password Protection Creating Keys Creating Keys with a Passphrase Using Key-Based Authentication in Our
More informationFtp Command Line Commands Linux Example Windows Put
Ftp Command Line Commands Linux Example Windows Put Examples of typical uses of the command ftp. This lists the commands that you can use to show the directory contents, transfer files, and delete files.
More informationIntroduction to Unix The Windows User perspective. Wes Frisby Kyle Horne Todd Johansen
Introduction to Unix The Windows User perspective Wes Frisby Kyle Horne Todd Johansen What is Unix? Portable, multi-tasking, and multi-user operating system Software development environment Hardware independent
More informationTELE301 Lab16 - The Secure Shell
TELE301 Lab16 - The Secure Shell Department of Telecommunications May 7, 2002 Contents 1 Introduction 2 2 OpenSSH 2 3 Replacing Telnet 2 4 Logging in without a password 2 5 SSH Agent 3 5.1 SSH Add..............................
More informationUnit: Making a move (using FTP)
Data Introduction to Unix and HPC (HPC for Wimps) Unit: Making a move (using FTP) Goals: Can login via Secure FTP and see home directory. Can transfer a file from local machine via FTP to home directory.
More informationGetting Started With Cpsc (Advanced) Hosted by Jarrett Spiker
Getting Started With Cpsc (Advanced) Hosted by Jarrett Spiker Advanced? - Assuming that everyone has done at least a year of CPSC already, or has a strong base knowledge. - If not, there is a Beginners
More information