HowTo: FermiGrid for MAP Users
|
|
- Junior Lawson
- 5 years ago
- Views:
Transcription
1 HowTo: FermiGrid for MAP Users Tom Roberts, Muons, Inc. March 2010 Introduction... 1 Requirements... 1 Background... 2 Glossary...2 Security and Privacy...4 Basic Properties of FermiGrid... 4 One Time Tasks... 5 Anatomy of Grid Jobs... 5 Preparation and Submission...6 Execution...6 Completion...7 Submitting Multiple Jobs using G4beamline... 7 Example Investigate a simple target in a solenoid...9 Gotchas...10 Introduction Fermilab has several Linux clusters that are combined into a grid-based system called FermiGrid [ There are thousands of CPUs in these clusters, and members of the Muon Acceleration Program can use them for MAP-related computations. This document is a synopsis of what is required to use FermiGrid, specifically for members of MAP, and a description of how to do useful work with it. Tom Roberts is the MAP liaison to FermiGrid, and questions or problems should ordinarily be directed to him. If he is unavailable, Scott Berg is an alternate. Requirements There are several requirements for using FermiGrid: 1. A Fermilab Kerberos principal plus the knowledge and tools required to use it to access Fermilab computer systems. 2. Awareness of the conditions in the Fermilab policy for using FermiGrid, and acceptance of them [ 3. A login on the FermiGrid login machines (fnpcsrv1, fnpcsrv2, fnpcsrv532, fnpcsrv564). Here s how to satisfy these requirements: 2010/02/02 TJR HowTo: FermiGrid for MAP Users 1
2 1. Fermilab Kerberos principal this is available from the Fermilab Computing Division. Participating in MAP is sufficient reason to obtain one. If you don t already have one, talk to Tom Roberts, who will guide you through the process; expect it to take several days. The MAP director will need to approve your request (should be a rubber stamp). If you already have one, you don t need a new one. 2. Policy read the link Note that incidental use is not permitted, so only MAP-related computations can use grid resources using the map group. If you don t agree with these policies, don t use FermiGrid. 3. Login on FermiGrid login machines tell Tom Roberts and he will request this for you. He will also request that you be added to the map group and to the Fermilab/map virtual organization. Background A useful resource containing tutorials on using FermiGrid is Note that FermiGrid 101 spends a lot of time on installing the tools; you can skip that as the tools are already installed on the FermiGrid login machines you will be using. The FermiGrid consists of several clusters of Linux machines. Some are dedicated to specific experiments, but the GP cluster (General Purpose) is available to us. The GP cluster currently has about 1900 job slots (cores). It often runs at >99% utilization, but still jobs are usually handled in a timely fashion. At present, MAP is set up only to use FermiGrid, not any of the external grids. This could be expanded in the future. Glossary The grid uses nomenclature all its own, and to understand both the documents and what is going on you need to become familiar with common terms. NOTE: these are my personal interpretations of these terms; they are not at all official. Google is your friend, as is Here are the terms I have found useful (in a logical order, not alphabetical): Term Grid (computing) Virtual Organization (VO) Group Grid Trust Meaning A collection of resources and virtual organizations that share a common system of grid trust, and thus share computing resources. A set of standards to facilitate this sharing has been published by the Open Science Grid organization (OSG) [ NOTE: European grids often use a different set of standards (e.g. the MICE grid). A collection of users sharing a common purpose or project, and utilizing the same grid trust. Fermilab has created a VO named Fermilab. A subset of a VO. This is done to reduce the overheads of creating a VO for a small project. The Muon Accelerator Program has a group named map this is a group within the Fermilab VO called fermilab/map. An agreement between the VO and grid providers (such as FermiGrid) that describes what resources can be used. After approval, it is embodied 2010/02/02 TJR HowTo: FermiGrid for MAP Users 2
3 Certificate Proxy Cluster (computing) Login Machine Worker node Compute Element Condor Cluster (condor) as a certificate. MAP has executed a grid trust agreement with Fermilab. How you obtain your personal grid certificate (really a proxy) is described below. A computer-readable cryptographic file that represents a user s or system s identity; often includes a list of authorized privileges. A secondary certificate that stands in for an underlying grid trust certificate for a short time (~1 week). Used so the underlying semipermanent certificate is less exposed to attack by bad guys. A collection of computing resources. Usually consists of a few login machines plus a larger number of worker nodes. Usually all run Linux. Usually all machines share common disk space. Note that FermiGrid consists of several clusters, most of which are dedicated to specific experiments. We are using the GP Cluster (General Purpose). A machine in a computing cluster that permits external logins. Usually ssh is used for access (on FermiGrid, Kerberized ssh is required). Usually shares disk storage with the worker nodes of the cluster. Used to assemble jobs, to submit them to worker nodes via some batch processing system, and then to obtain results. On FermiGrid, condor is used as the batch system (quite common). A machine in a cluster used for running jobs. Usually external logins are not allowed. Usually shares disk resources with the login nodes. An interface machine that permits the submission of Grid jobs. On FermiGrid this gateway uses condor. But you run condor_submit on a login node it talks to the CE to actually submit the job on the cluster. A batch submission system that provides detailed control of jobs. Used on FermiGrid to submit jobs, monitor them, and cancel them when necessary. It is very flexible, and therefore rather complex. Most of the complexity is hidden from you when you use the job-submit scripts I have written. A collection of related jobs all submitted at one time, sharing a single executable file (arguments for individual jobs can differ). Do not confuse this with a computing cluster. /grid/data Disk space on FermiGrid for the input and output of jobs. Currently 24 Terabytes 59% used. Verbally called grid data. Our quota is currently 0.4 Terabytes (can be increased). Not backed up, so be sure to copy important files elsewhere. /grid/app Disk space on FermiGrid for applications. Currently 0.3 Terabytes 73% used. Verbally called grid app. Mounted read-only on worker nodes; read-write on login nodes. Our quota is currently 0.08 Terabytes (can be increased). Backed up daily to tape. OSG_DATA OSG_APP OSG_WN_TMP An environment variable set to /grid/data on FermiGrid. Called $DATA in some older documentation. An environment variable set to /grid/app on FermiGrid. Called $APP in some older documentation. An environment variable set to local worker-node disk space. Called 2010/02/02 TJR HowTo: FermiGrid for MAP Users 3
4 HOME AFS WN_TMP in some older documentation. The usual environment variable set to the home directory of your login. Valid on login machines, but not on worker nodes. This is local to FermiGrid and is different from your AFS home on other Fermilab machines. The Andrew File system, which is a worldwide filesystem used at Fermilab for home directories and other storage on most other Fermilab machines. Fermilab requires a Kerberos ticket to access Fermilab data. Available on login nodes but not on worker nodes. Security and Privacy On the grid, there is strong security to prevent unauthorized users from accessing resources (implemented cryptographically using certificates). Once authorized and admitted to a grid system, however, there is almost no security, and certainly no expectation of privacy. So, for instance, on FermiGrid you must have a valid Kerberos ticket and use ssh to access the login machines, and you must have a valid grid certificate to submit jobs. But on these machines everyone is a member of the group fnalgrid, and virtually all files and directories are group readable and writable. This is so because all jobs on worker nodes run as user fnalgrid and group fnalgrid. Consequently, any user on a login machine can read and write essentially anybody s files. This obviously requires some care and discipline on the part of all users. Basic Properties of FermiGrid The GP cluster used by MAP currently has about 1900 job slots (cores), and often runs with >99% occupancy. Some systems implementing worker nodes consist of 8 cores, 24 Gigabytes of RAM, and a local disk drive with >200 Gigabytes available; others have 4 nodes, 4 Gigabytes of RAM, and ~100 Gigabytes of disk available. So individual jobs should not expect more than 1 Gigabyte of RAM or about 10 Gigabytes of disk (with advanced condor usage you can require nodes with more RAM or disk space). If your job has multiple threads, they should not total more than ~101% of a single CPU; most grid jobs are single-threaded. Neither G4beamline nor ICOOL are likely to approach these limits on individual jobs. Remarkably, there is no CPU time limit on a job; instead, a reaper process runs every Sunday, killing jobs more than a week old (i.e. jobs it has already seen once). So jobs that start on Monday can get almost 2 weeks of runtime. There are disk quotas, and at present MAP has a quota of 400 Gigabytes on /grid/data and 80 Gigabytes on /grid/app. It this proves insufficient, we can obtain more space. There is a limit on the number of simultaneous jobs, which varies for each VO. At present MAP can have 100 simultaneous jobs. If this proves insufficient, we can get it increased. Because of the job submission and startup overheads, it is undesirable for jobs to run less than about 10 minutes. Because of both common sense and the reaper, it is undesirable for jobs to run more than about 48 hours. Generally you should make a trial run of your job on a login or worker 2010/02/02 TJR HowTo: FermiGrid for MAP Users 4
5 node before submitting a large number of jobs this is a check of its sanity and provides an estimate of its runtime which you can use to set the # events per job. The login nodes are for preparing, submitting, and monitoring jobs only. You should not attempt to perform large computations on them. But building programs and running modest analysis programs are OK. The disks used on FermiGrid are implemented on a BlueArc RAID system that is accessible by all login and worker nodes. While it is a high-capacity and high-performance system, it cannot possibly handle hundreds of jobs accessing the same files simultaneously (hundreds of jobs simultaneously accessing different files is generally OK as the different files are located on different physical disk drives). So to avoid a focused overload, it is necessary to have each job copy input data onto node-local disk space, run using node-local disk space, and copy output data back out from node-local space. Moreover, in case many jobs start nearly simultaneously, the copies should be protected by a throttling mechanism that limits the number of simultaneous copies for the entire job series. It is the responsibility of all users to avoid such overloads; the scripts I have written do so. One-Time Tasks Once you have met all the requirements, you must set up your environment to use FermiGrid. I ll describe this for bash users, but other shells can be used with similar commands. The primary login machine is fnpcsrv1, but it shares home directories and user names among all FermiGrid login machines. You will want to set up your customary Linux environment on it, possibly by copying.bash_profile and.bashrc from some other machine. I ll assume that initially you will just use the job submission scripts that I have written, leaving creating new types of jobs until after you have gained experience with FermiGrid. So you should do the following: (on your local machine) kinit ssh fnpcsrv1.fnal.gov (now on fnpcsrv1... login banner, etc.) mkdir /grid/data/$user # edit.bash_profile to put /grid/app/$user into PATH # possibly put cd /grid/data/$user into.bash_profile Then log out and login again. Since your HOME directory is not visible to worker nodes, I have found it convenient to work exclusively in /grid/data/$user essentially as if it were my HOME. Using that directory will distinguish your files from everybody else s; this is just good etiquette on a shared filesystem. Beware: it is not backed up, and is subject to being seen, and even accidentally overwritten or deleted by other users. Anatomy of Grid Jobs There are three basic aspects to grid jobs: preparation and submission, execution, and completion. The first and last are usually performed manually on a login node, while the 2010/02/02 TJR HowTo: FermiGrid for MAP Users 5
6 execution occurs on worker nodes, usually many in parallel. In some cases the combination of output files is large enough to justify using worker nodes. Preparation and Submission Stage Description preparation Performed manually. You construct a directory in /grid/data/$user for this specific set of jobs. Generally you must include all input files the jobs will use, and you usually create a compressed tarball containing them (to minimize disk I/O overheads). The job-submit scripts I write permit you to symbolically-link in files and directories; the script follows the links and includes all files and directories in the tarball. submission 1. To submit jobs to the grid, you need a proxy certificate. This is described in the documentation, but it s easiest to just use my script proxy it converts your Kerberos certificate to a grid certificate valid for the remaining renewal time of the former. 2. Condor requires the creation of a submit file, which tells condor how to submit each job. The job-submit scripts I write do this internally, and then execute the condor_submit command to submit them. Execution The execution of a single job on a worker node generally consists of three stages: setup, run, finalize. Stage setup run Description Sets up the input files in $OSG_WN_TMP. Generally this consists of: 1. Obtain a copy ticket 2. Copy files to $OWG_WN_TMP 3. Release the copy ticket 4. Un-tar the copied files if necessary The use of a copy ticket is designed to avoid a focused overload on /grid/data when a large number of similar jobs execute simultaneously. This is good etiquette on a shared filesystem. Executes the program, reading and writing $OSG_WN_TMP. Note that $OSG_WN_TMP is on a local drive that is not subject to overloads from multiple jobs. finalize Copies output files from $OSG_WN_TMP to /grid/data. Generally this consists of: 1. Obtain a copy ticket 2. Copy files to /grid/data/$user/ Release the copy ticket The use of a copy ticket is designed to avoid a focused overload on /grid/data when a large number of similar jobs execute simultaneously. This is good etiquette on a shared filesystem. 2010/02/02 TJR HowTo: FermiGrid for MAP Users 6
7 Completion In some cases there may be no need for this. Stage Description completion Combine multiple output files from multiple jobs into a single file for analysis. Submitting Multiple Jobs using G4beamline A common task in MAP is to scan various parameter values, looking for an optimal set of parameters for a given configuration. The submit_g4bl script is specifically written to address this efficiently and easily. It has these features: Command-line syntax similar to g4bl; the parameter names and values are passed to g4bl, so the parameter names you use depend on the parameters used in the g4bl input file Simple command-line syntax to loop over values of a parameter Simple command-line syntax to run multiple identical jobs with different events Multiple parameter loops can be nested, including the event loop All parameter values are put into the names of the output files, including event #s For setup, creates a tarball of the current directory, following all symbolic links Adheres to the canons of good etiquette (using my throttle script) This script constructs a.submit file for condor, and a.run shell script that is executed by each job. The arguments to.run are: name, Job-id, first, last; these are given in the.submit file to condor. At execution time, the actual parameters to g4bl are parsed from the name. Here is the help text from submit_g4bl help : submit_g4bl - submit multiple g4bl jobs to FermiGrid USAGE: submit_g4bl input.g4bl a=b c=1,10,2 d=1:5:17 ev=1,10000,1000 This script will tar up the contents of the current directory, FOLLOWING symbolic links; the jobs it submits start out by un-taring into OSG_WN_TMP and using files from there; the jobs finish by copying g4beamline.root to the submit directory, using the name of the job. The "throttle" command is used to avoid overloads on /grid/data. This script creates a "Jobs.log" file containing the status of the jobs it submits. All output files are put into the Output directory. The current files in Output are moved to OldOutput at the start of this script. NOTE: arguments cannot contain spaces or semicolons, and they cannot contain commas or colons unless they are defining a loop or list. Job names are generated by the script, using the parameter names=values from the command line, separated by commas. The job name also defines the parameters to g4bl on its command line. The above command line generates 150 jobs with names: a=b,c=1,d=1,ev= /02/02 TJR HowTo: FermiGrid for MAP Users 7
8 a=b,c=1,c=1,ev= more with varying events a=b,c=1,d=5,ev= more with varying events a=b,c=1,d=17,ev= more with varying events a=b,c=3,d=1,ev= etc. The output files will be named for the job (with.root appended). The "combine" script can be used to combine the 150 output files into 15, putting them into the submit directory. Loops are defined by an argument like c=1,10,2 -- this generates a loop in the usual DO-loop fashion from 1 to 10 (inclusive) incrementing by 2; values are floats, not ints. The increment cannot be omitted. Lists are defined by an argument like d=1:5:17 -- this generates a loop that iterates the value of d over the elements in the list; values are strings, and can thus be any type (int, float, or string). If multiple loops or lists are given, the first is outermost and the last is innermost. Note also that ev=a,b,c is special and if present gives the innermost loop on event numbers (see below). Events are handled specially with an argument ev=1,10000, this generates the innermost loop over event numbers, defining the variables first and last to g4bl, but appending ev= to the first job name, ev= to the second job name, etc. Regardless of where ev=a,b,c appears in the command line, it always generates the innermost loop and comes last in the job name. (".." is the Ada operator for an inclusive interval.) The input.g4bl file MUST follow these conventions: * histofile is not set (so the output file is g4beamline.root) * output format is root (the default) * input.g4bl and any auxiliary files it uses should be referenced from the current directory (e.g. magnet field maps, window profiles, etc.) -- they should be symbolically linked to the current directory before submit_g4bl is run, so they will be transferred to the worker node during job startup. Symbolic links to subdirectories are fine. * the only exception to the previous point is a large beam file, which should be referenced via an absolute pathname starting /grid/data/... * the beam command should include "firstevent=$first lastevent=$last", because the ev=... argument to submit_g4bl will be converted to "first=1 last=1000" (etc.) on the g4bl command-line. * The parameters used on the submit_g4bl command line (except ev=) should be used in input.g4bl to control the simulation This script throttles job startup and completion to avoid a focused overload on the disk (/grid/data). 2010/02/02 TJR HowTo: FermiGrid for MAP Users 8
9 Example Investigate a simple target in a solenoid An example of using FermiGrid is my investigation of a simple target in a solenoid magnetic field. This is primarily a demonstration tool, and not really a design for a real-world target station. G4beamline input file: target.g4bl * target.g4bl TJR * * 8 GeV proton beam into a Tungsten target in a uniform Bz field. param -unset R=6 L=300 Bz=20 Angle=0 physics QGSP_BERT beam gaussian particle=proton firstevent=$first lastevent=$last beamz=0.0 \ sigmax=1.5 sigmay=1.5 sigmaxp=0.005 sigmayp=0.005 rotation=x$angle \ meanmomentum= sigmap=8 meant=0.0 sigmat=3 trackcuts kill=nu_e,anti_nu_e,nu_mu,anti_nu_mu,e+,e-,neutron,gamma fieldexpr Field radius=2000 length= Bz=$Bz place Field z=0 tubs Pipe innerradius=2000 outerradius=2001 length=80000 kill=1 place Pipe z=0 front=1 cylinder Target outerradius=$r length=$l material=w color=.6,.6,.6 place Target z=$l/2 rotation=x$angle y=$l/2*tan(-$angle* /180) particlefilter Filter radius=2000 length=0.1 keep=pi+,pi-,mu+,mu- color=1,0,0 place Filter zntuple zloop=1000,60000,1000 place Filter z=60001 Note that target.g4bl uses these parameters: R radius of the tungsten target L length of the tungsten target Bz Z component of the uniform solenoid field Angle angle (degrees) of the target relative to the solenoid axis The submit_g4bl script is used to scan over R, Bz, and Angle simultaneously (a loop over L as well was considered too many jobs; a previous scan gave 300 as its optimal value). Before submitting a large number of jobs to FermiGrid, I ran a short test locally on the login node, to determine how many events to run in each job: g4bl target.g4bl R=6 L=300 Angle=2 Bz=20 first=1 last=1000 From the runtime of this command, I determined that 20,000 events should take about an hour. Command Line submit_g4bl target.g4bl R=3:6:10:15 L=300 Angle=0:1:2:3:5 Bz=2:5:10:20 \ ev=1,20000, /02/02 TJR HowTo: FermiGrid for MAP Users 9
10 This command submits a total of 80 jobs. They scan 4 values of R, 5 values of Angle, and 4 values of Bz. Output Files: lf Output R=10,L=300,Angle=0,Bz=10,ev= log R=10,L=300,Angle=0,Bz=10,ev= out R=10,L=300,Angle=0,Bz=10,ev= root R=10,L=300,Angle=0,Bz=2,ev= log R=10,L=300,Angle=0,Bz=2,ev= out R=10,L=300,Angle=0,Bz=2,ev= root... total of 240 files For this task, sufficient events were generated in each job, so there s no need to combine multiple output files. To analyze the output files, I wrote a Root macro that obtains the values of parameters from the file names, and writes a summary file consisting of one row per file, with columns containing the parameters and interesting values from each run (# muons, # pions, transverse sigmas, etc.), as a function of Z (1 to 60 meters). This is too long to include here; ask for details. Gotchas There are a number of rather unwelcome surprises when using FermiGrid. Here are the ones I have found: A shell script that is to be run by condor must begin #!/bin/bash. Usually this is optional, but for condor it is mandatory. $OSG_WN_TMP is not unique to your job, so if you assume that it is and two of your jobs land on different cores of the same worker node, there probably will be trouble. My scripts work in $OSG_WN_TMP/job-name. Your proxy certificate may run out while your jobs are running. When this happens, they are automatically put on hold status in condor. Fortunately, you can fix this simply by obtaining a new proxy certificate, and issuing the condor_release command. But only you can do this. 2010/02/02 TJR HowTo: FermiGrid for MAP Users 10
A Guide to Condor. Joe Antognini. October 25, Condor is on Our Network What is an Our Network?
A Guide to Condor Joe Antognini October 25, 2013 1 Condor is on Our Network What is an Our Network? The computers in the OSU astronomy department are all networked together. In fact, they re networked
More informationOSG Lessons Learned and Best Practices. Steven Timm, Fermilab OSG Consortium August 21, 2006 Site and Fabric Parallel Session
OSG Lessons Learned and Best Practices Steven Timm, Fermilab OSG Consortium August 21, 2006 Site and Fabric Parallel Session Introduction Ziggy wants his supper at 5:30 PM Users submit most jobs at 4:59
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationCS Fundamentals of Programming II Fall Very Basic UNIX
CS 215 - Fundamentals of Programming II Fall 2012 - Very Basic UNIX This handout very briefly describes how to use Unix and how to use the Linux server and client machines in the CS (Project) Lab (KC-265)
More informationIntroduction to Linux and Supercomputers
Introduction to Linux and Supercomputers Doug Crabill Senior Academic IT Specialist Department of Statistics Purdue University dgc@purdue.edu What you will learn How to log into a Linux Supercomputer Basics
More informationCernVM-FS beyond LHC computing
CernVM-FS beyond LHC computing C Condurache, I Collier STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot, OX11 0QX, UK E-mail: catalin.condurache@stfc.ac.uk Abstract. In the last three years
More information8 MANAGING SHARED FOLDERS & DATA
MANAGING SHARED FOLDERS & DATA STORAGE.1 Introduction to Windows XP File Structure.1.1 File.1.2 Folder.1.3 Drives.2 Windows XP files and folders Sharing.2.1 Simple File Sharing.2.2 Levels of access to
More informationParallel Computing with Matlab and R
Parallel Computing with Matlab and R scsc@duke.edu https://wiki.duke.edu/display/scsc Tom Milledge tm103@duke.edu Overview Running Matlab and R interactively and in batch mode Introduction to Parallel
More informationIntroduction to High-Performance Computing (HPC)
Introduction to High-Performance Computing (HPC) Computer components CPU : Central Processing Unit cores : individual processing units within a CPU Storage : Disk drives HDD : Hard Disk Drive SSD : Solid
More informationDiskSavvy Disk Space Analyzer. DiskSavvy DISK SPACE ANALYZER. User Manual. Version Dec Flexense Ltd.
DiskSavvy DISK SPACE ANALYZER User Manual Version 10.3 Dec 2017 www.disksavvy.com info@flexense.com 1 1 Product Overview...3 2 Product Versions...7 3 Using Desktop Versions...8 3.1 Product Installation
More informationDupScout DUPLICATE FILES FINDER
DupScout DUPLICATE FILES FINDER User Manual Version 10.3 Dec 2017 www.dupscout.com info@flexense.com 1 1 Product Overview...3 2 DupScout Product Versions...7 3 Using Desktop Product Versions...8 3.1 Product
More informationCreating Workflows. Viewing the Task Library. Creating a Workflow. This chapter contains the following sections:
This chapter contains the following sections: Viewing the Task Library, page 1 Creating a Workflow, page 1 Example: Creating a Workflow, page 13 Resolving Workflow Validation Issues, page 16 Changing Flow
More informationEducAide Software. Instructions for producing an Acces-compatible database module (program version 3.41 or higher)
EducAide Software Instructions for producing an Acces-compatible database module (program version 3.41 or higher) Requirements In order to produce your own database module, you need: a license for Acces,
More informationInstalling and Configuring VMware Identity Manager Connector (Windows) OCT 2018 VMware Identity Manager VMware Identity Manager 3.
Installing and Configuring VMware Identity Manager Connector 2018.8.1.0 (Windows) OCT 2018 VMware Identity Manager VMware Identity Manager 3.3 You can find the most up-to-date technical documentation on
More informationLaserfiche Rio 10.3: Deployment Guide. White Paper
Laserfiche Rio 10.3: Deployment Guide White Paper January 2018 Table of Contents How Laserfiche Licensing Works... 4 Types of Licenses... 4 Named User Licenses... 4 WebLink Public Portal Licenses... 6
More informationNBIC TechTrack PBS Tutorial
NBIC TechTrack PBS Tutorial by Marcel Kempenaar, NBIC Bioinformatics Research Support group, University Medical Center Groningen Visit our webpage at: http://www.nbic.nl/support/brs 1 NBIC PBS Tutorial
More informationOperating System Interaction via bash
Operating System Interaction via bash bash, or the Bourne-Again Shell, is a popular operating system shell that is used by many platforms bash uses the command line interaction style generally accepted
More informationA Hands-On Tutorial: RNA Sequencing Using High-Performance Computing
A Hands-On Tutorial: RNA Sequencing Using Computing February 11th and 12th, 2016 1st session (Thursday) Preliminaries: Linux, HPC, command line interface Using HPC: modules, queuing system Presented by:
More informationNetwork Install Guide
131 Hartwell Avenue Lexington, MA 02421 (800) 225-8697 www.wrightsoft.com Network Install Guide Overview and Installation Information for Wrightsoft Right-Suite Universal Network Version 1 Introduction
More informationSecurity Correlation Server Backup and Recovery Guide
CorreLog Security Correlation Server Backup and Recovery Guide This guide provides information to assist administrators and operators with backing up the configuration and archive data of the CorreLog
More informationEurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio. (INFN Padova) on behalf of Eurogrid support group
Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio (INFN Padova) on behalf of Eurogrid support group CDF computing model CDF computing model is based on Central farm
More informationUnix Tutorial Haverford Astronomy 2014/2015
Unix Tutorial Haverford Astronomy 2014/2015 Overview of Haverford astronomy computing resources This tutorial is intended for use on computers running the Linux operating system, including those in the
More informationT H E I N T E R A C T I V E S H E L L
3 T H E I N T E R A C T I V E S H E L L The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform. Ada Lovelace, October 1842 Before
More informationUser interface for a computational cluster: resource description approach
User interface for a computational cluster: resource description approach A. Bogdanov 1,a, V. Gaiduchok 1,2, N. Ahmed 2, P. Ivanov 2, M. Kamande 2, A. Cubahiro 2 1 Saint Petersburg State University, 7/9,
More informationNBIC TechTrack PBS Tutorial. by Marcel Kempenaar, NBIC Bioinformatics Research Support group, University Medical Center Groningen
NBIC TechTrack PBS Tutorial by Marcel Kempenaar, NBIC Bioinformatics Research Support group, University Medical Center Groningen 1 NBIC PBS Tutorial This part is an introduction to clusters and the PBS
More informationCS 215 Fundamentals of Programming II Spring 2019 Very Basic UNIX
CS 215 Fundamentals of Programming II Spring 2019 Very Basic UNIX This handout very briefly describes how to use Unix and how to use the Linux server and client machines in the EECS labs that dual boot
More informationUnit 8: Working with Actions
Unit 8: Working with Actions Questions Covered What are actions? How are actions triggered? Where can we access actions to create or edit them? How do we automate the sending of email notifications? How
More informationKerberos & HPC Batch systems. Matthieu Hautreux (CEA/DAM/DIF)
Kerberos & HPC Batch systems Matthieu Hautreux (CEA/DAM/DIF) matthieu.hautreux@cea.fr Outline Kerberos authentication HPC site environment Kerberos & HPC systems AUKS From HPC site to HPC Grid environment
More informationGetting started with the CEES Grid
Getting started with the CEES Grid October, 2013 CEES HPC Manager: Dennis Michael, dennis@stanford.edu, 723-2014, Mitchell Building room 415. Please see our web site at http://cees.stanford.edu. Account
More informationLinux Essentials. Smith, Roderick W. Table of Contents ISBN-13: Introduction xvii. Chapter 1 Selecting an Operating System 1
Linux Essentials Smith, Roderick W. ISBN-13: 9781118106792 Table of Contents Introduction xvii Chapter 1 Selecting an Operating System 1 What Is an OS? 1 What Is a Kernel? 1 What Else Identifies an OS?
More informationWho am I? I m a python developer who has been working on OpenStack since I currently work for Aptira, who do OpenStack, SDN, and orchestration
Who am I? I m a python developer who has been working on OpenStack since 2011. I currently work for Aptira, who do OpenStack, SDN, and orchestration consulting. I m here today to help you learn from my
More informationCSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209
CSC209 Software Tools and Systems Programming https://mcs.utm.utoronto.ca/~209 What is this Course About? Software Tools Using them Building them Systems Programming Quirks of C The file system System
More informationNo More Passwords (with SSH)
No More Passwords (with SSH) Ted Dustman March 30, 2009 Contents 1 Introduction 1 1.1 Local or Remote?................................. 1 1.2 SSH Command Set................................ 1 2 Authentication
More informationVirtual CD TS 1 Introduction... 3
Table of Contents Table of Contents Virtual CD TS 1 Introduction... 3 Document Conventions...... 4 What Virtual CD TS Can Do for You...... 5 New Features in Version 10...... 6 Virtual CD TS Licensing......
More informationTable of Contents. Table Of Contents. Access to parameters (lesson 2)) 26 Surprised? 26 Key Points for Lesson 1: 26 Quiz 26
Preface 9 Why you should buy this book 9 What is parametric programming? 10 A word about CAM systems 10 Scope 10 Versions of Custom Macro 10 Machine types 10 Prerequisites 11 Optional status 11 Lessons
More informationDesign Choices 2 / 29
File Systems One of the most visible pieces of the OS Contributes significantly to usability (or the lack thereof) 1 / 29 Design Choices 2 / 29 Files and File Systems What s a file? You all know what a
More informationGraham vs legacy systems
New User Seminar Graham vs legacy systems This webinar only covers topics pertaining to graham. For the introduction to our legacy systems (Orca etc.), please check the following recorded webinar: SHARCNet
More informationClient Installation and User's Guide
IBM Tivoli Storage Manager FastBack for Workstations Version 7.1 Client Installation and User's Guide SC27-2809-03 IBM Tivoli Storage Manager FastBack for Workstations Version 7.1 Client Installation
More informationUnit 2 : Computer and Operating System Structure
Unit 2 : Computer and Operating System Structure Lesson 1 : Interrupts and I/O Structure 1.1. Learning Objectives On completion of this lesson you will know : what interrupt is the causes of occurring
More informationDiskBoss DATA MANAGEMENT
DiskBoss DATA MANAGEMENT File Delete and Data Wiping Version 9.3 May 2018 www.diskboss.com info@flexense.com 1 1 Product Overview DiskBoss is an automated, policy-based data management solution allowing
More informationWhite Paper Arcserve Backup Greatly Reduces the Cost of Backup Using All-Flash Arrays with the Latest LTO Ultrium Technology
White Paper Arcserve Backup Greatly Reduces the Cost of Backup Using All-Flash Arrays with the Latest LTO Ultrium Technology Unlimited Backup Capacity and Number of Generations Adoption of all-flash arrays
More informationLecture 3: Web Servers / PHP and Apache. CS 383 Web Development II Monday, January 29, 2018
Lecture 3: Web Servers / PHP and Apache CS 383 Web Development II Monday, January 29, 2018 Server Configuration One of the most common configurations of servers meant for web development is called a LAMP
More informationCSC209. Software Tools and Systems Programming. https://mcs.utm.utoronto.ca/~209
CSC209 Software Tools and Systems Programming https://mcs.utm.utoronto.ca/~209 What is this Course About? Software Tools Using them Building them Systems Programming Quirks of C The file system System
More informationIntroduction to Linux and Cluster Computing Environments for Bioinformatics
Introduction to Linux and Cluster Computing Environments for Bioinformatics Doug Crabill Senior Academic IT Specialist Department of Statistics Purdue University dgc@purdue.edu What you will learn Linux
More information24 Writing Your First Script
In the preceding chapters, we have assembled an arsenal of command line tools. While these tools can solve many kinds of computing problems, we are still limited to manually using them one by one on the
More informationIntroduction to BioHPC
Introduction to BioHPC New User Training [web] [email] portal.biohpc.swmed.edu biohpc-help@utsouthwestern.edu 1 Updated for 2015-06-03 Overview Today we re going to cover: What is BioHPC? How do I access
More informationComputer Science 2500 Computer Organization Rensselaer Polytechnic Institute Spring Topic Notes: C and Unix Overview
Computer Science 2500 Computer Organization Rensselaer Polytechnic Institute Spring 2009 Topic Notes: C and Unix Overview This course is about computer organization, but since most of our programming is
More informationPractical 5. Linux Commands: Working with Files
Practical 5 Linux Commands: Working with Files 1. Ps The ps command on linux is one of the most basic commands for viewing the processes running on the system. It provides a snapshot of the current processes
More informationShell / Python Tutorial. CS279 Autumn 2017 Rishi Bedi
Shell / Python Tutorial CS279 Autumn 2017 Rishi Bedi Shell (== console, == terminal, == command prompt) You might also hear it called bash, which is the most widely used shell program macos Windows 10+
More informationDo not start the test until instructed to do so!
Instructions: Print your name in the space provided below. This examination is closed book and closed notes, aside from the permitted one-page formula sheet. No calculators or other electronic devices
More informationXcalar Installation Guide
Xcalar Installation Guide Publication date: 2018-03-16 www.xcalar.com Copyright 2018 Xcalar, Inc. All rights reserved. Table of Contents Xcalar installation overview 5 Audience 5 Overview of the Xcalar
More informationTeamcenter Enterprise: Rule System
Teamcenter Enterprise: Rule System Volume ITI-MTI-E Written by Erich Brauchli March 2, 2008 Version 2.10 Reflecting Teamcenter Enterprise 2005 Email: erich@brauchli.info Internet: http://www.brauchli.info
More informationUNIT V. Dr.T.Logeswari. Unix Shell Programming - Forouzan
UNIT V UNIX SYSTEM COMMUNICATION Dr.T.Logeswari 1 Electronic mail or email is easiest way of communication on unix. Fast and cheap Used to exchange graphics, sound and video files 2 3 Elements of a communication
More information${Unix_Tools} exercises and solution notes
${Unix_Tools exercises and solution notes Markus Kuhn Computer Science Tripos Part IB The shell Exercise : Write a shell command line that appends :/usr/xr6/man to the end of the environment variable $MANPATH.
More informationEUSurvey Installation Guide
EUSurvey Installation Guide Guide to a successful installation of EUSurvey May 20 th, 2015 Version 1.2 (version family) 1 Content 1. Overview... 3 2. Prerequisites... 3 Tools... 4 Java SDK... 4 MySQL Database
More informationLinux Command Line Primer. By: Scott Marshall
Linux Command Line Primer By: Scott Marshall Draft: 10/21/2007 Table of Contents Topic Page(s) Preface 1 General Filesystem Background Information 2 General Filesystem Commands 2 Working with Files and
More information1Z Oracle Linux Fundamentals (Oracle Partner Network) Exam Summary Syllabus Questions
1Z0-409 Oracle Linux Fundamentals (Oracle Partner Network) Exam Summary Syllabus Questions Table of Contents Introduction to 1Z0-409 Exam on Oracle Linux Fundamentals (Oracle Partner Network)... 2 Oracle
More informationInstalling Design Room ONE
Installing Design Room ONE Design Room ONE consists of two components: 1. The Design Room ONE web server This is a Node JS server which uses a Mongo database. 2. The Design Room ONE Integration plugin
More informationCreating Workflows. Viewing the Task Library. Creating a Workflow. This chapter contains the following sections:
This chapter contains the following sections: Viewing the Task Library, page 1 Creating a Workflow, page 1 Example: Creating a Workflow, page 12 Resolving Workflow Validation Issues, page 15 Changing Flow
More informationMy Favorite bash Tips and Tricks
1 of 6 6/18/2006 7:44 PM My Favorite bash Tips and Tricks Prentice Bisbal Abstract Save a lot of typing with these handy bash features you won't find in an old-fashioned UNIX shell. bash, or the Bourne
More informationCallManager Server: Use PsList to Troubleshoot a Memory Leak Problem
CallManager Server: Use PsList to Troubleshoot a Memory Leak Problem Document ID: 66967 Contents Introduction Prerequisites Requirements Components Used Conventions Background Usage Setup PsList on the
More informationSetting up PostgreSQL
Setting up PostgreSQL 1 Introduction to PostgreSQL PostgreSQL is an object-relational database management system based on POSTGRES, which was developed at the University of California at Berkeley. PostgreSQL
More informationWeb Services for Relational Data Access
Web Services for Relational Data Access Sal Valente CS 6750 Fall 2010 Abstract I describe services which make it easy for users of a grid system to share data from an RDBMS. The producer runs a web services
More informationDataMan. version 6.5.4
DataMan version 6.5.4 Contents DataMan User Guide 1 Introduction 1 DataMan 1 Technical Specifications 1 Hardware Requirements 1 Software Requirements 2 Ports 2 DataMan Installation 2 Component Installation
More informationName Department/Research Area Have you used the Linux command line?
Please log in with HawkID (IOWA domain) Macs are available at stations as marked To switch between the Windows and the Mac systems, press scroll lock twice 9/27/2018 1 Ben Rogers ITS-Research Services
More informationHTCondor Essentials. Index
HTCondor Essentials 31.10.2017 Index Login How to submit a job in the HTCondor pool Why the -name option? Submitting a job Checking status of submitted jobs Getting id and other info about a job
More informationWhen talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used:
Linux Tutorial How to read the examples When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: $ application file.txt
More informationShell Programming Overview
Overview Shell programming is a way of taking several command line instructions that you would use in a Unix command prompt and incorporating them into one program. There are many versions of Unix. Some
More informationWhite Paper NetVault Backup Greatly Reduces the Cost of Backup Using All-Flash Arrays with the Latest LTO Ultrium Technology
White Paper NetVault Backup Greatly Reduces the Cost of Backup Using All-Flash Arrays with the Latest LTO Ultrium Technology Unlimited Backup Capacity and Number of Generations Adoption of all-flash arrays
More informationServer guides for the GIRAF project
Server guides for the GIRAF project A joint guide produced by SW611 & SW616 Aalborg University Contents 1 Accessing the GIRAF servers (SW611)........................... 3 2 Using Docker (SW611)..........................................
More informationVersion 11. NOVASTOR CORPORATION NovaBACKUP
NOVASTOR CORPORATION NovaBACKUP Version 11 2009 NovaStor, all rights reserved. All trademarks are the property of their respective owners. Features and specifications are subject to change without notice.
More informationOPERATING SYSTEM. Functions of Operating System:
OPERATING SYSTEM Introduction: An operating system (commonly abbreviated to either OS or O/S) is an interface between hardware and user. OS is responsible for the management and coordination of activities
More informationUsing LINUX a BCMB/CHEM 8190 Tutorial Updated (1/17/12)
Using LINUX a BCMB/CHEM 8190 Tutorial Updated (1/17/12) Objective: Learn some basic aspects of the UNIX operating system and how to use it. What is UNIX? UNIX is the operating system used by most computers
More informationHTC Brief Instructions
HTC Brief Instructions Version 18.08.2018 University of Paderborn Paderborn Center for Parallel Computing Warburger Str. 100, D-33098 Paderborn http://pc2.uni-paderborn.de/ 2 HTC BRIEF INSTRUCTIONS Table
More informationLabels and Information Flow
Labels and Information Flow Robert Soulé March 21, 2007 Problem Motivation and History The military cares about information flow Everyone can read Unclassified Few can read Top Secret Problem Motivation
More informationOutline. ASP 2012 Grid School
Distributed Storage Rob Quick Indiana University Slides courtesy of Derek Weitzel University of Nebraska Lincoln Outline Storage Patterns in Grid Applications Storage
More informationXP: Backup Your Important Files for Safety
XP: Backup Your Important Files for Safety X 380 / 1 Protect Your Personal Files Against Accidental Loss with XP s Backup Wizard Your computer contains a great many important files, but when it comes to
More informationLinux+ Guide to Linux Certification, Third Edition. Chapter 2 Linux Installation and Usage
Linux+ Guide to Linux Certification, Third Edition Chapter 2 Linux Installation and Usage Objectives Install Red Hat Fedora Linux using good practices Outline the structure of the Linux interface Enter
More informationConfiguring the Oracle Network Environment. Copyright 2009, Oracle. All rights reserved.
Configuring the Oracle Network Environment Objectives After completing this lesson, you should be able to: Use Enterprise Manager to: Create additional listeners Create Oracle Net Service aliases Configure
More informationLab 1 Introduction to UNIX and C
Name: Lab 1 Introduction to UNIX and C This first lab is meant to be an introduction to computer environments we will be using this term. You must have a Pitt username to complete this lab. NOTE: Text
More informationAn overview of batch processing. 1-June-2017
An overview of batch processing 1-June-2017 One-on-one Your computer Not to be men?oned in this talk Your computer (mul?ple cores) (mul?ple threads) One thread One thread One thread One thread One thread
More informationFileCatalyst HotFolder Quickstart
FileCatalyst HotFolder Quickstart Contents 1 Installation... 2 1.1 Verify Java Version... 2 1.2 Perform Installation... 3 1.2.1 Windows... 3 1.2.2 Mac OSX... 3 1.2.3 Linux, Solaris, *nix... 3 1.3 Enable
More informationSetting up the Seagate D4 NAS with Hard Disk Sentinel Pro. By Gary Ryan. Adapted from:
Setting up the Seagate D4 NAS with Hard Disk Sentinel Pro By Gary Ryan Adapted from: https://www.hdsentinel.com/how_to_monitor_network_attached_storage_nas_status.php How to: monitor Network Attached Storage
More informationRead the relevant material in Sobell! If you want to follow along with the examples that follow, and you do, open a Linux terminal.
Warnings 1 First of all, these notes will cover only a small subset of the available commands and utilities, and will cover most of those in a shallow fashion. Read the relevant material in Sobell! If
More informationA Big Step. Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers
A Big Step Shell Scripts, I/O Redirection, Ownership and Permission Concepts, and Binary Numbers Copyright 2006 2009 Stewart Weiss What a shell really does Here is the scoop on shells. A shell is a program
More informationComputer Principles and Components 1
Computer Principles and Components 1 Course Map This module provides an overview of the hardware and software environment being used throughout the course. Introduction Computer Principles and Components
More informationAnaplan Connector Guide Document Version 2.1 (updated 14-MAR-2017) Document Version 2.1
Document Version 2.1 (updated 14-MAR-2017) Document Version 2.1 Version Control Version Number Date Changes 2.1 MAR 2017 New Template applied Anaplan 2017 i Document Version 2.1 1 Introduction... 1 1.1.
More informationHitchhiker s Guide to VLSI Design with Cadence & Synopsys
Hitchhiker s Guide to VLSI Design with Cadence & Synopsys David Money Harris 17 January 2009 The VLSI design tools at Harvey Mudd College are hosted on a Linux server named chips. This document introduces
More informationGMI-Cmd.exe Reference Manual GMI Command Utility General Management Interface Foundation
GMI-Cmd.exe Reference Manual GMI Command Utility General Management Interface Foundation http://www.gmi-foundation.org Program Description The "GMI-Cmd.exe" program is a standard part of the GMI program
More informationPurpose. Target Audience. Prerequisites. What Is An Event Handler? Nagios XI. Introduction to Event Handlers
Purpose This document describes how to use event handlers in to take predefined actions when the hosts or services you are monitoring change state. Event handlers are used to automate processes taken when
More informationFILE SYSTEMS. CS124 Operating Systems Winter , Lecture 23
FILE SYSTEMS CS124 Operating Systems Winter 2015-2016, Lecture 23 2 Persistent Storage All programs require some form of persistent storage that lasts beyond the lifetime of an individual process Most
More informationManaging the Cisco APIC-EM and Applications
Managing Cisco APIC-EM Using the GUI, page 1 Cisco APIC-EM Application Separation, page 1 Information about Backing Up and Restoring the Cisco APIC-EM, page 4 Updating the Cisco APIC-EM Software, page
More informationCS 1550 Project 3: File Systems Directories Due: Sunday, July 22, 2012, 11:59pm Completed Due: Sunday, July 29, 2012, 11:59pm
CS 1550 Project 3: File Systems Directories Due: Sunday, July 22, 2012, 11:59pm Completed Due: Sunday, July 29, 2012, 11:59pm Description FUSE (http://fuse.sourceforge.net/) is a Linux kernel extension
More informationIntroduction to High-Performance Computing (HPC)
Introduction to High-Performance Computing (HPC) Computer components CPU : Central Processing Unit cores : individual processing units within a CPU Storage : Disk drives HDD : Hard Disk Drive SSD : Solid
More informationHistory of SURAgrid Deployment
All Hands Meeting: May 20, 2013 History of SURAgrid Deployment Steve Johnson Texas A&M University Copyright 2013, Steve Johnson, All Rights Reserved. Original Deployment Each job would send entire R binary
More informationCSC209H Lecture 1. Dan Zingaro. January 7, 2015
CSC209H Lecture 1 Dan Zingaro January 7, 2015 Welcome! Welcome to CSC209 Comments or questions during class? Let me know! Topics: shell and Unix, pipes and filters, C programming, processes, system calls,
More informationInstall Certificate on the Cisco Secure ACS Appliance for PEAP Clients
Install Certificate on the Cisco Secure ACS Appliance for PEAP Clients Document ID: 64067 Contents Introduction Prerequisites Requirements Components Used Conventions Microsoft Certificate Service Installation
More informationCMSC 201 Spring 2017 Lab 01 Hello World
CMSC 201 Spring 2017 Lab 01 Hello World Assignment: Lab 01 Hello World Due Date: Sunday, February 5th by 8:59:59 PM Value: 10 points At UMBC, our General Lab (GL) system is designed to grant students the
More informationCloud Control Panel User Manual v1.1
Cloud Control Panel User Manual v1.1 March 2011 Page: 1 / 27 Contents 1 Introduction...3 2 Login procedure...4 3 Using the Dashboard...7 3.1 Enabling the Detailed View...8 3.2 Stopping the component...9
More informationDeploying a Production Gateway with Airavata
Deploying a Production Gateway with Airavata Table of Contents Pre-requisites... 1 Create a Gateway Request... 1 Gateway Deploy Steps... 2 Install Ansible & Python...2 Deploy the Gateway...3 Gateway Configuration...
More information