Getting Started with XSEDE Andrew Grimshaw and Karolina Sarnowska- Upton
|
|
- Alexia Logan
- 5 years ago
- Views:
Transcription
1 January 10, 2012 Getting Started with XSEDE Andrew Grimshaw and Karolina Sarnowska- Upton
2 Audience End users and developers who want to Access and use NSF funded XSEDE compute, data, and storage resources Create a secure shared resource environment with collaborators around the world Access and use compute, data, and storage resources available via XSEDE located at other institutions Advanced user support personnel who work with end users and developers
3 Goals At the end of this tutorial you will Understand underlying system and resource model of XSEDE Be able to install and configure client-side tools Understand grid command shell and basics of GUI Be able to define and run jobs on XSEDE Be able to use Global Federated File System Be able to share data resources into XSEDE 3
4 Agenda XSEDE Architecture Overview and Context XSEDE Genesis II Client Installation Using the client interfaces Running a job with XSEDE Adding data resources 4
5 January 10, 2012 XSEDE Architecture Overview & Context Andrew Grimshaw and Karolina Sarnowska-Upton
6 Initial XSEDE architecture: High-order bits Don t disrupt the user community! Maintain existing TeraGrid services Focus on user-facing access layer For power users, first, do no harm For other users, expand use via interfacees, new hosted XSEDE User Access Services (XUAS) and Global Federated File System (GFFS) Promote standards and best practices to enhance interoperability, portability, and implementation choice 6
7 XSEDE provides capabilities Access and share data between campuses and centers Access data on center resources from the campus, campus resources from a center, or campus A resources from campus B Access and share compute resources from home, campus, or center to run a job directly on a particular resource submit to one or more global queues to execute a workflow 7
8 XSEDE Architecture Access Layer Applications, GUIs, Portals and Gateways, XUAS Transparent access via the file system APIs and CLIs Services & Web Services Infrastructure XSEDE Enterprise Services JSDL/BES HPC-BP GridFTP WSI-BSP RNS/ByteIO Community Provided GRAM5 Services REST/RMI Amazon EC2 Application Deployment Resources Core Enterprise Resources, e.g., RP resources Other Resources, e.g. Campus centers, Amazon, Research group data 8
9 Implementations and Architecture The architecture defines the interfaces, communication, and interactions between software components The architecture defines how quality attributes are realized Security, reliability, availability, performance,.. Architecture components (that implement interfaces) may have more than one implementation Thus, we distinguish between the architecture and the implementation 9
10 Implementation Choices We have made initial choices of implementations we will use Process to evolve architecture & implementations Three major configuration items (software systems) providing implementations. They are (in alphabetical order) Genesis II :CLIs, APIs, GUI, GFFS, XES services Globus: XAUS (XD-Data), gridftp UNICORE 6: GUI, XES (BES at the SPs) XES services run on Grid Interface Units 10
11 XSEDE is a System of Systems XSEDE is a system of systems: Different organizations may be running different standards-compliant software stacks. 11
12 A Typical Service Provider Setup Supercomputer and local storage Login nodes Local scheduler e.g., PBS Grid Interface Unit(s) Site backbone Connection to internet Supercomputer and local storage Site wide file system and archival storage Data
13 A Typical Campus Setup Campus cluster Researcher data set Grid Interface Unit(s) Connection to Campus backbone internet Researcher cluster Department file system
14 Simple Grid Interface Unit Grid Interface Unit Local queuing systems Web Service Container Local distributed file systems Local disk
15 January 10, 2012 XSEDE Genesis II Client Installation Andrew Grimshaw and Karolina Sarnowska-Upton
16 Agenda Install Genesis II grid client 16
17 Acquire Installer Installers are delivered with Increment 1 TRR materials. Select the installer for the appropriate operating system platform. Run the installer. 17
18 The Installation Process Questions OK to install? License follows Apache license agreement 18
19 The Installation Process Questions Installation directory path? where code and configuration files will be placed **Directory to store container state will be created at ~/.genesisii
20 Grid Choice Question Shows supported grids; pick XSEDE for Increment 1 Deliverable. 20
21 Installer Progress...
22 22 Voila Client Installation Complete
23 January 10, 2012 XSEDE Genesis II Client Usage Andrew Grimshaw and Karolina Sarnowska-Upton
24 Agenda Prerequisites Client installed Access grid via: Cmd-line grid shell GUI client FUSE file system mount Learn access control basics 24
25 Using the Grid Client Multiple access methods Cmd-line grid shell GUI client-ui FUSE file system mount You will learn to: Login Navigate namespace Use GUI Manage access control Setup FUSE mount 25
26 Login via the CLI Note: All of the things we will talk about can also be done from the grid shell without using the GUI, it is just not as convenient Login using your grid credentials login Check grid credentials whoami 26
27 Fire up the GUI Type grid At the command line type client-ui You should see something like this Let s look around /queus /users /home /groups 27
28 /users versus /home /users is a directory of end user identities Used to log in and to add people to access control lists, e.g., chmod myfile +r /users/karolina /home shows home directories in GFFS of users you can put files and directories there E.g., /home/grimshaw/data.txt 28
29 GUI Grid Client: Start-Up Basics Browse to /home Click on your directory icon Open GUI sub shell Select Tools, then grid shell Shell as tab completion, history, help, etc. 29
30 GUI Grid Client: Tearing off a Browser Create additional GUI browser of grid global namespace by: Clicking Tear icon and draging to tear off browser 30
31 GUI Grid Client: View Access Control To view access control information: Browse to and highlight resource, then select Security tab Exercise: Give read access to your neighbor 31
32 GUI Grid Client: Edit Access Control Select credential to be added Add specific user by browsing to user identity under /users Add everyone by selecting Everyone icon Add specific username/password token by filling in dialog box and selecting icon Drag and drop credential to add desired rwx permission 32
33 That s it for the GUI for now Let s look at mapping the directory structure into the local file system using FUSE 33
34 FUSE Mounting the Grid: Overview Filesystem in Userspace (FUSE) is a loadable kernel module for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code We use FUSE to provide accesses to gird resources directly from your Linux file system via a directory mount point 34
35 FUSE Mounting the Grid: Setup Basics Ensure you are logged into the grid GenesisII/grid whoami Create empty Unix directory to use as mount point mkdir XSEDE Mount grid at mount point nohup GenesisII/grid fuse --mount local:xsede & Now you can access XSEDE via your file system Can add command to your Unix login dotfile to setup FUSE mount automatically on Unix login 35
36 Result XSEDE resources regardless of location can be accessed via the file system Files and directories can be accessed by programs and shell scripts as if they were local files Jobs can be started by copying job descriptions into directories One can see the jobs running or queued by doing an ls. One can cd into a running job and access the working directory where the job is running directly More on this later 36
37 GUI Grid Client: Editing Files Edit files in default editor (from client-ui sub-shell or grid shell) edit <filename> In Linux, EDITOR environment variable needs to be set before running grid client; e.g.: export EDITOR=/usr/bin/vim 37
38 GUI Grid Client: Configuring Preferences Select Preferences under File menu to configure: Credential verbosity Shell fonts Default job history level XML display mode 38
39 January 10, 2012 Running a Job with XSEDE Andrew Grimshaw and Karolina Sarnowska-Upton
40 Audience & Goals Audience End users and developers who want to Access and use NSF funded XSEDE compute resources Create secure shared compute environment with collaborators around the world Access and use compute resources available via XSEDE located at other institutions Advanced user support personnel who work with end users and developers Goals: at the end of this tutorial you will Be able to define and run jobs on XSEDE
41 Prerequisites Installed Genesis II client software Grid account with permission to run jobs Basic grid shell and client GUI understanding 41
42 XSEDE Activities (a.k.a. jobs) What are jobs in XSEDE? How are jobs executed? How are jobs specified? How to interact with jobs while they are running? Compute Grid Use module JSDL tool Grid queue Interacting with jobs Job state change notification
43 What are Jobs in XSEDE? A job is a unit of work that executes a program Really pretty generic: much like PBS or LSF job Program may be sequential, threaded, hybrid GPGPU program, or traditional parallel using MPI or OpenMP Programs can be command line programs or shell scripts that take zero or more parameters Jobs MAY specify files to be staged in before execution and out after execution This MAY include executables and libraries Jobs MAY specify file systems to mount, e.g., SCRATCH or GFFS (Global Federated File System) Jobs MAY specify resource requirements such as operating system, amount of memory, number of CPU s, or other matching criteria Jobs MAY be parameter sweep jobs with arbitrary number of dimensions 43
44 How are Jobs Executed? Job are executed by grid resources that implement the OGSA Basic Execution Services (BES) interface These are referred to as BESes Users submit jobs directly to BES or to a grid queue 44
45 BESes: Basic Execution Services BESes run jobs on particular compute resources Manage data staging for jobs Monitor job progress/completion Maintains job state Compute resources may be workstations, clusters, or supercomputers Each BES has a set of resource properties such as operating system, memory, number of cores, etc. that can be used to match jobs to BESes for execution XCG Tutorial
46 Grid Queues Work much like any other queuing system Grid users submit jobs to grid queue Maintain: List of (BES) compute resources available for scheduling Description of capabilities of each compute resource List of jobs and statuses Match jobs to available compute resources Ask matching resources to run jobs Monitor job progress/completion Cmd-line and GUI tools to manage jobs in queue qsub, qstat, qkill, qcomplete, queue manager XCG Tutorial
47 Grid Queues Cmd-line View Check queue/job status with: qstat <queue-path> XCG Tutorial
48 Grid Queues GUI Queue Manager Queue Manager presents information about jobs and resources currently managed by queue Click in the Max Slots column in the row for the desired resource, type in a number, and save. 48
49 Grid Queues Job Execution Grid-Queue job1 job2 job3 job4 BES1 BES executes job BES2 BES3 XCG
50 Job Execution The Working Directory User submits job/queue schedules on BES BES1 BES creates unique working dir for each job activities my_job_data job1 job2 job3 working-dir runa runb BES stages data to/from job working dir as specified in JSDL XCG
51 How are Jobs Specified? Jobs are specified using the Open Grid Forum standard Job Specification Description Language 1.0 (JSDL) XML-based language Widely adopted Not intended for human consumption Job information that is specified Identity Application description Resource requirements Data staging 51
52 JSDL Fragment Gdfg Job Name Resource Requirement Application Description XCG Tutorial
53 Creating JSDL Files using the Grid Job Tool Manual Creation: Use editor to create XML file Difficult and error-prone due to XML s eccentricities Easiest method: start with existing JSDL and modify (carefully) Using Grid Job Tool: GUI builder for JSDL files User describes job in GUI Description can be saved as GridJobTool project file edit/re-use project to create new JSDL files Automatically generates XML from user provided description Started with grid command job-tool XCG Tutorial
54 How to Launch Job Tool from GUI Browser Select directory where you want JSDL project file located OR Select execution container (BES or queue) where you want to execute job 54
55 How are Jobs Submitted for Execution? Recall: Jobs submitted to BES or grid-queue Jobs can be submitted via Grid shell run (to BES) or qsub (to queue) commands JSDL tool menu option from GUI grid shell Copying JDSL file to BES s submission-point pseudo-directory 55
56 Using run to Execute Jobs Check command syntax help run EXAMPLE run command for gnomad run --jsdl=<jsdf-file> <path-to-bes> 56
57 Using a Grid Queue to Execute Jobs General purpose XSEDE grid queue location /queues/grid-queue Submission syntax qsub <queue-path> <JSDL-file> OR cp <queue-path>/submission-point <JSDL-file> Example submission qsub /queues/grid-queue local:gnomad.xml 57
58 Job Submission Exercises GOAL: Run some simple jobs Create and execute hostname.jsdl Single job and parameter sweep Example files located at /examples 58
59 Interact with Jobs via Queue Manager You can stop, check status, examine job history, or reschedule a job You can interact with a job s working directory if job is in a running state on a (Genesis II) BES 59
60 View Job Information in Queue Manager Status QUEUED: job waiting to be scheduled on BES resources REQUEUED: job failed execution at least once and has been automatically re-queued ERROR: job failed the maximum allowable execution attempts and will not be re-queued On <BES name>: job passed to <BES name> for execution Note: Does not connote status within BES (job may be running, queued, staging data, etc.) FINISHED: job executed successfully Attempts Number of times queue has tried schedule job for execution Some failures do not increment attempts grid software failures job preempted due to local BES policies Ticket Unique ID assigned by grid queue to job on submission Queue keeps status of active and completed jobs Jobs in final status (ERROR and FINISHED) need to be cleaned up by user qcomplete <queue name> { --all <job ticket>+ } XCG Tutorial
61 Examine Job History in Queue Manager Right-clicking on job provides information about job s history in different levels of detail 61
62 Scratch file system Persists on BES between runs Good for caching large or frequently used files 62
63 Interact with Job Working Directory When using Genesis II BES resources, job working directory is accessible via GFFS Working-directory is located in queue where job was submitted at <queue-path>/jobs/mine/running For each running job, there is a directory with job ticket number with two entries: status file containing state of job (e.g. queued, running) working-dir session execution directory of running job read/write/create/delete files here to interact with running job If job was submitted directly to BES, job directory is located at <bes-path>/activities 63
64 JSDL File Contents Explained Identifier Info Descriptive information about job, e.g. job name Resource Requirements Describe resources job requires Memory OS Architecture Number of processors Run time <JobIdentification> <JobName>Adder</JobName> </JobIdentification> <Resources> <OperatingSystem> <OperatingSystemType> <OperatingSystemName>LINUX </OperatingSystemName> </OperatingSystemType> </OperatingSystem> </Resources> XCG Tutorial
65 JSDL File Contents Explained Application Description Describe execution Executable name Arguments Routing for stdout and stderr <Application> <POSIXApplication> <Executable>adder.sh</Executable> <Output>stdout</Output> <Argument>seven.dat</Argument> <Argument>fourty-two.dat</Argument> <Argument>sum.dat</Argument> <Argument>10</Argument> </POSIXApplication> </Application> XCG Tutorial
66 JSDL File Contents Explained DataStaging Describe data to copy in/out Several transport options: http scp (secure copy) RNS (grid directory structure) (out only) Copy in (data staging source): Source is URL of remote file to be copied in FileName is name within job working directory where file will be copied to Copy out (data staging target): Target is URL of remote file to be copied to FileName is name within job working directory of file to be copied out Other file handling info <DataStaging> <FileName>adder.sh</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</DeleteOnTermination> <Source> <URI> </Source> </DataStaging> <DataStaging> <FileName>sum.dat</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</DeleteOnTermination> <Target> <URI>rns:sum.dat</URI> </Target> </DataStaging> XCG Tutorial
67 Example JSDL Gdfg Job Name Resource Requirement Application Description XCG Tutorial
68 Gdfg Data Staging Requests XCG
69 January 10, 2012 Adding DATA Resources into the Grid Andrew Grimshaw
70 Ways to Add Data into the Grid Create files and directories Export file system directory 70
71 Creating Files in the Grid Creating a file (or directory) places its state on same grid container as its containing directory For example, all these following commands place files and directories in container where /home/bob resides echo hello > /home/bob/newfile mkdir /home/bob/testdir cp local:testfile grid:/home/bob/testfile 71
72 Creating Directories on Specific Containers Files can be created on other containers by specifying creating a containing directory on target container Directory placement location can be changed by explicitly specifying grid container to be used Path to service on target container is given to directory creation command (service is EnhancedRNSServicePortType) mkdir --rns-service=<rns-service-path> <new-dir-path> 72
73 Exports: Mapping Data into the Grid Basic idea: create grid resource that securely proxies access to local files and directories via RNS and ByteIO web services We use an export service to proxy a local file system directory tree into grid To create export, create instance of LightWeightExportPortType Via the command line Via the GUI (for local hosts) 73
74 Exporting: Mapping a local directory structure into the global namespace User runs export command Export Service myfiles Export service mounts local directory into global namespace user /home Export services redirects calls from grid export to local file system myexport XCG Tutorial
75 Export Creation Example: Cmdline Creating an export maps specified directory on container host into specified GFFS path To run export command, you need to know Location of files you want to export On which container you will create export resource (service is LightWeightExportPortType) Location in global namespace where you want to mount export export --create <path-to-service> <local-path-to-files> <GFFS-path-for-export> Quitting export turns off export service (underlying files in local file system are left intact) export --quit <GFFS-path-for-export> 75
76 Export Creation Example: GUI Provide: Location of files you want to export Location in global namespace where you want to mount files 76
77 Export Security Settings Recommendation Give users extended access control to enable export creation Allow only admin users to create exports 77
78
XSEDE Architecture Level 3 Decomposition
XSEDE Architecture Level 3 Decomposition June 30, 2013 Version 0.971 Felix Bachmann, Ian Foster, Andrew Grimshaw, David Lifka, Morris Riedel, Steven Tuecke XSEDE Architecture Level 3 Decomposition v.0.97
More informationGetting Started with XSEDE. Dan Stanzione
November 3, 2011 Getting Started with XSEDE Dan Stanzione Welcome to XSEDE! XSEDE is an exciting cyberinfrastructure, providing large scale computing, data, and visualization resources. XSEDE is the evolution
More informationA Hands-On Tutorial: RNA Sequencing Using High-Performance Computing
A Hands-On Tutorial: RNA Sequencing Using Computing February 11th and 12th, 2016 1st session (Thursday) Preliminaries: Linux, HPC, command line interface Using HPC: modules, queuing system Presented by:
More informationIndependent Software Vendors (ISV) Remote Computing Usage Primer
GFD-I.141 ISV Remote Computing Usage Primer Authors: Steven Newhouse, Microsoft Andrew Grimshaw, University of Virginia 7 October, 2008 Independent Software Vendors (ISV) Remote Computing Usage Primer
More informationUoW HPC Quick Start. Information Technology Services University of Wollongong. ( Last updated on October 10, 2011)
UoW HPC Quick Start Information Technology Services University of Wollongong ( Last updated on October 10, 2011) 1 Contents 1 Logging into the HPC Cluster 3 1.1 From within the UoW campus.......................
More informationXSEDE New User Tutorial
April 2, 2014 XSEDE New User Tutorial Jay Alameda National Center for Supercomputing Applications XSEDE Training Survey Make sure you sign the sign in sheet! At the end of the module, I will ask you to
More informationXSEDE Campus Bridging Tools Rich Knepper Jim Ferguson
April 3, 2014 XSEDE Campus Bridging Tools Rich Knepper (rich@iu.edu) Jim Ferguson (jwf@utk.edu) What is Campus Bridging? Bridging the gap between local researcher cyberinfrastructure, campus CI, and national
More informationXSEDE Campus Bridging Tools Jim Ferguson National Institute for Computational Sciences
September 26, 2014 XSEDE Campus Bridging Tools Jim Ferguson National Institute for Computational Sciences jwf@utk.edu Quick Advertisement: Student Programs Research Experience 12-18 students per summer,
More informationX-WAVE Canonical Use Case 3 Remote File Access Level 3 Version 0.92
X-WAVE Canonical Use Case 3 Remote File Access Level 3 Version 0.92 Template 02November2004 Revision history: RELEASE/REVISION: RELEASE/REVISION DATE: SUMMARY OF CHANGES 0.9 June 26, 2013 First Draft of
More informationSession 1: Accessing MUGrid and Command Line Basics
Session 1: Accessing MUGrid and Command Line Basics Craig A. Struble, Ph.D. July 14, 2010 1 Introduction The Marquette University Grid (MUGrid) is a collection of dedicated and opportunistic resources
More informationGridbus Portlets -- USER GUIDE -- GRIDBUS PORTLETS 1 1. GETTING STARTED 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4
Gridbus Portlets -- USER GUIDE -- www.gridbus.org/broker GRIDBUS PORTLETS 1 1. GETTING STARTED 2 1.1. PREREQUISITES: 2 1.2. INSTALLATION: 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4 3.1. CREATING
More informationSiemens PLM Software. HEEDS MDO Setting up a Windows-to- Linux Compute Resource.
Siemens PLM Software HEEDS MDO 2018.04 Setting up a Windows-to- Linux Compute Resource www.redcedartech.com. Contents Introduction 1 On Remote Machine B 2 Installing the SSH Server 2 Configuring the SSH
More informationNew User Tutorial. OSU High Performance Computing Center
New User Tutorial OSU High Performance Computing Center TABLE OF CONTENTS Logging In... 3-5 Windows... 3-4 Linux... 4 Mac... 4-5 Changing Password... 5 Using Linux Commands... 6 File Systems... 7 File
More informationXSEDE New User Training. Ritu Arora November 14, 2014
XSEDE New User Training Ritu Arora Email: rauta@tacc.utexas.edu November 14, 2014 1 Objectives Provide a brief overview of XSEDE Computational, Visualization and Storage Resources Extended Collaborative
More informationName Department/Research Area Have you used the Linux command line?
Please log in with HawkID (IOWA domain) Macs are available at stations as marked To switch between the Windows and the Mac systems, press scroll lock twice 9/27/2018 1 Ben Rogers ITS-Research Services
More informationSupercomputing environment TMA4280 Introduction to Supercomputing
Supercomputing environment TMA4280 Introduction to Supercomputing NTNU, IMF February 21. 2018 1 Supercomputing environment Supercomputers use UNIX-type operating systems. Predominantly Linux. Using a shell
More informationQosCosGrid Middleware
Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space PLGrid Plus QosCosGrid Middleware Domain-oriented services and resources
More informationSUG Breakout Session: OSC OnDemand App Development
SUG Breakout Session: OSC OnDemand App Development Basil Mohamed Gohar Web and Interface Applications Manager Eric Franz Senior Engineer & Technical Lead This work is supported by the National Science
More informationIntroduction to Linux and Cluster Computing Environments for Bioinformatics
Introduction to Linux and Cluster Computing Environments for Bioinformatics Doug Crabill Senior Academic IT Specialist Department of Statistics Purdue University dgc@purdue.edu What you will learn Linux
More informationTable of Contents. Table of Contents Job Manager for remote execution of QuantumATK scripts. A single remote machine
Table of Contents Table of Contents Job Manager for remote execution of QuantumATK scripts A single remote machine Settings Environment Resources Notifications Diagnostics Save and test the new machine
More informationContents. Note: pay attention to where you are. Note: Plaintext version. Note: pay attention to where you are... 1 Note: Plaintext version...
Contents Note: pay attention to where you are........................................... 1 Note: Plaintext version................................................... 1 Hello World of the Bash shell 2 Accessing
More informationParallel Programming Pre-Assignment. Setting up the Software Environment
Parallel Programming Pre-Assignment Setting up the Software Environment Authors: B. Wilkinson and C. Ferner. Modification date: Aug 21, 2014 (Minor correction Aug 27, 2014.) Software The purpose of this
More informationIntroduction to Unix The Windows User perspective. Wes Frisby Kyle Horne Todd Johansen
Introduction to Unix The Windows User perspective Wes Frisby Kyle Horne Todd Johansen What is Unix? Portable, multi-tasking, and multi-user operating system Software development environment Hardware independent
More informationUsing the vrealize Orchestrator Operations Client. vrealize Orchestrator 7.5
Using the vrealize Orchestrator Operations Client vrealize Orchestrator 7.5 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments
More informationUsing ISMLL Cluster. Tutorial Lec 5. Mohsan Jameel, Information Systems and Machine Learning Lab, University of Hildesheim
Using ISMLL Cluster Tutorial Lec 5 1 Agenda Hardware Useful command Submitting job 2 Computing Cluster http://www.admin-magazine.com/hpc/articles/building-an-hpc-cluster Any problem or query regarding
More informationX-WAVE Canonical Use Case 1 Run A Remote Job Level 3 Version 0.94
X-WAVE Canonical Use Case 1 Run A Remote Job Level 3 Version 0.94 Review completed by SD&I November, 2013 Template 02November2004 Revision history: RELEASE/REVISION: RELEASE/REVISION DATE: SUMMARY OF CHANGES
More informationCampus Bridging: What is it and how do we do it? Rich Knepper
September 11, 2014 Campus Bridging: What is it and how do we do it? Rich Knepper rich@iu.edu What is Campus Bridging? Bridging the gap between local researcher cyberinfrastructure, campus CI, and national
More informationGPU Cluster Usage Tutorial
GPU Cluster Usage Tutorial How to make caffe and enjoy tensorflow on Torque 2016 11 12 Yunfeng Wang 1 PBS and Torque PBS: Portable Batch System, computer software that performs job scheduling versions
More informationExercise 1: Connecting to BW using ssh: NOTE: $ = command starts here, =means one space between words/characters.
Exercise 1: Connecting to BW using ssh: NOTE: $ = command starts here, =means one space between words/characters. Before you login to the Blue Waters system, make sure you have the following information
More informationLinux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades Q2
Linux Operating System Environment Computadors Grau en Ciència i Enginyeria de Dades 2017-2018 Q2 Facultat d Informàtica de Barcelona This first lab session is focused on getting experience in working
More informationIntroduction: What is Unix?
Introduction Introduction: What is Unix? An operating system Developed at AT&T Bell Labs in the 1960 s Command Line Interpreter GUIs (Window systems) are now available Introduction: Unix vs. Linux Unix
More informationXSEDE New User Tutorial
June 12, 2015 XSEDE New User Tutorial Jay Alameda National Center for Supercomputing Applications XSEDE Training Survey Please remember to sign in for today s event: http://bit.ly/1fashvo Also, please
More informationPresented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory CONTAINERS IN HPC WITH SINGULARITY
Presented By: Gregory M. Kurtzer HPC Systems Architect Lawrence Berkeley National Laboratory gmkurtzer@lbl.gov CONTAINERS IN HPC WITH SINGULARITY A QUICK REVIEW OF THE LANDSCAPE Many types of virtualization
More informationXSEDE New User Tutorial
May 13, 2016 XSEDE New User Tutorial Jay Alameda National Center for Supercomputing Applications XSEDE Training Survey Please complete a short on-line survey about this module at http://bit.ly/hamptonxsede.
More informationIntroduction to Linux and Supercomputers
Introduction to Linux and Supercomputers Doug Crabill Senior Academic IT Specialist Department of Statistics Purdue University dgc@purdue.edu What you will learn How to log into a Linux Supercomputer Basics
More informationSoftware Installation - Accessing Linux and Checking your Environmental Variables
Accessing Linux and Checking your Environmental Although you may be fortunate enough to have a powerful multi-processor desktop running Linux, most of our sponsors do not. Most of our sponsors will have
More informationQuick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing
Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Contents User access, logging in Linux/Unix
More informationCS Fundamentals of Programming II Fall Very Basic UNIX
CS 215 - Fundamentals of Programming II Fall 2012 - Very Basic UNIX This handout very briefly describes how to use Unix and how to use the Linux server and client machines in the CS (Project) Lab (KC-265)
More informationIntroduction to the Linux Command Line January Presentation Topics
1/22/13 Introduction to the Linux Command Line January 2013 Presented by Oralee Nudson ARSC User Consultant & Student Supervisor onudson@alaska.edu Presentation Topics Information Assurance and Security
More informationAnsible Tower Quick Setup Guide
Ansible Tower Quick Setup Guide Release Ansible Tower 2.4.5 Red Hat, Inc. Jun 06, 2017 CONTENTS 1 Quick Start 2 2 Login as a Superuser 3 3 Import a License 4 4 Examine the Tower Dashboard 6 5 The Setup
More informationConfiguring the Oracle Network Environment. Copyright 2009, Oracle. All rights reserved.
Configuring the Oracle Network Environment Objectives After completing this lesson, you should be able to: Use Enterprise Manager to: Create additional listeners Create Oracle Net Service aliases Configure
More informationScheduling in SAS 9.4, Second Edition
Scheduling in SAS 9.4, Second Edition SAS Documentation September 5, 2017 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2016. Scheduling in SAS 9.4, Second Edition.
More informationIntroduction to HPC Resources and Linux
Introduction to HPC Resources and Linux Burak Himmetoglu Enterprise Technology Services & Center for Scientific Computing e-mail: bhimmetoglu@ucsb.edu Paul Weakliem California Nanosystems Institute & Center
More informationOGCE User Guide for OGCE Release 1
OGCE User Guide for OGCE Release 1 1 Publisher s Note Release 2 begins the migration to open standards portlets. The following has been published by the Open Grids Computing Environment: OGCE Release 2
More informationXSEDE Software and Services Table For Service Providers and Campus Bridging
XSEDE Software and Services Table For Service Providers and Campus Bridging 24 September 2015 Version 1.4 Page i Table of Contents A. Document History iv B. Document Scope v C. 1 Page ii List of Figures
More informationIntroduction to application management
Introduction to application management To deploy web and mobile applications, add the application from the Centrify App Catalog, modify the application settings, and assign roles to the application to
More informationArcGIS for Server: Administration and Security. Amr Wahba
ArcGIS for Server: Administration and Security Amr Wahba awahba@esri.com Agenda ArcGIS Server architecture Distributing and scaling components Implementing security Monitoring server logs Automating server
More informationBasic UNIX commands. HORT Lab 2 Instructor: Kranthi Varala
Basic UNIX commands HORT 59000 Lab 2 Instructor: Kranthi Varala Client/Server architecture User1 User2 User3 Server (UNIX/ Web/ Database etc..) User4 High Performance Compute (HPC) cluster User1 Compute
More informationHigh Performance Computing (HPC) Club Training Session. Xinsheng (Shawn) Qin
High Performance Computing (HPC) Club Training Session Xinsheng (Shawn) Qin Outline HPC Club The Hyak Supercomputer Logging in to Hyak Basic Linux Commands Transferring Files Between Your PC and Hyak Submitting
More informationEMC Documentum Composer
EMC Documentum Composer Version 6.5 SP2 User Guide P/N 300-009-462 A01 EMC Corporation Corporate Headquarters: Hopkinton, MA 01748-9103 1-508-435-1000 www.emc.com Copyright 2008 2009 EMC Corporation. All
More informationOur Workshop Environment
Our Workshop Environment John Urbanic Parallel Computing Scientist Pittsburgh Supercomputing Center Copyright 2015 Our Environment Today Your laptops or workstations: only used for portal access Blue Waters
More informationAn Introduction to Cluster Computing Using Newton
An Introduction to Cluster Computing Using Newton Jason Harris and Dylan Storey March 25th, 2014 Jason Harris and Dylan Storey Introduction to Cluster Computing March 25th, 2014 1 / 26 Workshop design.
More informationCS CS Tutorial 2 2 Winter 2018
CS CS 230 - Tutorial 2 2 Winter 2018 Sections 1. Unix Basics and connecting to CS environment 2. MIPS Introduction & CS230 Interface 3. Connecting Remotely If you haven t set up a CS environment password,
More informationQuick Start Guide. by Burak Himmetoglu. Supercomputing Consultant. Enterprise Technology Services & Center for Scientific Computing
Quick Start Guide by Burak Himmetoglu Supercomputing Consultant Enterprise Technology Services & Center for Scientific Computing E-mail: bhimmetoglu@ucsb.edu Linux/Unix basic commands Basic command structure:
More informationGrid Scheduling Architectures with Globus
Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents
More informationCENG 334 Computer Networks. Laboratory I Linux Tutorial
CENG 334 Computer Networks Laboratory I Linux Tutorial Contents 1. Logging In and Starting Session 2. Using Commands 1. Basic Commands 2. Working With Files and Directories 3. Permission Bits 3. Introduction
More informationManaging Software Images Using Software Management
CHAPTER 8 Managing Software Images Using Software Management Manually upgrading your devices to the latest software version can be an error-prone, and time-consuming process. To ensure rapid, reliable
More informationPartner Integration Portal (PIP) Installation Guide
Partner Integration Portal (PIP) Installation Guide Last Update: 12/3/13 Digital Gateway, Inc. All rights reserved Page 1 TABLE OF CONTENTS INSTALLING PARTNER INTEGRATION PORTAL (PIP)... 3 DOWNLOADING
More informationGetting started with the CEES Grid
Getting started with the CEES Grid October, 2013 CEES HPC Manager: Dennis Michael, dennis@stanford.edu, 723-2014, Mitchell Building room 415. Please see our web site at http://cees.stanford.edu. Account
More informationHelsinki 19 Jan Practical course in genome bioinformatics DAY 0
Helsinki 19 Jan 2017 529028 Practical course in genome bioinformatics DAY 0 This document can be downloaded at: http://ekhidna.biocenter.helsinki.fi/downloads/teaching/spring2017/exercises_day0.pdf The
More informationDCLI User's Guide. Data Center Command-Line Interface 2.9.1
Data Center Command-Line Interface 2.9.1 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about this documentation, submit
More informationGrid Computing in SAS 9.4
Grid Computing in SAS 9.4 SAS Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2013. Grid Computing in SAS 9.4. Cary, NC: SAS Institute Inc. Grid Computing
More informationCISC 220 fall 2011, set 1: Linux basics
CISC 220: System-Level Programming instructor: Margaret Lamb e-mail: malamb@cs.queensu.ca office: Goodwin 554 office phone: 533-6059 (internal extension 36059) office hours: Tues/Wed/Thurs 2-3 (this week
More informationDCLI User's Guide. Data Center Command-Line Interface
Data Center Command-Line Interface 2.10.2 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about this documentation, submit
More informationExeco tutorial Grid 5000 school, Grenoble, January 2016
Execo tutorial Grid 5000 school, Grenoble, January 2016 Simon Delamare Matthieu Imbert Laurent Pouilloux INRIA/CNRS/LIP ENS-Lyon 03/02/2016 1/34 1 introduction 2 execo, core module 3 execo g5k, Grid 5000
More informationGlobus Research Data Management: Campus Deployment and Configuration. Steve Tuecke Vas Vasiliadis
Globus Research Data Management: Campus Deployment and Configuration Steve Tuecke Vas Vasiliadis Presentations and other useful information available at globus.org/events/xsede15/tutorial 2 Agenda Globus
More informationNBIC TechTrack PBS Tutorial
NBIC TechTrack PBS Tutorial by Marcel Kempenaar, NBIC Bioinformatics Research Support group, University Medical Center Groningen Visit our webpage at: http://www.nbic.nl/support/brs 1 NBIC PBS Tutorial
More informationUser Scripting April 14, 2018
April 14, 2018 Copyright 2013, 2018, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and
More informationTyler Dashboard. User Guide Version 6.3. For more information, visit
Tyler Dashboard User Guide Version 6.3 For more information, visit www.tylertech.com. TABLE OF CONTENTS Tyler Dashboard... 4 Tyler Dashboard Features... 4 Search... 5 Browse... 5 Page... 6 Dashboard...
More informationComputational Skills Primer. Lecture 2
Computational Skills Primer Lecture 2 Your Background Who has used SCC before? Who has worked on any other cluster? Do you have previous experience working with basic linux and command line usage (CLI)?
More informationSetting Up Resources in VMware Identity Manager (SaaS) Modified 15 SEP 2017 VMware Identity Manager
Setting Up Resources in VMware Identity Manager (SaaS) Modified 15 SEP 2017 VMware Identity Manager Setting Up Resources in VMware Identity Manager (SaaS) You can find the most up-to-date technical documentation
More informationTable of Contents DevOps Administrators
DevOps Administrators Table of Contents DevOps Administrators Overview for DevOps Admins Managing Images, Projects, Users Configure a Registry Create Users Assign the Administrator Role Create a Project
More informationSAS Model Manager 2.3
SAS Model Manager 2.3 Administrator's Guide SAS Documentation The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2010. SAS Model Manager 2.3: Administrator's Guide. Cary,
More informationHow to build Scientific Gateways with Vine Toolkit and Liferay/GridSphere framework
How to build Scientific Gateways with Vine Toolkit and Liferay/GridSphere framework Piotr Dziubecki, Piotr Grabowski, Michał Krysiński, Tomasz Kuczyński, Dawid Szejnfeld, Dominik Tarnawczyk, Gosia Wolniewicz
More informationIntellicus Cluster and Load Balancing- Linux. Version: 18.1
Intellicus Cluster and Load Balancing- Linux Version: 18.1 1 Copyright 2018 Intellicus Technologies This document and its content is copyrighted material of Intellicus Technologies. The content may not
More informationScheduling in SAS 9.2
Scheduling in SAS 9.2 The correct bibliographic citation for this manual is as follows: SAS Institute Inc. 2009. Scheduling in SAS 9.2. Cary, NC: SAS Institute Inc. Scheduling in SAS 9.2 Copyright 2009,
More informationIntroduction to UNIX. Logging in. Basic System Architecture 10/7/10. most systems have graphical login on Linux machines
Introduction to UNIX Logging in Basic system architecture Getting help Intro to shell (tcsh) Basic UNIX File Maintenance Intro to emacs I/O Redirection Shell scripts Logging in most systems have graphical
More informationg-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.
g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse
More informationALICE Grid/Analysis Tutorial Exercise-Solutions
WLCG Asia Workshop, 2.12.2006 ALICE Grid/Analysis Tutorial Exercise-Solutions Andreas-Joachim Peters CERN www.eu-egee.org cern.ch/lcg http://cern.ch/arda EGEE is a project funded by the European Union
More informationGNU/Linux 101. Casey McLaughlin. Research Computing Center Spring Workshop Series 2018
GNU/Linux 101 Casey McLaughlin Research Computing Center Spring Workshop Series 2018 rccworkshop IC;3df4mu bash-2.1~# man workshop Linux101 RCC Workshop L101 OBJECTIVES - Operating system concepts - Linux
More informationWhen talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used:
Linux Tutorial How to read the examples When talking about how to launch commands and other things that is to be typed into the terminal, the following syntax is used: $ application file.txt
More informationVersion Installation Guide. 1 Bocada Installation Guide
Version 19.4 Installation Guide 1 Bocada Installation Guide Copyright 2019 Bocada LLC. All Rights Reserved. Bocada and BackupReport are registered trademarks of Bocada LLC. Vision, Prism, vpconnect, and
More informationOperating Systems. Copyleft 2005, Binnur Kurt
3 Operating Systems Copyleft 2005, Binnur Kurt Content The concept of an operating system. The internal architecture of an operating system. The architecture of the Linux operating system in more detail.
More informationIntroduction to parallel computing
Introduction to parallel computing using R and the Claudia Vitolo 1 1 Department of Civil and Environmental Engineering Imperial College London Civil Lunches, 16.10.2012 Outline 1 Parallelism What s parallel
More informationLinux Command Line Interface. December 27, 2017
Linux Command Line Interface December 27, 2017 Foreword It is supposed to be a refresher (?!) If you are familiar with UNIX/Linux/MacOS X CLI, this is going to be boring... I will not talk about editors
More informationIntroduction to the Linux Command Line
Introduction to the Linux Command Line May, 2015 How to Connect (securely) ssh sftp scp Basic Unix or Linux Commands Files & directories Environment variables Not necessarily in this order.? Getting Connected
More informationOperating Systems 3. Operating Systems. Content. What is an Operating System? What is an Operating System? Resource Abstraction and Sharing
Content 3 Operating Systems The concept of an operating system. The internal architecture of an operating system. The architecture of the Linux operating system in more detail. How to log into (and out
More informationXSEDE New User Tutorial
October 20, 2017 XSEDE New User Tutorial Jay Alameda National Center for Supercomputing Applications XSEDE Training Survey Please complete a short on line survey about this module at http://bit.ly/xsedesurvey.
More informationEMC Documentum Composer
EMC Documentum Composer Version 6.0 SP1.5 User Guide P/N 300 005 253 A02 EMC Corporation Corporate Headquarters: Hopkinton, MA 01748 9103 1 508 435 1000 www.emc.com Copyright 2008 EMC Corporation. All
More informationZadara Enterprise Storage in
Zadara Enterprise Storage in Google Cloud Platform (GCP) Deployment Guide March 2017 Revision A 2011 2017 ZADARA Storage, Inc. All rights reserved. Zadara Storage / GCP - Deployment Guide Page 1 Contents
More informationPrototype DIRAC portal for EISCAT data Short instruction
Prototype DIRAC portal for EISCAT data Short instruction Carl-Fredrik Enell January 19, 2017 1 Introduction 1.1 DIRAC EGI, first European Grid Initiative, later European Grid Infrastructure, and now simply
More informationvcloud Director Administrator's Guide
vcloud Director 5.5 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of
More informationDCLI User's Guide. Data Center Command-Line Interface 2.7.0
Data Center Command-Line Interface 2.7.0 You can find the most up-to-date technical documentation on the VMware Web site at: https://docs.vmware.com/ The VMware Web site also provides the latest product
More informationZENworks Reporting System Reference. January 2017
ZENworks Reporting System Reference January 2017 Legal Notices For information about legal notices, trademarks, disclaimers, warranties, export and other use restrictions, U.S. Government rights, patent
More informationChapter 2 Introduction to the WS-PGRADE/gUSE Science Gateway Framework
Chapter 2 Introduction to the WS-PGRADE/gUSE Science Gateway Framework Tibor Gottdank Abstract WS-PGRADE/gUSE is a gateway framework that offers a set of highlevel grid and cloud services by which interoperation
More informationDCLI User's Guide. Modified on 20 SEP 2018 Data Center Command-Line Interface
Modified on 20 SEP 2018 Data Center Command-Line Interface 2.10.0 You can find the most up-to-date technical documentation on the VMware website at: https://docs.vmware.com/ If you have comments about
More informationIntroduction. Overview of HCM. HCM Dashboard CHAPTER
CHAPTER 1 This chapter describes the Hosted Collaboration Mediation (HCM) software. It includes: Overview of HCM, page 1-1 Terminology Used in HCM, page 1-2 HCM Dashboard Architecture, page 1-3 Starting
More informationForeScout CounterACT. (AWS) Plugin. Configuration Guide. Version 1.3
ForeScout CounterACT Hybrid Cloud Module: Amazon Web Services (AWS) Plugin Version 1.3 Table of Contents Amazon Web Services Plugin Overview... 4 Use Cases... 5 Providing Consolidated Visibility... 5 Dynamic
More informationOracle Enterprise Manager. 1 Introduction. System Monitoring Plug-in for Oracle Enterprise Manager Ops Center Guide 11g Release 1 (
Oracle Enterprise Manager System Monitoring Plug-in for Oracle Enterprise Manager Ops Center Guide 11g Release 1 (11.1.3.0.0) E18950-03 November 2011 This document provides describes the System Monitoring
More informationBatch Systems & Parallel Application Launchers Running your jobs on an HPC machine
Batch Systems & Parallel Application Launchers Running your jobs on an HPC machine Partners Funding Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike
More information