XSEDE and XSEDE Resources

Transcription

1 October 26, 2012 XSEDE and XSEDE Resources Dan Stanzione Deputy Director, Texas Advanced Computing Center Co-Director, iplant Collaborative

2 Welcome to XSEDE! XSEDE is an exciting cyberinfrastructure, providing large scale computing, data, and visualization resources. XSEDE is the evolution of the NSF TeraGrid. Today s session is a general overview of XSEDE for new XSEDE/TeraGrid users; it is not going to teach you computational science or programming! Use the webcast chat window to ask questions!

3 Outline What is XSEDE? How do I get started? XSEDE User Portal User Responsibility & Security Applying for an Allocation Accessing Resources Managing Data and File Transfers Other Resources Picking the right resource for your job. Running Jobs Managing Your Software Environment Getting Help Next Steps

4 What is XSEDE? The Extreme Science and Engineering Discovery Environment (XSEDE): The most powerful integrated advanced digital resources and services in the world. Funded by NSF. A single virtual system that scientists can use to interactively share computing resources, data, and expertise. 9 supercomputers, 3 visualization systems, and 9 storage systems provided by 16 partner institutions (Service Providers or SPs) 4

5 What is XSEDE? The successor to the TeraGrid, XSEDE is an NSF-funded, advanced, nationally distributed open cyberinfrastructure, consisting of: Supercomputing (and other computing) Storage Visualization Data Collections Network Science Gateways Unified Policies and Programs 5

6 XSEDE Service Providers NCSA, Illinois PSC, Pitt/Carnegie Mellon NICS, Tennessee/ORNL TACC, Texas SDSC, UC San Diego OSC, Ohio State Cornell Virginia Indiana Purdue Rice Shodor Foundation Argonne UC-Berkeley U Chicago SURA Open Science Grid 6

7 Allocation of XSEDE Resources XSEDE resources are allocated through a peerreviewed process. Open to any US open science researcher (or collaborators of US researchers) regardless of funding source. XSEDE resources are provided at NO COST to the end user through NSF funding (~$100M/year). 7

8 How do I get started using XSEDE To get started using XSEDE a researcher needs to: Apply for an Allocation, or Get added to an existing allocation To do either of these things, you should start with the XSEDE User Portal 8

9 XSEDE User Portal (XUP) Web-based single point of contact that provides: Continually updated information about your accounts. Access to your XSEDE accounts and allocated resources: A single location from which to access XSEDE resources. One can access all accounts on various machines from the Portal. Interfaces for data management, data collections, and other user tasks and resources Access to the Help Desk 9

10 The XSEDE.org Home Page From here, you can create a web account at any time! 10

11 User Responsibilities and Security The first time you login to the Portal, at the beginning of each allocation term, you will be asked to accept the User Responsibilities form: Explains acceptable use to protect shared resources and intellectual property. Acknowledgment in publications, etc. You are responsible for your account: Do not share accounts User is responsible for protecting the passwords: Includes not sharing passwords, not writing passwords down where they can be easily found, and not using tools which expose passwords on the network This includes private keys: make sure they are password-protected. Appropriate Behavior Protecting computing, closing SSH terminals when done, logging out of the User Portal when done, etc. Report Suspicious Activity. If you have any suspicion that your account or personal computer has been compromised send to or call 24/ immediately. 11

12 Getting an Allocation If you do not yet have an allocation, you can use the portal to acquire one. If you are a first time investigator, request a startup allocation. 12

13 Creating an Allocation 13

14 Once your allocation is approved: The PI (principle investigator), Co-PI, or Allocation Manager can add users to an existing allocation through the portal. XSEDE UserPortal:My XSEDE->Add/Remove User Takes the portal name of the user you want to add/remove. Accounts at certain Service Providers need to be activated before they can be accessed. 14

15 Accessing XSEDE Resources Several methods are possible: Direct login access Single Sign On (SSO) through portal SSO between resources Through Science Gateways Your choice of method may vary with: How many resources you use How much you want to automate file transfers, job submission, etc. 15

16 Accessing Resources(2) SSO is the default method; you ll need to file a ticket to request a direct access password to the machine. Direct access: Use a secure shell (ssh) client. From Linux or Mac terminal window: ssh l <username> <machinename> E.g: ssh l dstanzi ranger.tacc.utexas.edu From Windows: Download one of many ssh clients Free ones include putty Most campuses have a site license for a fancier one. 16

17 Single Sign On Single Sign-On (SSO)allows you to use just one username and password (your User Portal one) to log into every digital service on which you have an account. The easiest way to use SSO is via the XSEDE User Portal, but you can also use SSO via a desktop client or with an X.509 certificate. Stand-alone client: After you authenticate using SSO with your User Portal username and password, you will be recognized by all XSEDE services on which you have account, without having to enter your login information again for each resource. 17

18 SSO thru user portal Make sure you are logged into the XSEDE User Portal Go to My XSEDE tab Go to the Accounts link Resources you have access to will be indicated by a login link Click on the login link of the resource you would like to access. 18

19 SSO Thru User Portal A Java Applet will talk you may be asked permission to allow it to run. After the applet starts, a blank terminal window will appear in your web browser. The window will fill with text indicating that you have been successfully logged into the resource of your choice. You can now work on this machine, and connect to other machines from this terminal, using the command gsissh machine-name 19

20 Another Access Path: Science Gateways There are many sites that give you web-based, domain-specific access to applications running on XSEDE. Collectively, we call them Science Gateways View a list of them on the User Portal, in the Resources tab. Access methods vary; click on the specific gateway to find out more (dozens available, across many fields!). iplant s DE is one of these gateways! 20

21 The Mobile User Portal Allows browsing of all XSEDE systems, file downloading, and third-party transfers. It provides several features for mobile users such as one touch file publishing to the user's public folder, simple creation of shared groups for any file/folder, and one click permission management of all XSEDE systems, file downloading, and third-party transfers. 21

22 XUP Resource Monitor View system information: TFLOPS, memory, today s load, jobs running in queue. Status: up or down: takes you to the news announcements that tells you when the machine is expected to come back up. 22

23 User Portal: User Forums The User Forums are a great place to ask question, get help, or discuss ideas about XSEDE. 23

24 Running Jobs Each system in XSEDE has some local options you will need to know about to run jobs. To learn about the specifics of each system, check out the user guides: In the portal, under Documentation select User Guides Pay particular attention to: File Systems Batch job submission 24

25 File Systems on XSEDE Resources Where your data resides on XSEDE and the appropriate storage is your responsibility. In general, all resources provide: HOME: Permanent space, but small. A good choice for building software and working file collections of small to medium sized files, where a medium sized file is less than 50 MB. SCRATCH: More space, but temporary; use while you are running your jobs. Scratch space is temporary; it is not backed up and has limited redundancy, and is periodically purged of old files! Archival Storage: Long term storage of large amounts of data (often tape); slower access, accessible from all sites. 25

26 Batch Jobs All XSEDE compute resources use some form of batch scheduler. Compute jobs *can not* be run on the login nodes (no faster than a normal workstation!) There are several batch systems in use, but all work basically the same way. Request number/type of nodes you need. Specify how long you need to run. Specify where your output files go. Jobs typically described with a job script:

27 Sample Job Script for Grid Engine on TACC Lonestar: #!/bin/bash #$ -N mympi # Job Name #$ -j y # Combine stderr and stdout #$ -o $JOB_NAME.o$JOB_ID # Name of the output file #$ -pe 12way 24 # Requests 12 tasks/node, 24 cores total #$ -q normal # Queue name "normal" #$ -l h_rt=01:30:00 # Run time (hh:mm:ss) hours ibrun./a.out # Run the MPI executable named "a.out"

28 Submitting/manipulating batch jobs Submit the script that you have created: Actual commands are machine specific, but they follow general principles. qsub jobname qstat a qstat -u username qdel jobid man qsub 28

29 Managing Your Environment: Modules Allows you to manipulate your environment. module list shows currently loaded modules. module avail shows available modules. module show <name> describes module. % module load gcc/3.1.1 % which gcc /usr/local/gcc/3.1.1/linux/bin/gcc % module switch gcc/3.1.1 gcc/3.2.0 % which gcc /usr/local/gcc/3.2.0/linux/bin/gcc % module unload gcc % which gcc gcc not found 29

30 File Transfers: Small Files (<2GB) To transfer small files between XSEDE Resources and/or your own workstation you can use scp or sftp. From Linux or Mac, you can run these commands directly from the terminal From Windows, use your ssh client to do this (putty has free downloads for these tools, too! just Google putty sftp ). These are easy to use and secure, but provide poor performance for large files. 30

31 File Transfer: User Portal Log into the XSEDE User Portal Select Resources tab Select File Manager tab (now wait for Java Applet to load) May need to allow access for applet to run by clicking OK. You will see a list of all machines. This includes: your local machine. XSEDE$Share: 2GB of space to collaborate. Allows you to share files with your collaborators. 31

32 Transferring Large Files with User Portal For large file transfers, we need to set a few parameters. Before clicking on the resource, Right click on the resource you re going to transfer data from and select Edit. This will bring up the file transfer parameters: Click the checkbox next to Stripe Transfers - Click OK Repeat for the other panel using the destination resource Repeat this every time you change Resources Drag and drop the file from source to destination to transfer. 32

33 Large Files Command Line Transfers Within XSEDE, you can use the command line to transfer large files with uberftp or globus-url-copy. Example: From PSC Blacklight to TACC Ranger : Optimized for large files globus url copy -stripe tcp bs gsiftp:// gridftp.blacklight.psc.teragrid.org/scratcha/joeuser/file gsiftp:// gridftp.ranger.tacc.teragrid.org/scratch/joeuser Look here for names of gridtftp servers at each site: support/transfer_location speedpage.psc.edu: provides information on transfer speeds you can expect using globus-url-copy with the optimized parameters above. 33

34 What is Globus Online? Initial implementation of XSEDE User Access Services (XUAS) Reliable data movement service High performance: Move terabytes of data in thousands of files Automatic fault recovery Across multiple security domains Designed for researchers Easy fire and forget file transfers No client software installation New features automatically available Consolidated support and troubleshooting Works with existing GridFTP servers Ability to move files to any machine (even your laptop) with ease "We have been using Globus Online to move files to a TeraGrid cluster where we analyze and store tens of terabytes of data... I plan to continue using GO to access these resources within XSEDE to easily get my files where they need to go. -- University of Washington user The service is reliable and easy to use, and I look forward to continuing to use it with XSEDE. I've also used the Globus Connect feature to move files from TeraGrid sites to other machines -- this is a very useful feature which I'm sure XSEDE users will want to take advantage of. -- NCSA user 34

35 ECSS Extended Collaborative Support Services Expertise available in a wide range of areas Performance analysis Petascale optimization Gateways and web portals Specialized scientific software Can solicit ECSS support at any time though the Allocations tab in the XSEDE User Portal Requires written justification and a project plan Inquire at help@xsede.org 35

36 ECSS can include Porting applications to new resources Providing help for portal and gateway development Implementing algorithmic enhancements Implementing parallel math libraries Improving scalability of codes to higher processor counts Optimizing codes to efficiently utilize specific resources Assisting with visualization, workflow, data analysis, and data transfer 36

37 Questions? Need Help? First, try searching the knowledge base or other documentation Next, submit a ticket portal.xsede.org -> My XSEDE -> Tickets Send help@xsede.org Or call the Help Desk

38 Need more training? portal.xsede.org -> Training Course Calendar On-line training 38

39 October 26, 2012 Selecting the right XSEDE resource Dan Stanzione Deputy Director, TACC

40 Overview What kinds of architectures are out there? Will my program run (well) on them? If not, what can I do about it? What are the current XSEDE systems? What s coming down the pipe?

41 Parallel Computing Architectures Shared Memory - threads Distributed Memory tasks Something in between (most everything).

42 Shared Memory Parallelism Processors have access to the same memory Limited number of processors Fast communication via memory access

43 Shared Memory Model Tasks share a common address space, which they read and write asynchronously. Advantage: no need for explicit data exchange between tasks. Disadvantage: understanding performance and managing data locality become more difficult. Implementation The native compiler translates user program variables into actual memory addresses, which are global. No common distributed memory platform implementation currently exist.

44 Distributed Memory Parallelism Processors do not have access to each other s memory Many processors available (Large scale parallelism possible) Communication speed determined by network (switch)

45 Distributed Memory Model Advantages Easy to scale with number of processors No need to maintain cache coherency Faster local access to memory No interference or overhead Cost effective; can use commodity, off-the-shelf processors and networking Disadvantages Programmer is responsible for data exchange between processors. Non-uniform memory access (NUMA) times Difficulty mapping some data structures to this memory architecture.

46 Parallel Programming Models Shared Memory: Create *threads* primarily with OpenMP (or Pthreads). Distributed Memory: Create *tasks* primarily with MPI. Use a Single Program Multiple Data model. None of this applies to accelerators, exactly, we will get back to that. All of this assumes that C/C++ and Fortran are the only languages in the universe

47 What if my application is not C/C++ or Fortran? Your mileage will definitely vary. In response to the Java question, very senior architects at very large chip makers answer I thought we were talking about performance. Same kinds of answers for Python, and they get worse for less mainstream things. Some systems will not support these at all: i.e. Cray systems have no Java runtime on compute nodes (Kraken, Blue Waters). Most will support them at some level, but you will have difficulty getting maximum performance. If you just need a high througput system, you may well not care!

48 Accelerators and Co-Processors Architectures are becoming much more heterogeneous, particularly with the wide use of: GPU (nvidia, AMD Fusion) Intel MIC (Many Integrated Core) FPGA (to a lesser extent, e.g. Convey). Some custom networks.

49 Accelerators and Co-Processors The biggest problem with the new accelerated models is the lack of standard programming models. No existing MPI/OpenMP codes will work out of the box. A few have been ported to one system or the other. Options: CUDA (most stable, most popular): nvidia extensions to C to support a streaming model. (C-only; PGI provides a subset in Fortran) OpenCL: An attempt at a standard for offload extensions (C-only) OpenACC: A fork off of OpenMP to do offload like things for OpenMP (much to their dismay). All of these models do two things: Partition your code between a piece that runs on the host, and a piece that runs on the accelerator. Express your accelerated code in a highly parallel, long vector, streaming model to make memory access efficient.

50 XSEDE Resource Picking Rules of Thumb. Do I want to write code? (If yes, you have more options). Is my code parallel? (What language is it in? Does it have parallel support?). YES: Do I need shared memory or distributed memory?

51 XSEDE Resource Picking Rules of Thumb. Does it use MPI? --Probably Distributed memory systems. Does it use OpenMP Probably shared memory systems. If both, probably distributed memory, but: How much memory do I need per task? (If more than ~32GB, back to shared memory). This is per task total memory can be 200TB; this is an important distinction! Does it support CUDA, or anything else custom? (Then you can think about accelerators ).

52 XSEDE Resource Picking Rules of Thumb. NO, my code isn t parallel Do you want to make it parallel? This may be a big undertaking Do you have to do a lot of runs? If no, you may be in the wrong place. If yes How big is one run? If you can fit a single run in 24-32GB memory, and hours, you may be able to use a distributed memory resource in high throughput mode. If more RAM than that, we have a few high memory resources. But remember, running a non-parallel code, on, say Blacklight to get 2TB of RAM, means leaving 511 empty cores to on a $4M taxpayer machine to support you only at 0.2% of peak at best!!!

53 Current XSEDE Resources Compute (Shared and Distributed Memory) Visualization Storage Special Purpose

54 Ranger: World Class Supercomputing Capability!

55 Ranger System Summary Peak Performance Teraflops 3,936 Sun four-socket blades 15,744 AMD Barcelona processors Quad-core, four flops/clock cycle Total Memory Terabytes 2 GB/core, 32 GB/node 123 TB/s aggregate bandwidth Interconnect 1 GB/s, sec latency Sun Data Center Switches (2), InfiniBand, up to x ports each 7.8 TB/s backplane Ranger will be decommissioned in 2013 (after more than 3 million jobs for more than 5,000 users).

56 Kraken The next of the Track 2 systems, awarded the year after Ranger. Oak Ridge/NICS So, newer, faster cores, even more peak performance (~1PF); still the largest system in XSEDE! Cray, with AMD processors. Single socket nodes >100,000 total cores. Cray Linux on nodes; really good for C/Fortran MPI codes, less support for much else.

57 Forge The first XSEDE resource targeted exclusively at production GPU computing. ~150TF peak, 4TB total RAM 36 nodes, 16 cores of AMD per node, plus 6 nvidia 2070GPUs per node (216 total). Really relies on CUDA apps! Leaves XSEDE pool September 1 (Keeneland will bring new GPU capability).

58 Queen Bee/Steele LSU/Purdue Older, but still solid, distributed memory Linux clusters. 50TF(QB)/60TF (Steele) of Intel Quad core processors (~5k/6k processor cores) Steele has a GigE connection; throughput or coarse-grain parallel implementations. Queen Bee has an Infiniband interconnect (like Ranger), better for MPI jobs.

59 Trestles Appro Linux distributed memory cluster. AMD Magny Cours processors, 324 nodes, 10,368 cores. 100TF peak performance Local flash disks on nodes. QDR infiniband (40Gbps) Focused more on interactive jobs; support for jobs generated by web gateways (shorter max run times and shorter queue waits).

60 Lonestar 4 Dell Intel 64-bit Xeon Linux Cluster 22,656 CPU cores (302 TFlops) 44 TB memory, 2.4 PB disk

61 Lonestar: The Stats 302 TeraFlops (Trillion floating point operations per second). 22,656 Intel Westmere Processor Ghz in 1,888 Dell Blades. 2GB RAM per core, 24GB per node, 44.3 TB total. 40Gbps QDR Infiniband interconnect in a fully non-blocking fat tree (Mellanox). 1.2Petabytes of disk in parallel filesystem (DataDirect, Lustre)

62 Specialized Subsystems Lonestar is a comprehensive system for science and engineering High Performance Computing High Throughput Computing Visualization subsystem Large memory subsystem GPU/Accelerated Computing Subsystem

63 Blacklight The premiere shared memory resource in XSEDE; 32TB shared RAM. SGI Ultraviolet system 4096 Cores, 37TF peak performance Peak is not as high, but focused on running large shared memory problems that will not run on the big systems. Pittsburgh Supercomputer Center

64 Gordon San Diego Supercomputing Center Focus on Data Intensive Applications Just started production 1024 nodes/16,384 cores of Intel Sandy Bridge 341TF peak performance Dual-rail QDR interconnect Features for data intensive applications: 300TF of Flash storage as a fast cache ScaleMP to provide virtual shared memory of up to 2TB (slower than real shared memory, but much faster than not having it!!!).

65 Nautilus, Longhorn, Spur Visualization/Data Analysis Systems Longhorn; GPUs to be used for the novel purpose of graphics (at TACC). A system for remote visualization; 2048 Intel Nehalem cores (Dell) and 512 nvidia GPUs; 4GB or 12GB per core, QDR infiniband. Nautilus; SGI UV system at NICS (like Blacklight) for larger memory, with 8 nvidia GPUs (less vis than longhorn, more statistical analysis). Spur; the original remote visualization system, 8 high memory nodes with GPUs integrated on Ranger.

66 Condor Pool Purdue University A specialized resource to take advantage of a server farm Focus on throughput computation; large numbers of single node jobs (similar to what you might run on a commercial cloud, e.g. Amazon). ~150TF peak, ~4,500 various speed CPUs, from 0.5GB to 16GB.

67 Storage Systems A variety of large scale storage systems; mostly focused on archive, or swapping data between resources: Lustre-WAN (PSC), Data Capacitor (Indiana); online Lustre storage. Data Replication Service (TACC) HPSS (6PB, NICS), NCSA Tape (10PB, NCSA), Ranch (60PB, TACC) ;; all long term primarily tape archives. New Data SuperCell at PSC (disk-based archive).

68 Coming XSEDE Resources Keeneland Fall12 Stampede End 12/Start of 13 Blue Waters Mid-2012/Start of 13.

69 Keeneland Track 2D Experimental System Georgia Tech Oak Ridge/NICS First system to bring GPU computing to production at large scale. Lots of focus on evolving GPU programming model. Supporting CUDA, OpenCL Developing Ocelot model.

70 Keeneland Architecture KID: Keeneland initial delivery in production now. 7 racks, 200TFLOPS. 3 nvidia 2070 GPUs per node In early user phase now Production system: About triple the peak performance of KID nvidia 2090 GPUs. Full specs available soon

71 Stampede Two petaflop, 100,000+ core Xeon based cluster to be the new workhorse of the NSF/XSEDE open science community Eight additional petaflops, several hundred thousand cores of Intel Many-Integrated- Core processors to provide a revolutionary innovative capability.

72 Stampede: Programming Models A 2PF Xeon-only system (MPI, OpenMP) An 8PF MIC-only system (MPI, OpenMP) A 10PF heterogeneous system (MPI, OpenMP) Run separate MPI tasks on Xeon vs. MIC, or use OpenMP extensions for offload for hybrid programs.

73 Power/Physical Stampede will physically use U cabinets. Power density (after upgrade in 2015) will exceed 40kW per rack. Estimated 2015 peak power is 6.2MW.

74 Blue Waters Not technically an XSEDE Resource, but too big not to talk about! Cray (like a newer, much larger Kraken), but also a large number of GPUs (>3,000). Detailed specs not yet public: ~25,000 nodes AMD processors/nvidia GPUs 4GB per core Lots and lots of I/O 11.5PF peak performance Focus on a handful of full-system, sustained petaflops apps (requiring 100 s of thousands of cores).

75 Thanks for listening, and welcome to XSEDE! 75