Introduc)on to the RCE September 21, 2010 Len Wisniewski

Transcription

1 Introduc)on to the RCE September 21, 2010 Len Wisniewski

2 IQSS technical services Resource support Research Compu)ng Environment (RCE): cluster compu)ng for sta)s)cal research Desktop support Computer lab services Data hos)ng Network storage Web site hos)ng Research Tools (next lunch!) Dataverse: archive of research data sets OpenScholar: templates and hos)ng of personal / project web sites Introduc)on to Research Compu)ng Environment (RCE) 2

3 RCE in detail RCE = Research Compu)ng Environment, remote infrastructure and scalable compu)ng power for research analysis Used to run common sta)s)cal applica)ons R, GAUSS, Mathema)ca, MATLAB, Octave, SAS, S- PLUS, Stata RCE has three types of nodes Login nodes User logs in via NX (similar to VNC) and gets a desktop session User can launch an applica)on directly from the desktop Compute- on- demand nodes 25 compute cores on 9 nodes User has special RCE Powered Applica)ons menu to launch applica)ons on machines with large memory resources (up to 64GB) Batch nodes 572 compute cores on 79 nodes Used typically for non- interac)ve, long- running, scalable jobs Most jobs use R Introduc)on to Research Compu)ng Environment (RCE) 3

4 RCE architecture and configura)on Interac)ve nodes Key Applica+ons User Resource Manager Secure login via NX Allocate and manage resources R GAUSS Mathema)ca MATLAB Octave SAS (COD only) S- PLUS Stata (SE and MP) R GAUSS Mathema)ca MATLAB Octave SAS S- PLUS Stata (SE and MP) Local Batch nodes Remote Introduc)on to Research Compu)ng Environment (RCE) 4

5 RCE Current Metrics 471 ac)ve user accounts 105 HKS, 70 IQSS, 43 HSPH, 30 Gov, 26 GSE 3TB of data over 263 ac)ve project spaces 1.6 million hours of compute )me on batch cluster ac)ve batch users per month Average 50% u)liza)on of batch pool Pool includes 378 processor cores 100+ thousand hours of compute )me on interac)ve cluster ac)ve interac)ve users per month Average 80% u)liza)on of interac)ve pool Extended periods of overflow into batch Pool has grown from 12 to 36 processor cores Introduc)on to Research Compu)ng Environment (RCE) 5

6 Sta)s)cal applica)ons R Gauss Mathema)ca MATLAB Octave SAS (COD and Batch only) S- PLUS Stata (SE and MP) Introduc)on to Research Compu)ng Environment (RCE) 6

7 Produc)vity applica)ons Text editors and news clients Text processing tools Office produc)vity suite (OpenOffice) Graphical and command- line system tools Introduc)on to Research Compu)ng Environment (RCE) 7

8 Documenta)on hmp://support.hmdc.harvard.edu/kb- 14/research_compu)ng_environment Accessing the RCE Connec)ng to the RCE (via NX) Working in the RCE (produc)vity tools) Developing Sooware (CVS and R packaging tools) Using Sta)s)cal Programs Working with {R, GAUSS, Mathema)ca, MATLAB, Octave, SAS, S- PLUS, Stata, RCE Powered} Recent Updates RCE FAQ hmp://support.hmdc.harvard.edu/kb- 13/cluster_compu)ng Working with Batch Servers Working with Interac)ve Servers Cluster Compu)ng FAQ Introduc)on to Research Compu)ng Environment (RCE) 8

9 Future direc)ons Become more adap)ve to resource needs Single flexible pool of resources for batch- interac)ve jobs Leverage scalable resources Understand how users problems scale Understand how users are solving their scalable problems Migrate more users to take advantage of scalable resources Make account renewals and job extensions easier Use cloud resources for elas)cally launching large R jobs Introduc)on to Research Compu)ng Environment (RCE) 9

10 Compute- on- demand node applica)on usage Introduc)on to Research Compu)ng Environment (RCE) 10

11 Batch nodes applica)on usage Introduc)on to Research Compu)ng Environment (RCE) 11

12 RCE batch resource usage Introduc)on to Research Compu)ng Environment (RCE) 12

13 The hybrid model: RCE and the cloud Interac)ve User Resource Manager Local Secure login via NX Allocate and manage resources Batch nodes Key Applica+ons R GAUSS Mathema)ca MATLAB Octave SAS (COD only) S- PLUS Stata (SE and MP) R GAUSS Mathema)ca MATLAB Octave SAS S- PLUS Stata (SE and MP) Remote R Octave Introduc)on to Research Compu)ng Environment (RCE) 13

14 Training and advice RCE training Once a semester half- day HPC course (Star)ng in Oct) Monthly one- hour Intro to RCE for new users (Star)ng in Spring) Monthly one- hour Advanced RCE sessions Cool tricks and solu)ons by IQSS staff and users Online materials for Intro to RCE and various common advanced techniques Advice from you, the expert users E- mail me directly (lwisniewski@iq.harvard.edu) or stop by K316 Discussions with Bob Kinney, Mam Cox, and the Core staff Send in feature requests via RT Introduc)on to Research Compu)ng Environment (RCE) 14

15 Ques)ons? Any ques)ons, e- mail Web sites: RT (Request Tracker): help.hmdc.harvard.edu IQSS IT support: support.hmdc.harvard.edu Introduc)on to Research Compu)ng Environment (RCE) 15