Infrastructure Underpinnings of the GFDL Workflow

Transcription

1 Infrastructure Underpinnings of the GFDL Workflow A Trio Of Lightning Talks For ISENES2 Chandin Wilson, Engility Amy Langenhorst, NOAA Karen Paffendorf, Princeton University Erik Mason, Engility Jeffrey Durachta, Engility V. Balaji, Princeton University National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory Princeton, NJ

2 3 topics in 25 minutes From Silos to Enclave with Certificates General Copy Program FMS FRE Chaco rewrite

3 Certificate Based Authentication Without User Torture SILOS TO ENCLAVE

4 Early 2000s

5 Transition Foundations Single HPCS procurement Unify user namespace Standardize authentication mechanism

6 NOAA RDHPCS Enclave Bastion GFDL HPCS (Princeton, NJ) Users Post Processing Storage Nwave Zeus HPCS (Fairmont, WV) Compute Storage Bastion Jet HPCS (Boulder, CO) Users Compute Gaea HPCS (Oakridge, TN) Compute Bastion

7 Enclave Logical distinction between HPCS and user workstations Clear security demarcation points Unified login for all HPCS access Redundant access Multiple private 10Gbit links

8 Certificates X509 User certificates Issued at Bastions for authentication inside enclave Limited lifetime GSI based implementation LDAP backend storage Distributed administration and signing

9 User Experience A few extra steps upon initial login An additional passphrase to remember Identical ways to access all HPCS Unattended high speed data transfers just work All certificate enabled actions just work without further ado on the HPCS

10 Admin Experience 99% of all administration done by simple TUI tool accessible via ssh Distributed across centers; site outages do not affect operations Typical daily action is simply signing certificates as notified.

11 Implementation Internal x509 root Certificate Authority Bastion scripts handle all certificate requests, retrieval, and proxy generation upon user login GSI-SSHD and Globus Toolkit on all HPCS nodes Gridftp on all Data Transfer Nodes

12 Further Details Implementing a Certificate Based Authentication Infrastructure Without User Torture LOPSA 2013 Presentation

13 Implementing Cloud Transfer for HPC: Abstracting the Network for Scientific Data Movement GENERAL COPY PROGRAM

14 A Copy Wrapper The copy tool to use in the enclave Simple scp -like usage Hides all the details about actual copy program and transfer nodes used Reliable and high throughput using dedicated transfer nodes Centralized logging

15 Motivation Distributed Computing Users Toolsets Filesystems Connectivity Reliability Maintainability

16 Tool evolution Predates NOAA RDHPCS Enclave External collaborators wanted simple way to pull or push data to GFDL Improving Cluster Filesystem copies Lustre, CXFS, have specific best tools HPCS installs and upgrades change filesystem characteristics Exploring WAN transfer options

17 GCP 1 Perl, non-oo design Flexible, but not configurable Excellent proving ground for ideas. Understood two or three sites Predates DTN infrastructure Mutated hydra after 24 months

18 GCP 2 Design Perl, OO concepts Driver, Agent, Transport, Method design Configuration abstracted Behavior changes based on invoking node, source and destination filesystem Logging centralized and flexible Able to implement per-user trace logging Extensive retries and timeouts on all operations

19 Data Transfer Nodes, 1 Cluster filesystems Fast and expensive: not usually attached to batch nodes NFS serving introduces a layer FTP-like copies between batch nodes and cluster filesystem nodes perform fastest

20 Data Transfer Nodes, 2 Workstation (local) filesystems Isolated from HPCS, but transfer access needed. Served via NFS from workstations Remote filesystems WAN transfers introduce latency and bandwidth considerations different than LAN transfers.

21 DTN Graphic

22 GCP 2 Features Supports five sites, inside and outside the enclave Handles a dozen filesystems across nine node types Easily expandable for additional sites, filesystems, nodes, transports, and agents.

23 GCP Testing and Production Unit and end-to-end tests run hourly via Jenkins Production test runs validate functionality of entire transfer infrastructure and most of filesystems

24 GCP Logs Extensive tunable logging via Log4perl Syslog as the transport All transfers logged with unique session identifier

25 GCP Log example May 29 23:57:56 pp045 gcp_info[10824]: ts= t23:57:56.561; uid=383b1edb b5b-b8a a291e0d; p=info;where=gcp::driver::log_end.1414;event=gov.noaa.rdhpcs.gcp.end;transfer _size= ;dtn_d estination=pnfs06- ha.princeton.rdhpcs.noaa.gov;file_count=1;status=0;prog=/usr/local/gcp/2.2/gcp;gcp _version=2.2.1;node=pp045.princeton.rdhpcs.noaa.gov;transport_count=1;level=inf o;transfer_time=10;pid=1082 4;error=none;user=Bill.Hurlin;gcp_call=/usr/local/gcp/2.2/gcp \-v mu_di\.nc \/archive\/bill\.hurlin\/siena_201305\/esm2m_pi\- control_c2_14jun2013_t_flux_3\/gfdl\.ncrc2\-intel\-prod\-openmp\/pp\/ocean_topaz_ misc\/ts\/monthly\/5yr\/ocean_topaz_misc\ \ \.mu_di\.nc;

26 Act the same, but be all different FMS FRE CHACO REWRITE

27 FMS FRE Chaco Rewrite of FRE, starting with post processing Maintain compatibility and historical behavior, yet standardize tool behavior Old (user) interfaces available, e.g. drop in replacement where possible Improve visibility and control of experiments

28 FRE Workflows Two component time average and time series task flow, year 1 and 2 This test case produces several postprocessing products on atmos variables that have been regridded onto pressure levels (zinterp). Chunk lengths and averaging intervals are 3 and 6 years, so only intermediate products are stored at year 1 & 2.

29 FRE Workflows Two component time average and time series task flow, year 3 Chunk lengths and averaging intervals are 3 and 6 years: store 3 yr product in archive, and stage for 6yr products.

30 Chaco Goals Remove year boundary for post processing Improve seasonal support Support hires models Better calendar support Modular not monolithic Testable Extensible

31 Chaco Features Parallel actions where possible Data and tool run inventory Where did this file come from? What tool created it? How long did the tool run? Logging Log4perl framework Centralized logging easy implemented

32 Chaco Tech Perl OO via Moose and friends Path to Perl 6 CPAN all things Test driven development with continuous integration tactics. Strong source code practices, code documentation and project management tools to enable the team.

33 Chaco Cylc Cylc for the dependency and scheduling engine Cylc suites generated by FRE Chaco Allows for different grouping of tasks based on arguments, XML, or other factors. User accessible / modifiable, but hidden Leverage Cylc task management and job submission Cylc GUI gives visibility and control

34 Chaco Tool Outline Modularize the monolithic workflow Standardize interface fre CATEGORY ACTION [OPTIONS] Can operate independent of workflow On-disk data structure (FRED) stashes parsed experiment details File inventory and tool run databases Tools know input and output files

35 Chaco Tools Cylc preparation and running Data movement get, store, stage Transformations Split into month and variable shards xyinterp, zinterp time series, time averages

36 Informative Chaco Tools, cont. Experiment configuration and run status Files produced By what tool, when Tools run Files consumed and produced Output Timings

37 Chaco Perl perlbrewed Perl-5.18, Moose, Err, Try::Tiny, etc The path to perl 6 DateTime Log4perl MCE MLDBM

38 Das Ende