Production Petascale Climate Data Replication at NCI Lustre and our engagement with the Earth Systems Grid Federation (ESGF)
|
|
- Abigayle Thompson
- 5 years ago
- Views:
Transcription
1 Joseph Antony, Andrew Howard, Jason Andrade, Ben Evans, Claire Trenham, Jingbo Wang Production Petascale Climate Data Replication at NCI Lustre and our engagement with the Earth Systems Grid Federation
2 MOTIVATION
3 International Climate Change Research The CMIP projects The UN s International Panel on Climate Change (IPCC) prepares an intergovernmental assessment report every 6 years This effort requires significant scientific and HPC/HPD resources to back it The most recent of these activities was the Coupled Model Intercomparison Project 5 (CMIP5) The NCI is a major data node within the ESGF federation In this talk I will share with you a view from the coalface, replicating ~2PB of data
4
5 CMIP DATA VOLUMES
6 CMIP1 thru CMIP5 Data Volumes Taken from Dean Williams ESGF Internet2 presentation, 2014
7 ESGF NODE ARCHITECTURE
8 The ESGF Data Archival and Retrieval System The ESGF is a federated peer-to-peer international data archival and retrieval system Incorporates singlesign-on for end-users It has publication and version management tools Supports data aggregations and can notify users if datasets have been modified
9 THE END-USER PERSPECTIVE
10 The Last-Mile Problem Data is too large to move onto desktop for analysis CMIP3 to CMIP5 Users want versioned, curated data to be able to jump right into scientific analysis At NCI An integrated eco-system exists for dataintensive science Data Repositories Virtual Laboratories The ICNWG effort to solve the Last Mile Problem for networking
11 ICNWG Activities
12 Okay so where s Lustre in all of this you ask?
13 Okay so where s Lustre in all of this you ask? We use Lustre as our distributed filesystem for a set of dedicated WAN data transfer nodes (DTNs)
14 Okay so where s Lustre in all of this you ask? We use Lustre as our distributed filesystem for a set of dedicated WAN data transfer nodes (DTNs) But first a detour
15 1Gbps == 125 MB/sec Courtesy Eli Dart, ESnet
16 Courtesy Eli Dart, ESnet
17 Courtesy Eli Dart, ESnet
18 Courtesy Eli Dart, ESnet
19 Courtesy Eli Dart, ESnet
20
21 AARNet International Links
22 NCI s DTN Nodes
23 CBR-SYD and onto the CONUS via SXtransport
24 SXtransport Physical Layout Cable Station Network Segment
25 SXtransport Logical Network Layout
26 What are some of the world s longest submarine cables you ask? 39,000 Km of submarine fibre
27 What are some of the world s longest submarine cables you ask? 39,000 Km of submarine fibre 28,900 Km of submarine fibre 1,600 Km of terrestrial fibre
28 Networking Topology for Data Replication Courtesy Mary Hester, ESnet
29 Initial Transfer Rates from NCI Graph shows the data rate vs. the volume of data transferred Different lines in the graph represent how many data streams were required to obtain the given performance. The results of the graph indicate that it is possible to get a line-rate of 1GB/s (8Gbps) between Australia and the United States, however, it requires configuring transfers to run more than 100 parallel streams
30 Data replication and Science DMZs Currently we ve replicated ~1.5PB Working on improving these rates by employing a Science DMZ model and dedicated data transfer nodes
31 Globus Online Globus Online is a hosted data-transfer-asa-service offering, run by the University of Chicago It makes the job of large data transfers easy for both instrument owners and end-users
32 Globus Online Architecture
33
34
35
36
37 Using Dedicated DTNs January 2015
38 Using Dedicated DTNs March 2015
39 State of the Union Numbers from the ICNWG Consortium
40 Conclusion Non-trivial to get various ducks lined-up 10GigE WAN networking Mellanox tuning work for 10GigE Ethernet and 56Gbp FDR Being NUMA aware is critical for the GridFTP daemon!
41 THE END
42 VERIFIED, CURATED SCIENTIFIC DATASETS
43 Centralized Quality Control for Data Processing Multi-layered QC Initial Level 1 QC done at data nodes DKRZ performs L2 QC Further metadata and variable checking is done to get to L3 QC At every step, end-users can see the QC Level for their data Replicated data has passed QC Level 3 and receives a DOI
International Climate Network Working Group (ICNWG) Meeting
International Climate Network Working Group (ICNWG) Meeting Eli Dart ESnet Science Engagement Lawrence Berkeley National Laboratory Workshop on Improving Data Mobility & Management for International Climate
More informationClimate Science s Globally Distributed Infrastructure
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. Climate Science s Globally Distributed Infrastructure
More informationIdentifier Infrastructure Usage for Global Climate Reporting
Identifier Infrastructure Usage for Global Climate Reporting IoT Week 2017, Geneva Tobias Weigel Deutsches Klimarechenzentrum (DKRZ) World Data Center for Climate (WDCC) Scientific driver: Global climate
More informationSLIDE 1 - COPYRIGHT 2015 ELEPHANT FLOWS IN THE ROOM: SCIENCEDMZ NATIONALLY DISTRIBUTED
SLIDE 1 - COPYRIGHT 2015 ELEPHANT FLOWS IN THE ROOM: SCIENCEDMZ NATIONALLY DISTRIBUTED SLIDE 2 - COPYRIGHT 2015 Do you know what your campus network is actually capable of? (i.e. have you addressed your
More informationLawrence Berkeley National Laboratory Lawrence Berkeley National Laboratory
Lawrence Berkeley National Laboratory Lawrence Berkeley National Laboratory Title Scaling the Earth System Grid to 100Gbps Networks Permalink https://escholarship.org/uc/item/80n7w3tw Author Balman, Mehmet
More informationThe NCI High Performance Computing (HPC) and High Performance Data (HPD) Platform to Support the Analysis of Petascale Environmental Data Collections
ESSI 2015-8273 The NCI High Performance Computing (HPC) and High Performance Data (HPD) Platform to Support the Analysis of Petascale Environmental Data Collections Ben Evans 1, Lesley Wyborn 1, Tim Pugh
More informationCMIP5 Datenmanagement erste Erfahrungen
CMIP5 Datenmanagement erste Erfahrungen Dr. Michael Lautenschlager Deutsches Klimarechenzentrum Helmholtz Open Access Webinare zu Forschungsdaten Webinar 18-17.01./28.01.14 CMIP5 Protocol +Timeline Taylor
More informationClare Richards, Benjamin Evans, Kate Snow, Chris Allen, Jingbo Wang, Kelsey A Druken, Sean Pringle, Jon Smillie and Matt Nethery. nci.org.
The important role of HPC and data-intensive infrastructure facilities in supporting a diversity of Virtual Research Environments (VREs): working with Climate Clare Richards, Benjamin Evans, Kate Snow,
More informationEngagement With Scientific Facilities
Engagement With Scientific Facilities Eli Dart, Network Engineer ESnet Science Engagement Lawrence Berkeley National Laboratory Global Science Engagement Panel Internet2 Technology Exchange San Francisco,
More informationClimate Data Management using Globus
Climate Data Management using Globus Computation Institute Rachana Ananthakrishnan (ranantha@uchicago.edu) Data Management Challenges Transfers often take longer than expected based on available network
More informationBuilding a Global Data Federation for Climate Change Science The Earth System Grid (ESG) and International Partners
Building a Global Data Federation for Climate Change Science The Earth System Grid (ESG) and International Partners 24th Forum ORAP Cite Scientifique; Lille, France March 26, 2009 Don Middleton National
More informationInternational Big Science Coming to Your Campus Soon (Sooner Than You Think )
International Big Science Coming to Your Campus Soon (Sooner Than You Think ) Lauren Rotman ESnet Science Engagement Group Lead April 7, 2014 ESnet Supports DOE Office of Science Office of Science provides
More informationImplementing a Data Quality Strategy to simplify access to data
IN43D-07 AGU Fall Meeting 2016 Implementing a Quality Strategy to simplify access to data Kelsey Druken, Claire Trenham, Ben Evans, Clare Richards, Jingbo Wang, & Lesley Wyborn National Computational Infrastructure,
More informationEnhancing Infrastructure: Success Stories
Enhancing Infrastructure: Success Stories Eli Dart, Network Engineer ESnet Network Engineering Group Joint Techs, Winter 2012 Baton Rouge, LA January 24, 2012 Outline Motivation for strategic investments
More informationIndex Introduction Setting up an account Searching and accessing Download Advanced features
ESGF Earth System Grid Federation Tutorial Index Introduction Setting up an account Searching and accessing Download Advanced features Index Introduction IT Challenges of Climate Change Research ESGF Introduction
More informationIntroduction to Grid Computing
Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able
More informationThe Changing Role of Data Stewardship in Creating Trustworthy, Transdisciplinary High Performance Data Platforms for the Future
AGU Fall Meeting 2016 IN31-G The Changing Role of Data Stewardship in Creating Trustworthy, Transdisciplinary High Performance Data Platforms for the Future Clare Richards, Ben Evans, Lesley Wyborn, Jingbo
More informationThe Science DMZ: Evolution
The Science DMZ: Evolution Eli Dart, ESnet CC-NIE PI Meeting Washington, DC May 1, 2014 Why Are We Doing This? It s good to build high-quality infrastructure As network engineers, we like building networks
More informationDesign patterns for data-driven research acceleration
Design patterns for data-driven research acceleration Rachana Ananthakrishnan, Kyle Chard, and Ian Foster The University of Chicago and Argonne National Laboratory Contact: rachana@globus.org Introduction
More informationData Management Components for a Research Data Archive
Data Management Components for a Research Data Archive Steven Worley and Bob Dattore Scientific Computing Division Computational and Information Systems Laboratory National Center for Atmospheric Research
More informationData Issues for next generation HPC
Data Issues for next generation HPC Bryan Lawrence National Centre for Atmospheric Science National Centre for Earth Observation Rutherford Appleton Laboratory Caveats: Due to time, discussion is limited
More informationUniform Resource Locator Wide Area Network World Climate Research Programme Coupled Model Intercomparison
Glossary API Application Programming Interface AR5 IPCC Assessment Report 4 ASCII American Standard Code for Information Interchange BUFR Binary Universal Form for the Representation of meteorological
More informationChallenges of Big Data Movement in support of the ESA Copernicus program and global research collaborations
APAN Cloud WG Challenges of Big Data Movement in support of the ESA Copernicus program and global research collaborations Lift off NCI and Copernicus The National Computational Infrastructure (NCI) in
More informationBigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer. BigData Express Research Team November 10, 2018
BigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer BigData Express Research Team November 10, 2018 Many people s hard work FNAL: ESnet: icair/starlight: KISTI: Qiming
More informationImplementing a Data Quality Strategy to simplify access to data
Implementing a Quality Strategy to simplify access to data Kelsey Druken Implementing a Quality Strategy to simplify access to data Kelsey Druken, Claire Trenham, Lesley Wyborn, Ben Evans National Computational
More informationShort Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy
Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy François Tessier, Venkatram Vishwanath Argonne National Laboratory, USA July 19,
More informationCOMPUTE CANADA GLOBUS PORTAL
COMPUTE CANADA GLOBUS PORTAL Fast, user-friendly data transfer and sharing Jason Hlady University of Saskatchewan WestGrid / Compute Canada February 4, 2015 Why Globus? I need to easily, quickly, and reliably
More informationDiscovery, Unconstrained by Geography
Discovery, Unconstrained by Geography ACAT 2016 Valparaiso, Chile January 21, 2016 Gregory Bell, Ph.D. Director, Energy Sciences Network (ESnet) Director, ScienKfic Networking Division Lawrence Berkeley
More informationZhengyang Liu University of Virginia. Oct 29, 2012
SDCI Net: Collaborative Research: An integrated study of datacenter networking and 100 GigE wide-area networking in support of distributed scientific computing Zhengyang Liu University of Virginia Oct
More informationData near processing support for climate data analysis. Stephan Kindermann, Carsten Ehbrecht Deutsches Klimarechenzentrum (DKRZ)
Data near processing support for climate data analysis Stephan Kindermann, Carsten Ehbrecht Deutsches Klimarechenzentrum (DKRZ) Overview Background / Motivation Climate community data infrastructure Data
More informationTuning I/O Performance for Data Intensive Computing. Nicholas J. Wright. lbl.gov
Tuning I/O Performance for Data Intensive Computing. Nicholas J. Wright njwright @ lbl.gov NERSC- National Energy Research Scientific Computing Center Mission: Accelerate the pace of scientific discovery
More informationInfraStructure for the European Network for Earth System modelling. From «IS-ENES» to IS-ENES2
InfraStructure for the European Network for Earth System modelling From «IS-ENES» to IS-ENES2 Sylvie JOUSSAUME, CNRS, Institut Pierre Simon Laplace, Coordinator ENES European Network for Earth System modelling
More informationModeling groups and Data Center Requirements. Session s Keynote. Sébastien Denvil, CNRS, Institut Pierre Simon Laplace (IPSL)
Modeling groups and Data Center Requirements. Session s Keynote. Sébastien Denvil, CNRS, Institut Pierre Simon Laplace (IPSL) Outline Major constraints (requirements' DNA) Modeling center requirements/constraints
More informationTHE GLOBUS PROJECT. White Paper. GridFTP. Universal Data Transfer for the Grid
THE GLOBUS PROJECT White Paper GridFTP Universal Data Transfer for the Grid WHITE PAPER GridFTP Universal Data Transfer for the Grid September 5, 2000 Copyright 2000, The University of Chicago and The
More informationThe CEDA Archive: Data, Services and Infrastructure
The CEDA Archive: Data, Services and Infrastructure Kevin Marsh Centre for Environmental Data Archival (CEDA) www.ceda.ac.uk with thanks to V. Bennett, P. Kershaw, S. Donegan and the rest of the CEDA Team
More informationIntro to CMIP, the WHOI CMIP5 community server, and planning for CMIP6
Intro to CMIP, the WHOI CMIP5 community server, and planning for CMIP6 Caroline Ummenhofer, PO Overview - Background on IPCC & CMIP - WHOI CMIP5 server - Available model output - How to access files -
More informationThe Earth System Grid Federation: Delivering globally accessible petascale data for CMIP5
Proceedings of the Asia-Pacific Advanced Network 2011 v. 32, p. 121-130. The Earth System Grid Federation: Delivering globally accessible petascale data for CMIP5 Dean N. Williams 1, Bryan N. Lawrence
More informationChallenges and Solutions for Future Modeling Data Analysis Systems
Challenges and Solutions for Future Modeling Data Analysis Systems Tsengdar Lee tsengdar.j.lee@nasa.gov NASA Headquarters Dan Duffy, NASA GSFC Seungwon Lee, JPL Rama Nemani, NASA ARC Duane Waliser, JPL
More informationLong Term Data Preservation for CDF at INFN-CNAF
Long Term Data Preservation for CDF at INFN-CNAF S. Amerio 1, L. Chiarelli 2, L. dell Agnello 3, D. De Girolamo 3, D. Gregori 3, M. Pezzi 3, A. Prosperini 3, P. Ricci 3, F. Rosso 3, and S. Zani 3 1 University
More informationData Intensive Science Impact on Networks
Data Intensive Science Impact on Networks Eli Dart, Network Engineer ESnet Network Engineering g Group IEEE Bandwidth Assessment Ad Hoc December 13, 2011 Outline Data intensive science examples Collaboration
More informationProgrammable Information Highway (with no Traffic Jams)
Programmable Information Highway (with no Traffic Jams) Inder Monga Energy Sciences Network Scientific Networking Division Lawrence Berkeley National Lab Exponential Growth ESnet Accepted Traffic: Jan
More informationAn NDN Testbed for Large-scale Scientific Data
An NDN Testbed for Large-scale Scientific Data Huhnkuk Lim Korea Institute of Science & Technology Information (KISTI) NDNComm 2015 Sep. 28, 2015 Motivations on NDN for Large-scale Scientific Application
More informationPortfolio of Services. NATIONAL COMPUTATIONAL Portfolio INFRASTRUCTURE
Portfolio of Services NATIONAL COMPUTATIONAL Portfolio INFRASTRUCTURE of Services 1 National Computational Infrastructure The Australian National University 143 Ward Road Acton ACT 2601 T +61 2 6125 9800
More informationAchieving the Science DMZ
Achieving the Science DMZ Eli Dart, Network Engineer ESnet Network Engineering Group Joint Techs, Winter 2012 Baton Rouge, LA January 22, 2012 Outline of the Day Motivation Services Overview Science DMZ
More informationZhengyang Liu! Oct 25, Supported by NSF Grant OCI
SDCI Net: Collaborative Research: An integrated study of datacenter networking and 100 GigE wide-area networking in support of distributed scientific computing Zhengyang Liu! Oct 25, 2013 Supported by
More informationFile Transfer: Basics and Best Practices. Joon Kim. Ph.D. PICSciE. Research Computing 09/07/2018
File Transfer: Basics and Best Practices Joon Kim. Ph.D. PICSciE Research Computing Workshop @Chemistry 09/07/2018 Our goal today Learn about data transfer basics Pick the right tool for your job Know
More informationNetwork and Host Design to Facilitate High Performance Data Transfer
Network and Host Design to Facilitate High Performance Data Transfer Jason Zurawski - ESnet Engineering & Outreach engage@es.net globusworld 2014 April 15 th 2014 With contributions from S. Balasubramanian,
More informationResearch Cyberinfrastructure Upgrade Proposal - CITI
10/02/2015 Research Cyberinfrastructure Upgrade Proposal - CITI Bill Labate, Director Research Technology Group RCI Upgrade Executive Summary REQUEST Support for the funding request for upgrades to UCLA
More informationThe Future of ESGF. in the context of ENES Strategy
The Future of ESGF in the context of ENES Strategy With a subtext of the important role of IS-ENES2 In addressing solutions to the following question: Two thirds of data written is never read! WHY NOT?
More informationBUCKNELL S SCIENCE DMZ
BUCKNELL S SCIENCE #Bisonet Param Bedi VP for Library and Information Technology Principal Investigator Initial Science Design Process Involving Bucknell faculty researchers Library and Information Technology
More informationRESEARCH DATA DEPOT AT PURDUE UNIVERSITY
Preston Smith Director of Research Services RESEARCH DATA DEPOT AT PURDUE UNIVERSITY May 18, 2016 HTCONDOR WEEK 2016 Ran into Miron at a workshop recently.. Talked about data and the challenges of providing
More informationPacific Wave: Building an SDN Exchange
Pacific Wave: Building an SDN Exchange Will Black, CENIC - Pacific Wave Internet2 TechExchange San Francisco, CA Pacific Wave: Overview Joint project between CENIC and PNWGP Open Exchange supporting both
More informationJoint DOE, NASA, NOAA, NSF, IS-ENES, and ANU/NCI Conference
Partnerships for development of next-generation software for distributed access and analysis of simulated, observed, and reanalysis data from the climate and weather communities. Page 1 of 6 Registration:
More informationComputer Science Section. Computational and Information Systems Laboratory National Center for Atmospheric Research
Computer Science Section Computational and Information Systems Laboratory National Center for Atmospheric Research My work in the context of TDD/CSS/ReSET Polynya new research computing environment Polynya
More informationData Staging: Moving large amounts of data around, and moving it close to compute resources
Data Staging: Moving large amounts of data around, and moving it close to compute resources PRACE advanced training course on Data Staging and Data Movement Helsinki, September 10 th 2013 Claudio Cacciari
More informationTransferring a Petabyte in a Day. Raj Kettimuthu, Zhengchun Liu, David Wheeler, Ian Foster, Katrin Heitmann, Franck Cappello
Transferring a Petabyte in a Day Raj Kettimuthu, Zhengchun Liu, David Wheeler, Ian Foster, Katrin Heitmann, Franck Cappello Huge amount of data from extreme scale simulations and experiments Systems have
More informationCMIP6 Data Citation and Long- Term Archival
CMIP6 Data Citation and Long- Term Archival Authors: Martina Stockhause, Frank Toussaint, Michael Lautenschlager Date: 2015-08- 05 Version: 3bnl (Exec summary and objective modified by Bryan) Version:
More informationSCA19 APRP. Update Andrew Howard - Co-Chair APAN APRP Working Group. nci.org.au
SCA19 APRP Update Andrew Howard - Co-Chair APAN APRP Working Group 1 What is a Research Platform Notable Research Platforms APRP History Participants Activities Overview We live in an age of rapidly expanding
More informationIRNC:RXP SDN / SDX Update
30 September 2016 IRNC:RXP SDN / SDX Update John Hess Darrell Newcomb GLIF16, Miami Pacific Wave: Overview Joint project of CENIC (California regional research and
More informationEuropean and international background
IS-ENES2 General Assembly 11-13th June 2014 Barcelona European and international background Metadata and ES-DOC (including statistical downscaling) Sébastien Denvil, Mark Greenslade, Allyn Treshansky,
More informationBigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer. Fermilab, May 2018
BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer Fermilab, May 2018 BigData Express Research Team FNAL Wenji Wu (PI) Qiming Lu Liang Zhang Amy Jin Sajith Sasidharan
More informationWelcome! Presenters: STFC January 10, 2019
Welcome! Presenters: Vas Vasiliadis vas@uchicago.edu Brendan McCollam bjmc@globus.org STFC January 10, 2019 Agenda Morning topics Introduction to the Globus SaaS Service overview & architecture Demo: A
More informationData Staging: Moving large amounts of data around, and moving it close to compute resources
Data Staging: Moving large amounts of data around, and moving it close to compute resources Digital Preserva-on Advanced Prac--oner Course Glasgow, July 19 th 2013 c.cacciari@cineca.it Definition Starting
More informationReproducibility and Replication in Climate Science
May 9 2018 Reproducibility and Replication in Climate Science Gavin Schmidt, NASA GISS For the National Academies of Sciences, Engineering, and Medicine, Committee on Reproducibility and Replicability
More informationMeta4+CMIP5. Bryan Lawrence, Gerry Devine and many many others
Meta4+CMIP5 Bryan Lawrence, Gerry Devine and many many others, Outline A metadata taxonomy: a place for everyone (and everyone in their place :-) Introduction to Key Metafor Concepts CIM Metafor and IS-ENES
More informationUltraScience Net Update: Network Research Experiments
UltraScience Net Update: Network Research Experiments Nagi Rao, Bill Wing, Susan Hicks, Paul Newman, Steven Carter Oak Ridge National Laboratory raons@ornl.gov https://www.csm.ornl.gov/ultranet February
More informationFile Access Optimization with the Lustre Filesystem at Florida CMS T2
Journal of Physics: Conference Series PAPER OPEN ACCESS File Access Optimization with the Lustre Filesystem at Florida CMS T2 To cite this article: P. Avery et al 215 J. Phys.: Conf. Ser. 664 4228 View
More informationData publication and discovery with Globus
Data publication and discovery with Globus Questions and comments to outreach@globus.org The Globus data publication and discovery services make it easy for institutions and projects to establish collections,
More informationThe Pacific Research Platform (PRP)
The Pacific Research Platform (PRP) John Silvester CENIC International Relations University of Southern California Professor of Electrical Engineering Many slides courtesy of Professor Larry Smarr., CALIT2,
More informationGrid Computing. MCSN - N. Tonellotto - Distributed Enabling Platforms
Grid Computing 1 Resource sharing Elements of Grid Computing - Computers, data, storage, sensors, networks, - Sharing always conditional: issues of trust, policy, negotiation, payment, Coordinated problem
More information400G: Deployment at a National Lab
400G: Deployment at a National Lab Chris Tracy (Esnet) *Jason R. Lee (NERSC) June 30, 2016-1 - Concept - 2 - Concept: Use case This work originally began as a white paper in December 2013, in which Esnet
More informationChapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT.
Chapter 4:- Introduction to Grid and its Evolution Prepared By:- Assistant Professor SVBIT. Overview Background: What is the Grid? Related technologies Grid applications Communities Grid Tools Case Studies
More informationOperating two InfiniBand grid clusters over 28 km distance
Operating two InfiniBand grid clusters over 28 km distance Sabine Richling, Steffen Hau, Heinz Kredel, Hans-Günther Kruse IT-Center University of Heidelberg, Germany IT-Center University of Mannheim, Germany
More informationCERA: Database System and Data Model
CERA: Database System and Data Model Michael Lautenschlager Frank Toussaint World Data Center for Climate (M&D/MPIMET, Hamburg) NINTH WORKSHOP ON METEOROLOGICAL OPERATIONAL SYSTEMS ECMWF, Reading/Berks.,
More informationShared File System Requirements for SAS Grid Manager. Table Talk #1546 Ben Smith / Brian Porter
Shared File System Requirements for SAS Grid Manager Table Talk #1546 Ben Smith / Brian Porter About the Presenters Main Presenter: Ben Smith, Technical Solutions Architect, IBM smithbe1@us.ibm.com Brian
More informationNetwork Support for Data Intensive Science
Network Support for Data Intensive Science Eli Dart, Network Engineer ESnet Network Engineering Group ARN2 Workshop Washington, DC April 18, 2013 Overview Drivers Sociology Path Forward 4/19/13 2 Exponential
More informationBigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer. Wenji Wu Internet2 Global Summit May 8, 2018
BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer Wenji Wu wenj@fnal.gov Internet2 Global Summit May 8, 2018 BigData Express Funded by DOE s office of Advanced Scientific
More informationData Management. Parallel Filesystems. Dr David Henty HPC Training and Support
Data Management Dr David Henty HPC Training and Support d.henty@epcc.ed.ac.uk +44 131 650 5960 Overview Lecture will cover Why is IO difficult Why is parallel IO even worse Lustre GPFS Performance on ARCHER
More informationCMIP5 Update. Karl E. Taylor. Program for Climate Model Diagnosis and Intercomparison (PCMDI) Lawrence Livermore National Laboratory
CMIP5 Update Karl E. Taylor Program for Climate Model Diagnosis and Intercomparison () Lawrence Livermore National Laboratory Presented to the WCRP Working Group on Coupled Modelling Hamburg, Germany 24
More informationExArch: Climate analytics on distributed exascale data archives Martin Juckes, V. Balaji, B.N. Lawrence, M. Lautenschlager, S. Denvil, G. Aloisio, P.
ExArch: Climate analytics on distributed exascale data archives Martin Juckes, V. Balaji, B.N. Lawrence, M. Lautenschlager, S. Denvil, G. Aloisio, P. Kushner, D. Waliser, S. Pascoe, A. Stephens, P. Kershaw,
More informationNetworking European Digital Repositories
Networking European Digital Repositories What to Network? Researchers generate knowledge This is going to become an amazing paper I only hope I will be able to access it Knowledge is wrapped in publications
More informationThe Best Defense is a Good Offense Creating Networks That Work (the first time)
The Best Defense is a Good Offense Creating Networks That Work (the first time) Lauren Rotman, ESnet Jennifer Schopf, Indiana University April 24, 2017 Internet2 Global Summit 2017 Presentation Overview
More informationThe Practical Obstacles of Data Transfer: Why researchers still love scp
The Practical Obstacles of Data : Why researchers still love scp Hai Ah Nam Oak Ridge National Laboratory Scientific Computing 1 Bethel Valley Road Oak Ridge, TN 37830 namha@ornl.gov Jason Hill Oak Ridge
More informationHow Five International Networks are Enabling International Data-Intensive Research. Internet2 Global Summit 2014
How Five International Networks are Enabling International Data-Intensive Research Internet2 Global Summit 2014 CONTENTS Brief introduction to EYR and EYR-Global Introduction to 2 selected projects Large
More informationIntroduction to The Storage Resource Broker
http://www.nesc.ac.uk/training http://www.ngs.ac.uk Introduction to The Storage Resource Broker http://www.pparc.ac.uk/ http://www.eu-egee.org/ Policy for re-use This presentation can be re-used for academic
More informationImproving Network Infrastructure to Enable Large Scale Scientific Data Flows and Collaboration (Award # ) Klara Jelinkova Joseph Ghobrial
Improving Network Infrastructure to Enable Large Scale Scientific Data Flows and Collaboration (Award # 1659348) Klara Jelinkova Joseph Ghobrial NSF Campus Cyberinfrastructure PI and Cybersecurity Innovation
More informationC2CAMP. (A Working Title) International Coordination for Science Data Infrastructure: A Symposium 1 Nov 2017
C2CAMP (A Working Title) International Coordination for Science Data Infrastructure: A Symposium 1 Nov 2017 Larry Lannom C2CAMP (Cross-Continental Collection & Management Pilot) Proposed multi-party distributed
More informationShared Parallel Filesystems in Heterogeneous Linux Multi-Cluster Environments
LCI HPC Revolution 2005 26 April 2005 Shared Parallel Filesystems in Heterogeneous Linux Multi-Cluster Environments Matthew Woitaszek matthew.woitaszek@colorado.edu Collaborators Organizations National
More informationBig Data infrastructure and tools in libraries
Line Pouchard, PhD Purdue University Libraries Research Data Group Big Data infrastructure and tools in libraries 08/10/2016 DATA IN LIBRARIES: THE BIG PICTURE IFLA/ UNIVERSITY OF CHICAGO BIG DATA: A VERY
More informationBeyond Petascale. Roger Haskin Manager, Parallel File Systems IBM Almaden Research Center
Beyond Petascale Roger Haskin Manager, Parallel File Systems IBM Almaden Research Center GPFS Research and Development! GPFS product originated at IBM Almaden Research Laboratory! Research continues to
More informationData Movement and Storage. 04/07/09 1
Data Movement and Storage 04/07/09 www.cac.cornell.edu 1 Data Location, Storage, Sharing and Movement Four of the seven main challenges of Data Intensive Computing, according to SC06. (Other three: viewing,
More informationThe Stampede is Coming Welcome to Stampede Introductory Training. Dan Stanzione Texas Advanced Computing Center
The Stampede is Coming Welcome to Stampede Introductory Training Dan Stanzione Texas Advanced Computing Center dan@tacc.utexas.edu Thanks for Coming! Stampede is an exciting new system of incredible power.
More informationData Replication: Automated move and copy of data. PRACE Advanced Training Course on Data Staging and Data Movement Helsinki, September 10 th 2013
Data Replication: Automated move and copy of data PRACE Advanced Training Course on Data Staging and Data Movement Helsinki, September 10 th 2013 Claudio Cacciari c.cacciari@cineca.it Outline The issue
More informationAPAN Global Collaboration Linking the World with Light
APAN Global Collaboration Linking the World with Light 1 Andrew Howard Coordinating Engineer Advanced Communications Services and International Development Massive increase in International connectivity
More informationAn Assessment of Data Transfer Performance for Large- Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6
An Assessment of Data Transfer Performance for Large- Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6 Eli Dart, Michael F. Wehner, Prabhat Lawrence Berkeley National
More informationSPINOSO Vincenzo. Optimization of the job submission and data access in a LHC Tier2
EGI User Forum Vilnius, 11-14 April 2011 SPINOSO Vincenzo Optimization of the job submission and data access in a LHC Tier2 Overview User needs Administration issues INFN Bari farm design and deployment
More informationScientific data processing at global scale The LHC Computing Grid. fabio hernandez
Scientific data processing at global scale The LHC Computing Grid Chengdu (China), July 5th 2011 Who I am 2 Computing science background Working in the field of computing for high-energy physics since
More informationAnalisi Tier2 e Tier3 Esperienze ai Tier-2 Giacinto Donvito INFN-BARI
Analisi Tier2 e Tier3 Esperienze ai Tier-2 Giacinto Donvito INFN-BARI outlook Alice Examples Atlas Examples CMS Examples Alice Examples ALICE Tier-2s at the moment do not support interactive analysis not
More informationExperiences of the Development of the Supercomputers
Experiences of the Development of the Supercomputers - Earth Simulator and K Computer YOKOKAWA, Mitsuo Kobe University/RIKEN AICS Application Oriented Systems Developed in Japan No.1 systems in TOP500
More informationHPC File Systems and Storage. Irena Johnson University of Notre Dame Center for Research Computing
HPC File Systems and Storage Irena Johnson University of Notre Dame Center for Research Computing HPC (High Performance Computing) Aggregating computer power for higher performance than that of a typical
More information