BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer. Wenji Wu Internet2 Global Summit May 8, 2018

Transcription

1 BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer Wenji Wu Internet2 Global Summit May 8, 2018

2 BigData Express Funded by DOE s office of Advanced Scientific Computing Research (ASCR) Collaborative effort by Fermilab and Oak Ridge National Laboratory KISTI joined as a unfunded partner at 2017 ESnet provides WAN service A three-year research project Start: Oct 1, 2015 End: Sep 30,

3 BigData Express Research Team FNAL Wenji Wu (PI) Qiming Lu Liang Zhang Amy Jin Sajith Sasidharan Phil DeMar ORNL Nageswara Rao Gary Liu KISTI Syed Asif Shah Seo-Young Noh Jin Kim Note: KISTI and ESnet are unfunded project partners

4 DOE Leadership facilities offer computing and Big datacomputing enables scientific discoveries storage resources needed to process and analyze science data BigData Express Goal The efficient movement of science data from their sources into processing and storage facilities and ultimately on to user A distributed middleware system that provides a analysis is critical to the success of any such endeavor. schedulable, predictable, and high-performance transfer for function the DOE s large-scale Datadata transfer is now service an essential for science discoveries, particularly within datacollaborators. environments. science facilities andbig their

5 Why BigData Express? Targeted at optimizing data transfers in high-speed networks Large-scale data movement of Big Data Science High-speed network environments (40/100GE+) Builds on Multicore-Aware Data Transfer Middleware (MDTM) mdtmftp: a high-performance data transfer tool Pipelined I/O-centric design to streamline data transfer MDTM optimizes use of underlying multicore system Extremely efficient in transferring of Lots Of Small Files (LOSF) Orchestrates system (DTN), storage, & network (SDN) resources To provide full end-to-end performance optimization

6 BigData Express versus SENSE BigData Express is data transfer middleware Uses SENSE for WAN SDN services SENSE is a network service Provides higher-level applications with SDN-type services BigData Express is an application to SENSE BigData Express and SENSE are each stand-alone services in their own right BigData Express works fine without SENSE WAN component is simply Best Effort SENSE is agnostic to higher-level applications using its services

7 BigData Express Major Components BDE Web Portal Allow users to access BigData Express data transfer services BDE Scheduler DTN as a service Co-scheduling of DTN, storage, and network BDE AmoebaNet Network as a service mdtmftp a high-performance data transfer engine

8 BigData Express Major Components (cont.) DTN Agents Manage and configure DTNs Collect and report the DTN configuration and status Storage Agents Manage and configure storage systems Data Transfer Launching Agent Launch data transfer jobs Support different data transfer protocols

9 BigData Express -- Distributed A Peer-to-Peer model

10 BigData Express -- Flexible DTNs DTNs Data Transfer Federation Data Transfer Federation Flexible to set up data transfer federations Data Transfer Federation Networks DTNs Providing inherent support for incremental deployment DTNs DTNs

11 BigData Express -- Scalable BDE Web Portal SDN Agent SDN Agent (AmoebaNet) Message Queue BDE Scheduler BDE Scheduler DTN Agent DTN Agent DTN Agent Storage Agent Storage Agent Storage Agent Data Transfer Data Transfer Launching Data Transfer Agent Launching Agent Launching Agent BigData Express scheduler manages site resources through agents Use RabbitMQ as message bus

12 BigData Express -- Extensible BDE Web Portal SDN Agent SDN Agent (AmoebaNet) Message Queue BDE Scheduler BDE Scheduler DTN Agent DTN Agent DTN Agent Data Transfer Launching Agent Data Transfer Launching Agent Data Transfer Launching Agent Storage Agent Storage Agent Storage Agent mdtmftp Plugin GridFTP Plugin SRM Plugin XrootD Plugin Extensible Plugin framework to support various data transfer protocols mdtmftp, GridFTP, SRM, XrootD,

13 BigData Express -- End-to-End Data Transfer Model Site A - Smart E2E Data Transfer Orchestrator Web Portal Scheduler Data Transfer Launching Agent mdtmftp DTN Agent Resource negotiation & brokering AmoebaNet Resource negotiation & brokering SENSE Resource negotiation & brokering AmoebaNet Site B - Smart E2E Data Transfer Orchestrator Web Portal Scheduler Data Transfer Launching Agent mdtmftp DTN Agent Application-aware network service o On-demand programming Fast-provisioning of end-to-end network paths with guaranteed QoS Storage Agent LAN WAN LAN Storage Agent Distributed resource negotiation & brokering Storage Edge DTN Edge DTN Storage A End-to-End Data Transfer Loop with Guaranteed QoS

14 BigData Express -- Three Types of Data Transfer Real-time data transfer Deadline-bound data transfer Best-effort data transfer

15 BigData Express vs. Globus Online Features BigData Express Globus Online Architecture Supported Protocols SDN Support Supported Data Transfers Error Handling Distributed service Flexible to set up data transfer federations Extensible plugin framework to support multiple protocols: o mdtmftp o GridFTP, XrootD, SRM (coming soon) Yes, Network as a service Fast-provisioning end-to-end network paths with guaranteed QoS Real-time data transfer Deadline-bound data transfer Best-effort data transfer Checksum Retransmit Centralized service GridFTP No Best-effort data transfer Checksum Retransmit

16 BigData Express SC 17 DEMO BigData Express: a schedulable, predictable, and high-performance data transfer service QoS-guaranteed data transfer DTN as a service Network as a service Distributed resource brokering/matching A DOE/SC/ASCR-sponsored research project Software is available at:

17 A Cross-Pacific SDN Testbed HP Z91000 AmoebaNet KREONET STP STP 40GE GE BDE Web Protal DTN2 BDE Scheduler StarLight STP GE GE DTN3 DTN ESNET OSCARS KISTI, South Korea KISTI SW 600W Chicago To Internet STP FNAL Border router bde-hp1.fnal.gov ESnet NSI Circuit Service AmoebaNet 40GE 4/1 73 4/3 Pica8 P5101 4/5 4/7 40GE 47 Pica8 P yosemite.fnal.gov 4/2 74 4/4 4/6 4/ BDE Web Protal BDE Scheduler 40GE 40GE 40GE GE 40GE BDE1 BDE2 BDE3 BDE-hp5 BDE4 Lustre file system wwwld1 (mgt) Infiniband Switch wwwld5 (oss) wwwld2 (oss) wwwld3 (oss) wwwld4 (oss) wwwld6 (oss) FNAL, US

18 BigData Express Deployment Completed deployment: KISTI, UMD, StarLight, FNAL Ongoing deployment: KSTAR, ESnet Work with StarLight to deploy BDE at XRPs Pacific Research Platform (PRP) National Research Platform (NRP) Global Research Platform (GRP) The European Research Platform (ERP) Asia Research Platform (ARP) Collaborate with SENSE for BDE+SENSE deployment Work with US CMS to deploy BDE at US CMS sites

19 Fusion community Support Science Work with KSTAR, KISTI, PPPL, and ORNL to transfer/stream data from KSTAR to US research institutions XRPs (PRP, NRP, GRP, ERP, ARP) Work with StarLight to deploy BDE at XRPs to support various science HEP community Work with US CMS to deploy BDE at US CMS sites PI has been invited to give a BDE demo for US CMS Tentatively scheduled for the last week of May, 2018

20 More information about BigData Express PI: Wenji Wu, Fermilab