Challenges of Big Data Movement in support of the ESA Copernicus program and global research collaborations

Transcription

1 APAN Cloud WG Challenges of Big Data Movement in support of the ESA Copernicus program and global research collaborations

2 Lift off

3 NCI and Copernicus The National Computational Infrastructure (NCI) in Australia hosts the Asia Regional Copernicus data Hub A free and open data resource available for the Asia Pacific region This requires a high speed and continuous network and storage capability to capture, store and publish the satellite data in close to real time Highly critical for Disaster management Real time response to extreme climate and seismic events Delivering this data to Australia and to the region involves many networks AARNet Australia's Academic and Research Network TEIN the Trans Eurasian Information Network SingAREN in Singapore Internet2 in the USA GÉANT the pan European network

4 The Copernicus Program

5 Problem Space High bandwidth, high latency networks Traditional transfer tools perform at one to two orders of magnitude below available bandwidth The greater the distance the lower the performance Science DMZ handles the last mile problem of firewalls, traffic shapers and other network restricting devices but is not the magic bullet for big data movement Most remote data servers use protocols designed in the 90 s (http) Tools are optimised and tested within a regional environment Europe, Americas, Asia Oceania (Australia and New Zealand) are large data consumers As a TCP/IP based protocol they suffer from TCP related tuning issues Window Sizes Congestion control Highly sensitive to even small <1% packet loss with sever performance degradation All sites must be tuned DTN to DTN is performant but is a small subset of the transfer requirement These do not scale well on a global scale Commercial response by FANG (Facebook, Amazon, Netflix, Google) is global distributed CDN network to bring data closer to the user

6 Network capabilities The network is capable of delivering data at global scales Existing tools and protocols are the bottleneck CERN and the High Energy Physics community are the most successful and have lead the way in performant big data transfers at a global scale Multi-tiered approach Canberra/Singapore 1Gbs (~6000km) Average Time NCI and NSCC have collaborated to utilise new protocols and techniques InfiniCortex Extend InfiniBand globally to create a highly effective high speed data transfer system InfiniCloud A global cloud application platform BeeGFS A high performance Global file system Squid based caching infrastructure Singapore/Canberra 10Gbs (~30,000km) 14h 13h 12h 11h 10h 9h 8h 7h 6h 5h 4h 3h 2h 1h 0 12h 33m 9h 4h 5m 24m rsync A*Star to NCI R&E path rsync A*Star to NCI 10Gbs IPoIB DSYNC 1Gbs IB path DSYNC 10Gbs IB path (~26,000Km) 1,143Gb data set

7 Friction Free Data movement We need to provide our HPC centres and researchers with a friction free data transfer system Secure using a Federated Access system Easy to use The network and tools should have the data in the right location at the right time Able to effectively use different storage tiers SSD Spinning Disk Tape The researcher creates a Data Intent definition Data Source Data Target Storage performance (SSD, Disk, Tape)

8 APRP - Building the Platform for Regional and Global research The Asia Pacific Research Platform and other Research Platforms are an essential component of transferring large data sets in a performant, secure and reliable manner between regions Earth Monitoring Genetics Climate data NCI in Australia and NSCC in Singapore have collaborated to build a shared high performance data transfer system for the benefit of Asia and Oceania based on the European model of an extreme Data Cloud combined with the US model of Science DMZs pioneered by ESNet in the USA Our joint challenge is to transform our tools from a custom solution to an open platform

9 Regional Infrastructure Our best current solution to this challenge has been to implement a set of regional based DTN services Optimised for the prevalent regional network tuning Custom built for specific organisations Not a general purpose facility Not scalable beyond participating organisational collaboration Our next phase will build out a set of regional hubs each tuned for regional network conditions and connected together using our high speed R&E networks London (Europe) Singapore (Asia) Los Angeles (Americas) Canberra (Oceania)

10 extreme DataCloud FTS INDIGO PaaS Orchestrator INDIGO CDMI Server

11 What is the extreme Data Cloud? The extreme DataCloud is a software development and integration project Develops scalable technologies for federating storage resources and managing data in highly distributed computing environments Focus efficient, policy driven and Quality of Service based DM The targeted platforms are the current and next generation e- Infrastructures deployed in Europe European Open Science Cloud (EOSC) The e-infrastructures used by the represented communities

12 Intelligent & Automated Dataset Distribution A typical workflow Initially the data will be stored on low latency devices for fast access To ensure data safety, the data will be replicated to a second storage device and will be migrated to custodial systems, which might be tape or S3 appliances Eligible users will get permission to restore archived data if necessary After a grace period, Access Control will be changed from private to open access Data management based on access pattern

13 What is the extreme Data Cloud Intelligent & Automated Dataset Distribution Orchestration to realise a policy-driven data management Data distribution policies based on Quality of Service (i.e. disks vs tape vs SSD) supporting geographical distributed resources (cross-sites) Software lifecycle management Data pre-processing during ingestion Data management based on access patterns Move to glacier-like storage unused data, move to fast storage hot data at infrastructure level Smart caching Transparent access to remote data without the need of a-priori copy Metadata management Sensitive data handling secure storage and encryption

14 XDC components Storage Rucio Orchestration INDIGO Orchestrator xrootd Cache QoS CDMI c Federation

15 Caching Smart caching A global caching infrastructure supporting the following building blocks: dynamic integration of satellite sites by existing data centres creation of standalone caches modelled on existing web solutions federation of the above to create a large scale caching infrastructure

16 ONEDATA A storage federator that allows users to store, process and publish data using global data storage backed by resource providers worldwide Providers deploy Oneprovider services near physical storage resources Users use Onezone web interfaces APIs available Local mounting on users machines available Storage is organised into Zones federations of providers enable the creation of closed or interconnected communities

17 OneData Unified data access platform at a PaaS level at the Exascale Advanced metadata management with no pre-defined schema Encryption Services and Secure Storage Sensitive data management and key storage within Onedata

18 XDC high level architecture

19 Conclusion Our Research and Education networks connect the world Traditional transfer tools perform at one to two orders of magnitude below available bandwidth Science DMZs are not the magic bullet for big data movement The European extreme Data Cloud provides a model for Federated, Secure data movement and storage distributed between regions We welcome your participation For further information please contact andrew.howard@anu.edu.au

20 Acknowledgements Thanks to Patrick Fuhrmann DESY Germany Daniele Cesini INFN Italy For much of the material used in this presentation. extreme DataCloud is co-funded by the Horizon2020 Framework Program Grant Agreement Copyright Members of the XDC Collaboration,

21

22 Acknowledgements