METADATA SYNCHRONIZATION PROTOCOL FOR A DECENTRALIZED NETWORK OF DATA PROVIDERS

Transcription

1 METADATA SYNCHRONIZATION PROTOCOL FOR A DECENTRALIZED NETWORK OF DATA PROVIDERS ACK CYFRONET AGH, Kraków, Poland AGH University of Science and Technology, Kraków, Poland Faculty of Computer Science, Electronics and Telecommunications Department of Computer Science Łukasz Opioła Łukasz Dutka Renata G. Słota Jacek Kitowski

2 AGENDA Global data access for modern science Challenges of global data access Metadata synchronization protocol - requirements Proposed concept of metadata synchronization protocol Conclusions

3 GLOBAL DATA ACCESS FOR MODERN SCIENCE GROUP 2 GROUP 3 GROUP 1

4 CHALLENGES OF GLOBAL DATA ACCESS GROUP 2 GROUP 3 GROUP 1

5 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Bob GROUP 3 GROUP 1 Bob

6 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Cross-border collaboration Bob Alice GROUP 3 GROUP 1 Bob

7 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Cross-border collaboration Security & efficiency Bob Alice GROUP 3 GROUP 1

8 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Cross-border collaboration Security & efficiency Bob Alice Openness of the network GROUP 3 GROUP 1 James GROUP 4

9 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Cross-border collaboration Security & efficiency Openness of the network!!!! GROUP 3 Autonomy (lack of trust) GROUP 1!! GROUP 4

10 CHALLENGES OF GLOBAL DATA ACCESS Transparent data access GROUP 2 Cross-border collaboration Security & efficiency Openness of the network GROUP 3 Autonomy (lack of trust) GROUP 1 Reflecting existing hierarchies GROUP 4

11 METADATA SYNCHRONIZATION PROTOCOL

12 METADATA SYNCHRONIZATION GROUP 2 GROUP 3 GROUP 1 GROUP 4

13 METADATA SYNCHRONIZATION - REQUIREMENTS Decentralized (retaining autonomy) Globally consistent Scalable hundreds of providers Secure decentralized AAI* Location discovery mechanisms * AAI authentication and authorization infrastructure

14 PROPOSED SOLUTION Metadata synchronization protocol based on hybrid, multi-tier architecture

15 PROPOSED SOLUTION PROVIDER SCOPE Provider service basic building block Gathers knowledge using metadata sync Manages data on underlying storages AAI Authentication and Authorization Infrastructure

16 PROPOSED SOLUTION ZONE SCOPE Zone service oversees a group of providers Gathers and serves metadata Trusted authority & mediator AuthN & AuthZ center Macaroon based tokens Reflects existing hierarchies

17 PROPOSED SOLUTION GLOBAL SCOPE Cooperation between Zones hybrid architecture ZONE A Decentralized, P2P backbone ZONE B Local authority center Local authority center PROVIDER 1 PROVIDER 2 PROVIDER 3

18 PROPOSED SOLUTION METADATA SYNC Metadata handled by the Zone of origin local AuthN & AuthZ center ZONE A AAI CREATE / UPDATE / DELETE AAI ZONE B READ READ CREATE / UPDATE / DELETE CREATE / UPDATE / DELETE READ PROVIDER 1 PROVIDER 2 PROVIDER 3 No concurrent modification conflicts Metadata sync overheads evenly distributed

19 PROPOSED SOLUTION METADATA SYNC Multi-tier client-server architecture ZONE A ZONE B PROVIDER 2 PROVIDER 1 PROVIDER 3 web interface native client web interface

20 PROPOSED SOLUTION METADATA SYNC Publish / subscribe and multi-tier caching Vast performance improvement ZONE A FETCH & SUBSCRIBE CACHE ZONE B CACHE USER META FETCH & SUBSCRIBE USER META CACHE USER META FETCH & SUBSCRIBE PROVIDER 2 USER META PROVIDER 1 FETCH & SUBSCRIBE PROVIDER 3 CACHE FETCH & SUBSCRIBE USER META CACHE web interface native client web interface

21 PROPOSED SOLUTION METADATA SYNC Publish / subscribe and multi-tier caching Eventual consistency MODIFIED ZONE A CACHE ZONE B PUBLISH PUBLISH PUBLISH CACHE CACHE PROVIDER 2 PUBLISH PROVIDER 1 PROVIDER 3 CACHE PUBLISH CACHE web interface native client web interface

22 PROPOSED SOLUTION AAI Macaroon-based decentralized authorization ZONE A FETCH & SUBSCRIBE ZONE B DISCOVERY MECHANISM FETCH & SUBSCRIBE PROVIDER 2 PROVIDER 1 FETCH & SUBSCRIBE PROVIDER 3 location public bd3f798c749b TOKEN <caveats> sig a7e108fa16baeffab0 web interface native client web interface

23 PROPOSED SOLUTION ZONE MODE A) Open B) Restricted C) Isolated

24 METADATA SYNC PROTOCOL IN GLOBAL DATA ACCESS GROUP 2 ZONE 2 GROUP 3 ZONE 3 GROUP 1 ZONE 1 GROUP 4 ZONE 4

25 CONCLUSIONS Global data access can be achieved by creating a decentralized network of data providers We propose a metadata synchronization protocol for such network, based on hybdrid, multi-tier architecture with P2P backbone Proposed concept is being implemented in Onedata, a distributed virtual file system Zone service acting as central authority and SP server Synchronization Protocol (SP) for single zone scope Macaroon based AAI for single zone scope Data provider service employing the SP Cross Zone cooperation support Sync protocol supporting global, cross zone scale

26 THANK YOU The CGW'18, authorsoctober gratefully22-24, acknowledge KRAKÓW the financial support of this work from AGH-UST