Chelonia a lightweight self-healing distributed storage Zsombor Nagy (zsombor@niif.hu) Salman Toor (salman.toor@it.uu.se) Jon Kerr Nilsen (j.k.nilsen@fys.uio.no)
Motivation How to easily... Create a storage resource from available disk space on any computer Connect storage resources through the Internet Provide an easy-to-understand way for users to upload and share their files Use these files in a Grid environment
Design The system should be... self-healing easy to deploy easy to maintain lightweight no single point of failure
Decisions Files Replication Berkeley DB HA HTTP(S) file transfer SOAP messaging X509 certificates
NORDUGRID Grid Solution for Wide Area Computing and Data Handling Chelonia Developed by the KnowARC project Will be maintained by the NorduGrid Collaboration Running in NorduGrid ARC s web service container Written in Python Running on Linux and Mac OS X (and Windows soon)
The Chelonia Cloud Storage cloud for anyone Can be created by anyone Can be shared with anyone Can be used by anyone
Global namespace Global hierarchical namespace Files are organized in collections All users see the same tree Can use logical names Paths in Chelonia namespace Similar to paths in a regular local filesystem
Replication Files are replicated User specifies how many replicas are needed Chelonia itself takes care of the replication Easy to create a new storage node
Access Control Access policies to files and collections Can grant access to Individual users If in a Grid environment, to entire virtual organizations
Client tools Several ways to access Chelonia Command line interface FUSE module In an ARC enabled Grid environment Through job specification file With ARC client tools
FUSE module We have a FUSE module to mount Chelonia as a local filesystem You can upload and download files, create collections (directories) with your OS s CLI and GUI tools (e.g. drag and drop) Linux and Mac OS X
Components of Chelonia Consists of four services The Bartender The Shepherd The Librarian The A-Hash Can have multiple instances of all services
The Shepherd Manages the storage node with the actual file data Replication requires more storage nodes, each managed by a Shepherd Adding a Shepherd, replica will automatically be created (if needed)
The Librarian and A-Hash The Librarian manages the metadata Uses the A-Hash to store the data Both metadata and Librarians can be replicated More fault tolerant Better load balance
The Bartender Serves the users by negotiating with the Librarian to query files and collections the Shepherd to upload and download files More than one Bartender to eliminate single point of failure Users can contact any Bartender and get the same result
Downloading A-H A-H A-H 2. traverse(ln) B 1. getfile(ln) A-H L 3. file's metadata S 5. TURL 4. get(referenceid) 6. TURL 7. downloading the file USER S E S
Uploading A-H 2. traverse(ln) A-H A-H A-H L 3. parent's data 4. new(metadata) 5. file's GUID 6. add entry to parent 8. list of Shepherds 10. register new replica 9. put(...) 11. TURL B S 1. putfile(ln) 12. TURL 13. uploading the file USER 14. report state change S E S
Depth test
Width test
Multiple users
Number of replicas Five Shepherds were used for this test User Side ALIVE OFFLINE THIRDWHEEL CREATING 10 files of size 100MB have 3 replicas and now user: Increase the number replicas to 5. Decrease the number of replicas to 2 Number of files in the system 50 40 30 20 10 0 50 40 30 20 10 0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 15 30 45 60 75 90 105 120 135 105 165 180 195 210 225 240 255 Time in seconds taken by the system to achieve the needed replicas System Side 10 files with 4 replicas are in the system. One Shepherd went OFFLINE for sometime Became ONLINE Number of files in the system 50 40 30 20 10 0 15 30 45 60 75 90 105 120 135 150 165 Time in seconds taken by the system to achieve the needed replicas
Memory usage 8 days running with lots of file uploads and downloads
Better clients? SRM? WebDAV? Encryption? AAI? Transfer protocols? Transfer security? Versioning? Resuming uploads? SOAP over HTTPS? Space reservation? Future?
ClusterGrid Maintained by NIIF, Hungary Since 2002 connecting a few hundred PCs located at several universities in Hungary Previously using NIIF s own grid middleware Soon it will use ARC and Chelonia
Chelonia and ClusterGrid Soon there will be a purchase of some storage solutions which will be deployed at chosen Hungarian universities We will deploy Chelonia storage nodes on these The ClusterGrid users will be able to upload and download (input/output) files to the Chelonia cloud and refer to these files in the job descriptions
Questions / Demo
Thank You! web: http://www.knowarc.eu/chelonia/ paper: http://arxiv.org/abs/1002.0712 video: http://www.youtube.com/watch?v=neuwzghhghc Zsombor Nagy, NIIF (Hungary), zsombor@niif.hu