CERNBox. deployment experience and status. Data & Storage Services. CERNBox. Jakub T. Mościcki Massimo Lamanna CERN IT- DSS TNC 2014

Transcription

1 Data & Storage Services CERNBox deployment experience and status CERN IT Department CH-1211 Geneva 23 Switzerland Jakub T. Mościcki Massimo Lamanna CERN IT- DSS TNC 2014 CERNBox

2 Content Background & context Experience deploying owncloud CERNBox: next steps Conclusions 2

3 Main data storage services from IT/DSS group AFS: general purpose filesystem 2.5 billion files, 300 TB on disk 3 millions read/writes per day (34Hz) 300 disk drives, RAID model EOS: low- latency physics storage fast file namespace (20KHz file stats) scalable IO (22GB/s) ~50 PB on disk, 170 million files 30K disk drives, RAIN model CASTOR: archival and bulk data ~100 PB on tape 3

4 The origins of the CERNBox project Missing link? 4500 dis`nct IPs in DNS from cern.ch to *.dropbox.com (daily...) What we are missing easy access cloud storage for end users files go automa`cally to the cloud and are available always everywhere broken laptop data lost offline access to data work on the plane and rsync when back online keep files in sync across devices access on mobile clients (easy) sharing of files with colleagues CERNBox s`ll surprisingly difficult Can we have this? for documents (small files, ojen ppts, text, ) for science data (integrated into data processing workflows and exis`ng infrastructure) 4

5 CERNBox beta service Address the classical Dropbox use- case first for the documents usability and easiness over high- performance Based on 5

6 CERNBox beta deployment Setup 100% RH6 on standard hardware Guaranteed failover (redundant nodes) MySQL server 48GB RAM diagram source: owncloud.com Apache, PHP 5.4 (SCL1.0) mod_proxy_balancer 64 core, 64GB RAM NFS servers, async, RAID JBODs Ini`al space: 20 TB 6

7 Usage of CERNBox CERNBox Beta 2014 March April May users 190 (*) files 191K 907K 1.6M size 480GB 1TB 1.5TB 15% 1% Size per user 84% <10GB >10GB up to 100GB (*) users inherited from the ini`al prototype deployment Files per user 1% 5% 94% < 5K 5K- 20K up to 100K 7

8 User feedback very posi`ve but we see many idle accounts (~50%) is our feedback survey skewed by the 5% enthusias`c users? I start using the cernbox since I'm a heavy user of Dropbox and I recently reached the limit of free disk space (5Gb). For work it will be great to have at least 50Gb of personal space I would like to have is a free client for Android, which should be much more stable. I find the service perfect to be able to get always the files/sources/documents I need independently of the place and connec`on. On my Macbook Air I no`ced that the bavery was draining much faster than usual. I checked on the ac`vity monitor and CERNBOX was consuming 80-95% of the total energy. What I would like to do in the future is to combine my private data like my photos for example on my home owncloud server, and my work data on the Cern owncloud server. I'm very glad that CERN has launched the service using the OwnCloud plaxorm. I hope that you will be suppor`ng this service officially soon!! 8

9 File access paherns GET/PUT ra`o: 2/1 File type distribu`on: 1200 different file extensions! 30%.c.h.C 30%.jpg.png 15% no extension (UNIX world!) 25% other:.pdf,.txt,.ppt,.docx,.root,.py,.eps,.tex ~100 URL shares, ~40 synced shares 9

10 TesIng owncloud in the lab Our lab tests par`ally automated github.com/opensmashbox Client issues Change detec`on: full scans of the local/remote folders (takes long `me, drains bavery) Confusing client behaviour if filesystem modified while syncing File conflicts resolu`on - YES, directory conflict resolu`on - NO Corner cases of illegi`mate file dele`ons (symlinks, character cases) A safety net required to prevent data loss (e.g. local trashbin ) Centralized admin control needed (e.g. excluded file pavern list and other capabili`es) File versioning and trashbin are a must Some conflict situa`ons may only be resolved manually by the user à Owncloud client improvements are under heavy development Stability? How quickly can we ramp- up with fixes? 10

11 Towards large- scale data sync and share Currently deployed CERNBox beta works OK so far for the classical Dropbox use- case low- frequency document sync and share But can we bring this system to the next level? Our core- business workflows and large- scale workloads expose PBs of exis%ng data from day 1 integra`on into physics data processing eco- system central services: batch, interac`ve data analysis applica`ons sync higher data volumes at higher rates Can we s`ll keep the simplicity of cloud storage access? 11

12 Current architecture Sync client (webdav) Web access (hvps) HTTPS LB Data flow Metadata flow USER AS OC AS OC AS OC Image courtesy of Keeps track of sync state for every file in the system SQL overheads (Hz metadata ops) DB filesystem (POSIX) Files not exposed directly to the user STORAGE

13 Revised architecture Sync client (webdav) Web access (hvps) HTTPS HTTPS LB HTTPS LB HTTPS LB LB Data flow Metadata flow Data directly accessible by the user USER hvp (public data) hvps (private data) hvp (internal) KHz metadata ops OC fuse All sync state as metadata in the storage Files wriwen with USER credenxals STORAGE (EOS) IO redirect disk servers (1000s) namespace

14 Immediate access to all our data! 14

15 Massive scaling at reduced cost? No need to keep track of all files and directories in the database avoids explosive growth of your DB infrastructure Our file number es`mate? With 10K users we have 2.5 billion files in AFS already! What is your number for 100K users? Before we start throwing hardware at the problem consider the cost of running the service Fixed: hardware purchase, service deployment, infrastructure Scaling: hardware incidents, user support; backup; integrity checks; upgrades Infrastructure: space, electricity and cooling in the data center For massive scaling we need to keep TCO under control profit from exis`ng large- scale opera`ons and support of our storage services exploit economies of scale We have regular discussions on scaling issues with several partners CNRS: DB hardware scaling analysis for 100K users h)p://xstra.u- strasbg.fr/lib/exe/fetch.php?media=doc:josy- cloud:josy- cloud projet- owncore.pdf CERNBox - J.T.Moscicki, M.Lamanna - TNC2014 Dublin 15

16 Integrated storage ecosystem for scienific research sync / share / offline access 2.0 USER webdav & hvps:// online file- system access fuse CERNBox Analysis cluster Central Services high- performance applica`on access xrootd:// batch access xrdcopy LARGE- SCALE STORAGE

17 CERNBox 2.0 Advanced prototype stage Adapted exis`ng webdav interface in EOS to be compa`ble with owncloud sync clients Test environment (EOS PPS) standard hardware namespace node with Xeon 2.2GHz, 16 cores, 24GB RAM 50 disk servers: cheap JBODs (1000 disks), total 800TB usable space Storage layout: 2 replicas in RAIN mode à every file PUT = 2 copies of the file on two independent storage nodes (with adler32 checksums of content) Event- based hvp(s) load- balancer (nginx) Underlying storage scalability (EOS Prod) Max observed IO: 28GB/s from 160 clients reading with total of 1KHz file reads Max observed file stats: 10s KHz Thousands of connected clients Server should never be a bovleneck for CERNBox 17

18 First performance numbers Sync lots of small files to a laptop in campus WIFI 30Hz download, 50Hz upload Sync big files on fixed 1Gb network upload: 20MB/s download: ~1Gb/s owncloud client 1.6rc2 + external redirects your sync client is directly talking to 100s fileservers go as fast as your network goes! 18

19 First performance numbers User- perceived performance (client) Metadata opera`on (pycurl with SSL sessions) PROPFIND with 1 entry: 90 Hz PROPFIND with 1K entries: 8.5 KHz PROPFIND with 10K entries: 10KHz ops/s Download Upload Delete Small files (10KB) pycurl seq pycurl P=10 pycurl P=50 owncloud sync client owncloud sync client (1.6.2rc2) TH=10 19

20 Summary beta service for document sharing provide a useful func`onality already now appreciated by beta users sync clients need further improvements heading towards large sync and share layer for science research all our data exposed from day 1 massive scalability, high performance integrated into exis`ng workflows - new capabili`es! small overhead on top of our exis`ng opera`ons and development TCO control and s`ll as easy to use as Dropbox.com CERNBox 20

21 Backup slides 21

22 Basic sync algorithm client owncloud server PROPFIND: get remote ETAG PUT, GET, MOVE files ino`fy local changes sqlite DB propagate ETAG on file change Notes: ETAG is a standard HTTP header for cache control ETAG is a unique iden`fier generated by the server No file diffs over the wire icons: hvp:// 22

23 CERNBOX - J.T.Moscicki - White Area Lecture, April

24 File versioning and trashbin 24

25 Sharing Named shares Shared files to known users are synced automa`cally Share with hashed link Anonymous access It is possible to share a folder for web- based upload 25

26 Choices for CERN Dropbox service buy service private cloud on 3rd party storage - dropbox, wuala,... we could also buy other services: gmail,. private cloud on own storage with closed sojware licensing costs? vendor lock- in? bit torrent sync: mixed mode service private cloud with open source sojware large choice but products immature develop our own solu`on plaxorm integra`on? can we design good interfaces? none of these solu`ons is ideal... for various reasons 26

27 Pilot limitaions Move On the origin client move is propagated to the server On the other clients it is propagated as COPY/DELETE (subop`mal) Symlinks are not supported Ignored files:, :? * < > We currently recommend one sync folder setup: ~/cernbox High per- file overhead Expect 2-5Hz PUT Expect ~10Hz GET Transfer rates Expect 10-30MB download Expect 5-10MB upload Larger files: 400MB file on my desktop hvps/upload: ~25MB/s, hvps/download: ~60MB/s For wireless devices, laptops, phones do we care about transfer rates? 27

28 Protocol WebDAV (extension to HTTP with XML body) OwnCloud Server is RFC 2518 compliant (litmus check) Protocol is HTTP with XML body so it is bloated Basic metadata query for a file ~0.5KB Compresses well: metadata for 1000 files ~16KB Some good points Integra`on with other web- services Desktop browsers: OSX/Finder, Simply curl/wget to GET, PROPFIND, PUT, DELETE, MOVE Fuse mount (davfs2) HTTP POST/GET 4GB file limit for upload currently (browser limita`ons) Sync client: PUT Chunked upload (10MB chunks) Extension: OC- CHUNKED header avribute Sharing: REST API 28

29 Desktop sync client Implementa`on: Qt, libneon Keep sync state in sqlite db on the client side Basic workflow Check- for- updates Propagate Reconcile No file locking Change detec`on trigger a tree scan periodically asynchronously (ino`fy events via QFileSystemWatcher) Conflict resolu`on Local file is renamed: FILE_conflict txt No delta sync 29

30 Client issues Efficient cross- plaxorm client implementa`on is not THAT easy But it is improving. oc client 1.4 oc client 1.5 hvps://github.com/owncloud/mirall/issues/