Overview. Jakub T. Mościcki, IT/DSS. Meeting with Palestinian fellows

Transcription

1 Overview Jakub T. Mościcki, IT/DSS Meeting with Palestinian fellows

2 What is CERNBox? CERNBox provides a cloud synchronisation service Available for all CERN users (1TB/user) Synchronise files (data at CERN) Easy way to share with other users All major platforms supported Based on owncloud integrated with EOS fs webdav xroot sync share mobile web ACLs EOS Physical Storage 2

3 Cloud storage for all devices powered by 3

4 Access Methods: Sync fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 4

5 Access Methods: Sharing fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 5

6 Access Methods: Mobile & Web fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 6

7 Access Methods: WebDAV/SMB fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 7

8 Access Methods: FUSE fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 8

9 Access Methods: xroot & ROOT Embedded ROOT viewer in CERNBox browser fs webdav xroot sync share mobile web ACLs EOS Physical Storage Namespace 9

10 Service Overview 10

11 How much space do I have? Starting with 1 TB per user Quotas: maximum file size is 8 GB 4 GB web upload limit file quota of 1 million files 11

12 CERNBox Service Numbers EOS offers virtually unlimited cloud-storage for our end-users Users 5074 # files 55 Million # dirs 7.2 Million Quota Used Space Deployed Space 1TB/user 104 TB 1.3 PB Mar-14 Apr-14 May-14 Jun-14 Jul-14 Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 Users Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15 Jan-16 20% 60% 20% User community very active Very positive feedback Several useful suggestions Important contributions happy to help testing new features Engineers Physicists Services & Administration 12

13 EOS/CERNBox HTTP Operations Some Peak Requests at 1.1kHz Weekly Cycles Weekends CERN end-of-the-year closure 1200"Hz" 800"Hz" 400"Hz" 0"Hz" 7:42:40& 7:42:45& 7:42:50& 7:4 PUT and GET Requests Dis$nct IPs Millions Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15 0 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15 13

14 Nov 2015: Geolocation Active Users Background OpenStreetMap & contributors; image available under CC-BY-SA 14

15 Dec 2015: Geolocation Active Users Background OpenStreetMap & contributors; image available under CC-BY-SA 15

16 November 2015 OpenStreetMap contributors December 2015 OpenStreetMap contributors 16

17 E-Science Enable non-experts to easily use CERN Storage resources Powerful integration with the batch system Simple to share result with collaborators Thanks to Mauro Arcorace, members of UNITAR/UNOSAT and CIMA foundation for the material provided 17

18 CERN Press Office native client (xrdcp) profit from native performance in transferring 30-50GB files Desktop Sync Client CERNBox used by the Press Office to share immediately videos to the media for download CERNBox share 18

19 Documentation 19

20 Try out and give feedback to Dorothee 20

21 Tour of Features 21

22 Tour of Features Desktop sync client Limitations Recommended configuration Data protection Trashbin File versions Sharing Share by link Share with registered users or egroups Online access Webdav and fuse mounts Web goodies ROOT files viewer 22

23 Synchronization client install cernbox clients from (for service management reasons we do not support arbitrary upstream client versions) 23

24 Recommended configuration One sync folder to your data (this is the default) local path: ~/cernbox > remote path: home 24

25 Limitations Synchronization filenames with special characters do not sync e.g., :? * > < ignored file patterns (e.g. Thumbs.db.DS_Store ~*.tmp) symlinks are not supported (and may be dangerous) do not sync into AFS! definitive knowledge source: KB

26 Ignored file patterns 26

27 Choose what to sync 27

28 Data protection features 28

29 Trashbin Self-service file recovery 29

30 Versions Restore up to last 10 versions 30

31 Sharing features 31

32 Share by link Share files with anyone, password protected 32

33 Share by link Allow upload into a directory via link 33

34 Share with registered users Only directories (currently limited to top-level) 34

35 Share with registered users notification 35

36 Syncing shared folder 36

37 Syncing shared folder 37

38 Syncing shared folder 38

39 Syncing shared folder 39

40 Share with e-groups Only directories (currently limited to top-level) 40

41 Online access to files 41

42 Online file access from Desktop 42

43 Online file access from Linux If you know EOS CLI you can use it directly export EOS_MGM_URL=root://eosuser.cern.ch eos ls -l /eos/user/k/kuba lxplus fuse mounts for beta testers in the future available to all users ls /eos/user/k/kuba 43

44 Technical Architecture 44

45 Components involved 45

46 CERNBox architecture fuse xroot 1094 sync, webdav https 443 cernbox gateway http 8000 http 8001 EOS https 443 web browser REST API oc server xroot 1094

47 CERNBox internals Identity service (AD) ldaps ldap(s?) auth ldaps MGM slave nginx eos metadata MGM master blender data oc oc server FST FST FST server DB

48 Summary 48

49 Summary Possible use-cases administration: document sharing scientific collaboration and data analysis on-the-field tasks with no internet connectivity good for a large spectrum of users, including non-technical persons 49