dcache NFSv4.1 Tigran Mkrtchyan Zeuthen, dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 1

Transcription

1 dcache NFSv4.1 Tigran Mkrtchyan Zeuthen, dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 1

2 Outline NFSv41 basics NFSv4.1 concepts PNFS Id mapping Industry standard dcache implementation dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 2

3 Classic NFS BigData never fits into a single server. Big administrative overhead to keep data on multiple servers Single NFS server becomes bottleneck dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 3

4 NFSv4.1 Provides access to files and directories Stateful Keeps track of OPEN/CLOSE ( LOCK/UNLOCK ) Detects client/server reboot Client controlled reply cache and EOS Aware of multihomed servers Detects retransmits Recovery from network disconnect dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 4

5 Classic NFS DISK DISK NFS DISK DISK dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 5

6 DISK DISK NFS DISK DISK dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 6

7 CPU or network become a bottleneck with growing number of clients DISK DISK NFS DISK DISK dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 7

8 Parallel NFS NFS DS NFS DS NFS MDS NFS DS NFS DS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 8

9 Parallel + Striping read(block1-block4) Read block1 DS b1 Read block2 DS b2 NFS Read block3 DS b3 Read block4 DS b4 dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 9

10 Parallel NFS Single Namespace, distributed data Client talks to Meta Data Server for metadata only Bandwidth and performance grow with number of Data Server nodes File striping ( like raid0 ) Enforces the same security on DS (pools) as on MDS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 10

11 Security To verify client credentials RPCSEC_GSS is used Krb5 implementation supported by all clients/server Three type of Quality of Protection (QOP): NONE Auth only. Checksum protection of RPC header INTEGRITY Checksum protection of RPC messages PRIVACY full encryption of RPC messages Security flavor used on mount enforced to IO traffic as well dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 11

12 Security NFS client 'sec=krb5i' NFS server (MDS) NFS server (DS) Client always will use the same security flavor and Quality of Protection for all RPC traffic to MDS and DS. dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 12

13 ID mapping All principals are utf 8 strings No UID, no GID To resolve conflict NFSDOMAIN used: tigran@desy.de vs. tigran@cern.ch E.g. talk to corresponding mapping service Mapping delegated to client and server Client and server may use different mapping services/sources Windows client and server do not need numeric ID Best with ldap or nis Linux client can use numeric strings 124 => 124 dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 13

14 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 14

15 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 15

16 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 16

17 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 17

18 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 18

19 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 19

20 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 20

21 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 21

22 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 22

23 idmapping App. euid Chimera NFS client 'owner : tigran' NFS server dcache tigran tigran idmapd gplazma LDAP/NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 23

24 Why so complicated? Your favorite OS may not use numeric id MS Windows uses principals Your numeric ID on client and server may be different My ID on laptop 500, NIS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 24

25 Existing servers NETAPP (ONTAP 8.1 cluster mode, running at DESY) dcache Production ready In production since XXX SONAS (IBM), Panasas, EMC, LinuxBox will be ready by 2Q 2013 dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 25

26 Existing clients Linux first usable kernel Part of RHEL 6 (SL6), Fedora >= 15, Debianunstable Oracle UEK2 for RHEL5 (sl5) + updates! No AFS and some other kernel modules Windows 7 64 bit Opensource client from CITI VMware ESX (pnfs client in VMware hypervisor) dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 26

27 dcache implementation dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 27

28 dcache NFS in one slide JVM JVM JVM Message passing layer Door (MDS) Pool Manager PnfsManager Pools (DSs) Pools Pools DBMS (Data Server) (Data Server) dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 28

29 Implementation Status No file striping yet Supports RPCGSS_SEC (kerberos5 only) Krb5, krb5i and krb5p Even with Windows AD as KDC Supports IPv6 Can be integrated with NIS and/or local passwd file Implemented as gplazma plugins. dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 29

30 Still Missing Extended attribute support Will provide native access (setfattr/getfattr) to checksum, AL and RP Only unix permission bits are supported ACL expected by dcache 2.4, set/getfacl works today. No striping Striping on read will come soon. No IO through Meta Data Server (door) A problem if pool crashed/restarted dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 30

31 Limitations dcache files are immutable No support of byte range locks No support of multiple writers dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 31

32 ACLs with NFSv4.1 in dcache Will be available in dcache 2.4 (or 2.3) Main focus on predictable semantic Unix mode and ACL coexistence NFSv3, FTP and NFS4 coexistence dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 32

33 Prerequisites for NFSv4.1 1.Configured gplazma 2.Kerberos5 keytab with nfs principals for NFS door and all pools. 3.Open TCP port 2049 in NFS door 4.Open TCP range 'net.lan.port.min:net.lan.port.max' on pools Default is 33115:33145 (shared with dcap and xrootd) 5.Client with pnfs aware kernel (SL6.2 and FC16 recommended) 6.Configured rpc.idmapd to match NFS DOMAIN with server 7.Kerberos5 keytab with host principal 8.Start dcache, mount on client and access the data. dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 33

34 Terminology dcache equivalent MDS Meta Data Server dcache NFSv4.1 door DS Data server dcache pool QOP Quality Of Protection dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 34

35 Typical IO scenario Open Read/write close dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 35

36 OPEN+READ/WRITE+CLOSE Open Open/Create a file Find appropriate pool Start a mover Read/write READ/WRITE Close Release mover Close the file dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 36

37 OPEN+READ/WRITE+CLOSE App NFS Door PnfsManager PoolManager Pool OPEN OPEN ID GET STORAGE INFO STORAGE INFO GET POOL SELECT POOL POOL POOL/MOVER ID START MOVER MOVER ID READ/WRITE CLOSE STOP MOVER dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 37

38 Setup: enable door # layout file # nfs 4.1 section [nfsdoordomain/nfsv41] nfs.domain=desy.de nfsioqueue=nfs # NFS uses direct access to Chimera database chimera.db.name = chimera chimera.db.host = localhost chimera.db.user = chimera chimera.db.password = # dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 38

39 Setup: tweak rpcbind # /etc/sysconfig/rpcbind # enable insecure mode RPCBIND_ARGS=" i" # dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 39

40 Setup: start door/pool # layout file [pooldomain/pool] name=pool xxx... poolioqueue=nfs # # use one of the ports from range for movers. # only single port used per pool net.lan.port.min=33115 net.lan.port.max=33145 # dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 40

41 Setup: tweak pool # layout file [pooldomain/pool] name=pool xxx... poolioqueue=nfs # # use one of the prots from range for movers. # only single port used per pool net.lan.port.min=33115 net.lan.port.max=33145 # dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 41

42 Setup: gplazma Identity plugin required Nsswitch uses local accounts, based on /etc/nsswitch.conf Uses accounts from gplazma host! Nis uses site NIS server nfs.domain on door MUST match with 'Domain' on client Linux conf at /etc/idmapd.conf dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 42

43 Setup: gplazma # layout file [gplazmadomain/gplazma] gplazma.nis.server = nisserv6.desy.de gplazma.nis.domain = desy.de # # gplazma.conf identity requisite nis dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 43

44 Setup: tweak idmap # dcache.conf # maximal number of entries in the cache nfs.idmap.cache.size = 512 # cache entry maximal lifetime nfs.idmap.cache.timeout = 30 # Time unit used for timeout. # SECONDS MINUTES HOURS DAYS) nfs.idmap.cache.timeout.unit = SECONDS dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 44

45 Setup: read to go dcache start dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 45

46 Anatomy of NFS package NFSv41 (rfc 5661) RPC (rfc 1831) RPCCES_GSS (rfc 2203) XDR (rfc 1832) dcache NFSv4.1 Tigran Mkrtchyan 4/13/12 Page 46