NFS on Steroids: Building Worldwide Distributed File System Gregory Touretsky Intel IT

Transcription

1 NFS on Steroids: Building Worldwide Distributed File System Gregory Touretsky Intel IT

2 Legal Notices This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. * Other names and brands may be claimed as the property of others. Copyright 2013, Intel Corporation. All rights reserved.

3 Agenda Intel IT and Design Computing infrastructure Global Data Access challenges Considered solutions Selected direction Remote NFS access Kerberos authentication Implementation details 3

4 2013 Intel IT Vital Statistics 6,500 IT 59 global IT sites >95,200 Intel 164 Intel sites / 63 countries 68 Data Centers >147,000 Devices 4

5 Design Site Infrastructure Samba Interactive pool Remote Desktop Name Space NFS file servers Batch pool NIS Cron servers Application servers Configuration management Events monitoring 5

6 Many sites, many projects Project A Project B Project C Challenge: provide global data access solutions without compromising security 6

7 Cross-site data access: 2012 Site B Client1 Client2 Client3 Batch & Interactive NFS NFS1 Export11 Export12 Export13 Rsync++ NFS3 Export31 Export32 Export33 NFS Client1 Client2 Client3 NFS2 Export21 Export22 Site A Site C NFS4 Export41 Export42 Export43 Client1 Client2 Client3 Replication is time consuming. Hard to define specific dataset A subset of data is available at remote sites 16 GIDs constraint

8 Global User / Group accounts Global project requires global user, group accounts User may have 16 groups in site A, different 16 groups in site B and no account at site C 8

9 Goals Every file to be accessible from anywhere Same path access Server downtime shouldn t impact clients which aren t accessing data on this server WAN-friendly Every user account at every site. Every group at every site Solve 16 GIDs limitation Local IO performance shouldn t be compromised Leverage existing NAS infrastructure

10 Remote access - options OpenAFS Cloud storage NFS client-side caching WAN optimization NFS site-level caching Proprietary Open source (NFS Ganesha) In-house development *Other names and brands may be claimed as the property of others.

11 Direct NFS mount over WAN optimized tunnel Limited optimizer cache size Wrap the data store in days No data pinning NFS ops termination at the remote site (getattr/access) 220ms * 100,000 NFSops = 22,000 sec Multiple potential routes cache miss Same client going via different routes (experienced during the testing) different response from the other side 11

12 Selected: Site-level NFS caching and Kerberos CL1 CL2 CL3 NFS+KRB NFS1 Export11 Export12 Export13 Cache Site B NFS2 Export21 Export22 Cache NFS+KRB CL1 CL2 CL3 Export11 Cache Export21 Export52 WAN optimization NFS3 Export31 CL1 CL2 Export Cache Export11 Site A Site C Read/Write cache appliance mounts remote file system and re-exports it Kerberized NFS access increased number of GIDs per user CL3

13 Cache implementations First implementation: vendor-specific Evaluating alternatives Read Cache vs. Read/Write Cache Consistency vs. Performance Attribute cache timeout / Max writeback delay Optimizations Delegations Proactive attribute validation for hot files Cache pre-population

14 Where it s problematic? Read once over high latency link First read large file interactive work Large percentage of non-cacheable ops Seldom access beyond actimeo Business Continuity requirements 14

15 Cache management implementation Goal: Self-Service management for data caching Today: requires customer and several storage administrators Use cases: Cache my disk at sites A, B Modify cache parameters Remove cache Migrate source/cache Get cache usage statistics Shared capacity management Expanding our Storage and Replication management framework

16 Management system - example > stodstatus areas --cell pdx --fields project,path,sizegb,usagegb,cachedcells 'cachedcells=~"png"' Project Path SizeGB UsageGB CachedCells P1 /nfs/pdx/proj/path png P2 /nfs/pdx/proj/path png P3 /nfs/pdx/disks/path png P3 /nfs/pdx/disks/path png cr P3 /nfs/pdx/disks/path png P4 /nfs/pdx/disks/path ibw png > stodcache create --cell pdx --path /nfs/pdx/proj/path1 --cache iil Cached area with path /nfs/pdx/proj/path1 (id pdx ) successfully created on iil 16

17 Cache capacity planning Goal: every file to be accessible on-demand everywhere Track cache usage by Org/Project Shared cache capacity, multi-tenant Initial rule of thumb : 7-10% of the source capacity Seeding capacity at key locations

18 Usage models Remote validation Write once, Read many Get results back from remote sites Write once, Read once Drop box Generate in one site, get anywhere Single home directory Quick remote environment setup Data access from branch locations 18

19 NFS (RPC) authentication AUTH_SYS RPCSEC_GSS (KRB5) NFS client NFS UID Primary GID GID1-16 NFS server NFS client NFS Ticket NFS server KDC User is authenticated by Linux client Client tells server User ID and list of Group IDs (up to 16) User is authenticated via KDC on the client Server defines which groups the user belongs to (128/256/1024)

20 Bringing Kerberos? Samba Interactive pool Remote Desktop Name Space NFS file servers Batch pool NIS Cron servers Application servers Configuration management Events monitoring 20

21 Touching everything Linux client NFS file servers SSH Batch scheduler Remote Desktop / interactive servers Name space and automounter Trusted hosts vs. regular hosts Samba Setuid / sudo Cron jobs and service accounts Keytab management system Transparent Kerberos environment in the Linux world 21

22 Supporting Transition Can t jeopardize projects at critical phases Unknown stability issues Unknown tools issues Unknown performance issues AUTH_SYS RPCSEG_GSS transition will be gradual How to allow same the same NFS client 16 GIDs limit for AUTH_SYS mounts >16 GIDs for RPCSEC_GSS mounts 22

23 Solution: 2 nd NIS domain 2 distinct group views are needed during transition from non-krb to KRB: NFS File servers: full group membership NFS clients: up to 16 GIDs Site s NIS domain must be partially mirrored for the storage, with a slightly different view On the NIS master, the domain will be mirrored to <domain>.storage Some maps require data munging, as the domain name may be in keys and values 23

24 Welcome on board GDA Airlines Customer 1: Works like magic! It s like running in site A, while it s actually in site B! Customer 2: It enables transparency across sites that is just not possible thus far. we just do see a big gain and I m more than happy to pioneer this with IT! This is actually one of the biggest methodology advancements I see in years from database perspective! Customer 3: A true cloud computing Three cheers to you guys who brought this solution I am sure this will revolutionize remote site usage 24

25 Summary NFSv3 can be accessed over WAN using NFS caching proxy Some workloads do better than others NFSv3 environment can be Kerberized Major effort is required Transition is challenging It would be as challenging for NFSv4/KRB Facing similar challenges? Contact 25

26 Sharing Intel IT Best Practices with the World Learn more about Intel IT s Initiatives at 26