An NFS Replication Hierarchy

Transcription

1 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 1 of 18 An NFS Replication Hierarchy Sun Microsystems, Inc

2 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 2 of 18 The Problem: /usr/local Software packages and other shared data Replication for high availability, scalability, network traffic Support for heterogeneous systems Consistency for failover.

3 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 3 of 18 Example: Sun s /usr/dist In each replica: - 11 Gigabytes of data 300,000 files packages 400 Servers worldwide in 6 level hierarchy Current distribution daily via hierarchical rdist rdist rdist

4 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 4 of 18 Problems with rdist Complex setup Clients must register with distribution server Point-to-point design (no hierarchical support) Unix only Recovery can be difficult No support for differential update Push rather than pull Other alternatives: - mirror (ftp via Perl script) - rsync (rsh)

5 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 5 of 18 NFS Hierarchy Overview Nodes are NFS clients/servers Data pulled down hierarchy by polling for changes Changes made only at master node End nodes are diskless clients or replicas themselves. NFS NFS

6 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 6 of 18 Advantages of NFS High capacity, kernel-resident server good performance! Concurrent service to multiple clients Easier setup & configuration Resistant to network/node failure NFS server not responding Protocol designed for remote access to file hierarchies - Access to directories, files, symbolic links, hard links NFS already installed or readily available from multiple vendors. Administrators already familiar with NFS /usr/local servers are already NFS servers.

7 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 7 of 18 Replication Hierarchy Detail Replication daemon runs at intermediate and leaf nodes Initial setup via network copy or tape Thereafter maintenance by automatic network updates Master Node archive ctl Poll Intermediate Node Updates archive ctl Poll Leaf Node Updates archive Updates ctl

8 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 8 of 18 upd.curr Control Directory upd.157 Updates update foo/bar update this/that update that/theother update file2 update file3 create file9 upd.157 upd.curr upd.156 upd.curr upd.156 upd.156 l upd.155 upd.154 l upd.153 l upd.152 : GETATTR Child polls file mtime lkjkjljlkjljlkj ljdljd dllkjkjljlkjl- lkjkjljlkjljlkj ljdljd dljdldkjdldkdl djldkdldkd- lkjkjljlkjljlkj ljdljd dllkjkjljlkjljlkj ljdljd dl- : upd.156 upd.155 upd.154 upd.153 upd.152 update fil3 update file4 update file6 update racoon update thisfilej update thatfile update file1 update file2 update file3 update file4 create file9 delete file11 delete file12 delete file13 update file27 delte usr/dist/two delete usr/dsit/three delete usr/dist/four delete usr/dist/five create file9 create file10 create file11 create file12 : upd.156 upd.155 upd.154 upd.153 upd.152

9 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 9 of 18 Update Files (contd) Child polls parent s ctl/upd.curr file for changed mtime When updates complete, child sets upd.curr mtime to parent s. Configuration is transitive when updates complete the child s archive (incl ctl) directories are identical to parent s Child may use several updates to catch up due to: - Network outage - Disconnection/reconnection (mobile computer)

10 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 10 of 18 Update File Syntax Operations to bring replica up to date, one per line. All pathnames relative to archive root. update pathname - Create or update a regular file, directory or symbolic link delete pathname - Remove file, directory (& subdirs) or symbolic link rename pathname1 pathname2 - Rename a file, directory or symlink link pathname1 pathname2 - Create a new hard link

11 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 11 of 18 copy pathname1 pathname2 - Make a copy of a file or directory exec command args... - Execute local command

12 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 12 of 18 Generating Updates Comparing client with server hierarchy can be expensive: - Walk /usr/dist tree with find : 17 minutes (mtimes & sizes) - Read all files: 105 minutes (checksums) - Repeated for each client - Used by rdist, rsync Compute once at server - Take snapshot - Make changes - Compare hierarchy with snapshot - Generate list of created, updated, deleted & renamed files - Add manual updates

13 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 13 of 18 Partial Update What if node loses contact with parent during update? Updates are non-idempotent - cannot be restarted. NFS is well known for dogged persistence. Just keep trying until server responds and resume update Assumption that update time << update frequency Can configure child to failover to grandparent.

14 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 14 of 18 Multiple Hierarchies Nodes can host multiple hierarchies - Each disjoint, configured independently

15 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 15 of 18 An Internet Hierarchy Internet distribution using NFS over TCP All servers export read-only Servers have no knowledge of clients.com.com.gov.edu.net.org.nz...au

16 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 16 of 18 Why NFS? Could build a hierarchy with SMB, AFS or DCE/DFS or any distributed filesystem that you have lying around the house. Easy setup for sites that already have NFS servers. NFS servers handle high loads with good response time - Specbench SFS 93: max 27,862 18ms (rw) NFS clients recover automatically from server crash or lost TCP connection. Can use NFS v2, though V3 provides larger transfers, piggyback attributes, READDIRPLUS better where latency is high.

17 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 17 of 18 NFS Security? General use of trusted host on Intranets - ro=engineering,rw=admin1 Used on Internet to export public data readonly Could use secure tunnels on Internet RPC security: Diffie-Hellman & Kerberos v4 key exchange - Not widely available RPCSEC-GSS - IETF Working Group - Implements pluggable security based on GSS-API - Authentication, Integrity, Privacy

18 W3C Push Technologies Workshop An NFS Replication Hierarchy Slide 18 of 18 Work in Progress Still a prototype - not yet deployed within Sun Need to meet all requirements of rdist system: - Update logging - Error reporting - Customized installations - In-place updates Requirement for stable server snapshot - When is it safe for client to download updates? - When is it safe for server to download updates? Automatic generation of updates on master - Compare one snapshot with another and emit updates, renames, deletes - Merging automatic updates with manual updates