CITI Technical Report 08-1 Parallel NFS Block Layout Module for Linux

Size: px

Start display at page:

Download "CITI Technical Report 08-1 Parallel NFS Block Layout Module for Linux"

Lesley Baker
6 years ago
Views:

1 CITI Technical Reprt 08-1 Parallel NFS Blck Layut Mdule fr Linux William A. Adamsn, University f Michigan andrs@citi.umich.edu Frederic Isaman, University f Michigan iisaman@citi.umich.edu Jasn Glasgw, EMC Glasgw_Jasn@emc.cm ABSTRACT This psitin statement presents CITI's Linux prttype f NFSv4.1 pnfs client blck layut mdule and reviews ur implementatin apprach. CITI's prttype implements the IETF draft specificatin draft-ietf-nfsv4-pnfs-blck and is ne f three layut mdules being develped alng with the Linux pnfs generic client, which implements the draft-ietf-nfsv4-minrversin1 specificatin. The blck layut mdule prvides fr an I/O data path ver iscsi directly t client SCSI devices identified by the pnfs blck server. February 12, 2008 Center fr Infrmatin Technlgy Integratin University f Michigan 535 W. William St., Suite 3100 Ann Arbr, MI

3 Intrductin Parallel NFS Blck Layut Mdule fr Linux William A. Adamsn, University f Michigan andrs@citi.umich.edu Frederic Isaman, University f Michigan iisaman@citi.umich.edu Jasn Glasgw, EMC Glasgw_Jasn@emc.cm This psitin statement presents CITI's Linux prttype f NFSv4.1 pnfs client blck layut mdule and reviews ur implementatin apprach. CITI's prttype implements the IETF draft specificatin draft-ietf-nfsv4-pnfs-blck and is ne f three layut mdules being develped alng with the Linux pnfs generic client, which implements the draft-ietf-nfsv4-minrversin1 specificatin. The blck layut mdule prvides fr an I/O data path ver iscsi directly t client SCSI devices identified by the pnfs blck server. CITI has als develped a Pythn-based NFSv4.1 test envirnment -- an LVM-based pnfs blck layut server that supprts SCSI disks emulated in RAM and the iscsi prtcl -- t test direct blck I/O and cmplex vlume tplgies alng with the pnfs and ther NFSv4.1 peratins. We refer t draft-ietf-nfsv4-minrversin1-17.txt Sectin 12 fr a detailed descriptin f NFSv4.1 Parallel NFS. Here are snippets f the intrductin t set the stage Intrductin pnfs is a set f ptinal features within NFSv4.1; the pnfs feature set allws direct client access t the strage devices cntaining file data. When file data fr a single NFSv4 server is stred n multiple and/r higher thrughput strage devices (by cmparisn t the server's thrughput capability), the result can be significantly better file access perfrmance. pnfs takes the frm f OPTIONAL peratins that manage prtcl bjects called 'layuts' which cntain data lcatin infrmatin. The NFSv4.1 pnfs feature has been structured t allw fr a variety f strage prtcls t be defined and used. The NFSv4.1 prtcl directly defines ne strage prtcl, the NFSv4.1 strage type, and its use. Examples f ther strage prtcls that culd be used with NFSv4.1's pnfs are: Blck/vlume prtcls such as iscsi ([35]), and FCP ([36]). The blck/vlume prtcl supprt can be independent f the addressing structure f the blck/vlume prtcl used, allwing mre than ne prtcl t access the same file data and enabling extensibility t ther blck/vlume prtcls. Object prtcls such as OSD ver iscsi r Fibre Channel [37]. Other strage prtcls, including PVFS and ther file systems that are in use in HPC envirnments.

4 The pnfs blck layut mdule is specified in draft-ietf-nfsv4-pnfs-blck-05.txt The pnfs peratins carry an paque paylad, which cnfrms t a strage prtcl layut type, between the pnfs client and the pnfs server. The Linux pnfs client and server prttype implementatins reflect this design. The Linux pnfs generic client perfrms tasks cmmn t all layut types. Per layut type paylads are paque t this generic cde. The generic cde passes an paque paylad t the apprpriate registered layut mdule via a layut mdule API, which runs cde specific t the layut type. The Linux NFSv4.1 pnfs server exprts pnfs capable file systems. The pnfs server design is similar t the pnfs client. The pnfs server perfrms tasks cmmn t all layut types. Opaque paylads are passed t and frm the exprted pnfs capable file system via a new set f peratins in struct exprt_peratins. A team f engineers frm CITI, Netwrk Appliance, Panasas, and IBM are implementing the Linux pnfs client and server prttypes. The Linux blck layut mdule is based n the EMC MPFS file system client, which shares many design pints with pnfs. The Linux pnfs client and server prttypes have been tested at fur interperability events where the Linux pnfs client demnstrated supprt fr simultaneus multiple layut mdules, and the Linux pnfs server was used t exprt bject (Panasas) and NFSv4.1 (IBM and Netwrk Appliance) based file systems. Prtcl Requirements Here is a list f the requirements placed n the pnfs blck layut client by the prtcl. We are cnsidering nly the iscsi strage prtcl transprt at this time. 1) Identify strage vlumes by cntent 2) Supprt arbitrarily cmplex vlume tplgies per file system id 3) Break dwn and reset lgical disk/vlume tplgy 4) I/O requires mapping file ffset extent vlume lgical ffset physical disk + ffset 5) Blck I/O t SCSI disk 6) 0 write size server file system blck size 7) Fail ver t NFSv4.1 server 8) Specify the maximum I/O time f the I/O stack 9) Cpy-n-write supprt Here are the kernel interfaces that ur blck layut mdule prttype uses t implement the requirements. Prttype Kernel Interfaces The first jb is t identify which f the SCSI disks the client can see belng t the pnfs server. 1) Identify strage vlumes by cntent. Our prttype des the fllwing. EXPORT drivers/scsi/hsts.c: shst_class symbl lists all SCSI devices. We walk the list, and identify all SCSI disks that can be pen_by_dev() and bd_claim(). We retrieve a list f device IDs frm the pnfs blck server and crrespnding cntent at an ffset. Fr each device ID, the prttype reads each SCSI disk cmparing cntent at the ffset. When we find a match, the pnfs server device ID is assciated with the disk. We bd_release() and clse all un-assciated disks. INTERFACE: a) shst_class b) pen_by_dev c) bd_claim d) bis read

5 The next three requirements inspired ur use f the LVM interface. 2) Supprt arbitrarily cmplex vlume tplgies per file system id 3) Break dwn and reset lgical disk/vlume tplgy 4) I/O requires mapping file ffset extent vlume lgical ffset physical disk + ffset These requirements seemed t be a gd match fr the user land LVM2 sftware. When ur prttype gets a vlume tplgy with disks identified via cntent, we call the LVM2 services in the kernel. Create a dm device (a la user land LVM ictl interface) t represent vlume tplgy. Fr I/O, the dm device handles the lgical ffset t physical disk + ffset mapping fr us. INTERFACE: create a dm device This des mst f what we want. We culd write ur wn dm target. We use the ictl interface in the kernel. Splitting dm functinality t service bth the existing ictl interface and the desired kernel interface remains an issue. The dm device gives us a device fr perfrming I/O. Next, we are faced with the need t chse the best kernel interface fr the fllwing requirements. 5) Blck I/O t SCSI disk 6) 0 write size server file system blck size 7) Fail ver t NFSv4.1 server INTERFACE: We have implemented tw paths: Use the blck_read/write_full_page() interface. We need ur wn callback rutine fr cleanup and errr recvery. Calls bis, and des it right! Large server file system blck size is an issue because we might have t deal with fs blck sizes larger than a page. Cde ur wn bis rutines. Write case is cmplex It is easy t cde ur wn callback rutine Issues We have yet t address these remaining requirements: 8) Specify the maximum I/O time f the I/O stack 9) Cpy-n-write supprt Knwing when t give up n I/O t a disk is an issue. We have nt yet addressed the cpy-n-write requirement. We hpe t discuss this and ther pnfs psitin statements at the Linux File Systems Wrkshp, which will help us mve pnfs int the Linux kernel ver the next 1 2 years

Dynamic Storage (ECS)

Dynamic Storage (ECS) User Guide Dynamic Strage (ECS) Swisscm (Schweiz) AG 1 / 10 Cntent 1 Abut Dynamic Strage... 3 2 Virtual drive, the EMC CIFS-ECS Tl... 4 3 Amazn S3 Brwer... 6 4 Strage Gateway Appliance... 9 5 Amazn S3