Fast Indexing: Support for Size-Changing Algorithms in Stackable File Systems

Transcription

1 Fas Indexing: Sppor for Size-Changing Algorihms in Sackable File Sysems Erez Zadok SUNY a Sony Brook ezk@cs.snysb.ed Johan M. Andersen, Ion Bădlesc, and Jason Nieh Colmbia Universiy Absrac Sackable file sysems can provide exensible file sysem fncionaliy wih minimal performance overhead and developmen cos. However, previos approaches provide only limied fncionaliy. In pariclar, hey do no sppor size-changing algorihms (SCAs), which are imporan and sefl for many applicaions sch as compression and encrypion. e propose fas indexing, a echniqe for efficien sppor of SCAs in sackable file sysems. Fas indexing provides a page mapping beween file sysem layers in a way ha can be sed wih any SCA. e se index files o sore his mapping. Index files are designed o be recoverable if los and add less han.1% disk space overhead. e have implemened fas indexing sing porable sackable emplaes, and we have sed his sysem o bild several example file sysems wih SCAs. e demonsrae ha fas index files have low overhead for ypical ser workloads sch as large compilaions, only 2.3% over oher sacked file sysems and 4.7% over non-sackable file sysems. Or sysem can deliver beer performance wih SCAs han serlevel applicaions, as mch as five imes faser. 1 Inrodcion Size-changing algorihms (SCAs) are hose ha ake as inp a sream of daa bis and prodce op of a differen nmber of bis. These SCAs share one qaliy in common: hey are generally inended o work on whole sreams of inp daa, from he beginning o he end of he sream. Some of he applicaions of sch algorihms fall ino several possible caegories: Compression: Algorihms ha redce he overall daa size o save on sorage space or ransmission bandwidhs. Encoding: Algorihms ha encode he daa sch ha i has a beer chance of being ransferred, ofen via , o Appeared in proceedings of he 21 Annal USENIX Technical Conference. is inended recipiens. For example, Uencode is an algorihm ha ses only he simples prinable ASCII characers and no more han 72 characers per line. In his caegory we also consider ransformaions o sppor inernaionalizaion of ex as well as nicoding. Encrypion: These are algorihms ha ransform he daa so i is more difficl o decode i wiho an ahorizaion a decrypion key. Encrypion algorihms can work in varios modes, some of which change he daa size while some modes do no [23]. Typically, encrypion modes ha increase daa size are also more secre. There are many sefl ser-level ools ha se SCAs, sch as encode, compress, and pgp. These ools work on whole files and are ofen sed manally by sers. This poses addiional inconvenience o sers. hen yo encryp or decompress a daa file, even if yo wish o access js a small par of ha file, yo sill have o encode or decode all of i nil yo reach he porion of ineres an acion ha consmes many resorces. SCAs do no provide informaion ha can help o decode or encode only he porion of ineres. Frhermore, rnning ser-level SCA ools repeaedly coss in addiional overhead as daa ms be copied beween he ser process and he kernel several imes. User-level SCA ools are herefore neiher ransparen o sers nor do hey perform well. Insead, i wold be sefl for a file sysem o sppor SCAs. File sysems are (1) ransparen o sers since hey do no have o rn special ools o se files, and (2) perform well since hey ofen rn in he kernel. File sysems have proven o be a sefl absracion for exending sysem fncionaliy. Several SCAs (ofen compression) have been implemened as exensions o exising disk-based file sysems [2, 3, 18]. Their disadvanages are ha hey only work wih specific operaing sysems and file sysems, and hey only sppor hose specific SCAs. Spporing general-prpose SCAs on a wide range of plaforms was no possible. 1

2 Sackable file sysems are an effecive infrasrcre for creaing new file sysem fncionaliy wih minimal performance overhead and developmen cos [1, 12, 22, 24, 28, 29, 25]. Sackable file sysems can be developed independenly and hen sacked on op of each oher o provide new fncionaliy. Also, hey are more porable and are easier o develop [29]. For example, an encrypion file sysem can be moned on op of a naive file sysem o provide secre and ransparen daa sorage [27]. Unfornaely, generalprpose SCAs have never been implemened in sackable file sysems. The problem we se o o solve was how o sppor general-prpose SCAs in a way ha is easy o se, performs well, and is available for many file sysems. e propose fas indexing as a solion for spporing SCAs in sackable file sysems. Fas indexing provide a way o map file offses beween pper and lower layers in sackable file sysems. Since he fas indexing is js a mapping, a lower-layer file sysem does no have o know anyhing abo he deails of he SCA sed by an pper-level file sysem. e sore his fas indexing informaion in index files. Each encoded file has a corresponding index file which is simply sored in a separae file in he lower-layer file sysem. The index file is mch smaller han he original daa file, resling in negligible sorage reqiremens. The index file is designed o be recoverable if i is somehow los so ha i does no compromise he reliabiliy of he file sysem. Finally, fas indexing is designed o deliver good file sysem performance wih low sacking overhead, especially for common file operaions. e have implemened fas indexing sing sackable emplaes [28, 29, 25]. This allows s o provide ransparen sppor for SCAs in a porable way. To demonsrae he effeciveness of or approach, we bil and esed several size-changing file sysems, inclding a compression file sysem. Or performance resls show (1) ha fas index files have low overhead for ypical file sysem workloads, only 2.3% over oher nll-layer sackable file sysems, and (2) ha sch file sysems can deliver as mch as five imes beer performance han ser-level SCA applicaions. This paper describes fas index files and is organized as follows. Secion 2 reviews he sacking file-sysem infrasrcre sed for his work and discsses relaed work in SCA sppor in file sysems. Secion 3 deails he design of he index file. Secion 4 describes he sage of he index file in relaion o common file operaions and discsses several opimizaions. Secion 5 deails or design for a consisen and recoverable index file. Secion 6 smmarizes imporan implemenaion isses. Secion 7 describes he file sysems we bil sing his work and evalaes or sysem. Finally, we presen conclsions and discss direcions for fre work. 2 Backgrond In his secion we discss exensibiliy mechanisms for file sysems, wha wold be reqired for sch file sysems o sppor SCAs, and oher sysems ha provide some sppor for compression SCAs. 2.1 Sacking Sppor Sackable file sysems allow for modlar, incremenal developmen of file sysems by layering addiional fncionaliy on anoher file sysem [13, 15, 21, 24]. Sacking provides an infrasrcre for he composiion of mliple file sysems ino one. User Process wrie() CompressFS EXT2FS FS_RITE() iral File Sysem (FS) compressfs_wrie() ex2fs_wrie() User Kernel Figre 1: An example sackable compression file sysem. A sysem call is ranslaed ino a generic FS fncion, which is ranslaed ino a file-sysem specific fncion in or sackable compression file sysem. CompressFS hen modifies (compresses) he daa passed o i and calls he file sysem sacked below i wih he modified daa. Figre 1 shows he srcre for a simple single-level sackable compression file sysem called CompressFS. Sysem calls are ranslaed ino FS calls, which in rn invoke heir CompressFS eqivalens. CompressFS receives ser daa o be wrien. I compresses he daa and passes i o he nex lower layer, wiho any regard o wha ype of file sysem implemens ha layer. Sackable file sysems were designed o be modlar and ransparen: each layer is independen from he layers above and below i. In ha way, sackable file sysem modles cold be composed ogeher in differen configraions o provide new fncionaliy. Unfornaely, his poses problems for SCAs becase he decoded daa a he pper layer has differen file offses from he encoded daa a he lower layer. CompressFS, for example, ms know how mch compressed daa i wroe, where i wroe i, and wha original offses in he decoded file did ha daa represen. Those pieces of informaion are necessary so ha sbseqen reading operaions can locae he daa qickly. 2

3 If CompressFS canno find he daa qickly, i may have o resor o decompression of he complee file before i can locae he daa o read. Therefore, o sppor SCAs in sackable file sysems, a sackable layer ms have some informaion abo he encoded daa offse informaion. B a sackable file sysem ha ges ha informaion abo oher layers violaes is ransparency and independence. This is he main reason why pas sacking works do no sppor SCAs. The challenge we faced was o add general-prpose SCA sppor o a sacking infrasrcre wiho losing he benefis of sacking: a sackable file sysem wih SCA sppor shold no have o know anyhing abo he file sysem i sacks on. Tha way i can add SCA fncionaliy aomaically o any oher file sysem. 2.2 Compression Sppor Compression file sysems are no a new idea. indows NT sppors compression in NTFS [18]. E2compr is a se of paches o Linx s Ex2 file sysem ha add block-level compression [2]. Compression exensions o log-srcred file sysems resled in halving of he sorage needed while degrading performance by no more han 6% [3]. The benefi of block-level compression file sysems is primarily speed. Their main disadvanage is ha hey are specific o one operaing sysem and one file sysem, making hem difficl o por o oher sysems and resling in code ha is hard o mainain. The ATTIC sysem demonsraed he seflness of aomaic compression of leas-recenly-sed files [5]. I was implemened as a modified ser-level NFS server. hereas i provided porable code, in-kernel file sysems ypically perform beer. In addiion, he ATTIC sysem decompresses whole files which slows performance. HURD [4] and Plan 9 [19] have an exensible file sysem inerface and have sggesed he idea of sackable compression file sysems. Their primary focs was on he basic minimal exensibiliy infrasrcre; hey did no prodce any working examples of size-changing file sysems. Spring [14, 16] and Fics [11] discssed a similar idea for implemening a sackable compression file sysem. Boh sggesed a nified cache manager ha can aomaically map compressed and ncompressed pages o each oher. Heidemann s Fics work provided addiional deails on mapping cached pages of differen sizes. 1 Unfornaely, no demonsraion of hese ideas for compression file sysems was available from eiher of hese works. In addiion, no consideraion was given o arbirary SCAs and 1 Heidemann s earlier work [13] menioned a prooype compression layer bil dring a class projec. In personal commnicaions wih he ahor, we were old ha his prooype was implemened as a block-level compression file sysem, no a sackable one. how o efficienly handle common file operaions sch as appends, looking p file aribes, ec. 3 The Index File In a sacking environmen ha sppors SCAs, daa offses may change arbirarily. An efficien mapping is needed ha can ell where he saring offse of he encoded daa is for a given offse in he original file. e call his mapping he index able. The index able is sored in a separae file called he index file, as shown in Figre 2. This file serves as he fas meadaa index ino an encoded file. For a given daa file, we creae an index file called.idx. Many file sysems separae daa and mea daa; his is done for efficiency and reliabiliy. Mea-daa is considered more imporan and so i ges cached, sored, and pdaed differenly han reglar daa. The index file is separae from he encoded file daa for he same reasons and o allow s o manage each par separaely and simply. Decoded (original) File Encoded Daa File Index File Upper Layer Lower Layer Figre 2: Overall srcre of size-changing sackable file sysems. Each original daa file is encoded ino a lower daa file. Addiional mea-daa index informaion is sored in an index file. Boh he index file and he encoded daa files reside in he lower level file sysem. Or sysem encodes and decodes whole pages or heir mliples which maps well o file sysem operaions. The index able assmes page-based operaions and sores offses of encoded pages as hey appear in he encoded file. Consider an example of a file in a compression file sysem. Figre 3 shows he mapping of offses beween he pper (original) file and he lower (encoded) daa file. To find o he byes in page 2 of he original file, we read he daa byes 3 72 in he encoded daa file, decode hem, and rern o he FS ha daa in page 2. To find o which encoded byes we need o read from he lower file, we consl he index file, shown in Table 1. The index file ells s ha he original file has 6 pages, ha is original size is 215 byes, and hen i liss he ending offses of he encoded daa for an pper page. Finding he lower offses for he pper page 2 is a simple linear dereferencing of he daa in he index file; we do no have o search he index file linearly. Noe ha or design of he 3

4 Decoded File (pper) 4K 8K 12K 16K Page Page 1 Page 2 Page 3 Page 4 Page 5 11 Page Page 1 Page 2 3 4K K (EOF) Encoded File (lower) 215 (EOF) 2K Figre 3: An example of a 32-bi file sysem ha shrinks daa size (compression). Each pper page is represened by an encoded lower chnk. The mapping of offses is shown in Table 1. index file sppors boh 32-bi and 64-bi file sysems, b he examples we provide here are for 32-bi file sysems. ord Represening Reglar ih Fas (32/64 bis) IDX File Tail (f) 1 (12 bis) flags ls=,f=,... ls=,f=1,... 1 (2/52 bis) # pages orig. file size page page page page page page Table 1: Forma of he index file for Figres 3 and 4. Fas Tails are described in Secion 4.2. The firs word encodes boh flags and he nmber of pages in he index file. The ls (large size) flag is he firs bi in he index file and indicaes if he index file encodes a 32-bi () or 64-bi (1) file sysem. The index file sars wih a word ha encodes flags and he nmber of pages in he corresponding original daa file. e reserve he lower 12 bis for special flags sch as wheher he index file encodes a file in a 32-bi or a 64- bi file sysem, wheher fas ails were encoded in his file (see Secion 4.2), ec. The very firs bi of hese flags, and herefore he firs bi in he index file, deermines if he file encoded is par of a 32-bi or a 64-bi file sysem. This way, js by reading he firs bi we can deermine how o inerpre he res of he index file: 4 byes o encode page offses on 32-bi file sysems or 8 byes o encode page offses on 64-bi file sysems. e se he remaining 2 bis (on a 32-bi file sysem) for he nmber of pages becase 4KB pages (he ypical page size on i386 and SPARCv8 sysems) wold give s he exac maximm file size we can encode in 4 byes on a 32-bi file sysem, as explained nex; similarly 4KB pages is he exac maximm file size on a 64-bi file sysem. The index file also conains he original file s size in he second word. e sore his informaion in he index file so ha commands like ls -l and ohers sing sa(2) wold work correcly; a process looking a he size of he file hrogh he pper level file sysem wold ge he original nmber of byes and blocks. The original file s size can be comped from he saring offse of he las daa chnk in he encoded file, b i wold reqire decoding he las (possibly incomplee) chnk (byes in he encoded file in Figre 3) which can be an expensive operaion depending on he SCA. Soring he original file size in he index file is a speed opimizaion ha only consmes one more word in a physical daa block ha mos likely was already allocaed. The index file is small. e sore one word (4 byes on a 32-bi file sysem) for each daa page (sally 496 byes). On average, he index able size is 124 imes smaller han he original daa file. For example, an index file ha is exacly 496 byes long (one disk block on an Ex2 file sysem formaed wih 4KB blocks) can describe an original file size of 122 pages, or 4,186,112 byes (almos 4MB). Since he index file is relaively small, we read i ino kernel memory as soon as he main file is open and maniplae i here. Tha way we have fas access o he index daa in memory. The index informaion for each page is sored linearly and each index enry ypically akes 4 byes. Tha les s compe he needed index informaion simply and find i from he index able sing a single dereference ino an array of 4-bye words (inegers). To improve performance frher, we wrie he final modified index able only afer he original file is closed and all of is daa flshed o sable media. The size of he index file is less imporan for SCAs which increase he daa size, sch as nicoding, encoding, and some forms of encrypion. The more he SCA increases he daa size, he less significan he size of he index file becomes. Even in he case of SCAs ha decrease daa size (e.g., compression) he size of he index file may no be as imporan given he savings already gained from compression. Since he index informaion is sored in a separae file, i ses p one more inode. e measred he effec ha he consmpion of an addiional inode wold have on ypical file sysems in or environmen. e fond ha disk daa block sage is ofen 6 8 imes greaer han inode ilizaion on disk-based file sysems, leaving pleny of free inodes o se. To save resorces even frher, we efficienly sppor zero-lengh files: a zero-lengh original daa file is represened by a zero-lengh index file. For reliabiliy reasons, we designed he index file so i cold be recovered from he daa file in case he index file is los or damaged (Secion 5.) This offers cerain improvemens over ypical Unix file sysems: if heir mea-daa (inodes, inode and indirec blocks, direcory daa blocks, ec.) 4

5 is los, i rarely can be recovered. Noe ha he index file is no needed for or sysem o fncion: i represens a performance enhancing ool. iho he index file, sizechanging file sysems wold perform poorly. Therefore, if i does no exis (or is los), or sysem aomaically regeneraes he index file. 4 File Operaions e now discss he handling of file sysem operaions in fas indexing as well as specific opimizaions for common operaions. Noe ha mos of his design relaes o performance opimizaions while a small par (Secion 4.4) addresses correcness. Becase he cos of SCAs can be high, i is imporan o ensre ha we minimize he nmber of imes we invoke hese algorihms and he nmber of byes hey have o process each ime. The way we sore and access encoded daa chnks can affec his performance as well as he ypes and freqencies of file operaions. As a resl, fas indexing akes ino accon he fac ha file accesses follow several paerns: The mos poplar file sysem operaion is sa(2), which resls in a file lookp. Lookps accon for 4 5% of all file sysem operaions [17, 2]. Mos files are read, no wrien. The raio of reads o wries is ofen 4 6 [17, 2]. For example, compilers and ediors read in many header and configraion files, b only wrie o a handfl of files. Files ha are wrien are ofen wrien from beginning o end. Compilers, ser ools sch as cp, and ediors sch as emacs wrie whole files in his way. Frhermore, he ni of wriing is sally se o mach he sysem page size. e have verified his by rnning a se of common edi and bild ools on Linx and recorded he wrie sar offses, size of wrie bffers, and he crren size of he file. Files ha are no wrien from beginning o end are ofen appended o. The nmber of appended byes is sally small. This is re for varios log files ha reside in /var/log as well as eb server logs. ery few files are wrien in he middle. This happens in wo cases. Firs, when he GNU ld creaes large binaries, i wries a sparse file of he arge size and hen seeks and wries he res of he file in a non-seqenial manner. To esimae he freqency of wries in he middle, we insrmened a nll-layer file sysem wih a few coners. e hen measred he nmber and ype of wries for or large compile benchmark (Secion 7.3.1). e coned 9193 wries, of which 58 (.6%) were wries before he end of a file. Second, daa-base files are also wrien in he middle. e srveyed or own sie s file servers and worksaions (several hndred hoss oaling over 1TB of sorage) and fond ha hese files represened less han.15% of all sorage. Of hose, only 2.4% were modified in he pas 3 days, and only 3% were larger han 1MB. All oher operaions (ogeher) accon for a small fracion of file operaions [17, 2]. e designed or sysem o opimize performance for he more common and imporan cases while no harming performance ndly when he seldom-execed cases occr. e firs describe how he index file is designed o sppor fas lookps, file reads, and whole file wries, which ogeher are he mos common basic file operaions. e hen discss sppor for appending o files efficienly, handling he less common operaion of wries in he middle of files, and ensring correcness for he infreqen se of rncae. 4.1 Basic Operaions To handle file lookps fas, we sore he original file s size in he index able. De o localiy in he creaion of he index file, we assme ha is name will be fond in he same direcory block as he original file name, and ha he inode for he index file will be fond in he same inode block as he encoded daa file. Therefore reading he index file reqires reading one addiional inode and ofen only one daa block. Afer he index file is read ino memory, rerning he file size is done by copying he informaion from he index able ino he size field in he crren inode srcre. Remaining aribes of he original file come from he inode of he acal encoded file. Once we read he index able ino memory, we allow he sysem o cache is daa for as long as possible. Tha way, sbseqen lookps will find files aribes in he aribe cache. Since mos file sysems are srcred and implemened inernally for access and caching of whole pages, we also encode he original daa file in whole pages. This improved or performance and helped simplify or code becase inerfacing wih he FS and he page cache was more naral. For file reads, he cos of reading in a daa page is fixed: a fixed offse lookp ino he index able gives s he offses of encoded daa on he lower level daa file; we read his encoded seqence of byes, decode i ino exacly one page, and rern ha decoded page o he ser. Using enire pages made i easier for s o wrie whole files, especially if he wrie ni was one page size. In he case of whole file wries, we simply encode each page size ni, add i o he lower level encoded file, and add one more enry o he index able. e discss he cases of file appends and wries in he middle in Secions 4.2 and 4.3, respecively. 5

6 e did no have o design anyhing special for handling all oher file operaions. e simply rea he index file a he same ime we maniplae he corresponding encoded daa file. An index file is creaed only for reglar files; we do no have o worry abo symbolic links becase he FS will only call or file sysem o open a reglar file. hen a file is hard-linked, we also hard-link he index file sing he name of he new link wih a he.idx exension added. hen a file is removed from a direcory or renamed, we apply he same operaion o he corresponding index file. 4.2 Fas Tails One common sage paern of files is o append o hem. Ofen, a small nmber of byes is appended o an exising file. Encoding algorihms sch as compression and encrypion are more efficien when hey encode larger chnks of daa. Therefore i is beer o encode a larger nmber of byes ogeher. Or design calls for encoding whole pages whenever possible. Table 1 and Figre 3 show ha only he las page in he original file may be incomplee and ha incomplee page ges encoded oo. If we append, say, 1 more byes o he original (pper) file of Figre 3, we have o keep i and he index file consisen: we ms read he 12 byes from 248 nil 215, decode hem, add he 1 new byes, encode he new 13 seqence of byes, and wrie i o in place of he older 12 byes in he lower file. e also have o pdae he index able in wo places: he oal size of he original file is now 2151, and word nmber 8 in he index file may be in a differen locaion han 112 (depending on he encoding algorihm, i may be greaer, smaller, or even he same). The need o read, decode, append, and re-encode a chnk of byes for each append grows worse as he nmber of byes o append is small while he nmber of encoded byes is closer o one fll page. In he wors case, his mehod yields a complexiy of in he nmber of byes ha have o be decoded and encoded, mliplied by he cos of he encoding and decoding of he SCA. To solve his problem, we added a fas ails rnime mon opion ha allows for p o a page size worh of nencoded daa o be added o an oherwise encoded daa file. This is shown in he example in Figre 4. In his example, he las fll page ha was encoded is page 4. Is daa byes end on he encoded daa file a offse 1 (page 2). The las page of he original pper file conains 12 byes (215 less 2K). So we sore hese 12 byes direcly a he end of he encoded file, afer offse 1. To aid in comping he size of he fas ail, we add wo more byes o he end of he file pas he fas ail iself, lising he lengh of he fas ail. (Two byes is enogh o lis his lengh since ypical page sizes are less han byes long.) The final size of he encoded file is now 1122 byes long. Decoded File (pper) 4K 8K 12K 16K Page Page 1 Page 2 Page 3 Page 4 Page 5 11 Page Page 1 Page 2 3 4K K 1 Fas Tail 1122 (EOF) FT Size Encoded File (lower) 215 (EOF) 2K Figre 4: Size-changed file srcre wih fas-ail opimizaion. A file sysem similar o Figre 3, only here we sore p o one page fll of nencoded raw daa. hen enogh raw daa is colleced o fill a whole fas-ail page, ha page is encoded. ih fas ails, he index file does no record he offse of he las ail as can be seen from he righ-mos colmn of Table 1. The index file, however, does record in is flags field (firs 12 bis of he firs word) ha a fas ail is in se. e p ha flag in he index able o speed p he compaions ha depend on he presence of fas ails. e append he lengh of he fas ail o he encoded daa file o aid in reconsrcion of a poenially los index file, as described in Secion 5. hen fas ails are in se, appending a small nmber of byes o an exising file does no reqire daa encoding or decoding, which can speed p he append operaion considerably. hen he size of he fas ail exceeds one page, we encode he firs page worh of byes, and sar a new fas ail. Fas ails, however, may no be desirable all he ime exacly becase hey sore nencoded byes in he encoded file. If he SCA sed is an encrypion one, i is insecre o expose plainex byes a he end of he cipherex file. For his reason, fas ails is a rnime global mon opion ha affecs he whole file sysem moned wih i. The opion is global becase ypically sers wish o change he overall behavior of he file sysem wih respec o his feare, no on a per-file basis. 4.3 rie in he Middle User processes can wrie any nmber of byes in he middle of an exising file. ih or sysem, whole pages are encoded and sored in a lower level file as individal encoded chnks. A new se of byes wrien in he middle of he file may encode o a differen nmber of byes in he lower level file. If he nmber of new encoded byes is greaer han he old nmber, we shif he remaining encoded file oward o make room for he new byes. If he nmber of 6

7 byes is smaller, we shif he remaining encoded file inward o cover nsed space. In addiion, we adjs he index able for each encoded daa chnk which was shifed. e perform shif operaions as soon as or file sysem s wrie operaion is invoked, o ensre seqenial daa consisency of he file. To improve performance, we shif daa pages in memory and keep hem in he cache as long as possible: we avoid flshing hose daa pages o disk and le he sysem decide when i wans o do so. Tha way, sbseqen wrie-inhe-middle operaions ha may resl in addiional inward or oward shifs will only have o maniplae daa pages already cached and in memory. Any daa page shifed is marked as diry, and we le he paging sysem flsh i o disk when i sees fi. Noe ha daa ha is shifed in he lower level file does no have o be re-encoded. This is becase ha daa sill represens he acal encoded chnks ha decode ino heir respecive pages in he pper file. The only hing remaining is o change he end offses for each shifed encoded chnk in he index file. e examined several performance opimizaion alernaives ha wold have encoded he informaion abo inward or oward shifs in he index able or possibly in some of he shifed daa. e rejeced hem for several reasons: (1) i wold have complicaed he code considerably, (2) i wold have made recovery of an index file difficl, and (3) i wold have resled in fragmened daa files ha wold have reqired a defragmenaion procedre. Since he nmber of wries in he middle we measred was so small (.6% of all wries), we do consider or simplified design as a good cos vs. performance balance. Even wih or simplified solion, or file sysems work perfecly correcly. Secion shows he benchmarks we ran o es wries in he middle and demonsraes ha or solion prodces good overall performance. 4.4 Trncae One design isse we faced was wih he rncae(2) sysem call. Alhogh his call occrs less han.2% of he ime [17, 2], we sill had o ensre ha i behaved correcly. Trncae can be sed o shrink a file as well as enlarge i, poenially making i sparse wih new holes. e deal wih for cases: 1. Trncaing on a page bondary. In his case, we rncae he encoded file exacly afer he end of he chnk ha now represens he las page of he pper file. e pdae he index able accordingly: i has fewer pages in i. 2. Trncaing in he middle of an exising page. This case resls in a parial page: we read and decode he whole page and re-encode he byes wihin he page represening he par before he rncaion poin. e pdae he index able accordingly: i now has fewer pages in i. 3. Trncaing in he middle of a fas ail. In ha case we js rncae he lower file where he fas ail is acally locaed. e hen pdae he size of he fas ail a is end and pdae he index file o indicae he (now) smaller size of he original file. 4. Trncaing pas he end of he file is akin o exending he size of he file and possibly creaing zero-filled holes. e read and re-encode any parially filled page or fas ail ha sed o be a he end of he file before he rncaion; we have o do ha becase ha page now conains a mix of non-zero daa and zeroed daa. e encode all sbseqen zero-filled pages. This is imporan for some applicaions sch as encrypion, where every bi of daa zeros or oherwise shold be encryped. 5 Index File Consisency ih he inrodcion of a separae index file o sore he index able, we now have o mainain wo files consisenly. Normally, when a file is creaed, he direcory of ha file is locked. e keep boh he direcory and he encoded daa file locked when we pdae he index file. This way boh he encoded daa file and he index file are garaneed o be wrien correcly. e assme ha encoded daa files and index files will no become corrp inernally de o media failres. This siaion is no worse han normal file sysems where a random daa corrpion may no be possible o fix. However, we do concern orselves wih hree poenial problems wih he index file: parially wrien file, a los file, and rivial corrpions. An index file cold be parially wrien if he file sysem is fll or he ser ran o of qoa. In he case where we were nable o wrie he complee index file, we simply remove i and log a warning message hrogh syslog(3) where he message cold be passed on o a cenralized logging faciliy ha moniors and generaes appropriae alers. The absence of he index file on sbseqen file accesses will rigger an in-kernel mechanism o recover he index file. Tha way he index file is no necessary for or sysem o fncion; i only aids in improving performance. An index file cold be los if i was removed inenionally (say afer a parial wrie) or ninenionally by a ser direcly from he lower file sysem. If he index file is los or does no exis, we can no longer easily ell where encoded byes were sored. In he wors case, wiho an index file, we have o decode he complee file o locae any 7

8 arbirary bye wihin. However, since he cos of decoding a complee file and regeneraing an index able are nearly idenical (see Secion 7.6), we chose o regenerae he index able immediaely if i does no exis, and hen proceed as sal as he index file now exiss. e verify he validiy of he index file when we se he index able. e check ha all index enries are monoonically increasing, ha i has he correc nmber of enries, file size maches he las enry, flags sed are known, ec. The index file is regeneraed if an inconsisency is deeced. This helps or sysem o srvive cerain mea-daa corrpions ha cold occr as a resl of sofware bgs or direc ediing of he index file. e designed or sysem so ha he index file can be recovered reliably in all cases. For imporan pieces of informaion are needed o recover an index file given an encoded daa file. These for are available in he kernel o he rnning file sysem: 1. The SCA sed. 2. The page size of he sysem on which he encoded daa file was creaed. 3. heher he file sysem sed is 32-bi or 64-bi. 4. heher fas ails were sed. To recover an index file we read an inp encoded daa file and decode he byes nil we fill o one whole page of op daa. e rely on he fac ha he original daa file was encoded in nis of page size. The offse of he inp daa where we finished decoding ono one fll page becomes he firs enry in he index able. e conine reading inp byes and prodce more fll pages and more index able enries. If fas ails were sed, hen we read he size of he fas ail from he las wo byes of he encoded file, and we do no ry o decode i since i was wrien nencoded. If fas ails were no sed and we reached he end of he inp file, ha las chnk of byes may no decode o a whole op page. In ha case, we know ha was he end of he original file, and we mark he las page in he index able as a parial page. hile we are decoding pages, we sm p he nmber of decoded byes and fas ails, if any. The oal is he original size of he daa file, which we record in he index able. e now have all he informaion necessary o wrie he correc index file and we do so. 6 SCA Implemenaion Or SCA sppor was inegraed ino FiST [29, 25]. The FiST sysem incldes porable sackable file sysem emplaes for several operaing sysems as well as a highlevel langage ha can describe new sackable file sysems [26, 28]. Mos of he work was p ino he sackable emplaes, where we added sbsanially more code o sppor SCAs: 2119 non-commen lines of C code, represening a 6% increase in he size of he emplaes. Becase his addiional code is sbsanial and carries an overhead wih i ha is no needed for non-size-changing file sysems (Secion 7), we made i opional. To sppor ha, we added one addiional declaraion o he FiST langage, o allow developers o decide wheher or no o inclde his addiional sppor. To se FiST o prodce a size-changing file sysem, developers need o inclde a single FiST declaraion in heir inp file and hen wrie only wo roines: encode daa and decode daa. The main advanage of sing FiST for his work has been he ease of se for developers ha wan o wrie size-changing file sysems. All he complexiy is placed in he emplaes and is mosly hidden from developers view. Developers need only concenrae on he core implemenaion isses of he pariclar algorihm hey wish o se in heir new file sysem. The FiST sysem has been pored o Linx, Solaris, and FreeBSD. Crren SCA sppor is available for Linx 2.3 only. Or primary goal in his work was o prove ha sizechanging sackable file sysems can be designed o perform well. hen we feel ha he design is sable and addresses all of he algorihmic isses relaed o he index file, we will por i o he oher emplaes. e wold hen be able o describe an SCA file sysem once in he FiST langage; from his single porable descripion, we cold hen prodce a nmber of working file sysems. There are wo implemenaion-specific isses of ineres: one concerning Linx and he oher regarding wries in he middle of files. As menioned in Secion 3, we wrie any modified index informaion o when he main file is closed and is daa flshed o sable media. In Linx, neiher daa nor mea-daa are aomaically flshed o disk. Insead, a kernel hread (kflshd) rns every 5 seconds and asks he page cache o flsh any file sysem daa ha has no been sed recenly, b only if he sysem needs more memory. In addiion, file daa is forced o disk when eiher he file sysem is nmoned or he process called an explici fflsh(3) or fsync(2). e ake advanage of his delayed wrie o improve performance, since we wrie he index able when he res of he file s daa is wrien. To sppor wries in he middle correcly, we have o make an exra copy of daa pages ino a emporary locaion. The problem is ha when we wrie a daa page given o s by he FS, we do no know wha his daa page will encode ino, and how mch space ha new encoding wold reqire. If i reqires more space, hen we have o shif daa oward in he encoded daa file before wriing he new daa. For his firs implemenaion, we chose he simplified approach of always making he emporary copy, which affecs performance as seen in Secion 7. hile or code shows good performance, i has no been opimized mch ye; we discss avenes of fre work in Secion 9. 8

9 7 Evalaion To evalae fas indexing in a real world operaing sysem environmen, we bil several SCA sackable file sysems based on fas indexing. e hen condced exensive measremens in Linx comparing hem agains non-sca file sysems on a variey of file sysem workloads. In his secion we discss he experimens we performed on hese sysems o (1) show overall performance on general-prpose file sysem workloads, (2) deermine he performance of individal common file operaions and relaed opimizaions, and (3) compare he efficiency of SCAs in sackable file sysems o eqivalen ser-level ools. Secion 7.1 describes he SCA file sysems we bil and or experimenal design. Secion 7.2 describes he file sysem workloads we sed for or measremens. Secions 7.3 o 7.6 presen or experimenal resls. 7.1 Experimenal Design e ran or experimens on five file sysems. e bil hree SCA file sysems and compared heir performance o wo non-sca file sysems. The hree SCA file sysems we bil were: 1. Copyfs: his file sysem simply copies is inp byes o is op wiho changing daa sizes. Copyfs exercises all of he index-managemen algorihms and oher SCA sppor wiho he cos of encoding or decoding pages. 2. Uencodefs: his is a file sysem ha sores files in encoded forma and decodes files when hey are read. I is inended o illsrae an algorihm ha increases he daa size. This simple algorihm convers every 3-bye seqence ino a 4-bye seqence. Uencode prodces 4 byes ha can have a mos 64 vales each, saring a he ASCII characer for space (2 ). e chose his algorihm becase i is simple and ye increases daa size significanly (by one hird). 3. Gzipfs: his is a compression file sysem sing he Deflae algorihm [7] from he zlib package [9]. This algorihm is sed by GNU zip (gzip) [8]. This file sysem is inended o demonsrae an algorihm ha (sally) redces daa size. The wo non-sca file sysems we sed were Ex2fs, he naive disk-based file sysem mos commonly sed in Linx, and rapfs, a sackable nll-layer file sysem we rivially generaed sing FiST [25, 29]. Ex2fs provides a measre of base file sysem performance wiho any sacking or SCA overhead. rapfs simply copies he daa of files beween layers b does no inclde SCA sppor. By comparing rapfs o Ex2fs, we can measre he overhead of sacking and copying daa wiho fas indexing and wiho changing is conen or size. Copyfs copies daa like rapfs b ses all of he SCA sppor. By comparing Copyfs o rapfs, we can measre he overhead of basic SCA sppor. By comparing Uencodefs o Copyfs, we can measre he overhead of an SCA algorihm incorporaed ino he file sysem ha increases daa size. Similarly, by comparing Gzipfs o Copyfs, we can measre he overhead of a compression file sysem ha redces daa size. One of he primary opimizaions in his work is fas ails as described in Secion 4.2. For all of he SCA file sysems, we ran all of or ess firs wiho fail-ails sppor enabled and hen wih i. e repored resls for boh whenever fas ails made a difference. All experimens were condced on for eqivalen 433Mhz Inel Celeron machines wih 128MB of RAM and a Qanm Fireball lc1 9.8GB IDE disk drive. e insalled a Linx pre3 kernel on each machine. Each of he for sackable file sysems we esed was moned on op of an Ex2 file sysem. For each benchmark, we only read, wroe, or compiled he es files in he file sysem being esed. All oher ser iliies, compilers, headers, and libraries resided oside he esed file sysem. Unless oherwise noed, all ess were rn wih a cold cache. To ensre ha we sed a cold cache for each es, we nmoned all file sysems which paricipaed in he given es afer he es compleed and moned he file sysems again before rnning he nex ieraion of he es. e verified ha nmoning a file sysem indeed flshes and discards all possible cached informaion abo ha file sysem. In one benchmark we repor he warm cache performance, o show he effeciveness of or code s ineracion wih he page and aribe caches. e ran all of or experimens 1 imes on an oherwise qie sysem. e measred he sandard deviaions in or experimens and fond hem o be small, less han 1% for mos micro-benchmarks described in Secion 7.2. e repor deviaions which exceeded 1% wih heir relevan benchmarks. 7.2 File Sysem Benchmarks e measred he performance of he five file sysems on a variey of file sysem workloads. For or workloads, we sed five file sysem benchmarks: wo general-prpose benchmarks for measring overall file sysem performance, and hree micro-benchmarks for measring he performance of common file operaions ha may be impaced by fas indexing. e also sed he micro-benchmarks o compare he efficiency of SCAs in sackable file sysems o eqivalen ser-level ools. 9

10 7.2.1 General-Prpose Benchmarks Am-ils: The firs benchmark we sed o measre overall file sysem performance was am-ils (The Berkeley Aomoner) [1]. This benchmark configres and compiles he large am-ils sofware package inside a given file sysem. e sed am-ils-6..4: i conains over 5, lines of C code in 96 files. The bild process begins by rnning several hndred small configraion ess inended o deec sysem feares. I hen bilds a shared library, abo en binaries, for scrips, and docmenaion: a oal of 265 addiional files. Overall his benchmark conains a large nmber of reads, wries, and file lookps, as well as a fair mix of mos oher file sysem operaions sch as nlink, mkdir, and symlink. Dring he linking phase, several large binaries are linked by GNU ld. The am-ils benchmark is he only es ha we also ran wih a warm cache. Or sackable file sysems cache decoded and encoded pages whenever possible, o improve performance. hile normal file sysem benchmarks are done sing a cold cache, we also fel ha here is vale in showing wha effec or caching has on performance. This is becase ser level SCA ools rarely benefi from page caching, while file sysems are designed o perform beer wih warm caches; his is wha sers will experience in pracice. Bonnie: The second benchmark we sed o measre overall file sysem performance was Bonnie [6], a file sysem es ha inensely exercises file daa reading and wriing, boh seqenial and random. Bonnie is a less general benchmark han am-ils. Bonnie has hree phases. Firs, i creaes a file of a given size by wriing i one characer a a ime, hen one block a a ime, and hen i rewries he same file 124 byes a a ime. Second, Bonnie wries he file one characer a a ime, hen a block a a ime; his can be sed o exercise he file sysem cache, since cached pages have o be invalidaed as hey ge overwrien. Third, Bonnie forks 3 processes ha each perform 4 random lseeks in he file, and read one block; in 1% of hose seeks, Bonnie also wries he block wih random daa. This las phase exercises he file sysem qie inensively, and especially he code ha performs wries in he middle of files. For or experimens, we ran Bonnie sing files of increasing sizes, from 1MB and dobling in size p o 128MB. The las size is imporan becase i mached he available memory on or sysems. Rnning Bonnie on a file ha large is imporan, especially in a sackable seing where pages are cached in boh layers, becase he page cache shold no be able o hold he complee file in memory Micro-Benchmarks File-copy: The firs micro-benchmark we sed was designed o measre file sysem performance on ypical blk file wries. This benchmark copies files of differen sizes ino he file sysem being esed. Each file is copied js once. Becase file sysem performance can be affeced by he size of he file, we exponenially varied he sizes of he files we ran hese ess on from byes all he way o 32MB files. File-append: The second micro-benchmark we sed was designed o measre file sysem performance on file appends. I was sefl for evalaing he effeciveness of or fas ails code. This benchmark read in large files of differen ypes and sed heir byes o append o a newly creaed file. New files are creaed by appending o hem a fixed b growing nmber of byes. The benchmark appended byes in hree differen sizes: 1 byes represening a relaively small append; 1 byes represening a ypical size for a log enry on a eb server or syslog daemon; and 1 byes, represening a relaively large append ni. e did no no ry o append more han 4KB becase ha is he bondary where fas appended byes ge encoded. Becase file sysem performance can be affeced by he size of he file, we exponenially varied he sizes of he files we ran hese ess on from byes all he way o 32MB files. Compression algorihms sch as sed in Gzipfs behave differenly based on he inp hey are given. To accon for his in evalaing he append performance of Gzipfs, we ran he file-append benchmark on for ypes of daa files, ranging from easy o compress o difficl o compress: 1. A file conaining he characer a repeaedly shold compress really well. 2. A file conaining English ex, acally wrien by sers, colleced from or Usene News server. e expeced his file o compress well. 3. A file conaining a concaenaion of many differen binaries we locaed on he same hos sysem, sch as hose fond in /sr/bin and /sr/x11r6/bin. This file shold be more difficl o compress becase i conains fewer paerns sefl for compression algorihms. 4. A file conaining previosly compressed daa. e ook his daa from Microsof NT s Service Pack 6 (sp6i386.exe) which is a self-narchiving large compressed execable. e expec his file o be difficl o compress. File-aribes: The hird micro-benchmark we sed was designed o measre file sysem performance in geing file aribes. This benchmark performs a recrsive lising (ls -lrf) on a freshly npacked and bil am-ils benchmark file se, consising of 1225 files. ih or SCA sppor, he size of he original file is now sored in he index file, no in he inode of he encoded daa file. Finding his size reqires reading an addiional inode of he index 1

11 R R R R R R S S S S S S S r r r r r r s s s s s s file and hen reading is daa. This micro-benchmark measres he addiional overhead ha resls from also having o read he index file File Sysem vs. User-Level Tool Benchmarks To compare he SCAs in or sackable file sysems verss ser-level ools, we sed he file-copy micro-benchmark o compare he performance of he wo sackable file sysems wih real SCAs, Gzipfs and Uencodefs, agains heir eqivalen ser-level ools, gzip [8] and encode, respecively. In pariclar, he same Deflae algorihm and compression level (9) was sed for boh Gzipfs and gzip. In comparing Gzipfs and gzip, we measred boh he compression ime and he resling space savings. Becase he performance of compression algorihms depends on he ype of inp, we compared Gzipfs o gzip sing he filecopy micro-benchmark on all for of he differen file ypes discssed in Secion General-Prpose Benchmark Resls Am-Uils Figre 5 smmarizes he resls of he am-ils benchmark. e repor boh sysem and elapsed imes. The op par of Figre 5 shows sysem imes spen by his benchmark. This is sefl o isolae he oal effec on he CPU alone, since SCA-based file sysems change daa size and hs change he amon of disk I/O performed. rapfs adds 14.4% overhead over Ex2, becase of he need o copy daa pages beween layers. Copyfs adds only 1.3% overhead over rapfs; his shows ha or index file handling is fas. Compared o Copyfs, Uencodefs adds 7% overhead and Gzipfs adds 69.9%. These are he coss of he respecive SCAs in se and are navoidable wheher rnning in he kernel or ser-level. The oal size of an nencoded bild of am-ils is 22.9MB; a Uencoded bild is one-hird larger; Gzipfs redces his size by a facor of 2.66 o 8.6MB. So while Uencodefs increases disk I/O, i does no ranslae o a lo of addiional sysem ime becase he Uencode algorihm is rivial. Gzipfs, while decreasing disk I/O, however, is a coslier algorihm han Uencode. Tha s why Gzipfs s sysem ime overhead is greaer overall han Uencodefs s. The addiional disk I/O performed by Copyfs is small and relaive o he size of he index file. The boom par of Figre 5 shows elapsed imes for his benchmark. These figres are he closes o wha sers will see in pracice. Elapsed imes facor in increased CPU imes he more expensive he SCA is, as well as changes in I/O ha a given file sysem performs: I/O for index file, increased I/O for Uencodefs, and decreased I/O for Gzipfs. On average, he cos of daa copying wiho sizechanging (rapfs compared o Ex2fs) is an addiional Sysem Time (seconds) Elapsed Time (seconds) Cold Reglar Cold Fas arm Reglar arm Fas N N N N N N O O O O O O O 88 YY / / / / / / / T T T T T T U U U U U U U ex2fs wrapfs copyfs encodefs gzipfs File Sysem Cold Reglar Cold Fas arm Reglar arm Fas ww ŒŒ ex2fs wrapfs copyfs encodefs gzipfs File Sysem Figre 5: The Am-ils large-compile benchmark. Elapsed imes shown on op and sysem imes shown on boom. The sandard deviaions for his benchmark were less han 3% of he mean. 2.4%. SCA sppor (Copyfs over rapfs) adds anoher 2.3% overhead. The Uencode algorihm is simple and adds only 2.2% addiional overhead over Copyfs. Gzipfs, however, ses a more expensive algorihm (Deflae) [7], and i adds 14.7% overhead over Copyfs. Noe ha he elapsed-ime overhead for Gzipfs is smaller han is CPU overhead (almos 7%) becase whereas he Deflae algorihm is expensive, Gzipfs is able o win back some of ha overhead by is I/O savings. Using a warm cache improves performance by 5 1%. Using fas ails improves performance by a mos 2%. The code ha is enabled by fas ails ms check, for each read or wrie operaion, if we are a he end of he file, if a fas ail already exiss, and if a fas ail is large enogh ha i shold be encoded and a new fas ail sared. This code has a small overhead of is own. For file sysems ha do no need fas ails (e.g., Copyfs), fas ails add an overhead of 1%. e deermined ha fas ails is an opion bes sed for expensive SCAs where many small appends are occrring, a conclsion demonsraed more visibly in Secion

12 7.3.2 Bonnie Figre 6 shows he resls of rnning Bonnie on he five file sysems. Since Bonnie exercises daa reading and wriing heavily, we expec i o be affeced by he SCA in se. This is confirmed in Figre 6. Over all rns in his benchmark, rapfs has an average overhead of 2% above Ex2fs, ranging from 2 73% for he given files. Copyfs only adds an addiional 8% average overhead over rapfs. Uencodefs adds an overhead over Copyfs ha ranges from 5% o 73% for large files. Gzipfs, wih is expensive SCA, adds an overhead over Copyfs ha ranges from 22% o 418% on he large 128MB es file. Elapsed Time Overhead (%) gzipfs/copyfs encodefs/copyfs copyfs/wrapfs wrapfs/ex2 1M 2M 4M 8M 16M 32M 64M 128M File Size (byes) [log] Figre 6: The Bonnie benchmark performs many repeaed reads and wries on one file as well as nmeros random seeks and wries in hree concrren processes. e show he oal cmlaive overhead of each file sysem. Noe ha he overhead bands for Gzipfs and Uencodefs are each relaive o Copyfs. e repor he resls for files 1MB and larger, where he overheads are more visible. Figre 6 exhibis overhead spikes for 64MB files. Or es machines had 128MB of memory. Or sackable sysem caches wo pages for each page of a file: one encoded page and one decoded page, effecively dobling he memory reqiremens. The 64MB files are he smalles es files ha are large enogh for he sysem o rn o of memory. Linx keeps daa pages cached for as long as possible. hen i rns o of memory, Linx execes an expensive scan of he enire page cache and oher in-kernel caches, prging as many memory objecs as i can, possibly o disk. The overhead spikes in his figre occr a ha ime. Bonnie shows ha an expensive algorihm sch as compression, copled wih many wries in he middle of large files, can degrade performance by as mch as a facor of 5 6. In Secion 9 we describe cerain opimizaions ha we are exploring for his pariclar problem. 7.4 Micro-Benchmark Resls File-Copy Figre 7 shows he resls of rnning he file-copy benchmark on he differen file sysems. rapfs adds an average overhead of 16.4% over Ex2fs, which goes o 6% for a file size of 32MB; his is he overhead of daa page copying. Copyfs adds an average overhead of 23.7% over rapfs; his is he overhead of pdaing and wriing he index file as well as having o make emporary daa copies (explained in Secion 6) o sppor wries in he middle of files. The Uencode algorihm adds an addiional average overhead of 43.2% over Copyfs, and as mch as 153% overhead for he large 32MB file. The linear overheads of Copyfs increase wih he file s size de o he exra page copies ha Copyfs ms make, as explained in Secion 6. For all copies over 4KB, fas-ails makes no difference a all. Below 4KB, i only improves performance by 1.6% for Uencodefs. The reason for his is ha his benchmark copies files only once, whereas fas-ails is inended o work beer in siaions wih mliple small appends. Elapsed Time Overhead (%) K encodefs/copyfs copyfs/wrapfs wrapfs/ex2 16K 64K 256K 1M File Size (byes) [log] Figre 7: Copying files ino a esed file sysem. As expeced, Uencodefs is coslier ha Copyfs, rapfs, and Ex2fs. Fas-ails do no make a difference in his es, since we are no appending mliple imes File-Append Figre 8 shows he resls of rnning he file-append benchmark on he differen file sysems. The figre shows he wo emerging rends in effeciveness of he fas ails code. Firs, he more expensive he algorihm, he more helpfl fas ails become. This can be seen in he righ colmn of plos. Second, he smaller he nmber of byes appended o he file is, he more savings fas ails provide, becase he SCA is called fewer imes. This can be seen as he 4M 16M 12