ANNOTATION FOR AUTOMATION: RAPID GENERATION OF FILE SYSTEM TOOLS

Transcription

1 1 ANNOTATION FOR AUTOMATION: RAPID GENERATION OF FILE SYSTEM TOOLS Kuei (Jack) Sun, Daniel Fryer, Angela Demke Brown, and Ashvin Goel PLOS 2013, Nemacolin, Pennsylvania Nov 3, 2013

2 File-System Aware Applications 2 File system checker Ensures that all file system metadata is consistent Differentiated storage service Uses type-specific caching policies to improve performance All require knowledge of file system format

3 Problem 3 File system tools are currently strongly coupled with file-system specific details Imagine n file systems and k file system tools n k programs! Goal Decouple file system format from file system tools Effort reduced to n + k Minimize effort for file system developers

4 Approach 4 1. Annotated File System Compiler Annotation Parser Template Engine 2. Application Template 3. printf( hello file system! ); Generated Code

5 Challenge 5 Need specification of file system format to traverse file system graph Require information about nodes and edges Can we extract it from file system data structure definitions? Problem: data structures that define nodes and edges are incomplete

6 Missing Field Information 6 Nodes are under-specified struct baz { short name_len; /* name goes here */ }; 2 bytes name_len bytes 8 plostalk name_len name Data structure definition in C for baz on-disk layout of the baz data structure

7 Missing Pointer Information 7 Edges are implicit E.g., does struct foo point to struct bar? struct foo { struct bar * a; }; In-memory pointer representation struct foo { int bar_block_ptr; }; On-disk pointer representation Problem: we can only attempt to deduce semantics from field name.

8 File System Annotations 8 Add a minimal set of annotations to specify the missing information Enables traversal of file system as a graph Allows reuse of existing structure definitions

9 Annotation Language 9 Identify on-disk data structures Specify pointer information the type of structure referenced conversion to disk address Specify conditional interpretation of fields i.e. tagged unions Specify variable array parameters

10 Example: TestFS Inode 10 TestFS a simplified ext2-like file system Original TestFS inode structure definition struct inode { u32 i_type; u32 i_mod_time; u32 i_size; u32 i_block_nr[ NR_DIRECT_BLOCKS ]; u32 i_indirect; };

11 Annotated TestFS Inode 11 FSSTRUCT identifies on-disk metadata FSSTRUCT inode { u32 i_type; u32 i_mod_time; u32 i_size; POINTER(type=dir_block, when = self.i_type == I_DIR) u32 i_block_nr[ NR_DIRECT_BLOCKS ]; POINTER(type=data_indirect_block, when = self.i_type!= I_DIR) POINTER(type=dir_indirect_block, when = self.i_type == I_DIR) u32 i_indirect; };

12 Annotated TestFS Inode 13 FSSTRUCT inode { u32 i_type; u32 i_mod_time; u32 i_size; i_indirect points to data indirect block when the inode is a regular file, and direct indirect block when the inode is a directory. POINTER(type=dir_block, when = self.i_type == I_DIR) u32 i_block_nr[ NR_DIRECT_BLOCKS ]; POINTER(type=data_indirect_block, when = self.i_type!= I_DIR) POINTER(type=dir_indirect_block, when = self.i_type == I_DIR) u32 i_indirect; };

13 Example: TestFS Directory Entry 14 FSSTRUCT dirent { u32 d_name_len; s32 d_inode_nr; VECTOR( name = d_name, type = u8, size = self.d_name_len ); }; dirent is a fixed sized structure followed by a variable-length array of size d_name_len 4 bytes 4 bytes 8 3 d_name_len d_inode_nr d_name_len bytes plostalk d_name

14 Generated Code 15 FSSTRUCT inode { u32 i_type, i_mod_time, i_size; POINTER( type=dir_block, when = self.i_type == I_DIR ) u32 i_block_nr[nr_direct_blocks]; POINTER( type=data_indirect_block, when=self.i_type!= I_DIR ) POINTER( type=dir_indirect_block, when=self.i_type == I_DIR ) u32 i_indirect; }; Compiler associates pointer Code template uses type with metadata definitions the when argument //... from the POINTER for ( int i = 0; i < NR_DIRECT_BLOCKS ; i++ ) annotation { to generate if ( self->i_block_nr[i]!= 0 ) { this if statement if ( self->i_type == I_DIR ) { const char * blkdata = read_block(fsimg, } //... self->i_block_nr[i]); testfs_dir_block_parse(fsimg, (const char *)blkdata, BLOCK_SIZE);

15 Evaluation 16 Recon Checks consistency of file system at runtime Re-implemented Recon for TestFS Added 15 lines of annotation Using generated code to parse metadata Running 2,000 file system operations No-Recon Handwritten Generated User Time 0.34 ± 0.04s 0.55 ± 0.02s 0.53 ± 0.03s System Time 1.84 ± 0.06s 1.80 ± 0.04s 1.82 ± 0.04s

16 Future Work 17 Annotate real file systems Example: ext3, btrfs, etc Improve code generation Custom template language Generate parsing library Implement file-system aware applications Infer annotations from file system operations

17 Summary 18 Annotate file system data structures Specify metadata pointers Add missing field information Generate metadata parsers Simplify file system tools development

18 19 Demo Available! Any Questions? Presented by Kuei (Jack) Sun

19 1 Annotation For Automation: Rapid Generation of File System Tools Kuei (Jack) Sun, Daniel Fryer, Angela Demke Brown, and Ashvin Goel PLOS 2013, Nemacolin, Pennsylvania Demo Session

20 TestFS Graph Super Block block freemap inode freemap Inode 1 Direct Pointers inode table dir indirect dir indirect dir indirect block Indirect pointer Dirent 1 d_name_len 0 d_inode_nr 0 dir block dir block

21 Code Template Django Template Language Text substitution Supports sub-templates Example: file system dump tool Traverse file system as a graph Prints out all file system metadata

22 File System Dump Tool Generate a parsing function for each file system metadata type {% for st in structs %} static void {{ fs_name }}_{{ st.ident }}_parse(file * fsimg, const {{ st. type }} * self, int size) { {% for fd in st.fields %} {% include "print_field.c" with field=fd struct =st %} {% endfor %} {% for fd in st.fields %} {% if fd.pointer %} {% include "chase_pointer.c" with field =fd struct=st %} {% endif %} {% endfor %} } {% endfor %}

23 File System Dump Tool For every file system metadata object Print every field in that object {% for st in structs %} static void {{ fs_name }}_{{ st. ident }}_parse(file * fsimg, const {{ st. type }} * self, int size) { {% for fd in st.fields %} {% include "print_field.c" with field=fd struct =st %} {% endfor %} {% for fd in st.fields %} {% if fd.pointer %} {% include "chase_pointer.c" with field =fd struct=st %} {% endif %} {% endfor %} } {% endfor %}

24 File System Dump Tool If the field is a pointer Dereference the pointer and parse the object {% for st in structs %} static void {{ fs_name }}_{{ st. ident }}_parse(file * fsimg, const {{ st. type }} * self, int size) { {% for fd in st.fields %} {% include "print_field.c" with field=fd struct =st %} {% endfor %} {% for fd in st.fields %} {% if fd.pointer %} {% include "chase_pointer.c" with field =fd struct=st %} {% endif %} {% endfor %} } {% endfor %}