Design of a WORM Filesystem Terry Stokes Dell EMC/Isilon

Transcription

1 Design of a WORM Filesystem Terry Stokes Dell EMC/Isilon

2 Who am I? 15+ years in network protocols. Previously worked at Blue Coat Systems, RadioFrame and Microsoft Anti-trust (MCPP). Co-founded a network capture device startup in Currently work in the AIMA group at Isilon. 2

3 Disclaimers Not Isilon Smartlock design. Dated design, developed in I m not a filesystems expert. 3

4 Founded in 2003, folded in Product called the Compliance Appliance. Capture and archive electronic communications. Incorporated a WORM filesystem. 10+ sweat equity employees. Two WORM filesystem patents. Failed to get funding. 4

5 Other Traffic O Enterprise Data Center Traffic Capture ( and IM) Inspection, Analysis, Discovery and Reporting Compliance Appliance Store & Archive 5

6 To Archiver Structured Objects Message Handler MSN-IM Message Handler Exchange Message Handler SMTP Message Handler HTTP Packet Capture Network Tap Network 6

7 IT Litigation & Compliance LDAP Directory SOAP Query & Analysi s Application Dashboard Reports Discovery Workflow SEARCH Messages And Message Relationships Policy-Based Capture, Retention, Access Control Index Store Message Archive Indexer: Message Relationships Archiver: Structured Objects External Storage IM Web Existing Archives 7 Network

8 Remote Office Compliance Officer Indexing and Search Devices Messaging Servers 5 Message Stores Capture Device 2 Capture Device 3 Corp Counsel Internet LOB User Capture Device 1 Firewall Outside Parties: Legal Discovery, Regulatory Enforcement, Audit, Administrator Remote Office Capture Device 4 8

9 WORM Storage 2003 Optical storage (CD-R, optical tape, etc). EMC Centera CE Content-Addressable Storage Content hashed for storage location. NetApp Snaplock Plugin for Data OnTap Software prevents file deletion. 9

10 Customer Research Talked to potential customers. Banks, Brokers, Healthcare, Manufacturers investment firms. Several had home grown software solutions. A few simply backed up optical storage. Issue was greater than simply providing a better WORM storage solution. 10

11 Regulations SOX Section 404, 802 HIPAA Protected Health info Securities Regulations Gramm Leach Bliley Act SEC 17a-4 NASD 3010/3110 NYSE 440/472 Fin Privacy Rule Safeguards Rule Basel-II Supervisory Control CA SB 1386 Privacy and Protection US Patriot Act Cust Identification Pgm ISO Retention & Safeguarding DOD STD ERecords Management 11

12 Multiple Protocols SEC rules are ambiguous: All broker-dealer communications must be held for 3 years on non-rewriteable and nonerasable storage. Originally SEC only enforced storage. Added instant messaging requirement. What about broker web pages? VoIP? Each protocol requires a separate solution. 12

13 Slow Message Retrieval Message discovery requests (from legal and SEC) have a time limit. Many SEC requests involved several brokers, with each broker search taking 2-3 days. By the time search was complete, only a day or two left to review before handover. Didn t know what was being handed over. Legal Discovery industry born from this. 13

14 Deletion of Expired Messages s can hurt you! Harmful s have been used in several notable trials, Microsoft for one. One unexpired can indirectly hold hundreds of expired s on optical storage media. It is important for a company to delete s as soon as regulations allow. 14

15 Survey of Filesystems Reviewed top ten filesystems for ideas. ReiserFS: Uses B+ trees for file objects (inodes, directories and file data). Employs tail packing. Sun XFS: Use B+ trees for tracking free extents and free inodes. Uses extents instead of blocks. 15

16 Extents 16

17 Distributed Storage 17

18 Distributed Storage 18

19 Unsupported Operations 19

20 Specialized Storage Formatted for electronic communications. Instant Messaging HTTP Web Pages VoiceOverIP File content and meta data encrypted. 20

21 Specialized Storage Restricted File Writes Write out entire file or not at all. No appends or modifications. NVRAM backed journal for crashes. Restricted File Reads. Only two types: Read inode (for searches) Read entire file (for viewing/gathering) 21

22 Layered Filesystem 22

23 Data Encryption/Compression Layer 23

24 WORM Layer 24

25 Volume Layout 25

26 Structure Data Message Relationships Msg1 Msg2 Msg3 Attachment Associations INDEX SPACES Att-1 Att-2 Headers: Relational DB Content + Att: Reverse Keyword Structured Objects Msg Protocol Msg Headers Msg Content Msg Attachments Msg Protocol 26

27 Superblock Format 27

28 AG Header Format 28

29 Inode Format 29

30 File Format 30

31 Disk File Format 31

32 Automatic File Deletion Hourly process to delete expired files. Deletion holds table: 32

33 WORM Characteristics Disk structures and data encrypted. Non-standard disk format. Unsupported file and directory operations. Access to file contents restricted via software. 33

34 Performance Characteristics Use of extents. File contents optimized on disk. Tail packing in inode. Read operations optimized for archival. Structured file format. B+ trees for free inodes and extents. 34

35 If I did it again Investigate SED drives. Drop support for off-box storage. Include disk defragmentation scheme. Add rudimentary directory structure. Remove distributed storage by location. 35

36 Questions? 36