IBM Active Cloud Engine/Active File Management Kalyan Gunda kgunda@in.ibm.com
Agenda Need of ACE? Inside ACE Use Cases
Data Movement across sites How do you move Data across sites today? FTP, Parallel FTP SCP Backup to tape and Fedex Issues Pre planned, user initiated Replica Mgmt What if this data needs to move to multiple sites very frequently
Data Movement between sites What if there is a tool That pulls data on demand No explicit user initiation That moves data periodically & smartly That moves only changed data That effectively uses the network Manages these replicas keeping staleness in control? Is there such a tool?
Panache/ACE/AFM ACE Global provides Seamless data movement between clusters On demand Periodically Continuously Provide a persistent scalable POSIX-compliant cache for remote filesystem Even during disconnection
Moving data between locations can be slow and data copies itself can become stale Once And data is not persistent But customers need to collaborate immediately with up to date changes Write Read Read Read
Inside ACE
Panache Overview: Reads Remote user reads local edge device for file /home/appl/data/web/spreadsheet.xls /home /appl /data On demand-read from home site /web /home/appl/data/web/drawing.ppt Can run disconnected Local cache to disk Gateway node NFS CIFS HTTP VFS Read Interface node Interface node Storage node GPFS Storage node Panache Scale out cache Storage Array Panache Home Site Cluster
Asynchronous write back Remote user writes file to local edge device /home /appl /home/appl/data/web/spreadsheet.xls /home/appl/data/web/drawing.ppt /data /web Periodically, or when nw is connected Local cache to disk Log write to memory Q 1. Write Interface node Interface node Storage node Storage node Panache scale out cache Panache Home cluster
Asynchronous Updates (write, create, remove) Updates at cache site are pushed back lazily Mask the latency of the WAN Data is written to GPFS at cache site synchronously GW node queues the update for later execution Performance identical to a local file system update Writeback is asynchronous Configurable asynch delay GW nodes queue updates and write back to home as network bandwidth permits Write back tends to coalesce updates and accommodate out-of-order and parallel writes to files and directories maximizing WAN bandwidth utilization Users can force a sync if needed
Expiration of Data Staleness Control Defined based on time since disconnection Once cache is expired, no access is allowed to cache Manual expire/unexpire option for admin Allowed onlys for ro mode cache Disabled for SW & LU as they are sources of data themselves
Panache WAN Caching Features Feature Panache support Writable cache Granularity Policy based pre-fetching Yes Fileset (dir tree) Yes (uses GPFS policy engine rules) Policy based cache eviction Disconnected mode operations Data Transport protocol Streaming support Locking support Sparse file support Namespace caching Parallel data transfer Yes (uses GPFS policy engine rules) Yes (can also expire based on configured timeout) NFS (uses standard to move data from any filer) Yes (GPFS policy rules select files to replicate) No (only local cluster wide locks) Yes (can read as sparse files) Yes ( gets dir struct along with data) Yes
Use Cases
Use Case: Central/Branch Office Periodic Prefetch On Demand Pull HQ Primary Site (Writer) Central Site Data is created, maintained, updated/changed. Branch/edge sites periodically prefetch (via policy) or pull on demand Data is revalidated when accessed A typical scenario for this is itunes like music sites Edge site (Reader)
Use Case: Non-Dependent Writers Each site writes to the site s decidated fileset/directory. UseUser A s home directory (writer) r A s home directory (writer) UseUser B s home directory (writer) A central system which will have all home dirs and backup/hsm will be managed out of this. Backup Site UseBackujp site
Use Case: Ingest and Disseminate Data Ingest on location(writer) Backup site Backup Site On Demand Pull Central site gets updates frequently Regional/edge sites can periodically prefetch or pull on demand Data is revalidated Periodic Pull Periodic pre-fetch
Use Case: Global Namespace (Mesh) Clients connect to: SONAS:/data1 SONAS:/data2 SONAS:/data3 SONAS:/data4 SONAS:/data5 SONAS:/data6 Clients connect to: SONAS:/data1 SONAS:/data2 SONAS:/data3 SONAS:/data4 SONAS:/data5 SONAS:/data6 SONAS2.ibm.com Home for data3 and data4 File System: store2 Cache Filesets: /data1 /data2 Local Filesets: /data3 /data4 Cache Filesets: /data5 /data6 SONAS3.ibm.com File System: store2 HOME FOR DATA5 AND DATA6 Cache Filesets: /data1 /data2 Cache Filesets: /data3 /data4 Local Filesets: /data5 /data6 Every fileset is accessibile from all sites Each cache site will export same namespace view File System: store1 Local Filesets: /data1 /data2 Cache Filesets: /data3 /data4 Cache Filesets: /data5 /data6 SONAS1.ibm.com Home for data1 and data2 Clients connect to: SONAS:/data1 SONAS:/data2 SONAS:/data3 SONAS:/data4 SONAS:/data5 SONAS:/data6
Thank You