I/O in scientific applications

Transcription

1 COSC 4397 Parallel I/O (II) Access patterns Spring 2010 I/O in scientific applications Different classes of I/O operations Required I/O: reading input data and writing final results Checkpointing: data written periodically as insurance against hardware failures Data staging: support for applications, whose data does not fit in memory (out-of-core computations) 1

2 Options for I/O in parallel applications Sequential I/O A single process executes file operations Leads to load imbalance Individual I/O Each process has its own files Pre/Post-processing required Parallel I/O Define interfaces how multiple processes can access the same file (efficiently) Parallel I/O Goals Several process should be able to access the same file concurrently Several process should be able to access the same file efficiently 2

3 UNIX file access model: multi-process scenarios Multiple processes can open a file concurrently. Each process will have its own file pointer. No conflicts occur, when multiple processes read the same file. If several processes write at the same location, most UNIX file systems guarantee sequential consistency. e.g. two processes write 8 bytes at the same location in the file: data from one of the processes will be available in the file, but not a mixture of both Concurrent file access in a parallel file system Number of compute and storage servers will typically not be identical Blocks from compute nodes Logical view ( shared file ) storage server Disks 3

4 Concurrent file access opening a file Each storage server holds a subset of the file blocks File system needs to look up where the file resides Two alternatives: Each storage server maintains its own directory information or Centralized name service (Metadata server) File system needs to determine the striping factor depending on file system, this parameter might be fixed Creating a new file file systems has to choose different storage servers for holding the first block in order to avoid contention Concurrent write operations How to ensure sequential consistency? File locking Prevents parallelism even if processes write to different locations in the same file (false sharing) Better: locking of individual blocks Parallel file systems often offer two consistency models Sequential consistency Relaxed consistency model application is responsible for preventing overlapping write-operations 4

5 File pointers Individual file pointers: each process has a separate file pointer operations of different processes do not influence each other Shared file pointers: maintained jointly by a group of processes I/O operations of other processes influence each other might lead to non-deterministic behavior Examples: writing a parallel log file reading chunks of work from a file Explicit file offset operations: each process tells the file system where to read/write in the file UNIX I/O equivalent: pread()/pwrite() Buffering and caching Client buffering: buffering at compute nodes Consistency problems (e.g. one node writes, another tries to read the same data) Server buffering: buffering at storage servers Prevents concatenating several small requests to a single large one increases network traffic 5

6 Discontiguous access fseek(fh, offset=21, SEEK_SET); read(fh, length=2) fseek(fh, offset=29, SEEK_SET); read(fh, length=2) fseek(fh, offset=37, SEEK_SET); read(fh, length=2) fseek(fh, offset=45, SEEK_SET); read(fh, length=2) e.g. reading a subblock of a two-dimensional matrix produces a series of discontiguous requests of small amounts of data Handling discontiguous access Merge small requests into a single operation Single data transfer operation Enables prefetching of blocks Possible I/O interfaces Algorithmic description: compact interface for regular access patterns with constant strides List I/O: for irregular access patterns 6

7 Algorithmic description Contiguous in memory, discontiguous on disk read_strided (file, buffer, file_stride, segment_size); buffer file_stride=3, segment_size=1 disc Discontiguous in memory, discontiguous on disk read_strided2 (file, buffer, file_stride, mem_stride, segment_size); buffer disc file_stride=3, mem_stride = 2, segment_size=1 Algorithmic description in MPI Derived Datatypes MPI_Type_vector( ); MPI_Type_contiguous( ) MPI_Type_subarray ( ) 7

8 List I/O interfaces Contiguous in memory, discontiguous on disk read_list ( file, buffer, count, offsets[], length[] ); Discontiguous in memory, contiguous on disk readv ( file, const struct iovec *vec, int count); struct iovec { void* iov_base; /*starting address */ size_t iov_len; /*length in bytes */ } Discontiguous in memory, discontiguous on disk read_list2 ( ) gather/scatter interface: gathers data from disc and scatters it in memory List I/O Interfaces in MPI Derived Datatypes MPI_Type_indexed( ); MPI_Type_struct ( ); 8

9 Optimization for discontiguous I/O: Data sieving Ignore the gaps when reading from disk One large contiguous access instead of many small requests Works well if gaps are small Overhead can be dominating for large gaps user buffer temporary buffer disc read(); Collective I/O (I) Example: reading a sub-block of a two-dimensional matrix produces a series of non-contiguous requests of small amounts of data Process 2: 0: 1: 3: read(, offset=4, offset=0, offset=2, offset=6, length=2) read(, offset=12, offset=8, offset=10, offset=14, length=2) read(, offset=20, offset=16, offset=18, offset=22, length=2) read(, offset=28, offset=24, offset=26, offset=30, length=2) 9

10 Collective I/O (II) Estimate for the costs of the previous operation: T read = np * ( n op * (l disk + n / b disk )) with np: number of processes n op : number of read operations per process l disk : disk latency ( 7-12ms) b disk : disk transfer rate ( MB/s) n: number of bytes read per operation For small requests T read np * n op *l disk Collective I/O (III) I/O requests of multiple processes can be combined to larger read/write operations if (rank == root ) { read(, offset=0, length=32) } MPI_Scatter ( ); 10

11 Collective I/O operations (IV) Estimate of the costs for the alternative algorithm T read = (l disk + n*n op *np /b disk ) + (np-1) (l net + n*n op /b net ) with l net : network latency (e.g. IB 4 µs) b net : network bandwidth ( e.g. IB: 1 GB/s) Collective I/O Merges separate I/O requests across multiple processes Collective read: retrieve large chunks from disk and distribute to multiple processes Collective write: gather data from multiple processes before writing to disk Eliminates false sharing! Two classes of collective I/O techniques on parallel file systems Client-based collective I/O Server-based collective I/O 11

12 Data layout per process Client-based collective I/O Uses the message-passing network to rearrange data (shuffle) before sending contiguous chunks to the I/O node Process 0 Process 1 Process 2 Process 3 Logical Data layout on intermediary processes Data layout on I/O nodes I/O node 0 I/O node 1 Client-based collective I/O continued Consists of two steps (=> often called two-phase I/O) Shuffle I/O operation Problems to worry about Number of intermediary processes: either number of application processes or number of I/O nodes Additional buffer space: segmenting of data might be required Schedule for accessing I/O nodes: avoid that all intermediary processes send first to I/O node 0, than to I/O node 1 etc. Introduces additional copy and data transfer operations. improves performance if costs of copy and data transfer operations are smaller than the gain through the improved I/O performance. 12

13 Server-based I/O Collect and merge requests on the server Data layout per process Process 0 Process 1 Process 2 Process 3 I/O nodes gather data to fill blocks I/O nodes write previous blocks to disk while continuing to gather data I/O node 0 I/O node 1 Server-based I/O continued Steps for a write operation Compute processes send a description of the planned data transfer (without data) Each I/O node determines which file blocks are under its control Each I/O node determines which processes hold data for each block For each block, I/O nodes request the data from the compute nodes 13

14 Server-based I/O cont. again Eliminates the need for extra buffer space on compute nodes Data travels only once over the network Many server-based I/O techniques are designed to handle only a few blocks at a time minimizes buffer space requirements on the I/O nodes might require multiple messages between compute process and I/O node for large read/write operations Hints Performance of any I/O technique depends on Machine parameters Application parameters Implementation of the I/O library I/O library can not determine the best/fastest method to handle I/O operations for a wide range of application scenarios Application have to give hints to the I/O library about their I/O characteristics 14

15 Hints and optimization possibilities Hint Read-only Write-only Consecutive access Strided access Random access Large consecutive access No overlapping access Possible optimization Aggressive prefetching Turn-off prefetching Prefetch blocks in sequence for read-access files Prefetch according to strided pattern; delay writing if other process will fill in data Turn off prefetching; use largest possible cache and buffer; delay writing as long as possible Turn off caching and buffering Turn off concurrency control 15