MPI IO. Timothy H. Kaiser, Ph.D. Friday, August 12, 11
|
|
- Everett Giles Houston
- 5 years ago
- Views:
Transcription
1 MPI IO Timothy H. Kaiser, Ph.D.
2 Purpose Introduce Parallel IO Introduce MPI-IO Not give an exhaustive survey Explain why you would want to use it Explain why you wouldn t want to use it Give a nontrivial and useful example
3 References software/libs/io/mpiio.php Parallel I/O for High Performance Computing. John May Using MPI-2 Advanced Features of the Message Passing Interface. William Gropp, Ewing Lusk and Rajeev Thakur
4 What & Why of parallel IO
5 What & Why of parallel IO Same motivation of going parallel initially You have lots of data You want to do things fast Parallel IO will (hopefully) enable you to move large amounts of data to/from disk quickly
6 What & Why of parallel IO Parallel implies that some number (or all) of your processors (simultaneously) participate in an IO operation Good parallel IO shows speedup as you add processors I write about 300 Mbytes/second, others faster
7 A Motivating Example Earthquake Model E3d Finite difference simulation with the grid distributed across N processors On BlueGene we run at sizes of 7509 x 7478 x 250 = 14,021,250,000 cells or 56 GBytes per volume, output 3 velocity volumes per dump For a restart file we write 14 volumes
8 Simple (nonmpi) Parallel IO Each processor dumps its portion of the grid to a separate unique file char* unique(char *name,int myid) { static char unique_str[40]; int i; for(i=0;i<40;i++) unique_str[i]=(char)0; sprintf(unique_str,"%s%5.5d",name,myid); return unique_str; }
9 Simple (nonmpi) Parallel module stuff contains function unique(name,myid) character (len=*) name character (len=20) unique character (len=80) temp write(temp,"(a,i5.5)")trim(name),myid unique=temp return end function unique end module
10 Why not just do this?
11 Why not just do this? Might write thousands of files Could be very slow Output is dependent on the number of processors We might want the data in a single file
12 MPI-IO to the rescue MPI has over 55 calls related to file input and output Available in most modern MPI libraries Can produce exceptional results Support striping A collection of distributed files look like one We will look at outputs to a single file
13 Why not? Some functionality might not be available 3d data types
14 Why not? Some functionality might not be available 3d data types More likely to have/introduce bugs Memory leak File system overload Just hangs
15 Why not? More complex than normal output
16 Why not? More complex than normal output Need support from the file system for good performance Have seen 200 bytes/second NOT Megabytes Have run out of file locks
17 Our Real World Example We have a 3d volume of some data v distributed across N processor The size and distribution are input and not the same on each processor We are outputting some function of v, V=f(v) We do not Consider a spherical cow Each processor writes its values to a common file
18 Typical Small Case Grid size 472 x 250 x 242 on 16 processors Color shows processor number vista --raw minmax skip 36 \ -x 640 -y outformat png --swapbytes -r \ --fov 30 -g \ -a opacity 0.01 volume dsmpi
19 Special Considerations We are calculating our output on the fly Create a buffer Fill the buffer and write Different processors will have different number of writes
20 Special Considerations We want to use a collective write operation Each process must call the collective write the same number of times Each process must determine how many writes it needs to do The total number of writes is the max Some processors might call write with no data
21 Procedure # 1 Allocate a temporary output buffer Open the file Set the view of the file to the beginning Process 0 writes the file header (36 bytes)
22 Procedure #2 Create a description of how the data is distributed Defined data type Hardest part of the whole process Set the view of the file to this description Determine how many writes are needed and AllReduce into do_call_max
23 Procedure #3 Loop over the grid Fill buffer If buffer is full Write it Adjust offset do_call_max=do_call_max-1 Call write with no data until do_call_max=0
24 The MPI-IO Routines MPI_File_open(MPI_COMM_WORLD,fname,(MPI_MODE_RDWR MPI_MODE_CREATE),MPI_INFO_NULL,&fh); MPI_File_set_view(fh,disp,MPI_INT,filetype,"native",MPI_INFO_NULL); MPI_File_write_at(fh, 0, header, hl, MPI_INT,&status); MPI_File_write_at_all(fh, offset, ptr, i2, MPI_INT,&status); MPI_File_close(&fh);
25 Synopsis: Opens a file MPI_File_open int MPI_File_open(MPI_Comm comm, char *filename, int amode, MPI_Info info, MPI_File *fh) Input Parameters comm communicator (handle) filename name of file to open (string) amode file access mode (integer) info info object (handle) Output Parameters fh file handle (handle)
26 Synopsis: Sets the file view MPI_File_set_view int MPI_File_set_view(MPI_File fh, MPI_Offset disp, MPI_Datatype etype, MPI_Datatype filetype, char *datarep, MPI_Info info) Input Parameters fh file handle (handle) disp displacement (nonnegative integer) etype elementary datatype (handle) filetype filetype (handle) datarep data representation (string) info info object (handle)
27 MPI_File_write_at Synopsis: Write using explicit offset, not collective int MPI_File_write_at(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Status *status) Input Parameters fh file handle (handle) offset file offset (nonnegative integer) buf initial address of buffer (choice) count number of elements in buffer (nonnegative integer) datatype datatype of each buffer element (handle) Output Parameters status status object (Status)
28 MPI_File_write_at_all Synopsis: Collective write using explicit offset int MPI_File_write_at_all(MPI_File fh, MPI_Offset offset, void *buf, int count, MPI_Datatype datatype, MPI_Status *status) Input Parameters fh file handle (handle) offset file offset (nonnegative integer) buf initial address of buffer (choice) count number of elements in buffer (nonnegative integer) datatype datatype of each buffer element (handle) Output Parameters status status object (Status)
29 MPI_File_close Synopsis: Closes a file int MPI_File_close(MPI_File *fh) Input Parameters fh file handle (handle)
30 The Data type Routines Our preferred routine creates a 3d description MPI_Type_create_subarray(3,gsizes,lsizes,istarts,MPI_ORDER_C,old_type,new_type); On some platforms we need to fake it MPI_Type_contiguous(sx,old_type,&VECT); MPI_Type_struct(sz,blocklens,indices,old_types,&TWOD); MPI_Type_commit(&TWOD);
31 MPI_Type_create_subarray Synopsis: Creates a datatype describing a subarray of an N dimensional array int MPI_Type_create_subarray(int ndims, int *array_of_sizes, int *array_of_subsizes, int *array_of_starts, int order, MPI_Datatype oldtype, MPI_Datatype *newtype) Input Parameters ndims number of array dimensions (positive integer) array_of_sizes number of elements of type oldtype in each dimension of the full array (array of positive integers) array_of_subsizes number of elements of type oldtype in each dimension of the subarray (array of positive integers) array_of_starts starting coordinates of the subarray in each dimension (array of nonnegative integers) order array storage order flag (state) oldtype old datatype (handle) Output Parameters newtype new datatype (handle)
32 MPI_Type_contiguous Synopsis: Creates a contiguous datatype int MPI_Type_contiguous( int count,mpi_datatype old_type,mpi_datatype *newtype) Input Parameters count replication count (nonnegative integer) oldtype old datatype (handle) Output Parameter newtype new datatype (handle)
33 MPI_Type_struct Synopsis: Creates a struct datatype int MPI_Type_struct( int count, int blocklens[], MPI_Aint indices[], MPI_Datatype old_types[], MPI_Datatype *newtype ) Input Parameters count number of blocks (integer) -- also number of entries in arrays array_of_types, array_of_displacements and array_of_blocklengths blocklens number of elements in each block (array) indices byte displacement of each block (array) old_types type of elements in each block (array of handles to datatype objects) Output Parameter newtype new datatype (handle)
34 Our Program...! MPI_Init(&argc,&argv);! MPI_Comm_rank( MPI_COMM_WORLD, &myid);! MPI_Comm_size( MPI_COMM_WORLD, &numprocs);! MPI_Get_processor_name(name,&resultlen);! printf("process %d running on %s\n",myid,name); /* we read and broadcast the global grid size (nx,ny,nz) */! if(myid == 0) {!! if(argc!= 4){!!! printf("the grid size is not on the command line assuming 100 x 50 x 75\n");!!! gblsize[0]=100;!!! gblsize[1]=50;!!! gblsize[2]=75;!! }!! else {!!! gblsize[0]=atoi(argv[1]);!!! gblsize[1]=atoi(argv[2]);!!! gblsize[2]=atoi(argv[3]);!! }! }! MPI_Bcast(gblsize,3,MPI_INT,0,MPI_COMM_WORLD); /********** a ***********/!
35 /* the routine three takes the number of processors and returns a 3d decomposition or topology. this is simply a factoring of the number of processors into 3 integers stored in comp */! three(numprocs,comp);! /* the routine mpdecomposition takes the processor topology and the global grid dimensions and maps the grid to the topology. mpdecomposition returns the number of cells a processor holds and the starting coordinates for its portion of the grid */! if(myid == 0 ) {! printf("input mpdecomposition %5d%5d%5d%5d%5d%5d\n",gblsize[1],gblsize[2],gblsize[0],! comp[1], comp[2], comp[0]); }! mpdecomposition( gblsize[1],gblsize[2],gblsize[0],comp[1],comp[2],comp[0],myid,dist); printf(" out mpdecomposition %5d%5d%5d%5d%5d%5d%5d\n",myid,dist[0],dist[1],dist[2], dist[3],dist[4],dist[5]); /********** b ***********/!
36 Example Distribution Processor Size X Size Y Size Z Start X Start Y Start Z Global size 50 x 200 x 100 on 8 processors
37 Back to our program... /* global grid size */ nx=gblsize[0];! ny=gblsize[1];! nz=gblsize[2]; /* amount that i hold */! sx=dist[0];! sy=dist[1];! sz=dist[2]; /* my grid starts here */! x0=dist[3];! y0=dist[4];! z0=dist[5]; /********** c ***********/!
38 /* allocate memory for our volume */! vol=getarrayf3d((long)sy,(long)0,(long)0, (long)sz,(long)0,(long)0, (long)sx,(long)0,(long)0); /* fill the volume with numbers 1 to global grid size */ /* the program from which this example was derived, e3d, holds its data as a collection of vertical planes. plane number increases with y. that is why we loop on y with the outer most loop. */! k=1+(x0+nx*z0+(nx*nz)*y0);! for (ltmp=0;ltmp<sy;ltmp++) {!! for (mtmp=0;mtmp<sz;mtmp++) {!!! for (ntmp=0;ntmp<sx;ntmp++) {!!!! val=k+ntmp+ mtmp*nx! + ltmp*nx*nz;!!!! if(val > (long)int_max)val=(long)int_max;!!!! vol[ltmp][mtmp][ntmp]=(int)val;!!! }!! }! } /********** d ***********/!
39 /* create a file name based on the grid size */! for(j=1;j<80;j++) {!! fname[j]=(char)0;! } sprintf(fname,"%s_%3.3d_%4.4d_%4.4d_%4.4d","mpiio_dat", numprocs,gblsize[0],gblsize[1],gblsize[2]); /* we open the file fname for output, info is NULL */ ierr=mpi_file_open(mpi_comm_world, fname,(mpi_mode_rdwr MPI_MODE_CREATE), MPI_INFO_NULL, &fh); /* we write a 9 integer header */ hl=3; /* set the view to the beginning of the file */ ierr=mpi_file_set_view(fh, 0, MPI_INT, MPI_INT, "native",mpi_info_null); /* process 0 writes the header */ if(myid == 0) { header[0]=nx; header[1]=ny; header[2]=nz; /* MPI_File_write_at is not a collective so only 0 calls it */ ierr=mpi_file_write_at(fh, 0, header, hl, MPI_INT,&status); } /********** 01 ***********/
40 /* we create a description of the layout of the data */ /* more on this later */! printf("mysubgrid0 %5d%5d%5d%5d%5d%5d%5d%5d%5d%5d\n",myid,nx,ny,nz,sx,sy,sz,x0,y0,z0);! mysubgrid0(nx, ny, nz,sx, sy, sz, x0, y0, z0, MPI_INT,&disp,&filetype); /* length of the header */! disp=disp+(4*hl); /* every processor "moves" past the header */ ierr=mpi_file_set_view(fh, disp, MPI_INT, filetype, "native",mpi_info_null); /********** 02 ***********/
41 /* we are going to create the data on the fly */ /* so we allocate a buffer for it */ t3=mpi_wtime(); isize=sx*sy*sz; buf_size=num_vals*sizeof(flt); if( isize < NUM_VALS) { buf_size=isize*sizeof(flt); } else { buf_size=num_vals*sizeof(flt); } ptr=(flt*)malloc(buf_size); offset=0; /* find the max and min number of isize of each processors buffer */ ierr=mpi_allreduce ( &isize, &max_size, 1, MPI_INT, MPI_MAX, MPI_COMM_WORLD); ierr=mpi_allreduce ( &isize, &min_size, 1, MPI_INT, MPI_MIN, MPI_COMM_WORLD); /********** 03 ***********/
42 /* find out how many times each processor will dump its buffer */ i=0; i2=0; do_call=0; sample=1; grid_l=y0+sy; grid_m=z0+sz; grid_n=x0+sx; /* could just do division but that would be too easy */ for(l = y0; l < grid_l; l = l + sample) { for(m = z0; m < grid_m; m = m + sample) { for(n = x0; n < grid_n; n = n + sample) { i++; i2++; if(i == isize i2 == NUM_VALS){ do_call++; i2=0; } } } } /* get the maximum number of many times a processor will dump its buffer */ ierr= MPI_Allreduce ( &do_call, &do_call_max, 1, MPI_INT, MPI_MAX, MPI_COMM_WORLD); /********** 04 ***********/
43 /* finally we start to write the data */ i=0; i2=0; /* we loop over our grid filling the output buffer */ for(l = y0; l < grid_l; l = l + sample) { for(m = z0; m < grid_m; m = m + sample) { for(n = x0; n < grid_n; n = n + sample) { ptr[i2] = gets3d(vol,l, m, n,y0,z0,x0); i++; i2++; /********** 05 ***********/
44 /* when we have all our data or the buffer is full we write */ if(i == isize i2 == NUM_VALS){ t5=mpi_wtime(); t7++; if((isize == max_size) && (max_size == min_size)) { /* as long as every processor has data to write we use the collective version */ /* the collective version of the write is MPI_File_write_at_all */ ierr=mpi_file_write_at_all(fh, offset, ptr, i2, MPI_INT,&status); do_call_max=do_call_max-1; } else { /* if only I have data to write then we use MPI_File_write_at */ /*ierr=mpi_file_write_at(fh, offset, ptr, i2, MPI_INT,&status);*/ /* Wait! Why was that line commented out? Why are we using MPI_File_write_at_all? */ /* Answer: Some versions of MPI work better using MPI_File_write_at_all */ /* What happens if some processors are done writing and don't call this? */ /* Answer: See below. */ ierr=mpi_file_write_at_all(fh, offset, ptr, i2, MPI_INT,&status); do_call_max=do_call_max-1; } offset=offset+i2; i2=0; t6=mpi_wtime(); dt[5]=dt[5]+(t6-t5); } }}} /********** 06 ***********/
45 /* Here is where we fix the problem of unmatched calls to MPI_File_write_at_all*/ /* If a processor is done with its writes and others still have */ /* data to write the the done processor just calls */ /* MPI_File_write_at_all but this 0 values to write */ /* All processors call MPI_File_write_at_all the same number of */ /* times so everyone is happy */ while(do_call_max > 0) { ierr=mpi_file_write_at_all(fh, (MPI_Offset)0, (void *)0, 0, MPI_INT,&status); do_call_max=do_call_max-1; } /* We finally close the file */ ierr=mpi_file_close(&fh); /********* ierr=mpi_info_free(&fileinfo); *********/ MPI_Finalize(); exit(0); /********** 07 ***********/
46 Our output: vista --rawtype int --minmax skip 12 -x 640 -y outformat png --fov 30 bonk --raw r g a opacity 0.01
47 Source
48 void mpdecomposition(int l, int m, int n, int nx, int ny, int nz, int node, int *dist) {! int nnode, mnode, rnode;! int grid_n,grid_n0,grid_m,grid_m0,grid_l,grid_l0; /* x decomposition */! rnode = node%nx;! mnode = (n%nx);! nnode = (n/nx);! grid_n = (rnode < mnode)? (nnode + 1) : (nnode);! grid_n0 = rnode*nnode;! grid_n0 += (rnode < mnode)? (rnode) : (mnode); /* z decomposition */! rnode = (node%(nx*nz))/nx;! mnode = (m%nz);! nnode = (m/nz);! grid_m = (rnode < mnode)? (nnode + 1) : (nnode);! grid_m0 = rnode*nnode;! grid_m0 += (rnode < mnode)? (rnode) : (mnode); /* y decomposition */! rnode = node/(nx*nz);! mnode = (l%ny);! nnode = (l/ny);! grid_l = (rnode < mnode)? (nnode + 1) : (nnode);! grid_l0 = rnode*nnode;! grid_l0 += (rnode < mnode)? (rnode) : (mnode);!! dist[0]=grid_n; dist[1]=grid_l; dist[2]=grid_m;! dist[3]=grid_n0; dist[4]=grid_l0; dist[5]=grid_m0; } /* the routine mpdecomposition takes the processor topology (nx, ny,nz) and the global grid dimensions (l,m,n) and maps the grid to the topology. mpdecomposition returns the number of cells a processor holds, dist[0:2], and the starting coordinates for its portion of the grid dist[3:5] */
49 void mysubgrid0(int nx, int ny, int nz, int sx, int sy, int sz, int x0, int y0, int z0, MPI_Datatype old_type, MPI_Offset *location,mpi_datatype *new_type) {! MPI_Datatype VECT; #define BSIZE 5000! int blocklens[bsize];! MPI_Aint indices[bsize];! MPI_Datatype old_types[bsize];! MPI_Datatype TWOD;! int i;! if(myid == 0)printf("using mysubgrid version 1\n");! if(sz > BSIZE)mpi_check(-1,"sz > BSIZE, increase BSIZE and recompile");! ierr=mpi_type_contiguous(sx,old_type,&vect); ierr=mpi_type_commit(&vect); for (i=0;i<sz;i++) {!blocklens[i]=1;!old_types[i]=vect;!indices[i]=i*nx*4; }! ierr=mpi_type_struct(sz,blocklens,indices,old_types,&twod); ierr=mpi_type_commit(&twod); for (i=0;i<sy;i++) {!blocklens[i]=1;!old_types[i]=twod;!indices[i]=i*nx*nz*4; }! ierr=mpi_type_struct(sy,blocklens,indices,old_types,new_type); ierr=mpi_type_commit(new_type); *location=4*(x0+nx*z0+(nx*nz)*y0); } /* we have two versions of mysubgrid0, the routine that creates the description of the data layout. /* /* This version of mysubgrid0 builds up the description from primatives. we start with x, then create VECT which is a vector of x values. we then take a collection of VECTs and create a vertical slice, TWOD. note that the distance between each VECT in TWOD is given in indices[i]. we next take a collection of vertical slices and create our volume. again we have the distances between the slices given in indices[i] */
50 /* This one is actually preferred. it uses a single call to the mpi routine MPI_Type_create_subarray with the the grid description as input. what we get back is a data type that is a 3d strided volume. Unfortunately, MPI_Type_create_subarray does not work for 3d arrays for some versions of MPI, in particular LAM. */ void mysubgrid0(int nx, int ny, int nz, int sx, int sy, int sz, int x0, int y0, int z0,!!!! MPI_Datatype old_type,!!!! MPI_Offset *location,!!!! MPI_Datatype *new_type) {! int gsizes[3],lsizes[3],istarts[3];! gsizes[2]=nx; gsizes[1]=nz; gsizes[0]=ny;! lsizes[2]=sx; lsizes[1]=sz; lsizes[0]=sy;! istarts[2]=x0; istarts[1]=z0; istarts[0]=y0;! if(myid == 0)printf("using mysubgrid version 2\n");! ierr=mpi_type_create_subarray(3,gsizes,lsizes,istarts,mpi_order_c,old_type,new_type);! ierr=mpi_type_commit(new_type);! *location=0; }
MPI, Part 3. Scientific Computing Course, Part 3
MPI, Part 3 Scientific Computing Course, Part 3 Non-blocking communications Diffusion: Had to Global Domain wait for communications to compute Could not compute end points without guardcell data All work
More informationMPI Parallel I/O. Chieh-Sen (Jason) Huang. Department of Applied Mathematics. National Sun Yat-sen University
MPI Parallel I/O Chieh-Sen (Jason) Huang Department of Applied Mathematics National Sun Yat-sen University Materials are taken from the book, Using MPI-2: Advanced Features of the Message-Passing Interface
More informationMPI, Part 2. Scientific Computing Course, Part 3
MPI, Part 2 Scientific Computing Course, Part 3 Something new: Sendrecv A blocking send and receive built in together Lets them happen simultaneously Can automatically pair the sends/recvs! dest, source
More informationParallel I/O Techniques and Performance Optimization
Parallel I/O Techniques and Performance Optimization Lonnie Crosby lcrosby1@utk.edu NICS Scientific Computing Group NICS/RDAV Spring Training 2012 March 22, 2012 2 Outline Introduction to I/O Path from
More informationAdvanced Parallel Programming
Advanced Parallel Programming Derived Datatypes Dr Daniel Holmes Applications Consultant dholmes@epcc.ed.ac.uk Overview Lecture will cover derived datatypes memory layouts vector datatypes floating vs
More informationAdvanced Parallel Programming
Advanced Parallel Programming Derived Datatypes Dr David Henty HPC Training and Support Manager d.henty@epcc.ed.ac.uk +44 131 650 5960 16/01/2014 MPI-IO 2: Derived Datatypes 2 Overview Lecture will cover
More informationPart - II. Message Passing Interface. Dheeraj Bhardwaj
Part - II Dheeraj Bhardwaj Department of Computer Science & Engineering Indian Institute of Technology, Delhi 110016 India http://www.cse.iitd.ac.in/~dheerajb 1 Outlines Basics of MPI How to compile and
More informationIntroduction to I/O at CHPC
CENTER FOR HIGH PERFORMANCE COMPUTING Introduction to I/O at CHPC Martin Čuma, m.cumautah.edu Center for High Performance Computing Fall 2015 Outline Types of storage available at CHPC Types of file I/O
More informationMPI-IO. Warwick RSE. Chris Brady Heather Ratcliffe. The Angry Penguin, used under creative commons licence from Swantje Hess and Jannis Pohlmann.
MPI-IO Chris Brady Heather Ratcliffe The Angry Penguin, used under creative commons licence from Swantje Hess and Jannis Pohlmann. Warwick RSE Getting data in and out The purpose of MPI-IO is to get data
More informationCompressible Fluid Dynamics
Compressible Fluid Dynamics Fluids: Almost Everything 99% of the visible matter in the Universe is in the form of fluids Most of the astrophysical systems we don t fully understand, it s the fluid dynamics
More informationParallel I/O with MPI
Parallel I/O with MPI March 15 2017 Benedikt Steinbusch Jülich Supercomputing Centre Parallel I/O with MPI Part I: Introduction March 15 2017 Benedikt Steinbusch Features of MPI I/O Standardized I/O API
More informationUSER-DEFINED DATATYPES
Advanced MPI USER-DEFINED DATATYPES MPI datatypes MPI datatypes are used for communication purposes Datatype tells MPI where to take the data when sending or where to put data when receiving Elementary
More informationHigh Performance Computing Course Notes Message Passing Programming III
High Performance Computing Course Notes 2009-2010 2010 Message Passing Programming III Blocking synchronous send the sender doesn t return until it receives the acknowledgement from the receiver that the
More informationHigh Performance Computing Course Notes Message Passing Programming III
High Performance Computing Course Notes 2008-2009 2009 Message Passing Programming III Communication modes Synchronous mode The communication is considered complete when the sender receives the acknowledgement
More informationDerived Datatypes. MPI - Derived Data Structures. MPI Datatypes Procedure Datatype Construction Contiguous Datatype*
Derived Datatypes MPI - Derived Data Structures Based on notes from Science & Technology Support High Performance Computing Ohio Supercomputer Center MPI Datatypes Procedure Datatype Construction Contiguous
More informationBuffering in MPI communications
Buffering in MPI communications Application buffer: specified by the first parameter in MPI_Send/Recv functions System buffer: Hidden from the programmer and managed by the MPI library Is limitted and
More informationIntrodução ao MPI-IO. Escola Regional de Alto Desempenho 2018 Porto Alegre RS. Jean Luca Bez 1 Francieli Z. Boito 2 Philippe O. A.
Introdução ao MPI-IO Escola Regional de Alto Desempenho 2018 Porto Alegre RS Jean Luca Bez 1 Francieli Z. Boito 2 Philippe O. A. Navaux 1 1 GPPD - INF - Universidade Federal do Rio Grande do Sul 2 INRIA
More informationParallel I/O. Steve Lantz Senior Research Associate Cornell CAC. Workshop: Data Analysis on Ranger, January 19, 2012
Parallel I/O Steve Lantz Senior Research Associate Cornell CAC Workshop: Data Analysis on Ranger, January 19, 2012 Based on materials developed by Bill Barth at TACC 1. Lustre 2 Lustre Components All Ranger
More informationIntroduction to MPI-2 (Message-Passing Interface)
Introduction to MPI-2 (Message-Passing Interface) What are the major new features in MPI-2? Parallel I/O Remote Memory Operations Dynamic Process Management Support for Multithreading Parallel I/O Includes
More informationParallel I/O for SwissTx
16 February 2000 Parallel I/O for SwissTx SFIO parallel I/O system implementation for Swiss-Tx. MPI-II I/O interface support for SFIO. LSP DI EPFL 1.0 Introduction 1.1 Requirements for parallel I/O 1.1.1
More informationNAME MPI_Address - Gets the address of a location in memory. INPUT PARAMETERS location - location in caller memory (choice)
Concurrent Programming in heterogeneous Distributed Systems 3-1 3 Manual pages Some manual pages are from the LAM-distribution and others from the mpich-distribution, when they weren't available in LAM.
More informationIntroduction to the Message Passing Interface (MPI)
Introduction to the Message Passing Interface (MPI) CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Introduction to the Message Passing Interface (MPI) Spring 2018
More informationPractical Scientific Computing: Performanceoptimized
Practical Scientific Computing: Performanceoptimized Programming Advanced MPI Programming December 13, 2006 Dr. Ralf-Peter Mundani Department of Computer Science Chair V Technische Universität München,
More informationParallel I/O for SwissTx Emin Gabrielyan Prof. Roger D. Hersch Peripheral Systems Laboratory Ecole Polytechnique Fédérale de Lausanne Switzerland
Fig. 00. Parallel I/O for SwissTx Emin Gabrielyan Prof. Roger D. Hersch Peripheral Systems Laboratory Ecole Polytechnique Fédérale de Lausanne Switzerland Introduction SFIO (Striped File I/O) software
More informationReusing this material
Derived Datatypes Reusing this material This work is licensed under a Creative Commons Attribution- NonCommercial-ShareAlike 4.0 International License. http://creativecommons.org/licenses/by-nc-sa/4.0/deed.en_us
More informationThe MPI Message-passing Standard Practical use and implementation (II) SPD Course 27/02/2017 Massimo Coppola
The MPI Message-passing Standard Practical use and implementation (II) SPD Course 27/02/2017 Massimo Coppola MPI communication semantics Message order is not guaranteed, Only communications with same envelope
More informationIntroduction to I/O at CHPC
CENTER FOR HIGH PERFORMANCE COMPUTING Introduction to I/O at CHPC Martin Čuma, m.cuma@utah.edu Center for High Performance Computing Fall 2018 Outline Types of storage available at CHPC Types of file I/O
More informationMPI Derived Datatypes
MPI Derived Datatypes Francesco Salvadore f.salvadore@cineca.it SuperComputing Applications and Innovation Department 1 Derived datatypes What are? MPI Derived datatypes are datatypes that are built from
More informationScalable I/O. Ed Karrels,
Scalable I/O Ed Karrels, edk@illinois.edu I/O performance overview Main factors in performance Know your I/O Striping Data layout Collective I/O 2 of 32 I/O performance Length of each basic operation High
More informationThe Message Passing Interface (MPI) TMA4280 Introduction to Supercomputing
The Message Passing Interface (MPI) TMA4280 Introduction to Supercomputing NTNU, IMF January 16. 2017 1 Parallelism Decompose the execution into several tasks according to the work to be done: Function/Task
More informationCluster Computing MPI. Industrial Standard Message Passing
MPI Industrial Standard Message Passing MPI Features Industrial Standard Highly portable Widely available SPMD programming model Synchronous execution MPI Outer scope int MPI_Init( int *argc, char ** argv)
More informationECE 587 Hardware/Software Co-Design Lecture 09 Concurrency in Practice Message Passing
ECE 587 Hardware/Software Co-Design Spring 2018 1/14 ECE 587 Hardware/Software Co-Design Lecture 09 Concurrency in Practice Message Passing Professor Jia Wang Department of Electrical and Computer Engineering
More informationA Parallel API for Creating and Reading NetCDF Files
A Parallel API for Creating and Reading NetCDF Files January 4, 2015 Abstract Scientists recognize the importance of portable and efficient mechanisms for storing datasets created and used by their applications.
More informationCS 470 Spring Mike Lam, Professor. Advanced MPI Topics
CS 470 Spring 2018 Mike Lam, Professor Advanced MPI Topics MPI safety A program is unsafe if it relies on MPI-provided buffering Recall that MPI_Send has ambiguous blocking/buffering If you rely on it
More informationIntroduction to MPI Programming Part 2
Introduction to MPI Programming Part 2 Outline Collective communication Derived data types Collective Communication Collective communications involves all processes in a communicator One to all, all to
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2019 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationMPI - Derived Data Structures
MPI - Derived Data Structures Based on notes from Science & Technology Support High Performance Computing Ohio Supercomputer Center Cluster Computing 1 Derived Datatypes MPI Datatypes Procedure Datatype
More informationMore MPI. Bryan Mills, PhD. Spring 2017
More MPI Bryan Mills, PhD Spring 2017 MPI So Far Communicators Blocking Point- to- Point MPI_Send MPI_Recv CollecEve CommunicaEons MPI_Bcast MPI_Barrier MPI_Reduce MPI_Allreduce Non-blocking Send int MPI_Isend(
More informationParallel I/O and MPI-IO contd. Rajeev Thakur
Parallel I/O and MPI-IO contd. Rajeev Thakur Outline Accessing noncontiguous data with MPI-IO Special features in MPI-IO for accessing subarrays and distributed arrays I/O performance tuning 2 Accessing
More informationAn introduction to MPI. (word cloud of all the MPI hydro code written for this course:
An introduction to MPI (word cloud of all the MPI hydro code written for this course: http://www.wordle.net) C MPI is a Library for Message-Passing Not built in to compiler Function calls that can be made
More informationMPI is a Library for Message-Passing
C MPI is a Library for Message-Passing Not built in to compiler Function calls that can be made from any compiler, many languages Just link to it Wrappers: mpicc, mpif77 Fortran MPI is a Library for CPU1
More informationFor developers. If you do need to have all processes write e.g. debug messages, you d then use channel 12 (see below).
For developers A. I/O channels in SELFE You need to exercise caution when dealing with parallel I/O especially for writing. For writing outputs, you d generally let only 1 process do the job, e.g. if(myrank==0)
More informationFabio AFFINITO.
Introduction to Message Passing Interface Fabio AFFINITO Collective communications Communications involving a group of processes. They are called by all the ranks involved in a communicator (or a group)
More informationPCAP Assignment I. 1. A. Why is there a large performance gap between many-core GPUs and generalpurpose multicore CPUs. Discuss in detail.
PCAP Assignment I 1. A. Why is there a large performance gap between many-core GPUs and generalpurpose multicore CPUs. Discuss in detail. The multicore CPUs are designed to maximize the execution speed
More informationMPI Runtime Error Detection with MUST
MPI Runtime Error Detection with MUST At the 27th VI-HPS Tuning Workshop Joachim Protze IT Center RWTH Aachen University April 2018 How many issues can you spot in this tiny example? #include #include
More informationParallel I/O. Steve Lantz Senior Research Associate Cornell CAC. Workshop: Parallel Computing on Ranger and Lonestar, May 16, 2012
Parallel I/O Steve Lantz Senior Research Associate Cornell CAC Workshop: Parallel Computing on Ranger and Lonestar, May 16, 2012 Based on materials developed by Bill Barth at TACC Introduction: The Parallel
More informationCOSC 6374 Parallel Computation. Introduction to MPI V Derived Data Types. Edgar Gabriel Fall Derived Datatypes
COSC 6374 Parallel Computation Introduction to MPI V Derived Data Types Edgar Gabriel Fall 2013 Derived Datatypes Basic idea: describe memory layout of user data structures e.g. a structure in C typedef
More informationLecture 16. Parallel Sorting MPI Datatypes
Lecture 16 Parallel Sorting MPI Datatypes Today s lecture MPI Derived Datatypes Parallel Sorting 2 MPI Datatypes Data types MPI messages sources need not be contiguous 1-dimensional arrays The element
More informationCS4961 Parallel Programming. Lecture 16: Introduction to Message Passing 11/3/11. Administrative. Mary Hall November 3, 2011.
CS4961 Parallel Programming Lecture 16: Introduction to Message Passing Administrative Next programming assignment due on Monday, Nov. 7 at midnight Need to define teams and have initial conversation with
More informationCS 179: GPU Programming. Lecture 14: Inter-process Communication
CS 179: GPU Programming Lecture 14: Inter-process Communication The Problem What if we want to use GPUs across a distributed system? GPU cluster, CSIRO Distributed System A collection of computers Each
More informationParallel I/O with MPI TMA4280 Introduction to Supercomputing
Parallel I/O with MPI TMA4280 Introduction to Supercomputing NTNU, IMF March 27. 2017 1 Limits of data processing Development of computational resource allow running more complex simulations. Volume of
More informationLesson 1. MPI runs on distributed memory systems, shared memory systems, or hybrid systems.
The goals of this lesson are: understanding the MPI programming model managing the MPI environment handling errors point-to-point communication 1. The MPI Environment Lesson 1 MPI (Message Passing Interface)
More informationParallel IO Benchmarking
Parallel IO Benchmarking Jia-Ying Wu August 17, 2016 MSc in High Performance Computing with Data Science The University of Edinburgh Year of Presentation: 2016 Abstract The project is aimed to investigate
More informationMPI. (message passing, MIMD)
MPI (message passing, MIMD) What is MPI? a message-passing library specification extension of C/C++ (and Fortran) message passing for distributed memory parallel programming Features of MPI Point-to-point
More informationTutorial 2: MPI. CS486 - Principles of Distributed Computing Papageorgiou Spyros
Tutorial 2: MPI CS486 - Principles of Distributed Computing Papageorgiou Spyros What is MPI? An Interface Specification MPI = Message Passing Interface Provides a standard -> various implementations Offers
More informationAnomalies. The following issues might make the performance of a parallel program look different than it its:
Anomalies The following issues might make the performance of a parallel program look different than it its: When running a program in parallel on many processors, each processor has its own cache, so the
More informationint MPI_Cart_shift ( MPI_Comm comm, int direction, int displ, int *source, int *dest )
Lecture 10 int MPI_Cart_shift ( MPI_Comm comm, int direction, int displ, int *source, int *dest ) comm - communicator with Cartesian structure direction - coordinate dimension of shift, in range [0,n-1]
More informationLecture 6: Parallel Matrix Algorithms (part 3)
Lecture 6: Parallel Matrix Algorithms (part 3) 1 A Simple Parallel Dense Matrix-Matrix Multiplication Let A = [a ij ] n n and B = [b ij ] n n be n n matrices. Compute C = AB Computational complexity of
More informationAn Introduction to MPI
An Introduction to MPI Parallel Programming with the Message Passing Interface William Gropp Ewing Lusk Argonne National Laboratory 1 Outline Background The message-passing model Origins of MPI and current
More informationLog into scinet with your account:
Log into scinet with your account: ssh -Y hpcsioxx@login.scinet.utoronto.ca ssh -Y gpc01 qsub -X -I -l nodes=1:ppn=8,walltime=4:00:00 cd ${SCRATCH} cp -r /scinet/course/pario. cd pario source parallellibs
More informationDistributed Memory Parallel Programming
COSC Big Data Analytics Parallel Programming using MPI Edgar Gabriel Spring 201 Distributed Memory Parallel Programming Vast majority of clusters are homogeneous Necessitated by the complexity of maintaining
More informationIntroduction to MPI. HY555 Parallel Systems and Grids Fall 2003
Introduction to MPI HY555 Parallel Systems and Grids Fall 2003 Outline MPI layout Sending and receiving messages Collective communication Datatypes An example Compiling and running Typical layout of an
More informationCOSC 6374 Parallel Computation. Derived Data Types in MPI. Edgar Gabriel. Spring Derived Datatypes
COSC 6374 Parallel Computation Derived Data Types in MPI Spring 2008 Derived Datatypes Basic idea: interface to describe memory layout of user data structures e.g. a structure in C typedef struct { char
More informationProgramming with MPI. advanced point to point. Type to enter text. Jan Thorbecke. Challenge the future. Delft University of Technology
Programming with MPI advanced point to point Jan Thorbecke Type to enter text Delft University of Technology Challenge the future Acknowledgments This course is partly based on the MPI courses developed
More informationMessage Passing with MPI
Message Passing with MPI PPCES 2016 Hristo Iliev IT Center / JARA-HPC IT Center der RWTH Aachen University Agenda Motivation Part 1 Concepts Point-to-point communication Non-blocking operations Part 2
More informationSlides prepared by : Farzana Rahman 1
Introduction to MPI 1 Background on MPI MPI - Message Passing Interface Library standard defined by a committee of vendors, implementers, and parallel programmers Used to create parallel programs based
More informationMPI Correctness Checking with MUST
Center for Information Services and High Performance Computing (ZIH) MPI Correctness Checking with MUST Parallel Programming Course, Dresden, 8.- 12. February 2016 Mathias Korepkat (mathias.korepkat@tu-dresden.de
More informationIntroduction to Parallel. Programming
University of Nizhni Novgorod Faculty of Computational Mathematics & Cybernetics Introduction to Parallel Section 4. Part 2. Programming Parallel Programming with MPI Gergel V.P., Professor, D.Sc., Software
More informationMore advanced MPI and mixed programming topics
More advanced MPI and mixed programming topics Extracting messages from MPI MPI_Recv delivers each message from a peer in the order in which these messages were send No coordination between peers is possible
More informationParallel I/O. Jonathan Dursi, SciNet
Parallel I/O Jonathan Dursi, SciNet ljdursi@scinet.utoronto.ca HPCS2011 Agenda Intro to I/O MPI-IO HDF5, NetCDF4 Parallel HDF5/NetCDF4 HPCS2011 Data is getting bigger Increase in computing power makes
More informationHigh performance computing. Message Passing Interface
High performance computing Message Passing Interface send-receive paradigm sending the message: send (target, id, data) receiving the message: receive (source, id, data) Versatility of the model High efficiency
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2017 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationCS4961 Parallel Programming. Lecture 18: Introduction to Message Passing 11/3/10. Final Project Purpose: Mary Hall November 2, 2010.
Parallel Programming Lecture 18: Introduction to Message Passing Mary Hall November 2, 2010 Final Project Purpose: - A chance to dig in deeper into a parallel programming model and explore concepts. -
More informationParallel Programming
Parallel Programming Prof. Paolo Bientinesi pauldj@aices.rwth-aachen.de WS 16/17 Point-to-point communication Send MPI_Ssend MPI_Send MPI_Isend. MPI_Bsend Receive MPI_Recv MPI_Irecv Paolo Bientinesi MPI
More informationMA471. Lecture 5. Collective MPI Communication
MA471 Lecture 5 Collective MPI Communication Today: When all the processes want to send, receive or both Excellent website for MPI command syntax available at: http://www-unix.mcs.anl.gov/mpi/www/ 9/10/2003
More informationCS 470 Spring Mike Lam, Professor. Distributed Programming & MPI
CS 470 Spring 2018 Mike Lam, Professor Distributed Programming & MPI MPI paradigm Single program, multiple data (SPMD) One program, multiple processes (ranks) Processes communicate via messages An MPI
More informationDPHPC Recitation Session 2 Advanced MPI Concepts
TIMO SCHNEIDER DPHPC Recitation Session 2 Advanced MPI Concepts Recap MPI is a widely used API to support message passing for HPC We saw that six functions are enough to write useful
More information15-440: Recitation 8
15-440: Recitation 8 School of Computer Science Carnegie Mellon University, Qatar Fall 2013 Date: Oct 31, 2013 I- Intended Learning Outcome (ILO): The ILO of this recitation is: Apply parallel programs
More informationMPI Workshop - III. Research Staff Cartesian Topologies in MPI and Passing Structures in MPI Week 3 of 3
MPI Workshop - III Research Staff Cartesian Topologies in MPI and Passing Structures in MPI Week 3 of 3 Schedule 4Course Map 4Fix environments to run MPI codes 4CartesianTopology! MPI_Cart_create! MPI_
More informationIntermediate MPI features
Intermediate MPI features Advanced message passing Collective communication Topologies Group communication Forms of message passing (1) Communication modes: Standard: system decides whether message is
More informationMPI Collective communication
MPI Collective communication CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) MPI Collective communication Spring 2018 1 / 43 Outline 1 MPI Collective communication
More informationLecture 33: More on MPI I/O. William Gropp
Lecture 33: More on MPI I/O William Gropp www.cs.illinois.edu/~wgropp Today s Topics High level parallel I/O libraries Options for efficient I/O Example of I/O for a distributed array Understanding why
More informationThe MPI Message-passing Standard Practical use and implementation (VI) SPD Course 08/03/2017 Massimo Coppola
The MPI Message-passing Standard Practical use and implementation (VI) SPD Course 08/03/2017 Massimo Coppola Datatypes REFINING DERIVED DATATYPES LAYOUT FOR COMPOSITION SPD - MPI Standard Use and Implementation
More informationIntroduction to MPI part II. Fabio AFFINITO
Introduction to MPI part II Fabio AFFINITO (f.affinito@cineca.it) Collective communications Communications involving a group of processes. They are called by all the ranks involved in a communicator (or
More informationLecture 34: One-sided Communication in MPI. William Gropp
Lecture 34: One-sided Communication in MPI William Gropp www.cs.illinois.edu/~wgropp Thanks to This material based on the SC14 Tutorial presented by Pavan Balaji William Gropp Torsten Hoefler Rajeev Thakur
More informationNCSU CSC 548 Parallel Systems. Homework 6. Project Report for Integration of. MPI I/O into Record Framework. By Lily Shi
NCSU CSC 548 Parallel Systems Homework 6 Project Report for Integration of MPI I/O into Record Framework By Lily Shi (username: xshi) URL: http://www4.ncsu.edu/~xshi/csc548prj4.html November 26, 2006,
More informationCS 426. Building and Running a Parallel Application
CS 426 Building and Running a Parallel Application 1 Task/Channel Model Design Efficient Parallel Programs (or Algorithms) Mainly for distributed memory systems (e.g. Clusters) Break Parallel Computations
More informationCOSC 6374 Parallel Computation. Remote Direct Memory Acces
COSC 6374 Parallel Computation Remote Direct Memory Acces Edgar Gabriel Fall 2013 Communication Models A P0 receive send B P1 Message Passing Model: Two-sided communication A P0 put B P1 Remote Memory
More informationTopics. Lecture 7. Review. Other MPI collective functions. Collective Communication (cont d) MPI Programming (III)
Topics Lecture 7 MPI Programming (III) Collective communication (cont d) Point-to-point communication Basic point-to-point communication Non-blocking point-to-point communication Four modes of blocking
More informationChapter 4. Message-passing Model
Chapter 4 Message-Passing Programming Message-passing Model 2 1 Characteristics of Processes Number is specified at start-up time Remains constant throughout the execution of program All execute same program
More informationProgramming Scalable Systems with MPI. Clemens Grelck, University of Amsterdam
Clemens Grelck University of Amsterdam UvA / SurfSARA High Performance Computing and Big Data Course June 2014 Parallel Programming with Compiler Directives: OpenMP Message Passing Gentle Introduction
More informationDistributed Memory Machines and Programming. Lecture 7
Distributed Memory Machines and Programming Lecture 7 James Demmel www.cs.berkeley.edu/~demmel/cs267_spr16 Slides from Kathy Yelick CS267 Lecture 7 1 Outline Distributed Memory Architectures Properties
More informationMPI-Hello World. Timothy H. Kaiser, PH.D.
MPI-Hello World Timothy H. Kaiser, PH.D. tkaiser@mines.edu 1 Calls we will Use MPI_INIT( ierr ) MPI_Get_Processor_name(myname,resultlen,ierr) MPI_FINALIZE(ierr) MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr
More informationCSE 613: Parallel Programming. Lecture 21 ( The Message Passing Interface )
CSE 613: Parallel Programming Lecture 21 ( The Message Passing Interface ) Jesmin Jahan Tithi Department of Computer Science SUNY Stony Brook Fall 2013 ( Slides from Rezaul A. Chowdhury ) Principles of
More informationMPI Runtime Error Detection with MUST
MPI Runtime Error Detection with MUST At the 25th VI-HPS Tuning Workshop Joachim Protze IT Center RWTH Aachen University March 2017 How many issues can you spot in this tiny example? #include #include
More informationProgramming with MPI: Advanced Topics
Programming with MPI: Advanced Topics Steve Lantz Senior Research Associate Cornell CAC Workshop: Introduction to Parallel Computing on Ranger, May 19, 2010 Based on materials developed by by Bill Barth
More informationMPI-Hello World. Timothy H. Kaiser, PH.D.
MPI-Hello World Timothy H. Kaiser, PH.D. tkaiser@mines.edu 1 Calls we will Use MPI_INIT( ierr ) MPI_Get_Processor_name(myname,resultlen,ierr) MPI_FINALIZE(ierr) MPI_COMM_RANK( MPI_COMM_WORLD, myid, ierr
More informationParallel Programming Using Basic MPI. Presented by Timothy H. Kaiser, Ph.D. San Diego Supercomputer Center
05 Parallel Programming Using Basic MPI Presented by Timothy H. Kaiser, Ph.D. San Diego Supercomputer Center Talk Overview Background on MPI Documentation Hello world in MPI Basic communications Simple
More informationlslogin3$ cd lslogin3$ tar -xvf ~train00/mpibasic_lab.tar cd mpibasic_lab/pi cd mpibasic_lab/decomp1d
MPI Lab Getting Started Login to ranger.tacc.utexas.edu Untar the lab source code lslogin3$ cd lslogin3$ tar -xvf ~train00/mpibasic_lab.tar Part 1: Getting Started with simple parallel coding hello mpi-world
More informationIntegrating logical and physical file models in the MPI-IO implementation for Clusterfile
Integrating logical and physical file models in the MPI-IO implementation for Clusterfile Florin Isailă, David Singh, Jesús Carretero, Félix Garcia Departement of Computer Science University Carlos III
More information