Data Space Oriented Tiling

Transcription

1 Dt Spce Oriented Tiling Mhmut Kndemir Deprtment of Computer Science nd Engineering The Pennsylvni Stte University University Prk, PA 16802, USA Abstrct. An optimizing compiler cn ply n importnt role in enhncing dt loclity in rry-intensive pplictions with regulr dt ccess ptterns. This pper presents compiler-bsed dt spce oriented tiling pproch (DST). In this strtegy, the dt spce (i.e., the rry index spce) is logiclly divided into chunks (clled dt tiles) nd ech dt tile is processed in turn. In processing dt tile, our pproch trverses the entire itertion spce of ll nests in the code nd executes ll itertions (potentilly coming from different nests) tht ccess the dt tile being processed. In doing so, it lso tkes dt dependences into ccount. Since dt spce is common cross ll nests tht ccess it, DST cn potentilly chieve better results thn trditionl tiling by exploiting inter-nest dt loclity. This pper lso shows how dt spce oriented tiling cn be used for improving the performnce of softwremnged scrtch pd memories. 1 Introduction Itertion spce tiling (lso clled loop blocking) [9,1] is loop-oriented optimiztion iming t improving dt loclity. The ide behind tiling is to divide given itertion spce into chunks such tht the dt elements ccessed by given chunk fit in the vilble cche memory cpcity. Previously-published work itertion spce tiling reports significnt improvements in cche miss rtes nd progrm execution times. Compilers use itertion spce tiling minly to crete the blocked version of given nested loop utomticlly. Note tht, in generl, it is difficult to gurntee tht the rry elements ccessed by given itertion spce tile will fit in the cche. This problem occurs becuse tile shpes nd tiling style re decided bsed on loop behvior rther thn the dt elements ccessed. In prticulr, most of the current pproches to tiling do not consider the shpe of the dt regions (from different rrys) touched by n itertion spce tile. In this pper, we discuss nd evlute dt spce oriented tiling (DST), vrint of clssicl itertion spce oriented tiling, to chieve better dt loclity thn clssicl tiling. Insted of tiling itertion spce first, nd then considering dt spce requirements of the resulting tiles (dt regions) in dt spce, DST tkes dt spce oriented pproch. Specificlly, it first logiclly divides dt spce into tiles (clled dt spce tiles or dt tiles for short), nd then processes D. Le Métyer (Ed.): ESOP 2002, LNCS 2305, pp , c Springer-Verlg Berlin Heidelberg 2002

2 Dt Spce Oriented Tiling 179 ech dt tile in sequence. Processing dt tile involves determining the set of loop itertions tht ccess the elements in tht dt tile nd executing these itertions tking into ccount dt dependences. Since it strts its nlysis from dt spce, DST hs two min dvntges over itertion spce tiling: Since dt spce of given rry is shred cross ll nests tht ccess the corresponding rry, DST hs more globl view of the progrm-wide ccess pttern (thn itertion spce tiling). This is especilly true if one cn come up with strtegies to summrize ccess ptterns of multiple nests on given rry. Working on dt spce llows compiler to tke lyout constrints into ccount better. For instnce, in selecting dt tile shpe, in ddition to other prmeters, the compiler cn lso consider memory lyout of the rry in question. This pper describes dt spce oriented tiling pproch nd presents strtegy for determining dt tile shpes nd sizes utomticlly. It lso shows how DST cn be used in conjunction with scrtch pd memory (SPM). The reminder of this pper is orgnized s follows. Section 2 revises clssicl itertion spce tiling nd discusses how it improves dt loclity. Section 3 presents description of dt spce oriented tiling, focusing in prticulr on issues such s selection of dt tile shpes, trversing itertion spce, nd hndling dt dependences. Section 4 discusses n ppliction of DST to optimizing the effectiveness of scrtch pd memories, which re compiler-mnged on-chip SRAMs. Section 5 concludes the pper with summry of our mjor contributions. 2 Review of Itertion Spce Tiling An importnt technique used to improve cche performnce (dt loclity) by mking better use of cche lines is itertion spce tiling (lso clled loop blocking) [1,2,9]. In tiling, dt structures tht re too big to fit in the cche re (logiclly) broken up into smller pieces tht will fit in the cche. In other words, insted of operting on entire columns or rows of given rry, tiling enbles opertions on multi-dimensionl sections of rrys t one time. The objective here is to keep the ctive sections of the rrys in fster levels of memory hierrchy (e.g., dt cches) s long s possible so tht when dt item (rry element) is reused, it cn be ccessed from the fster memory insted of the slower memory. For n illustrtion of tiling, consider the mtrix-multiply code given in Figure 1(). Let us ssume tht the lyouts of ll the rrys re row-mjor. It is esy to see tht, from the cche loclity perspective, this loop nest my not exhibit very good performnce (depending on the ctul rry sizes nd cche cpcity). The reson is tht, lthough rry U 2 hs temporl reuse in the innermost loop (the k loop) nd successive itertions of this loop ccess consecutive elements from rry U 0 (i.e., rry U 0 hs sptil reuse in the innermost loop), the successive ccesses to rry U 1 touch different rows of this rry. Obviously, this is not good style of ccess for row-mjor rry. Using stte-of-the-rt

3 180 Mhmut Kndemir optimizing compiler technology (e.g., [5]), we cn derive the code shown in Figure 1(b), given the code in Figure 1(). In this optimized code, the rry U 0 hs temporl reuse in the innermost loop (the j loop now) nd the rrys U 1 nd U 2 hve sptil reuses, mening tht the successive itertions of the innermost loop touch consecutive elements from both the rrys. However, unless the fster memory in question is lrge enough to hold the entire N N rry U 1, mny elements of this rry will probbly be replced from the cche before they re reused in successive itertions of the outermost i loop. Insted of operting on individul rry elements, tiling chieves reuse of rry sections by performing the clcultions (in our cse mtrix multipliction) on rry sections (in our cse sub-mtrices). Figure 1(c) shows the tiled version of Figure 1(b). This tiled version is from [1]. In the tiled code, the loops kk nd jj re clled the tile loops, wheres the loops i, k, ndj re clled the element loops. It is importnt to choose the tile size (blocking fctor) B such tht ll the B 2 +2NB rry items ccessed by the element loops i, k, j should fit in the fster memory (e.g., cche). In other words, the tiled version of the mtrixmultiply code opertes on N B sub-mtrices of rrys U 0 nd U 2,ndB B sub-mtrix of rry U 1 t one time. Assuming tht the mtrices in this exmple re in min memory to begin with, ensuring tht B 2 +2NB rry elements cn be kept in cche might be sufficient to obtin high levels of performnce. In prctice, however, depending on the cche size, cche ssocitivity, nd bsolute rry ddresses in memory, cche conflicts cn occur. Consequently, the tile size B is set to much smller vlue thn necessry [1]. for(i =1;i N; i ++) for(j =1;j N; j ++) for(k =1;k N; k ++) U 2[i][j]+ = U 0[i][k] U 1[k][j]; () for(i =1;i N; i ++) for(k =1;k N; k ++) for(j =1;j N; j ++) U 2[i][j]+ = U 0[i][k] U 1[k][j]; (b) for(kk =1;kk N; kk = kk + B) for(jj =1;jj N; jj = jj + B) for(i =1;i N; i ++) for(k = kk; k min(n, kk + B 1); k ++) for(j = jj; j min(n, jj + B 1); j ++) U 2[i][j]+ = U 0[i][k] U 1[k][j]; (c) Fig. 1. () Mtrix-multiply nest. (b) Loclity-optimized version. (c) Tiled version.

4 3 Description of Dt Spce Oriented Tiling Dt Spce Oriented Tiling 181 The trditionl itertion spce tiling tiles the itertion spce tking into ccount dt dependences in the code. In such tiling strtegy, it is not gurnteed tht the rry elements ccessed by given itertion spce tile form region (which we cn refer to s dt tile) tht exhibits loclity. Also, since ech nest is tiled independently from other nests in the code, it my not be possible to exploit potentil dt reuse between different nests due to common rry regions ccessed. In contrst, dt spce oriented tiling (DST) tkes different pproch. Insted of focussing on itertion spce, it first considers dt spce (e.g., n rry). Specificlly, it divides given rry into logicl prtitions (dt tiles) nd processes ech dt tile in turn. In processing dt tile, it trverses the entire itertion spce of ll nests in the code nd executes ll itertions (potentilly coming from different nests) tht ccess the current dt tile being processed. In doing so, it tkes dt dependences into ccount. Note tht since dt spce is common cross ll nests tht ccess it, DST cn potentilly chieve better results thn trditionl tiling by exploiting inter-nest loclity. Note lso tht, s opposed to trdition tiling, this dt oriented tiling pproch cn lso hndle imperfectly-nested loops esily s it is not restricted by the wy the loops in the nests re structured. In [4], Kodukul et l. present dt oriented tiling strtegy clled dt shckling, which is similr to our pproch in spirit. However, there re significnt differences between these two optimiztion strtegies. First, the wy tht dt tile shpe is determined in [4] is experimentl. Since they minly focus on liner lgebr codes, they decided tht using squre (or rectiliner) tiles would work well most of the time; tht is, such tiles led to legl (semntics-preserving) ccess ptterns. In comprison, we first summrize the ccess ptterns of multiple nests on dt spce using dt reltion vectors, nd then select suitble tile shpe so s to minimize the communiction volume. Second, their optimiztion strtegy considers only single imperfectly-nested loop t time, while we ttempt to optimize ll the nests in the code simultneously. Therefore, our pproch is expected to exploit inter-nest dt reuse better. Third, we lso present n utomted strtegy to hndle dt dependences. Insted, their work is more oriented towrds determining leglity for given dt tile (using polyhedrl tool). Finlly, the ppliction domins of these two techniques re lso different. The pproch discussed in [4] is specificlly designed for optimizing cche loclity. As will be explined lter in the pper, we insted minly focus on improving memory energy consumption of scrtch pd memory bsed rchitecture. Therefore, in our cse, optimizing inter-nest reuse is more importnt. To illustrte the difference between itertion spce oriented tiling nd dt spce oriented tiling, we consider the scenrio in Figure 2 where n rry is mnipulted using three seprte nests nd there re no intr-nest or inter-nest dt dependences in the code. As shown on the upper-left portion of the figure, the rry is divided into two sections (regions): nd b. The itertion spces of the nests re divided into four regions. Ech region is identified using letters or b to indicte the dt region it ccesses. Figure 2 lso shows three possi-

5 182 Mhmut Kndemir ble execution orders. In the execution order (I), the trditionl tiling pproch is shown, ssuming tht the sections in the itertion spce re processed from left-to-right nd top-to-bottom. We clerly see tht there re frequent trnsitions between -blocks nd b-blocks, which is not good from the dt loclity perspective. The execution order (II) illustrtes dt oriented pproch which restricts its optimiztion scope to single nest t time (s in [4]). Tht is, it hndles nests one-by-one, nd in processing nest it clusters itertions tht ccess given dt region. Consequently, it does not incur trnsitions between -blocks nd b-blocks in executing given nest, big dvntge over the scheme in (I). However, in going from one nest to nother, it incurs trnsitions between -blocks nd b-blocks. Finlly, the execution order (III) represents our pproch. In this strtegy, we process dt regions one-by-one, nd in processing region, we execute ll itertions from ll nests tht ccess the sid region. Therefore, our pproch first executes itertions (considering ll nests) tht ccess -block, nd then executes ll itertions tht ccess b-blocks. Consequently, there is only one trnsition between -blocks nd b-blocks. However, there re mny issues tht need to be ddressed. First, in some cses, inherent dt dependences in the progrm my not llow interleving loop itertions from different nests. Second, in generl, given code my contin multiple rrys tht need to be tken into ccount. Third, shpe of the dt regions might lso hve significnt impct on the success of the strtegy (in prticulr, when we hve dt dependences). Dt spce oriented tiling is performed in two steps: (i) selecting n rry (clled the seed rry) nd determining suitble tile shpe for tht rry, nd (ii) iterting through dt tiles nd for ech tile executing ll itertions (from ll nests in the code) tht mnipulte rry elements in the dt tile. In the reminder of this pper, we ddress these issues in detil. Dt Spce 1st Itertion Spce 2nd Itertion Spce 3rd Itertion Spce b b b b b b b (I) b b b b b b (II) b b b b b b (III) b b b b b b Fig. 2. Comprison of itertion spce oriented tiling nd dt spce oriented tiling. The loop itertors surrounding ny sttement cn be represented s n n 1 column vector: i =[i 1,i 2,,i n ] T, where n is the number of enclosing loop itertors. The loop bounds of the itertors cn be described by system of inequlities which define the polyhedron Ai b where A is n l n integer

6 Dt Spce Oriented Tiling 183 mtrix nd b is n l vector. The integer vlues tken by i define the itertion spce of the itertors. The dt storge of n rry U 0 cn lso be viewed s (rectiliner) polyhedron. The index domin of rry U 0 cn be described using index vectors: = [ T 1, 2,, dim(u0)], where dim(u0 ) refers to the dimensionlity of U 0. The index vectors hve certin rnge which describe the size of the rry, or dt spce: µ LB µ UB,wherethedim(U 0 ) 1 vectors µ LB nd µ UB correspond to lower nd upper bounds of the rry, respectively. In this pper, we ssume tht µ LB =[1, 1,, 1, 1]; tht is, the lowest index vlue in ech subscript position is 1. The subscript function for reference to rry U 0 represents mpping from itertion spce to dt spce. An itertion vector i is sid to ccess (or reference) n rry element indexed by if there exists subscript function (or rry reference) R U0 (.) such tht R U0 (i) =. In our context, n rry reference cn be written s n ffine mpping tht hs the form Li + o, where L is dim(u 0 ) n mtrix nd o is dim(u 0 ) 1 vector. For exmple, n rry reference such s U 0 [i 1][i + j +2]intwo-levelnestedloop(wherei is the outer loop nd j is the inner loop) cn be represented s R U0 (i) =Li + o = [ 10 11][ i j ] + [ 1 2 ], where i =[i j] T. When there is no confusion, we write R U0 N k to indicte tht the reference R U0 (.) ppers in nest N k. 3.1 Arry Selection nd Tile Shpes The first step in DST is selecting suitble rry (clled the seed rry) from mong the rrys declred in the code nd determining suitble dt tile shpe for this rry. Once the shpe of the tile hs been determined, its sizes in different dimensions cn be found by scling up its shpe. Let us ssume for now tht we hve lredy selected seed rry, U 0.Adt tile corresponds to set of rry elements in dt spce (rry). To define suitble dt tile for given seed rry, we need to consider the ccess pttern of ech nest on the sid rry. For given seed rry U 0 nd nest N i, the seed element of U 0 with respect to nest N i, denoted s U0,N i, is the lexicogrphiclly smllest element of the rry ccessed by N i. Bsed on this definition, the globl seed element g U0 for rry U 0 is the smllest rry element ccessed by ll nests in the code. In cses where there is not such globl seed, we select n element which is ccessed by most of the nests. Using this globl seed element, we determine seed itertion for ech nest s follows. The seed itertion of nest N i with respect to rry U 0 is n itertion i su0,n i tht mong the elements ccessed by this itertion, g U0 is the smllest one in lexicogrphic sense. If there re multiple seed itertions (for given nest), we select the lexicogrphiclly smllest one. Then, we define the footprint of nest N i with respect to rry U 0 (denoted F U0,N i ) s the set of elements ccessed by i su0,n i. More precisely, F U0,N i = {f f = R U0 (i su0,n i ) for ll R U0 N i }. Let us define set of vectors (V U0,N i ), clled dt reltion vectors, on the dt spce of U 0 using the elements in the footprint F U0,N i. Specificlly, let F U0,N i =

7 184 Mhmut Kndemir {f 1, f 2,, f k }, where the elements in this set re ordered lexicogrphiclly, f 1 being the lexicogrphiclly smllest one. Ech v j Vrepresents vector between f i nd f k, where k>i. In other words, by doing so, we define set of lexicogrphiclly positive vectors between ll dt point pirs in F U0,N i. We cn write V U0,N i s mtrix [v 1 ; v 2 ; ; v L ]. This mtrix is termed s the locl dt reltion mtrix. The globl dt reltion mtrix of rry U 0 (denoted G U0 )isthecombintion of locl dt reltion mtrices coming from individul nests; tht is, G U0 =[V U0,N 1 ; V U0,N 2 ; V U0,N P ], where P is the number of nests in the code. If desired, the (column) vectors in G U0 cn be re-ordered ccording to their frequency of occurrence. Our pproch uses G U0 to define tile shpes on dt spce. Specificlly, we first find the vectors in G U0 ndcovertheentiredtspce(of the rry in question) using these vectors. The positions of these vectors on the dt spce is used in selecting dt tile shpe. Our objective in selecting dt tile shpe is to ensure tht, when executing group of itertions tht ccess the elements in given dt tile, the number of non-tile elements ccessed should be minimized s much s possible. Obviously, the shpe of the dt tile plys mjor role in determining the number of the non-tile elements ccessed. Arry Spce Itertion Spce (II) Dt Tile Off-Tile Element (I) Itertion Tile Fig. 3. Going from dt tile to itertion tile nd off-tile (non-tile) elements. We next define communiction volume s the number of non-tile elements ccessed during the execution of itertions tht mnipulte the elements in the tile. It should be noted tht, for given dt tile, the execution of ech nest might incur non-zero communiction volume. We then try to minimize the globl (over ll nests) communiction volume. It should lso be noted tht non-tile element ccess occurs due to reltion vector tht crosses tile boundry (i.e., one of its end-points re inside the tile wheres the other end-point lies outside the tile). As n exmple, consider the itertion spce nd dt spce shown in Figure 3. Considering the dt tile on the left side of the figure, our pproch determines n itertion tile (on the itertion spce). This ctivity is mrked (I) in the figure. The itertions in the itertion tile re the ones tht ccess the rry elements in the dt tile. We will mke more ccurte definition of itertion tile lter in the pper. Next, the entire set of rry elements ccessed by this itertion tile is determined. This step corresponds to (II) in the figure. These rry elements re delimited using dshed box in the figure. The rry elements

8 Dt Spce Oriented Tiling 185 tht re within the dshed box but outside the dt tile re clled off-tile (or non-tile) elements. The objective of our tile selection strtegy is to minimize the number of off-tile elements. A given dt tile cn be defined using set of hyperplnes. Specificlly, dt tiles in n M-dimensionl spce cn be defined by M fmilies of prllel hyperplnes (or plnes), ech of which is n (M 1)-dimensionl hyperplne. Dt tiles so defined re prllelepipeds (except for those ner the boundry of the dt spce) nd ech tile is n M-dimensionl subset of the dt spce. Thus, the shpe of the tiles is defined by the fmilies of plnes nd the size of the tiles is defined by the distnce of seprtion between djcent pirs of prllel plnes in ech of the M fmilies. We cn represent given tile to rry U 0 using M vectors, where the ith vector p i (1 i M) corresponds to the ith boundry of the tile. These vectors cn collectively be written s mtrix P U0 =[p 1 ; p 2 ; ; p M ]. Alterntively, given dt tile cn be defined using nother mtrix, H U0,ech row of which is perpendiculr to given tile boundry. It cn be shown tht H U0 = P 1 U0. Consequently, to define dt tile, we cn either specify the columns of P U0 or the rows of H U0. We then try to select tile shpe such tht the number of dt reltion vectors intersected by tile boundries will be minimum. As mentioned erlier, ech such vector (lso referred to s the communiction vector) represents two elements, one of which is within the tile wheres the other is outside the tile. Note tht such vectors re the most importnt ones to concentrte on s the vectors with both the ends re outside cn be converted to either the communiction vectors or the vectors which re contined completely in the tile by mking the tile lrge enough. It should lso be noted tht using H U0 nd G U0,we cn represent the communiction requirements in concise mnner. Specificlly, since dt tiles re seprted by tile boundries (defined by H U0 ), communiction vector must cross the tile boundry between the tiles. A non-zero entry in G U 0 = H U0 G U0,sytheentryin(i, j), implies tht communiction is incurred due to the jth communiction vector poking the ith tile boundry. The mount of communiction cross tile boundry, defined by the ith row of H U0,is function of the sum of the entries in the ith row of G U 0. Bsed on this, we cn formulte the problem of finding tiling plnes s tht of finding trnsformtion H U0 such tht the communiction volume (due to communiction vectors) will be minimum. Note tht the communiction volume is proportionl to: M S M h i,k v k,j. i=1 j=1 k=1 As n exmple, let us consider the code frgment given below, which consists of two seprte nests. Figure 4() shows the locl dt reltion vectors for ech nest s well s the globl dt reltion vector (only the first 3 3 portion of the rry is shown for clrity). Figure 4(b) shows how the globl dt reltion vectors cn be used to cover the entire dt spce. This picture is then used to select suitble tile shpe. It should be noted tht the globl dt reltion mtrix in this exmple is: [ ] 01 G U0 =. 11

9 186 Mhmut Kndemir for(i =1;i N 1; i ++) for(j =1;j N 1; j ++) {U 0[i][j],U 0[i +1][j +1]}; for(i =1;i N; i ++) for(j =1;j N 1; j ++) {U 0[i][j],U 0[i][j +1]}; Assuming dt tile cpcity (size) of six elements, Figure 4(c) shows three lterntive dt tile shpes with their communiction vectors. It should be noted tht ech tile in this figure hs different communiction volume. For exmple, the dt tile in (I) hs communiction volume of 12, corresponding to six in-coming edges nd six out-going edges. The tile in (II), on the other hnd, hs communiction volume of 14. Finlly, the tile (III) hs communiction volume of 10. Consequently, for the best results, tile (III) should be selected. In fct, it is esy to see tht, in this exmple, if the dimension sizes of the rectngulr dt tile (s in (I) nd (II)) re n (verticl) nd m (horizontl), then the communiction volume is 4n +2(m 1). For the tile in (III), on the other hnd, the corresponding figure is 2(m + n). As n exmple, if n = m = 50, the communiction volume of the tile in (III) is 32% less thn the one in (I). To show how our pproch derives the tile shown in (III) for the exmple code frgment bove, let us define H U0 s: H U0 = [ ] h11 h 12. h 21 h 22 Consequently, [ ][ ] [ ] G h11 h U 0 = H U0 G U0 = h11 + h = 12 h 12. h 21 h h 21 + h 22 h 22 To minimize the communiction volume, the sum of (the bsolute vlues of) the entries in this lst mtrix should be minimum. This is becuse, s we hve discussed bove, ech non-zero entry in G U 0 represents communiction long one surfce of the tile. In mthemticl terms, we need to select h 11, h 12, h 21, nd h 22 such tht h 11 + h 12 + h 12 + h 21 + h 22 + h 22 should be minimized. A possible set of vlues for minimizing this is h 11 =1,h 12 =0,h 21 = 1, nd h 22 = 1, respectively, which gives us: [ ] 10 H U0 =, 1 1 which, in turn, mens [ ] P U0 = H 1 10 U0 =. 11 Recll tht ech column of the P U0 mtrix represents boundry of dt tile. So, the P U0 mtrix bove represents the dt tile (III) illustrted in Figure 4(c). We next explin how dt tile is ctully scled up. After selecting dt tile shpe, it is scled up in ech dimension. In scling up dt tile, we consider the itertions tht follow the seed itertion in execution order. The left prt of Figure 4(d) shows the globl dt reltion vectors defined by the seed itertion nd three other itertions tht follows it. We

10 Dt Spce Oriented Tiling 187 (b) () (c) (I) (II) (III) (d) Fig. 4. () Locl nd globl dt reltion vectors. (b) Dt spce covered by globl dt reltion vectors. (c) Three different dt tile shpes with their communiction vectors. (d) Scling up tile size bsed on rry lyout. (e) Tiling the entire dt spce. cn include s mny itertions s possible s long s the mximum (llowble) cpcity of dt tile is not exceeded. As will be discussed lter in the pper, the mximum cpcity of dt tile depends on the ppliction t hnd nd the memory rchitecture under considertion. The itertions tht (follow the seed itertion nd) re included in determining the size of tile constitute n itertion tile. It should be noted, however, tht we do not necessrily include the itertions tht immeditely follow the seed. Insted, we cn tke into ccount rry lyout nd determine suitble itertion tile such tht the sptil loclity is exploited s much s possible. For exmple, s we cn see on the left portion of Figure 4(d), for this exmple, progressing the reltion vectors (tht is, stretching the dt tile) long the horizontl xis mkes sense since the rry lyout is row-mjor. If, however, the rry lyout ws column-mjor, it would be more beneficil to stretch the tile long the verticl exis s illustrted one the right side of Figure 4(d). Our current implementtion tkes the mximum cpcity of the tile nd the rry lyout into ccount, nd determines the itertions in the itertion tile. Note tht, for given dt tile, ech nest my hve different itertion tile. As will be explined in detil in the next section, once suitble dt tile (shpe/size) hs been selected, our pproch considers dt tiles one-by-one, nd for ech dt tile, executes itertions tht ccess the rry elements in the tile. It should be noted, however, itertions tht mnipulte elements in given tile my lso ccess elements from different rrys. Assuming tht we hve tiles for these rrys s well, these ccesses my lso incur communiction (i.e., ccesses to non-tile elements). Consequently, just considering the seed rry nd its communiction volume my not be sufficient in obtining n overll stisfctory performnce (tht is, minimizing the communiction volume due to ll rrys nd ll nets). It should lso be noted tht the selection of the seed rry is very importnt s it determines the execution order of loop itertions,

11 188 Mhmut Kndemir how the itertions from different nests re interleved, nd tile shpes for other rrys. Our current pproch to the problem of selecting the most suitble seed rry is s follows. Since the number of rrys in given code is smll, we consider ech rry in turn s the seed rry nd compute the size of the overll communiction set. Then, we select the rry which leds to overll minimum communiction when used s the seed. 3.2 Trversing Itertion Spce In this subsection, we ssume tht there exists no dt dependences in the code. Once seed rry hs been determined nd dt tile shpe/size hs been selected, our pproch divides the rry into tiles. The tiles re exct copies of ech other except mybe t the boundires of the rry spce. It then restructures the code so tht the (re-structured) code, when executing, reds ech dt tile, executes loop itertions (possibly from different nests) tht ccesses its elements, nd moves to the next tile. Since, s explined in the previous section, we re creful in selecting the most suitble tile shpe, in executing itertions for given tile, the number of off-tile (non-tile) elements will be minimum. However, we need to be precise in defining the itertions tht mnipulte the elements in given tile. This is becuse even itertions tht re fr prt from ech other cn occsionlly ccess the sme element in given tile. For this purpose, we use the concept of the itertion tile given bove. Let us focus on specific dt tile of T elements: DT U0 = { 1, 2,, T 1, T }, ssuming tht g U0 = 1 nd H U0 is the corrsponding tile mtrix. Let U0,N i = {i 1, i 2,, T 1, T } be the corresponding itertion tile for nest N i,wherei su0,n i = i 1. When DT U0 is processed, the corresponding U0 is determined. This,in turn, determines the dt tiles for the other rrys in the code. This is depicted in Figure 5, ssuming tht there re three rrys in the code nd single nest: U 0 (the seed rry), U 1,ndU 2. We first determine the dt tile for U 0 (the seed rry). Then, using this dt tile, we find the corresponding itertion tile. After tht, using this itertion tile, we determine dt tiles for rrys U 1 nd U 2. Once this itertion tile is executed, our pproch processes the next tile from U 0 nd so on. If there exist multiple nests in the code being optimized, when we process the dt tile, we execute ll itertions from the corresponding itertion tiles of ll nests. Let us number the tiles in given dt spce (rry) from 1 to Y. Let us lso denote DT U0 (j) thejth dt tile (from rry U 0 )nd U0,N i (j) the corresponding itertion tile from nest N i. We process dt tiles nd execute corresponding itertion tiles in the following order (in Y steps): DT U0 (1) : U0,N 1 (1), U0,N 2 (1),, U0,N P (1) DT U0 (2) : U0,N 1 (2), U0,N 2 (2),, U0,N P (2)..:... DT U0 (Y ): U0,N 1 (Y ), U0,N 2 (Y ),, U0,N P (Y )

12 Dt Spce Oriented Tiling 189 In other words, the itertions from different nests re interleved. This is possible s we ssumed tht no dt dependence exists in the code. When there re dt dependences, however, the execution order of loop itertions is somewht restricted s discussed in the next section. Itertion Spce Itertion Tile Arry U 1 Arry U 0 (seed) Dt Tile Arry U 2 Dt Tile Dt Tile Fig. 5. Determining dt tiles of non-seed rrys using the itertion tile defined by the dt tile of the seed rry. 3.3 Hndling Dt Dependences As mentioned erlier, idelly, we would like to execute loop itertions s follows. We consider dt tiles from the rry one-by-one, nd for ech dt tile, we execute ll loop itertions (nd only those itertions) tht ccess rry elements in the tile. However, if the communiction volume of dt tile is not zero, this idel execution pttern would not hppen. This is becuse, in executing some itertions, we might need to ccess elements from other tiles s well. Obviously, if we re ble to select good dt tile (using the strtegy explined erlier), the number of such non-tile ccesses will be minimized. Note tht, even in loop without dt dependences, we cn experience non-tile ccesses. However, when dt dependences exist in the code, we cn expect tht such off-tile ccesses will be more s the itertions tht ccess the elements in the current tile might involve in dt dependence reltionships with other itertions. For the ske of presenttion, let us ssume tht there re two nests in the code (N 1 nd N 2 ) nd single rry (U 0 ). Assume tht DT U0 (1) is dt tile for rry U 0 nd let U0,N 1 (1) nd U0,N 2 (1) be the corresponding itertion tiles for nests N 1 nd N 2. Assume further tht U 0,N 1 (1) is the set of itertions (in N 1 ) other thn those in U0,N 1 (1). Note tht U0,N 1 (1) nd U 0,N 1 (1) re disjoint nd their union gives the itertion spce of nest N 1. We cn define similr U 0,N 2 (1) set for nest N 2. Consider now the itertion sets U0,N 1 (1), U 0,N 1 (1), U0,N 2 (1), nd U 0,N 2 (1) shown in Figure 6(). If there re no dt dependences in the code, when processing DT U0 (1), we

13 190 Mhmut Kndemir cn execute U0,N 1 (1) followed by U0,N 2 (1). Note tht this corresponds to the idel cse s these two itertion sets, nmely, U0,N 1 (1) nd U0,N 2 (1), ccess the sme dt tile, so executing them one fter nother (without n intervening itertion from U 0,N 1 (1) or U 0,N 2 (1)) represents the best possible scenrio. Note lso tht even if there re dt dependences between itertions in U0,N 1 (1) (nd/or between itertions in U0,N 2 (1)) but not cross itertions of different sets, we cn still execute U0,N 1 (1) followed by U0,N 2 (1), provided tht we execute itertions in U0,N 1 (1) (nd lso in U0,N 2 (1)) in their originl execution order. This execution order is lso vlid if there re dependences from U0,N 1 (1) (resp. U0,N 2 (1)) to U 0,N 1 (1) (resp. U 0,N 2 (1)) only. These cses re superimposed in Figure 6(b). Once ll the itertions in U0,N 1 (1) nd U0,N 2 (1) hve been executed, we cn proceed with DT U0 (2). The dshed rrow in Figure 6(b) represents the execution order of these sets. Suppose now tht there exists dependence from n itertion i U 0,N 1 (1) to n itertion i U0,N 1 (1) s shown in Figure 6(c). Assume further tht there exists dependence from n itertion i U 0,N 2 (1) to n itertion i U0,N 2 (1). In this cse, it is not possible to execute U0,N 1 (1) followed by U0,N 2 (1) s doing so would modify the originl semntics of the code (i.e., violte dt dependences). To hndle this cse, our pproch breks U 0,N 1 (1) into two groups, U 0,N 1 (1) nd U 0,N 1 (1b), such tht there is dependence from U 0,N 1 (1) to U0,N 1 (1), but not from U 0,N 1 (1b)to U0,N 1 (1). This sitution is depicted in Figure 6(d). Note tht in the degenerte cse one of U 0,N 1 (1) nd U 0,N 1 (1b) cn be empty. Similrly, we lso divide U 0,N 2 (1) into two groups: U 0,N 1 (1) nd U 0,N 1 (1b). Then, suitble order of execution (during processing DT U0 (1)) is U 0,N 1 (1), U 0,N 2 (1), U0,N 1 (1), U0,N 2 (1), which is lso illustrted in Figure 6(d). It should be noticed tht, in this scenrio, lthough we need to execute sets U 0,N 1 (1), U 0,N 2 (1)before U0,N 1 (1), U0,N 2 (1), we re still ble to execute U0,N 1 (1) nd U0,N 2 (1) one fter nother, which is good from the loclity viewpoint. Let us now consider the scenrio in Figure 6(e) tht indictes dt dependences from U0,N 1 (1) nd U 0,N 1 (1) to U0,N 2 (1) nd U 0,N 2 (1). To hndle this cse, we brek U 0,N 1 (1) into two subsets, U 0,N 1 (1)nd U 0,N 1 (1b), such tht there re no dependences from the set U 0,N 1 (1b) totheset U 0,N 2 (1). Then, the preferred execution order is shown Figure 6(f). 4 Appliction of Dt Spce Oriented Tiling There re severl pplictions of dt spce oriented tiling. One of these is improving cche loclity in rry-dominted pplictions. Since DST cptures dt ccesses in globl (procedure-wide) mnner, it hs better potentil for improving cche loclity compred to conventionl itertion spce oriented tiling. In this section, however, we focus on similr yet different ppliction re: using dt spce oriented tiling for exploiting n on-chip scrtch pd memory (SPM). Scrtch pd memories (SPMs) re lterntives to conventionl cche memories in embedded computing world [7,8]. These smll on-chip memories, like

14 Dt Spce Oriented Tiling 191 U, 0 N (1) 1 U, 0 N (1) 1 U, 0 N (1) 1 U, 0 N (1) () (b) U, 0 N (1) 2 (c) U, 0 N (1) 2 U, 0 N (1) 1 U, 0 N (1) 1 U, N (1b) U, N (1) 0 1 U, N (1b) 0 1 (d) U, N (1) U, N (1b) (e) (f) Fig. 6. Different itertion sets nd dependences between them. Note tht solid rrow denotes dt dependence, wheres dshed rrow denotes legl execution order. cches, provide fst nd low-power ccess to dt nd instructions; but, they differ from conventionl dt cches in tht their contents re mnged by softwre insted of hrdwre. Since the softwre is in full control of wht the contents of the SPM will be t given time, it is esy to predict memory ccess times in n SPM-bsed system, desired property for rel-time embedded systems. Since there is lrge difference between ccess ltencies nd energy consumptions of these memories, it is importnt to stisfy s mny dt requests s possible from SPM. Our compiler-bsed pproch to SPM mngement determines the contents of the SPM (t every progrm point) nd schedules ll dt movements between the SPM nd off-chip dt memory t compile-time. The ctul dt movements (between SPM nd off-chip dt memory), however, tke plce t run-time. In other words, we divide the tsk of exploiting the softwre-controlled SPM between compiler nd run-time (hrdwre). It should lso be mentioned tht in order to benefit from n SPM, the energy (nd performnce) gins obtined through optimized loclity should not be offset by the runtime overheds (e.g., explicit dt copies between SPM nd off-chip memory). Tht is, dt item (rry element) should be moved to the SPM only if it is likely tht it will be ccessed from the SPM lrge number of times (tht is, if it exhibits high dt reuse). Our pproch works s follows. It first optimizes the code using DST s explined bove. It then reds dt tiles from off-chip dt memory to SPM nd executes ll itertions tht mnipulte the dt in the SPM. When these itertions re finished, new set of dt tiles re brought into SPM nd the corresponding loop itertions re executed, nd so on. It should be noted tht if, during its sty in the SPM, the dt tile hs been modified (through write commnd) it should be written bck to the off-chip memory when it needs to be replced by nother dt tile. Figure 7 gives sketch of the SPM optimiztion lgorithm bsed on dt spce oriented tiling. To keep the presenttion cler, we ssume tht ll rrys re of the sme size nd dimensionlity, nd ll rrys re

15 192 Mhmut Kndemir ccessed in ech nest. After determining the seed rry nd the most suitble tile shpe from the viewpoint of communiction volume, the first loop in this figure itertes over dt tiles. In ech itertion, we red the corresponding dt tiles from the off-chip memory to the SPM. Then, the second loop nest determines the corresponding itertion tiles from ll nests nd lso computes the set of itertions (I ) tht should be executed before these itertions (due to dt dependences). The compiler then genertes code to execute these itertions nd updtes the itertion sets by eliminting the lredy executed itertions from further considertion. It should be noted tht this is highly-simplified presenttion. In generl, the itertions in I nd U0,N j (i) might be dependent on ech other. In executing the itertions in given set, we stick to the originl execution order (not to violte ny dependences). After tht, it checks whether the dt tiles hve been updted while they re in the SPM. If so, they need to be written bck to the min memory. This concludes n itertion of the outermost for-loop in Figure 7. INPUT: setofnestsn j,1 j P ccessing K rrys I j: the itertion set of the jth nest ALGORHM: determine the seed rry U 0 nd dt tile shpe; for ech dt tile i, 1 i Y generte code to red DT U0 (i),dt U1 (i),,dt UK (i) from min memory; I res = ; for ech nest N j,1 j P I rem = I k,1 k j determine U0,N j (i) nd U 0,N j (i); determine I I rem such tht: (i) there is dependence from I to U0,N j (i) (ii) there is no dependence from (I rem I ) to U0,N j (i) I res = I res I ; endfor; for ech nest N j,1 j P generte code to execute itertions in I (if they hve not been executed so fr); generte code to execute itertions in U0,N j (i) (if they hve not been executed so fr); endfor; updte I k,1 k j for ech U l,1 l K if DT Ul (i) is modified, then write it bck to min memory; endfor; endfor; Fig. 7. An SPM optimiztion lgorithm bsed on DST.

16 Dt Spce Oriented Tiling Conclusions This pper presents compiler-bsed strtegy for optimizing dt ccesses in regulr rry-dominted pplictions. Our pproch, clled dt spce oriented tiling, is vrint of clssicl itertion spce tiling. It improves over the ltter by working with better dt tile shpes nd by exploiting inter-nest dt reuse. This pper lso shows how dt spce oriented tiling cn be used to improve the effectiveness of scrtch pd memory. References 1. S. Colemn nd K. McKinley. Tile size selection using cche orgniztion nd dt lyout. In Proc. the ACM SIGPLAN Conference on Progrmming Lnguge Design nd Implementtion, June , 179, 180, F. Irigoin nd R. Triolet. Super-node prtitioning. In Proc. the 15th Annul ACM Symposium on Principles of Progrmming Lnguges, pges , Jnury M. Kndemir, J. Rmnujm, M. Irwin, N. Vijykrishnn, I. Kdyif, nd A. Prikh. Dynmic mngement of scrtch-pd memory spce. In Proc. the 38th Design Automtion Conference, Ls Vegs, NV, June I. Kodukul, N. Ahmed, nd K. Pingli. Dt-centric multi-level blocking. In Proc. the SIGPLAN Conference on Progrmming Lnguge Design nd Implementtion, June , 181, 181, W. Li. Compiling for NUMA Prllel Mchines. Ph.D. Disserttion, Computer Science Deprtment, Cornell University, Ithc, NY, M. O Boyle nd P. Knijnenburg. Non-singulr dt trnsformtions: Definition, vlidity, pplictions. In Proc. the 6th Workshop on Compilers for Prllel Computers, pges , P. R. Pnd, N. D. Dutt, nd A. Nicolu. Efficient utiliztion of scrtch-pdmemory in embedded processor pplictions. In Proc. the Europen Design nd Test Conference (ED&TC 97), Pris, Mrch L. Wng, W. Tembe, nd S. Pnde. Optimizing on-chip memory usge through loop restructuring for embedded processors. In Proc. the 9th Interntionl Conference on Compiler Construction, Mrch , pp , Berlin, Germny M. Wolfe. High Performnce Compilers for Prllel Computing, Addison-Wesley Publishing Compny, , J. Xue nd C.-H. Hung. Reuse-driven tiling for dt loclity. In Lnguges nd Compilers for Prllel Computing, Z.Lietl.,Eds.,LectureNotesinComputer Science, Volume 1366, Springer-Verlg, 1998.