SW-G using new DryadLINQ(Argentia)

Transcription

1 SW-G using new DryadLINQ(Argentia) DRYADLINQ: Dryad is a high-perfrmance, general-purpse distributed cmputing engine that is designed t manage executin f large-scale applicatins n varius cluster technlgies, including Windws HPC Server 2008.DryadLINQ is based n LINQ, which was intrduced with Micrsft.NET Framewrk versin 3.5. The cre f DryadLINQ is a DryadLINQ prvider, which translates the applicatin s LINQ queries int a Dryad jb and runs the jb as a distributed applicatin n a Windws HPC cluster. A develper thus des nt need t knw much abut Dryad r even abut parallel r distributed cmputing t write a DryadLINQ applicatin. Hwever, develpers wh are familiar with these technlgies can take advantage f their knwledge t ptimize perfrmance. A DryadLINQ evaluatin query runs as fllws: An applicatin creates ne r mre DistributedData<TSurce> bjects t represent persistent data, and then it defines a query by applying ne r mre LINQ peratrs. DryadLINQ builds a DistributedQuery<TResult> t represent the query, but defers evaluatin. The applicatin triggers the bject evaluatin prcess. a One way t trigger evaluatin is by calling freach t enumerate a DistributedQuery<T> bject. If yu step thrugh a DryadLINQ applicatin in the debugger, yu will see that the freach call itself is the trigger. The freach peratr calls the bject s GetEnumeratr methd, which initiates the evaluatin. The actual enumeratin desn t take place until evaluatin is cmplete. Evaluatin can als be triggered by ther DryadLINQ methds, such as Execute() and ExecuteAsync(). LINQ passes the LINQ expressin t the DryadLINQ prvider. The DryadLINQ prvider: Generates an executin plan The executin plan is used by the Dryad Jb Manager t create a graph fr the jb. Because LINQ applicatins perate n data sets rather than individual items, the DryadLINQ prvider has cnsiderable flexibility in hw it translates the LINQ query int an efficient Dryad executin plan. Generates data prcessing cde fr the vertices. The data prcessing cde fr each stage is cmpiled t a.net assembly and then dispatched t the cluster s cmputers at the apprpriate stage f the peratin. Cllates any side-infrmatin that is necessary fr the cmputatin, particular lcal variables that have been referenced in the query via clsure. A Dryad jb is started n the cluster, including bth a Dryad Graph Manager and Dryad Vertex Hsts. The Dryad Graph Manager executes the Dryad vertices frm the Dryad graph n the cluster: The graph manager schedules vertices as Dryad Vertex Hsts are available. Each vertex runs.net cde t prcesses a data partitin and prepare the results fr subsequent stages r final utput. The input t a vertex can be either static data r data frm the preceding stage. The graph manager may apply runtime plicies and ptimizatins t the executin graph t ptimize perfrmance. When executin cmpletes, the results are written t ne r mre distributed data sets.

2 hw a Dryad jb runs n a cluster. Dryad Linq Applicatin with Partitined Data:

3 Abut SWG: In SWG we evaluate the perfrmance rmance f Smith Waterman Gth Dissimilarity Cmputatin(SW-G) using the new DryadLinq(Argentia). Upper Triangle N*N matrix brken t D*D blcks DryadLINQ MPI Calculate pairwise distances fr a cllectin f genes (used fr clustering, MDS) Fine grained tasks in MPI Carse grained tasks in DryadLINQ Perfrmed n 768 cres (Tempest Cluster)

4 Blcks in the upper triangle Each D cnsecutive blcks are merged t frm a set f rw blcks ; each with NxD elements prcess has wrklad f NxD element. We need t determine hw Dryad distributes this data acrss the cmpute ndes. Fr this purpse we recrd every blck inf dwn t the lg file, then analyze it t see hw blcks are distributed. Therefre we can knw hw Dryad distributed general data after reasning and analysis. We use the fllwing DryadLinq peratrs t distribute data acrss the ndes: AsDistributed() and AsDistributedFrmPartitins().The difference between the tw is that is AsDistributed() always divide data int default pieces, and AsDistributedFrmPartitins() can divide data as yu like.we als use a new peratr called ApplyPerPartitin() basically tells Dryad hw t deal with each individual piece f data.

5 Scalibility f Pairwise Distance Calculatin: Time per Actual Calculatin (ms) % 50% 40% 30% 20% 10% 0% -10% Perfrmance Degradatin n VM (Hadp) N. f Sequences DryadLinq SWG n WinHPC Prblems: 1) It's nt easy t debug in Argentia, sme exceptin infrmatin is nt f slid value. (Argentia SW- wrk such as G Debugging) 2) In cases where a query is perfrming a significant amunt f nn-cmpute-bund File I/O, it might be beneficial t specify a degree f parallelism greater than the number f cres n the machine