Reconfigurable Supercomputing with Scalable Systolic Arrays and In-Stream Control for Wavefront Genomics Processing

Transcription

1 Reconfigurable Supercomputing with Scalable Systolic rrays and In-Stream ontrol for Wavefront enomics Processing SP 1 uesday, July 13, 21. Pascoe (speaker),. Lawande,. Lam,. eorge NSF enter for igh-performance Reconfigurable omputing (RE), University of Florida Y. Sun, W. Farmerie Interdisciplinary enter for Biotechnology Research (IBR), University of Florida M. erbordt epartment of Electrical and omputer Engineering, Boston University

2 Motivation Impending roadblock in health-sciences research urrent-generation sequencing technology capable of producing millions of sequences per day Next-generation sequencing instruments poised to increase production more than 1-fold onventional sequence-analysis processing has not kept pace with raw sequence production Objective: evelop sustainable solution for increasing performance of sequence-analysis applications pproach: evelop in-stream control approach for systolic array to maximize hardware acceleration apitalize on reconfigurable computing at scale to achieve sustainable supercomputing performance 2

3 Presentation Outline Background Sequential Score alculation Backtrack to Produce String Representation Sequence-lignment cceleration ommon ardware cceleration echniques Sequence-lignment cceleration Issues Scalable Systolic rrays with In-Stream ontrol emonstration of In-Stream ontrol: ase Studies Needleman-Wunsch (NW) Smith-Waterman (SW) Needle-istance (N) Novo- System rchitecture iel PROStar-III Boards NW/SW/N Performance Evaluation on Novo- onclusions Q& 3

4 Background: Sequence-lignment Example Sequence S 1 : Sequence S 2 : Want to quantify similarity between S 1 and S 2 Use NW or SW or other algorithms to align sequences S( i, j) max S( i 1, j S( i ( i 1) Sub( i, 1, j) eog 1, j) e j) ( i, ( i, j) j) max max S( i ( i S( i, ( i, j j 1, j) 1, j) 1) 1) og eg og eg S( i, j ( i, j 1) 1) eog e S(, i) (, i) S( i,) ( i,) og og i eg for 1 i imax, 1 j jmax Sub( i, j) penalty matrix Needleman-Wunsch Equations 4

5 5 Sequential Score alculation Score alculation --- match = 5, mismatch =, ogap =, egap = S1 S2

6 6 Backtrack to Produce String Representation _ S1 S2

7 7 Sequence-lignment cceleration S1 S2 Score alculation --- match = 5, mismatch =, ogap =, egap = -.5

8 ommon ardware cceleration echniques onstruct systolic array of processing elements (PEs) to exploit parallelism Each PE responsible for calculating single column of scoring matrix O(X Y) time complexity in software reduced to O(X+Y) in hardware S(i max,) S(i-1,j-1) + Sub(i,j) S(i-1,j) + eog Max (i-1,j) + e S(,j max ) omputation of ell Score S(i max,j max ) } } S(i,j-1) + eog Exploit Parallelism along nti-diagonal (i,j-1) + e } 8

9 Sequence-lignment cceleration Issues Maximum possible performance directly proportional to: Number of PEs Operating clock frequency (e.g. 1 PEs x 1Mz = 1 UPS) iga ell Updates Per Second chieved performance can be much less due to overhead: Input data manipulation Input data transfer to FP FP initialization and setup atapath initialization PE configuration esign goals: Eliminate or overlap overhead with useful work Pipeline latency andshaking control between FP/PU Output data transfer to PU Output data manipulation Others Minimizing PE area to maximize the number of PEs per FP 9

10 Scalable Systolic rrays with In-Stream ontrol Worst-case Performance Low-area Overhead N (setup time + pipeline latency + PU/FP handshake + PE configuration time + time to record results + time to process streamed sequence) Best-case Performance igh-area Overhead pipeline latency + N (time to process streamed sequence) With In-Stream ontrol Near-best-case Performance Low-area Overhead pipeline latency + M (PE configuration + 2 clock cycles) + N (time to process streamed sequence + 2 clock cycles) Replace complex state machines and other supporting logic with control data inserted directly into data stream Signal state transitions rigger complex actions on application data Limit wasted cycles Reduce PE area Increase hardware utilization ssuming N successive alignments and M PE reconfigurations omplex-controller performance with simple-controller overhead 1

11 In-Stream ontrol: Needleman-Wunsch Needleman-Wunsch (NW) used for global sequence alignment onventional systolic-array architecture With simple PEs and simple control Input sequences consist of standard nucleotide alphabet (,,,, N) 3-bit encoding: dd 3 additional control characters (total of 8 characters): (L) load query, (R) reset to initial conditions, (P) push results Special 2-character sequence PN used to signal end of stream 11

12 In-Stream ontrol for NW: Simple Example Example set of short sequences: {,, N, N} ssume it is desired to perform NW between each unique sequence pair in set: (i.e. 1-2, 1-3, 1, 2-3, 2, and 3) Stream encoding LRPRNPRNPLR NPRNPLNRNPN fter (5 + pipeline length) FP clock cycles, the six results are ready for retrieval when PU has free cycles 12

13 In-Stream ontrol for NW: Summary Introduces notion of local control ontrol characters used as framing characters for each loaded or streamed sequence Simultaneously: PEs in pipeline between an L and R can be reconfiguring Earlier group of PEs between an R and P can be processing a query Even earlier group is doing something else Simplifies control: Meaningful outputs result from correct sequence of characters passing through systolic array No need for control signals from controller and PU Benefit: ost? Limits PE idle periods Reduces configuration overhead between runs Reduces PE area Only two additional clock cycles of overhead per sequence to process framing characters 13

14 In-Stream ontrol: Smith-Waterman From Prev. FP o FIFO Input Buffer Input FIFO Interface PE(1) PE(2) PE(N) Output Buffer Output FIFO Interface FIFO o Next FP SRM (atabase) Optimized for Short Sequences vs. Large atabase Smith-Waterman (SW) used for local sequence alignment Use NW control structure as basis then modify to add functionalities Switching between query and database stream Requires overloaded L character to toggle between streams Loading multiple queries against a single streamed database Introduce concept of push window and add extra bit to PU stream encoding for controlling query start and query extend modes Providing traceback (string representation of alignment) No changes to encoding scheme, modification to supporting software control Extending queries across multiple FPs No changes to encoding scheme, modification to supporting software control 14

15 In-Stream ontrol for SW: Simple Example Example set of short sequences: {,, N, N} ssume it is desired to perform SW between each sequence and a large database atabase Stream (3bit) Encoding: R PPPPL Query Stream (4bit) Encoding: (L)(1)()()()(1)()()(1)()(N) ()(1)(N)()()(R)(L)(P)(N) fter (2 + database length + pipeline length) FP clock cycles, the 4 results are ready for retrieval when PU has free cycles From Prev. FP o FIFO Input Buffer Input FIFO Interface SRM (atabase) PE(1) PE(2) PE(N) Output Buffer Output FIFO Interface FIFO o Next FP 15

16 In-Stream ontrol: Needle-istance Needle-istance (N) used for calculating optimal pairwise distance Original Software: 1. alculate NW score 2. enerate string representation 3. alculate custom normalized edit distance ardware: Score and distance calculations performed in parallel Eliminate computation of string representation Edit distance: = 2 otal distance: = 1 Normalized edit distance: 2/1 =.2 N PEs implemented by adding distance calculation modules to PEs of NW 16

17 Novo- System rchitecture 24+1 Linux servers in cluster 24 compute servers (2 boards/server) 1 head-node server for management 2 b/s non-blocking InfiniBand 1 b/s Ethernet 26 (24+2) quad-core Xeons Max. system power of ~8 KW 48 quad-fp boards iel PIe x8 PROStar-III Embedded-style boards, for both PE- & P-oriented research 4¼ B R2 attached to each FP ~ 1 B total RM in Novo- 192 Stratix-III E26 FPs Each ltera FP with x18 multipliers, 254K logic elements, 24K registers, & max. power ~18W Novo is Latin, "to make anew, refresh, revive, change, alter," essence of R; is for enesis or reen. 17

18 iel PROStar-III Board (one of 48) J for Signalap debug 2 2B = 4B R2 RM per FP 12.8 mm PIe x8 interface (4B/s) iel PROStarIII Board ypical frequencies 1-325Mz M channels 32 R2 module slots 8 NOE: iel developed multiboard interconnect; stream apps 312 across mm multiple PS-III boards within servers ltera Stratix-III E26 FP 254,4 Logic Elements 768 multipliers (18 18) 14,688 Kbits of embedded memory 5% less power than Stratix-II 65nm technology 18

19 Speedup verage Sequence Length (Nucleotides) Needleman-Wunsch Performance Evaluation on Novo- NW PEs/FP Baseline: , length 85 Sequence omparisons Software Runtime: 11,26 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 47, ,14 3, , , (est.) ,13 Scaling performance with varying number of FPs under optimal input conditions K 4K 16K 256K 1M 4M Number of Sequence omparisons Performance of single FP under varying input conditions 16M 32M NW UPS chieved/max 17,24/2,4 = 85% 19

20 Speedup atabase Length (Nucleotides) Speedup vg. Sequence Length (Nucleotides) SW/N Performance Evaluation on Novo- SW 65 PEs/FP 45 PEs/FP N verage Sequence Length (Nucleotides) 1K 64M 4M 256K 16K K 4K 16K 64K 256K 1M 4M 16M Number of Sequence omparisons Baseline: uman X hromosome v 192, length 65 Seqs Software Runtime: 5,481 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 23, ,966 3, , , (est.) ,366 SW UPS chieved/max 15,158/15,6 = 97% 2 Baseline: , length 45 istance alculations Software Runtime: 11,673 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 13,522 3,18 4 3,429 12, , , (est.) ,751 N UPS chieved/max 8,471/1,8 = 78%

21 onclusions eveloped in-stream control approach for systolic array to maximize hardware acceleration apitalized on reconfigurable computing at scale to achieve sustainable supercomputing performance emonstrated performance and sustainability results on Novo-: Baseline: , length 85 Sequence omparisons Software Runtime: 11,26 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 47, ,14 3, , , (est.) ,13 Baseline: uman X hromosome v 192, length 65 Seqs Software Runtime: 5,481 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 23, ,966 3, , , (est.) ,366 Baseline: , length 45 istance alculations Software Runtime: 11,673 PU hours on 2.4z Opteron # FPs Runtime (sec) Speedup 1 13,522 3,18 4 3,429 12, , , (est.) ,751 Results on Novo- for NW (left), SW (enter), and N (Right). Each table shows scaling performance with varying number of FPs under optimal input conditions. Estimated performance on Novo- comparable or better than biggest supercomputers on ORNL: 224,162 cores 2.4 z exacore Opterons; 6.95 MW Novo- Power LNL: 122,4 cores 3.2 z ells + 1.8z Opterons; 2.35 MW } 8 KW Max. 21

22 hanks for listening! Questions Questions? 22

23 References [1] W. Pearson and W. Miller, "ynamic programming algorithms for biological sequence comparison, Methods Enzymol. 21: 5751, [2] Y. Sun, Y. ai, L. Liu, F. Yu, M.L. Farrell, W. McKendree, W. Farmerie, ESPRI: Estimating Species Richness Using Large ollections of 16S rrn Pyrosequences, Nucleic cids Research, vol. 37, no. 1 e76, May 29. PMI: PM [3] S. Lloyd and Q. Snell, ardware ccelerated Sequence lignment with raceback, Int. Journal of R, vol. 29, rticle I [4] O. Storaasli, W. Yu,. Strenski, J. Maltby, Performance Evaluation of FP-Based Biological pplications, Proc. ray Users roup, Seattle, W, May 27. [5] K. Benkrid, Y. Liu, and. Benkrid, ighly Parameterized and Efficient FP-Based Skeleton for Pairwise Biological Sequence lignment, IEEE rans. LSI Systems, vol. 17, pp , 29. [6] Novo- architecture overview, [7] ESPRI download, plaza.ufl.edu/sunyijun/espri.htm 23