When Shorter Isn t Better

Transcription

1 DesignCon 2010 When Shorter Isn t Better Mike Steinberger, SiSoft msteinb@sisoft.com Paul Wildes, Sack Brothers Enterprises, LLC Paul@SackBrothers.com Mike Higgins, Cray Inc. mfh@cray.com Eric Brock, SiSoft ebrock@sisoft.com Walter Katz, SiSoft wkatz@sisoft.com

2 Abstract In system level measurements, the paths with the worst bit error rate were not the longest paths, but rather some of the paths that had a short length of transmission line between two connectors. Even within this class of paths, however, there was no easy way to predict which paths would be bad. Subsequent post-route analysis demonstrated that the problem was multiple reflections between the connectors, but that the problem was detected only when their timing was exactly wrong. This paper presents the analysis methods and results, and recommends unequalizable energy as a reliable metric for identifying potential problem paths. Author(s) Biography Dr. Michael Steinberger is currently responsible for leading the development of SiSoft's serial link analysis products. He has over 30 years experience in the design and analysis of very high speed electronic circuits. Prior to joining SiSoft, Dr. Steinberger worked at Cray Inc., where he designed very high density interconnects and increased the data rate and path lengths to the state of the art. Mike holds a B.S. from the California Institute of Technology and a Ph.D. from the University of Southern California, and has been awarded 13 U.S. patents. Paul Wildes has spent over 20 years building custom hardware platforms and software tools for the design, debug, and characterization of high-performance digital systems. Mr. Wildes received the BS and MS degrees in Electrical Engineering from the University of Wisconsin, and has been awarded 2 U.S. patents. He is the founder of Sack Brothers Enterprises, LLC. Previously he designed processor boards at Cray Inc., developed semicustom ICs at Bell Labs, and did Unix kernel development at Astronautics Corporation in Madison WI. Mike Higgins is a Principle Engineer at Cray Inc. He has 20 years experience designing, analyzing, and validating very dense, high-speed communication channels and memory subsystems for supercomputers while working for Cray Research, Inc., Silicon Graphics, Inc., and Cray Inc. Mike holds a B.S.E.E. from the University of Minnesota. Eric Brock is a Principal Member of Technical Staff at Signal Integrity Software Inc. (SiSoft). He received his BS in Electrical Engineering and a BS in Physics from Portland State University in After graduating, Eric joined Compaq Computer as a Signal Integrity Engineer in the Alpha Development Group, where he was a member of a system-level signal integrity group involved in interconnect analysis on Compaq's latest Alpha processor systems. Before leaving Compaq, he led a group of signal integrity engineers in the creation of design rules, simulations, waveform quality, timing analysis, and design verification of Compaq's next generation Alpha processor systems. Eric joined SiSoft in 2002 and has worked as a consultant on the design and verification of dozens of high-speed parallel and serial interfaces, both proprietary and industry standard. More recently, he has been involved in the design of multi-ghz serial links and SERDES modeling. When Shorter Isn t Better SiSoft, 2010 Page 2

3 Dr. Walter Katz, Chief Scientist for SiSoft, is a pioneer in the development of constraint driven printed circuit board routers. He developed SciCards, the first commercially successful auto-router. Dr. Katz founded Layout Concepts and sold routers through Cadence, Zuken, Daisix, Intergraph and Accel. More than 20,000 copies of his tools have been used worldwide. Dr. Katz developed the first signal integrity tools for a 17 MHz 32-bit minicomputer in the seventies. In 1991, IBM used his software to design a 1 GHz computer. Dr. Katz holds a PhD from the University of Rochester and a BS from Polytechnic Institute of Brooklyn. When Shorter Isn t Better SiSoft, 2010 Page 3

4 Background The analysis technique described in this paper was developed in response to a problem discovered during the development of a new large computing system. The system contained a very large number of high-performance SerDes channels, the vast majority of which functioned at or above the designed performance targets. However, a very small subset of channels, appearing to be no different from their neighbors, were found to experience very high error rates. It was found that traditional techniques for analysis and simulation, and conventional wisdom or rules-of-thumb, were not adequate to predict the phenomena observed. In the process of isolating the cause of the failures, and finding means to simulate the failing conditions, a new metric was developed to assess the quality of a transmission path, and to predict the likelihood of experiencing this class of failures. The metric has been termed unequalizable energy. System Description In the Spring of 2006, engineers at Cray Inc. began functional testing of a new custom SerDes circuit design for chip-to-chip communication. This circuit would eventually form the basis for all processor-to-memory and high-speed network communications for a new supercomputer system [1], [2]. The basic building block of the system is an 8- processor compute blade, with 32 memory DIMMs and a midplane connector carrying network connections. The floor plan for this blade is shown in Figure 1. Figure 1: Compute Blade Dual 4-way SMP Nodes, 16 memory DIMMs each. When Shorter Isn t Better SiSoft, 2010 Page 4

5 The image of a single PCB layer (Figure 2) shows the transmitters from a single processor in each node. There are connections to each memory DIMM in the corresponding SMP node, along with network connections to the midplane. Figure 2: Communication paths from one processor in each SMP node: Memory and Network. Eight compute blades (referred to as BW) are engaged in a chassis in a vertical orientation, mating to network connectors in a midplane (referred to as RA). On the other side of the midplane are four horizontally-oriented network router boards (referred to as R1), which provide store-and-forward network connectivity from every endpoint to every endpoint. Cables out the back end of the network router modules connect to other processor endpoints via the high-speed network fabric. Each full compute cabinet contains two of the chassis just described, stacked vertically. See the Figure 3 below. When Shorter Isn t Better SiSoft, 2010 Page 5

6 Figure 3: Compute Chassis Eight blades, 4 network router modules, cables. Transmission Path Topologies The system architecture described above utilizes SerDes communication circuits in three distinct topologies: 1) processor to memory; 2) processor to network; and 3) network (chassis-to-chassis). For reasons that will become obvious later, the subject of this paper is the processor to network channels, as illustrated in Figure 4. Figure 4: Topology #2: Processor to Network The SerDes design was programmable to run at 6.25 Gb/s, 5.0 Gb/s, or Gb/s. For all the testing and operation described here, channels were run at a full-rate bit rate of 6.25Gbps. When Shorter Isn t Better SiSoft, 2010 Page 6

7 The SerDes equalization consists of transmit de-emphasis and receive peaking filters. The transmit equalizer is a four tap synchronously spaced de-emphasis network with one precursor tap and two post-cursor taps. The receive equalizer has two identical stages of peaking filter than can be controlled independently. System Analysis and Test Pre-Design Analysis Extensive signal integrity (SI) analysis was done prior to committing system designs to manufacturing. The SI analysis was oriented primarily toward path loss, focused on minimizing trace loss and reducing impedance discontinuities. The primary tool for predesign analysis of the channels was the Cray-developed tool LASSIE (Linear Analysis of Supercomputer Systems Interconnect Experiments). This tool performs time domain simulations with semi-analytic bit error rate analysis. The transmission path is modeled as the concatenation of transmission lines and/or lumped elements, using transmission line models that include the effects of internal impedance, conductor roughness, and dielectric losses. Transmit equalization is modeled as a synchronously spaced or fractionally spaced tapped delay line, and the receive equalization is modeled as a rational transfer function. LASSIE allowed rapid simulation of transmission paths, plotting of eye diagrams, multiple 'what if' scenarios, and calculation of BER and timing margin, given a particular input data pattern. LASSIE also has a facility for finding the optimum tap coefficients for synchronous- or fractional-spaced tapped delay line pre-emphasis filters. Initial Hardware Testing Initial printed circuit board articles were tested for compliance to all electrical specifications, and were substantially in agreement. Some minor design changes were initiated, particularly in the area of the connectors, to minimize impedance discontinuities. Of particular interest in the transmission path integrity was the midplane connector. Crosstalk in particular was evaluated, first as a stand-alone component and then in the complete transmission path. When first silicon was received, a basic check of SerDes transmitter functionality was performed (data transmission, logic encoding, etc). In the detailed characterization work that was subsequently done, two areas were focused on: the output voltage swing, and the wave shaping due to the tapped delay-line pre-emphasis filter. All variations were tested, a histogram built, and settings were iteratively refined until satisfactory. When Shorter Isn t Better SiSoft, 2010 Page 7

8 The SerDes receiver was tested on a Network Router module, using direct programming of the SerDes macros. A single module was used, with connections being made from one router chip to the other, using loopback through the cable connectors. To test the receive sensitivity, two setups were used: ideal attenuators and long twinax cables. The ideal attenuators were a good test of the sensitivity since it removed the requirement of equalization from the receiver. Over 20dB of ideal SMA attenuators were inserted in the connector path and the channel could still run error free. The voltage levels were in the 50mV to 100mV range. That is around where the assumptions were made for the original LASSIE simulations. The long cables apply more towards the equalization testing than sensitivity but it was verified that the RX equalizers maintain very good sensitivity on the frequency dependent loss paths. The largest cable used was a 24AWG 12m cable. A 10m 26AWG cable was also run with success. The receive equalizer was analyzed with respect to signal to noise ratio (SNR), as well as empirically measured in the lab. The SNR was evaluated by calculating the link SNR due to crosstalk and noise, combined with the frequency response family of curves for the receive equalizer. This work was then applied to the SerDes RX macro, in a test environment using board, connector, and cable elements from the full compute system. The receive equalization was tested three ways: both equalization stages fixed, one fixed and one adaptive, and both adaptive. The greatest performance (from a loss perspective) was when the first stage was adaptive and the second stage was fixed at the minimum equalization setting (maximum baseband gain). Two basic recommendations came from the characterization work. First, use only enough TX pre-emphasis to account for roughly half of the path loss. This will balance the signal-to-noise ratio with the optimum equalization. For short links, RX equalization may be the best solution since the TX settings seem to be dependent on path discontinuities: this was also seen in LASSIE simulations. Second, the lossiest cable paths will likely have the best SNR due to the bandwidth effects. In other words, be careful of noise in the short paths or paths with NEXT (nearend crosstalk). All the early testing confirmed what had already been observed in pre-production test chips, namely that the new SerDes design was fully functional, and all signs indicated that performance in an actual system should meet or exceed all the design targets. A Problem Appears Eventually, all the pieces were in place for complete system validation. At this point, the testing focus shifted to empirical testing of channels under system load, at a full cabinet level. In this system, a cabinet contains two chassis, each with eight processor blades and When Shorter Isn t Better SiSoft, 2010 Page 8

9 four network router boards. Fully populated, a cabinet has 22,528 SerDes lanes, all operating continuously. The link control block used in all instances of the SerDes channels contains self-test features, along with mission mode operation that monitors for errors, and initiates automatic re-try on error. Instances of packet re-try are logged in error counters, which can be monitored by system diagnostic software during normal system operation, without interfering with the running software applications. This allowed the testing team to gather large amounts of data on very many continuously running channels over a long period of time. During the early part of this system operational testing, a transmission problem began to be evident. A small subset of transmission paths were reporting an elevated level of errors. At this point, it was not clear what class of channels were problematic, or the exact level of bit errors. The evident distribution of errors at that time, after initial measurements, is shown in green in Figure 5. The uniform target per lane of BER is shown in blue for reference. After identifying this problem, several months of focused testing ensued, utilizing the largest hardware platforms available, including one system of over 60,000 SerDes channels, operating under stress load testing for several hundred hours. A much more detailed picture emerged, shown in red in Figure 5. Figure 5: Channel performance distribution When Shorter Isn t Better SiSoft, 2010 Page 9

10 Notice from Figure 5 that all the higher BER channels were midplane channels; that is, topology #2 defined above. The main body of channels in this class had around BER; about 10% were significantly worse, around A very small number, less than 0.1%, were another order of magnitude worse yet. What made the problem interesting was the fact that there were no fundamental path differences between the good, the bad, and the ugly. They all shared The same tx/rx circuit; Sometimes the exact same chip, on both transmit and receive sides; Same number of vias; Similar trace lengths on all three path segments; Same basic connector pinout, hence similar crosstalk characteristics. Another significant point was that these were not the longest paths of this topology. After a significant amount of work to eliminate manufacturing defects or chip variation as a factor, it was determined that under a very specific set of parameters short midplane trace length, short router module trace length, combination of two or more fulllength via or minimum length vias. a midplane channel was a potential candidate for the bad class. However, the majority of channels with this profile did not fail at a higher level; it was a necessary but not sufficient condition for failure. Post-Layout Analysis A post-layout analysis was performed, with the goal of identifying the root cause of the channel failures. To assure objectivity and therefore increase the probability of detecting what seemed to be a previously unidentified failure mechanism, the post-layout analysis was performed by an independent organization without any a priori identification of the failing paths. In short, the post-layout analysis was performed without any hints as to what the right answer was. Since there were no predefined hypotheses to be tested, there was also no way to decide a-priori which details were important and which were not. The analysis was therefore performed at the most detailed level practical. Circuit topologies and transmission line dimensions were extracted from the layout databases for the PC boards. The SerDes transmitters and receivers were modeled using SiSoft s generic AMI models [3] configured to represent equalization data provided by the IC supplier. Via dimensions were extracted, and via S parameters were estimated using both a 3D field solver and closed form equations [4]. Connector S parameters were obtained from the connector manufacturer. Package S parameters were estimated using closed form equations; however, the packages were known to have a well matched characteristic impedance. The circuit topology is shown in Figure 6. When Shorter Isn t Better SiSoft, 2010 Page 10

11 Figure 6: Circuit topology used for resonance study Bit error rate estimates were performed using SiSoft s Quantum Channel Designer (QCD). Since there were so many bit error rates to estimate, most of the estimates were obtained by running QCD in statistical analysis mode. Fixed transmit tap settings were used along with a fixed receive equalizer configuration. This removed the possibility of variation due to changes in equalizer configuration. In the results below, it will be seen that large changes in bit error rate occurred for small changes in trace length. Thus, the equalizer configuration was close to optimum for the several cases that are compared. IBIS AMI Models A four tap IBIS-AMI model was developed to model the transmitter. This model, consisting of one pre-cursor tap, a main tap, and two post cursor taps, has exactly the same structure as the SerDes transmitter. Tap weights were neither limited to a specific range nor quantized to specific values available in the transmitter s configuration space; however, optimized values did fall within the range of tap values supported by the transmitter. An IBIS-AMI model for the receiver was developed that included a peaking filter with 32 configuration settings and auto-optimization. The peaking filter poles and zeros were determined by manually varying the pole and zero locations to match SPICE gain curves supplied by the IC supplier. The first 16 configurations were for a single stage of the peaking filter enabled, while the remaining 16 configurations were for both peaking filter stages enabled. The auto-optimization algorithm iterated through all available configurations looking for the best eye shape. To a first order, this criterion matches the control algorithm used in the receiver. All optimal configurations were in the first 16 configurations, matching receiver threshold measurements made at Cray. Connector Models An s48p connector model supplied by Amphenol was loaded into QCD to generate s12p (3 coupled differential pairs) and s4p (single differential pair) models for each of the connector paths: AB (shortest), DE, GH, and JK (longest). These models were used for all simulations. When Shorter Isn t Better SiSoft, 2010 Page 11

12 Via Models Two different via geometries were used in the system: one geometry for the BW and R1 boards and a different geometry for the RA board. These geometries are shown in Figures 7 and 8. Figure 7: BW and R1 via geometry Figure 8: RA via geometry When Shorter Isn t Better SiSoft, 2010 Page 12

13 Since these two via geometries are for opposite ends of the same connector, they re necessarily very similar. The major difference between the two geometries is that the antipads in Figure 7 (BW and R1 boards) were made as large as possible, consistent with providing minimal shielding between traces in different layers, whereas the antipads in Figure 8 (RA board) were considerably smaller. Thus, as demonstrated in [4], the characteristic impedance for the BW/R1 boards was significantly higher than that for the RA board. The characteristic impedance for both vias is substantially below that of the rest of the transmission path. All vias used on were back-drilled from the bottom of the board. In addition, on the RA board the vias were back-drilled from the top of the board as well. This back-drilling was included in all via models. 3D via models were developed by Physware using their PhysWave field-solver. In addition, distributed lumped models were generated by QCD for every via on all three boards. These were uncoupled 4-port models that correlated fairly well to the 3D Physware models. The lumped models were used to perform the analysis. Package Models No package models were available for the ICs at either end of the channel, so simple, uncoupled, 50ohm W-element models were used. Different lengths were used for each model based on which package row the signal s ball was on. The following length ranges were used: BW ASIC: 7.5, 10, 12.5, and 15mm R1 ASIC: 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, and 28mm Etch Models Based on the stackups and the physical dimensions extracted from the Allegro board files, lossy w-element models were created for each of the three boards. Since each of the boards use a symmetrical stackup, only a single model was required for each board. The models described in Table 1 were used for all simulations. Trace Width (mils) Trace Thickness (mils) Differential Separation (mils) Conductor Roughness µrms Differential Impedance Loss Board Er Tangent BW RA R Table 1: PC board trace models Receiver Sensitivity Receiver sensitivity, also called minimum latch overdrive or receiver threshold, was not derived from the Cray data, but rather was used as a variable to shift the bit error rate for the entire population. Values used ranged from 120mV to 150mV. When Shorter Isn t Better SiSoft, 2010 Page 13

14 Results As described above, the bit error rate was estimated for all of the high speed serial channels going either direction between the BW and R1 boards. Although there was some general correlation between total length and bit error rate, there is also considerable spread, indicating that there are other factors which are important. Figure 9 shows the scatter plot of predicted bit error rate vs. routing length on the RA board. Figure 9: Predicted bit error rate vs. RA routing length Figure 9 shows some unexpected behavior. In Figure 9, the lengths 1.5, 2.5, 3.4, 4.0, 4.5 and 5.0 seem to be particularly prone to high bit error rate, with 1.5 apparently worse than the others. The case circled in red was therefore chosen for further study. The routing details are given in Table 2. PC Board Routing Length Routing Layer Connector Row BW out of 10 JK (longest) R out of 8 JK (longest) RA out of 6 Table 2: Routing variation study parameters Note that this path fits the pattern that was observed in system testing: short routing length on RA, long via on one side of the trace, short via on the other side. A series of bit error rate estimates were calculated by varying each of these three routing lengths by plus or minus 0.3. A scatter plot of the results is shown in Figure 10. When Shorter Isn t Better SiSoft, 2010 Page 14

15 Figure 10: Routing length variation study In Figure 10, there is no correlation with routing length on the BW or R1 board, but very good correlation with routing length on the RA board. We therefore concluded that the phenomenon in question truly was occurring on the RA board. The bit error rates for the three RA routing lengths were Backplane length = BER = 7.9e-22 Backplane length = BER = 7.0e-15 Backplane length = BER = 3.9e-8 Figures 11 through 13 are the eye diagrams for the three RA routing lengths. They further confirm that the variation with RA routing length is real. When Shorter Isn t Better SiSoft, 2010 Page 15

16 Figure 11: Case 1: RA Length = Figure 12: Case 2: RA Length = When Shorter Isn t Better SiSoft, 2010 Page 16

17 Figure 13: Case 3: RA Length = The RA routing length was then swept over a larger range. The results are shown in Figure 14. Figure 14: Predicted bit error rate vs. RA routing length In Figure 14, the earlier test cases are circled in red. Note in Figure 14 that there is a wide variation in predicted bit error rate, and although that variation appears to have a generally periodic nature, it also appears to be a complicated function of routing length. When Shorter Isn t Better SiSoft, 2010 Page 17

18 Analysis Some insight into this phenomenon can be gained by looking at the pulse response for the three RA routing lengths, as shown in Figure 15. To make it easier to compare pulse responses, the pulses for the two longer RA routing lengths have been shifted forward in time to line up with the pulse response for the shortest routing length. Figure 15 also has vertical lines indicating the edges of bit times. Figure 15: Pulse response vs. RA routing length As can be seen in Figure 15, the main pulse shape is the same for all three routing lengths, but the pulse responses start to differ markedly after two bit times from the main pulse. The pulse amplitudes in this time region are significant, and the equalization can have only limited effect. What appears to be happening is that for some RA routing lengths, the pulse response ripples seem to occur mostly at the edge of the data eye whereas for other routing lengths, the pulse response ripples seem to have significant amplitude in the center of the eye. Thus, while one pulse response may produce a very open eye diagram, another one with slightly different timing could produce a much more closed eye diagram. This characteristic is exactly what was observed in the system level measurements reported above. Since intersymbol interference is the combined effect of all of the pulse response ripples, there will typically not be any one bit time one can look at to determine whether the phasing of the pulse response ripples is fortunate or unfortunate. Rather, the combined effect of the entire pulse response must be considered as a single measurement. When Shorter Isn t Better SiSoft, 2010 Page 18

19 Unequalizable Energy If one assumes that the equalization solution can only control the pulse response from one bit time before the main pulse (pre-cursor tap) to two bits times after the main pulse (two post-cursor taps), then the pulse response energy outside of this time window could be referred to as unequalizable energy. Furthermore, note that this energy should be normalized to the total energy in the pulse response so as to create a direct measure of intersymbol interference. With this motivation, and given pulse response p(t), bit time τ and maximum amplitude time t0, we therefore define the unequalizable energy as Exactly this concept, developed independently, is also mentioned in [5]. Note that this metric is very different from the concept of unequalizable jitter. Unequalizable jitter is generally accepted to be the total jitter that is not correlated with the data, and cannot therefore be reduced through equalization. Unequalizable energy, however, is completely correlated with the data and could be reduced if the equalization solution were better. Note also that standards such as IEEE 802.3ap, Annex 69B [6] address this type of phenomenon through the concept of insertion loss deviation, or more colloquially ripple. As this metric is usually defined, however, it does not address the number of ripples in the frequency domain that matter, or in other words the time range outside of which the time domain ripples matter. Another way to look at this is that equalizers can typically match smooth frequency responses rather well, but have difficulty matching frequency responses that vary rapidly. As currently written, the standards do not distinguish between the two cases. These insertion loss deviation metrics could be extended to be equivalent to unequalizable energy by stating the maximum number of frequency domain ripples that can be present within a given frequency range. Another way to achieve the same result would be to extend the LMS fitting routines in IEEE802.3ap to fit, say, a cubic polynomial to the loss curve and calculate the insertion loss deviation from that fit. Acceptable Limits In an effort to determine acceptable limits, the bit error rate and unequalizable energy was calculated for all of the paths in the system. Figure 16 shows the resulting scatter plot. When Shorter Isn t Better SiSoft, 2010 Page 19

20 Figure 16: Scatter plot of bit error rate vs. unequalizable energy In Figure 16, a red line has been drawn to indicate a typical system bit error rate requirement of 1E-12. Given the equalization solution used in this portion of the study, an unequalizable energy of 0.6% would be required. This can be considered a safe value for most systems. With a more nearly optimum equalization solution, the unequalizable energy might be allowed to reach 1.0%. Any value above 1.0%, however, must be considered cause for serious concern. Note that since unequalizable energy is supposed to be, by definition, insensitive to the details of the equalization solution, it can be applied to a passive electrical interconnect before the equalization solution has been chosen. Table 3 compares the unequalizable energy for the three RA routing length cases studied above, with and without equalization. RA routing length Unequalizable energy with equalization (%) Unequalizable energy for passive interconnect (%) Table 3: Unequalizable energy with and without equalization When Shorter Isn t Better SiSoft, 2010 Page 20

21 While the two measurements are not exactly the same in all cases, the unequalizable energy of the passive network does appear to be close enough to serve as a reliable early test for standing wave problems. Remediation Evaluation of TDR plots suggested that the unequalizable energy might be due to standing waves caused by reflections from the connector vias on the RA board. A set of analyses was therefore run using the via designs from the BW board in place of the vias designs for the RA board. Since the BW vias have a higher impedance, and therefore a lower reflection coefficient, the hypothesis was that they would reduce the level of the standing waves, and therefore the unequalizable energy. Eye height and eye width were also used as a measure of performance margin in place of the bit error rate estimates. Table 4 summarizes the results of this experiment. RA Length Unequalizable Energy(%) Eye Height (V) Eye Width (ps) RA via BW via RA via BW via RA via BW via Table 4: Via remediation experiment results These results confirm the hypothesis in that the BW vias dramatically reduce the unequalizable energy and consistently increase the eye height. In fact, it may very well be that if the BW vias had been used on the RA board, this study might never have been performed. Discussion System measurements and subsequent analysis demonstrated that the exact timing of reflections between closely spaced discontinuities in a transmission path can have an extreme effect on bit error rate. These performance problems have proven to be hard to detect using conventional analysis and measurement techniques. What makes these problems so hard to isolate is the fact that the channel performance is so sensitive to small variations in path length. For example, typical variations in dielectric constant could cause a given path to perform well on one PC board and very poorly on another board with the same design. The poor performance isn t detectable unless the phases add up in just the wrong way. Because of this extreme sensitivity, paths which are all but identical can have very different performance. This is what was experienced in extensive system level tests. We have proposed unequalizable energy as a metric which can be used to reliably detect the potential for this type of problem early in the design of a system. Unequalizable energy is much less sensitive to small variations in phase, and so can detect standing When Shorter Isn t Better SiSoft, 2010 Page 21

22 wave problems before they show up in the bit error rate. Unequalizable energy is also, by definition, insensitive to the details of the equalization solution, and therefore can be applied to the passive electrical interconnect before the equalization solution has been chosen. Acknowledgements Building the characterization infrastructure and collecting the large amounts of test data at Cray was a considerable undertaking. The authors would like to acknowledge the efforts of Roger Bethard, Tim Ingebritson, Kelly Marquardt, Steve Martin and Chris White. References [1] Steve Scott, Dennis Abts, John Kim, William J. Dally, The BlackWidow High- Radix Clos Network, Proceedings of the 33rd annual international symposium on Computer Architecture, p.16-28, June 17-21, 2006 [2] Dennis Abts, Abdulla Bataineh, Steve Scott, Greg Faanes, Jim Schwarzmeier, Eric Lundberg, Tim Johnson, Mike Bye, Gerald Schwoerer, The Cray BlackWidow: a highly scalable vector multiprocessor, Proceedings of the 2007 ACM/IEEE conference on Supercomputing, November 10-16, 2007, Reno, Nevada [3] Michael Steinberger, Todd Westerhoff, Christopher White, "Demonstration of SerDes Modeling Using the Algorithmic Model Interface (AMI) Standard", paper 7-TA3, DesignCon2008. [4] Chong Ding, Divya Gopinath, Steve Scearce, Mike Steinberger, Doug White, "A Simple Via Experiment", paper 5-TP2, DesignCon2009. [5] Troy Beukema, Challenges in Serial Electrical Interconnects at 5 to 10 Gb/s and Beyond, IEEE SSCS, Denver Section Seminar, Fort Collins, CO, March [6] IEEE Draft P802.3apTM/Draft 3.3, IEEE, copyright When Shorter Isn t Better SiSoft, 2010 Page 22