FB-DIMM Commands/Data and Lane Traffic Verification Preparing for FB-DIMM Fully buffered dual inline memory modules (FB-DIMMs) provide servers and workstations with greater memory capacities, higher operating reliability and easier serviceability. This note details how to verify FB-DIMM designs for conformance to manufacturer s specifications to ensure correct and reliable operation. Introduction Memory system and component designers face a wide variety of challenging tests as they verify their designs and bring their fully buffered dual inline memory module (FB-DIMM) products to the market. (Figure 1) These tests validate, characterize and debug their FB-DIMM designs. Validation to FB-DIMM specifications is required to ensure their products are compatible and reliable. FB-DIMM designs are implemented with fast edges, high clock frequencies and high-speed serial signals, resulting in problems designers may not have seen before. Older memory designs could tolerate a wider range of signal timing and amplitude variances. With FB-DIMMs, this is no longer the case. With FB-DIMM s faster speeds, design margins decrease and analog characteristics of digital signals are increasingly important for the FB-DIMM memory system signal integrity.
To keep pace with the more complex and shorter design cycles, designers need to optimize their FB-DIMM testing. In order for designers to be successful in these chal- FB-DIMM 0 FB-DIMM 1 FB-DIMM 7 lenging circumstances, Tektronix offers the TLA7000 Series of Logic Analyzers to enable you to implement a better FB-DIMM design, in less time. Memory Interface Southbound Traffic We begin by focusing on the verification of FB-DIMM Northbound Traffic commands and data, followed by verification of the FB-DIMM memory lane traffic. FB-DIMM Commands and Data Verification Clocks The (Advanced Memory Buffer) controls all the signals to the SDRAM (Double Data Rate2 Synchronous Dynamic Random Access Memory) on the FB-DIMM and is responsible for initializing the memory, refreshing the memory, writing data to the memory and reading data from the memory. The FB-DIMM designer needs to verify that these to SDRAM operations are correctly carried out to ensure reliable memory operation. Potential problem areas include incorrect initialization of the mode registers, wrong sequence of memory commands, refresh errors, command timing problems, data errors, etc. Comprehensive Test Solution The Tektronix TLA7000 Series modular logic analyzers, TLA7AA4 logic analyzer modules, P6860/P6864 logic analyzer probes and the Nexus Technology NEX-FBD-NEXVu are used to verify, test and debug the control of the SDRAM, as shown in Figure 2. The is responsible for all signals going to the SDRAM. This includes clock, command, address, bank address, data, data mask, data strobe, etc. The configures, writes data and reads data to the SDRAMs. And just like a DIMM the FB-DIMMs can have one or more ranks of memory. The Nexus Technology NEX-FBD-NEXVu is a FB-DIMM with, SDRAMs and logic analyzer probing footprints, as shown in Figure 3. The NEX-FBD-NEXVu also includes logic analyzer processor support software to configure the logic analyzer to acquire data from the NEX-FBD-NEXVu FB-DIMM and software to decode the acquired data. Figure 1. FB-DIMM architecture overview. Memory Interface Southbound Traffic Northbound Traffic FB-DIMM 0 TLA7016 Logic Analyzer NEX-FBD-NEXVu FB-DIMM 7 Figure 2. Monitoring the signals and commands between the and the SDRAM on a FB-DIMM. Figure 3. Nexus Technology NEX-FBD-NEXVu monitors the signals and commands between the and SDRAMs on the FB- DIMM. The is bottom center and the SDRAMs are on both sides of the. (Image courtesy of Nexus Technology, Inc.) 2 www.tektronix.com/memory
Figure 4. SDRAM mode register and extended mode register being programmed by. One Probe, Three Simultaneous Measurements The traces coming from the SDRAM s FBGAs and going to the logic analyzer footprints are isolated within circuit board layers with isolation resistors. These footprints are designed for Tektronix P6860/P6864 logic analyzer probes. For the highest signal fidelity the P6860/P6864 probes attach directly to the NEXVu circuit board. To reduce degrading the signals being measured, no probing connectors are used on the circuit board. The P6860/P6864 logic probes enable a single probing connection for both the logic analyzer and the oscilloscope. The result is three simultaneous measurements are being made through one probe. The logic analyzer simultaneously acquires the signals with 125 ps highresolution MagniVu timing and with up to 1.25 Gb/s state acquisition. The oscilloscope measures the same signals with up to 40 GS/s sample rate. Using the logic analyzer probe for the oscilloscope is like having a 2 GHz oscilloscope probe immediately available at each logic analyzer probe connection. If the signal is differential, like the SDRAM clock, the active logic analyzer probe converts the differential clock signal to a single-ended signal to be used by the oscilloscope. SDRAM Mode Registers During the SDRAM initialization the mode register (MR) and extended mode registers (EMR) are configured by the. The mode register is used to define the specific mode of operation for the SDRAM. This includes the column address select (CAS) latency, burst length, burst type, operating mode, digital delay-locked loop (DLL) reset, power-down (PD) mode and write recovery. The extended mode registers include additional functions such as on-die termination (ODT) values, enabling differential data strobes, digital delay-locked loop (DLL) enable/disable, output enable/disable, etc. The mode register and extended mode registers are configured with the SDRAM load mode command. The NEX-FBD-NEXVu traces the configuration of the mode register and the extended mode registers. It also decodes the register s bit fields and provides an easy to read description, as shown in Figure 4. In this example, sample 6 at the top shows the mode register being configured for standard power-down. The write recovery is set at 3, the DLL is in normal operation and is not being reset. The operating mode is normal and is not in the test mode which is only used by the SDRAM manufacturer. CAS latency is set at 4. Burst type is interleaved and not sequential. Burst length is 4, which determines the maximum number of column locations that are accessed for read or write command. SDRAMs support burst lengths of 4 or 8 which provides an increase in data throughput. Also in sample 7 in the bottom half of Figure 4, the extended mode register is being configured. www.tektronix.com/memory 3
Figure 5. A Micron MT47H256M4 1 Gb SDRAM memory which is configured as 32 Meg x 4 outputs x 8 banks. The 17 address lines are multiplexed between the row and column addresses. There are 16 outputs because this memory has 4 data outputs and it delivers a burst of 4 bits/data line output. (Image courtesy of Micron Technology, Inc.) SDRAM Memory Structure SDRAMs below 1 Gb have 4 banks and at 1 Gb and above they have 8 banks. The SDRAM s bank address pins determine which bank is being addressed. A row is also referred to as a memory page and contains memory organized by columns. All columns of the selected row are at the sense amplifiers, as shown in Figure 5. Once a row is activated multiple reads and writes can be performed on the different columns of the activated row. Reading SDRAM Data After initialization the mode register and extended mode registers the sends commands to read data and to write data to memory. As shown in Figure 6 in the top sample number 2039, the sends a precharge command. The precharge command deactivates the open row and needs to be sent before activating a different row in the same memory bank. Next the activate command is used to activate a row in a designated bank to start the command sequence needed to read data from memory. 4 www.tektronix.com/memory
Figure 6. reading data from the SDRAM on a FB-DIMM. The next command is a read command which selects the column in the row (page) to be read. The memory will output a burst of 4 or 8 memory locations depending upon the mode register burst length. Two more read commands are sent right after the first read command. SDRAM outputs bursts of 4 read data for each read command sent by the. FB-DIMM Commands and Data Verification Summary Table 1 summarizes the FB-DIMM to SDRAM verification tests and the test equipment needed to perform these tests. After completing these tests the FB-DIMM is ready for memory channel testing. How do I verify my design? What is my solution? What are the key instrument capabilities? to SDRAM initialization and commands 1 TLA7016 Logic Analyzer Modular Mainframe Simultaneous state acquisition and 125 ps SDRAM data valid windows 3 TLA7AA4 Logic Analyzer Modules high-resolution MagniVu timing acquisition Mode register and extended mode register settings 4 P6860 probes SDRAM mnemonics in state display is refreshing memory 4 P6864 probes Three types of memory storage 1 Nexus Technology NEX-FBD-NEXVu Low signal loading using connectorless probing with isolation resistors Table 1. to SDRAM verification test equipment summary. www.tektronix.com/memory 5
TLA7016 Logic Analyzer FB-DIMM 0 FB-DIMM 1 FB-DIMM 2 NEX-FBD-LAI Interposer Memory Interface Southbound Traffic Northbound Traffic To other FB-DIMMs Figure 7. Monitoring the FB-DIMM southbound and northbound traffic. FB-DIMM Memory Lane Traffic Interposer The northbound and southbound lanes between the memory controller and the first FB-DIMM carry all memory channel traffic going to and coming from all of the FB-DIMMs. To monitor all the memory traffic an interposer with its configured as a logic analyzer interface (LAI) is installed in the first FB-DIMM slot, as shown in Figure 7. Typically FB-DIMM memory systems support multiple memory channels. To monitor all memory channels an interposer is used in the first slot of each channel. LAI Mode The LAI mode is a special debugging mode of the. The data rate of the northbound and southbound lanes exceeds the capture data rate of a typical logic analyzer. The in LAI mode monitors the northbound and southbound traffic, de-serializes the data and outputs the data at SDRAM data rates which are six times slower than the northbound and southbound lane data rates. This slower, wider data is then captured at the outputs by the logic analyzer. 6 www.tektronix.com/memory
NEX-FBD-LAI The Nexus Technology NEX-FBD-LAI monitors northbound and southbound traffic with its in LAI mode. The NEX-FBD-LAI as shown in Figure 8 is designed so that the FB-DIMM replaced by the NEX-FBD-LAI is installed on its top edge of the NEX-FBD-LAI, as shown in Figure 9. This results in not losing a FB-DIMM when the NEX-FBD-LAI is in place. The NEX-FBD-LAI is powered from its own power supply and does not draw power from the computer motherboard. The NEX-FBD-LAI is controlled and configured with a SMBus interface that is separate from the computer motherboard s SMBus. A SMBus to USB interface cable is attached between the NEX-FBD-LAI and the USB port on the TLA7016 logic analyzer. SMBus software running on the logic analyzer simplifies configuration of the NEX-FBD-LAI. Figure 8. The NEX-FBD-LAI has its in the LAI mode to de-serialize the northbound/southbound traffic and provides data to the logic analyzer. (Image courtesy of Nexus Technology, Inc.). A NEX-FBD-LAI event bus connects multiple NEX-FBD-LAIs when multiple FB-DIMM channels are being monitored by multiple NEX-FBD-LAIs. Figure 9. The FB-DIMM is plugged into the top of the NEX-FBD-LAI so that there is no reduction in FB-DIMMs on the memory channel. (Image courtesy of Nexus Technology, Inc.). www.tektronix.com/memory 7
Analyzing Lane Traffic The logic analyzer captures the de-serialized data in a state acquisition listing display, as shown in Figure 10. The southbound traffic is shown in the listing view as 10 columns of data representing the 10 southbound lanes and the northbound traffic is shown as 14 columns of data representing the 14 northbound lanes. A FB-DIMM frame is 12 serial bits times the number of southbound or northbound lanes. For example, 12 serial bits times 14 northbound lanes equals 168 bits per northbound frame. As a result, the north bound frame can contain two 72 bit read words with CRC bits and transfers read data from the FB-DIMM to the memory controller hub at full memory speed. The Nexus Technology NEX-FBD-LAI software decodes southbound and northbound traffic in Figure 10 into a more descriptive representation, as shown in Figure 11. In Figure 11, the top sample is a beginning of four frames containing write data to the FB-DIMM from the memory controller hub. The memory controller hub sends a southbound command to activate DIMM: 0, Rank: 0, Bank: 0 and Address: 0000. The next command in the same frame is a 64-bit write data of 1111,1111,1111,1111 with ECC of 11. In this example, the memory operates in a burst length of four and the next three frames contain the remaining three data words to be written to memory. Also, during the second frame the northbound traffic is a status frame. Figure 10. Nexus Technology NEX-FBD-LAI software decodes the 10 southbound lanes and 14 northbound lanes (shown in columns) in JEDEC state format. 8 www.tektronix.com/memory
Figure 11. Nexus Technology NEX-FBD-LAI software decodes southbound and northbound traffic. The southbound traffic is an active command with write data to DIMM 0. FB-DIMM Memory Lane Traffic Summary Testing, verify and debugging the memory channel is the highest level of functional testing. At the memory system level the complete memory channel is being exercised with multiple FB-DIMMs operating. The memory controller initializes the memory channel, it writes data to the different FB-DIMMs on the channel and it reads data from the FB-DIMMs. Test equipment verifies memory channel initialization, channel operation and monitors it for faults, as shown in Table 2. How do I verify my design? What is my solution? What are the key instrument capabilities? Memory controller to communications 1 TLA7016 Logic Analyzer Modular Mainframe Logic analyzer simultaneous state acquisition FB-DIMM to FB-DIMM communications 3 TLA7AA4 Logic Analyzer Modules and high-resolution MagniVu timing acquisition Memory channel commands, 10 P6860 probes JEDEC & software state display data writes and data reads 1 Nexus Technology NEX-FBD-LAI Self powered interposer with in LAI mode merging northbound traffic repeating southbound traffic Table 2. Northbound and southbound commands, data and response testing and equipment used. www.tektronix.com/memory 9
Summary FB-DIMM memory system architecture provides an increase in memory capacity and an increase in memory data rates for designers of high-performance servers and workstations. However, this tremendous increase in data rates also presents new design challenges that the designers must resolve. In order for designers to be successful in these challenging circumstances, Tektronix offers the TLA7000 Series of Logic Analyzers to enable you to implement a better FB-DIMM design, in less time. Tektronix also offers a comprehensive tool set including industry-leading oscilloscopes and true differential TDRs to enable FB-DIMM designers to perform quick and accurate electrical testing and operational validation of their memory designs. Collectively, this tool set provides superior performance with unparalleled ease-of-use, making it an ideal solution for FB-DIMM verification and debugging. 10 www.tektronix.com/memory
www.tektronix.com/memory 11
Contact Tektronix: ASEAN / Australasia / Pakistan (65) 6356 3900 Austria +41 52 675 3777 Balkan, Israel, South Africa and other ISE Countries +41 52 675 3777 Belgium 07 81 60166 Brazil & South America 55 (11) 3741-8360 Canada 1 (800) 661-5625 Central East Europe, Ukraine and Baltics +41 52 675 3777 Central Europe & Greece +41 52 675 3777 Denmark +45 80 88 1401 Finland +41 52 675 3777 France & North Africa +33 (0) 1 69 86 81 81 Germany +49 (221) 94 77 400 Hong Kong (852) 2585-6688 India (91) 80-22275577 Italy +39 (02) 25086 1 Japan 81 (3) 6714-3010 Luxembourg +44 (0) 1344 392400 Mexico, Central America & Caribbean 52 (55) 56666-333 Middle East, Asia and North Africa +41 52 675 3777 The Netherlands 090 02 021797 Norway 800 16098 People s Republic of China 86 (10) 6235 1230 Poland +41 52 675 3777 Portugal 80 08 12370 Republic of Korea 82 (2) 528-5299 Russia & CIS 7 095 775 1064 South Africa +27 11 254 8360 Spain (+34) 901 988 054 Sweden 020 08 80371 Switzerland +41 52 675 3777 Taiwan 886 (2) 2722-9622 United Kingdom & Eire +44 (0) 1344 392400 USA 1 (800) 426-2200 For other areas contact Tektronix, Inc. at: 1 (503) 627-7111 Last Updated June 15 2005 For Further Information Tektronix maintains a comprehensive, constantly expanding collection of application notes, technical briefs and other resources to help engineers working on the cutting edge of technology. Please visit www.tektronix.com Copyright 2005, Tektronix, Inc. All rights reserved. Tektronix products are covered by U.S. and foreign patents, issued and pending. Information in this publication supersedes that in all previously published material. Specification and price change privileges reserved. TEKTRONIX and TEK are registered trademarks of Tektronix, Inc. All other trade names referenced are the service marks, trademarks or registered trademarks of their respective companies. 10/05 DV/WOW 52W-19113-1