OpenFabrics Alliance Interoperability Working Group (OFA-IWG)

Transcription

1 Cover Letter Nimrod Gindi Mellanox Technologie Hermon Building 4th Floor P.O. Box 586, Yokenam Irael OpenFabric Alliance Interoperability Working Group (OFA-IWG) December 2008 Logo Event Report UNH-IOL 121 Technology Drive, Suite 2 Durham, NH OpenFabric Interoperability Logo Group (OFILG) Hot ofalab@iol.unh.edu December 30, 2008 Report Rev1.02 Ofed Verion: 1.4 Encloed are the reult from OFA Logo teting performed on the following device under tet (DUT): Mellanox MHET2X-1TC (InfiniHot HCA, dual-port 10Gb/, PCI-X, M B) Mellanox MHEA28-1TC (InfiniHot III Ex HCA, dual-port 10Gb/, PCIe1.2 x8, M B) Mellanox MHEA28-XTC (InfiniHot III Ex HCA, dual-port 10Gb/, PCIe1.2 x8, Me mfree) Mellanox MHES18-XTC (InfiniHot III Lx HCA, ingle-port 10Gb/, PCIe1.2 x8, Me mfree) Mellanox MHES14-XTC (InfiniHot III Lx HCA, ingle-port 10Gb/, PCIe1.2 x4, Me mfree) Mellanox MHGA28-1TC (InfiniHot III Ex HCA, dual-port 20Gb/, PCIe1.2 x8, M B) Mellanox MHGA28-XTC (InfiniHot III Ex HCA, dual-port 20Gb/, PCIe1.2 x8, Me mfree) Mellanox MHGS18-XTC (InfiniHot III Lx HCA, ingle-port 20Gb/, PCIe1.2 x8, Me mfree) Mellanox MHEH28-XTC (ConnectX IB HCA, dual-port 10Gb/, PCIe2.0 x8, Me mfree) Mellanox MHGH28-XTC (ConnectX IB HCA, dual-port 20Gb/, PCIe2.0 x8, Me mfree) Mellanox MHGH29-XTC (ConnectX IB HCA, dual-port 20Gb/, PCIe2.0 x8 5GT/, MemFree) Mellanox MHJH29-XTC (ConnectX IB HCA, dual-port 40Gb/, PCIe2.0 x8 5GT/, Me mfree) Mellanox MHQH29-XTC (ConnectX IB HCA, dual-port 40Gb/, PCIe2.0 x8 5 GT/, Me mfree) The tet uite referenced in thi report i available at the OFA webite, at tet time releae 1.22 (Augut 29, 2008 DRAFT) wa ued: The following table highlight the Mandatory tet reult required for the OpenFabric Interoperability Logo for the DUT per the tetplan referenced above and the current OpenFabric Interoperability Logo Program (OFILP). Mandatory Tet Procedure IWG Tet Statu Reult/Note 10.1: IB Link Initialization Mandatory Paed ee com 10.2: IB Fabric Initialization Mandatory Paed no iue een 10.3: IB IPoIB Connected Mode Mandatory Paed no iue een 10.9: TI iser Mandatory Not Available 10.10: SRP Mandatory Paed ee com 10.11: SDP Mandatory Paed no iue een For pecific detail regarding iue pleae ee the correponding tet reult. Summary of all reult follow on the econd page of thi report. Teting pleted 12/23/2008 Review pleted 01/23/2009 Nickola Wood ndv2@iol.unh.edu Bob Noeworthy ren@iol.unh.edu

2 Table 1: Reult Summary OFA Logo Event Report December 2008 The following table ummarize all reult from the event pertinent to an IB device. Tet Procedure IWG Tet Statu Reult/Note 10.1: IB Link Initialization Mandatory Paed ee com 10.2: IB Fabric Initialization Mandatory Paed no iue een 10.3: IB IPoIB Connected Mode Mandatory Paed no iue een 10.4: IB IPoIB Datagram Mode Beta Paed no iue een 10.9: TI iser Mandatory Not Available 10.10: SRP Mandatory Paed ee com 10.11: SDP Mandatory Paed no iue een 10.12: IB SM Failover and Handover Beta Not Teted 10.13: TI MPI - OSU Beta Informative 10.14: TI MPI - Intel Beta Informative 10.15: HP MPI - HP Beta Informative 10.16: TI MPI - Open Beta Informative 10.17: TI udapl Beta Paed no iue een 10.19: IB FibreChannel Gateway Beta Not applicable to DUT 10.20: IB Ethernet Gateway Beta Not applicable to DUT 10.21: IB Reliable Datagram Socket Beta Not Teted : TI Baic RDMA Interoperability : TI RDMA Operation over Interconnect ponent Beta Beta Not Teted Not Teted Digital Signature Information Thi docu wa created uing an Adobe digital ignature. A digital ignature help to enure the authenticity of the docu, but only in thi digital format. For information on how to verify thi docu integrity proceed to the following ite: If the docu tatu till indicate Validity of author NOT confirmed, then pleae contact the UNH-IOL to confirm the docu authenticity. To further validate the certificate integrity, Adobe 6.0 hould report the following fingerprint information: MD5 Fingerprint: F6E2 1B99 28AD 0D25 E77E ADE5 479A 1E05 SHA-1 Fingerprint: AD30 8B08 DD3B B2E E BE47 1D49 890B UNH-IOL OFA OFILG 2 Report Rev1.02

3 Report Reviion Hitory v1.0 Initial Releae OFA Logo Event Report December 2008 v1.01 Added Firmware and Hardware Verion Number v1.02 Editorial Change Table 2: Reult Key The following table contain poible reult and their meaning: Reult: Decription: The Device Under Tet (DUT) wa oberved to exhibit conformant behavior. The DUT wa oberved to exhibit conformant behavior however an additional explanation of the ituation i included, uch a due to time limitation only a portion of the teting wa performed. FAIL Warning Informative Refer to Not Applicable Not Available Borderline Not Teted The DUT wa oberved to exhibit non-conformant behavior. The DUT wa oberved to exhibit behavior that i not recommended. Reult are for informative purpoe only and are not judged on a pa or fail bai. From the obervation, a valid pa or fail could not be determined. An additional explanation of the ituation i included. The DUT doe not upport the technology required to perform thi tet. Due to teting tation limitation or time limitation, the tet could not be performed. The oberved value of the pecified parameter are valid at one extreme and invalid at the other. Not teted due to the time contraint of the tet period. UNH-IOL OFA OFILG 3 Report Rev1.02

4 Table 3: DUT and Tet Setup Information Figure 1: The IB fabric configuration utilized for any tet requiring a multi-witch configuration i hown below. DUT #1 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHET2X-1TC Hardware Rev a1 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHET2X-1TC (InfiniHot HCA, dual-port 10Gb/, PCI-X, M B, Cou gar S D R ) DUT #2 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHEA28-1TC Hardware Rev a0 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHEA28-1TC (InfiniHot III Ex HCA, dual-port 10Gb/, PCIe1.2 x8, 128MB, Lion Cub S D R ) UNH-IOL OFA OFILG 4 Report Rev1.02

5 DUT #3 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHEA28-XTC Hardware Rev a0 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHEA28-XTC (InfiniHot III Ex HCA, dual-port 10Gb/, PCIe1.2 x8, Me mfree, Lion Mini S D R ) DUT #4 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHES18-XTC Hardware Rev a0 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHES18-XTC (InfiniHot III Lx HCA, ingle-port 10Gb/, PCIe1.2 x8, Me mfree, Cheetah S D R ) DUT #5 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHES14-XTC Hardware Rev a0 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHES14-XTC (InfiniHot III Lx HCA, ingle-port 10Gb/, PCIe1.2 x4, Me mfree, Tiger S D R ) DUT #6 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHGA28-1TC Hardware Rev a0 Speed DDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHGA28-1TC (InfiniHot III Ex HCA, dual-port 20Gb/, PCIe1.2 x8, 128MB, Lion Cub D D R ) DUT #7 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHGA28-XTC Hardware Rev 20 Speed DDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHGA28-XTC (InfiniHot III Ex HCA, dual-port 20Gb/, PCIe1.2 x8, Me mfree, Lion Mini D D R ) DUT #8 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHGS18-XTC Hardware Rev a0 Speed DDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHGS18-XTC (InfiniHot III Lx HCA, ingle-port 20Gb/, PCIe1.2 x8, Me mfree, Cheetah D D R ) DUT #9 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHEH28-XTC Hardware Rev a0 Speed SDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHEH28-XTC (ConnectX IB HCA, dual-port 10Gb/, PCIe2.0 x8, Me mfree ConnectX S D R ) DUT #10 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHGH28-XTC Hardware Rev a0 Speed DDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHGH28-XTC (ConnectX IB HCA, dual-port 20Gb/, PCIe2.0 x8, Me mfree ConnectX D D R ) UNH-IOL OFA OFILG 5 Report Rev1.02

6 DUT #11 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHGH29-XTC Hardware Rev a0 Speed DDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHGH29-XTC (ConnectX IB HCA, dual-port 20Gb/, PCIe2.0 x8 5 GT/, Me mfree) DUT #12 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHJH29-XTC Hardware Rev a0 Speed QDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHJH29-XTC (ConnectX IB HCA, dual-port 40Gb/, PCIe2.0 x8 5GT/, Me mfree) DUT #13 Detail Manufacturer Mellanox Ltd. Firmware Rev Model MHQH29-XTC Hardware Rev a0 Speed QDR 4x IP Addre in Fabric Vendor Provided Detail Mellanox MHQH29-XTC (ConnectX IB HCA, dual-port 40Gb/, PCIe2.0 x8 5 GT/, Me mfree) UNH-IOL OFA OFILG 6 Report Rev1.02

7 Mandatory Tet - IB Device Tet Summary Reult: The following table detail reult for tet identified by the OFA-IWG a mandatory tet for the OFA Interoperability Logo Program (OFILP) per the OFA-IWG Interoperability Tet Plan. 1 Dione MHGH29-XTC 2 Hyperion MHEH28-XTC 3 Teleto MHGS18-XTC 4 Pandora MHGA28-1TC 5 Titan MHES18-XTC 6 Phoebe MHEA28-XTC Tet #1: Group 1: IB Link Initialize 7 Calypo MHES14-XTC 8 Skathi MHJH29-XTC 9 Janu MHET2X-1TC 10 Tarvo MHEA28-1TC 11 Farbauti MHQH29-XTC 12 Atla MHGH28-XTC 13 Mima MHGA28-1TC Dicuion: Tet #1: Phy link up all port Phyical link initialization wa verified between thi device and every other device in the fabric. Link tatu wa oberved viually via tatu light on the device. Port width and link peed wa verified via ibdiagnet. Thi verion of the tet plan doe not explicitly call for teting of link peed, the next verion will, and logo teting in the 2 nd half of 2009 will require proper link peed between all link combination. Narvi and Skoll only link at SDR peed to all Mellanox DDR/QDR HCA' even though they are DDR card themelve. Link Partner Device QLogic SilverStorm 9024 (Switch) QLogic SilverStorm 9040 (Switch) Flextronic F- X (Switch) Flextronic F- X (Switch) Mellanox MTS3600 (Switch) Obidian Longbow- XR (Range Extender) Obidian Longbow- XR (Range Extender) Obidian Longbow Campu (Range Extender) Obidian Longbow UNH-IOL OFA OFILG 7 Report Rev1.02

8 Campu (Range Extender) LSI XBB1 (SRP Target) LSI XBB2 (SRP Target) DataDirect Network (SRP Target) Hot: Miranda HCA: QLogic QLE7140 SDR Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Narvi, G2 PCI Expre HCA: QLogic QLE7280 DDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A UNH-IOL OFA OFILG 8 Report Rev1.02

9 Connectx DDR Hot: Mima LionMini DDR Hot: Skoll HCA: QLogic QLE7240 DDR Hot: Pandora LionCub DDR N/A N/A UNH-IOL OFA OFILG 9 Report Rev1.02

10 Group 2: IB Fabric Initialization Tet #1: Port i Active all SM Dicuion: Tet #1: Verify all SM configure fabric The fabric configuration hown in Figure 1 wa ued for thi tet. 'ibdiagnet -c 1000' howed no Port error counter incre. Only one SM i run at a time. All witche are power cycled between SM trial. All link are validated via ue of 'ibdiagnet' and 'ibchecknet' wa ued to verify that there were no duplicate guid. Refer to the table below for SM detail. SM teted include: OFED OpenSM (SM Only), QLogic SilverStorm 9024 (Managed Switch), QLogic SilverStorm 9040 (Managed Switch) Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR For each SM lited above All port Armed/Active No Dup GUID No Port error UNH-IOL OFA OFILG 10 Report Rev1.02

11 Group 3: IPoIB Connected Mode Tet #1-3 Tet completed out error Dicuion: Tet #1 An automated tet cript wa ued to end ICMP Echo Requet packet payload of pecific ize between all hot on the configured fabric. Thi procedure wa repeated each ubnet manager independently managing the fabric. Dicuion: Tet #2 An HCA wa diconnected from the fabric and reconnected in a different location; the ICMP Echo Reply packet ceaed while the HCA wa diconnected, and then reumed when it wa reconnected. Thi procedure wa repeated once each ubnet manager independently managing the fabric. Dicuion: Tet #3 An automated tet cript wa ued to tranfer an 4MB file uing the SFTP protocol between all hot on the configured fabric. The file wa tranferred four time in each direction between all hot, and the content of the file wa verified after each tranfer. Thi procedure wa repeated each ubnet manager independently managing the fabric. For all tet cae, SM teted include: OFED OpenSM (SM Only), QLogic SilverStorm 9024 (Managed Switch), QLogic SilverStorm 9040 Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR For each SM lited above Tet 1 Tet 2 Tet 3 UNH-IOL OFA OFILG 11 Report Rev1.02

12 Group 9: TI iser Tet #1-4 Not Available Not Available Dicuion: Tet #1-4 Group 10: IB SRP Tet #1: Automated Tet Script Dicuion: Tet Reult The automated tet cript wa revied ince the verion publihed in the tet docu. The automated tet cript run the operation in the tet plan every available hot and log the reult. The log how that the data tranfer operation completed for each hot to each available volume on each target, in both the read and write direction while uing OpenSM to configure the fabric. The 9024 and the 9040 SM' produced error 2 mellanox HCA' during the SRP operation; Titan and Calypo. Thee error can be circumvented by tarting either QLogic SM before thee HCA' etablih link the fabric. Becaue of thi olution, thee device did not fail. The current tet plan doe not pecify when to tart an SM or which HCA' to ue. The next verion will. For DataDirect Network S2A 9900 Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR Verified Automated Tet Reult UNH-IOL OFA OFILG 12 Report Rev1.02

13 For LSI XBB2-IB (Dual Controller SRP Storage Sytem) Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR For LSI Engenio 0825 (SRP Storage Sytem) Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR Verified Automated Tet Reult Verified Automated Tet Reult UNH-IOL OFA OFILG 13 Report Rev1.02

14 Group 11: TI SDP Tet #1: Netperf Tet pleted out error Tet #2: FTP Tet pleted out error Tet #3: SCP Tet pleted out error Dicuion: Tet #1-3 The automated tet cript ued in the lat event wa ued again during thi event the addition of a wrapper program to control the cluter environ and facilitate better logging. The automated tet cript run the three part of the SDP procedure between every poible pair of hot out the hot connecting to themelve and record the reult to a log. The tet log how that no iue were een the procedure. Every operation completed for each pair. However, ome hot were noted to run ignificantly lower than other during the tranfer. Thi i not a failure a per the current tet plan, but it hould be noted that thi could become a topic of focu in future reviion of the Tet Plan. Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR Tet 1 Tet 2 Tet 3 UNH-IOL OFA OFILG 14 Report Rev1.02

15 Beta Tet - IB Device Tet Reult: The following table detail reult for tet identified by the OFA-IWG a beta tet for the OFA Interoperability Logo Program (OFILP) per the OFA-IWG Interoperability Tet Plan. Group 4: IPoIB Datagram Mode Tet #1-3 Tet ucceeded between all IPoIB device Dicuion: Tet #1 An automated tet cript wa ued to end ICMP Echo Requet packet payload of pecific ize between all hot on the configured fabric. Thi procedure wa repeated each ubnet manager independently managing the fabric. Dicuion: Tet #2 An HCA wa diconnected from the fabric and reconnected in a different location; the ICMP Echo Reply packet ceaed while the HCA wa diconnected, and then reumed when it wa reconnected. Thi procedure wa repeated once each ubnet manager independently managing the fabric. Dicuion: Tet #3 An automated tet cript wa ued to tranfer an 4MB file uing the SFTP protocol between all hot on the configured fabric. The file wa tranferred four time in each direction between all hot, and the content of the file wa verified after each tranfer. Thi procedure wa repeated each ubnet manager independently managing the fabric. For all tet cae, SM teted include: OFED OpenSM (SM Only), QLogic SilverStorm 9024 (Managed Switch), QLogic SilverStorm 9040 UNH-IOL OFA OFILG 15 Report Rev1.02

16 Hot: Tarvo LionCub SDR Hot: Hyperion Connectx SDR Hot: Janu Cougar SDR Hot: Phoebe LionMini SDR Hot: Titan Cheetah SDR Hot: Calypo Tiger SDR Hot: Skathi, G2 PCI Expre Connectx QDR Hot: Farbauti, G2 PCI Expre Connectx QDR Hot: Atla Connectx DDR Hot: Teleto Cheetah DDR Hot: Dione, G2 PCI Expre Connectx DDR Hot: Mima LionMini DDR Hot: Pandora LionCub DDR For each SM lited above Tet 1 Tet 2 Tet 3 Group 12: IB SM Failover and Handover Tet #1-4 Not Teted Dicuion: Tet Reult Not teted due to time contraint UNH-IOL OFA OFILG 16 Report Rev1.02

17 Group 13: TI MPI Ohio State Univ. Tet #1 PingPing Tet pleted out error and PingPong Tet #2 All Tet failed Informative Dicuion: Mvapich1, tet #2 The following error wa oberved in the log file for the AllGather 62 proce benchmark. Exit code -3 ignaled from titan-ib Killing remote procee...abort ignaled by rank 61: [miranda.ofa:61] Got completion error IBV_WC_RETRY_EXC_ERR, code=12, det rank=29 MPI proce terminated unexpectedly Dicuion: Mvapich2, tet #2 The following error wa oberved in the log file for the Gather 62 proce benchmark. end dec error [60] Abort: [] Got completion error 12, vendor code=0, det rank=28 at line 553 in file ibv_channel_manager.c MPI proce terminated unexpectedly Exit code -5 ignaled from titan-ib Tet 1 Tet 2 Mvapich1 Informative Mvapich2 Informative Group 14: MPI Intel Tet #1 PingPing Tet pleted out error and PingPong Tet #2 All Tet failed Informative Dicuion: Tet #2 Reult The following error wa oberved in the log file for the AllGather 62 proce benchmark. mpiexec_janu.ofa (handle_ig_occurred 1982): job ending due to env var MPIEXEC_TIMEOUT=360 UNH-IOL OFA OFILG 17 Report Rev1.02

18 Tet #1 Default No Error Tet # 2 No Error ExitPath Tet # 3 IMB Tet failed Informative -IBV -rdma mode Tet # 4 IMB Tet failed Informative -IBV -rq mode Tet # 5 IMB Tet failed Informative -UDAPL -rdma mode Tet # 6 fork Tet failed Informative Group 15: MPI Hewlett-Packard -IBV -rdma mode Tet # 7 fork Tet failed Informative -IBV -rq mode Tet # 8 fork Tet failed Informative -UDAPL -rdma mode Tet # 9 ring2 - Tet failed Informative IBV -rdma mode Tet # 10 ring2 Tet failed Informative -IBV -rq mode Tet # 11 ring2 Tet failed Informative -UDAPL -rdma mode Dicuion: Tet #1 Default IBV wa auto-detected on all DUT. Dicuion: Tet #2 ExitPath Each DUT ha the ability to be torn down abnormally while remaining table. Dicuion: Tet # 3 IMB -IBV -rdma mode The following error wa oberved in the log file for the Exchange 62 proce benchmark. hpmpitet.x: Rank 0:51: MPI_Recv: IBV connection to 52 on card 0 i broken hpmpitet.x: Rank 0:51: MPI_Recv: ibv_poll_cq(): bad tatu 12 hpmpitet.x: Rank 0:51: MPI_Recv: elf calypo.ofa peer koll-ib.ofa (rank: 52) hpmpitet.x: Rank 0:51: MPI_Recv: error meage: tranport retry exceeded error hpmpitet.x: Rank 0:51: MPI_Recv: Internal MPI error MPI Application rank 51 exited before MPI_Finalize() tatu 16 Dicuion: Tet # 4 IMB -IBV -rq mode The following error wa oberved in the log file for the Allgatherv 62 proce benchmark. MPI Application rank 8 killed before MPI_Finalize() ignal 9 MPI Application rank 20 killed before MPI_Finalize() ignal 9 MPI Application rank 36 killed before MPI_Finalize() ignal 9 MPI Application rank 24 killed before MPI_Finalize() ignal 9 MPI Application rank 48 killed before MPI_Finalize() ignal 9 MPI Application rank 28 killed before MPI_Finalize() ignal 9 UNH-IOL OFA OFILG 18 Report Rev1.02

19 MPI Application rank 32 killed before MPI_Finalize() ignal 9 Dicuion: Tet # 5 IMB -UDAPL -rdma mode The following error wa oberved in the log file for the Allgatherv 62 proce benchmark. Memory allocation failed. code poition: et_buf 1. tried to alloc byte MPI Application rank 15 killed before MPI_Finalize() ignal 11 Memory allocation failed. code poition: et_buf 1. tried to alloc byte MPI Application rank 0 killed before MPI_Finalize() ignal 11 Memory allocation failed. code poition: et_buf 1. tried to alloc byte MPI Application rank 17 killed before MPI_Finalize() ignal 11 Dicuion: Tet # 6 fork -IBV -rdma mode No uable data in the log file. Dicuion: Tet # 7 fork -IBV -rq mode No uable data in the log file. Dicuion: Tet # 8 fork -UDAPL -rdma mode No uable data in the log file. Dicuion: Tet # 9 ring2 -IBV -rdma mode The following error wa oberved in the log file for the Ring 19 tet. phae 1 part 1 (loop(i){loop(r){iend;recv;wait}}) c-int chk [t:n/a] hpmpitet.x: Rank 0:56: MPI_Waitall: IBV connection to 9 on card 0 i broken hpmpitet.x: Rank 0:56: MPI_Waitall: ibv_poll_cq(): bad tatu 12 hpmpitet.x: Rank 0:57: MPI_Waitall: IBV connection to 10 on card 0 i broken hpmpitet.x: Rank 0:57: MPI_Waitall: ibv_poll_cq(): bad tatu 12 hpmpitet.x: Rank 0:56: MPI_Waitall: elf miranda.ofa peer mima-ib.ofa (rank: 9) hpmpitet.x: Rank 0:56: MPI_Waitall: error meage: tranport retry exceeded error hpmpitet.x: Rank 0:56: MPI_Waitall: Internal MPI error hpmpitet.x: Rank 0:57: MPI_Waitall: elf miranda.ofa peer mima-ib.ofa (rank: 10) hpmpitet.x: Rank 0:57: MPI_Waitall: error meage: tranport retry exceeded error hpmpitet.x: Rank 0:57: MPI_Waitall: Internal MPI error MPI Application rank 56 exited before MPI_Finalize() tatu 16 Dicuion: Tet # 10 ring2 -IBV -rq mode The following error wa oberved in the log file for the Ring 19 tet. phae 1 part 1 (loop(i){loop(r){iend;recv;wait}}) c-int chk [t:n/a] hpmpitet.x: Rank 0:57: MPI_Recv: IBV connection to 51 on card 0 i broken hpmpitet.x: Rank 0:57: MPI_Recv: ibv_poll_cq(): bad tatu 12 hpmpitet.x: Rank 0:57: MPI_Recv: elf miranda.ofa peer calypo-ib.ofa (rank: 51) hpmpitet.x: Rank 0:57: MPI_Recv: error meage: tranport retry exceeded error hpmpitet.x: Rank 0:57: MPI_Recv: Internal MPI error MPI Application rank 57 exited before MPI_Finalize() tatu 16 Dicuion: Tet # 11 ring2 -UDAPL -rdma mode The following error wa oberved in the log file for the Ring 19 tet. phae 1 part 1 (loop(i){loop(r){iend;recv;wait}}) c-int chk [t:n/a] miranda.ofa:5924: DTO completion ERR: tatu 12, op OP_RDMA_WRITE, vendor_err 0x hpmpitet.x: Rank 0:56: MPI_Waitall: dat_evd_wait: bad tatu: 8 hpmpitet.x: Rank 0:56: MPI_Waitall: Internal MPI error MPI Application rank 56 exited before MPI_Finalize() tatu 16 UNH-IOL OFA OFILG 19 Report Rev1.02

20 Tet #1 Tet pleted out error Group 16: MPI Open Tet #2 Tet failed Informative Dicuion: Tet #2 Reult The following error wa oberved in the log file for the Sendrecv 62 proce benchmark. mpirun noticed that job rank 0 PID 6489 on node atla-ib exited on ignal 15 (Terminated). 60 additional procee aborted (not hown) Group 17: TI udapl Tet #1 Automated cript ued Dicuion: Tet Reult The automated tet cript provided in the tet plan wa ued. Thi cript' output wa captured to a log file and pared. No error were found. Group 19: IB FibreChannel Gateway Tet #1-10 Not applicable to DUT Not Applicable Dicuion: Tet Reult Group 20: IB Ethernet Gateway Tet #1-10 Not applicable to DUT Not Applicable Dicuion: Tet Reult Group 21: IB Reliable Datagram Socket Tet #1-10 Not teted Not Teted Dicuion: Tet Reult Not teted due to time contraint Group 22-23: TI Baic RDMA Tet #1-10 Not teted Not Teted Interoperability Dicuion: Tet Reult Not teted due to time contraint Group 24-25: TI RDMA Operation over Tet #1-10 Not teted Not Teted Interconnect ponent Dicuion: Tet Reult Not teted due to time contraint UNH-IOL OFA OFILG 20 Report Rev1.02