Xpander Rack Mount 2 Gen 3 HPC Version User Guide

Transcription

1 Xpander Rak Mount 2 Gen 3 HPC Version User Guide Xpander Rak Mount 2 is a 2U rak mount PCI Express (PCIe) expansion enlosure that enables onnetion of two passively-ooled aelerators to a host omputer. The host must have a PCIe x16 expansion slot available for the host interfae ontroller (HIC). Rak Mount 2 provides two PCIe x16 slots. Connet a HIC in one host omputer to one Xpander. With a HIC installed, a single host omputer an support two double-wide, passively-ooled aelerators. It onnets to one HIC in the host omputer. Rak Mount 2 provides two PCIe Generation 3 (Gen 3) x16 slots for graphial proessing units (GPUs) or other aelerators. For example, use NVIDIA Compute Unified Devie Arhiteture (CUDA) -enabled GPUs or Intel Xeon Phi oproessors for high-performane omputing (HPC). A HIC in a host omputer s PCIe Gen 3.0 x16 slot provides the full onnetion transfer rate with Xpander. For example, the HIC in a PCIe Gen 3.0 x16 slot provides a theoretial transfer rate of 8.0Gbs x16 = 128Gbps. Xpander Rak Mount 2 Gen 3 Tehnial Speifiations Expansion slots, x16 std PCI Express Gen 3.0 x16 mehanial / eletrial Number of slots 2 slots in one mainboard Form fator support 2 double-wide, full-length, full-height aelerators Conneting able External PCI Express x16, 136-pin able Cable / onnetor x16 / 136-pin Length, standard 2.00meters (78.74inhes or 6.56feet) Length, optional 1.00meter (39.37inhes or 3.28feet) Connetor duty yle 250 mating yles maximum per onnetor PCIe x16 HIC Host interfae ontroller, install in a PCIe x16 slot PCIe Gen 3.0 slot Provides a theoretial 128Gbps data transfer rate PCIe Gen 2.0 slot Provides a theoretial 80Gbps data transfer rate PCIe Gen 1.0 slot Provides a theoretial 40Gbps data transfer rate Front panel LCD Main menu = left button; sub-menu = right button Cubix Corporation MIB assembly and revision; management firmware revision Power supply status Standard main power Temperature status 3 sensor readouts in degrees Centigrade (C) Fan status Fan status, voltage perent (%) speed PCIe link status x16 able onneted; 8.0Gbps x16 link speed x16 able Conneted or not onneted status PCIe Gen 3.0, 8.0Gbps x16 = 128.0Gbps Slot 1-2 x16 transfer PCIe Gen 2.0, 5.0Gbps x16 = 80.0Gbps rate, theoretial PCIe Gen 1.0, 2.5Gpbs x16 = 40.0Gbps Cubix Corporation 2800 Lokheed Way Carson City, NV Page 1 Sales sales@ubix.om xpander-rakmount2-g

2 Rear panel 1 power input reeptale and 1 x PCIe x16 able reeptale Cooling fans 2 x fans run at 102fm (ubi feet per minute) maximum Air flow diretion Front to rear Xpander power up Fan is full on for 10s (seonds) Fan speed, minimum 50% (6Volts) at 25 degrees C or less Fan speed, maximum 100% (12Volts) at 50 degrees C or more Fan speed, range 25 degrees C to 50 degrees C in gradual, linear progression Normal operation D1 LED = 1 pulse of 10ms (milliseond) on for every 700ms off Error indiation D1 LED = 4 pulses of 10ms on for every 700ms off Error onditions [1] No fan tahometer or [2] no temperature sensor onneted AC power supply Single-phase, auto-ranging, VAC, Amps, 50 / 60Hz; total output power is not to exeed 800Watts Operating environment 0 to 30 Celsius (32 to 86 Fahrenheit) temperature 5% to 80% non-ondensing humidity Dimensions (2U) 3.50" H x 19.00" W x 16.00" D (8.89m H x 48.26m W x 40.64m D) Weight 18.50lbs (8.39kg) with no GPUs Warranty / period Parts & labor return to manufaturer / 2 years Ext. warranty period Optional 3 years Inluded Rak Mount 2 inludes two ooling fans, an 800W power supply and a power ord as well as one 2-slot main board with risers, one PCIe x16 HIC and one external PCIe x16 able. It does not inlude installation media beause installation requires no driver or other software. The only drivers you install are those for the devies you install. Cubix provides a standard, two-year warranty with the purhase of eah Rak Mount 2. Optional The standard external PCIe able length is 2meter. Optional 1m PCIe able is also available. The maximum duty yle for PCIe ables is 250 mating yles. Cubix offers a third-year warranty option on Xpander produts for a total of three years. Quik Start Without Detail Ignore this step if you wish. Install a GPU in the host system and install devie drivers. One you have verified operation, shut down, move the GPU to Xpander, install the HIC in the same slot within the host omputer, onnet Xpander to the HIC using the able provided, onnet power to Xpander Rak Mount 2. Boot the host omputer. If the host operating system (OS) is plug-n-play, the OS detets the hardware devies and loads devie drivers. Start-up / Test / Installation Proedure With Detail Chek your host omputer motherboard vendor s Website for devie driver and firmware updates, espeially those that may affet multiple aelerators. During this proedure, plae the Rak Mount 2 on a stable surfae. Use a Philips #2 srewdriver for this proedure. Do not install graphis or other ontrollers until instruted to do so within this proedure. Cubix reommends that you do not install your first Rak Mount 2 enlosure in the rak mount abinet until you have finished. 1. Swith off AC power to the host omputer. Leave the power ord onneted. 2. Swith off AC power to Xpander Rak Mount Install the x16 HIC in a PCIe x16 slot within the host omputer. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 2 Sales sales@ubix.om xpander-rakmount2-g

3 4. Remove the srews holding in plae the Xpander top over, remove the top over and set it aside. 5. Connet the external PCIe able from the Xpander to the HIC in the host omputer. Make sure the able onnetor seats firmly in the HIC port. You will feel the able onnetor lik into plae. See the image of a properly-seated onnetor at the right. 6. Connet a power ord to the reeptale at the Xpander enlosure rear. 7. Power on the host omputer. Xpander also powers on. If you have properly seated the HIC in the host PCIe slot, you will see the 8 LEDs at the top edge of the HIC light steady green. 8. Chek the mainboard Cable (P0) LED, whih should light steady green. This indiates that the onnetion between the HIC and Xpander mainboard is good. The 5VSTBY LED should also light steady green. See the image at the top of the next page. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 3 Sales sales@ubix.om xpander-rakmount2-g

4 Confirm that interior Cable LED (P0) is lit green. The unit fans are spinning. Xpander is funtioning properly one the Cable LED (P0) lights green. Proeed to install adapters. 9. Swith off AC power to the host omputer. Leave the power ord and the external PCIe able to the host omputer onneted. 10. Swith off power and remove the power ord from Xpander. Warning Before installing or removing ards in Xpander, swith off AC power and wait for the mainboard 5V Standby LED to power off. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 4 Sales sales@ubix.om xpander-rakmount2-g

5 11. Remove the two srews holding the end-braket retaining panel from Xpander rear and remove the retaining panel. See the image below. 12. Install the aelerators in the appropriate riser slots. Make sure the baffle plate remains in plae. The baffle plate is required for passively-ooled aelerators like NVIDIA or AMD GPUs or Intel Xeon Phi oproessors. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 5 Sales sales@ubix.om xpander-rakmount2-g

6 13. Install blank end-brakets, if required. Find 3 x blank end-brakets in Xpander pakaging. 13. Connet PCIe auxiliary (aux) power ables to the aelerators. Aux power onnetors are loated at the rear of eah aelerator. Rak Mount 2 provides three sets of auxiliary power onnetors. See the image of aux power onnetors at the right. 14. Replae the two srews holding the end-braket retaining panel on Xpander rear. 15. Install the hex-head srews that seat the end-brakets in plae. 16. Connet GPU video port(s), if needed, to monitors. This step is espeially important for Linux, beause X Windows rashes if you have not onneted video ports to monitors. If you annot onnet monitors to installed GPU video ports, you an edit the Devie setion of /et/x11/xorg.onf to define whih GPU X Windows should use as the primary display devie. For example: Setion "Devie" Identifier "Devie0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "Quadro M6000" BusID "PCI:8:0:0" EndSetion To determine the BusID for the GPU you wish to use as the primary display, run the sanpi tool disussed in Step Install the single-wide ards at the far right faing the rear of Xpander. 18. Power up the host omputer and Xpander to onfirm that the appropriate Xpander slot link LEDs are lit green, whih onfirms that you have installed the ontroller(s) orretly in eah slot. Note Slot LEDs should light steady green. If they are flashing, it may be beause the GPUs in the slots do not have a proessing load. Apply a proessing load on the GPUs. If the slot LEDs still flash green, hek the vendor s link for a driver update. 19. Confirm that your operating system (OS) has deteted the new ontroller(s). In a PC running Mirosoft Windows, hek Windows Devie Manager. Within Windows, you an also run GPU-Z, a utility available for free download here. In a PC running Linux, run the following: root# > XFree86 sanpi Read more about sanpi here. 20. If all the Cable Link and slot LEDs are green but your OS does not detet the GPUs, shut down the OS graefully, power down Xpander and reseat the GPUs in eah PCIe slot. Make sure you seat eah GPU firmly in its slot and that you seat all aux power onnetors firmly. 21. Power up the host omputer and Xpander. 22. Install the appropriate graphis or other ontroller driver(s) using the vendor s proedure. 23. Replae the Xpander front top over and replae the srews. Warning Replae the over as soon as possible to allow for proper ooling. Do not run Xpander without the over for more than a few minutes. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 6 Sales sales@ubix.om xpander-rakmount2-g

7 Normal Power On/Off Proedure 1. With Xpander onneted to the host omputer and running, shut down the host normally. 2. With Xpander onneted to the host omputer and with power onneted to both, swith on Xpander first. Swith on the onneted host omputer; both will boot. Install Xpander in a Cabinet For this proedure, use a Philips #2 srewdriver, a flat-blade srewdriver and a 3/8 wrenh or soket and drive. When lifting the Xpander into plae, ask for assistane. Mount it lose enough to the host omputer so that the external PCIe able will reah. Mount Xpander either immediately above or immediately below the host omputer(s) in the abinet. Within pakaging, you will find four brakets that mount within the four orners of your rak mount abinet, srews, nuts and G-nuts for use in mounting Xpander within a abinet. See the image at the right. 1. Shut down the host and Xpander normally and disonnet all data and power ables onneted to Xpander. 2. Find in pakaging the rakmount slide rails with a bag of 8 x mounting srews taped to one of the rails. 3. Remove the inner slide rail, or slide mount, from the outer rail and mount it to Xpander using the Xpander slide mounting holes. To release eah outer slide mount, slide it toward the rear of the enlosure until it stops. Press the ath to release and remove the outer slide mount as you an see in the image at the right. 4. Determine where to mount the Xpander slide mounts in your abinet. 5. Mount the front and rear mounting brakets (Part # S32760A) to eah slider assembly with the bolts and nuts provided in the pakaging. 6. Repeat the proess for the other front and two rear mounting brakets and slider assembly. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 7 Sales sales@ubix.om xpander-rakmount2-g

8 7. Tightly seure the srews holding the sliders to the front and two rear mounting brakets on eah side of the abinet. 8. Chek the two rails on eah side and hek from front to rear to make sure that you have mounted the rails so that they are level within the abinet. 9. Lift the Xpander, line up the inside slider with the slider assembly mounted in your abinet and arefully slide it into plae. Warning Before lifting Xpander into plae, request assistane. 10. Try sliding the Xpander in and out slightly to make sure it moves freely. 11. Seure Xpander into plae within the abinet using the front-panel srews provided. Use the Philips #2 srewdriver. 12. Connet the PCIe x16 external able to the HIC in the host omputer. 13. Connet power to Xpander. Aess Rak Mount 2 Interior For this proedure, use a Philips #2 srewdriver. For aessing Xpander to add, move or replae GPUs or other adapters, use the following proedure. 1. Shut down graefully the host omputer onneted to Xpander. 2. Swith off power and disonnet the power ord. 3. Remove the two front-panel srews that seure Rak Mount 2 to the rakmount abinet. 4. Gently pull the Xpander toward you. 5. Remove the srews holding in plae the Xpander front top over, remove the top over and set it aside. 6. Make sure the 5V Standby LED in Xpander is off. Warning Before installing or removing ards from GPU-Xpander, swith off AC power and wait for the 5V Standby LED to power off. 7. Remove the srews holding eah of the end-brakets. 8. Remove the two srews holding the end-braket retaining panel from Xpander rear and remove the retaining panel. 9. Remove and replae the aelerators in the appropriate riser slots. Make sure the baffle plate remains in plae. The baffle plate is required for passively-ooled aelerators like NVIDIA or AMD GPUs or Intel Xeon Phi oproessors. 14. Install blank end-brakets, if required. Find 3 x blank end-brakets in Xpander pakaging. 24. Connet PCIe auxiliary (aux) power ables to the aelerators. Aux power onnetors are loated at the rear of eah aelerator. Rak Mount 2 provides three sets of auxiliary power onnetors. See the image of aux power onnetors at the right. 25. Replae the two srews holding the end-braket retaining panel on Xpander rear. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 8 Sales sales@ubix.om xpander-rakmount2-g

9 10. Replae the Xpander top over and srews. 11. Gently slide Xpander bak in plae. 12. Seure Xpander into plae within the abinet using the front-panel srews provided. Use the Philips #2 srewdriver. 13. Connet the PCIe x16 external able to the HIC in the host omputer. 14. Connet power to Xpander. Front Panel LCD Xpander Rakmount 2 front panel LCD provides enlosure details and status. Beneath the LCD are two buttons: left button srolls through main-menu and right button srolls through sub-menu. When you first onnet power, Xpander omes up in standby mode. The LCD displays Cubix on the top line and Corporation on the bottom line. This is the Start menu. Aess all other main menu items by pressing the left button. The upper right orner of the LCD shows a flashing asterisk *. This indiates that the on-board Miro-Controller Unit (MCU) is running properly. If the asterisk is not flashing or the LCD is not responding, reset the MCU by pressing and holding the left button for 3 seonds. A orresponding HB (heartbeat) LED on the mainboard also shows the MCU is working. When a fault ours, the LCD will jump to that fault and the LCD baklight will flash on / off. Press the left button to aknowledge the fault and the LCD will stop flashing and enable normal operation. The LCD will ontinue to display the fault at the appliable sub-menu until you lear the fault. Start Menu Menu Sub-menu Sub-menu Sub-menu Cubix Cubix Corporation A09150 Rev 0B 0A Main menu: Cubix Corporation Sub-menu: PCB Assembly Board Number EEPROM Revision Board Revision Sub-menu: MCU (Miro-Controller Unit) Firmware Release Number Release Date Sub-menu: Cubix Serial Number MCU Firmware Rel Serial Number Cubix Corporation 2800 Lokheed Way Carson City, NV Page 9 Sales sales@ubix.om xpander-rakmount2-g

10 Power Supply Status Menu Sub-menu Sub-menu Power Supply Power Supply Power Supply Status PWR ON-Good RPS ON-Good Menu: Power Supply Status Sub-Menu: Power Supply Status: PWR Standby = Main power is off; system is in standby PWR ON-Good = Main power is on; +3.3V, +5V, +12, -12V are within spe. PWR ON-FAULT = Main power is on; +3.3V, +5V, +12V, or -12V is out of spe. Temperature Status Menu Sub-menu Temperature Status Menu: Temperature Status Sub-Menu: Display the temperature of the 3 temperature sensors in degrees Celsius. Note: If a temperature sensor fails, FLT (fault) appears under the failed sensor. Fan Status Tmp1 Tmp2 Tmp3 24C 25C 26C Menu Sub-menu Sub-menu Fan Status Fan 1 Running 7V 70% Fan 2 Running 7V 70% Menu: Fan Status Sub-Menu: Indiates fan status for fans 1-2 Note: Fan speed is ontrolled by adjusting fan voltage based on sensed system temperature. Fan voltage adjusts from 6V min (25deg C or lower) at 50% to 12V max (50deg C or higher) at 100%. Fan voltage is the same for all fans. Status: PWR OFF is 0V at 00%; system is in standby. Status: Running aav at pp%; fan is good; aa = fan voltage and pp = fan power perent (%). Status: STOPPED is 12V at 100%; fan fault: fan voltage is set to full on. Status: NO FAN aav at pp%; Xpander an be onfigured with 1, 2, 3, or 4 fans. No fan means fan is not installed. PCIe Link Status Menu Sub-menu Sub-menu Sub-menu PCIe Link Status PCIe X16 Cable Conneted PCIe Cable 8.0Gb/s X16 PCIe Slot n 8.0Gb/s X8 Menu: PCIe Link Status Sub-Menu: Indiates x16 PCIe able installation status Conneted x16 able is properly onneted to the HIC NOT-Conneted x16 able is not properly onneted to the HIC Sub-Menu: x16 PCIe able link speed and link width. Speed and width depend on the host apability. Speed: 8.0Gb/s Cubix Corporation 2800 Lokheed Way Carson City, NV Page 10 Sales sales@ubix.om xpander-rakmount2-g

11 Width: x1, x2, x4, x8, x16 and NO Link when the host did not suessfully link or Xpander is in standby power. Sub-Menu: Link speed and link width for slots 1-4. Speed: 8.0Gb/s Slot 1-4 width: x1, x2, x4, x8, and NO Link when the slot is empty, unsuessful link, or Xpander is in standby power. LCD Baklight Control Press right button to toggle the LCD baklight On/Off. Fan Control Xpander fan ontrol runs the fan at full on for 10s (seonds) at power up. After 10s, fan ontrol starts and runs at minimum fan speed for up to 25C degrees. From 25C to 50C, fan speed inreases in a gradual, linear progression. At 50C and up, fan ontrol runs at maximum. During normal operation, the fan ontrol D1 LED pulses on for 10ms (milliseonds) every 700ms off. When an error ondition exists, the fan ontrol D1 LED pulses on 4 times at 10ms eah every 700ms. Two error onditions are possible: [1] no fan tahometer or [2] no temperature sensor is onneted. See the fan ontrol illustration below. Shipping Proedure Before shipping Xpander, remove any installed graphis or other ontroller(s) using the following proedure: 1. Swith off the host omputer and disonnet power. 2. Disonnet Xpander from the HIC. 3. Gently pull Xpander toward you. 4. Lift Xpander down from the abinet and set it on a stable surfae. Warning Before lifting Xpander out of a abinet, ask for assistane. 5. Remove the srews holding in plae the Xpander front top over, remove the top over and set it aside. 6. Remove the srews holding eah of the end-brakets. 7. Remove the two srews holding the end-braket retaining panel from Xpander rear and remove the retaining panel. 8. Remove the GPUs and other PCIe ards and blank end-brakets. 9. Replae the two srews seuring the end-braket retaining panel to Xpander rear. 10. Replae the srews for holding eah of the end-brakets. Warning Remove GPUs or other adapters before shipping Xpander. 11. Pak the graphis ards or other ontrollers within the boxes in whih you reeived them. 12. Replae the Xpander front top over, replae the srews and remove the power ords. 13. Remove the HIC(s) from the host omputer. 14. Within the original pakaging, pak the Xpander, the power ords, the HIC(s) and the boxed graphis ards. Inlude a opy of these instrutions. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 11 Sales sales@ubix.om xpander-rakmount2-g

12 Troubleshooting Proedure The front-panel LCD asterisk does not flash. Reset the LCD s MCU by pressing and holding the left button for 3 seonds while Xpander is running. The HB LED inside Xpander flashes green. This is normal; it indiates that the on-board MCU is running properly. Cable LED inside Xpander does not light green. Chek the following: Confirm that the HIC is firmly seated in a PCIe slot within the host omputer. Confirm that the able onnetor is firmly seated HIC onnetor port. Cable LED inside Xpander flashes green even though I have a PCIe x16 HIC and able. The internal Cable LED should light steady green. If the Cable LED flashes, it is transferring data at a redued bandwidth beause you installed the PCIe x16 HIC in a Gen 1 slot or a PCIe x8 (eletrial) slot. If your HIC is PCIe x8, you installed it in a Gen 1 slot. The HIC operates at full bandwidth when installed in a PCIe Gen 2 slot. Slot LED inside Xpander does not light green. Shut down the OS graefully, power down Xpander and reseat the GPUs in eah PCIe slot. Make sure that you seat eah GPU firmly in its slot and that you onnet all aux power onnetors firmly. Slot LED inside Xpander flashes green even though I installed a PCIe x16 Gen 3 GPU or ard. Slot LEDs should light steady green. If they are flashing, it may be beause the GPUs in the slots do not have a proessing load. Apply a proessing load on the GPUs. If the slot LEDs still flash green, hek the vendor s link for a driver update. All Cable and Slot LEDs are lit green but your OS does not detet the GPUs. Shut down the OS graefully, power down Xpander and reseat the GPUs in eah PCIe slot. Make sure that you seat eah GPU firmly in its slot and that you onnet all aux power onnetors firmly. The front-panel LCD sreen baklight is flashing on/off, whih indiates a fault. Aknowledge the fault by pressing the Menu button, whih will stop the LCD flashing and the LCD will remain at the fault. After aknowledging a fault, the front-panel LCD sreen baklight ontinues flashing on/off, whih indiates two or more faults. If multiple faults have ourred, when you aknowledge the first fault, the LCD will jump to the next fault and the LCD baklight will one again flash on/off. This ontinues until you lear all faults. Aknowledge the fault by pressing the Menu button, whih will either stop the LCD flashing and the LCD will remain at the fault, or the LCD will jump to the next fault and the LCD baklight will one again flash on/off. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 12 Sales sales@ubix.om xpander-rakmount2-g

13 The front-panel LCD sreen either does not respond or is blank even though Xpander is operating. Press and hold the MIB Selet button to reset the Miro-Controller Unit (MCU). The front-panel LCD sreen either does not respond or is blank for a partiular MIB even though Xpander and the MIB are both operating. Press and hold the Menu (left) button for 3 seonds to reset the Miro-Controller Unit (MCU). If the Xpander is running, the Xpander fans will turn on full speed for 10 seonds and then slow down to the operating speed set by the MIB loal temperatures. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 13 Sales sales@ubix.om xpander-rakmount2-g