Xpander Rack Mount 8 5U Gen 3 with Redundant Power [Part # XPRMG3-81A5URP] User Guide

Transcription

1 Xpander Rak Mount 8 5U Gen 3 with Redundant Power [Part # XPRMG3-81A5URP] User Guide Xpander Rak Mount 8 5U Gen 3 with Redundant Power (RP) supplies is a rak mount PCI Express (PCIe) expansion enlosure that enables onnetion of 8 double-wide graphis or other ontrollers with rear-onneted auxiliary power ables to a host omputer. Rak Mount 8 5U RPS supports graphial proessing units (GPUs) with ative ooling and rear- or top-onneted auxiliary power ables. The host omputer must have one PCIe x16 slot (preferably PCIe Gen 3.0) available for the host interfae ontroller (HIC). Rak Mount 8 5U RP provides one Main Interfae Board (MIB) onneted to a HIC in a host omputer with a PCIe x16 able. The MIB features eight PCIe x16 slots to support a total of eight double-wide GPUs like NVIDIA Quadro / GeFore GTX Series or AMD Firepro W9100 Series. Contat Cubix Customer Servie (ustomerservie@ubix.om) if you have questions. The MIB onnets via a PCIe x16 able to a single HIC in a host omputer. Rak Mount 8 provides one set of eight PCIe x16 slots for GPUs for use with high-performane omputing (HPC) and other Compute Unified Devie Arhiteture- (CUDA-) or OpenCL / OpenGL-enabled tasks. With 8 x Intel Xeon Phi oproessors that feature ative ooling, it supports parallel proessing appliations. Cubix designed this Rakmount 8 with a single main board speifially for appliations that require the fastest peer-to-peer (P2P) transfer rates possible. For example, Cubix has measured P2P transfer rates of 12.5GBytes/seond simultaneously in both diretions among 8 x NVIDIA GeFore GTX Series GPUs. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 1

2 Cubix Corporation 2800 Lokheed Way Carson City, NV Page 2

3 Xpander Rak Mount 8 5U RPS Tehnial Speifiations Expansion slots PCI Express Generation 3.0 x16 Number of slots (std) 8 slots in 1 x MIB for a total of 8 slots Form fator support 8 full-height, full-length, double-wide ontrollers or adapters in 1 MIB Auxiliary power able support Rear-onneted or top-onneted auxiliary power ables 8 x sets of 6-pin + 8-pin (6P + 8P) Conneting able External PCI Express able Connetor type x16, 136-pin Length, standard 2.00meters (78.7inhes or 6.6feet) total length 1.16meters (45.6inhes or 3.8feet) usable length Connetor duty yle 250 mating yles maximum per onnetor PCIe x16 HIC Install in a PCIe x16 slot of the host omputer PCIe Gen 3 slot in host Provides a theoretial 128.0Gbps data transfer rate PCIe Gen 2 slot in host Provides a theoretial 80.0Gbps data transfer rate PCIe Gen 1 slot in host Provides a theoretial 40.0Gbps data transfer rate Front panel LCD Main menu = left button Sub-menu = right button (See detail on page 9.) Cubix Corporation MIB assembly and revision; management firmware revision Power supply status Redundant power Temperature status 3 sensor readouts in degrees Centigrade (C) Fan status Fan status, voltage perent (%) speed PCIe link status x16 able onneted; 8.0Gbps, 5.0Gbps or 2.5Gbps x16 link speed x16 able Conneted or not onneted status PCIe Gen 3.0Gbps x16 = 128.0Gbps Slot 1-8 x16 transfer PCIe Gen 25.0Gbps x16 = 80.0Gbps rate, theoretial PCIe Gen 12.5Gpbs x16 = 40.0Gbps LCD panel baklight Toggle on / off using right button Rear panel Power supply module input reeptales Adapter ports If available, on installed ontrollers or adapters Reeptale x AC reeptale for eah power supply module Cooling fans 4 fans run at 112fm (ubi feet per minute) maximum Xpander power up Fans are full on for 10s (seonds) Fan speed, minimum 25C degrees Fan speed, maximum 50C degrees Fan speed, in range 25C degrees to 50C degrees in gradual, linear progression AC power supply, 1 MIB 3,000W power with hot-plug modules: eah module provides autoranging Amps Auxiliary power ables Standard: 8 x sets of 6-pin + 8-pin (6P + 8P) Operating environment 0 to 30 Celsius (32 to 86 Fahrenheit) temperature 5% to 80% non-ondensing humidity Altitude Maximum Operating 10,000ft (3,050meters) Storage 50,000ft (15,240meters) Dimensions 8.75" H x 19.00" W x 26.00" D (22.22m H x 48.26m W x 66.04m D) Weight 52.00lbs (24.50kg) Warranty / period Parts & labor return to manufaturer / 2 year Ext. warranty period Optional 3 years Cubix Corporation 2800 Lokheed Way Carson City, NV Page 3

4 Inluded Xpander Rakmount 8 inludes four ooling fans, one 3,000W power supply with four redundant modules and four power ords as well as one MIB, one PCIe x16 HIC and one external PCIe x16 able. It does not inlude installation media beause installation requires no driver or other software. The only drivers you install are for the devies you install in Xpander. Quik Start without Detail Install a GPU in the host system and install the devie driver. One you have verified operation, shut down, move the GPU to Xpander, install the HIC in the same slot within the host omputer, onnet Xpander to the HIC using the able provided and onnet power to Xpander Rakmount 8. Boot the host omputer. If the host operating system (OS) is plug-n-play, the OS detets the GPUs and loads devie drivers. Start-up / Installation / Test Proedure with Detail Eah Rak Mount 8 provides one MIB with eight PCIe x16 slots. The MIB supports only one host omputer. During this proedure, plae the Rak Mount 8 enlosure on a stable surfae. Use a Philips #2 srewdriver for this proedure. Make sure all AC power to Xpander is disonneted. Do not install graphis or other ontrollers until instruted to do so within this proedure. 1. Swith off AC power from the host omputer. Leave the power ord to the host onneted. 2. Install the standard x16 HIC in a PCIe x16 slot within the host omputer. Preferably, the HIC should be in a PCIe Gen 3.0 x16 slot. 3. Connet the external PCIe able from its MIB inside the Rak Mount 8 enlosure to the HIC port. Make sure the onnetor snaps into plae within the HIC port as shown in the illustration at the right. Cubix inludes a lowprofile end braket for the HIC, if required. 4. Remove the srews holding in plae the Rak Mount 8 top over and remove the top over. 5. Connet a power ord to the reeptale in eah of the four power supply modules at the Rak Mount 8 enlosure rear. 6. Power on the host omputer. Rak Mount 8 enlosure also powers on. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 4

5 7. Confirm that the PEX1 -Cable Link LED lights green for the MIB inside the Rak Mount 8 enlosure. Xpander MIB is funtioning properly one the PEX1-Cable LED lights green. Proeed to the next step to install adapters. 8. Swith off AC power to the host omputer. Leave the power ords and the external PCIe ables to the host omputer onneted. 9. Disonnet AC power from Xpander. 10. Remove the two srews at eah end of the rear ventilation panel above the rear Input / Output (I/O) panel, remove the ventilation panel and set it aside temporarily. See the image with the rear ventilation panel at the right. 11. Remove the srews for holding down the PCIe ard end brakets. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 5

6 12. Install the graphis or other ontroller(s) in the Rak Mount 8 PCIe slot(s) and seure eah with the srews provided in the top of the I/O panel. Before installing or removing ards in the Xpander Rak Mount 8, remove AC power and wait for the 5V Standby LED to power off. 13. Connet PCIe auxiliary (aux) power ables. The power supply with redundant modules inside Rak Mount 8 enlosure provides eight sets of aux power onnetors inluding 6P + 8P (6-pin and 8-pin) onnetors. See the image at the right. 14. Make sure blank end brakets (L-brakets) are installed for eah slot without an aelerator. Make sure that blank end brakets (L-brakets) are installed for eah slot that has no PCIe ard. Blank end brakets are required to assure proper ooling of the GPUs. 14. Replae the ventilation panel and tighten the two srews holding it in plae. See the image on the previous page. 15. Connet GPU video port(s), if present, to monitors. This step is espeially important for Linux, beause X Windows will rash if you have not onneted video ports to monitors. If you annot onnet monitors to installed GPU video ports, you an edit the Devie setion of /et/x11/xorg.onf to define whih GPU X Windows should use as the primary display devie. For example: Setion "Devie" Identifier "Devie0" Driver "nvidia" VendorName "NVIDIA Corporation" BoardName "Quadro M6000" BusID "PCI:8:0:0" EndSetion To determine the BusID for the GPU you wish to use as the primary display, run the sanpi tool disussed in Step Power up the omputer(s) and the Rak Mount 8 to onfirm that the appropriate slot link LEDs are lit green, whih onfirms that you have installed a ontroller orretly in eah slot. Refer to the appropriate image above to identify slot LEDs. 17. Confirm that your operating system (OS) has deteted the new ontroller(s). For instane, in a PC running Mirosoft Windows, hek Windows Devie Manager. Within Windows, you an also run GPU-Z, a utility available for free download here. In a PC running Linux, run the following: root# > XFree86 sanpi Read more about sanpi here. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 6

7 18. Connet power to Xpander and boot up the host omputer.install the appropriate graphis or other ontroller driver(s) using the vendor s proedure: e.g., NVIDIA Quadro / GeFore GTX Series. 19. Replae the Rak Mount 8 front top overs and replae the srews holding it in plae. Replae the over as soon as possible to allow for proper ooling. Do not run Rak Mount 8 without the over for more than a few minutes. Normal Power On/Off Proedure 1. With Rak Mount 8 onneted to the host omputer and running, shut down the host omputer normally. 2. With Rak Mount 8 onneted to the host omputer and powered down, swith on the onneted host omputer; both Xpander and host will boot. 3. To run the host PC without Xpander onneted, power down the host, remove power from Xpander and boot the host. When you are ready to run Xpander again, power down the host, onnet power to Xpander and boot the host. Install Xpander in a Cabinet For this proedure, use a Philips #2 srewdriver, a flat-blade srewdriver and a 3/8 wrenh or soket and drive. When lifting the Xpander into plae, ask for assistane. Mount it lose enough to the host omputer so that the external PCIe able will reah. Mount Xpander either immediately above or immediately below the host omputer(s) in the abinet. Within pakaging, you will find four brakets that mount within the four orners of your rak mount abinet, srews, nuts and G-nuts for use in mounting Xpander within a abinet. See the image on the previous page. 1. Shut down the host and Xpander normally and disonnet all data and power ables onneted to Xpander. 2. Find in pakaging the rakmount slide rails with a bag of 8 x mounting srews taped to one of the rails. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 7

8 3. Remove the inner slide rail, or slide mount, from the outer rail and mount it to Xpander using the Xpander slide mounting holes. To release eah outer slide mount, slide it toward the rear of the enlosure until it stops. Press the ath to release and remove the outer slide mount as you an see in the image at the right. 4. Determine where to mount the Xpander slide mounts in your abinet. 5. Mount the front and rear mounting brakets (Part # S32760A) to eah slider assembly with the bolts and nuts provided in the pakaging. 6. Repeat the proess for the other front and two rear mounting brakets and slider assembly. 7. Tightly seure the srews holding the sliders to the front and two rear mounting brakets on eah side of the abinet. 8. Chek the two rails on eah side and hek from front to rear to make sure that you have mounted the rails so that they are level within the abinet. 9. Lift the Xpander, line up the inside slider with the slider assembly mounted in your abinet and arefully slide it into plae. Before lifting Xpander into plae, ask for assistane. 10. Try sliding the Xpander in and out slightly to make sure it moves freely. 11. Seure Xpander into plae within the abinet using the front-panel srews provided. 12. Connet the PCIe x16 external able to the HIC in the host omputer. 13. Connet power to Xpander. Use this step if you need extender brakets for mounting in a 1000mm deep abinet. 1. Mount the extender braket on the outer slide mount behind the front braket (S32760 A). Mount the rear braket (S32760 A) to the rear end of the extender braket. Leave the nuts loose so you an slide the rear braket along the extension braket to Cubix Corporation 2800 Lokheed Way Carson City, NV Page 8

9 mount on the abinet s vertial rails and fit inside the abinet. Chek to make sure the mounting brakets and slide rails are level from front to rear before tightening the srews and nuts. Align the front of the slider with the front of the abinet. 2. If you do not need the extender brakets, mount eah of the two sliders into the front and rear mounting brakets (Part # S18500A) and slider assembly with the bolts and G-nuts provided in the pakaging. Use the Philips #2 srewdriver and the 3/8 wrenh or soket to seure snugly but not tightly eah slider to the front and rear mounting brakets. Chek to make sure the mounting brakets and slide rails are level from front to rear before tightening the srews and nuts. Align the front of the slider with the front of the abinet. 3. Repeat the proess for the other two front and two rear mounting brakets (as well as the extender braket if needed) and slider assembly on the right side of the abinet. 4. Chek to make sure that the slide rails are level, both from front to bak and from right to left. One you re sure the slide rails are level, you re ready to tighten the mounting srews. See the image at the right. 5. Tightly seure the srews holding the sliders to the front and two rear mounting brakets on eah side of the abinet. 6. Extend the slide rails almost all the way out the front of the abinet. 7. Lift the Rak Mount 8 enlosure, line it up arefully and slide it into plae. Before lifting the Rak Mount 8 enlosure into plae, ask for assistane. 8. Try sliding the Rak Mount 8 in and out slightly to make sure it moves freely. 9. Connet the AC power ords into the Rak Mount 8 rear reeptales. 10. Make sure the host omputer(s) with the host adapters installed are powered down. 11. Connet the PCIe x16 external ables to the host adapters in the host omputer(s). 12. Power up the host omputer(s) with the host adapters installed. Aess Xpander Rak Mount 8 Interior For this proedure, use a Philips #2 srewdriver. For aessing Rak Mount 8 to add, move or replae GPUs or other adapters, use the following proedure. 1. Shut down graefully the host omputer(s) onneted to the MIB(s) you intend to aess. 2. Remove the power ord(s) onneted to the power supply modules. 3. Remove the seure srews holding the Rak Mount 8 in the abinet. 4. Gently pull the Rak Mount 8 toward you a foot or so. 5. Remove the srews holding in plae the Rak Mount 8 top over, remove the top over and set it aside. 6. Remove the two srews holding the ventilation panel in plae above the I/O panel. Before installing or removing ards in the Rak Mount 8, swith off AC power and wait for the 5V Standby LED to power off for the affeted MIB. 7. Add, remove or replae the GPUs or other ards, making sure to seat eah firmly in the slot, seure the end braket srews and to onnet aux power ables, if neessary. Make sure that blank end brakets (L-brakets) are installed for eah slot that has no PCIe ard, espeially if the installed ards are passively-ooled GPUs. Blank end brakets are required to assure proper ooling of the GPUs. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 9

10 8. Replae the Rak Mount 8 front top over and replae the four srews (front, rear and one on eah side) holding it in plae. 9. Gently slide the Xpander Rak Mount 8 bak in plae. 10. Replae the seure srews holding the Rak Mount 8 in the abinet. Front Panel LCD Rakmount 8 front panel LCD provides enlosure details and status. Beneath the LCD are two buttons: left button srolls through main-menu items and right button srolls through sub-menu items. When you first onnet power, Xpander omes up in standby mode. The LCD displays Cubix on the top line and Corporation on the bottom line. This is the Start menu. Aess all other main menu items by pressing the left button for a MIB, the middle button for a menu seletion and then the right button for a sub-menu seletion. The upper right orner of the LCD shows a flashing asterisk *. This indiates that the on-board Miro-Controller Unit (MCU) is running properly. If the asterisk is not flashing or the LCD is not responding, reset the MCU by pressing and holding the left button for 3 seonds while Xpander is running. A orresponding HB (for heartbeat) LED on the MIB also onfirms that the MCU is working. When a fault ours, the LCD will jump to that fault and the LCD baklight will flash on / off. Press the left button to aknowledge the fault and the LCD will stop flashing and enable normal operation. The LCD will ontinue to display the fault at the appliable sub-menu until you lear the fault. Menu 1: Start Menu Menu 1 Sub-menu 1 Sub-menu 2 Sub-menu 3 Cubix PCB Assembly Corporation A09180 Rev 1A-0A Main menu 1: Cubix Corporation Sub-menu 1: PCB Assembly Board Number EEPROM Revision Board Revision Sub-menu 2: MCU Firmware Release Number Release Date Sub-menu 3: PCB Serial Number MCU Firmware Rel Serial Number Cubix Corporation 2800 Lokheed Way Carson City, NV Page 10

11 Menu 2: Power Supply Status Menu 2 Sub-menu 1 Sub-menu 2 Power Supply Power Supply Power Supply Status PWR ON-Good RPS ON-Good Main Menu 2: Power Supply Status Sub-Menu 1: Main Power Status: PWR Standby = Main power is off; system is in standby PWR ON-Good = Main power is on; +3.3V, +5V, +12, -12V are within spe. PWR ON-FAULT = Main power is on; +3.3V, +5V, +12V, or -12V is out of spe. Sub-Menu 2: Optional redundant power supply (RPS) status RPS Standby = Main Power is off, System is in standby RPS ON-Good = Main Power is on, power supply/modules are working RPS ON-FAULT = Main Power is on, power supply/module has failed Menu 3: Temperature Status Menu 3 Sub-menu 1 Sub-menu 2 Temperature Status Tmp1 Tmp2 Tmp3 24C 25C 26C IC Temperature 65C Menu 3: Temperature Status Sub-Menu 1: Display the temperature of the 3 temperature sensors in degrees Celsius. Note: If a temperature sensor fails, FLT (fault) appears under the failed sensor. Sub-Menu 2: Bridge (left), Bridge (right) IC Temperature (Range: 20C-125C, ---C when power is off) Menu 4: Fan Status Menu Sub-menu Sub-menu Sub-menu Submenu Fan Status Fan 1 Running 7V 70% Menu 4: Fan Status Sub-Menu 1-4: Indiates fan status for fans 1-4 Note: Fan speed is ontrolled by adjusting fan voltage based on sensed system temperature. Fan voltage adjusts from 6V min (25deg C or lower) at 50% to 12V max (50deg C or higher) at 100%. Fan voltage is the same for all fans. PWR OFF is 0V at 00%; system is in standby. Running aav at pp%; fan is good; aa = fan voltage and pp = fan power perent (%). FAULT is 12V at 100%; fan fault: fan voltage is set to full on. NO FAN is aav at pp%; Xpander an be onfigured with 1, 2, 3, or 4 fans. No fan means fan is not installed. Menu 5: PCIe Link Status Fan 2 Running 7V 70% Fan 3 Running 7V 70% Fan 4 Running 7V 70% Menu Sub-menu Sub-menu Sub-menu PCIe Link Status PCIe X16 Cable Conneted PCIe Cable 8.0Gb/s X16 Menu 5: PCIe Link Status Sub-Menu 1: Indiates x16 PCIe able installation status Conneted x16 able is properly onneted to the HIC PCIe Slot n 8.0Gb/s X16 Cubix Corporation 2800 Lokheed Way Carson City, NV Page 11

12 NOT-Conneted x16 able is not properly onneted to the HIC Sub-Menu 2: x16 PCIe able link speed and link width. Speed and width depend on the host apability. Speed: 8.0Gb/s, 5.0Gb/s or 2.5Gb/s Width: x1, x2, x4, x8, x16 and NO Link when the host did not suessfully link or Xpander is in standby power. Sub-Menu 3-10: Link speed and link width for slots 1-6. Speed: 2.5Gb/s or 5Gb/s Slot 1-2 width: x1, x2, x4, x8, and NO Link when the slot is empty, unsuessful link, or Xpander is in standby power. Slot 3-6 width: x1, x2, x4, x8, x16, and NO Link when the slot is empty, unsuessful link, or Xpander is in standby power LCD Baklight Control Press right swith to toggle the LCD baklight On/Off. Replae a Power Supply Module All power supply modules show a steady green LED when operating normally. A module with a red LED is either disonneted from power or has a fault. Chek that the power ord is firmly seated. If an LED remains red, replae the module. Shipping Proedure Before shipping Xpander, remove any installed graphis or other ontroller(s) using the following proedure: 1. Swith off the host omputer onneted to Xpander. 2. Disonnet Xpander from the HIC(s). Cubix Corporation 2800 Lokheed Way Carson City, NV Page 12

13 3. Remove the seure srews holding the Xpander in the abinet. 4. Gently pull the Xpander toward you. 5. Lift Xpander down from the abinet and set it on a stable surfae. Before lifting Xpander out of a abinet, ask for assistane. 6. Remove the srews holding Xpander top over, remove it and set it aside. 7. Remove the graphis or other ontrollers and pak them in the original pakaging. Remove GPUs or other adapters before shipping Xpander. 8. Replae the hold-down brakets and tighten the aptured srews. 9. Pak the graphis ards or other ontrollers within the boxes in whih you reeived them. 10. Replae the Xpander Rakmount Elite front top over, replae the srews and remove the power ords. 11. Remove the two outer slide mounts by sliding eah out until it stops. Press the ath to release and remove the outer slide mount. 12. Remove the two slide mounts from the Xpander sides. 13. Store all of the srews, nuts and G-nuts in the plasti bag inluded. Tape the bag to the rails and wrap the rails together. 14. Remove the HIC(s) from the host omputer. 15. Within the original pakaging, pak the Xpander, the power ords, the HIC(s) and the boxed graphis ards. Inlude a opy of these instrutions. Troubleshooting Issue: Cable LED inside Xpander does not light green. Resolution: Chek the following: Confirm that the HIC is firmly seated in a PCIe slot within the host omputer. Confirm that the able onnetor is firmly seated HIC onnetor port. Issue: Resolution: Issue: Resolution: Cable LED inside Xpander flashes green even though I have a PCIe x16 HIC and able. The internal Cable LED should light steady green. If the Cable LED flashes, it is transferring data at a redued bandwidth beause you installed the PCIe x16 HIC in a Gen 1 slot or a PCIe x8 (eletrial) slot. If your HIC is PCIe x8, you installed it in a Gen 1 slot. The HIC operates at full bandwidth when installed in a PCIe Gen 2 slot. Slot LED inside Xpander does not light green. Shut down the OS graefully, power down Xpander and reseat the GPUs in eah PCIe slot. Make sure that you seat eah GPU firmly in its slot and that you onnet all aux power onnetors firmly. Issue: Slot LED inside Xpander flashes green even though I installed a PCIe x16 Gen 3 GPU or ard. Resolution: Slot LEDs should light steady green. If they are flashing, it may be beause the GPUs in the slots do not have a proessing load. Apply a proessing load on the Cubix Corporation 2800 Lokheed Way Carson City, NV Page 13

14 GPUs. If the slot LEDs still flash green, hek the vendor s link for a driver update. Issue: Resolution: Issue: Resolution: Issue: Resolution: Issue: Resolution: All Cable and Slot LEDs are lit green but your OS does not detet the GPUs. Shut down the OS graefully, power down Xpander and reseat the GPUs in eah PCIe slot. Make sure that you seat eah GPU firmly in its slot and that you onnet all aux power onnetors firmly. The front-panel LCD sreen baklight is flashing on/off, whih indiates a fault. Press the MIB button momentarily to identify the MIB with the fault. Aknowledge the fault by pressing the Menu button, whih will stop the LCD flashing and the LCD will remain at the fault. After aknowledging a fault, the front-panel LCD sreen baklight ontinues flashing on/off, whih indiates two or more faults. If multiple faults have ourred, when you aknowledge the first fault, the LCD will jump to the next fault and the LCD baklight will one again flash on/off. This ontinues until you lear all faults. Press the MIB button momentarily to identify the MIB with the fault. Aknowledge the fault by pressing the Menu button, whih will either stop the LCD flashing and the LCD will remain at the fault, or the LCD will jump to the next fault and the LCD baklight will one again flash on/off. The front-panel LCD sreen either does not respond or is blank even though Xpander is operating. Press and hold the MIB Selet button to reset the Miro-Controller Unit (MCU). Issue: Resolution: The front-panel LCD sreen either does not respond or is blank for a partiular MIB even though Xpander and the MIB are both operating. Selet the appropriate MIB using the MIB Selet (left) button. Press and hold the Menu (middle) button for 3 seonds to reset the Miro-Controller Unit (MCU). If the Xpander is running, the Xpander fans will turn on full speed for 10 seonds and then slow down to the operating speed set by the MIB loal temperatures. Cubix Corporation 2800 Lokheed Way Carson City, NV Page 14