Data Center Practices the Good, the Bad, & the Ugly

Size: px

Start display at page:

Download "Data Center Practices the Good, the Bad, & the Ugly"

Amice Webster
6 years ago
Views:

Co-chair BICSI Data Center Subcommittee Chair TIA TR-42.

1 Data Center Practices the Good, the Bad, & the Ugly Image by Thiago Rodrigues Santos Jonathan Jew - J&M Consultants, Inc. Co-chair BICSI Data Center Subcommittee Chair TIA TR-42.6 Telecom Administration Vice-chair TIA TR-42.3 Pathways & Spaces Vice-chair USTAG to ISO/IEC JTC 1 SC 25 WG 3

2 Lessons Learned from 30+ years of 30+ years 90+ data center projects data center design Co-Chair BICSI Data Center Design & Implementation Subcommittee US Project Lead for ISO/IEC and ISO/IEC international data center cabling standards Primary contributor ANSI/TIA-942, ANSI/TIA-942-A, and editor of ANSI/TIA-942-B revisions (data centers) Primary contributor ISO/IEC TR telecom identifiers, ANSI/TIA-606-B telecom administration, and editor ANSI/TIA-606-C revision

3 Learn something from every project Best practice what to do Poor practice what not to do New ideas different way of doing something (e.g., using foam inserts instead of brushes around cable openings) New standards and practices Changes in technology and in the industry mean that common practices may become bad practices (e.g., facing cabinets all the same way or not sealing cable openings)

4 I8 I7 I6 I5 I4 I3 I2 I1 FILE FILE FILE FILE AT&T DEFINITY UNIT Use Standards-Based Structured Cabling Single-use cabling, removing and installing new cables is disruptive Difficult to troubleshoot and manage A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS Heater - A/C Heater - A/C Heater - A/C Heater - A/C POMONA A/C UNIT #8 TABLE DESK DESK TAPE RACK IBM IBM DASD DASD STORAGE TAPE RACK IBM S-390 R34 TABLE H8 H7 H6 H5 H4 H3 H2 H1 TABLE G7 G6 G5 G4 G3 G2 G1 POMONA A/C UNIT #7 F6 F5 F4 F3 F2 LAN F1 E7 E6 E5 E4 E3 E2 E1 IBM CAB IBM IBM A SAA D2 SAA D1 IBM 3745-A11 IBM D IBM 3745-L13 POMONA A/C UNIT #6 C6 C5 C4 C3 C2 C1 CD ROM SCANNER B7 B6 B5 B4 B3 B2 B1 HP PRINTER A6 A5 A4 A3 A2 PAC BELL A1 CISCO ROUTER CONSOLE HALON PANEL HALON ABORT HALON RELEASE F/X STORAGE TAPE RACK IBM AS-400 IBM AS-400 INSTALL A CABLE WHEN YOU NEED TABLE IT (SINGLE-USE, UNORGANIZED CABLING) STORAGE LIEBERT PDU-C STORAGE LOCKER 16 Example of point-to-point cabling (mid-1990s mainframe data center)

5 I8 I7 I6 I5 I4 I3 I2 I1 FILE FILE FILE FILE AT&T DEFINITY UNIT Use Standards-Based Structured Cabling - Organized, Reusable, Flexible Can install new connections quickly Changes are less disruptive Easier to manage and troubleshoot = higher availability A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS Heater - A/C Heater - A/C Heater - A/C Heater - A/C POMONA A/C UNIT #8 TABLE DESK STORAGE DESK TAPE RACK TAPE RACK IBM IBM IBM S-390 DASD DASD R34 IDF IDF Mainframe IDF TABLE H8 H7 H6 H5 H4 H3 H2 IDF H1 TABLE G7 G6 G5 G4 G3 G2 IDF G1 POMONA A/C UNIT #7 IDF FIBER MDF & CORE F6 F5 F4 F3 F2 LAN F1 COPPER MDF E7 E6 E5 E4 E3 E2 E1 IBM 3745s IBM CAB IBM IBM 3745-A IBM D IBM A IBM 3745-L13 SAA D2 SAA D1 POMONA A/C UNIT #6 C6 C5 C4 C3 C2 IDF C1 IDF CD ROM SCANNER B7 B6 B5 B4 B3 B2 IDF B1 HP PRINTER IDF A6 A5 A4 A3 A2 PAC BELL A1 CISCO ROUTER CONSOLE HALON PANEL HALON ABORT HALON RELEASE F/X STORAGE TAPE RACK IBM AS-400 IBM AS-400 STRUCTURED CABLING SYSTEM (ORGANIZED, TABLE REUSABLE, FLEXIBLE CABLING) LIEBERT STORAGE PDU-C STORAGE LOCKER 16 Example of structured cabling (data center redesigned in 2000s)

6 ADMINISTRATION & LABELING

7 Label Patch Panels & Patch Cords Easy to troubleshoot and make changes Difficult to troubleshoot and make changes All ports and cords labeled (with near and far end port IDs) Only one label (ne pas disconnect )

8 ISO/IEC & TIA Labeling ISO/IEC TR , TIA-606-B, and TIA- 606-C (in progress) have compatible labeling schemes Good labeling and documentation reduces troubleshooting time and errors, and thus improves availability

9 Cabinet Naming Scheme "X" COORDINATE AA AB AC AD AE AF AG AH AI AJ HOT AISLE ( REAR) AD02 AD03 AD04 AD05 AD07 AD08 AD09 AD10 COLD AISLE ( FRONTS) AG03 AG04 AG05 AG06 AG08 AG09 AG10 AG11 HOT AISLE ( REAR) Label cabinets and racks with ID on front and back 12 AD12 AG13 13 "Y" COORDINATE = cabinet corner used for grid location ID (right front facing the cabinet)

10 Grid coordinates on walls

11 606-A-1 Patch Panel ID Scheme RU # s on rails

12 Labeling Patch Panels & Cables At minimum, label patch panel with ID of patch panel and 1 st or last port of every subpanel Even high density patch panels must be labeled (maybe on the edge, on a cover, or a horizontal cable manager above or below)

13 Labeling of Patch Cords Place labels where they can be read without disconnecting a cable Labels should be permanent, machine printed labels Labels are the same on both ends of the cable, however, the order should be reversed on each end Local Equip - Rack/cabinet ID - Equip/panel ID - Slot/Port or Port Optional Label with Additional Information (e.g. circuit id, server id, business partner, or name of equipment and ends of channel Local Equip - Rack/cabinet ID - Equip/panel ID - Slot/Port or Port To Cabinet AD04 Switch sfcoresw01 Slot 3 port 2 AD04 sfcoresw01:03/02 To AD07-42:24 sfcoresw01 To sfhda03sw02 sfcoresw01 To sfhda03sw02 AD07-42:24 To AD04 sfcoresw01:03/02 To Cabinet AD07 Panel 42 port 24 Remote Equip - Rack/cabinet ID - Equip/panel ID - Slot/Port or Port AD07-42:24 to AD04 sfcoresw01:03/02 Remote Equip - Rack/cabinet ID - Equip/panel ID - Slot/Port or Port

14 Life Cycle Refresh Rule of thumb is to replace data center server and network equipment ~ 3 to 5 years for maximum availability. 5 to 7 years is more common Old equipment has higher failure rate Replacement often required to deploy new technologies Replace equipment before end of support by manufacturer

15 Impact of Documentation, Labeling, & Equipment Refresh Major reduction in outages at a top ten global technology company

16 Impact of Documentation, Labeling, & Equipment Refresh Prioritize work to maximize impact 1 st year was to improve the data centers and network core 2 nd + years were to improve the edge (office areas and labs)

17 Color-code patch cords Color-code patch cords to simplify identification. In this example for a data center in the USA: - Yellow = Production A - Red = Production B - White = Management

18 Color-coded, labeled, locking power cords Label power cords at both ends to avoid disconnecting the wrong power cord Consider color coding power cords and power strips Consider locking power cords, receptacles, or retention clips

19 ELECTRICAL & BONDING

Labeled, color-coded, & locking electrical

receptacles should be locking to prevent

20 Labeled, color-coded, & locking electrical receptacles Label electrical receptacles with PDU, RPP, breaker Consider color coding electrical circuits Electrical receptacles should be locking to prevent accidental disconnection (twistlock or IP67 watertight)

21 Non-locking, unlabeled, not color-coded electrical receptacles This data center in Australia has electrical receptacles that: - Do not lock (can easily be pulled out) - Are not labeled - Are not colorcoded

Load Power Strips to 40% Maximum and Track Load on Every Power Strip Load power strips to 40% maximum to avoid over-loading circuits if a circuit or power strip fails Maximum load by US code is 80%

22 Load Power Strips to 40% Maximum and Track Load on Every Power Strip Load power strips to 40% maximum to avoid over-loading circuits if a circuit or power strip fails Maximum load by US code is 80% (to ensure that circuit breaker is not tripped) Best practice is Data Center Infrastructure Management (DCIM) system centralized monitoring Alternatively, use power strips with local display that shows current load Check load on power strips before adding equipment

23 Attaching Mesh-BN to Pedestals Do not use cable ties to bond computer room mesh bonding network to pedestals Use bonding hardware designed for the purpose and listed by a testing lab

24 Cable trays properly bonded to computer room mesh bonding network

25 Network equipment not bonded to rack grounding bar as recommended by manufacturer bonding cable comes with equipment Racks & cabinets bonded serially and not individually to computer room mesh-bn as specified in ISO/IEC 30129, TIA-607-C, BICSI-607

26 Power & UTP cables are too close

27 Separation of UTP/STP from unshielded power cables run in parallel # of Circuits Electrical Circuit Type Separation Distance 1 20A 120/230V 1-phase unshielded 2 (E 1 ) - 50 mm (E 2 +) 2 20A 120/230V 1-phase unshielded 10 (E 1 ) - 50 mm (E 2 +) 3 20A 120/230V 1-phase unshielded 20 (E 1 ) - 50 mm (E 2 +) 4 20A 120/230V 1-phase unshielded 24 (E 1 ) - 50 mm (E 2 +) A 120/230V 1-phase unshielded 50 mm A 120/230V 1-phase unshielded 100 mm A 120/230V 1-phase unshielded 200 mm A 120/230V 1-phase unshielded 300 mm A 120/230V 1-phase unshielded 600 mm 1 100A 415V 3-phase unshielded 300 mm Distances are ½ if power cables are shielded. Distances vary based on E environmental classification (MICE) No separation required if there is a properly earthed 1.5 mm thick galvanized steel containment around either the power or twisted-pair cables A 415V 3-phase unshielded 600 mm

28 TELECOMMUNICATIONS PATHWAYS

29 Small diameter fiber patch cords and trunks should not be mixed with heavier cables that can damage them

30 Install Patch Cords and <= 4mm Diameter Fiber Cables in Solid Bottom Trays or Duct Solid bottom in duct prevents micro-bends that increase attenuation applies to both standard and bendinsensitive fiber - in next revision to TIA-942 and proposed for revision to ISO/IEC

31 Install Patch Cords and <= 4mm Diameter Fiber Cables in Solid Bottom Trays or Duct Solid bottom can be installed in cable ladder or wire basket tray

32 Don t install fiber cords or <=4mm diameter fiber cable in wire basket tray without bottom Nor in ladder that does not have cross members with large rounded edges

33 Improper use of cable ties as supports Do not use non-continuous supports (ties, hooks) without large rounded edges for fiber patch cords and other small diameter fiber cables Use Velcro instead of cable ties for UTP/STP

34 Keep patch cords off the floor Dress patch cords properly to keep them off the floor or the base of racks, where they can be stepped on

35 Leave room for growth in cable trays and carefully plan cable tray intersections Tray is full with area of cable at 50% of area of tray Intersection is too full when two full trays meet

36 Cable Trays Minimum clearance above cable trays 200 mm and preferably at least 300 mm (in TIA-569-D and revision to ISO/IEC ) Confirm with raised floor designer or ceiling structural engineer that they can support cable tray load No sharp edges or exposed threaded rods that can damage cables Maximum depth of cables is 150 mm Max depth of fiber patch cords 100 mm

37 Overhead Cabling Best practice where ceiling heights permit as under floor cabling can create airflow disruption & turbulence Must be higher than tallest anticipated cabinet or rack Suspend cable tray from ceiling to allow cabinets to be removed Separation from fluorescent lights 130 mm minimum

38 Overhead Cable Tray Cable tray and duct suspended from ceiling using threaded rod Best practice to cover threaded rod to avoid cable abrasion

39 Under Floor Cabling Full cable trays could potentially block airflow if not properly planned & coordinated (place in hot aisles avoid front of air conditioning) Upper cable trays should be narrower than lower trays to allow access Recommend at least 50 mm from top of cable tray to bottom of floor tiles to allow cables to exit

Example of Wire Basket Cable Trays For Cabling Under 900 mm Raised Floor Top tray at least 50 mm from bottom of tile 24" RAISED FLOOR TILE 8" 2" FIBER 4" 1.375" 6" 1.

40 Example of Wire Basket Cable Trays For Cabling Under 900 mm Raised Floor Top tray at least 50 mm from bottom of tile 24" RAISED FLOOR TILE 8" 2" FIBER 4" 1.375" 6" 1.375" 6" 24" RAISED FLOOR TILE 2" 4" 8" FIBER 1" SUPPORT STRUT SUPPORT STRUT Main aisle fiber tray 1" 4" 36" 12" min clearance 300 mm between 12" cable 12" traysbetween trays 18" 18" COPPER CABLING 6" 6" COPPER CABLING 1" SUPPORT STRUT SUPPORT STRUT 1" Main aisle copper tray 6" HOT AISLE " 4.625" HOT AISLE Server Row Server Row Main aisle and server cabinet aisle trays at different levels to avoid congestion at intersections

41 Spacing between trays not 200 mm in this data center in Tokyo

42 These commonly used electrical trays have sharp edges that damage cables

contact with earth to avoid moisture Use cable

43 No cables on concrete in contact with earth Cable should not be installed directly on concrete in contact with earth to avoid moisture Use cable matting or other floor cover to avoid placing cables on bare concrete

44 Remove unused cables

45 S AND RACKS

Equipment Cabinets Front rails of cabinets must be set back to provide adequate room for patch cables and wire managers Adequate space for cable

46 Equipment Cabinets Front rails of cabinets must be set back to provide adequate room for patch cables and wire managers Adequate space for cable management at front, sides, and rear Arrange switches and patch panels to minimize patching between cabinets & racks One edge of cabinets placed at edge of tile

These switches don t have adequate clearance to cool properly - 150 mm to

47 These switches don t have adequate clearance to cool properly mm to side panel with Catalyst 65xx, 68xx mm to side panel with Nexus 7000

48 SD SD SD SD SD SD SD SD SD Power Supply 1 Power Supply 2 Catalyst 6500 SERIES SD SD Power Supply 1 Power Supply 2 Catalyst 6500 SERIES Plan for growth & consider airflow SAMPLE HDA FOR 20 SERVER S (24 UTP and 36 pair MMF per cabinet) Rack AU82 Rack AU81 Rack AU80 Rack AU79 Rack AU78 Leave space for new equipment & patch panels Equipment needs adequate space for side-to-side airflow Fiber Patch Panel Fiber Patch Panel Fiber Patch Panel Space for Future SAN Switch SAN Switch A A B C Backbone Fiber Panel Fiber Patch Panel Fiber Patch Panel SAN Switch B Space for Future SAN Switch A D B C Fiber Growth Space for Future LAN Switch LAN Switch A Cab xxx1 Cab xxx2 Cab xxx3 Cab xxx4 Cab xxx5 Cab xxx6 Cab xxx7 Cab xxx8 Cab xxx9 Cab xx10 Cab xx11 Cab xx12 Cab xx13 Cab xx14 Cab xx15 Cab xx16 Cab xx17 Cab xx18 Cab xx19 Cab xx20 Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Cat 6A panel Copper Growth A B C D E F FG H I IJ K L M N O P Q FR S T 24 UTP to NET-MDA-1 Backbone Cat 6A Backbone UTP Patch Pnl Copper Growth Space for Future LAN Switch LAN Switch B A 7 FOOT RACK 7 FOOT RACK 7 FOOT RACK 7 FOOT RACK 7 FOOT RACK

49 Horizontal & vertical cable management Provide adequate vertical and horizontal cable management for fully populated cabinet

50 Energy Efficiency Use blanking panels in open spaces in cabinets & racks Use brushes & grommets for cable openings Use baffles for equipment cooled side-to-side

51 Seismic cabinets and racks In seismically active locations consider seismic cabinets or racks They must be supported from the structural floor or on a steel stand attached to the structural floor

52 Steel Seismic Rack Aluminum is brittle so seismic racks are typically made of steel Rack on the left is bolted to a steel stand attached to the concrete floor below (it is not on the raised floor)

53 Light color cabinets enhance lighting Front edge of cabinets should be on edge of floor tile to permit two full tiles for ventilated tiles

54 BICSI COMMUNITITY & BICSI-002 ARE GOOD SOURCES FOR DATA CENTER BEST PRACTICES

55 BICSI-002 Data Center Standard BICSI Data Center Design & Implementation Best Practices is an update to the 2011 version Supplements applicable local data center standard. It references ANSI/TIA-942-A (US & Canada), CENELEC EN (Europe), ISO/IEC (International) Basis for BICSI Data Center Design Consultant (DCDC) certification Available in Spanish

56 DC Operations & Maintenance New BICSI 009 Data Center Operations & Maintenance standard being developed Includes participants from a wide variety of organizations Include participants from several countries on every continent except Antarctica

57 Questions? Jonathan Jew President J&M Consultants, Inc. Website: Co-chair BICSI data center subcommittee Editor TDMM 13 th Edition Telecom Administration Chair TIA TR-42.6 telecom administration subcommittee Vice-Chair TIA TR-42.3 pathways & spaces Vice-chair USTAG to ISO/IEC JTC 1 SC 25 WG 3 Editor ISO/IEC TR telecom admin identifiers SME BICSI TI&M Telecommunications Administration Secretary TIA TR-42

58 PLANNING AHEAD

59 Preparing for Category 8 Plan rows, pods, distributor (IDF, HDA, ZD) locations to permit migration to Cat 8 24 m max permanent link (panel-to-panel) m channels (3-4 m patch cords) No intermediate patch panels

60 12 & 32-fiber MPOs 12-fiber MPO has 4 of 12 unused fibers for 40G and 100G 400G Ethernet will use 32- fiber MPO and waste no fibers Expect 3 x 12-fiber MPO to 32-fiber MPO adapter cables (that waste only 4 fibers)

Cabling a Data Center to TIA-942 Standard

Cabling a Data Center to TIA-942 Standard BICSI 2006 Winter Conference Wednesday, January 25 th 2006 at 9:15 am Gary J. Bernstein, RCDD HellermannTyton Corporation Agenda Overview of TIA-942 Data Center