Data Center Design and the ANSI/TIA-942 Standard

Transcription

1 Data Center Design and the BICSI Regional Meeting Charleston, WV Rick Foster RCDD,TLT Communications Infrastructure Group April 20, 2006 Maurice Zetena Fiber Connect

2 Standards TIA-942 NFPA-70 Article 645 NFPA-75 NFPA-76 International Standards & BICSI Practical Applications of ANSI/TIA-942 Any to All = Structure Trunk Cabling Copper & Fiber

3 THE TECHNOLOGY GAP PERFORMANCE TECHNOLOGY GAP CODES & STANDARDS EVOLUTION TIME

4 CODES vs STANDARDS CODE: A regulation that is enforceable. It is the Law! STANDARD: A recommend practice and performance requirements. It is our Profession!

5 Business Risk Proper Design = Redundancy = Continuity Room Layout Equipment Types Environmental - HVAC & Power Change Control Fire & Safety Security Issues IT Infrastructure Physical Layer Structured Design approach

6 TR s CP Backbone WAO EF s Admin Horizontal Enterprise Cabling Enterprise Cabling Are they really any different? Data Center Cabling EDA ZDA Backbone MDA DATA CENTER CR HDA Horizontal

7 Why another standard? Why another standard? The purpose of this standard is to provide requirements and guidelines for the design and installation of a data center or computer room. It is intended for use by designers who need a comprehensive understanding of the data center design including the facility planning, the cabling system and the network design. Source: ANSI/TIA

8 Data Center Design ANSI/TIA-942 Data Center Design ANSI/TIA-942 Design Process = Critical Coordination Telecommunications cabling system Equipment floor plan Electrical plans Architectural plan HVAC Security Lighting systems

9 TIA 942 Design Considerations Source: ANSI/TIA

10 Defines new terms Recommends media selections Correlation to existing ANSI/TIA/EIA 568 & 569 Source: ANSI/TIA Defines the physical environment

11 Spaces Computer Room Telecommunications Room Entrance Room Main Distribution Area (MDA) Horizontal Distribution Area (HDA) Equipment Distribution Area (EDA) Zone Distribution Area (ZDA) Source: ANSI/TIA

12 Cabling & Topology Backbone Horizontal Main cross-connect Source: ANSI/TIA Horizontal cross-connect Cross-connect in the entrance room or MDA Zone outlet EDA outlet

13 Spaces Data Center Computer Room MDA EDA HAD ZDA Source: ANSI/TIA

14 Data Center Telecommunications Spaces Data Center Telecommunications Spaces Main Distribution Area (MDA) - It s the central space where the point of distribution for the structured cabling system in the data center is located - The data center shall have at least one MDA - The core routers and core switches for the data center are often located in or near the MDA Could be a Equipment Room

15 Data Center Telecommunications Spaces Data Center Telecommunications Spaces Horizontal Distribution Area (HDA) - It s the space that supports cabling to the equipment distribution areas - LAN, SAN and KVM switches that support the end equipment are also typically located in the HDA - There should be a minimum of one horizontal distribution area per floor Could be a Telecom Room

16 Data Center Telecommunications Spaces Data Center Telecommunications Spaces Equipment Distribution Areas (EDA) - Are spaces allocated for the end equipment, including computer systems and communications equipment -End equipment is usually standing equipment or equipment mounted in cabinets or racks - Horizontal cables are terminated in equipment distribution areas or connecting hardware mounted in cabinets or racks - Point-to-point cabling is permitted between equipment located in the equipment distribution area. Cable should be greater than 15 m and should connect equipment in adjacent racks or cabinets in the same row Could be a Work Area

17 Data Center Telecommunications Spaces Data Center Telecommunications Spaces Zone Distribution Area (ZDA) - Should be limited to serving a maximum of 288 coaxial or twisted pair connections - Cross-connection shall not be used in the ZDA - No more than one ZDA shall be used within the same horizontal cable run - There shall be no active equipment in the zone distribution area, except DC powering equipment Could be a Consolidation Point

18 Recognized Media 100Ω ohm twisted-pair copper cable Cat-5e Cat-6 recommended Multimode fiber optic cable 62.5/125 µm or 50/125 µm 50/125 µm 850 nm laser optimized MMF recommended Single Mode Fiber Optic Cable 75 Ω ohm Coax Cable

19 Data Center Redundancy Data Center Redundancy Tiers - Annex G of ANSI/TIA-942 Tier 1 Basic data center Tier 2 Redundant components Tier 3 Concurrently maintainable Tier 4 Fault tolerant Corresponds to the tier ratings defined by the Uptime Institute

20 Data Center Tiers Source: ANSI/TIA

21 Source: Uptime Institute Data Center Tiers Tier I Tier II Tier III Tier IV Building Type Tenant Tenant Stand-alone Stand-alone Staffing None 1 Shift 1 + Shifts "24 by Forever" Useable for Critical Load 100% N 100% N 90% N 90% N Initial Gross Watts per Square Foot (W/ft 2 ) (typical) Ultimate Gross W/ft 2 (typical) ,2, ,2 Uninterruptible Cooling None None Maybe Yes Support Space to Raised-Floor Ratio 20% 30% 80-90% % Raised-Floor Height (typical) 12" 18" 30-36" " 2 Floor Loading Lbs/ft 2 (typical) Utility Voltage (typical) 208, , kv kv 2 Single Points-of Failure Many + human error Many + human error Some + human error None + human error Annual Site-Caused IT Downtime 28.8 hours 22.0 hours 1.6 hours 0.4 hours (actuals) Site Availability % % % % Months to Implement Year First Deployed Construction Cost ( + -30%) 1,2,3 Raised Floor Useable UPS Output $220/ft 2 $10,000/kW $220/ft 2 $11,000/kW $220/ft 2 $20,000/kW $220/ft 2 $22,000/kW Uptime Institute 1100 W/ft 2 maximum for air-cooling over large areas, water or alternate cooling methods greater than 100 W/ft 2 (added cost excluded). 2Greater W/ft 2 densities require greater support space (100% at 100 W/ft 2 and up to 2 or more times at greater densities), higher raised floor, and, if required over large areas, medium voltage service entrance. 3Excludes land; architectural, engineering, and commissioning fees; permits and other fees; interest; and abnormal civil cost. These can be several million dollars. Assumes minimum of 15,000 ft 2 of raised floor, architecturally plain, one-story building, with power backbone sized to achieve ultimate capacity with installation of additional components or systems. Make adjustments for NYC, Chicago, and other high cost areas.

22 Summary ANSI/TIA Standard specifies data center planning and design procedures. Structured approach. 2. Provides guidelines and recommendations for data centers implementation, embraces much more than telecommunications infrastructure (power, pathways, fire protection, doors, walls treatment, and others) 3. Determines four tiers for data center redundancy 4. Considers 568-B, 569-B, 606-A and 607 for the cabling, pathways, administration and earthing design, installations and other procedures

23 National Electric Code National Electric Code Protects people and property from hazards arising from the use of electricity. It is not a DESIGN SPECIFICATION or training manual for the untrained individuals Scope. This article covers equipment, power-supply wiring, equipment interconnecting wiring, of information technology equipment and systems, including terminal units, in an information technology equipment room. Source: NEC 2005 NFPA 70 FPN: For further information, see NFPA , Standard for the Protection of Electronic Computer/Data Processing Equipment.

24 Source: NEC 2005 NFPA 70 Source: NEC 2005 NFPA 70

25 National Electric Code NFPA-70 Article Modifications to Chapter Special Requirements for Information Technology Equipment Room. This article shall apply, provided all the following conditions are met: (1) Disconnecting means complying with are provided. (2) A separate heating/ventilating/air-conditioning (HVAC) system is provided that is dedicated for information technology equipment use and is separated from other areas of occupancy. Any HVAC system that serves other occupancies shall be permitted to also serve the information technology equipment room if fire/smoke dampers are provided at the point of penetration of the room boundary. Such dampers shall operate on activation of smoke detectors and also by operation of the disconnecting means required by FPN: For further information, see NFPA , Standard for the Protection of Electronic Computer/Data Processing Equipment, Chapter 8, 8-1, 8-1.1, 8-1.2, and Conductor sizing derated for some equipment 2. Connection methods attachment plugs - 4 to Interconnecting cable types nonplenum rated and allows vendor specific interconnecting cabling under the raised floor Check with the (AHJ) (3) Listed information technology equipment is installed. (4) The room is occupied only by those personnel needed for the maintenance and functional operation of the installed information technology equipment. (5) The room is separated from other occupancies by fireresistantrated walls, floors, and ceilings with protected openings. FPN: For further information on room construction requirements, see NFPA , Standard for the Protection of Electronic Computer/Data Processing Equipment, Chapter Wiring methods for HVAC Require smoke detection and shutdown. 5. Room construction defined Source: NEC 2002 NFPA 70

26 NFPA 75 Standard for the Protection of Electronic Computer/Data Processing Equipment. NFPA-70 National Electric Code Scope. This article covers equipment, power-supply wiring, equipment interconnecting wiring, of information technology equipment and systems, including terminal units, in an information technology equipment room. FPN: For further information, see NFPA , Standard for the Protection of Electronic Computer/Data Processing Equipment. ANSI/TIA-942 Source: ANSI/TIA

27 NFPA Standard NFPA - 75 NFPA Standard NFPA Scope. This standard covers the requirements for the protection of information technology equipment and information technology equipment areas. Source: NFPA * Purpose. The purpose of this standard is to set forth the minimum requirements for the protection of information technology equipment and information technology equipment areas from damage by fire or its associated effects namely, smoke, corrosion, heat, and water. How to build it A systems approach

28 NFPA Standard NFPA - 75 Reduce Risk = Business Continuity Chapter 4 Risk Considerations 4.1* Risk Factors. The following factors shall be considered where determining the need for protecting the environment, equipment, function, programming, records, and supplies: (1) Life safety aspects of the function (e.g., process controls, air traffic controls) (2) Fire threat of the installation to occupants or exposed property (3) Economic loss from loss of function or loss of records (4) Economic loss from value of equipment Source: NFPA Telecommunications Risks. In assessing and evaluating the damage and interruption potential of the loss of information technology equipment room operations, attention shall be given to the impact of the loss of data and communications lines. If these functions are vital to the operation, rooms housing the services shall be constructed in accordance with Chapter 5 and protected in accordance with Chapter 8. These rooms shall be secured, locked, and free of extraneous combustibles.

29 Source: NFPA -75

30 NFPA Standards NFPA - 76 NFPA Standards NFPA - 76 Central Office Space 1.1* Scope. This recommended practice provides recommendations for fire protection of telecommunications facilities where telephone, data, cellular, internet, and video services are rendered. Telecommunications facilities include telecommunications equipment spaces, cable entrance facilities, power areas and battery spaces, main distribution frames, standby engine areas, technical support areas, administrative areas, and building services and support areas within both large and small facilities. This recommended practice includes fire protection of telecommunications service, property protection, and life safety for people in telecommunications facilities. It provides for both performance-based and prescriptive options. Source: NFPA -76

31 NFPA Standards NFPA - 76 Reduce Risk = Business Continuity 4.2 Communications Risks. In assessing and evaluating the damage and interruption potential of the loss of communication operations, attention should be given to the impact of the loss of data, voice, and video communications links. The complexity and scope of switching equipment operations can make it necessary to provide internal redundancy, alternative routing, and in some cases dual access terminal in order to prevent communication outages. Source: NFPA -76

32 Practical Infrastructure Topologies Practical Infrastructure Topologies Structured Cabling System in the Data Center - (TIA/EIA 942) Any to All = Structure Change equipment not infrastructure

33 Don t Let This Happen To You Improper Bend Radius No Clear Migration Path

34 Proper Planning Makes The Difference Equipment Placement - Cabinet Layout Switch Location Server Cabinet Patch Panel Use of Cable Management Panels Use of Vertical Cable Managers Example of Poor Use of Rack Space, No Migration Planning, & Poor Cable Management.

35 Any to All = Structure Any to All = Structure HDA or EDA Connectivity design will support multiple equipment and applications. Change out equipment not infrastructure MDA or HDA

36 Critical Design Factors Talk the same Language ie. Strands vs. Channels Enclosure / Cabinets Connector types Define Polarity Method Cable Types Equipment & Migration Path Redundancy Environment

37 Structured Advantages Structured Advantages Allows for faster roll outs Designed Zones with multiple connections as required by equipment design Reduce the number of cable pulls Managed cabling strategy Trunk cables Adapt to frequently changing connection models or media types

38 Traditional Point to Point - Non Structured Jumper Cable / Point to Point (Traditional) Difficult to administer Requires Under Floor Access to Increase Capacity Not Suitable for Rapid Deployment Inconsistent Channel to Channel Characteristics Difficult to Maintain Day 2 organization

39 Fiber / Copper - Structured

40 MDA / HDA - Structured Fiber SAN

41 Structured Approach Design Strategy Utilizing Trunk Cabling

42 Trunk Cables What are the Advantages? What s Available? Plug & Play?

43 Why use Trunk Cables? Why use Trunk Cables? Media Choice: Fiber Copper Performance: Quality Assurance Application Warranty Availability: Schedule Install Time Logistics Environment Price: Material Cost Labor Cost Less Scrap Installation: Design Future Growth Installer Skill Level Accessibility

44 Copper Trunk Cables Copper Trunk Cables

45 MDA / HDA - Structured Non-MTP

46 MDA / HDA - Structured MTP

47 Fiber Trunk Cable Assemblies Fiber Trunk Cable Assemblies Cables can be doubled ended or single ended Factory Terminated and Tested Connectors can be different types SC,LC,MTP, MU, other

48 Plug & Play Assemblies What Are They? Factory terminated modules that utilize multifiber (MTP/MPO) connector with breakout kit to individual fiber connectors for interconnect

49 Plug & Play: Alternate Building Blocks Plug & Play: Alternate Building Blocks Is Plug & Play Limited to Cassettes? MTP Brackets: Reduce number of Hops. Reduce System Loss. Reduce System Cost. Work above the Floor.

50 Plug & Play: Alternate Building Blocks Plug & Play: Alternate Building Blocks MTP - X Harness Assemblies: The Mix & Match Advantage MTP - SC Duplex MTP - MT-RJ MTP - LC Duplex

51 Trunk Cabling Systems Advantages Simplified Installation Reduce human errors * Less time in the floor less disruption and debris Factory terminated and pre-tested minimizes points of failure Uniform design and consistent documentation scheme

52 ADVANTAGES OF MTP TRUNK / HARNESS SWITCH CABLING Trunk and Harness Technology: Trunk / Harness Technology Simplifies administration Simplifies Visual Identification Future Growth Is Built In Allows Pre-Cabling to Cabinets for Rapid Deployment (servers) Consistent Channel to Channel Characteristics

53 EXAMPLE OF MTP TRUNK / HARNESS SWITCH CABLING EXAMPLE OF MTP TRUNK / HARNESS SWITCH CABLING Trunk / Harness Configuration Fan Intake Remains Clear Proper Bend Radius Adherence Trunk Cable Feed From Under floor

54 UPS PDU COLD CORRIDORE PDU UPS Fiber Connect, Inc. Hot and Cold Aisles Relocated Equipment Future Equipment Grow th UPS PDU UPS PDU UPS PDU UPS PDU COLD CORRIDORE HOT CORRIDORE COLD CORRIDORE HOT CORRIDORE zzzzzzzzzzzzz HOT CORRIDORE COLD CORRIDORE HOT CORRIDORE COLD CORRIDORE HOT CORRIDORE COLD CORRIDORE PDU UPS PDU UPS PDU UPS PDU UPS PDU UPS AC Package Unit AC Package Unit AC Package Unit DB-A DB-B

55 Hot and Cold Aisles

56 Don t Forget Cable Routing Cable Routing Defined Pathways Minimize Zig Zag Overhead Distribution Under floor Distribution Coordination with other room utilities

57 Additional Standards International CENELEC EN Information Technology Generic Cabling Systems Part 5 Data Centers CENELEC EN Annex on Data Center planning ISO/IEC Proposed Generic Cabling or Data Centers Data Center sub-committee to standards group Best Practices Document

58 References American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) - ASHRAE Thermal Guidelines for Data Processing Environments Electronic Industries Association (EIA) EIA-310-D Cabinets, Racks, Panels, and Associated Equipment National Fire Protection Association (NFPA) - NFPA 75 Standard for the Protection of Information Technology Equipment NFPA 70 National Electrical Code are and NFPA 101 Life Safety Code also applicable

59 References Telecommunications Industry Association (TIA) - TIA-942 (Proposed) Telecommunications Infrastructure Standard for Data Centers Other TIA cabling standards (568, 569, 606, 607, 758) are also applicable. Telecordia Technologies - Telcordia GR-63-CORE (NEBS) Requirements: Physical Protection A central office specification, but referenced by TIA-942 for seismic specifications, vibration testing, floor loading capacity measurement and test methods.

60 Data Center User Groups and Information Exchanges 7x24 Exchange - AFCOM Institute for Data Center Professionals (IDCP) International Facility Management Association (IFMA) Uptime Institute Construction Specification Institute The Data Center Journal -

61 Thank you! & Questions