Data Center Cabling Technology Trends and Best Practices

Size: px

Start display at page:

Download "Data Center Cabling Technology Trends and Best Practices"

Audrey Higgins
6 years ago
Views:

1 Data Center Cabling Technology Trends and Best Practices Lisa Huff, CDCP Data Center Applications Engineer Berk-Tek, A Nexans Company March 31, 2009

2 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 2

3 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 3

4 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 4

5 LAN and Data Center Growth Drivers BECAUSE YOUR BUSINESS RUNS THROUGH US Data Policy Directive (DPD) European Union (E.U.) Basel Capital Accord (2006) HIPAA (U.S.), PIPEDA (Canada) Sarbanes-Oxley Act (U.S.) Securities & Exchange Commission (U.S.) Redundancy to protect against catastrophic loss Web Enterprise Storage Growth Million Terabytes 5

6 Higher Speed Market Drivers Higher Speed Ethernet Market Adopters Internet exchanges ISP Backbones Content Providers Campus Backbones Corporate Backbones Corporate Data Centers ISP Aggregation High Performance Computing (HPC) Broadband Aggregation BECAUSE YOUR BUSINESS RUNS THROUGH US Year 6

7 Typical Enterprise Data Centers Facility Purpose Typical Data Rates Typical # of Racks Typical Business TR Cross-Connections & Switches 100 Mbps < 5 All Small DC (Reduced) Medium DC (Basic) Servers, CPU, UPS, Switches Servers, CPU, UPS, POE, Local SAN, Switches 1 Gbps 5-20 Small Financial Hospitals 1-10 Gbps Banks Drug Co. Universities Large DC (Distributed) Servers, CPU, UPS, POE, Local & Remote SANs, Switches, Asset Management 1-10 Gbps > 100 Large Financial Retail Insurance Large Research Labs 7

8 Server Forecast x86 Servers by Ethernet Connection Speed (40G and 100G) Source: Intel & Broadcom (Nov 2007) Millions 10 year transition for 1G Ethernet 5 years for 10GBE 5 years for 40GBE 8

9 Server Trends Market Penetration by Data Rate BECAUSE YOUR BUSINESS RUNS THROUGH US Ethernet Speed (Mbps) Time to Reach (years) Million Port Sale (after Ratification) % Market Penetration , , , , PROJECTIONS 9

10 Server Trends: Rationale for 40G Ethernet High cost relegates 10G primarily to aggregation 40G Rationale Better match to speed of next generation servers PCIe Significantly less expensive than 100G Technology already developed InfiniBand SONET 10

11 Server Trends Source: IDC (2007) Extrapolated with 10% CAGR % servers IDC/Co-location Data Center Program pedestal rack blade 11

channel host bus adaptor (HBA) and other input/output (IO)

12 Blade Servers Servers on a card, containing processors, memory, integrated network controllers, an optional fiber channel host bus adaptor (HBA) and other input/output (IO) ports. IDC/Co-location Data Center Program Rack Server 12 Blade Server

Blade Servers: Impact on Cabling Pedestal Server Rack Server Blade Server Less than 100m to switch across DC Copper or fiber cabling Base-T or Base-SR/LR

13 Blade Servers: Impact on Cabling Pedestal Server Rack Server Blade Server Less than 100m to switch across DC Copper or fiber cabling Base-T or Base-SR/LR Less than 15m to switch at rack top or in row Copper cabling Base-CX4 or Base-T Less than 30 inches to switch in back of chassis Copper PCB Base-KX4 or Base-KR 13

14 Blade Servers: Impact on Cabling Will reduce client-to-access and patch cabling links Adoption of blades, virtualization and FCoE will increase network data rates 10G copper and fiber 40 and 100G fiber IDC/Co-location Data Center Program 14

15 Switch and Server Vendors Collide Cisco announces blade servers as part of its California Unified Computer System (UCS) In response, IBM positioning to buy SUN 15

16 Data Center Trends Top considerations Energy and cooling Space savings and density 16

17 Data Center Heat Load Trends kw/ft 2 kw/rack ASHRAE, Datacom Equipment Power Trends and Cooling Applications, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., 17

18 Modular Designs Based on Cooling Requirements Hot aisle containment POD with APC in-row cooling Hot aisle containment POD with Liebert XDV overhead spot cooling Source: Sun Microsystems 18

19 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 19

20 Standards: Fibre Channel Fibre Channel Roadmap Specification and Market Availability Striped FC-Base10 for ISLs Only Solid FC-Base2 for all end devices Year Source: Fibre Channel Industry Association (FCIA) 20

1Qau, Virtual Bridged Local Area Networks Reducing number of server network cards and

21 Standards: FCoE and CEE Converged data center network Mapping FC frames over Ethernet frames T11.3 FC-BB-5, Fibre Channel Backbone 5 Making Ethernet lossless IEEE 802.1Qau, Virtual Bridged Local Area Networks Reducing number of server network cards and therefore interconnections Leverages the best of FC and Ethernet and installed base Diagrams Source: FCIA 21

22 Standards: InfiniBand Switched fabric communications link primarily used in high-performance computing. Data Rate (Gbps) Single Double Quad 1X X X

23 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 23

24 Cabling for HSE: SMF vs LOMF Fiber Type SMF LOMF OM3 LOMF OM4 Cable Cost Low High High Transceiver Type Transceiver Cost CWDM Parallel Optics Parallel Optics High Low Low Optical 10G Expected 40/100G Availability of 40/100G 10 km 300 m 500 m 10 km 100 m 200 m 2H09 Yes Yes 24

25 Cabling Options Transmission Methods: Single Mode Fiber Serial 40G lasers are too expensive 100G lasers still in R&D CWDM 40G - Four 25G lasers 100G Ten 10G lasers 1 Tx fiber, 1 Rx fiber Must have low/zero water peak SM for CWDM in the 1300 nm region BECAUSE YOUR BUSINESS RUNS THROUGH US 25

26 Cabling Options: Low/Zero Water Peak Fiber BECAUSE YOUR BUSINESS RUNS THROUGH US 1.2 O wavelength E (nm) S C L Lower loss Loss (db/km) Conventional SM Fiber Wavelength (nm) Enables Full-Spectrum CWDM Coarse Wavelength Division Multiplexing (20 nm spacing ITU-T G.694.2) 26

27 Cabling Options: LOMF 12 Channel Duplex 100 Gb/s 850 VCSEL Array 12 x 10 Gb/s Two 12 fiber ribbons/mpo Installed serial 1G/10G 1 Tx fiber, 1 Rx fiber 40/100G parallel optics 10 fibers of 10G Tx 10 fibers at 10G Rx 12-fiber MTP/MPO connector 24 fiber LOMF cable 27

28 Cabling Options: Parallel Optics Specifications What happened to OM2? BECAUSE YOUR BUSINESS RUNS THROUGH US Parallel optics manufacturing process Mounting 12 VCSELs on wafer difficult Transceiver manufacturers want a relaxed spec Spectral width Encircled flux Result may be increased DMD Higher DMD results in shorter distance OM2 distance with new VCSELs too short OM3 becomes minimum bandwidth required 28

29 Cabling Options: OM4 Fiber 4700 MHz km targeted bandwidth 500 meters at 10G Targeted distance of over 200 meters for 40/100G 4900 MHz km current maximum bandwidth fiber available today GIGAlite 10XB 600m at 10G Best shot at longer distance 40G/100G 29

30 Cabling Options Summary: Preparing for 40G/100G Fiber BECAUSE YOUR BUSINESS RUNS THROUGH US Specify minimum of OM3 fiber OM4 (4700 MHz minimum) as an option Add SM (low water peak) to cables Design data centers for 100 meter maximum lengths Consider higher fiber count requirements 2 fibers per link becomes 24 fibers MPO/MTP connectors will become standard transceiver interface 30

31 Cabling Options: The Future for Copper IEEE work just starting on 40/100G copper Preliminary modeling Berk-Tek and Penn State University Transceiver manufacturers Possible to look at UTP/FTP technology and draw reasonable conclusions about copper roadmap 31

32 Cabling Options: Noise Susceptibility Alien Crosstalk BECAUSE YOUR BUSINESS RUNS THROUGH US Two options for reducing alien crosstalk Shielding Spacing Cat 6 Cat 6a 32

MHz ISO 11801 Category 7A 1000 MHz proposed ISO

33 Cabling Options: Shielded Cable Designs F/UTP or FTP (Class E) 500 MHz Category 7 (Class F) 600 MHz ISO Category 7A 1000 MHz proposed ISO standard Category MHz proposed ISO standard 33

34 The future for outlets 4 Pairs on top 250 MHz 8 top contacts 100% RJ45 compatible 4 new contacts For >600MHz 2 Pairs on top 2 Pairs on bottom 600 MHz Switch mechanism Only 8 contacts at a time 34

35 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 35

36 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 36

37 Airflow: Hot Aisle/Cold Aisle BECAUSE YOUR BUSINESS RUNS THROUGH US CEILING CABLE TRAYS or LADDER RACK 3-4 CABLE TRAYS or LADDER RACK CABLE TRAYS or LADDER RACK Front Cabinet or Rack Cabinet or Rack Cabinet or Rack Rear Rear RAISED FLOOR CABLE TRAYS POWER CABLES PERFED TILES CABLE TRAYS POWER CABLES PERFED TILES 37

38 Airflow: CAT 6 vs 48-Fiber MDP Cable BECAUSE YOUR BUSINESS RUNS THROUGH US Wire Basket Air flow Raised Floor channel Cat 6 UTP solution 242 cables at 50% Fill Ratio channel Fiber solution 50/125 LOMF MDP 10 cables, Fill Ratio < 1 % *channels include 2 fibers 38

39 Airflow: Fiber vs Copper Cost Trade-off BECAUSE YOUR BUSINESS RUNS THROUGH US Optical ports currently 60-percent higher installed cost of copper Currently studying ROI for copper versus fiber Includes total system cost of ownership Power and cooling efficiency calculations Space savings 39

40 Airflow: Fiber vs Copper Cost Comparison Summary BECAUSE YOUR BUSINESS RUNS THROUGH US Connection type Port cost Materials: Cable, patch cords, connectivity Installation labor per channel Total per channel cost LOMF: Field Terminated 5, $5, LOMF: Pre-Terminated 5, $5, UTP Copper 2, $2, F/UTP Copper 2, $2, Cost per channel based on: Cisco 6500 switch with seven 48-port GigE copper/fiber blades; One 8-port 10-Gig blade with X2 modules; distribution pricing; 2 meter patch cords; labor rate of $60/hour 40

41 Airflow: Heat Load Considerations Copper vs Fiber at 10G Cisco Catalyst 6500 Switch 41 BECAUSE YOUR BUSINESS RUNS THROUGH US Watts per Network Port Copper Solution Copper Solution Optical Solution 8.1 Power Optical consumption solution based saves on: 1.7 kw/rack Common Items 1W power savings yields 2.84W total WS-C6509-E-FAN (chassis & fan tray) WS-X6K-SUP1A-2GE (supervisor engine) Total possible savings = WS-X G-3C (8-port 10G optical blade w/modules) Copper Blades 4.9 kw/rack 1: WS-X6748-GE-TX (48-port 1G copper) 2: WS-X6548-GE-TX (48 port 1G copper) Optical Blades WS-X6748-SFP (48-port 1G optical)

42 Airflow: Fiber versus Copper The Cascade Effect BECAUSE YOUR BUSINESS RUNS THROUGH US 1 Watt saved at the electronic component level results in cumulative savings of about 2.84 Watts in total consumption 42 42

43 Cable Management 43

44 Cable Management Simplifies system maintenance Extends useful life of system 44

45 Cable Management Incorrect Correct 45

46 Actual Installation Shots Best Practice Cat 6 (KVM, out of band mgmt) Cat 6A (data) 46

47 Agenda Higher Speed Networks Market Drivers and Technology Trends Networking Applications Standards Overview Cabling Options Data Center Cabling Best Practices Airflow and Cable Management Density Questions 47

48 Density: Wire rack-mounted Patch Panels BECAUSE YOUR BUSINESS RUNS THROUGH US 48

49 High Density Fiber Options Modular pre-terminated cassette-based systems Simplified and compact data center cabling MTP TM /MPO and LC available Optimized fiber cable and patch cord management 49

50 Tx Density: Cassette Connectivity Methods Per TIA-568B.1 Addendum 7 NetClear Recommended Method Method A Rx (Enlargement of fiber array) BECAUSE YOUR BUSINESS RUNS THROUGH US Rx Tx 1 Method A Cassette Method A Cassette A to B Patch Cord Type A Array Cable A to A Patch Cord MPO Method A Adaptor 50

Data Center Cabling Best Practices Hot aisle/cold aisle arrangement Hot-air return to CRAC unit Cable management raised floor and rack Innovative rack designs (optimum airflow) Blanking panels

51 Data Center Cabling Best Practices Hot aisle/cold aisle arrangement Hot-air return to CRAC unit Cable management raised floor and rack Innovative rack designs (optimum airflow) Blanking panels for open rack spaces Proper raised floor static pressure Gasket openings Fiber versus copper density, airflow and heat load considerations Thermal modeling BECAUSE YOUR BUSINESS RUNS THROUGH US 51

52 Questions?

DATA CENTER OPTICS ECOC MARKET FOCUS, SEPTEMBER 24, 2013

DATA CENTER OPTICS ECOC MARKET FOCUS, SEPTEMBER 24, 2013 Lisa Huff, CDCP Principal Analyst, Discerning Analytics Chief Technology Analyst, DataCenterStocks.com lisa.huff@discerninganalytics.com AGENDA