100G and Beyond: high-density Ethernet interconnects Kapil Shrikhande Sr. Principal Engineer, CTO Office Force10 Networks MIT MicroPhotonics Center Spring Meeting April 5, 2011 [ 1 ]
Ethernet applications BROADBAND ACCESS, WIRELESS BACKHAUL DATA CENTERS, CONTENT PROVIDERS Broadband Access Networks Data Center Networks Internet Backbone Networks Enterprise Networks Government, Research, HPC Networks Internet exchange and Interconnection Points ENTERPRISE RESEARCH, EDUCATION & GOVERNMENT FACILITIES [ 2 ]
Example data center switches: speeds and densities N x 10G today N x 10/40/100 Gb/s tomorrow 16 x 10G XFP line-card Spine Leaf Modular switch 40 x 10G SFP+ line-card N x 10G today N x 10/40/100 Gb/s links ToR switch 1 Gb/s today 10 Gb/s tomorrow 48 x 10G SFP+ 4 x 40G QSFP+ Link reach (Ethernet optics): few meters to 10s of km Data rates (Ethernet optics): 1G, 10G, 40G, 100Gb/s SFP+: high density form-factor for 10 Gb/s; allows ~400 Gb/s per line-card [ 3 ]
10G Ethernet optics: historical perspective 10GBASE-SR: 300m over OM3 MMF 10GBASE-LR, -ER: 10km, 40km over SMF Others Courtesy, Finisar [ 4 ]
IEEE 802.3 40Gb/s and 100Gb/s Physical Layer Specifications PMD Spec. Description 40GbE 100GbE 40G-KR4 At least 1m backplane, 4 parallel lanes 40G-CR4 100G-CR10 At least 7m Cu (twin-ax) cable, 4 or 10 parallel lanes 40G-SR4 100G-SR10 At least 100m OM3 MMF (150m OM4), 4 or 10 parallel lanes 40G-FR At least 2km SMF, serial 40G-LR4 100G-LR4 At least 10km SMF, 4 channel WDM 100G-ER4 At least 40km SMF, 4 channel WDM [ 5 ]
40/100G Ethernet pluggable modules 12x10G connector use as 4x10G or 10x10G 10x10G or 4x10G parallel optics (2x12 or 1x12 MPO connector) 4x or 4x10G WDM optics (duplex LC connector) 100G SMF / MMF 40G SMF / MMF 12x10G connector use as 10x10G 10x10G parallel optics (2x12 MPO connector) 100G MMF only 4x10G connector 4x10G parallel optics (1x12 MPO) 4x10G WDM optics (duplex LC) 40G SMF / MMF Module electrical interface Module optical interface [ 6 ]
Front-panel density 10/40/100G Line card illustrations a. 48 ports SFP+ @ 10GbE = 480 Gb/s b. 44 ports QSFP @ 40GbE = 1.76 Tb/s c. 32 ports CXP@ 100GbE= 3.2 Tb/s d. 4 ports CFP @ 100GbE= 400 Gb/s 100G MMF using the CXP can allow higher density implementations (if host capacity also scales), but reach limited to 100 m over OM3 100G SMF implementations currently limited by CFP module size and power (density lower than 10G SFP+ based line-card) a b c d Source: 100GbE Electrical Backplane/Cu Cable CFI [ 7 ]
Role of SM optics in data networks Reach of MM optics reducing with each rate jump 1 Gb/s : 500 m 10 Gb/s : 300 m 100 Gb/s : 100 m Application v. reach <100 m for HPC & small-size data 200m to 400m for mid-size data 500m to 2km for Internet Data Centers 2km for carrier client 10km, 40km for campus, metro, aggregation Increasing role of SM optics in data networks [ 8 ]
Need for low cost, high-density SM optics But cost of SM optics generally greater than MM optics Switch face-plate capacity has to scale to meet bandwidth demands doubling of capacity every X years High-density form-factors not available for 100 Gb/s Ethernet SM optics Low cost, high-density 100G SM solutions needed for data networks [ 9 ]
Factors determining line-card capacity (simplified) Switching capacity Module design Fiber type # fibers = L Module Linecard Ics (ASICs, FPGAs) M PHY N Interface ICs K TOSA/ ROSA Fiber Switching, MAC, & PHY functions Electrical Connector PHY Optical Connector (MDI) PHY Electrical path Number of electrical lanes x bit-rate per lane M = lanes on back-end IC (e.g. ASIC) N = lanes on module interface Number of optical lanes (K) x bit rate per lane TOSA, ROSA technology Retimed, partial or non-retimed module interface Interface IC inside module (gearbox, CDR, none) [ 10 ]
100G-LR4 SM CFP module Courtesy, Finisar [ 11 ]
100G-LR4 CFP evolution: Gen1 & Gen2 100BASE-LR4 10x10G 10:4 Mux EMLs, discrete TOSAs and Mux MD MD MD MD EML EML EML EML O-MUX SMF 10x10G 100BASE-LR4 10:4 Mux EMLs or DMLs, discrete or integrated TOSA LD LD LD LD DML DML DML DML O-MUX SMF TECs TEC 10x10G 4:10 De- Mux O- DEMUX SMF 10x10G 4:10 De- Mux O- DEMUX SMF I/O ICs SiGe Gearbox discrete ROSAs and De-mux Gen 1 CFP I/O ICs CMOS Gearbox discrete or integrated ROSA Gen 2 CFP Gen1 CFP: EMLs, discrete optics, SiGe Gearbox (~ 20-24 W, 2010-11) Gen2 CFP: could see subset of changes -- DMLs, CMOS gearbox (< 15 W, ~ 2012-13?) [ 12 ]
Next-gen 100GbE SM module options: using electrical I/O 100BASE-LR4 4x 4x 4x CDR 4x CDR DMLs, discrete or integrated TOSA LD LD LD LD DML DML DML DML TEC O-MUX O- DEMUX SMF SMF 4x 4x 100BASE-nR4 4x buffe r 4x buffe r DML array, integrated TOSA LD LD LD LD DML DML DML DML TEC O-MUX O- DEMUX SMF SMF I/O ICs CDR integrated ROSA Retimed module using I/O I/O ICs Buffers integrated ROSA Un-retimed module using I/O Module with retimed I/O: gearbox replaced with CDRs (< ~12 W, ~ 2013-14?) Module with un-retimed electrical interface: no CDRs (< ~5W, ~2014-15?) Increased use of photonic integration Open question with un-retimed interface: can 100G-LR4 optical budget be met? [ 13 ]
Host architecture using 4x signaling 4x module interface + ASIC interface can enable low-power high-density modules and host systems to scale to higher port counts Back-end ICs / ASIC (<=10G I/O) Back-end ICs / ASIC (<= 10G I/O) 10x10G (CAUI) 10x10G (CAUI) 4x (CAUI-4) Gearbox ICs (10:4) Module 4x MD MD MD MD EML EML EML EML Module WDM Mux WDM De- Mux Gearbox CDR TOSA (10:4) (4:4) ROSA LC / SC Fiber Fiber 100G-LR4 100G-LR4 4x CAUI-4 re-timed I/F CMOS Gearbox CDR Smaller connector Photonic integration Narrower Module (e.g. CFP2) Back-end ICs / ASIC (<= I/O) 4x (CAUI-4) PHY (4:4) 4x (CPPI-4) Buffer Module TOSA ROSA Fiber 100G-nR4, New PMD? 4x CPPI-4 un-retimed I/F ASIC IO PIC TOSA, PIC ROSA Narrower module (e.g. CFP4) Note that the above are example implementations. Other configurations may be possible such as partially retimed modules; un-retimed modules with 10G ASIC I/O; retimed modules with ASIC I/O, etc. [ 14 ]
100G MM optics using 4x Move to 4x optical and 4x electrical optics: reduce number of transceivers from 10 to 4 Reduce number of fibers from 20 to 8 (actually 24 to 12 assuming MPO connectors) CAUI CPPI Module (e.g. CXP) Parallel Fibers Back-end ICs / ASIC (10Gb/s I/O) PHY 10x10G TOSA (10-TX)... ROSA (10-RX)... 2x12 MPO 100G-SR10 Module Back-end ICs / ASIC (b/s I/O) 4x (CAUI4) PHY 4x (CPPI4) 4x TOSA (4-TX) ROSA (4-RX) 1x12 MPO Parallel Fibers 100G-SR4 Figure shows non re-timed module only, but retimed, partially retimed is possible as well. Also, figure shows ASIC IO -- implementations with 10G ASIC IO followed by Gearbox (10:4) also possible [ 15 ]
CFP MSA roadmap Work underway in the CFP MSA to build higher density CFP2 and the CFP4 modules (candidates for higher density 100GbE optics) Courtesy CFP MSA (from Roadmap, 3/4/2011) front-panel densities 400 Gb/s (possible today) 800 Gb/s (201X) 1600 Gb/s (201Y) 3200 Gb/s (201Z) [ 16 ]
What remains? We re at the beginning of the 100G Ethernet optics evolution Enabling technologies 1310nm DMLs [1,2] 850nm VCSELs [3,4] Photonic integration for 40/100G Ethernet optics [5, 6] signaling specifications for chip-chip, chip-module [7] CMOS Gearbox, CDR, capable module connector Various IC and connector companies working on these products Question of reach Can the 100G-LR4 budget be met with a 4x non-retimed interface, or do we need a new optical budget (and reach)? What will be the reach of 4x based 100GbE MMF optics same as 100G-SR10 (100m over OM3), or lower? Potential project in the IEEE to help define necessary specifications to enable high-density, lower cost 100GbE over optical fiber [ 17 ]
Growing bandwidth demand Studies show 40-50% annual growth in Internet traffic 2009 Atlas Internet Observatory Report: http://www.nanog.org/meetings/nanog47/abstracts.php?pt=mtq3nszuyw5vzzq3&nm=nanog47 MINTS http://www.dtc.umn.edu/mints/ Bandwidth measurements from AMS-IX (Amsterdam Internet Exchange) show similar trend Left: peak traffic from 2005-2010 Right: peak and average traffic over > 1 yr (peak crossing 1 Tb/s in Sep-Oct 2010) 1 Tb/s Data Provided by Henk Steenman, Amsterdam Internet Exchange (AMS-IX) [ 18 ]
Rate Mb/s Looking Beyond 100GbE Industry being challenged on two fronts Low cost, high density 100GbE 1,000,000 Source: IEEE 802.3 Next Rate of Ethernet Market Need Data Centers 100,000 Core Networking Doubling 18 mos 100 Gigabit Ethernet 40 Gigabit Ethernet Internet Exchanges 10,000 10 Gigabit Ethernet Carriers The economics of the application will dictate the solution 1,000 Server I/O Doubling 24 mos Gigabit Ethernet 100 1995 2000 2005 2010 2015 2020 Date [ 19 ]
Considering the Options Bit rate, Gb/s Gb/s per Lane Number of lanes increased lane rate advanced modulation 100 10 10 25 4 25 16 400 40 10 1000 50 8 25 40 50 20 more λs more fibres Beyond 100G, technology challenged on both electrical and optical fronts Electrical and optical signaling > 25 Gb/s requires major technological advances Tougher trade-offs between feasibility, power, cost Increased need for photonic integration (E.g. x10, x16, x20 transceivers) [ 20 ]
Considering the Options Open questions How to choose between 400Gb/s, 1Tb/s,? Considerations based on application (line side, client side)? Should Ethernet and OTN align on next rate? When is it needed? Is the network ready for >100G? What lessons to apply from 40G and 100G? What s needed to get us there New technologies, research Industry eco-system & standards [ 21 ]
IEEE 802.3 Ethernet Bandwidth Assessment Ad Hoc Charter and Scope Evaluate Ethernet wireline bandwidth needs of the industry Reference material for a future activity The role of this ad hoc is to gather information, not make recommendations or create a CFI Webpage - http://www.ieee802.org/3/ad_hoc/bwa/index.html Reflector - http://www.ieee802.org/3/ad_hoc/bwa/reflector.html Meetings will be face-to-face and teleconferences. The Ad Hoc needs data. If interested in contributing contact Chair, John D Ambrosia, Force10 Networks (jdambrosia@ieee.org) [ 22 ]
Summary The IEEE 802.3ba 40/100G Ethernet standard was ratified in 2010 Commercial systems using 40/100G Ethernet optics are becoming available At the same time, industry has already started work on the nextgeneration of 100GbE optics Focus on improving density, power and cost of 100GbE optics Number of technical challenges exist, and are being addressed The role of SM optics in data networks is increasing, which puts further pressure to provide high density, low cost SM optics Looking beyond 100G, options on the table are 400G and 1 Tb/s Will present major challenges on all fronts: optical signaling, electrical signaling, integration, packaging, etc. [ 23 ]
APPENDIX [ 24 ]
References 1. DML-based 100G-LR4 CFP demo at OFC 2011, Finisar press release 2. K. Adachi et al, 25-Gb/s Multi-channel 1.3-μm Surface-emitting Laser for Massive Data Links, ECOC 10, Th.9.D.2 3. VCSEL demo by Avago at OFC 2011 4. VCSEL status at T11 technical committee (Finisar) 5. C. Cole at al, Photonic Integration for high-volume, low cost applications, IEEE Communications Magazine, March 2009 6. T. Fujisawa et al, 4 25-Gbit/s, 1.3-um, Monolithically Integrated Light Source for 100-Gbit/s Ethernet, ECOC 10, Th.9.D.1 7. OIF CEI-28G-SR and VSR standards [ 25 ]
Overview of IEEE Std 802.3ba -2010 Interfaces Non-retimed M A C P C S P M A CPPI 100mm Retimed M A C P C S P M A CAUI (chip-to-module) 200mm P M A Combination M A C P C S P M A CAUI (chip-to-chip) 250mm P M A CPPI 100mm M A C P C S P M A CAUI (chip-to-chip) 250mm P M A CAUI (chip-to-module) 200mm P M A [ 26 ]