10GbE and Beyond. Jeff Cain. Joint Techs Council February 13, PDF Free Download

Outline 10GbE Current state of 10GbE Future Network drivers, Form Factors, Projections, Challenges Beyond 10GbE - 100GbE Market Drivers Why 100GbE? 100GbE Standard 100GbE Technologies Summary Wrap Up & Questions 2

A quick look back 10GbE costs are still high against standard benchmark of 3-4x for 10x Rate increase Form factor choice is dictated by system constraints: Power/Thermal Electrical Interface Size Xenpak & X2 will represent ~ 90%+ of Cisco volume for next two years Xenpaks shipping since 2003 X2s started in late 04 4

Transitions Distance Increased Choice of Flavors IEEE PMDS CX4 15m 30m T SR T 75m 100m SX LX LX4 LR LRM 220m 300m 10 km Distance ER 40 km ZX ZR 80 km WDM WDM 2002-04 2006-07 80+ km Time Standard/ Product Available 5

Transceiver PMD Mix - 2006 10GbE was ~ 5% of 2006 (GbE + 10GbE) Fiber ports 13xx lasers increase PMDs (LX4, LR) share over 850 nm sources (SR) Cisco 10GbE dominant form factor was still Xenpak 6

New 10GbE Standards 10GBaseLRM 220m FDDI grade MMF Single laser solution comparable to LX4 First standard to utilize Equalizer Dispersion Compensation Make silicon do the work for the optics This won t be the last time EDC is used to make up channel issues Modules starting to hit marketplace now 7

New 10GbE Standards 10GBaseT 10GbE over unshielded twisted pair Cat6A cable is target for 100m operation Phys now becoming reality NIC adoption this calendar year Thermal challenges abound with first generation 8

Network Architecture 2005 & Today Enterprise & Data Center Access Desktop Wiring Closet Media Cat5 copper 100m 2005 10/100 10/100/1000 2007 10/100/1000 MMF (FDDI/ OM1) 220m to 550m GbE 10GbE Distribution MMF/ SMF 550m/ 10km GbE 10GbE 10GbE Core SMF Distribution MMF/ SMF 550m/ 10km MMF OM2 100m to 300m 10GbE GbE 10GbE 10GbE Access Data Center Server MMF/ Copper 100m 10/100/1000 10/100/1000 10GbE 9

10G Transceiver Drivers & Influence Bandwidth Demand Datacenter Videos need to be stored somewhere Enterprise web applications drive increased server interconnect and storage interconnect using Ethernet Wiring Closet (Campus Buildings) Bandwidth design goals for distribution/core over-subscription ratio Product Availability - Most switch products now using/transitioning to 10G uplinks vs. 1G Per Port Costs 10G Price premium is an inhibitor today This is particularly true for MMF & SMF links up to 10Km 10

Server 10GbE Connection Projections (Based on Blade Server Adoption) Data Center Driving Demand for Higher Speed Ethernet 3 # Servers (M) 2.5 2 1.5 1 0.5 0 2004 2005 2006 2007 2008 2009 2008-2009 ~3M ~3M blade servers = ~500K blade chassis @ 2 X 10GbE uplinks per per chassis = 1M 1M 10GbE ports Year Source: IDC 11

10GbE PMD Challenges for Optics - Today Lower COST Lower SIZE / Higher PORT DENSITY Higher THERMAL / Lower POWER CONSUMPTION 12

10GbE Optical Transceiver Evolution Transceiver Power (W) 1 4 Year 2004 2006 2008 2010 XFP X2 SFP+ Xenpak 6-8 4-6 8-16 32 48 Max 10 GbE ports per line card Higher Port Counts (4 48) & Lower Power (6W 1W) 13

SFP+ Board/ Host SERDES/CDR + EDC SFI 1 x 8.5G 1x 10.3125 1x 10.5188 1x 11.09 Optic Module LD + TOSA TIA + ROSA SFP+ MOTIVATION Maximize port density per blade, minimize power, optimize cost PERFORMANCE Smallest 10GbE form factor Power Class 1: < 1 W ( multi-cage designs) Class 2 : 1.5W Rx interface: Linear Rx interface: Supports 10GBASE-LRM products with host-based EDC Limiting Rx interface: supports all other PMD types COST Most electronics out of optics to host board IC IC integration lowers costs; IC on host board enables quad/octal Serdes Higher Density, Lower Power, Lower Cost 14

Cisco 10 GbE Mix Forecast 2007 & Beyond 1GbE-10GbE Transceivers Unit Mix 100% 80% 60% 40% 10GbE 1GbE 20% 0% 2006 2007 2008 2009 2010 2011 10GbE MMF Transceiver Cost Curves Relative cost to 1GbE SX 30 25 20 15 10 5 0 2006 2007 2008 2009 2010 2011 15

10GbE Summary Market growth driven by use of 10GbE in datacenters 13xx nm PMDs (10G-LRM, LX4, LR) increasingly significant in mix 850 nm PMDs (10G-SR) still significant in mix distance good enough for datacenter Form Factors: SFP+: Routers XFP Switching X2, SFP+ Density, Power, Cost Enable significant 10GbE market adoption by lowering the cost of links <10Km Project ~50% share of Cisco 10GbE ports in 2010 16

Driving Bandwidth Requirements Consumers Increasing penetration to home Increasing bandwidth needs Content Increasing bandwidth requirements Networking Pulling it all together Service Providers Other Data Centers High Performance Computing 18

Network Impact 19

Applications Consumer & Business Driven All Devices Networked PCs, Desktop Phones, Cell Phones, Radios, TVs, Security Cameras, Vehicles, Sensors, Actuators, Doors, portable gadgets Everything on Ethernet Wired, Wireless.. Everything over IP Data, voice, video, wireless 20

Content Providers Personalized Video content is a killer application Google, Yahoo!, MSN, & YouTube deliver personalized content via high BW Yahoo! (Source: Adam Bechtel, Yahoo!, The Need for 100GE, Designcon 2006 Management Forum Panel) Ethernet everywhere 10 GbE used in data center, metro and WAN Q106: already using 6 x 10 GbE and 4 x 10 GbE LAG Yahoo! Asia Pacific (source: Chris Choi, Yahoo! Asia Pacific) 40+ Gbps aggregate bandwidth need to support Major League Baseball streaming in Asia Pacific YouTube (Source: Colin Corbett, YouTube, Peering of Video, NANOG 37) Personal video site, >50 million videos viewed per day ( >20 Gbps traffic) ~20% monthly traffic growth, all unicast traffic Initiatives for online DVD rentals by NetFlix and Amazon 21

Market Drivers - Increased Bandwidth Content Providers Video on demand Consumer & Broadband Access Service Providers & IXCs Data Center High Performance Computing 22

High Performance Computing Average Gigaflops per second (top 500, '93 - '05) 10000000 1000000 100000 10000 1000 100 10 1 50K to 110K GF/s- HSSG Deployment 10GE Std GE Std 7000 GF/s- 10GE Deployment 585 GF/s- GE Deployment 94 96 98 00 02 04 06 08 10 12 14 Year Source - Top500.org Historical Observations 4 years between standard and deployment GE captures 50% of market in 3 years (2002-2005) 10GE enters top500 in 2006 (included in Gigabit Ethernet # s) 12x increase in Avg Gigaflops: 10x increase in Ethernet Interconnect Forecast 10x increase needed at 84,000 Gigaflops (2010) 23

Use of parallel nx10gbps links Effectiveness of load distribution depends on Hash algorithm Number of flows/size of flows Diversity of hash input (variability in SRC/DEST IP address) Flow-based hash mechanism CANNOT guarantee equal distribution of load or QOS Un-equal load distribution can results in under-utilization of available capacity May potentially cause artificial congestion and packet loss 10x10Gbps is not the same as 1x100Gbps from a real throughput perspective How many parallel 10Gbps links needed to match usable BW of a 100Gbps link? Depends on traffic characteristics 24

Why 100GbE now? The need for a higher speed bandwidth link beyond 10Gb is already emerging today Standards are required to gain broad market acceptance for any high speed Ethernet beyond 10GbE Standards are time consuming Realistically, estimate new standard completion by late 09 Need a solution consistent with the technology avail in the timeframe when the standard is completed 100Gb is well aligned to future ASIC switching speed capability Given limited industry resources, it is better to focus on 100GbE for 09 release 25

Possible Timeline for 100GbE Standard TASK FORCE Formed WG Ballot HSSG Formed PAR Approved TF Review LMSC Ballot STD!!! 2006 2007 2008 2009 2010 J U L S E P N O V J A N M A R M A Y J U L S E P N O V J A N M A R M A Y J U L S E P N O V J A N M A R M A Y J U L S E P N O V J A N M A R M A Y J U L S E P N O V Objectives Drafted D 1 D 2 D 3 D 4 D 5 PAR Drafted Last New Proposal Last Feature Last Technical Change 26

Standard Considerations Define small number of PMDs Avoid the proliferation as with 10GbE Optimize Standard for Technology Advances / Time to Market / Cost 27

Progress! Six objectives were adopted by the Study Group: Support full-duplex operation only Preserve the 802.3/Ethernet frame format at the MAC Client service interface Preserve current 802.3 Std minimum and maximum FrameSize Support a speed of 100 Gb/s at the MAC/PLS service interface Support at least 10km on SMF Support at least 100 meters on OM3 MMF This set of objectives could be considered enough to move forward with a project request 28

Defining interfaces Standards goal is to leverage existing technologies while being flexible enough to easily incorporate future emerging technologies MAC PCS PMA PMD 10λ @ 10G 5λ @ 20G 4λ @ 25G 2λ @ 50G 1λ @ 100G? Need a universal PCS and a single common interface to optics module 29

Observations about interface rates Data Rate (Gbps) 1000 100 10 1 0.1 Network and Router I/F Speeds (Gbit/s) 140M TDM Transport 565M OC48 POS OC3 OC192 OC192 OC12 OC48 1GE Ethernet 0.01 80 85 90 95 00 05 Year of Deployment OC768 10GE OC768 10 Going back over last 25 years, SONET TDM led introduction of high speed BW technology into network Over time, the gap between introduction of high speed BW into the SONET TDM and Ethernet network has shrunk Beyond 10G, Ethernet data interfaces are likely to be the push for higher BW optical technology 30

Study Group PMD objectives Support at least 10km on SMF single mode fiber support considered a requirement actual distance will be a trade-off between reach, cost, complexity Support at least 100 meters on OM3 MMF parallel optical technology will likely be the lowest cost implementation belief that the cost benefits will outweigh the operational challenges of the ribbon fiber 31

10GbE+ - 100GbE Summary Bandwidth bottlenecks are real and increasing Standards are required to gain broad market acceptance for any high speed Ethernet beyond 10GbE Today, SONET OC-768 is the only standard beyond 10Gb 100Gb is better aligned to future ASIC switching speed capability and network bandwidth needs than lower rates Given pace of standards and the development resources needed for new PMDs it is best to focus on 100GbE for 09 release 32

Thank you! Questions? 33

10GbE and Beyond. Jeff Cain. Joint Techs Council February 13, 2007