Sky Core Network Positioning for Massive Video Demand UKNOF - 17 Jan Tim Rossiter Core Network Architect, Sky Network Services

Transcription

1 Sky Network Positioning for Massive Video Demand UKNOF - 17 Jan 2018 Tim Rossiter Network Architect, Sky Network Services

2 Positioning Sky Network for Massive Video Demand UK Broadband Usage and Pricing Sky Network Structure and Evolution 2017/18 Capacity Upgrade Cost Reduction by Content Delivery () Localisation Sky Network, UKNOF, 17Jan

3 Uk & International Historic Traffic Trends YoY Growth USA 20% UK 58% 90% in % in 2017 Source: Ofcom International & UK Communication Market Reports Sky Network, UKNOF, 17Jan

4 Volume - Sky Traffic Growth (10 Year History) Sky Experience Similar to UK Total Traffic x100 over 10 years => 58% YoY Per Sub traffic x25 => 38% YoY Sky Network, UKNOF, 17Jan

5 UK Broadband Price Trend Since 2000 Internet connectivity cost ~ p.m (also inflation ~50% ) Access Data rates increased x 100 in 12 years Cost per bit Reduces x 67% YoY, or 10 ->1 in 6 years. ISP per sub revenue unchanged, Access rate increased by x100 Sky Network, UKNOF, 17Jan

6 Sky Network Structure (Simplified) P Router NCS6k CPE ange DSL or GEA Rem PoP P&T P&T Sky Network, UKNOF, 17Jan

7 Resilience and Provisioning Policy Resilience or 1 : N Dual Rings, Dual feed to each PoP Dual P&T sites Multiple (1:N ) 2 or 3 Entry points to each - site Dimensioning Serve peak traffic in the event of a single major failure Major failure: Loss of one entire site or fibre (up to 80 layer 3 paths) Capacity Consequences Largest Router Capacity Traffic Served ( In+Out ) x Protection + Interconnect + Extras Partial Fill 8 Tb Traffic Requires 8 Tb Router Sky Network, UKNOF, 17Jan

8 Router Evolution 2005 Initial Cisco CRS1 Traffic < 50G, 40Gb per slot => 640 Gb chassis 2011 Upgrade to CRS3. Fabric, Line cards Traffic 350G, 140Gb per slot => 2240 Gb chassis (x3.5) CRS1 CRS3 NCS6k NCS5k NCS6k B2B Replace with NCS6008 -> New Chassis, Fork Lift Traffic 2 Tb, 8Tb per chassis (x12.5) 2017/18 - Add NCS5516 -> New Chassis Largest NCS6008 almost full Traffic ~ 8 Tb, 55 Tb per chassis (x 90) Retain NCS6008 in Back to Back Config, 16 Tb per chassis Sky Network, UKNOF, 17Jan

9 NCS6008 and NCS5516 NCS Slots, 1Tb per slot => 8 Tb node 100G or ( 10x10G) ports High Functionality Roadmap: B2B, 2Tb per slot => Tb NCS Slots, 3.6Tb per slot => 55 Tb node 36 x ( 100G or (4 x 10G) ) ports/slot Low touch MPLS P-Router, No B2B Sky Network, UKNOF, 17Jan

10 NCS6008 Back to Back 96 x 100G CXP SR10 Connectors => ~1000 Fibre Pairs! 2 x 40G Ethernet for processor links Shelves can be ~ 100m apart Sky Network, UKNOF, 17Jan

11 100G Optics CFP, CPAK, QSFP CFP Original 100G Bulky, Expensive CPAK, Cisco Proprietary CXP SR10 (10x10G) CPAK QSFP-28 (4x25G) Standard SR4, LR4 + Cost Effective Variants Large Quantities (Data Centre Usage) SR-4 very low cost Sky Network, UKNOF, 17Jan

12 Matching Broadband Cost Trend? Port savings: 10G -> 100G, Router silicon, Moore s Law, Reduced Functionality, LR->SR NCS low cost QSFP SR4 nearly matches BB price erosion, 10 : 1 in 6 years Sky Network, UKNOF, 17Jan

13 Migration Steps for NCS5516 in -core Different approach from previous NCS6008 whole ring introduction Growth slower => Longer timescale for transitions No 10G -> 100G transition issues -core transmission fully 100G Not a big bang migration Upgrade one node at a time (in multiple stages) Birmingham first Staged migration modelled to verify capacity for all failures Sky Network, UKNOF, 17Jan

14 Migration 1: Insert NCS65516 P Router NCS6k NCS5k CPE Birmingham P&T P&T Sky Network, UKNOF, 17Jan

15 Migration 2a: Migrate some feeds and consumers P Router NCS6k NCS5k CPE P&T P&T Sky Network, UKNOF, 17Jan

16 Migration 2b, 2c : Migrate remaining feeds and consumers P Router NCS6k NCS5k CPE P&T P&T Sky Network, UKNOF, 17Jan

17 Migration 3: Isolate Original NCS6008 & Decom P Router NCS6k NCS5k CPE P&T P&T Sky Network, UKNOF, 17Jan

18 Migration 5: Old NCS6k Makes B2B NC6k on Other Ring P Router NCS6k NCS5k CPE NCS6008 B2B 16 Tb with 1Tb cards SR4 CPAK Accepts 2Tb cards (-> 32 Tb) QSFP/CPAK P&T P&T Sky Network, UKNOF, 17Jan

19 Migration Waterfall Jul 17 Oct 17 Jan 18 Apr 18 Jul 18 pr2 Lon1 N6k B2B Leeds N6k B2B Birmingham N6k B2B Lon2 N6k B2B pr3 Lon1 NCS5500 Leeds NCS5500 Birmingham NCS5500 Lon2 NCS5500 Red triangles: Slot exhaust time, Green cells: Quarter for upgrade NCS6k replaced by new NCS5k Released NCS6k chassis incorporated into NCS6k B2B configuration Staged operation, One upgrade per quarter

20 Capacity Issue Sorted New NCS5516 Ring sufficient for 5+ years Remaining NCS6008 B2B exhaust earlier, Replace with NCS5516? New Cost point with NCS5516 silicon, low-touch and SR4 QSFP Router port price trend close to Broadband price Trends Any problems? Sky Network, UKNOF, 17Jan

21 Transponder Price Trend Transponder price (per Gb) falling at half rate of router port cost ( 10:1 in 12 years) 10 Gb Easy (in retrospect) => On-Off Modulation, FEC 100G Very Complex => QAM Modulation, Coherent Detection ( 2 Detectors I&Q ) Dual Polarisation ( 2 lasers, 2 x 2 Detectors: H&V x I&Q ), Ultra High Speed A/D Converters Dispersion Compensation DSP, Stronger FEC Distance is becoming (relatively) more expensive Localise Traffic Sources Sky Network, UKNOF, 17Jan

22 How to achieve further cost reductions? Most traffic originates from (On-Net or Off-Net) 1. Private Peering ( v.s. Transit and Public Peering) 2. On-Net - 65% of traffic sourced On-Net 3. Localise Source How much can be Localised to minimise carriage across network? Practicalities Matter Diminishing Returns, Long Tails Sky Network, UKNOF, 17Jan

23 Deployment Options Off-Net via P&T P Router CPE Cache At least 4 transmission links (Including Protection) Peering Interconnects Multiple Router Transitions P&T P&T OffNet OffNet Sky Network, UKNOF, 17Jan

24 Deployment On-Net, Remote CPE Multiple transmission links Fewer Router Transitions Rem P&T Sky Network, UKNOF, 17Jan

25 Deployment On-Net, CPE Minimal Links Traffic still on PoP Links Constraints Space Limited Expensive Power restrictions Sky Network, UKNOF, 17Jan

26 Deployment Options PoP CPE PoP No Primary Traffic on or PoP links If local fails, sufficient capacity on PoP links for central fall-back Requires per PoP mapping to localise traffic source Sky Network, UKNOF, 17Jan

27 Deployment PoP feeding remote PoP CPE Local serves other PoPs Upstream for free PoP Sky Network, UKNOF, 17Jan

28 Deployment ange, Set Top Box STB PoP CPE ange Problematic Insufficient traffic Power/Space constraint BNG architecture Home Pre-Caching? Pre-load by satellite side channel Limited application - For very popular material Sky Network, UKNOF, 17Jan

29 PoP Early Deployment Low Traffic PoPs, A few large + Long tail 5 larger providers (+ many more off-net) Representative PoP sizing ( actually 60 PoPs) Provider Fraction Prov1 21% Prov2 17% Prov3 13% Prov4 6% Prov5 5% Need > 10Gb to warrant PoP => Green Cells Early Deployment: 9 Locations, 31% Localised Other Locations ( < 10G localised) serve Local+Other PoPs. Sky Network, UKNOF, 17Jan

30 PoP Higher Traffic Levels Double Traffic => 27 deployments, 58% of traffic Provider Fraction Prov1 21% Prov2 17% Prov3 13% Prov4 6% Prov5 5% Further Growth: 4 x Traffic => 63 Deployments, 88% Localised 130% More Deployments, 50% More traffic localised Diminishing Returns Provider Fraction Prov1 21% Prov2 17% Prov3 13% Prov4 6% Prov5 5% Sky Network, UKNOF, 17Jan

31 Localisation History Easy to see Benefits of PoP Not so easy to realise them Sky Specifies Localisation BBPE & IPv6 Enhancements 5 x providers Each with own approach DNS, HTTP Redirect, + Secret Sauce All or nothing Specify Local and Central Mapping Allow headroom for overflow BBPE Software Upgrades Rolled out to 100 BBPEs Separate BBPE features for IPv6 Provider Localisation Sky Network, UKNOF, 17Jan

32 IPv6 Deployment Dual Stack alongside IPv4 (eg. Not DS-Lite) Both LLU (PPP) and Fibre (IPoE) services eligible > 5 Million customers eligible and enabled (>95%) Support all Sky Hub models (1,2,3) and legacy Sky-supplied Sagem routers. > 90% uptake. (Remaining assumed to be user-disabled and/or 3 rd party an or very old routers without compatible IPv6 support) New customer default since April 2016 /56 Prefix Delegation 2 year exercise 20% -> 90% Sky Network, UKNOF, 17Jan

33 Acknowlegements Rapid, well executed program through 2017 Made Possible by Great Team within Sky Networks Network Architecture & Planning - Forecast, Modelling, Migration Planning Network Development Testing, Verification, Product Advice Network Implementation - Configuration, Installation Network Operations - Many overnight migrations Project Mgt - Complex Project, Juggling Resources Cisco - Early testing support, Expedited delivery Sky Network, UKNOF, 17Jan

34 Sky Network Positionined for Massive Video Demand Thank you! Storm Surfers