Robotron: Top-down Network Management at. Yu-Wei Eric Sung, Xiaozheng Tie, Starsky H.Y. Wong, Hongyi Zeng ACM SIGCOMM 2016 August 25, 2016

Transcription

1

2 Robotron: Top-down Network Management at Scale Yu-Wei Eric Sung, Xiaozheng Tie, Starsky H.Y. Wong, Hongyi Zeng ACM SIGCOMM 2016 August 25, 2016

3 Scale of Facebook Community 1.7 Billion 1 Billion 500 Million on Facebook Monthly on Whatsapp Monthly on Instagram Monthly 1 Billion on Messenger Monthly

4 Network Management at Facebook... What s involved? R Goals: Build and evolve FB network Example tasks: circuit/device turnup, network monitoring Human interactions -> outages R `......

5 Network Management at Facebook Why is it hard? Distributed Configurations Multiple Domains Versioning Dependency Vendor Differences

6 Network Management at Facebook Early days Manual Configuration and Monitoring with ad-hoc scripts

7 Contribution Robotron started Manual Configuration and Monitoring with ad-hoc scripts Our Paper Shed light on Network management tasks Robotron s usage Evolution of Roboron Our experiences using Robotron

8 Overview of Facebook s Network Lifecycle of user requests Users Internet POPs Backbone Data Centers

9 Point of Presence (POP) Standardized topology Services: LB, Cache Common tasks Build/upgrade a cluster Provisioning new peering circuits Users Internet POPs Backbone Data Centers

10 Backbone Irregular, demand-driven topology Common tasks: Add/migrate circuits Add/remove routers Users Internet POPs Backbone Data Centers

11 Datacenter Standardized topology Services: Web, Cache, Database Common tasks Build/decomm a cluster Cluster capacity upgrade Users Internet POPs Backbone Data Centers

12 (normalized) Overview of Facebook s Network Multiple versions of FB cluster architectures co-exist POP DC # of clusters (normalized # of clusters (normalized) Gen2 Gen1 Gen3V6 Gen3 Gen2V6 Gen2-D Gen2-C Gen2-B Gen2-A Gen1 Time 8 generations Time

13 Robotron: Top-Down Network Management Overview Network Design Config Generation Deployment Monitoring FBNet DB

14 FBNet: Modeling the Network Example 4-post POP cluster BB 1 BB 2 Internet PR 1 PR 2 20G 4-post POP PSW a PSW b PSW c PSW d Cluster To Top-of-Rack switches & servers

15 FBNet: Modeling the Network Object 2001::1 2001::2 ae0 ae1 et1/1 10G et2/1 Circuit PSW a et1/2 et3/1 PR 1 PhysicalInterface 10G AggregatedInterface ebgp session Linecard Circuit Networkswitch Linecard PhysicalInterface V6Prefix BgpV6Session Circuit

16 FBNet: Modeling the Network Value 2001::1 2001::2 ae0 ae1 et1/1 10G et2/1 Circuit PSW a et1/2 10G et3/1 PR 1 PhysicalInterface name=et1/1 AggregatedInterface speed=10g ebgp session Linecard Circuit Networkswitch name=psw a Linecard slot=1 model=x PhysicalInterface name=et1/2 name=ae0 V6Prefix prefix=2001::1 BgpV6Session Circuit speed=10g

17 FBNet: Modeling the Network Relationship PSW a ae0 ebgp session Linecard Circuit 2001::1 2001::2 et1/1 et1/2 10G 10G ae1 et2/1 et3/1 Networkswitch name=psw a PR 1 Linecard slot=1 model=x device= PhysicalInterface name=et1/1 linecard= agg_interface= It s complicated PhysicalInterface name=et1/2 agg_interface= linecard= AggregatedInterface name=ae0 V6Prefix prefix=2001::1 interface= Circuit a_endpoint= z_endpoint= speed=10g BgpV6Session a_prefix= z_prefix= Circuit a_endpoint= z_endpoint= speed=10g

18 FBNet Model Snippet class PhysicalInterface(Interface): linecard = models.foreignkey(linecard) agg_interface = models.foreignkey( AggregatedInterface)

19 FBNet Model Snippet Related models class PhysicalInterface(Interface): linecard = models.foreignkey(linecard) agg_interface = models.foreignkey( AggregatedInterface)

20 FBNet Model Snippet Model inheritance class PhysicalInterface(Interface): linecard = models.foreignkey(linecard) agg_interface = models.foreignkey( AggregatedInterface)

21 FBNet: Architecture API Layer RPC services Read: fine-grained permodel query Write: task-based High Availability: Multiple replicas per DC Read API Read API Write Service Read API Read API Read Service FBNet

22 FBNet: Architecture API Layer 1 primary, multiple secondary DBs Scalability: 1 slave per DC Read API Read API Write Service Read API Read API Read Service Primary Slave Slave Secondary Replication Stream FBNet

23 Robotron s management life cycle Network Design Config Generation Deployment Monitoring FBNet DB

24 Network Design Design intent à FBNet objects Template for a POP cluster Cluster( devices={ PR: DeviceSpec( hardware= Router_Vendor1 num_devices=2) PSW: DeviceSpec( hardware= Switch_Vendor2 num_devices=4) }, Link_groups=[ LinkGroup( a_device=pr, z_device=psw, pifs_per_agg=2, ip=v6) ] ) FBNet objects PR 1 PR 2 PSW a PSW b PSW c PSW d BackboneRouters: 2 NetworkSwitches: 4 Circuits: 16 PhysicalInterfaces: 32 AggregatedInterfaces: 16 V6Prefixes: 16 BgpV6Sessions: 8 94 objects across 7 models

25 Config Generation FBNet objects à Device configs FBNet PR 1 PR 2 struct Device { 1: list<aggregatedinterface> aggs, } struct AggregatedInterface { 1: string name, 2: i32 number, 3: string v4_prefix, 4: string v6_prefix, 5: list<physicalinterface> pifs, } struct PhysicalInterface { 1: string name, } PR 1 PSW a PSW b PSW c PSW d Config Schema PR 2 PSW a PSW b PSW c PSW d FBNet objects Per-device objects Vendor agnostic

26 Config Generation FBNet objects à Device configs FBNet PR 1 PR 2 PSW a PSW b PSW c PSW d FBNet objects {% for agg in device.aggs %} interface {{agg.name}} mtu 9192 no switchport load-interval 30 {% if agg.v4_prefix %} ip addr {{agg.v4_prefix}} {% endif %} {% if agg.v6_prefix %} ipv6 addr {{agg.v6_prefix}} {% endif %} no shutdown! {% endfor %} PR 1 Config Schema PSW Vendor 1 c PSW d Vendor 2 interface template BGP template MPLS template PR 1 config PR 2 config PR 2 PSW a PSW b interface template BGP template MPLS template PSW a config PSW c config PSW b config PSW d config Per-device objects Vendor-specific Device Configs Vendor agnostic Vendor Specific

27 Config Generation FBNet objects à Device configs FBNet PR 1 PR 2 PSW a PSW b PSW c PSW d FBNet objects {% for agg in device.aggs %} interface {{agg.name}} mtu 9192 no switchport load-interval 30 {% if agg.v4_prefix %} ip addr {{agg.v4_prefix}} {% endif %} {% if agg.v6_prefix %} ipv6 addr {{agg.v6_prefix}} {% endif %} no shutdown! {% endfor %} PR 1 Config Schema PSW Vendor 1 c PSW d Vendor 2 interface template BGP template MPLS template PR 1 config PR 2 config PR 2 PSW a PSW b interface template BGP template MPLS template PSW a config PSW c config PSW b config PSW d config Per-device objects Vendor-specific Device Configs Vendor agnostic Vendor Specific

28 Config Generation FBNet objects à Device configs FBNet PR 1 PR 2 PSW a PSW b PSW c PSW d FBNet objects {% for agg in device.aggs %} interface {{agg.name}} mtu 9192 no switchport load-interval 30 {% if agg.v4_prefix %} ip addr {{agg.v4_prefix}} {% endif %} {% if agg.v6_prefix %} ipv6 addr {{agg.v6_prefix}} {% endif %} no shutdown! {% endfor %} PR 1 Config Schema PSW Vendor 1 c PSW d Vendor 2 interface template BGP template MPLS template PR 1 config PR 2 config PR 2 PSW a PSW b interface template BGP template MPLS template PSW a config PSW c config PSW b config PSW d config Per-device objects Vendor-specific Device Configs Vendor agnostic Vendor Specific

29 Config Generation FBNet objects à Device configs FBNet PR 1 PR 2 PSW a PSW b PSW c PSW d FBNet objects {% for agg in device.aggs %} interface {{agg.name}} mtu 9192 no switchport load-interval 30 {% if agg.v4_prefix %} ip addr {{agg.v4_prefix}} {% endif %} {% if agg.v6_prefix %} ipv6 addr {{agg.v6_prefix}} {% endif %} no shutdown! {% endfor %} PR 1 Config Schema PSW Vendor 1 c PSW d Vendor 2 interface template BGP template MPLS template PR 1 config PR 2 config PR 2 PSW a PSW b interface template BGP template MPLS template PSW a config PSW c config PSW b config PSW d config Per-device objects Vendor-specific Device Configs Vendor agnostic Vendor Specific

30 Usage Statistics # of FBNet model change? # changed FBNet objects per design change? Frequency and size of config change?

31 FBNet Model Changes How much does FBNet model change over time? Still many changes over time Reasons: new models, values, relationships

32 Design Changes How many FBNet object are changed per design change? POP/DC CDF across design changes ,000 10,000 # of FBNet objects All Interface Circuit v6 Prefix v4 Prefix Device Backbone CDF across design changes ,000 10,000 # of FBNet objects All Interface Circuit v6 Prefix v4 Prefix Device

33 Design Changes How many FBNet object are changed per design change? POP/DC Backbone CDF across design changes CDF across design changes All 0.5 Interface Circuit 0.25 v6 Prefix v4 Prefix Device 0 POP/DC: 1 bigger 10 design 100 changes 1,000 10,000 # of FBNet objects 1 Backbone: smaller design changes ,000 10,000 # of FBNet objects All Interface Circuit v6 Prefix v4 Prefix Device

34 Configuration Changes What s the frequency and size of configuration change? Median number of config lines changed per week POP/DC devices: 500 lines Backbone devices: <100 lines POP/DC: few bigger config changes Avg number Backbone: of times many changes smaller happen configper changes week POP/DC devices: 2.53 Backbone devices: 12.46

35 Evolution of Robotron Bottom-up, experience driven FBNet modeling started Active monitoring Passive monitoring Basic Deployment Basic design and config generation Robotron

36 Experience: Modeling is laborious Problem Scenario: new ebgp session configuration A new ebgp session needed a proper import policy Lesson: Modeling is hard Robotron was used without proper support à egress link Open problem: Lack of a network model saturated widely accepted by vendors Most development time spent on model changes

37 Experience: Coupling changes is key Problem Scenario: POP cluster switch turnup 1. An engineer Lesson: updated Network FBNet design, to add config a new generation rack, but forgot to generate and config deployment should be tightly coupled 2. The engineer Open pushed problem: stale config 3. The rack added never came online Atomicity Conflict resolution

38 Experience: Fallback is important Problem Scenario: Robotron-less management Engineer Lesson: bypassed Bypassing Robotron mechanism to manually is configure needed devices SSH into device Open problem: Make config How change to reliably account for such Log out activities? Needed upon How emergencies to safely revert such activities? Passively curtail with config monitoring

39 Conclusion First work sharing experience on a production network management system Open research problems: Network modeling Atomicity and conflict resolution across management tasks Make network management system work with manual fallback mechanisms

40 Questions? Poster session on Thursday

41 Overview of Facebook s Network Backbone: Interconnecting POPs/DCs Irregular, demand-driven topology PRs/DRs form an ibgp mesh Common tasks: Add/migrate circuits Add/remove BBs/PRs/DRs DR 1 To DCs DR 2 BB BB BB BB BB BB BB PR 1 PR 2 To POPs & Internet

42 Overview of Facebook s Network Point of Presence (POP) Standardized topology Internet BB 1 BB 2 Services: LB (Proxygen), Cache PR 1 PR 2 Common tasks Build/upgrade a cluster Provisioning new peering circuits POP Clusters

43 Overview of Facebook s Network Data Center Standardized topology Services: Web, Cache (TAO), BB 3 BB 4 Database DR 1 DR 2 Common tasks Build/decomm a cluster Cluster capacity upgrade DC Clusters

44 FBNet: Modeling the Network Object, Value, and Relationship 2001::1 2001::2 ae0 ae1 et1/1 10G et2/1 Circuit PSW a ebgp session Linecard Circuit et1/2 10G et3/1 Networkswitch name=psw a PR 1 Linecard slot=1 model=x device= PhysicalInterface name=et1/1 linecard= agg_interface= PhysicalInterface name=et1/2 agg_interface= linecard= AggregatedInterface name=ae0 V6Prefix prefix=2001::1 interface= a_endpoint= z_endpoint= speed=10g BgpV6Session a_prefix= z_prefix= Circuit a_endpoint= z_endpoint= speed=10g

45 Dependencies between FBNet models CDF across models # of related models

46 Experience: Fallback is needed Problem Scenario: manual changes to devices Manual config changes on devices are error-prone Ideal: All changes made through Robotron Reality: Robotron has latency, bugs and missing features. Quick fixes needed upon emergency Alternatives to discourage manual changes: Config monitoring Automatic config override after emergency window

47 Related Work Bottom-up config analysis: [Benson11,Sung09,Kim11, ] Abstraction-driven design and config generation: Top down config optimization: [Condor, Sun13] Centralized platform for network management: [Onix, Statesman] Template based config generation: [Enck09]

48 FBNet: Modeling the Network Desired versus Derived Desired Derived A =? A B C B C FBNet

49 Deployment Device configs à Devices New device: full config replacement Existing devices: Incremental Live updates Dryrun, Atomic, Phased, etc

50 Monitoring Is the network healthy? Passive monitoring Active monitoring Config monitoring

51