Stateless Multicast with Bit Indexed Explicit Replication

Stateless Multicast with Bit Indexed Explicit Replication IJsbrand Wijnands, Distinguished Engineer

Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#

Agenda Introduction Solution Overview Encapsulation Sets and Areas Forwarding ECMP BIER Overlay Deployment Scenarios Conclusion

Introduction

BIER history A team was formed to investigate solutions for multicast in the context of Segment Routing. Encoding a Sourced routed Multicast tree path using MPLS labels is difficult. The packet header would get very large, and its very hard to parse such header. 6

The BIER Epiphany Only encode the end-receivers (as a Bit Position) in the packet header. Not the intermediate nodes. Encode the Bit String in the packet header. Using some sort of encapsulation. Create a Bit Forwarding Table on all BIER nodes to allow multicast packet forwarding using the Bit String in the packet. Derived from the RIB, SPF based. We call it, Bit Indexed Explicit Replication (BIER). 8

IETF The BIER idea was presented in a BOF at the IETF in Hawaii. November 2014. A new BIER Working Group has been formed (bier@ietf.org) Vendors collaborating (co-authoring) with us; 9

IETF drafts draft-ietf-bier-problem-statement draft-ietf-bier-architecture (now rfc8279) draft-ietf-bier-encapsulation-mpls (now rfc8296) draft-ietf-bier-use-cases draft-ietf-l3vpn-mvpn-bier draft-ietf-ospf-bier-extensions draft-ietf-bier-isis-extensions draft-xu-idr-bier-extensions 10

Solution Overview

What is the problem with Multicast Each Tree has its own unique receiver population. To efficiently forward/replicate there is a Tree per flow! This means State is created in the network. S5 S4 S3 S2 S1 12

Multicast Routing State State is created in the network using a Multicast routing protocol like PIM, mldp, RSVP-TE. State means resources consumed, memory/cpu. Convergence is impacted by the amount of State. In order to manage the network, the protocol needs to be understood by the network operator. Different levels of complexity based on protocol choice. 13

What is BIER BIER is a new forwarding paradigm to forward and replication multicast packets through the network. Packets are forwarded using a special Header that is embedded into the packet. Routers build a special forwarding table to forward/replicate using the BIER header. BIER forwarding is State Less! 14

How does BIER work We give the Egress routers an identifier. The Ingress router includes the identifiers in the packet. Egress 1 Ingress 2 IP Mcast 1, 2, 3, 4 3 4 15

How does BIER work Packet is forwarded hop-by-hop using the identifier Each identifier is forwarded along the unicast (SPF) path. Egress 1 IP Mcast Ingress 2 IP Mcast IP Mcast 1, 2, 3, 4 3 IP Mcast 4 IP Mcast 16

How does BIER work The smaller the identifier, the more we can fit into a single packet, how small can an identifier be? A single Bit!!! With BIER the Egress Identifier is a Bit Position. We include a BitString of 256 bits into the packet. Manipulations of a BitString is much easier compared to including a list of numbers. 17

How does BIER work Each Egress BIER router has a unique Bit Position. Routers maintain a forwarding table of Bitmask's Egress Ingress BIER Fwd TBL Nbr BM E 0011 F 1100 E 0001 0010 IP Mcast IP Mcast IP Mcast 1111 F 0100 IP Mcast 1000 IP Mcast 18

Bit Index Forwarding Table BM Nbr BM Nbr BM Nbr 0111 B A 0011 C 0100 E B 0001 D 0010 F C D BFR-ID 1 BS:0001 BM-ER BM Nbr 0011 C B E BFR-ID 3 BS:0100 BM-ER F BFR-ID 2 BS:0010 BM-ER Based on shortest path route to RID, the Bit Mask Forwarding Table is created 19

Forwarding Packets BM Nbr BM Nbr BM Nbr AND 0001 0111 B A &0111 0011 C &0011 0001 D &0001 AND AND 0100 E 0010 F 0001 B 0001 C D BFR-ID 1 BS:0001 Suppose A leans about D s interest, in the blue multicast flow. (via BGP, SDN, STATIC, etc ) Nbr 0011 C E BFR-ID 3 BS:0100 F BFR-ID 2 BS:0010 B 20

Forwarding Packets Nbr Nbr Nbr AND 0101 0111 B A &0111 0011 C &0011 0001 D &0001 AND AND 0100 E &0100 0010 F 0101 B 0001 C D BFR-ID 1 BS:0001 Suppose A leans about D and E s interest, in the blue multicast flow. (via BGP, SDN, STATIC, etc ) AND Nbr 0011 C B E BFR-ID 3 BS:0100 F BFR-ID 2 BS:0010 21

Forwarding Packets Nbr Nbr Nbr AND 0111 0111 B A &0111 0011 C &0011 0001 D &0001 AND AND 0100 E &0100 0010 F &0010 0111 B 0011 C D BFR-ID 1 BS:0001 Suppose A leans about D, E and F s interest, in the blue multicast flow. (via BGP, SDN, STATIC, etc ) AND Nbr 0011 C B E BFR-ID 3 BS:0100 F BFR-ID 2 BS:0010 22

Forwarding Packets As you can see from the previous slides, the result from the bitwise AND (&) between the Bit Mask in the packet and the Forwarding table is copied in the packet for each neighbor. This is the key mechanism to prevent duplication. Look at the next slide to see what happens if the bits are not reset If the previous bits would not have been reset, E would forward the packet to C and vice versa. 23

0111 Forwarding Packets Nbr Nbr Nbr AND 0111 0111 B A &0111 AND 0111 0011 C 0100 E B &0011 &0100 AND 0111 0001 D 0010 F C D BFR-ID 1 BS:0001 AND Nbr 0011 C B E BFR-ID 3 BS:0100 F BFR-ID 2 BS:0010 24

Encapsulation

How many Bits and Where? The number of multicast egress routers that can be addressed is depending on the number of Bits that can be included in the BitString The BitString length is dependent on the encapsulation type and router platform. We ve analyzed the MPLS option, CRS, ASR1K and ASR9K platform. Both these platforms can do 256 bits. Other vendors confirmed they can do 256. We identified 5 different encoding options, most attractive below; 1. MPLS, below the bottom label and before IP header. 2. IPv6, extensions header. 26

EOS EOS MPLS encapsulation The Top Label is allocated by BIER from the downstream platform label space. The BIER Header follows directly below the BIER label. There is a single BIER label on top, unless the packet is re-encapsulated into a unicast MPLS tunnel. The VPN label is allocated from the upstream context label space (optional). BIER Label BIER Header VPN Label Payload MPLS Label BIER header Upstream Label (optional) IPv4/IPv6/L2 27

BIER Header 256bits = 32bytes 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 1 0 1 Ver Len Entropy +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ BitString (first 32 bits) ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ ~ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ~ BitString (last 32 bits) +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ OAM Reserved Proto BFIR-id +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ http://www.ietf.org/id/draft-ietf-bier-mpls-encapsulation-01.txt 28

Sets and Areas

BIER Sets A 1:0001 Set BM Nbr 1 0111 I G B 1:0010 Set 1 2 0111 I J 1:0111 2:0111 Note, we create different forwarding entries for each Set I Note, Bit Positions 1,2,3 appear in both Sets, and do not overlap due to Sets. H C D E 1:0100 2:0001 2:0010 Set 2 To increase the scale we group the egress routers in Sets. F 2:0100 30

BIER Sets Set BM Nbr 1 0111 I G A B 1:0001 Set 1 1:0010 2 0111 I J 1:0111 2:0111 I Set 2 C 2:0001 Note, we create different forwarding entries for each Set There is no topological restriction which set an egress belongs to H Set 1 D E F 1:0100 2:0010 Set 2 2:0100 31

BIER Sets If a multicast flow has multiple receivers in different Sets, the packet needs to be replicated multiple times by the ingress router, for each set once. Is that a problem? We don t think so The Set identifier is part of the packet. Can be implemented as MPLS label. 32

BIER Areas Area 1 BIER Forwarding Area BM Nbr Area 0 0:001 ABR G 1:010 A B 1:001 1:010 0 0011 I C 1:100 J 0:011 I D 2:001 Area 2 0:010 H 2:010 E 2:010 BIER area s are like IGP area s, you only need reachability to the ABR ABR F 2:100 33

BIER Areas The ABR removes the BIER header from Area 0, and imposes a new BIER header for Area 1 and 2. The new BIER header can be determined by a Group/Label lookup. Look for the inner IPv4/6 packet group address, do a lookup in the MFIB Requires flow state on the ABR. Similar to Segmented Inter-AS MVPN 34

Sub-domains

BIER Sub-domains Domain 1 A B C D PIM/ IGMP PIM/ IGMP E F G H Domain 2 Nodes are configured to be in a BIER sub-domain. BIER MPLS labels are not installed between sub-domains 36

BIER Sub-domains Domain 1 X A B C D PIM/ IGMP PIM/ IGMP E F G H Domain 2 When IGP changes SPF for B to F, traffic is dropped. BIER traffic is never leaked between sub-domains. Routers can be part of multiple sub-domains. This is a very easy mechanism to force dual plane. 37

The BFR-id

The BFR-id A BFER is uniquely identified by a two tuple {Set ID, Bit Position} The number of Bit Positions is depending on the length of the support BitString in the network. To make the BFER identifier independent of the BitString length we defined the BFR-id as a number between [1,65535] We auto-generate the BFR-id into a {Set ID, Bit Position}, based on the BitString Length. 39

The BFR-id Formula: SI = (BFR-id -1) / BitStringLength BP = ((BFR-id -1) modulo BitStringLength) + 1 Example BFR-id = 129. BitString Length 128 -> {SI = 1, BP = 1} BitString Length 256 -> {SI = 0, BP = 129} 40

The BFR-id By decoupling the BFR-id from the SetID and Bit Position tuple, the BFR identifier is agnostic to the supported BitString length in the network. This is very useful during migration. If a network supports multiple BitString lengths, an egress router only needs one BFR-id, and is reachable via each BitString length. 41

BIER Forwarding

BIER Forwarding We define two different forwarding methods for BIER. 1. Neighbor based forwarding. 2. Bit Indexed forwarding 43

BIER Forwarding, neighbor based. BIER Forwarding Table 5 4 3 2 1 Nbr Packet IN 1 1 A 2 10101 & 1 B & 10101 3 4 1 C 1 1 D & 10101 = = 00100 10001 Packet OUT Packet OUT A packet BitString is matched against each Neighbor in the BFT. This model works well for systems that have multi-cores. Its works less well for serialized processing as it requires a neighbor walk for each packet 44

5 4 3 2 1 BIER Forwarding, Bit Indexed BFT Neighbors 5 4 3 2 1 Nbr 1 A 1 B 1 C 1 1 D BFT Indexed 54321 Nbr 10001 D 00010 A 00100 B 01000 C 10001 D We translate the BFT neighbor table in to a table sorting on Bit Position (so not by neighbor) We walk the Bit Mask in the packet and Index into the FIB table. 45

5 4 3 2 1 BIER Forwarding, Bit Indexed Walk Bit Mask 1 BIER FIB Label Forwarding Table Packet IN 10101 Copy 10101 & 54321 Nbr 10001 D = 10001 Packet OUT & 01110 00010 A 00100 B 01000 C 10001 D We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 46

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 00100 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = & 10001 Packet OUT 00010 A 00100 B 01000 C 10001 D We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 47

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 2 00100 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = & 10001 Packet OUT 00010 A 00100 B 01000 C 10001 D We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 48

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 3 00100 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = & 10001 Packet OUT 00010 A 00100 B 01000 C 10001 D We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 49

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 00100 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = & 10001 Packet OUT Copy 00010 A 00100 & 00100 B 01000 C 10001 D We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 50

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 00100 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = Copy & 00100 & 00010 A 00100 B 01000 C 10001 D 10001 Packet OUT = 00100 Packet OUT We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 51

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 00100 & 11011 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = Copy & 00100 & 00010 A 00100 B 01000 C 10001 D 10001 Packet OUT = 00100 Packet OUT We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 52

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 00100 00000 & 11011 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = Copy & 00100 & 00010 A 00100 B 01000 C 10001 D 10001 Packet OUT = 00100 Packet OUT We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 53

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 4 00100 00000 & 11011 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = Copy & 00100 & 00010 A 00100 B 01000 C 10001 D 10001 Packet OUT = 00100 Packet OUT We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 54

5 4 3 2 1 BIER Forwarding, Bit Indexed Packet IN Walk Bit Mask 5 00100 00000 & 11011 Copy BIER FIB Label Forwarding Table 54321 Nbr 10101 10001 D = Copy & 00100 & 00010 A 00100 B 01000 C 10001 D 10001 Packet OUT = 00100 Packet OUT We walk the Bits in the packet, as soon as we hit a 1, we copy the packet, index into the FIB table with the position of the Bit. The Bit Mask entry is reverse & with the Bit Mask in the packet. This resets the Bits that where processed. 55

BIER Forwarding, Bit Indexed Walking the bits in a Bit String takes less clock cycles compared to walking a list of neighbors. For that reason its faster to walk the Bit String and index into the neighbor table. The table is a NxN bit matrix, where N is the Bit String length. Bits that where already processed are reset so we don t processes them if they appears later in a Bit String. This way we avoid multiple copies being forwarded. 57

BIER Forwarding, Bit Indexed Walking the Bit String in the packet is basically a repeating of Find First Set (FFS) Bit operation. Could be optimized to record the last position. http://en.wikipedia.org/wiki/find_first_set http://graphics.stanford.edu/~seander/bithacks.html Checkout Bruijn sequence (Count the consecutive zero bits) Is supported in compilers FFS and can be done in HW fairly easily. 58

ECMP

ECMP It is possible the same Bit Position is reachable over different interfaces if there are ECMP paths. We distinguish two different ECMP behaviors ECMP via parallel interfaces to a single neighbor. ECMP via different neighbors. The handling is different for both cases. If the ECMP path is going to a single neighbor, the Bit Mask is the same for each candidate path, no special processing needed. If the ECMP path is to different neighbors, the Bit Mask will be different for each neighbor because the Bit String in the packet is the end result of the lookup in BIER, which is different for each nbr. 60

ECMP Duplicate bit positions need to be resolved, ECMP logic needs to select based on Hash. In the example we selected C Nbr Nbr Nbr AND 0111 0111 B A &0111 0011 C &0011 0001 D &0001 AND AND 0110 E &0110 0010 F &0010 0111 B 0011 C D 0001 Nbr E 0100 F 0010 0011 C AND B 0010 F 61

ECMP BFT Neighbor ECMP Table 1 ECMP Table 2 ECMP Table 3 ECMP Table 4 5 4 3 2 1 Nbr 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 1 1 1 A 1 1 1 1 1 A 1 1 1 B 1 1 1 1 1 1 1 B 1 1 C 1 1 1 C 1 1 1 D 1 1 1 1 1 D We distribute the Bit Positions over multiple tables. Each Bit Position only appears once in each table. Table selection is based on entropy, done before Bit Index lookup 62

ECMP The number of tables is depending on the number of ECMP paths to different neighbors. If the max number of ECMP paths is 4, but there is a Bit Position reachable via 3 paths, this will cause unequal distribution. 63

Deployment Scenarios

Native BIER

Native BIER With Native BIER there is NO PIM involved, just IGMP and BIER. The Source and Receiver(s) are connected to BIER router. There are no RP s. There is no equivalent of PIM modes, like sparse, ssm, bidir etc.. We speak of single sender and multi sender, which is basically the same solution. The overlay can be BGP or SDN based. 66

Native BIER A E 0100 0001 1011 C C D 1110 D B 0011 D 0100 A 0001 E 0010 F F 1000 1000 B 1100 C 0010 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 67

Native BIER A 0100 E 0001 IGMP (*,G) 1011 C C D 1110 D B 0011 D 0100 A 0001 E 0010 F F 1000 1000 B 1100 C 0010 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 68

Native BIER (*,G):0:0001 A 0100 E 0001 IGMP (*,G) 1011 C C D 1110 D (*,G):0:000 1 B 1000 0011 D 0100 A 1000 B 0001 E 0010 F 1100 C (*,G):0:0001 F 0010 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 69

Native BIER (S1,G) (*,G):0:0001 A 0100 E 0001 IGMP (*,G) 1011 C C D 1110 D (*,G):0:000 1 B 1000 0011 D 0100 A 1000 B 0001 E 0010 F 1100 C (*,G):0:0001 F 0010 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 70

Native BIER (S1,G) (*,G):0:0001 A 0100 E 0001 IGMP (*,G) 1011 C C D 1110 D (*,G):0:000 1 B 1000 0011 D 0100 A 1000 B 0001 E 0010 F 1100 C (*,G):0:0001 F 0010 IGMP (*,G) 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 72

Native BIER (S1,G) (*,G):0:001 (*,G):0:0001 0 A 0100 (*,G):0:001 0 E 0001 IGMP (*,G) 1011 C C D 1110 D (*,G):0:001 (*,G):0:000 0 1 B 1000 0011 D 0100 A 1000 B 0001 E 0010 F 1100 C (*,G):0:0001 F 0010 IGMP (*,G) 0111 C 1101 D E and F announce their Group membership via overlay to all other routers. A BIER router connected to the Source can immediately start sending. 73

Native BIER (S1,G) (*,G):0:001 (*,G):0:000 0 1 A 0100 (*,G):0:001 0 E 0001 IGMP (*,G) 1011 C C D 1110 D (S2,G) (*,G):0:001 (*,G):0:000 0 1 B 1000 0011 D 0100 A 1000 B 0001 E 0010 F 1100 C (*,G):0:000 1 F 0010 IGMP (*,G) 0111 C 1101 D When B leans about a new source, it can immediately start sending. 75

MVPN over BIER

MVPN over BIER BIER replaces PIM, mldp, RSVP-TE or IR in the core. BIER represents a full mesh (P2MP) connectivity between all the PE s in the network. There is no need to explicitly signal any MDT s (or PMSI s). With MVPN there are many profiles, This is partly due to the tradeoff between State and Flooding. Different C-multicast signaling options. MVPN over BIER, there is one profile. BGP for C-multicast signaling. No need for Data-MDTs. 77

MVPN over BIER (S,G) PIM (*,G):0:0001 (*,G):0:0010 (*,G):0:0001 RR PIM (*,G) (S1,G) PIM A 0100 C 0001 PIM (*,G) BIER (*,G):0:0001 (S2,G) PIM B D PIM (*,G) 1000 0010 The BGP control plane defined for MVPN can be re-used. Big difference, there is no Tree per VPN!!! The BIER packets needs to carry Source ID and upstream VPN context label 78

Conclusion

Stateless There is no Multicast receiver or flow state in the core network (only edge). Imposition of the BIER Header may be done by application, removes state from ingress. There is no tree state in the network. There is no tree building protocol or logic in the network. There is only topology state for the BFER s, derived from unicast routing. 80

Scale Since there is no flow and tree state, converges as fast as unicast. Compared to Ingress Replication, saves 256x (minimum) 81

Simplicity No Reverse Path Forwarding (RPF) No Rendezvous Points No shared tree / source tree switchover No receiver driven tree building BIER is like unicast State is in the packet (like Segment Routing) 82

Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions 85

Thank you