Consistent Updates in Software Defined Networks: On Dependencies, Loop Freedom, and Blackholes

Transcription

1 onsistent Upates in Software Define Networks: On Depenencies, Loop Freeom, an Blackholes Klaus-Tycho Förster ETH urich Ratul Mahajan Microsoft Research Roger Wattenhofer ETH urich Abstract We consier the problem of fining efficient methos to upate forwaring rules in Software Define Networks (SDNs). While the original an upate set of rules might both be consistent, isseminating the rule upates is an inherently asynchronous process, resulting in potentially inconsistent states. We highlight the inherent trae-off between the strength of the consistency property an the epenencies it imposes among rule upates at ifferent switches; these epenencies funamentally limit how quickly the SDN can be upate. Aitionally, we consier the impact of resource constraints an show that fast blackhole free migration of rules with memory limits is NPhar for the controller. For the basic consistency property of loop freeom, we prove that maximizing the number of loop free upate rules is NP-har for interval-base routing an longestprefix matching. We also consier the basic case of just one estination in the network an show that the greey approach can be nearly arbitrarily ba. However, minimizing the number of not upate rules can be approximate well for estinationbase routing. For applying all upates, we evelop an upate algorithm that has a provably minimal epenency structure. We also sketch a general architecture for consistent upates that separates the twin concerns of consistency an efficiency, an lastly, evaluate our algorithm on ISP topologies. I. INTRODUTION The Internet as a whole is a wil place, full of autonomous participants. As such, it is naturally ifficult to control centrally; instea, routing an congestion control is achieve through a selection of istribute protocols such as BGP an TP. However, istribute protocols egrae performance, BGP cannot fin the least congeste path, an TP will only cruely approximate the available banwith on the path selecte by BGP. As a result, a loss of performance is to be expecte an accepte. Many esirable properties such as rop freeom of packets, goo utilization of links, or packet coherence are not as important as robustness. In contrast, iniviual networks that make up the Internet are controlle by single aministrative entites. These inclue enterprise networks, ISP networks, ata center networks, an wie area networks that connect the ata centers of large organizations. The owners of these networks want to get the maximum out of their massive financial investment, which often runs into hunres of millions of ollars per year (amortize). Towars this en, they have starte replacing inefficient istribute protocols. The technological river to this paraigm shift are so-calle Software Define Networks (SDNs): In an SDN, the ata plane ISBN c 2016 IFIP is separate from the control plane, allowing the ecision of where an how much ata is sent to be mae inepenent of the system that forwars the traffic itself. A centralize controller monitors the current state of the network, then calculates a set of forwaring rules, an istributes them to the routers an switches [1], [2], [3], [4]. Are centrally controlle SDNs the beginning of the en of istribute protocols? Not so fast! After all, the central SDN controller has to inform the switches about upates, an a network is an inherently asynchronous place, where noes might even be temporarily not accessible to the controller [4]! In this paper we will iscuss the problems that arise when upating rules in an asynchronous SDN-base network. We will show that espite the central control, istribute computing will have an important role, epening on the kin of consistency moel one expects from the network. One of the most basic consistency properties is that packets shoul not loop. As a result, this property, which we call loop freeom, is the starting point of our iscussion. We will then iscuss the broaer space of consistency properties an highlight the inherent trae-off between the strength of the property an the intricacy of epenencies it inuces among the actions of ifferent switches. These epenencies funamentally limit how quickly the SDN can be upate. We buil on our prior work [5], which showe that singleestination networks can be upate loop free in a istribute fashion, but i not consier the inherent computational complexity or ynamic architectures. We also exten the view on the consistency space, especially regaring blackholes. We start in Section III by formally moeling consistent single-/multi-estination network upates, an show that not all upates can be sent out in one flush. In Section IV, we follow up by stuying the NP-harness of loop free upates. In Section V, we stuy maximizing the number of sent out upates at once an how to buil a minimal epenency structure for applying all upates. Afterwars, in Section VI, we reveal the trae-off between consistency properties an upate epenencies. Aitionally, we consier the impact of resource constraints an show that fast blackhole free migration of rules with memory limits, i.e., a packet arriving at a switch must always have a matching rule to hanle it, is NPhar. We sketch a general architecture for consistent network upates in Section VII an conclue with Section VIII, where we present practical evaluation results.

2 II. BAKGROUND AND RELATED WORK From early papers on the topic (e.g., [6], [7]), we can learn that the primary promises of SDNs were that i) centralize control plane computation can eliminate the ill-effects of istribute computation (e.g., looping packets), an ii) separating control an ata planes simplifies the task of configuring the ata plane in a manner that satisfies iverse policy concerns. For example, to eliminate oscillations an loops that can occur in certain ibgp architectures, the Routing ontrol Platform (RP) [6], [7] propose a centralize control plane architecture that irectly configure the ata plane of routers in an autonomous system. However, as we gain more experience with this paraigm, a nuance story is emerging. Even with SDNs, packets can take paths that violate policy [8] an traffic greater than capacity can arrive at links [3]. What explains this gap between the promise an these inconsistencies? The root cause is that promises apply to the eventual behavior of the network, after the ata plane state has been change, but inconsistencies emerge uring ata plane state changes. Recent works have tackle specific pieces of this consistent upate problem. Reitblatt et al. [8], [9] propose a per-packet consistency solution that we call packet coherence each packet is route entirely using the ol rules or the rules, an never a mix of the two sets; Katta et al. [10] propose extensions to this solution to reuce switch memory overhea. SWAN [3] an [11], [12], [13] propose solutions to ensure that link capacity is not exceee. The work of Moses et al. [14] iscusses balancing upate performance versus perios of inconsistencies in a time-base upate approach. We make two contributions to this nascent line of work. First, beyon looking at consistency properties in isolation, we outline the broaer consistency space an the funamental harness of ensuring ifferent consistency properties. This perspective helps uncover the trae-off between the strength of the consistency property an the ifficulty of ensuring it. Secon, we investigate in etail loop freeom, a property that has not been consiere espite being basic, except for the recent parallel work of [15], [16], [17]. The packet stamping solution of Reitblatt et al. [8] can ensure loop freeom by aing version numbers to packets, but because it ensures the much stronger property of packet coherence, it is slow an has high memory overhea. The whole network nees to be upate first, before being able to use the system a long elay in upating single noe inuces a long elay for the complete network. Further, espite the extensions of Katta et al. [10], which trae-off switch memory for spee, packet stamping has high memory overhea because it simultaneously stores both ol an rules. Switch memory is a scarce commoity, with even future generations of switches reaching their memory limit easily when optimizing the network [3]. Our solutions, esigne specifically for loop freeom, are faster an memory efficient. Interestingly, a majority of the motivating examples in [8] o not nee packet coherence, only loop freeom. Francois et al. [18],[19] consier avoiing transient loops uring the convergence of link-state routing protocols. They argue that, ue to high reliability requirements nowaays, one shoul try to avoi all packet losses. For the case of single-estination rules, they consier the routing tree T of the estination, layere into ranks equivalent to the epth. The ranks are then upate after another, causing epth(t ) upates in total. Their mechanism esign can achieve fast convergence even in tier-1 ISPs an is carefully fine-tune for practical eployment [20]. Our work allows for upating noes from ifferent ranks in one upate. As such, our number of upates is not linke to the maximum chain length in the tree, but rather on the maximum chain length in the epenencies impose by the upate in general. Finally, Vanbever et al. [21] work on a relate problem, an stuy the migration of a conventional (non-sdn) network to a IGP protocol. The main ifferences to our work arise from the fact that they impose two restrictions on their moel: First, every noe must upate all its rules at once. Secon, only a single noe may be upate at a time, one after another. In contrast, we can upate iniviual forwaring entries for many noes in parallel. III. MODEL FOR LOOP FREE ROUTING UPDATES D, E D We moel a network as a set of connecte routers an switches (from now on, noes). Packets must be forware to their estination without loops. More formally, a network is a irecte multi-graph with a set of noes V, a set of estinations D V, an a set of estination-labele eges s.t. all eges labele with the same set of estinations will not contain a irecte loop. These eges form a irecte spanning tree with being the root an all eges being oriente towars. Definition 1: Let T = (V, E ) be a irecte graph with V being the set of noes, D being the sole estination, an E being the set of eges each labele with. The ege from u V to v V for estination is note as (u, v). The labele irecte graph T is a single-estination network, if T is a spanning tree with all irecte eges being oriente towars. Definition 2: Let V be a set of noes an D V be a set of estinations. For all D, let T = (V, E ) be a single-estination network an let E D = D E. Then the labele irecte multi-graph T D = (V, E D ) is a multiestination network. When a network nees to be upate, some (potentially all) noes receive a set of forwaring rules, leaving the network in a sort of limbo state. At some point all noes will be upate, but until then, the network might not be consistent, i.e., it might inuce loops. Definition 3: Let TD ol = (V, Eol D ) an TD = (V, E D ) be multi-estination networks for the same set of noes V an estinations D. Then U D = (V, E ol ) is calle a multiestination network upate. If the labele irecte multi-graph T D = (V, ED ol E D ) oes not contain any loops of eges with the same label, then the upate U D is calle consistent or loop free. A single-estination network upate U can be efine analogously.

3 v u x Fig. 1. Illustrating loop freeom. Not all upates can be sent out at once. Dotte eges are, soli eges are ol. For an introuctory example, consier the five-noe singleestination network in Figure 1. Assume that we want to upate the routing to estination from the ol pattern (soli eges) to the pattern (otte eges). A naïve metho is to sen out all upates (e.g., ask v to sen packets estine to to x) in one shot. However, uring application of these upates, it might happen that x upates its rule before y, introucing a routing loop between x an y. This loop will eventually isappear, once y upates its rule, but in an asynchronous system with possible message elays an losses, we cannot guarantee when this will happen. Asynchronicity is not a technicality, as noes in a prouction network can often react slowly (some switches might take up to 100 longer than average to upate [12]), or may not be accessible for some time to the controller [4]. Thus, solutions in which the network can quickly start using as many of the rules as possible, while maintaining the consistency properties, are preferable. IV. UPDATES AND DEFAULT RULES Interval routing an longest-prefix matching are common routing techniques for large networks. In interval routing (introuce in [22], cf. [23]), estinations { 1,..., D } are orere cyclically, an forwaring rules for a noe are efine as isjoint intervals over the estinations, cf. [24], [25], [26]. In contrast, longest-prefix routing efines forwaring rules via prefixes of the estination IDs, which may overlap: If two rules are in conflict, the one with the longer matching prefix is chosen, cf. [27], [28]. Both techniques have great practical avantages, since multi-estination routing oes not scale well: Even when consiering just IPv4 (an not IPv6), no router on the market coul store an iniviual rule for every IP aress. Furthermore, this fine graine information is not available, since the complete knowlege over a network is usually restraine to one s own Autonomous System. A subset of both techniques is multi-estination routing with the possibility of efault routes. Noes can either have iniviual forwaring rules for each estination or a efault rule, cf. [29], i.e., all packets go to a specific other noe (except for those that reache their estination at the current noe). In this section, we show that maximizing a loop free upate with efault rules is an NP-har problem an therefore also NP-har for both supersets. Definition 4: Let T D = (V, E D ) be a multi-estination network an let u, v V. If all outgoing eges from u point at v in E D, then those eges E u may be merge into a efault ege, labele with all labels from D (but packets for a estination u o not get forware from u). We enote such an ege with (u, v). I.e., we remove E u from E D an a {(u, v) }. Let the resulting set of eges of this iterate process be E D,. We call T D, = (V, E D, ) a multi-estination network with efault routes or multi-efault network. y Definition 5: Let TD, ol = (V, Eol D, ) an TD, = (V, E D, ) be multi-efault networks for the same set of noes V an estinations D. Then U D, = (V, ED, ol, E D, ) is calle a multi-efault network upate. If the labele irecte multigraph T D, = (V, ED, ol E D, ) oes not contain any loops of eges with the same label, then the upate U D, is calle consistent or loop free. v 1 v 2 v 3 Fig. 2. Illustrating circular epenencies with efault routes. Note that both in the ol an rules, no packet will loop: E.g., in the ol rules, a packet sent out from v 1 will be forware to v 3, an possibly to v 2, but never to v 1 again - as all possible estinations were alreay reache on the path. Let us start with an example of just three noes in Figure 2. We want to upate the three ol efault eges (rawn soli) to the three efault eges (rawn otte). However, ue to circular epenencies, not even a single ege can be upate without causing a loop. This problem can be hanle by relaxing the constraints of efault routing: One can prevent loops by breaking a single (efault) rule into one helper rule for each of the two other estinations, introucing these rules uring the upate process an then removing them later. In general, this is not esirable, as memory constraints on routers can easily prevent introucing these aitional helper rules, cf. [3]. Nonetheless, one can irectly check if a non-empty upate exists: heck each ege iniviually, since aing more eges cannot remove existing cycles. However, even if a multi-efault network can be upate with some eges, it is a har optimization problem. We efine the problem of upating multi-efault networks as fining the maximum number of eges that can be inclue in an upate at once: Problem 1: Let U D, = (V, ED, ol, E D, ) be a multi-efault network upate. Fin a set ED, max E max D,, s.t. i) UD, = (V, ED, ol, Emax D, ) is a loop free multi-efault network upate ii) for all loop free multi-efault network upates UD, other = (V, ED, ol, Eother D, ) with Eother D, ED, it hols that they o not contain more eges, i.e., ED, other Emax D,. Theorem 1: Problem 1 is NP-har. Proof: Our proof is a reuction from the classic NPcomplete satisfiability problem 3-SAT, in the variant with exactly three pairwise ifferent variables per clause [30]: 1) onsier the routes for estination in the trianglegaget from Figure 3. If noe X i upates, then noe X i cannot upate without inucing a loop for, an vice versa. hoosing one of the two upate rules correspons to a variable assignment for a variable x i in the instance I of 3-SAT: x i is either true or false, but not both. 2) Let be a clause in the instance I of 3-SAT. If there is a variable assignment S that satisfies I, then upating the triangle-gagets for the variables accoring to S oes not inuce a loop for any estination in the cycle-gaget for the corresponing clause in Figure 4. If no such variable assignment S exists, then at least one trianglegaget cannot be upate at all without causing a loop for a estination representing a clause.

4 X i X i i Fig. 3. Triangle-gaget for a variable x i. New eges are rawn otte, ol soli. X i i X i X i X i i Fig. 5. Extension of the triangle-gaget for a variable x i from Figure 3. New eges are rawn otte, ol soli. Eges not shown point at their estination. The four possible cycles for estination are i) X i, X i, ii) X i, X i, iii) X i, X i, i, iv) X i, X i, X i, X i, i. No other cycles are introuce. X 1 X 1 1 X 2 2 X 3 3 Fig. 4. ycle-gaget for the clause = (x 1 x 2 x 3 ). All eges not shown point irectly at their estination. Only if all three noes X 1, X 2, X 3 upate their forwaring rule for, then there is a loop for the label (via B X 1 X 1 X 2 X 2 X 3 3 B). E.g., = (x 1 x 2 x 3 ) coul only inuce a cycle via B X 1 1 X 2 2 X 3 X 3 B. X 2 B # in sequence conflicting clauses variable false variable true 1,, i,, i, X i,, i, X i 2 X i, X i X i, Xi X i, Xi 3 X i, X i X i, i X i, i 4 i Fig. 6. Table epicting the fastest possible migration scenarios for the noes in Figure 5. i) X i cannot upate before X i, ii) X i not before X i, iii) i not before X i or X i, an iv) X i or X i must upate before X i an X i an i can all three be upate. Note that,, i can always upate right away. However, if there are conflicting clauses (i.e., the corresponing instance is not satisfiable), then neither X i nor X i can upate right away, but must wait for the next upate to be sent out after the conflicts with the clauses have been cleare, thus requiring a sequence of length four. Else, one coul upate with a sequence of length three, as shown in the two rightmost columns. X 3 3) Let k be the number of variables in I. If k rules from the noes X i, X i in the triangle-gagets can be upate loop free, then there exists a variable assignment S that satisfies the instance I of 3-SAT. If less than k rules can be upate from the noes X i, X i in the triangle-gagets, then I cannot be satisfie. We now examine interval routing upates: Since the forwaring rules have to be isjoint, we may only apply upates that result in a vali state for each noe. I.e., after applying an upate, the forwaring rules have to cover all estinations an be isjoint. Removing all current rules an replacing them with a efault rule matches this requirement. In a similar fashion, we specify longest-prefix matching upates: A prefix rule may contain a set of rules it overries when the rule is inserte at a noe. Else, applying an upate might not change the routing behavior of a noe at all. orollary 1: Maximizing loop free upates for interval routing or longest-prefix matching is NP-har. A. Future Harware Even though asynchronicity is inherent in current harware solutions (e.g., noe failures [4] or highly eviating upate times [12]), one coul imagine these issues being tackle in future work. For example, the metho of upating routing information coul be ecouple from the remaining computational loa of a noe, resulting in roughly the same upate time for all noes in a network. Then one woul want to fin a shortest sequence of precompute upates that migrate the network from the current ol to the esire routing rules. I.e., the controller will sen out a first loop free multi-efault upate an wait until all affecte ege changes are confirme. This sening out of upates is iterate until all noes switche their eges to the esire routing rules. Nonetheless, this problem of upating a network remains har, i.e., how long is the sequence of upates that are sent out: Problem 2: Let U D, = (V, ED, ol, E D, ) be a multi-efault network upate. Fin a sequence of r loop free multi-efault network upates UD, 1 = (V, Eol D,, E1 D, ), U D, 2,..., UD, r with vertex sets V an corresponing pairwise isjoint ege sets E 1 D,, E2 D,,..., Er D, s.t. E1 D, E2 D, E r D, = E D, s.t. r N is minimal. Theorem 2: Problem 2 is NP-har. Proof: Note that the construction for the proof of Theorem 1 is not enough to show that Problem 2 is NP-har: While it is NP-har to ecie if k rules from the noes X i, X i in the triangle-gagets can be upate, the whole network in the proof can always be upate in a sequence of just two upates. In the first step, one woul upate all noes (except for the noes X i, X i in the triangle-gagets). Then, in the secon step, all the noes X i, X i in the triangle-gagets can be upate, since the possibility of loops in the gagets create from variables an clauses have vanishe after the first upate. However, we can exten our construction s.t. for a solution of sequence-length three, all k triangle-gagets nee to upate either X i, X i in the first element of the sequence of upates. Else, a sequence of length four woul be neee. The construction is escribe in the Figures 5 an 6. orollary 2: It is NP-har to approximate the length of the sequence of upates neee for Problem 2 with an approximation ratio strictly better than 4/3. V. ALGORITHMS FOR LOOP FREE ROUTING UPDATES We first consier variants for single-estination upates an then exten the iscussion to the other moels. While ynamic upates (i.e., upate as much as you can at once) are esirable ue to fault-tolerance (see Section I, e.g., a noe might be

5 u v w Fig. 7. Illustrating multiple maximal solutions. The noes u an v cannot upate together. a b u v y z Fig. 8. An upate of the noes a an b is a maximal upate, but an upate of the noes u, v,..., y, z an b woul be a maximum upate. temporarily unable to upate), we also stuy how to apply all upates in this section. Some proofs are in the Appenix. We start with single-estination upates: Given an upate U = (V, E ol, E ), fin a loop free upate U = (V, E ol, E ) with E E. We begin by setting E = : An upate is maximal, if aing more eges from E to E violates loop freeom. Maximal upates o not have to be unique, see Figure 7. Noe w may switch to the rule immeiately, but not noes u an v. If they both switch immeiately, an w is still using the ol rule, we get a loop. So, one of them must wait for w to switch. Either one is fine, i.e. either u must wait for w (an v, w may switch immeiately), or v must wait for w (an u, w may switch immeiately). Algorithm 1: 1) heck for an ege (u, v) = e E if the upate U = (V, E ol {e}) is loop free. This loop test can, E be performe, e.g., by a DF S from noe v to fin noe u on eges with label. 2) If aing e oes not introuce a loop, set E = E {e}. 3) Repeat step 1 until all eges were checke. Lemma 1: The upate calculate by Algorithm 1 is loop free an maximal. While a maximal solution might seem like a goo approach at first glance, it can be far from optimal regaring the number of upates sent out in one flush, see Figure 8: Even for just one estination, a maximum upate can be of size E 1, but a maximal might just be 2 eges. an we o better? Since we want to inclue as many eges as possible, we are essentially solving restricte instances of the NP-complete Feeback Arc Set Problem (FASP) [30]: Given a irecte graph, what is the minimum number of eges that nees to be remove to break all cycles. FASP can also be consiere in a variant with weighte eges: This allows us to exclue ol eges from removal, by giving all ol eges an arbitrarily high weight, an all eges a weight of just 1. The best known approximation algorithm for weighte FAS has an approximation ratio of O (log n log log n) [31], allowing us to enhance the greey algorithm for maximal upates: Algorithm 2: 1) Set the weight of all eges containe in E ol to, an the weight of all other eges to just 1. 2) alculate a FAS F for the weighte graph (V, E ol E ) accoring to [31]. 3) Set E = E \ F. 4) Apply Algorithm 1 to make the upate maximal. Lemma 2: The upate calculate by Algorithm 2 is loop free an maximal. The number of remove eges from E can at most be reuce by a factor of O (log n log log n). Proof: The removal of a FAS implies by efinition loop freeom for the network. However, ol eges are not allowe to be remove: But since all eges containe in the set of ol eges E ol = E ol ( ) E ol E have their weight set to infinity, there is always an infinitely better solution than removing any ol ege. One woul just set the eges being in E to, which results in a loop free network by efinition. Maximality is ensure by applying Algorithm 1 afterwar, which also preserves the loop free property for the network, see Lemma 1. Since Algorithm 1 can only a more eges to the upate, an not remove any, the approximation ratio of O (log n log log n) from [31] is still vali. Let us now consier how to apply the whole esire upate for a single estination via sening out multiple smaller loop free upates. In the worst case, we will nee E loop free upates, for example when reversing the links in a ring only one ege can be upate loop free at a time. Algorithm 3: 1) Use Algorithm 1/2 to sen out a first upate E,g1. 2) Once a set of noes has reporte back to the central controller that they have performe the rule upates E,g 1 E,g1 for estination (an iscare their ol rules E,g ol 1 ), the controller can calculate a current set of ol rules. Take into account that the noes applying the rules E,g1 \ E,g 1 are still in a limbo state: Either they applie the upate alreay or not, but it is not known ue to the asynchronicity until they report in. 3) alculate an sen out the next set of upate eges E,g2 ( ) E( (\E,g1 with Algorithm ) 1/2, which are erive from V, E ol \ E,g ol 1 E,g1, E \ E,g1 ). 4) Iterate the process until all eges are sent out. Algorithm 3 computes a series of loop free upates E,g1,...,E,gk, with k i=1 E,g i = E. For Algorithms 1 an 2, this can be unerstoo as a ynamic epenency forest, which is minimal in the sense that an ege e E,gj cannot be ae to E,gi, if i < j. Lemma 3: Iterating either Algorithm 1 or 2 to construct a ynamic epenency forest nees at most E non-empty upates to switch the network to the rules in E. Proof: If an upate is non-empty, then it contains at least one ege. Thus, E non-empty upates suffice to upate the network to only rules. We now show that we can always inclue at least one ege in an upate, once all sent out rules are applie. Assume that there is no noe that is currently applying a rule, i.e., all noes that receive a rule for applie it an reporte back to the controller. Thus, no noe is in a limbo state, where the noe was orere to apply a rule, but has not successfully reporte back yet. For contraiction, let us now assume that Algorithm 1 oes not fin any ege to be sent out as an upate. Thus, all not yet applie eges were checke, an each woul inuce a loop when aing it to the network in an upate. However, at least one ege exists that woul not inuce a loop. For ease of notation, let us call noes that still nee to apply a rule ol, an elsewise. Note that currently no

6 noes are in limbo. Start from an arbitrary ol noe, an move along the set of rules towars the estination. Since the estination is (by efinition), along this -rules path, there must be a last pair of noes c, p, where the ege of c points at p, an c is ol an p is. The ege (c, p) cannot inuce a loop: It points only to noes which are in the state alreay, that is, there are no more ol rules which can cause loops. Therefore, Algorithm 1 woul have foun at least one more ege to be inclue in a non-empty upate to be sent out (an thus, Algorithm 2 as well). Lemma 4: The structure of the ynamic epenency forest is minimal: Any e E,gj cannot be ae to E,gi, if i < j. Proof: W.l.o.g. let e E,gj an consier any upate E,gi with i < j. The set of eges for E,gi was maximal, i.e., no more eges coul have been ae, see the Lemmas 1 an 2. Note that the Algorithms 1, 2, an 3 can be applie to multi-estination network upates by treating them as a set of single-estination network upates: We can compute the variants separately for each label an apply upates in parallel, as eges with ifferent labels will not interfere with each other regaring loop freeom. A more complex case is where iniviual rules control routing to multiple estinations an ifferent rules control overlapping sets of estinations. (For non-overlapping estination sets, the situation is similar to above; replace estination sets with a virtual estination.) This situation can emerge in interval-base routing an longest-prefix matching. One can still use aapte versions of Algorithm 1 within Algorithm 3 for maximal loop free upates, but those upates might be empty: In this case, no (loop free) epenency forests to apply all rules may exist (cf. the network in Figure 2). We note that in practice, one shoul ivie the multi-graphs G = (V, E ol E ) into strongly connecte components (Ss), e.g., by implementing Tarjan s algorithm [32]: Eges from ifferent Ss cannot be part of the same loop, allowing to partition the problem into smaller instances. However, this oes not lea to better theoretical approximation bouns. Also, if we were able to calculate the set of all loops for each label in the multi-graph G inuce by an upate G = (V, E ol E ), then we can even improve the approximation ratio for some cases: First, consier each loop for each label as a set of eges, but only a eges to the sets. The set of ol eges was loop free, meaning there are no empty sets. Secon, solve the Minimum Hitting Set Problem (MHSP) [30] by choosing a minimum set of upate eges s.t. each loop is broken. MHSP is NP-complete as well, but a greey approach yiels an approximation ratio of H( E ) (with some improvement possible [33]), where H(n) is the n th harmonic number, H(n) ln n, cf. [34]. VI. ONSISTEN SPAE We now take a broaer view of the range of consistency properties. Table 9 helps frame this view. Its rows correspon to consistency properties. We efine loop freeom in Section III; the others are: Eventual consistency Blackhole freeom Loop freeom (Section V) Packet coherence ongestion freeom None Self Downstream subset Always guarantee Impossible A before remove Impossible (Lemma 5) Rule ep. forest Impossible (Lemma 6) Impossible (Lemma 7) Downstream all Per-flow ver. numbers TABLE 9 BASI ONSISTEN PROPERTIES & THEIR DEPENDENIES. Global Global ver. numbers [8] Stage partial moves [3], [11], [12], [13] Eventual consistency No consistency is provie uring upates. If the set of rules compute by the controller are consistent (by any efinition), the network will be eventually consistent. Blackhole freeom No packet shoul be blackhole uring upates. Blackholes occur if a packet arrives at a switch when there is no matching rule to hanle it. Packet coherence The set of rules seen by a packet shoul not be a mix of ol an rules; they shoul be either all ol or all rules. ongestion freeom The amount of traffic arriving at a link shoul not excee its capacity. Physical link capacity is a natural limit, but other limits may be interesting as well (e.g., margin for burstiness). ongestion freeom must be maintaine without ropping traffic; otherwise, we can trivially meet any limit. The consistency properties are liste in rough orer of strength, an satisfying a property lower on the list often (but not always) satisfies a property above it. Obviously, packet coherence implies blackhole an loop freeom (assuming that the ol an rules sets are free of blackholes an loops). Perhaps less obviously, congestion freeom implies loop freeom because flows in a loop will likely surpass any banwith limit. Note that flows may be splittable [35]. However, these properties cannot be totally orere. Packet coherence an congestion freeom are orthogonal, as packet coherence oes not aress congestion, an congestion freeom can be achieve with solutions beyon packet coherence. Blackhole freeom an loop freeom are also orthogonal. In fact, trivial solutions for one violates the other ropping packets before they enter a loop guarantees loop freeom, an just sening packets back to the sener provies blackhole freeom but creates loops. The columns in Table 9 enote epenency structures. They capture rules at which other switches must be upate before a rule at a switch can be use safely. Thus, the epenency is at rule level, not switch level; epenencies are often circular at switch level a rule on switch u epens on a rule on v, which in turn epens on u for other rules. The structures in Table 9 are: None The rule oes not epen on any other upate. Self The rule epens on upates at the same switch. Downstream subset The rule epens on upates at a subset of switches ownstream for impacte packets. Downstream all The rule epens on upates at all

7 switches ownstream for impacte packets. Global The rule epens on upates even at potentially all switches, incluing those that are not on the path for packets that use the rule. These epenency structures are qualitative, not quantitative. For instance, they o not capture the time it might take for the upate to complete. They also assume that switch resources, such as forwaring table memory or internal queues for unfinishe upates, are not a bottleneck. Resource limitations inuce aitional epenencies on the orer in which upates can be applie (see below). In general, upate proceures with fewer epenencies (i.e., to the left) are preferable. The cells in Table 9 enote whether a proceure exists to upate the network with the corresponing consistency property an epenency structure. We can prove that certain combinations are impossible (proofs are in the Appenix). For example, packet coherence cannot be achieve in a way that rules epen on upates at only a subset of ownstream switches. As we can see, weaker consistency properties (towars the top of Table 9) nee weaker epenency structures (towars the left). At one extreme, eventual consistency (i.e., no consistency uring upates) has no epenencies at all. Slightly stronger properties, such as blackhole freeom, have epenencies on other rules at the switch itself. A simple proceure for blackhole freeom is to a the rule in the switch before the ol rule is remove. When installe with higher priority, the rules become immeiately usable, without wait. At the other extreme, maintaining congestion freeom requires global coorination. The intuition here is that maintaining congestion freeom at a link requires coorinating all flows that use it, an some of these flows share links with other flows, an so on. Interestingly, all cells to the immeiate right of impossible cells are occupie in Table 9, which implies that, across past work an this paper, (qualitatively) optimal algorithms for maintain all these consistency properties are known. However, one must not infer from this observation that fining consistent upate proceures is a solve problem, for three reasons. First, some networks may nee ifferent properties, for which effective proceures or even best-case structures are unknown (e.g., loa balancing across links an maintaining packet orering within a flow). Secon, even for the properties in Table 9, the picture looks rosy partly because it assumes plentiful switch resources (e.g., forwaring table memory). If switch resources are constraine, maintaining consistency becomes harer. For instance, maintaining blackhole freeom with plentiful switch memory is straightforwar an inuces no epenencies across switches we can just a all rules with high priority before eleting any ol rules. But in the presence of switch memory limits, this becomes challenging because introucing a rule at a switch might require removing another rule first, which can only be remove after having ae a rule at some other switch. In fact, we can show that in the presence of memory limits, even maintaining a simple property like blackhole freeom is NP-har. Formally: Problem 3: Let c i N be the total interval rule memory of a switch v i, the combine number of interval rules in current use an the interval rules it can receive in one upate. Let G = (V, E) be the irecte graph on which packets can be route, with the estinations D V an the sources S V for the packets. In one roun, a central controller can sen out a set of any interval rules as an upate to each noe in the network. What is the minimum number of rouns, to migrate the network from a set of blackhole free ol rules to a set of blockhole free rules, if no blackholes shoul be introuce uring migration an routing shoul be possible at all times? Theorem 3: Problem 3 is NP-har. Proof: The proof for Theorem 3 is base on a reuction from the NP-har irecte Hamiltonian ycle problem (H), cf. [30]: Given a irecte graph G = (V, E), is there a cycle that visits each noe exactly once? The construction with further etails is shown in Figure 10: It is possible to migrate blackhole free in two rouns if an only if there is a Hamiltonian ycle in G, thus allowing to first use the cycle for intermeiate routing via efault rules, an then installing the rules; Else it will take three rouns, one for each rule. Thus, it is NP-har to ecie whether one can migrate in two or three rouns, even if the iameter is just two. The construction for the memory limit of c = 4 for all noes in V can be irectly extene to any c N with c 4. Furthermore, note that blackhole freeom is easy to guarantee for each noe in the presence of efault rules, if one oes not care about routing: Just set a efault rule to any neighboring noe. While packets might not arrive at all (an in aition violate other consistency properties, e.g., congestion freeom), blackhole freeom is guarantee. orollary 3: It is NP-har to approximate the number of rouns neee for Problem 3 with an approximation ratio v ol 2 v ol 1 v ol 3 G = (V, E) v 1 v 3 v 2 Fig. 10. The center noe represents the graph G = (V, E) from an instance I of the irecte Hamiltonian ycle problem, with noes v 1,..., v n. The sets of eges to (n each) an from (n/3 each) the outer six noes are bunle into single eges in this figure. Each noe in V = S = D has a memory limit c of four rules, with S being the set of packet sources an D being the set of packet estinations. The soli eges represent the eges use for the three ol rules v V, the otte eges the eges use for the three rules v V. All noes in V currently use the three noes v1 ol (for v 1,..., v (n/3) ), v2 ol (v (n/3)+1,..., v (2n/3) ), v3 ol (v (2n/3)+1,..., v n) on the left for 2-hop routing to the respective estinations in D = V, an want to migrate to use the noes v1 (for v 1,..., v (n/3) ), v2 (v (n/3)+1,..., v (2n/3) ), v3 (v (2n/3)+1,..., v n) on the right for 2-hop routing.

8 strictly better than 3/2. Thir, the table only shows the qualitative part of the story, ignoring quantitative effects, which may be equally important. Even though [8] an [3] both have global epenencies, [8] can always resolve the epenencies in two rouns, whereas [3] may nee more stages. Because of these three reasons, we believe that what is presente in this paper is just the tip of the iceberg for consistent upates in Software Define Networks. VII. AN ARHITETURE FOR SDN UPDATES We have argue that maintaining consistency uring rule upates is a key hurle towars realizing the promise of SDNs. The question is: how can we accomplish this in a flexible, efficient manner? A straightforwar possibility is that a single software moule (controller) ecies on rules an then micro manages the upate process in a way that maintains consistency. However, this monolithic architecture is unesirable because it mixes three separable concerns i) the rule set shoul be policy-compliant; ii) rules upates shoul maintain the esire consistency property; iii) the upate process shoul be efficient, which epens on the asynchronicity in the network. We propose an alternative architecture (Figure 11) with three parts, one for each concern above: i) the rule generator prouces policy-compliant rules; ii) the upate metho selector chooses the metho of how to apply the rules, base on ata from past upates; an iii) the upate executor scheules the upates efficiently in a ynamic fashion, taking current asynchronicity into account. Routing policy onsistency property Network behaviour using the ol rules, the rules, an the esire consistency property. Once a set of noes reporte back on the successful implementation of the rules, another batch of upates can be sent out into the network. Since the upate process is a ynamic one, faulty noes only inuce a limite elay, inepenent parts of the network can still be upate. Noes that i not report back yet have to be consiere in a limbo state: Either they applie the rules alreay or not, but to not break consistency properties, one has to assume that they are in both the an the ol state at the same time. An example for an upate executor woul be Algorithm 3: Maximal sets of loop free upates are sent out each time noes report back about the successful implementation of rules, inucing a minimal epenency structure in form of a ynamic epenency forest. VIII. EVALUATION We took Rocketfuel ISP topologies with intra-omain routing weights [36] an consiere link failures in these topologies, with our goal being loop free network upates from preto post-failure least-cost routing. Figure 12 plots the istribution of the length of epenency chains that emerge across ten trials, where a ranomly selecte link was faile in each. We see that roughly half of the upates epene on 0 or 1 other switch, an 90% of all forwaring rules were epenent on at most 3 other switches. In contrast, ha we use Reitblatt s proceure [8], which ensures the stronger property of packet coherence, rules woul have ha to wait for all other switches (well over a hunre in some cases), an a single slow switch can impee everyone. Rule generator New rules Upate metho selector Preferre metho Upate executor Fig. 11. Propose ynamic architecture for SDN upates The upate metho selector procees in two steps. It first generates, using the ol rules an collecte ata from past upates of the network, a moel of the current state of the network. This inclues, e.g., the mean an variance of applying an upate to a switch or the amount of unallocate memory/banwith. In the secon step, multiple methos of applying the upate are checke an simulate on the moel of the network. Depening on the outcome, a preferre metho for upates is selecte: For example, if the current amount of free memory on switches is small, packet stamping is not a viable upate metho. However, if a long chain of links nees to be reverse loop free, an memory is not an issue, packet stamping might be the best way to procee. In this step, it is also possible to issue helper rules, that are neither in the ol or set of rules, but allow consistent upates via a specific metho. onsier the network in Figure 2: One can prevent loops by breaking a single (efault) rule into one for each of the other estinations, introucing these rules uring the upate process an then removing them later. The upate executor computes a maximal set of upates that can be sent out immeiately with the selecte metho, Fig. 12. hain lengths in loop free upates in six Rocketfuel topologies. The x-axis label enotes the ASN. Francois et al. [18] evaluate their work on a tier-1 ISP with 200 noes an 800 links, resulting in chain lengths of 14. We ha a chain length of at most 7, even for tier-1 ISPs such as ASN 1239 (Sprintlink) with 547 noes an 1647 links. IX. SUMMAR We argue that consistent upates in Software Define Networks is an important an rich area for future research. We highlighte the trae-off between the strength of the consistency property an the epenency structure it imposes, an evelope minimal algorithms for loop freeom. For the basic consistency properties of loop an blackhole freeom, we showe that fast upates are NP-har optimization problems. We also sketche an architecture for consistent upates an showe that our loop freeom algorithm performs well in evaluations on ISP topologies.

9 AKNOWLEDGEMENTS We woul like to thank the anonymous reviewers for their helpful comments, which helpe us to improve the presentation of this paper. We woul also like to thank Stefan Schmi an Stefano Vissicchio for pointing us to [15], [16], [17] shortly before this article was accepte for publication. Klaus- Tycho Förster was supporte in part by Microsoft Research. REFERENES [1] M. Borokhovich an S. Schmi, How (Not) to Shoot in our Foot with SDN Local Fast Failover, in OPODIS, [2] M. asao, N. Foster, an A. Guha, Abstractions for software-efine networks, ommun. AM, vol. 57, no. 10, pp , Sep [3].-. Hong, S. Kanula, R. Mahajan, M. hang, V. Gill, M. Nanuri, an R. Wattenhofer, Achieving high utilization with software-riven WAN, in SIGOMM, [4] S. Jain, A. Kumar, S. Manal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanerer, J. hou, M. hu, J. olla, U. Hoelzle, S. Stuart, an A. Vahat, B4: Experience with a globally-eploye software efine WAN, in SIGOMM, [5] R. Mahajan an R. Wattenhofer, On consistent upates in software efine networks, in HotNets, [6] M. aesar, D. alwell, N. Feamster, J. Rexfor, A. Shaikh, an J. van er Merwe, Design an implementation of a routing control platform, in USENIX NSDI, [7] N. Feamster, H. Balakrishnan, J. Rexfor, A. Shaikh, an K. van er Merwe, The ase for Separating Routing from Routers, in SIGOMM Workshop on Future Directions in Network Architecture (FDNA), [8] M. Reitblatt, N. Foster, J. Rexfor,. Schlesinger, an D. Walker, Abstractions for network upate, in SIGOMM, [9] M. Reitblatt, N. Foster, J. Rexfor, an D. Walker, onsistent upates for software-efine networks: hange you can believe in! in 10th AM Workshop on Hot Topics in Networks, [10] N. P. Katta, J. Rexfor, an D. Walker, Incremental consistent upates, in HotSDN, [11] S. Brant, K.-T. Förster, an R. Wattenhofer, Augmenting anycast network flows, in IDN, [12] X. Jin, H. Liu, R. Ganhi, S. Kanula, R. Mahajan, J. Rexfor, R. Wattenhofer, an M. hang, Dionysus: Dynamic Scheuling of Network Upates, in SIGOMM, [13] S. Brant, K.-T. Förster, an R. Wattenhofer, On onsistent Migration of Flows in SDNs, in INFOOM, [14] T. Mizrahi, O. Rottenstreich, an. Moses, TimeFlip: Scheuling network upates with timestamp-base TAM ranges, in INFOOM, [15] A. Luwig, J. Marcinkowski, an S. Schmi, Scheuling Loop-free Network Upates: It s Goo to Relax! in POD, [16] S. Vissicchio an L. ittaini, FLIP the (Flow) Table: Fast LIghtweight Policy-preserving SDN Upates, in INFOOM, [17] A. Luwig, S. Duycz, M. Rost, an S. Schmi, Transiently Secure Network Upates, in Sigmetrics, [18] P. François an O. Bonaventure, Avoiing transient loops uring the convergence of link-state routing protocols, IEEE/AM Trans. Netw., vol. 15, no. 6, pp , [19] P. François,. Filsfils, J. Evans, an O. Bonaventure, Achieving subsecon IGP convergence in large IP networks, omputer ommunication Review, vol. 35, no. 3, pp , [20] P. Francois an O. Bonaventure, Loop-free convergence using ofib, Internet-Draft, IETF, [21] L. Vanbever, S. Vissicchio,. Pelsser, P. Francois, an O. Bonaventure, Lossless Migrations of Link-state IGPs, IEEE/AM Trans. Netw., vol. 20, no. 6, pp , Dec [22] N. Santoro an R. Khatib, Routing without routing tables, SS-TR-6, arleton University, Ottawa, Tech. Rep., [23]. Gavoille, A survey on interval routing, Theor. omput. Sci., vol. 245, no. 2, pp , [24] M. Flammini, G. Gambosi, an S. Salomone, Boolean Routing, in WDAG, [25] P. Fraigniau an. Gavoille, A characterization of networks supporting linear interval routing, in POD, [26] J. Van Leeuwen an R. B. Tan, Interval routing, The omputer Journal, vol. 30, no. 4, pp , [27] D. omer, E., Internetworking with TP/IP - Principles, Protocols, an Architectures, Fourth Eition. Prentice-Hall, [28] A. S. Tanenbaum an D. J. Wetherall, omputer Networks, 5th e. Upper Sale River, NJ, USA: Prentice Hall Press, [29] V. Fuller an T. Li, RF 4632, lassless Inter-omain Routing (IDR): The Internet Aress Assignment an Aggregation Plan, [30] M. R. Garey an D. S. Johnson, A Guie to the Theory of NP- ompleteness. New ork, N, USA: W. H. Freeman & o., [31] G. Even, J. Naor, B. Schieber, an M. Suan, Approximating minimum feeback sets an multicuts in irecte graphs, Algorithmica, vol. 20, no. 2, pp , [32] R. E. Tarjan, Depth-first search an linear graph algorithms, SIAM J. omput., vol. 1, no. 2, pp , [33] A. Srinivasan, Improve approximations of packing an covering problems, in STO, [34] R. L. Graham, D. E. Knuth, an O. Patashnik, oncrete Mathematics: A Founation for omputer Science, 2n e. Boston, MA, USA: Aison- Wesley Longman Publishing o., Inc., [35] R. ohen an G. Nakibly, Maximizing restorable throughput in mpls networks, IEEE/AM Trans. Netw., vol. 18, no. 2, pp , [36] R. Mahajan, N. Spring, D. Wetherall, an T. Anerson, Inferring link weights using en-to-en measurements, in IMW, X. APPENDIX FOR SETION 5 Proof of Lemma 1: We start with loop freeom: The invariant of the algorithm is that the current eges in the network are without loops. The invariant is true at the beginning, since no ege is inclue, an the ol eges form an in-tree to the estination. When a ege (u, v) is ae, a now existing loop must contain this ege, i.e., there is a path from v to u. If a DFS starting at v cannot reach u, then there is no path from v to u, an the network is loop free. We now look at maximality: The algorithm checks each ege once if it can be ae without inucing a loop. onsier an ege e = (x, y), that is being teste w.l.o.g. as the i-th ege, but cannot be ae to the network, because it woul inuce a loop x, y, z,..., x. If e is being teste again after the (j 1)-th ege, with i < j, coul e be ae to a loop free network without inucing a loop in the network? No, because it woul still inuce the same loop, as eges were never remove, only possibly ae. XI. APPENDIX FOR SETION 6 Lemma 5: Loop freeom epens on other noes. Proof: In Figure 1, noe x epens on noe y. Lemma 6: Packet coherence epens on all non-trivial ownstream switches. Proof: Let u be a switch router that is non-trivial, in the sense that u is affecte by a rule change, i.e. u s ol rule iffers from its rule. If the source starts to route packets accoring to the rule, switch u will forwar the packets wrongly, or rop them, which is not packet coherent. Lemma 7: ongestion freeom epens on all switches. Proof: Let f be a flow that wants to use a path p, or increase its capacity on an existing path. The network may be able to aapt to flow f, however, only if other flows use ifferent paths as well, which in turn may (recursively) move even other flows (some of which have no single switch/link in common with the path p). As such, any f may potentially epen on any single switch in the network.