Dual Subgradient Methods Using Approximate Multipliers

Transcription

1 Dual Subgradient Methods Using Approximate Multipliers Víctor Valls, Douglas J. Leith Trinity College Dublin Abstract We consider the subgradient method for the dual problem in convex optimisation with approximate multipliers, i.e., the subgradient used in the update of the dual variables is obtained using an approximation of the true Lagrange multipliers. This problem is interesting for optimisation problems where the exact Lagrange multipliers might not be readily accessible. For example, in distributed optimisation the exact Lagrange multipliers might not be available at the nodes due to communication delays or losses. We show that we can construct approximate primal solutions that can get arbitrarily close to the set of optima as step size α is reduced. Applications of the analysis include unsynchronised subgradient updates in the dual variable update and unsynchronised max-weight scheduling. Index Terms subgradient methods, Lagrange multipliers, max-weight scheduling, unsynchronised updates. I. INTRODUCTION In this paper we study the convergence of the subgradient method with constant step size in constrained convex optimisation with approximate subgradients. More specifically, the subgradient used in the update of the dual variables is computed using an approximation of the Lagrange multipliers instead of the true Lagrange multipliers. One of the motivations of considering approximate multipliers in the optimisation is that in some problems the exact Lagrange multipliers might not be available, but a noisy, delayed or perturbed version of the multipliers is available instead. For instance, in distributed optimisation a node might not have access to the exact Lagrange multiplier in the system due to transmission delays or losses. Another example is in networ problems where discrete valued queue occupancies can be identified with approximate scaled Lagrange multipliers []. We show that the subgradient method with approximate multipliers converges as long as the distance between the approximate and Lagrange multipliers can be controlled with step size α >. In particular, we show that the running average of the associated primal points can get arbitrarily close to the set of primal optima as step size α is reduced. Further, this running average is asymptotically attracted to a feasible point for every α>. These results establish conditions under which inaccurate nowledge of the Lagrange multipliers does not affect the convergence of the subgradient method. Important applications of the analysis include unsynchronised subgradient updates of the dual variable and unsynchronised max-weight scheduling. This wor was supported by Science Foundation Ireland under Grant No. /PI/77. A. Related Wor Subgradient methods in constrained convex optimisation have been extensively studied under various steps size rules. For example, see the early wors by Shor [] and Polya [3] or the more modern wors by Bertseas et. al. in [4] and [5]. Subgradient methods in constrained convex optimisation with noisy updates have been studied by Nedić and Bertseas in [6]. In particular, the authors study the case where the noise is deterministic and bounded, and show optimality using several step size rules. The wor in [7] considers unsynchronised subgradient updates where a subset of subgradients are updated at each iteration. The convergence of the algorithm is proved using a diminishing step size rule. The wor in [8] considers asynchronous subgradient updates with constant step size, and asymptotic optimality is proved using averaging schemes. Approximate solutions to convex problems under an averaging scheme have been studied by Nedić in[9]and []. The wor in [9] assumes that the dual function can be computed efficiently and the wor in [] considers a sequence of primal-dual subgradient updates. For a good reference on primal averaging schemes see the related wor section and references in [9]. The max-weight scheduling introduced by Tassiulas in [] has been extended in a wide range of papers to consider the minimise a convex utility function. For instance, energy minimisation [] or fairneess [3] among others. In [] we showed that Lagrange multipliers and queue occupancies in networ are related, and that max-weight resource allocation problems can be formulated as convex optimisation problems. B. Notation Vectors and matrices are indicated in bold type. Since we often use subscripts to indicate elements in a sequence, to avoid confusion we usually use a superscript x (j) to denote the j th element of a vector x. The j th element of operator [x] + equals x (j) (the j th element of x) whenx (j) > and otherwise equals when x (j) <. The subgradient of a convex function f at point x is denoted f(x). For two vectors x, y R m we use element-wise comparisons x y and y x to denote when y (j) x (j), y (j) >x (j) respectively for all j =,...,m.

2 II. PRELIMINARIES A. Problem Setup Consider the following convex optimisation problem P : minimise f(z) subject to g(z) where f,g (j) : R n R, j =,...,m are convex functions, g(z) =[g () (z),...,g (m) (z)] T and C is a bounded convex set from R n.letc := z C g(z) } and we will assume C is non-empty, i.e., problem P is feasible. Further, we will denote by C := arg min f(z) C the set of optima and f := f(z ), z C. B. Lagrange Penalty & Dual Function As in classical constrained convex optimisation we define the Lagrange penalty function L : R n R m R associated with problem P as L(z, λ) :=f(z)+λ T g(z) () where λ R m +. Note that L(z, λ) is convex in z for a fixed λ and linear in λ for a fixed z. We also define the Lagrange dual function as follows q(λ) :=L(z (λ), λ) =minl(z, λ) λ where z (λ) arg min L(z, λ). Importantly, since q(λ) is the infimum of a collection of linear functions it is concave and q(λ) L(z, λ) for all λ. The dual function is of particular interest when we have strong duality, i.e., the solution of the dual problem P D maximise λ q(λ) () and the primal problem P coincide. Namely, when we have that f =min max λ L(z, λ) =max λ min L(z, λ) =q(λ ) where λ := arg max λ q(λ). We are guaranteed to have strong duality when the following assumption holds. Assumption (Slater condition). The set C := z C g(z) } has non-empty relative interior, i.e., there exists a point z C such that g( z). A direct consequence of the Assumption is that the set of dual optima is bounded. The following lemma corresponds to Lemma in [9]. Lemma (Bounded Dual Set). Let Q δ := λ : q(λ) q(λ ) δ} with δ and let the Slater condition hold, i.e., there exists a vector z C such that g( z). Then, for every λ Q δ we have that λ Q:= υ (f( z) f + δ) (3) where υ := min j,...,m} g (j) ( z). C. Classical Subgradient Method Problem P D is an unconstrained optimisation problem that can be solved, for example, using the subgradient method. The subgradient method for the dual problem consists of updates z (λ ) arg min ) (4) λ + =[λ + α(g(z (λ )))] + (5) where α> is a step size. Subgradient methods can mae use of more complex step sizes (see [5] for example), however, in this paper we will only use a constant step size for its simplicity and practical importance. The convergence of the subgradient method for the dual problem [4] is based on decreasing the euclidean distance between λ and λ. In particular, the distance between λ and λ decreases when the difference between q(λ ) and q(λ ) is sufficiently large; otherwise λ remains in a ball around λ. The size of the ball λ converges depends on step size α>, having that λ Λ := arg max λ q(λ) as α. Further, since by strong duality arguments z (λ ) z as α we can intuitively expect z (λ ) to converge close to z C for α sufficiently small. III. SUBGRADIENT METHOD WITH APPROXIMATE MULTIPLIERS The classical subgradient updates (4) and (5) can be extended to mae use of approximate Lagrange multipliers µ R m +. Using an approximation of the Lagrange multiplies in the subgradient method is interesting, for example, to capture the fact that in some optimisation problems the exact Lagrange multipliers might not be nown or have errors. Suppose we have the following updates z (µ ) arg min ) (6) λ + =[λ + αg(z (µ ))] + (7) where µ R m +. We can thin of µ as being an approximate multiplier substituted into the classical subgradient updates (4) and (5). We are first interested in nowing when the multiplier λ generated by (7) will converge to a ball around λ.this is answered by the following lemma. Lemma. Consider the setup of problem P and updates (6) and (7) and suppose that λ µ ασ for all with α >, σ. Suppose the Slater condition is satisfied (Assumption ) and that λ R m +.Thenλ converges to a ball around λ. Further, the size of the ball depends on parameter α>, having that λ Λ := arg max λ q(λ) as α. Lemma tells us that all we require for λ to converge to a ball around λ is that the distance between µ and λ is uniformly bounded for all, i.e., λ µ ασ with α, σ >. As we will see later, this condition can be readily evaluated and is commonly satisfied in practical problems of interest. Further, observe that when σ =then µ = λ and so we recover the classical subgradient method.

3 3 In addition to convergence of the multipliers, we also have that the running average z := z i (µ i ) (8) i= converges to a feasible point as for every α>. This is formally presented in the following lemma. Lemma 3. (Feasibility) Consider the setup of problem P and updates (6) and (7) where λ µ ασ for all with α>, σ. Suppose the Slater condition is satisfied (Assumption ) and that λ R m +. Then, for all =,,... ( g(z ):=g i= ) z i (µ i ) λ α where λ λ := Q +max λ, Q + αmḡ}, ḡ := max g(z) and Q is given in Lemma with δ := αm (ḡ /+σ ḡ). As it follows that g(z ) and so z is feasible. Also, by strong duality arguments we can expect z to converge to a feasible point that is close to z for α sufficiently small. This intuition is captured by our main result, which establishs the convergence of f(z ) to f depending on step size α and iterate. Theorem. Consider the setup of problem P and updates (6) and (7) where λ µ ασ for all j =,...,m with α>, σ. Suppose the Slater condition is satisfied (Assumption ). Then, m λ α α m(ḡ /+σ ḡ) f(z ) f αm(ḡ + σ ḡ)+ 3m λ () α Proof: See appendix. Observe that the bounds in Theorem are not asymptotic but rather can be applied at finite times. As the bounds simplify to α m (ḡ +σ ḡ) f(z ) f αm (ḡ +σ ḡ) and can be made small by selecting step size α sufficiently small. Note from Theorem that if the Lagrange multipliers are accessible in the optimisation (µ = λ for all ) thenσ =and so we have a tighter bound. IV. UNSYNCHRONISED UPDATES Theorem has many useful applications. In addition to allowing use of noisy multipliers e.g., µ (j) =[λ (j) + αy ] + where y is a bounded random variable, in this section we show that convergence of the subgradient method with unsynchronised updates can be obtained as a corollary, avoiding the need for complex, specialised analysis machinery. To see this consider the following separable convex problem P s : minimise z (j) C j,,...,n f(z) := subject to g r (z) := f (j) (z (j) ) g (j) r (z (j) ) r =,...,m (9) where f (j),g r (j) : R R, j =,...,n, r =,...,m are convex functions and C := C C n. Here, the objective can be decomposed into functions f (j) which depend on element z (j) of vector z. Each element z (j) is constrained to lie in convex set C j. The constraints g r (z) can also be decomposed but are not separable since the sum on the LHS must satisfy the non-positivity constraint (which imposes a joint constraint on the g r (j) ). The Lagrangian of problem P s is given by L(z, λ) = = m f (j) (z (j) )+ L (j) (z (j), λ) r= λ (r) n g (j) r (z (j) ) where L (j) (z (j), λ) =f (j) (z (j) )+ m r (z (j) ) and λ := [λ (),...,λ (m) ] T. The dual function can be decomposed as the sum of n concave functions q(λ) := L (j) (z (j), λ) = q (j) (λ). () min z (j) C j,,...,n r= λ(r) g (j) where q (j) (λ) :=min z (j) C j L (j) (z (j), λ). While in () each function q (j) uses the same multiplier λ, this is not essential. Instead, we can allow each function q (j) to use a different approximate multiplier µ (j) and select z (j) (µ(j) [ λ (r) + = λ (r) ) arg min L (j) (w, µ (j) w C ) j =,...,n j () + α ] + g r (j) (z(j) (µ (j) )) r =,...,m. (3) Provided λ µ (j) ασ for all, then by Theorem we now that the updates ()-(3) converge to a ball around the optimum. In particular, consider now the situation where at each step only a subset of the elements z (j), j =,,nare updated i.e. updates are unsynchronised. The elements which are updated use multiplier µ (j) = λ (j). Those elements z(j) which are not updated at step can be formally thought of as being updated using the old multiplier value µ (j) = λ (j) where τ (j) τ (j) is the time step where z (j) was last updated. Provided τ (j) is uniformly upper bounded by σ then λ µ (j) is unformly bounded by ασ ḡ and so by Theorem updates ()-(3) converge to a ball around the optimum. That is, we have the following corollary. Corollary. (Unsynchronised Subgradient Updates) Consider the setup of problem P s and let µ (j) = λ (j), j =,...,n τ (j) with τ N n. Then, the bound () holds if there exists a constant σ such that τ σ for all. Note that the delay τ (j) may vary from time step to time step and from element to element in an arbitrary manner (including randomly) and this analysis still applies provided that τ (j) is

4 4 uniformly upper bounded. Extension to random delays which are not uniformly bounded can be made on a sample path basis. That is, we now have a probability of convergence with the probability being the fraction of sample paths for which satisfies a specified uniform upper bound. In this way the convergence of many consensus-lie approaches for distributed optimisation may be encompassed by Theorem. Further, observe that alternatively each element z (j), j =,,nmight be updated using a delayed value of multiplier τ (j) λ (j) (these might, for example, be due to propagation delay across a networ) and the same analysis applies. To illustrate these results we present a brief example with unsynchronised updates and approximate multipliers and compare it to the classical subgradient method (σ = and synchronised subgradient updates). Example. Consider the following optimisation problem minimise subject to z w Az b where z R 3, C := z R 3 z }, w := [ 3 ] [ ] T, A := and b := [ The Lagrangian is 4] T. L(z, λ) = z w + λ T (Az b) 3 = λ T b + (z (j) w (j) ) +(λ T A (j) )z (j), where λ := [λ (),λ () ] T and the dual variable updates are given by λ (j) + =[λ(j) + α(az (µ ) b)] + j =,. We study two cases. Case (i): µ (j) = λ (j) for all, j =, and the subgradients in the dual variable update are computed synchronously. Namely, at each iterate we have z (j) arg min (s s [,] w(j) ) +(λ T A(j) )s for all j =,, 3. Case (ii): approximate multipliers are given by µ (j) =[λ (j) + αy ] + where y is a realisation of a random variable uniformly distributed between and, and the primal variables z (j) are updated in an unsynchronised manner, i.e., z (j) arg min (s s [,] w(j) ) +(µ T A (j) )s where j,, 3} is selected uniformly at random at each time step. Note that case (i) corresponds to the classical subgradient method for the dual problem and that case (ii) extends it to mae use of approximate Lagrange multipliers and unsynchronised updates. We numerically solve this example with initial condition λ = and step size α =/5 for both cases. Figures a, b and c show respectively the convergence of λ, z (µ ), f(z ) to λ, z, f. Observe from Figure a that even though λ converges to a bigger ball around λ in case (ii) than in time slot case (i) - λ () case (i) - λ () case (ii) - λ () case (ii) - λ () (a) Convergence of the Lagrange multipliers. Optimal dual variables are λ () =.7 and λ () = time slot case (i) - z () case (i) - z () case (ii) - z () case (ii) - z () (b) Convergence of primal variables z (j) to z (j) for j =,. Straight lines corresponds to optimal values f time slot case(i) - f(z ) case(ii) - f(z ) (c) Convergence of the objective function in both cases to f =3.4. Fig. : Illustrating the convergence of Example with parameters λ = and α =/5. case (i), the value of f(z ) obtained in both cases is very close for large enough. V. UNSYNCHRONISED MAX-WEIGHT It was recently shown in [] and [4] that max-weight scheduling uses the queues within a networ as approximate scaled multipliers. This observation can be viewed as essentially a special case of Theorem which maes use of the fact that two queue-lie updates Q + =[Q + αδ ] + and

5 5 Q + Fig. : Networ of the example in Section V. =[ Q + α δ ] + satisfy Q Q ασ whenever max δ j δ j σ / (a property which also follows from the continuity of queues in the Sorohod metric). Using Theorem, the extension to unsynchronised updates is immediate. We illustrate this via a simple networ flow example. A. Networ Flow Example Consider the networ illustrated in Figure. Time is slotted at each node but nodes time slots are not synchronised. Exogenous pacets arrive into the system (enter the queue of node ) with mean rate pacet per slot. Node can transmit a single pacet to either node or node 3; the pacets transmitted by nodes and 3 leave the system. All nodes in the networ transmit a single pacet in a slot. Similarly to max-weight scheduling, we aim to design a pacet scheduling policy at each node that eeps the system stable (bounded queues), does not require us to now the traffic characteristics in advance and maes decisions based only on queues occupancies. Furthermore, nodes mae scheduling decisions in order to minimise a local convex utility function of the average lin throughput. 3 B. Fluid-lie Optimisation We can formulate the resource allocation problem as a convex optimisation min s.t. 4 f (j) (z (j) ) (4) z() + z (), z () z (3), z () z (4) where C := z R 4 + z,z () + z () }, i.e., it is the convex hull of actions set, } (where represents transmitting a pacet over a lin whereas to doing nothing) with the restriction that node can only transmit in only one lin in a time slot. Constraint z() +z () enforces the mean service rate of node to be bigger or equal than the mean arrival rate in the system. Similarly, constraints z () z (3) and z () z (4) enforce the service rate of nodes and 3 to be bigger or equal than the mean service rate of node over each lin. The Lagrangian is given by L(z, λ) = 4 f (j) (z (j) )+λ () ( z() z () ) + λ () (z () z (3) )+λ (3) (z () z (4) ). (5) Now consider the following subgradient updates with approximate multipliers [z () (µ ),z () (µ )] arg min w C f () (w () )+f () (w () ) µ () (w() + w () )+µ () w() + µ (3) w()}, (6) f (3) (w) µ () w}, (7) z (3) (µ ) arg min w [,] z (4) (µ ) arg min w [,] f (4) (w) µ (3) w}, (8) λ () + =[λ() + α( z () (µ ) z () (µ ))] +, (9) λ () + =[λ() + α(z () z (3) (µ ))] +, () λ (3) + =[λ(3) + α(z () z (4) (µ ))] +, () where C := z R + z }. Whenµ = λ then (9)-() are equivalent to the classical dual subgradient update. Since the objective function is separable and constraints are linear it can be seen that the optimisation can be solved in a distributed manner. Note from Equations (6)-(8) and Figure that we can compute the dual function without nowing the mean arrival rate at each node. However, updates (9)-() require us to now the mean pacet arrival rates. C. Optimisation with Unsynchronised Updates, Discrete Actions and Queue Occupancies as Approximate Multipliers Suppose now that at the beginning of each slot a node decides to transmit a pacet depending on a transmission policy. The transmission policy of a node at a given time t consists of a single action. The action set of node is [, ], [, ], [, ]} where action [, ] denotes transmitting a pacet to node, [, ] to node 3 and [, ] to doing nothing. Action set of nodes and 3 is, } where denotes transmitting a pacet out of the system and to doing nothing. The transmission policy of the system at a given time t is given by X :=, where the columns of matrix X contain the different permutations of possible transmission policies of the nodes in the system. Pacets arrive into the system (queue of node ) at discrete times t =,, 3,... and they satisfy that t i= (b i ) where b i, } denotes receiving a pacet or not. We use vector Q t = [Q () t,q () t,q (3) t ] T to denote the queue occupancy at nodes,, 3 respectively. Suppose the transmission policy of the system is updated at times t =,, 3,... and is obtained as shown in Equation (3) in [], ( ) x arg min z (µ i ) x i x () x X i= i=

6 6.3.5 f f(z ) time slot Fig. 3: Illustrating the converge of f(z ) to f =.5 in the unsynchronised networ example with step size α =. and initial condition Q =. where a vector x X is a transmission policy of the system and z (µ ):=[z () (µ ),...,z (4) (µ )] T isgivenbythe subgradient updates [z () (µ ),z () (µ )] arg min w C f () (w () )+f () (w () ) µ () (w() + w () )+µ () w() + µ (3) w()}, (3) f (3) (w) µ () w}, (4) z (3) (µ ) arg min w [,] z (4) (µ ) arg min w [,] f (4) (w) µ (3) w}, (5) where here µ = αq at each iterate with probability /5, otherwise µ = µ, i.e., updates are unsynchronised. Importantly, as shown in [] update () eeps the scaled queue occupancy close to the Lagrange multipliers, and therefore Theorem applies. Note as well from Equations (3)-(5) that as explained in the Section V-B the dual function can be computed without nowledge of the pacet arrivals mean rates. We simulate the networ with objective functions f (j) (z (j) )=(z (j) ) for all j =...,4 with initial condition Q = and step size α =.. See in Figure 3 that as expected from Theorem we have that f(z ) converges to f. Figure 4 shows the scaled queue occupancy of the networ and the Lagrange multipliers stay close and both converge to a ball around λ. VI. CONCLUSIONS We consider the subgradient method for the dual problem in convex optimisation with approximate multipliers, i.e., the subgradient used in the update of the dual variables is obtained using an approximation of the true Lagrange multipliers. This problem is interesting for optimisation problems where the exact Lagrange multipliers might not be readily accessible. We show that the running average of the associated primal points can get arbitrarily close to the set of primal optima as step size α is reduced. Further, this running average is asymptotically attracted to a feasible point for every α >. Applications time slot Fig. 4: Illustrating the convergence of the α-scaled queues to a ball around λ () =, λ () =.5 and λ (3) =.5. Blue, green and orange lines correspond, respectively, to the α- scaled queues Q (), Q () and Q (3) and blac lines correspond to the associated Lagrange multipliers. of the analysis include unsynchronised subgradient updates in the dual variable update and unsynchronised max-weight scheduling. REFERENCES [] V. Valls and D. Leith, On the relationship between queues and multipliers, in Communication, Control, and Computing (Allerton), 4 5nd Annual Allerton Conference on, Sept 4, pp [] N. Z. Shor, Minimization Methods for Non-Differentiable Functions. Springer Berlin Heidelberg, 985. [3] B. Polya, Introduction to Optimization. Optimization Software, Inc.,, 987. [4] D. P. Bertseas, Nonlinear programming. Athena Scientific, 999. [5] D. P. Bertseas, A. Nedić, and A. E. Ozdaglar, Convex Analysis and Optimization. Athena Scientific, 3. [6] A. Nedić and D. P. Bertseas, The effect of deterministic noise in subgradient methods, Mathematical programming, vol. 5, no., pp ,. [7] K. Kiwiel and P. Lindberg, Parallel subgradient methods for convex optimization, in Inherently Parallel Algorithms in Feasibility and Optimization and their Applications, ser. Studies in Computational Mathematics. Elsevier,, vol. 8, pp [8] N. Gatsis and G. Giannais, Asynchronous subgradient methods with unbounded delays for communication networs, in Decision and Control (CDC), IEEE 5st Annual Conference on, Dec, pp [9] A. Nedić and A. Ozdaglar, Approximate primal solutions and rate analysis for dual subgradient methods, SIAM Journal on Optimization, vol. 9, no. 4, pp , 9. [Online]. Available: [], Subgradient methods for saddle-point problems, Journal of Optimization Theory and Applications, vol. 4, no., pp. 5 8, 9. [] L. Tassiulas and A. Ephremides, Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networs, Automatic Control, IEEE Transactions on, vol. 37, no., pp , Dec 99. [] M. Neely, E. Modiano, and C. Rohrs, Power allocation and routing in multibeam satellites with time-varying channels, Networing, IEEE/ACM Transactions on, vol., no., pp. 38 5, Feb 3. [3] M. Neely, E. Modiano, and C. ping Li, Fairness and optimal stochastic control for heterogeneous networs, Networing, IEEE/ACM Transactions on, vol. 6, no., pp , April 8. [4] V. Valls and D. J. Leith, Max-weight revisited: Sequences of non-convex optimisations solving convex optimisations, Networing, IEEE/ACM Transactions on, in press.