Distributed Optimization via ADMM

Transcription

1 Distributed Optimization via ADMM Zhimin Peng Dept. Computational and Applied Mathematics Rice University Houston, TX Aug. 15, 2012 Main Reference: Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein Zhimin ADMM 1

2 Outline RICE 1 Alternating direction method of multipliers 2 Distributed Optimization 3 Lasso Zhimin ADMM 2

3 Alternating direction method of multipliers Alternating direction method of multipliers RICE ADMM problem form (with f, g convex) min s.t. f (x) + g(z) Ax + Bz = c - two sets of variables, with separable objective. Augmented Lagrangian Function L ρ (x, z, y) = f (x) + g(z) + y T (Ax + Bz c) + ρ 2 Ax + Bz c 2 2 ADMM: x k+1 z k+1 = arg min x L ρ (x, z k, y k ) = arg min z L ρ (x k+1, z, y k ) y k+1 = y k + ρ(ax k+1 + Bz k+1 c) Zhimin ADMM 3

4 Alternating direction method of multipliers Scaled Dual Form RICE Augmented Lagrangian Function: L ρ (x, z, y) = f (x) + g(z) + y T (Ax + Bz c) + ρ 2 Ax + Bz c 2 2 = f (x) + g(z) + ρ 2 Ax + Bz c + u const with u k = 1 ρ y k scaled form: x k+1 z k+1 = arg min x (f (x) + ρ 2 Ax + Bzk c + u k 2 2) = arg min z (g(z) + ρ 2 Ax k+1 + Bz c + u k 2 2) u k+1 = u k + (Ax k+1 + Bz k+1 c) Zhimin ADMM 4

5 Distributed Optimization Distributed Optimization RICE Consider global consensus problem with regularizer N min f i (x) + g(x) i=1 - x is the global variable ADMM form: min s.t. N f i (x i ) + g(z) i=1 x i z = 0 i = 1, 2,..., N - x i are local variables - z is the global variable, stored locally at each processor Zhimin ADMM 5

6 Distributed Optimization Augmented Lagrangian function L ρ (x, z, y) = g(z) + = g(z) + - with u i = 1 ρ y i - x i, u i are local variables ADMM: x k+1 i z k+1 N (f i (x i ) + yi T (x i z) + ρ 2 x i z 2 2) i=1 N (f i (x i ) + ρ 2 x i z + u i 2 2) + const i=1 = arg min x f i (x i ) + ρ 2 x i z k + u k i 2 2 = arg min g(z) + ρn z 2 z (uk + x k+1 ) 2 2 ui k+1 = ui k + xi k+1 z k+1 - with u k = 1 N N i=1 uk i, x k+1 = 1 N N i=1 x k+1 i Zhimin ADMM 6

7 Lasso RICE Lasso Problem: min 1 2 Ax b λ x 1 ADMM form: min 1 2 Ax b λ z 1 s.t. x z = 0 Scaled ADMM: x k+1 = (A T A + ρi ) 1 (A T b + ρ(z k u k )) z k+1 = S λ/ρ (x k+1 + u k ) u k+1 = u k + x k+1 z k+1 - updating x k+1 is very slow - S λ (x) is the soft-thresholding Q: How to parallelize it? Zhimin ADMM 7

8 ADMM form: min 1 2 Ax b λ z 1 s.t. x z = 0 Divide A and b into several blocks Distributed ADMM form: min s.t. 1 2 N A i x i b i λ z 1 i=1 x i z = 0, for i = 1, 2,..., N Zhimin ADMM 8

9 Distributed ADMM: ui k+1 = ui k + xi k z k, for i = 1, 2,..., N = (A T i A i + ρi ) 1 (A T i b + ρ(z k u k )), for i = 1, 2,..., N x k+1 i z k+1 = S λ/nρ (x k+1 + u k+1 ) - x i, u i are local variable, updated by local systems - z is a global variable In each iteration: - gather xi k+1, ui k+1 and average to get x k+1, u k+1 - scatter the average x k+1, u k+1 to processors - update z k+1 in each processor Zhimin ADMM 9

10 Zhimin ADMM 10

11 Zhimin ADMM 11

12 Zhimin ADMM 12

13 Zhimin ADMM 13

14 Zhimin ADMM 14

15 example Lasso RICE A = randn(1024, 3000) xs(randsample(1024, k)) = randn(k, 1), sparsity k = 200 b = Axs, no noise is introduced run on RICE STIC, with 1, 2, 4, 8, 16, 32 cores result: Termination Rule: xs z xs < 10 3 Zhimin ADMM 15

16 Progress with respect to xs z xs : Zhimin ADMM 16

17 Termination rules: ρ N z z prev < 10 5 nn y & x i z < 10 5 nn max( y, N z ) N is the number processors, n is the dimension of x Zhimin ADMM 17

18 Progress of prime and dual residual norm: Figure: iteration vs z z pre Figure: iteration vs z x Zhimin ADMM 18

19 Progress with respect to xs x xs 10 2 x xs # of iteration Figure: iteration vs xs x xs Zhimin ADMM 19

20 MPI & Cluster Lasso RICE MPI: Cluster: A library for message passing Designed for high performance massive parallel machine and workstation clusters MPI Init, MPI Comm rank, MPI Comm size, MPI Finalize Basic functions: MPI Send, MPI Recv, MPI Wait Collective MPI Communications: MPI Reduce, MPI Allreduce Compile: mpicc -o helloworld helloworld.c Run: mpiexec -n 4./helloworld Login: ssh username@stic.rice.edu -Y Transfer files: scp filename username@stic.rice.edu: Load applications: module load openmpi Zhimin ADMM 20

21 MPI Allreduce Lasso RICE Combines values from all processes and distributes the result back to all processes int MPI Allreduce( void sendbuf, void recvbuf, int count, MPI Datatype datatype, MPI Op op, MPI Comm comm) MPI Op: - MPI MAX : maximum value - MPI MAX : minimum value - MPI SUM: summation - MPI PROD: production - MPI LAND: logical and Other options: Zhimin ADMM 21

22 Amazon Elastic Compute RICE Features: Cloud computing Provide resizeable computing capacity Handle big data Connection: 1. Sign up for EC2. (aws.amazon.com) 2. Launch instance. (11 steps) 3. Connect to Linux Instance. (8 steps) 4. Terminate instance. Zhimin ADMM 22