LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

Transcription

1 LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning Tianyi Chen Georgios Giannakis Tao Sun Wotao Yin UMN, ECE UCLA, Math NeurIPS 208

2 Overview M := {,...,M} <latexit sha_base64="k7jru5gta9yb+ipfjpt/zlno0yy=">aaacaxicbvdlssnafj3uv62vqbvbzwarxjssseerhiibn4uk9gfnkjpjpb06mqkze6geuvfx3lhqxk/4c6/cdpmoa0hlhzouzd77wksrpv2ng+rslk6tr5r3cxtbe/s7tn7b20luoljcwsmzddaijdksuttzug3kqtfasodyhqz9tsprcoq+l0ej8sp0ydtigkkjds3jzipiwybk6trl3mrhgufvpwgn+nbzafqzacxizutmsjr7ntfxihwghoumunk9vwn0x6gpkaykunjsxvjeb6haekzylfmlj/nppjau6oemblsfndwpv6eyfcsdgotgem9fatelpxp6+x6ujszyhpuk04ni+kugagnm4yeglwzqndufyunmrxemkedymtjijwv8ezm0z6uuu3xvauv6ly+jci7bctgdlrgadxalmqafmhgez+avvflpovbn3mwwtwpnmi/sd6/aeiejx9</latexit> J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., Large-scale distributed deep networks, Proc. NIPS., Lake Tahoe, NV, pp , 202 2

3 Overview M := {,...,M} <latexit sha_base64="k7jru5gta9yb+ipfjpt/zlno0yy=">aaacaxicbvdlssnafj3uv62vqbvbzwarxjssseerhiibn4uk9gfnkjpjpb06mqkze6geuvfx3lhqxk/4c6/cdpmoa0hlhzouzd77wksrpv2ng+rslk6tr5r3cxtbe/s7tn7b20luoljcwsmzddaijdksuttzug3kqtfasodyhqz9tsprcoq+l0ej8sp0ydtigkkjds3jzipiwybk6trl3mrhgufvpwgn+nbzafqzacxizutmsjr7ntfxihwghoumunk9vwn0x6gpkaykunjsxvjeb6haekzylfmlj/nppjau6oemblsfndwpv6eyfcsdgotgem9fatelpxp6+x6ujszyhpuk04ni+kugagnm4yeglwzqndufyunmrxemkedymtjijwv8ezm0z6uuu3xvauv6ly+jci7bctgdlrgadxalmqafmhgez+avvflpovbn3mwwtwpnmi/sd6/aeiejx9</latexit> q Solvers: gradient descent (GD), momentum methods q Our method improves GD by same convergence rate in theory reduced communication in theory more than 90% communication saving in practice J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le et al., Large-scale distributed deep networks, Proc. NIPS., Lake Tahoe, NV, pp , 202 3

4 <latexit sha_base64="3ycjpjs6cm9hqn3faiohgutr/e=">aaacyxicbvfntxsxepuuli9qyeupxcwijkqkabchwqupopcewgkkakhxwm06tmlf9q7swdrotx+sg5de+kfqtxjoisnzfn5vzjx+zgolhcbxaxcurx/y2nzabu83n3bjz4d3lu8tfz0ek5y+5ibe0oa0uojsjwwvodolhjipt8a/efzwcdzc4ezqgw0ji0csq7oqtt6xbjcddm+6iobei9vm/zruv6uu6skdvuyauvknlwbsviydoj/rmhnifcyop+pus0yotb2ajqvadsoo/e86huqlegblommjv7ymoelfga5auf6svzgoaklkitrtjpraf8cmpr99cafm5qzr2q6bfnhnsuw78m0jn7b0uf2jvj+kwnohfvtyzcpfvlhf0okmmkeoxhi4tgpaky08zuoprwcfqzd4bb6welfaiwopppaxktkrdpfg/uzzpj3eluz9vd66udw+sqhjetkpal0ixfyq3peu5+b+vbbrax/am3wyg8wksgwblmm/kvwso/z9u58a==</latexit> <latexit sha_base64="3ycjpjs6cm9hqn3faiohgutr/e=">aaacyxicbvfntxsxepuuli9qyeupxcwijkqkabchwqupopcewgkkakhxwm06tmlf9q7swdrotx+sg5de+kfqtxjoisnzfn5vzjx+zgolhcbxaxcurx/y2nzabu83n3bjz4d3lu8tfz0ek5y+5ibe0oa0uojsjwwvodolhjipt8a/efzwcdzc4ezqgw0ji0csq7oqtt6xbjcddm+6iobei9vm/zruv6uu6skdvuyauvknlwbsviydoj/rmhnifcyop+pus0yotb2ajqvadsoo/e86huqlegblommjv7ymoelfga5auf6svzgoaklkitrtjpraf8cmpr99cafm5qzr2q6bfnhnsuw78m0jn7b0uf2jvj+kwnohfvtyzcpfvlhf0okmmkeoxhi4tgpaky08zuoprwcfqzd4bb6welfaiwopppaxktkrdpfg/uzzpj3eluz9vd66udw+sqhjetkpal0ixfyq3peu5+b+vbbrax/am3wyg8wksgwblmm/kvwso/z9u58a==</latexit> <latexit sha_base64="3ycjpjs6cm9hqn3faiohgutr/e=">aaacyxicbvfntxsxepuuli9qyeupxcwijkqkabchwqupopcewgkkakhxwm06tmlf9q7swdrotx+sg5de+kfqtxjoisnzfn5vzjx+zgolhcbxaxcurx/y2nzabu83n3bjz4d3lu8tfz0ek5y+5ibe0oa0uojsjwwvodolhjipt8a/efzwcdzc4ezqgw0ji0csq7oqtt6xbjcddm+6iobei9vm/zruv6uu6skdvuyauvknlwbsviydoj/rmhnifcyop+pus0yotb2ajqvadsoo/e86huqlegblommjv7ymoelfga5auf6svzgoaklkitrtjpraf8cmpr99cafm5qzr2q6bfnhnsuw78m0jn7b0uf2jvj+kwnohfvtyzcpfvlhf0okmmkeoxhi4tgpaky08zuoprwcfqzd4bb6welfaiwopppaxktkrdpfg/uzzpj3eluz9vd66udw+sqhjetkpal0ixfyq3peu5+b+vbbrax/am3wyg8wksgwblmm/kvwso/z9u58a==</latexit> <latexit sha_base64="3ycjpjs6cm9hqn3faiohgutr/e=">aaacyxicbvfntxsxepuuli9qyeupxcwijkqkabchwqupopcewgkkakhxwm06tmlf9q7swdrotx+sg5de+kfqtxjoisnzfn5vzjx+zgolhcbxaxcurx/y2nzabu83n3bjz4d3lu8tfz0ek5y+5ibe0oa0uojsjwwvodolhjipt8a/efzwcdzc4ezqgw0ji0csq7oqtt6xbjcddm+6iobei9vm/zruv6uu6skdvuyauvknlwbsviydoj/rmhnifcyop+pus0yotb2ajqvadsoo/e86huqlegblommjv7ymoelfga5auf6svzgoaklkitrtjpraf8cmpr99cafm5qzr2q6bfnhnsuw78m0jn7b0uf2jvj+kwnohfvtyzcpfvlhf0okmmkeoxhi4tgpaky08zuoprwcfqzd4bb6welfaiwopppaxktkrdpfg/uzzpj3eluz9vd66udw+sqhjetkpal0ixfyq3peu5+b+vbbrax/am3wyg8wksgwblmm/kvwso/z9u58a==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixvl86vqrn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlteldvesarh6kc0vo5wtdmajnchizrxwyasblykjnyqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha_base64="s6cfni5l/+zpwvoelvdqmdidqvu=">aaacdxicbva9swnbej3z2xg2tosbie28c5gs8hgwklbrcexw9xmkyzz2zt254rw5gfy+fdslbsxtfpfujekuoodzr7vzbazl0qvtot7x97c4tlyyuraemmjvlmxdkpn22sgs4apfgjuyvqciwajakje5sizcollinhueff/sgjjwjvqfrktox9rxssy7kpe7lknqykcxdjopdjjtxgml+lywsbwj2ju8pieghn8pc+ewu6n6dx8cnk+cgancdfedymfytxgwc0cobwtwe+pnamhyzuyl8lmiht5epui5ajgwnh2pjlsza6c0mw9xliniu3unxm5xrzy03xgsap7yve/7xwrr3tdi5mphqfpprlomelakxlrsce5q5ahyi92uja/qicexzcmfepw9ez40j+ubxw+ufvidpdihggrwamdwavfqaa6p8ayv8oy9es/e+zsubw+w2y78gvfxdconn0c=</latexit> <latexit sha_base64="s6cfni5l/+zpwvoelvdqmdidqvu=">aaacdxicbva9swnbej3z2xg2tosbie28c5gs8hgwklbrcexw9xmkyzz2zt254rw5gfy+fdslbsxtfpfujekuoodzr7vzbazl0qvtot7x97c4tlyyuraemmjvlmxdkpn22sgs4apfgjuyvqciwajakje5sizcollinhueff/sgjjwjvqfrktox9rxssy7kpe7lknqykcxdjopdjjtxgml+lywsbwj2ju8pieghn8pc+ewu6n6dx8cnk+cgancdfedymfytxgwc0cobwtwe+pnamhyzuyl8lmiht5epui5ajgwnh2pjlsza6c0mw9xliniu3unxm5xrzy03xgsap7yve/7xwrr3tdi5mphqfpprlomelakxlrsce5q5ahyi92uja/qicexzcmfepw9ez40j+ubxw+ufvidpdihggrwamdwavfqaa6p8ayv8oy9es/e+zsubw+w2y78gvfxdconn0c=</latexit> <latexit sha_base64="ad9+7z/64n0l6c5cvtcf9t7i5sq=">aaacghicbva9twjben3ze/eltbtzseywwtsblyk2fhayyefcizlbftiwt3fzntmhf36gjx/fxkjjbon8n+4bhyiv2ezlezozmrfeuhh03w9nzxvtfwmzt5xf3tnd2y8chnznlgjgayyskw4gylguitdqootnwhmia8kbwfam8xtpxbsrqqccxbwdql+jnmcavuouzn0fgytuzydp3bgt+ohol/wgkl0zcu2x+jjgcophyeacdqpft+xoqzejnydfmkeu5j43yglivfijbjt8twy2ylofezycd5pdi+bdahpw5yqcllpp9pdxvtukl3ai7r9culu/d2rqmiynwlcdgwi4m/ueeuxdtvoh4gs5yrnbvursjgiweu0kzrnkksxatlc7ujyadqxtlnkbgrd48jkpx5q9t+zdu8xk9tyohdkmj6rephjjkuswvemnmpjmxsk7+xbendfn0/mala44854j8gfo5aeuvkdg</latexit> <latexit sha_base64="s6z3bog3qzlpbrnhhzck7zmxid8=">aaacghicbvbns8naen34bf2qevsywas9eqepra9efbqsfvoyplst+3szsbstoqs+jo8+fe8efdea2/+gzdtdlp9sozjvrlm5owjfazd98uzmz2bxhcwi6trk6tb5q3txomtjxjdrblwn+hylguitdroot3ieyqhzlfhf3z3l975nqiwn3iiofbbf0looibwqlvpvqvhbiyn4gkl8pwlr+k7r4fxrjtbph9mh97hgh40m+dga54lbdmehf4hwkqgpct8ojvx2znoikmqrjmp6byjcbrsekh5b8paewb+6vgmpgoibibsfnqr7vmnttqztu0jh6s+odcktr2kri8cemfzy8t+vmwlnnmiesllkik0gdvjjmaz5srqtngcob5ya08luslkpndc0wzzscn70yx9j46jquvxv5rhsoyviwci7zjfse4+ckbq5inekthh5ii/kjbw7z86r8+f8tkpnnkjnm/ycm/oge5ygqg==</latexit> <latexit sha_base64="s6z3bog3qzlpbrnhhzck7zmxid8=">aaacghicbvbns8naen34bf2qevsywas9eqepra9efbqsfvoyplst+3szsbstoqs+jo8+fe8efdea2/+gzdtdlp9sozjvrlm5owjfazd98uzmz2bxhcwi6trk6tb5q3txomtjxjdrblwn+hylguitdroot3ieyqhzlfhf3z3l975nqiwn3iiofbbf0looibwqlvpvqvhbiyn4gkl8pwlr+k7r4fxrjtbph9mh97hgh40m+dga54lbdmehf4hwkqgpct8ojvx2znoikmqrjmp6byjcbrsekh5b8paewb+6vgmpgoibibsfnqr7vmnttqztu0jh6s+odcktr2kri8cemfzy8t+vmwlnnmiesllkik0gdvjjmaz5srqtngcob5ya08luslkpndc0wzzscn70yx9j46jquvxv5rhsoyviwci7zjfse4+ckbq5inekthh5ii/kjbw7z86r8+f8tkpnnkjnm/ycm/oge5ygqg==</latexit> <latexit sha_base64="s6z3bog3qzlpbrnhhzck7zmxid8=">aaacghicbvbns8naen34bf2qevsywas9eqepra9efbqsfvoyplst+3szsbstoqs+jo8+fe8efdea2/+gzdtdlp9sozjvrlm5owjfazd98uzmz2bxhcwi6trk6tb5q3txomtjxjdrblwn+hylguitdroot3ieyqhzlfhf3z3l975nqiwn3iiofbbf0looibwqlvpvqvhbiyn4gkl8pwlr+k7r4fxrjtbph9mh97hgh40m+dga54lbdmehf4hwkqgpct8ojvx2znoikmqrjmp6byjcbrsekh5b8paewb+6vgmpgoibibsfnqr7vmnttqztu0jh6s+odcktr2kri8cemfzy8t+vmwlnnmiesllkik0gdvjjmaz5srqtngcob5ya08luslkpndc0wzzscn70yx9j46jquvxv5rhsoyviwci7zjfse4+ckbq5inekthh5ii/kjbw7z86r8+f8tkpnnkjnm/ycm/oge5ygqg==</latexit> <latexit sha_base64="s6z3bog3qzlpbrnhhzck7zmxid8=">aaacghicbvbns8naen34bf2qevsywas9eqepra9efbqsfvoyplst+3szsbstoqs+jo8+fe8efdea2/+gzdtdlp9sozjvrlm5owjfazd98uzmz2bxhcwi6trk6tb5q3txomtjxjdrblwn+hylguitdroot3ieyqhzlfhf3z3l975nqiwn3iiofbbf0looibwqlvpvqvhbiyn4gkl8pwlr+k7r4fxrjtbph9mh97hgh40m+dga54lbdmehf4hwkqgpct8ojvx2znoikmqrjmp6byjcbrsekh5b8paewb+6vgmpgoibibsfnqr7vmnttqztu0jh6s+odcktr2kri8cemfzy8t+vmwlnnmiesllkik0gdvjjmaz5srqtngcob5ya08luslkpndc0wzzscn70yx9j46jquvxv5rhsoyviwci7zjfse4+ckbq5inekthh5ii/kjbw7z86r8+f8tkpnnkjnm/ycm/oge5ygqg==</latexit> Vanilla GD implementation + = X m2m rl m rl rl m rl <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> M q Per iteration communication overhead for M uploads (one per worker) 4

5 Prior art q Communication-efficient distributed learning Quantized gradient descent [Kashyap et al., 07], [Alistarh et al., 7], [Suresh et al., 7] Increasing computation before communication [Jaggi et al., 4], [Ma et al., 7], [Smith et al., 7] Sparse SGD with large entries [Aji-Heafield 7], [Sun et al., 7], [Lin et al., 8], [Stich et al., 8] Ø number of communication rounds is not reduced Our contribution Adaptively skip communication, provable communication reduction 5

6 <latexit sha_base64="xjaypflh2d2effhubkez30ozte=">aaac2hicbvjdixmxfm2mx2v92kqpvksl0ew2zmicvgglvvigsildxwzqccdno6fjzkjuccum+kcir/403/wv/guzbcwozeehm65offmi6+udjgkv6l4ytvr2/s3ezdun3n7n7/3vtv9awizevvwnpc3bcsspgkfgj88ok0lksz/nyvauffrlwydk8xulphowrs4lbwxuv/n8lln3eqhxtmsbelz0s+fps3llmv5yebvbbbhybhaz4zatzjoojbiddrwucugg7yn02mh5wb/siwycdcwpcswve+vgnz/l5wlickdfusjo4uchadnmhwt9qtjkkevg3qlbmqbjn/j5uvvnbcia/tuemavdjyffyjzoeq52ogc9hisybgtdctf36yrr6jdazoi9tmabpmr24w4n2bbchuwmwbldrys5tuup8xdrlu9uodn8umtekyknbv6yzaqvhtqoaujwhv8olsmax/iveuir098ixwemzuzqm0ndhg+oj7xxskyfkmrmsldwnx+qosfjwqnx5kmv0df4q/w5/hz/36tg0xbpa/jfxd/+ahy55xq=</latexit> <latexit sha_base64="xjaypflh2d2effhubkez30ozte=">aaac2hicbvjdixmxfm2mx2v92kqpvksl0ew2zmicvgglvvigsildxwzqccdno6fjzkjuccum+kcir/403/wv/guzbcwozeehm65offmi6+udjgkv6l4ytvr2/s3ezdun3n7n7/3vtv9awizevvwnpc3bcsspgkfgj88ok0lksz/nyvauffrlwydk8xulphowrs4lbwxuv/n8lln3eqhxtmsbelz0s+fps3llmv5yebvbbbhybhaz4zatzjoojbiddrwucugg7yn02mh5wb/siwycdcwpcswve+vgnz/l5wlickdfusjo4uchadnmhwt9qtjkkevg3qlbmqbjn/j5uvvnbcia/tuemavdjyffyjzoeq52ogc9hisybgtdctf36yrr6jdazoi9tmabpmr24w4n2bbchuwmwbldrys5tuup8xdrlu9uodn8umtekyknbv6yzaqvhtqoaujwhv8olsmax/iveuir098ixwemzuzqm0ndhg+oj7xxskyfkmrmsldwnx+qosfjwqnx5kmv0df4q/w5/hz/36tg0xbpa/jfxd/+ahy55xq=</latexit> <latexit sha_base64="xjaypflh2d2effhubkez30ozte=">aaac2hicbvjdixmxfm2mx2v92kqpvksl0ew2zmicvgglvvigsildxwzqccdno6fjzkjuccum+kcir/403/wv/guzbcwozeehm65offmi6+udjgkv6l4ytvr2/s3ezdun3n7n7/3vtv9awizevvwnpc3bcsspgkfgj88ok0lksz/nyvauffrlwydk8xulphowrs4lbwxuv/n8lln3eqhxtmsbelz0s+fps3llmv5yebvbbbhybhaz4zatzjoojbiddrwucugg7yn02mh5wb/siwycdcwpcswve+vgnz/l5wlickdfusjo4uchadnmhwt9qtjkkevg3qlbmqbjn/j5uvvnbcia/tuemavdjyffyjzoeq52ogc9hisybgtdctf36yrr6jdazoi9tmabpmr24w4n2bbchuwmwbldrys5tuup8xdrlu9uodn8umtekyknbv6yzaqvhtqoaujwhv8olsmax/iveuir098ixwemzuzqm0ndhg+oj7xxskyfkmrmsldwnx+qosfjwqnx5kmv0df4q/w5/hz/36tg0xbpa/jfxd/+ahy55xq=</latexit> <latexit sha_base64="xjaypflh2d2effhubkez30ozte=">aaac2hicbvjdixmxfm2mx2v92kqpvksl0ew2zmicvgglvvigsildxwzqccdno6fjzkjuccum+kcir/403/wv/guzbcwozeehm65offmi6+udjgkv6l4ytvr2/s3ezdun3n7n7/3vtv9awizevvwnpc3bcsspgkfgj88ok0lksz/nyvauffrlwydk8xulphowrs4lbwxuv/n8lln3eqhxtmsbelz0s+fps3llmv5yebvbbbhybhaz4zatzjoojbiddrwucugg7yn02mh5wb/siwycdcwpcswve+vgnz/l5wlickdfusjo4uchadnmhwt9qtjkkevg3qlbmqbjn/j5uvvnbcia/tuemavdjyffyjzoeq52ogc9hisybgtdctf36yrr6jdazoi9tmabpmr24w4n2bbchuwmwbldrys5tuup8xdrlu9uodn8umtekyknbv6yzaqvhtqoaujwhv8olsmax/iveuir098ixwemzuzqm0ndhg+oj7xxskyfkmrmsldwnx+qosfjwqnx5kmv0df4q/w5/hz/36tg0xbpa/jfxd/+ahy55xq=</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="0izsstpvxcdybhaas99noor5ey=">aaacghicbva9swnben2l3/erammzgityxdsrtbrtlcwima/ixtc32srl9vao3tkhhpkznv4vgwtfbnp5b9xlumjig2uf780wmy+iptdout9obml5zxvtfso/ubw9svy26+zkngmvkki90iwhapfk+iqmkbseyqbplxg8fn5tefudyiug84jhkrhj4sxceardqunpokagmpz0dsuhb8wprk/lbjdtmgnov9bhpeuapg8w5arekbtmdgc4sb0akzizkuzd2oxflqq6qstcm6bkxtllqkjjko7yfgb4dg0cpnyvehltsiehjeixvtq0g2n7fnkj+rsjhdbka9rkelbv5rm/m9rjti9bkvcxqlyxaaduomkgnesjdormjouq0uaawf3pawpghjalpm2bg/+5evsoyt7btm7py9exc/iwceh5iiuiecuybw5jrvsjyw8kfytj6cf+fn+xs+pqu5z9zzqp7agf8atcigjg==</latexit> <latexit sha_base64="kl3zlmnta00f+bxwcxva28jxtua=">aaacbxicbvbns8naen3urq/oh7sfgetyupgh4lxrwifewhnlfstpn26wytdzdccb48a948aciv/+dn/+n2zactj4yelw3w8y8iofmacf5sgplyyura8x0sbmvaovbvxvheqktrozgpzdogczgq0nnmc2okeegucwshwyuk37keqfosbpuraj0hfsjbroo3utq8zjxkor8a3q0+lgqind/hh69plp+jmgrejm5myylhv2p9el6zpbejttptque6i/yxizsihccllfssedkkfooykeohys+kxy3xslb4oy2lkadxvf09kjfjqfawmmyj6ooa9ifif0leo5ntcspbkfni8kuyx3jsss4xyrqzuegecqzurxtazgeahncyytgzr+8sjrviutu3ovtcq2axfeb+ginsaxnaeaukreaupaan9ijerufr2xqz3metbsuf2ud/yh8aysfmey=</latexit> <latexit sha_base64="kl3zlmnta00f+bxwcxva28jxtua=">aaacbxicbvbns8naen3urq/oh7sfgetyupgh4lxrwifewhnlfstpn26wytdzdccb48a948aciv/+dn/+n2zactj4yelw3w8y8iofmacf5sgplyyura8x0sbmvaovbvxvheqktrozgpzdogczgq0nnmc2okeegucwshwyuk37keqfosbpuraj0hfsjbroo3utq8zjxkor8a3q0+lgqind/hh69plp+jmgrejm5myylhv2p9el6zpbejttptque6i/yxizsihccllfssedkkfooykeohys+kxy3xslb4oy2lkadxvf09kjfjqfawmmyj6ooa9ifif0leo5ntcspbkfni8kuyx3jsss4xyrqzuegecqzurxtazgeahncyytgzr+8sjrviutu3ovtcq2axfeb+ginsaxnaeaukreaupaan9ijerufr2xqz3metbsuf2ud/yh8aysfmey=</latexit> <latexit sha_base64="kl3zlmnta00f+bxwcxva28jxtua=">aaacbxicbvbns8naen3urq/oh7sfgetyupgh4lxrwifewhnlfstpn26wytdzdccb48a948aciv/+dn/+n2zactj4yelw3w8y8iofmacf5sgplyyura8x0sbmvaovbvxvheqktrozgpzdogczgq0nnmc2okeegucwshwyuk37keqfosbpuraj0hfsjbroo3utq8zjxkor8a3q0+lgqind/hh69plp+jmgrejm5myylhv2p9el6zpbejttptque6i/yxizsihccllfssedkkfooykeohys+kxy3xslb4oy2lkadxvf09kjfjqfawmmyj6ooa9ifif0leo5ntcspbkfni8kuyx3jsss4xyrqzuegecqzurxtazgeahncyytgzr+8sjrviutu3ovtcq2axfeb+ginsaxnaeaukreaupaan9ijerufr2xqz3metbsuf2ud/yh8aysfmey=</latexit> <latexit sha_base64="kl3zlmnta00f+bxwcxva28jxtua=">aaacbxicbvbns8naen3urq/oh7sfgetyupgh4lxrwifewhnlfstpn26wytdzdccb48a948aciv/+dn/+n2zactj4yelw3w8y8iofmacf5sgplyyura8x0sbmvaovbvxvheqktrozgpzdogczgq0nnmc2okeegucwshwyuk37keqfosbpuraj0hfsjbroo3utq8zjxkor8a3q0+lgqind/hh69plp+jmgrejm5myylhv2p9el6zpbejttptque6i/yxizsihccllfssedkkfooykeohys+kxy3xslb4oy2lkadxvf09kjfjqfawmmyj6ooa9ifif0leo5ntcspbkfni8kuyx3jsss4xyrqzuegecqzurxtazgeahncyytgzr+8sjrviutu3ovtcq2axfeb+ginsaxnaeaukreaupaan9ijerufr2xqz3metbsuf2ud/yh8aysfmey=</latexit> <latexit sha_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixvl86vqrn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlteldvesarh6kc0vo5wtdmajnchizrxwyasblykjnyqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha_base64="s6cfni5l/+zpwvoelvdqmdidqvu=">aaacdxicbva9swnbej3z2xg2tosbie28c5gs8hgwklbrcexw9xmkyzz2zt254rw5gfy+fdslbsxtfpfujekuoodzr7vzbazl0qvtot7x97c4tlyyuraemmjvlmxdkpn22sgs4apfgjuyvqciwajakje5sizcollinhueff/sgjjwjvqfrktox9rxssy7kpe7lknqykcxdjopdjjtxgml+lywsbwj2ju8pieghn8pc+ewu6n6dx8cnk+cgancdfedymfytxgwc0cobwtwe+pnamhyzuyl8lmiht5epui5ajgwnh2pjlsza6c0mw9xliniu3unxm5xrzy03xgsap7yve/7xwrr3tdi5mphqfpprlomelakxlrsce5q5ahyi92uja/qicexzcmfepw9ez40j+ubxw+ufvidpdihggrwamdwavfqaa6p8ayv8oy9es/e+zsubw+w2y78gvfxdconn0c=</latexit> <latexit sha_base64="s6cfni5l/+zpwvoelvdqmdidqvu=">aaacdxicbva9swnbej3z2xg2tosbie28c5gs8hgwklbrcexw9xmkyzz2zt254rw5gfy+fdslbsxtfpfujekuoodzr7vzbazl0qvtot7x97c4tlyyuraemmjvlmxdkpn22sgs4apfgjuyvqciwajakje5sizcollinhueff/sgjjwjvqfrktox9rxssy7kpe7lknqykcxdjopdjjtxgml+lywsbwj2ju8pieghn8pc+ewu6n6dx8cnk+cgancdfedymfytxgwc0cobwtwe+pnamhyzuyl8lmiht5epui5ajgwnh2pjlsza6c0mw9xliniu3unxm5xrzy03xgsap7yve/7xwrr3tdi5mphqfpprlomelakxlrsce5q5ahyi92uja/qicexzcmfepw9ez40j+ubxw+ufvidpdihggrwamdwavfqaa6p8ayv8oy9es/e+zsubw+w2y78gvfxdconn0c=</latexit> <latexit sha_base64="ad9+7z/64n0l6c5cvtcf9t7i5sq=">aaacghicbva9twjben3ze/eltbtzseywwtsblyk2fhayyefcizlbftiwt3fzntmhf36gjx/fxkjjbon8n+4bhyiv2ezlezozmrfeuhh03w9nzxvtfwmzt5xf3tnd2y8chnznlgjgayyskw4gylguitdqootnwhmia8kbwfam8xtpxbsrqqccxbwdql+jnmcavuouzn0fgytuzydp3bgt+ohol/wgkl0zcu2x+jjgcophyeacdqpft+xoqzejnydfmkeu5j43yglivfijbjt8twy2ylofezycd5pdi+bdahpw5yqcllpp9pdxvtukl3ai7r9culu/d2rqmiynwlcdgwi4m/ueeuxdtvoh4gs5yrnbvursjgiweu0kzrnkksxatlc7ujyadqxtlnkbgrd48jkpx5q9t+zdu8xk9tyohdkmj6rephjjkuswvemnmpjmxsk7+xbendfn0/mala44854j8gfo5aeuvkdg</latexit> Our LAG implementation + = X Fresh gradient rl m ( ) m2m k Old gradient X rl m ˆ k m m2m/m k rl rl m q Select a subset of workers M k M to upload q Remaining workers in M/M k <latexit sha_base64="qvz+pv4h383let9dvojmypjlyi=">aaacanicbvdlsgmxfltx7w+rl2jm2arxjwzutblwy0boyj9qgcsmttthmyyq5iryldc+ctuxcjiq9w59+ytino64haytn33useiofmacf5sgorq2vrg8xn0tb2zu6evx/quneqcw2smmeye2bforo0qznmtjniiqoa03ywupz67xsqfyvfrr4ni/wqlcqeaynlopmo9gjq4nnujyddhp9w7us8toxzkblrm3j2xi0ejzn4/jmlehszmluq6tql9devncketkpcqmmaywgpanvtgico/m60wqadg6amwluyijwbq744mr0qno8burlgpai3ff/zuqkol/ymistvvjd5q2hkky7rna/uz5iszcegyckz+ssiqywx0sakgnbxvx5mbsqfdepudecr2axgeyzibm3dhhopwbqoaoeheiixeluerwfrzxqflxasvocq/sd6+aavu5bx</latexit> do not upload 6

7 <latexit sha_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixvl86vqrn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlteldvesarh6kc0vo5wtdmajnchizrxwyasblykjnyqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha_base64="0r4hplkwhklcrgku4unegnfajvs=">aaacw3icbvjnb9naef2brxikbk5cvlri7sgrzqwocc5igbqk0lbkptf4m7zxwx90dwpbc3+izd+ap+ddrwkad3sap/em9hmvtmkspsfp0kwjt379/cpbw9ojw8zon42ehp7zqjms5rhrlzhowqfwjckk8bw2cewi8szzfoj0sys0vlxln9rwucwgkwqjjcnvupf4r3kxduxjsqanjcg+ed2vyhezcciqftabgt/oue5erqxm045mqzn50buomjrf24zidtdbd9rzhjjrwpaurhxqhmsvp/2ue6dqhnh4ezdle9m2myni66s9xprqtxkfrnnofvw3ipthi+5itxj/fupjngsvjddyu4qimpqndsmpsr6kxwipcqiyld0so0c7dbh0tf+wznu8r409jfmf+w+ggsn3yprmayunrsohtevd6dulu2xdejayb5q2mlpfu93yttiosw89agmun5xlhlyh5h9az0j888m3wenrarxn468ro2av2et2zgl2hrjh9mmzzkmpgwxwffahbpwkmx7u8jg79tz9l+ep/4ayclksg==</latexit> <latexit sha_base64="0r4hplkwhklcrgku4unegnfajvs=">aaacw3icbvjnb9naef2brxikbk5cvlri7sgrzqwocc5igbqk0lbkptf4m7zxwx90dwpbc3+izd+ap+ddrwkad3sap/em9hmvtmkspsfp0kwjt379/cpbw9ojw8zon42ehp7zqjms5rhrlzhowqfwjckk8bw2cewi8szzfoj0sys0vlxln9rwucwgkwqjjcnvupf4r3kxduxjsqanjcg+ed2vyhezcciqftabgt/oue5erqxm045mqzn50buomjrf24zidtdbd9rzhjjrwpaurhxqhmsvp/2ue6dqhnh4ezdle9m2myni66s9xprqtxkfrnnofvw3ipthi+5itxj/fupjngsvjddyu4qimpqndsmpsr6kxwipcqiyld0so0c7dbh0tf+wznu8r409jfmf+w+ggsn3yprmayunrsohtevd6dulu2xdejayb5q2mlpfu93yttiosw89agmun5xlhlyh5h9az0j888m3wenrarxn468ro2av2et2zgl2hrjh9mmzzkmpgwxwffahbpwkmx7u8jg79tz9l+ep/4ayclksg==</latexit> <latexit sha_base64="ckjerppo0exvggranxgvbpzutg=">aaacznicbvjna9wwejxdr3t7twmpvyguheswi5legznpdawttbnaqvnmtbktlhzdqrxykuyxvv7eusp6p+ovn5cm3ha6phepgy0o6rs0miu/qrce/cfphy093jw5omz5y+g+y/pbfkblma8vkw5smakjbwyouqlliojoeiuoe/wp6+fi2mlax+iptklaritewlb/tucvibvzczu6fmq6lamq6kfmqwbutkdscsuq3spvcxy5glhozy3sqh4/78hndmlx+6dbjunl6u4oseeupqa4ixv3uwfkpojpvwwdvexq0f2njpvyroyrwy6h3cvuboplcbrnom3quydegrhzxxq5/mlwja8lozershyerxuuhbiuxilmwgorkubrymtcqw2fsau3xudd33pmrdps+korbtl/hq4k27btmwva3n7wwrjpm9eyvls4qasahezdobrwfeva7paupbeccyd4eb6xinpwq8u/q9ohxdffvjdchy0ianj/cuanrztxrfhxpm35ide5jicka9ksmaebx+dq+bb4mjpeb024fcunqx2nlfkvwh//afhrevw</latexit> LAG: GD under two alternative communication rules q Worker-side rule (LAG-WK): Include worker m in if Old gradient rl m rl m ˆ k m M Gradient innovation Optimization progress T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8,

8 <latexit sha_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixvl86vqrn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlteldvesarh6kc0vo5wtdmajnchizrxwyasblykjnyqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha_base64="0r4hplkwhklcrgku4unegnfajvs=">aaacw3icbvjnb9naef2brxikbk5cvlri7sgrzqwocc5igbqk0lbkptf4m7zxwx90dwpbc3+izd+ap+ddrwkad3sap/em9hmvtmkspsfp0kwjt379/cpbw9ojw8zon42ehp7zqjms5rhrlzhowqfwjckk8bw2cewi8szzfoj0sys0vlxln9rwucwgkwqjjcnvupf4r3kxduxjsqanjcg+ed2vyhezcciqftabgt/oue5erqxm045mqzn50buomjrf24zidtdbd9rzhjjrwpaurhxqhmsvp/2ue6dqhnh4ezdle9m2myni66s9xprqtxkfrnnofvw3ipthi+5itxj/fupjngsvjddyu4qimpqndsmpsr6kxwipcqiyld0so0c7dbh0tf+wznu8r409jfmf+w+ggsn3yprmayunrsohtevd6dulu2xdejayb5q2mlpfu93yttiosw89agmun5xlhlyh5h9az0j888m3wenrarxn468ro2av2et2zgl2hrjh9mmzzkmpgwxwffahbpwkmx7u8jg79tz9l+ep/4ayclksg==</latexit> <latexit sha_base64="0r4hplkwhklcrgku4unegnfajvs=">aaacw3icbvjnb9naef2brxikbk5cvlri7sgrzqwocc5igbqk0lbkptf4m7zxwx90dwpbc3+izd+ap+ddrwkad3sap/em9hmvtmkspsfp0kwjt379/cpbw9ojw8zon42ehp7zqjms5rhrlzhowqfwjckk8bw2cewi8szzfoj0sys0vlxln9rwucwgkwqjjcnvupf4r3kxduxjsqanjcg+ed2vyhezcciqftabgt/oue5erqxm045mqzn50buomjrf24zidtdbd9rzhjjrwpaurhxqhmsvp/2ue6dqhnh4ezdle9m2myni66s9xprqtxkfrnnofvw3ipthi+5itxj/fupjngsvjddyu4qimpqndsmpsr6kxwipcqiyld0so0c7dbh0tf+wznu8r409jfmf+w+ggsn3yprmayunrsohtevd6dulu2xdejayb5q2mlpfu93yttiosw89agmun5xlhlyh5h9az0j888m3wenrarxn468ro2av2et2zgl2hrjh9mmzzkmpgwxwffahbpwkmx7u8jg79tz9l+ep/4ayclksg==</latexit> <latexit sha_base64="ckjerppo0exvggranxgvbpzutg=">aaacznicbvjna9wwejxdr3t7twmpvyguheswi5legznpdawttbnaqvnmtbktlhzdqrxykuyxvv7eusp6p+ovn5cm3ha6phepgy0o6rs0miu/qrce/cfphy093jw5omz5y+g+y/pbfkblma8vkw5smakjbwyouqlliojoeiuoe/wp6+fi2mlax+iptklaritewlb/tucvibvzczu6fmq6lamq6kfmqwbutkdscsuq3spvcxy5glhozy3sqh4/78hndmlx+6dbjunl6u4oseeupqa4ixv3uwfkpojpvwwdvexq0f2njpvyroyrwy6h3cvuboplcbrnom3quydegrhzxxq5/mlwja8lozershyerxuuhbiuxilmwgorkubrymtcqw2fsau3xudd33pmrdps+korbtl/hq4k27btmwva3n7wwrjpm9eyvls4qasahezdobrwfeva7paupbeccyd4eb6xinpwq8u/q9ohxdffvjdchy0ianj/cuanrztxrfhxpm35ide5jicka9ksmaebx+dq+bb4mjpeb024fcunqx2nlfkvwh//afhrevw</latexit> <latexit sha_base64="isvjkbigrcoxdot70vzudktbfj4=">aaab8xicbva9swnbej2lxzf+rstfongfe5e0djgy2erwxxgcos9zsrzsrt37o4j4ci/slfqxnz/y+e/czncoykpbh7vztazl0oen9b3v73c2vrg5lzxu7szu7d/ud48apo4qwblbaxbkfuooakg5zbgeei5wrwfy0vpn5rsfuhsfqwu4sdcudkj7gjfonpwzdrgw5m/zkrzxq/4czjueoalajnqv/nxtxyyvqcwtjho4cc2zki2namclrqpwysymrixffjzowm88jwdo6znbrf0ps+bq74mmsmmmmnkdktqrwfzm4n9ej7wd6zdjkkktkrzynegfstgzvu/6xcozyuiizzq7wwkbuu2zdsgvxajb8surphlrdfxqch9zqv3mcrthbe7hhak4ghrcqh0awedbm7zcm2e8f+/d+i0frx85hj+wpv8atlpkiw=</latexit> <latexit sha_base64="isvjkbigrcoxdot70vzudktbfj4=">aaab8xicbva9swnbej2lxzf+rstfongfe5e0djgy2erwxxgcos9zsrzsrt37o4j4ci/slfqxnz/y+e/czncoykpbh7vztazl0oen9b3v73c2vrg5lzxu7szu7d/ud48apo4qwblbaxbkfuooakg5zbgeei5wrwfy0vpn5rsfuhsfqwu4sdcudkj7gjfonpwzdrgw5m/zkrzxq/4czjueoalajnqv/nxtxyyvqcwtjho4cc2zki2namclrqpwysymrixffjzowm88jwdo6znbrf0ps+bq74mmsmmmmnkdktqrwfzm4n9ej7wd6zdjkkktkrzynegfstgzvu/6xcozyuiizzq7wwkbuu2zdsgvxajb8surphlrdfxqch9zqv3mcrthbe7hhak4ghrcqh0awedbm7zcm2e8f+/d+i0frx85hj+wpv8atlpkiw=</latexit> <latexit sha_base64="isvjkbigrcoxdot70vzudktbfj4=">aaab8xicbva9swnbej2lxzf+rstfongfe5e0djgy2erwxxgcos9zsrzsrt37o4j4ci/slfqxnz/y+e/czncoykpbh7vztazl0oen9b3v73c2vrg5lzxu7szu7d/ud48apo4qwblbaxbkfuooakg5zbgeei5wrwfy0vpn5rsfuhsfqwu4sdcudkj7gjfonpwzdrgw5m/zkrzxq/4czjueoalajnqv/nxtxyyvqcwtjho4cc2zki2namclrqpwysymrixffjzowm88jwdo6znbrf0ps+bq74mmsmmmmnkdktqrwfzm4n9ej7wd6zdjkkktkrzynegfstgzvu/6xcozyuiizzq7wwkbuu2zdsgvxajb8surphlrdfxqch9zqv3mcrthbe7hhak4ghrcqh0awedbm7zcm2e8f+/d+i0frx85hj+wpv8atlpkiw=</latexit> <latexit sha_base64="isvjkbigrcoxdot70vzudktbfj4=">aaab8xicbva9swnbej2lxzf+rstfongfe5e0djgy2erwxxgcos9zsrzsrt37o4j4ci/slfqxnz/y+e/czncoykpbh7vztazl0oen9b3v73c2vrg5lzxu7szu7d/ud48apo4qwblbaxbkfuooakg5zbgeei5wrwfy0vpn5rsfuhsfqwu4sdcudkj7gjfonpwzdrgw5m/zkrzxq/4czjueoalajnqv/nxtxyyvqcwtjho4cc2zki2namclrqpwysymrixffjzowm88jwdo6znbrf0ps+bq74mmsmmmmnkdktqrwfzm4n9ej7wd6zdjkkktkrzynegfstgzvu/6xcozyuiizzq7wwkbuu2zdsgvxajb8surphlrdfxqch9zqv3mcrthbe7hhak4ghrcqh0awedbm7zcm2e8f+/d+i0frx85hj+wpv8atlpkiw=</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="hp+6lruf2d3tzaldqaqqvekmxyw=">aaab2xicbzdnsgmxfixvl86vqrn8eiucozbnqpuhfzwbzco5rm5k4bmskmyr2hdh0bf25efc93vo3pz0jbdwq+zknivscullqubn9ebwd3b/+gfugfnfzjk9nmo2fz0gjsilzl5jnmfpxu2cvjcp8lgzylffbj6f0i77+gstlxtzqrmmr4wmtuck7o6oyaraadlmw2ivxdc9yanb+gss7kddujxa0dhefbucunsafw7g9liwuxuz7ggupnm7rrtrxzzi6dk7a0n+5oykv394ukz9bostjdzdhn7ga2mp/lbiwlteldvesarh6kc0vo5wtdmajnchizrxwyasblykjnyqa8z3hysbg29d77odbu3wmya6nmmfxeein3ahd9cblghi4bxevyn35n2suqp569lo4i+8zx84xio4</latexit> <latexit sha_base64="exy2rvibegrsormj8+msfvxw2g=">aaacknicdzhpbtqwemad8k8sbraocmciqiqhrpje4fibkaavvcs2rbterhpvjlhwtoi9qvqlesuehbtvg7nzjgixksx/+ohm/4mrbvyfew/gvdgzvu37+zchd3bvf/g4fjr7qmrgitxkitd2fmuhgpv4pquatyvlyjjnz6ly7d9/uw7wqeq8iutapwbyeuvkqnkutl+czwy8ubl4pkltniltzl+aguvsnbdla9eadrusxwjuwixb3e3viscv3grwzbt3lwfoqexi4//eag6lmda+//ptiuohazkc3o5xojynerge9ekwge/lukn2goboengp8wiko3bkqqg52zxvno8butkauxgonfygxcjjmvszdo5u3a3y6/8gtbs8r6uxjf078rwjcuh9u/necfu5rr4bbcrkhs9bxvzd0qlnjoldwau8x7vfgfsihjr7waazwflcscvkhkf9qbef/98nvxejijo0n8jwi77cl7zvzzzf6xi/aenbapk8gt4cj4ehwmn4xvwupbrjdy+pay/rph598qd9cj</latexit> <latexit sha_base64="exy2rvibegrsormj8+msfvxw2g=">aaacknicdzhpbtqwemad8k8sbraocmciqiqhrpje4fibkaavvcs2rbterhpvjlhwtoi9qvqlesuehbtvg7nzjgixksx/+ohm/4mrbvyfew/gvdgzvu37+zchd3bvf/g4fjr7qmrgitxkitd2fmuhgpv4pquatyvlyjjnz6ly7d9/uw7wqeq8iutapwbyeuvkqnkutl+czwy8ubl4pkltniltzl+aguvsnbdla9eadrusxwjuwixb3e3viscv3grwzbt3lwfoqexi4//eag6lmda+//ptiuohazkc3o5xojynerge9ekwge/lukn2goboengp8wiko3bkqqg52zxvno8butkauxgonfygxcjjmvszdo5u3a3y6/8gtbs8r6uxjf078rwjcuh9u/necfu5rr4bbcrkhs9bxvzd0qlnjoldwau8x7vfgfsihjr7waazwflcscvkhkf9qbef/98nvxejijo0n8jwi77cl7zvzzzf6xi/aenbapk8gt4cj4ehwmn4xvwupbrjdy+pay/rph598qd9cj</latexit> <latexit sha_base64="fwqdngx8hsasjo47ghwur+5xsmo=">aaacnxicdvfnj9mwehxck+uncdaxyv0nlyktklhcsqeqaflrldrlr3o4nrjfbtjgtpkcpv/hw/hbv/bqcpeuywkax5em9gm36tkpajkjfqxjj5q3bd/budu7df/dw0xd/8amtgspflfeqmmcpwkfkkayouymz2gjqqrkzdpwu02ffhbgykr/huhyldxkpm8kbpzumfxwnmr2vobuklk3u0q6t45hirda89uhkwdddqln9llbhczt38ycufzzoc7uhwfw6zehp7+wzbqdqe9ffcfstvyfokreygv+squb4nkoirg0sbodrbvwyhs4yqz/mtlijdalmgvwdupoxoxdgxkrkq7yi0vnfav5gluyqla2ixbunvsl55z0qwy/pvin+zfhq607db2lrqwsfejtylzrvm3iyclosgrcn7qvmjkfa0oxvdsim4qruhwi30ujegdcu/ue7e+krx74oto/gctsov0ajydhwjj3yjlwgbyqmr8mefcanzep48dsybb+dt+hz8h4hh7ps8ng2/oe/bph7ddaedgy</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> <latexit sha_base64="yzvfko7wyha/yervzh/rngyfka=">aaacnxicdvfdi9nafj3er7v+bnu3fxcwcovdlmqr9leogsoqk9jtqqcbjtobzohmemduhdkbf+uv8c/46spolvwnap59zlvxnuwitpmyp+beg6zdu3tq7pbhz9979/egdh6e2aoyaqahuzc5sbkhjeqyoucfzbydrvmesxb3t9nl3mfzw5vdc7dqpc9ljgvhtyxdh8ejzm9kzi4osyutgvtk2nyapl2fhxico5uh9qm+tytduo272y5fkmsmy4uhwfwqygq0//yrhxdcf7+ua/k3aw/qqj8wjf9ilddabjcbsno03qqydeghhzxkky/mmwlwg0lcgut3yerzuuhdcohyj2wbolnrcrnspcw5jrsau3cbelzz2zpfll/curbti/oxzxtlvbv2qohb2sdequbd5g9nrhzfk3ckxob2wnoljr7lr0kq0ivgspudds70pfwb2h6a/amrbf/vjvcho0jqnx/oxlahk0twoppchpyagjyssyie/jczksetwojsgh4gp4nhwxhoef+9iw2py8iv9eopsnqvhrna==</latexit> LAG: GD under two alternative communication rules q Worker-side rule (LAG-WK): Include worker m in if Old gradient rl m rl m ˆ k m M Gradient innovation Optimization progress q Server-side rule (LAG-PS): Include worker m in if : smoothness of L m L m ˆ k m M LAG-PS is a sufficient condition for LAG-WK. T. Chen, G. B. Giannakis, T. Sun, and W. Yin "LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning," Proc. of NIPS, Montreal, Canada, December 3-8,

9 Iteration and communication complexity (nonconvex) Local loss is smooth. (convex) Loss is convex. (strongly convex) Loss is (restricted) strongly convex. Theorem In all cases, LAG enjoys the same convergence rate as GD. Theorem 2 If local objectives are heterogeneous, LAG requires smaller number of uploads to a given accuracy than GD; e.g., as small as /M. 9

10 Linear prediction q Real datasets distributed on M = 9 workers Cyc-/Num-IAG: cyclic/non-uniform update of incremental aggregated gradient M. Lichman, UCI machine learning repository, 203. [Online]: 0

11 Logistic regression q Real datasets distributed on M = 9 workers Ø LAG needs same number of iterations but fewer uploads

12 Conclusions q Adaptive communication rules for distributed learning q Not degrade convergence but reduce communication Thank You! Thu Dec 6th 05: :00 Room 20 & 230 AB #8 2