Coordination-Free Computations. Christopher Meiklejohn

Coordination-Free Computations Christopher Meiklejohn

LASP DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS CHRISTOPHER MEIKLEJOHN (BASHO TECHNOLOGIES, INC.) PETER VAN ROY (UNIVERSITÉ CATHOLIQUE DE LOUVAIN) 2

LASP MOTIVATION 3

SYNCHRONIZATION IS EXPENSIVE / IMPRACTICAL 4

MOBILE GAMES: SHARED STATE BETWEEN CLIENTS CLIENTS GO OFFLINE http://www.rovio.com/en/news/blog/261/263-million-monthly-active-users-in-december/ 5

INTERNET OF THINGS : DISJOINT STATE AGGREGATED UPSTREAM CLIENTS GO OFFLINE Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, architectural elements, and future directions." Future Generation Computer Systems 29.7 (2013): 1645-1660. 6

NO TOTAL ORDER: REPLICATED SHARED STATE WITH OFFLINE CLIENTS CLIENTS NEED TO MAKE PROGRESS Gilbert, Seth, and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." ACM SIGACT News 33.2 (2002): 51-59. 7

WALL CLOCKS: UNRELIABLE AT BEST NON-DETERMINISTIC IF USED IN COMPUTATIONS Corbett, James C., et al. "Spanner: Google s globally distributed database." ACM Transactions on Computer Systems (TOCS) 31.3 (2013): 8. 8

CONCURRENCY RECONCILED BY USER 9

set(1) set(2) R A 1 2? set(3) R B 3? 10

set(1) set(2) R A 1 2? set(3) R B 3? 11

set(1) set(2) R A 1 2? set(3) R B 3? 12

set(1) set(2) R A 1 2? set(3) R B 3? 13

CRDTs 14

CRDTs PROVIDE DETERMINISTIC RESOLUTION 15

CRDTs: MAPS, SETS, COUNTERS, REGISTERS, GRAPHS DETERMINISTIC RESOLUTION 16

CRDTs REALIZE MONOTONIC STRONG EVENTUAL CONSISTENCY CORRECT REPLICAS THAT HAVE DELIVERED THE SAME UPDATES HAVE EQUIVALENT STATE Shapiro, Marc, et al. "Conflict-free replicated data types." Stabilization, Safety, and Security of Distributed Systems. Springer Berlin Heidelberg, 2011. 386-400. 17

CRDTs EXAMPLE MAX REGISTER 18

set(1) set(2) max(2,3) R A 1 2 3 max(2,3) R B 3 3 set(3) 19

CRDTs EXAMPLE ORSET SET 20

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) R B {1} (1, {a, b}, {b}) add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) 21

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) R B {1} (1, {a, b}, {b}) add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) 22

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) R B {1} (1, {a, b}, {b}) add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) 23

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) R B {1} (1, {a, b}, {b}) add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) 24

FRAGMENTS & PROGRAMS 25

{1,2,3} {3,4,5} 26

{1,2,3} Intersection {3} {3,4,5} 27

{1,2,3} Intersection {3} Map {9} {3,4,5} 28

{1,2,3} Intersection {3} Map {9} {3,4,5} 29

{1,2,3} Intersection {3} Map {9} {3,4,5} 30

{1,2,3} Intersection {3} Map {9} {3,4,5} 31

{1,2,3} Intersection {3} Map {9} {3,4,5} {1,2,3} Intersection {3} Map {9} {3,4,5} 32

{1,2,3} Intersection {3} Map {9} {3,4,5} {1,2,3} Intersection {3} Map {9} {3,4,5} 33

{1,2,3} Intersection {3} Map {9} {3,4,5} {1,2,3} Intersection {3} Map {9} {3,4,5} 34

FUNCTION APPLICATION AND DATA COMPOSITION IS NONTRIVIAL 35

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) R B {1} (1, {a, b}, {b}) add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) fun(x) -> 2 end F(R C ) {2} {} {2} 36

add(1) R A {1} {1} (1, {a}, {}) (1, {a, b}, {b}) fun(x) -> 2 end F(R A ) {2} {2} {2} {2} add(1) remove(1) R C {1} {} {1} (1, {b}, {}) (1, {b}, {b}) (1, {a, b}, {b}) fun(x) -> 2 end F(R C ) {2} {} {2} 37

COMPOSITION: USER OBSERVABLE VALUE VS. STATE METADATA MAPPING IS NONTRIVIAL WITHOUT MAPPING METADATA; UNMERGABLE Brown, Russell, et al. "Riak dt map: A composable, convergent replicated dictionary." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014. Conway, Neil, et al. "Logic and lattices for distributed programming." Proceedings of the Third ACM Symposium on Cloud Computing. ACM, 2012. Meiklejohn, Christopher. "On the composability of the Riak DT map: expanding from embedded to multi-key structures." Proceedings of the First Workshop on Principles and Practice of Eventual Consistency. ACM, 2014. 41

LASP LANGUAGE 42

LASP: DISTRIBUTED RUNTIME IMPLEMENTED AS ERLANG LIBRARY USES RIAK-CORE 43

LASP: CRDTS AS STREAMS OF STATE CHANGES CRDTS CONNECTED BY MONOTONIC PROCESSES MANY TO ONE MAPPING OF CRDTS 44

PRIMITIVE OPERATIONS: MONOTONIC READ, UPDATE FUNCTIONAL: MAP, FILTER, FOLD SET-THEORETIC: PRODUCT, UNION, INTERSECTION LIFTED TO OPERATE OVER METADATA 45

EXAMPLE MAP 46

add(1) R A {1} (1, {a}, {}) {1} (1, {a, b}, {b}) fun(x) -> 2 end F(R A ) {2} {2} {2} {2} (2, {a}, {}) (2, {a, b}, {}) (2, {a, b}, {b}) (2, {a, b}, {b}) R B {1} (1, {a, b}, {b}) fun(x) -> 2 end F(R B ) {2} {} {} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) (2, {a, b}, {b}) add(1) remove(1) R C {1} (1, {b}, {}) {} (1, {b}, {b}) {1} (1, {a, b}, {b}) fun(x) -> 2 end F(R C ) {2} {} {2} {2} (2, {b}, {}) (2, {b}, {b}) (2, {a, b}, {b}) (2, {a, b}, {b}) 47

ARCHITECTURE 51

LASP STORE SHARED VARIABLE STORE 52

LASP BACKENDS LEVELDB, BITCASK, ETS 53

LASP CRDTs PROVIDED BY RIAK DT 54

LASP ARCHITECTURE CENTRALIZED SEMANTICS 55

STORE: SHARED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 56

LASP ARCHITECTURE DISTRIBUTED SEMANTICS 57

STORE: REPLICATED, SHARDED VARIABLE STORE PROCESSESS SYNCHRONIZE ON VARIABLES 58

REPLICATED: DISTRIBUTED WITH RIAK CORE QUORUM REQUESTS; ANTI-ENTROPY PROTOCOL 59

HYBRID: DISTRIBUTE PROGRAMS; R/W WITH LOCAL STORE CENTRALIZED EXECUTION 60

EXAMPLES 61

%% Create initial set. {ok, S1} = lasp:declare(type), %% Add elements to initial set and update. {ok, _} = lasp:update(s1, {add_all, [1,2,3]}, a), %% Create second set. {ok, S2} = lasp:declare(type), %% Apply map. ok = lasp:map(s1, fun(x) -> X * 2 end, S2),

%% Create initial set. {ok, S1} = lasp_core:declare(type, Store), %% Add elements to initial set and update. {ok, _} = lasp_core:update(s1, {add_all, [1,2,3]}, a, Store), %% Create second set. {ok, S2} = lasp_core:declare(type, Store), %% Apply map. ok = lasp_core:map(s1, fun(x) -> X * 2 end, S2, Store),

AD COUNTER 64

AD COUNTER: TRACKS AD IMPRESSIONS PUSHES ADVERTISEMENTS TO THE CLIENT DISABLES AD AT 50,000+ IMPRESSIONS CLIENTS DISPLAY ADS WHEN OFFLINE 65

AD COUNTER INFORMATION FLOW 66

Increment Rovio Ad Counter Read 50,000 Remove Rovio Ads Read Counter 1 Counter 2 Counter 3 Rovio Ad Counter Union Ads Product Ads Contracts Filter Ads With Contracts Counter 1 Counter 2 Counter 3 Riot Ad Counter Counter 1 Counter 2 Counter 3 Riot Ads Contracts Lasp Operation Riot Ad Counter User-Maintained CRDT Lasp-Maintained CRDT Counter 1 Counter 2 Counter 3 67

Rovio Ad Counter Union Ads Riot Ad Counter Riot Ads Contracts Riot Ad Counter 68

Rovio Ad Counter Increment Read 50,000 Remove Rovio Ads Rovio Ad Counter Union Ads Product Ads Contracts Filte Riot Ad Counter Riot Ads Contracts Lasp Operation Riot Ad Counter User-Maintained CRDT Lasp-Maintained CRDT 69

Union Ads Product Ads Contracts Filter Ads With Contracts Contracts Lasp Operation User-Maintained CRDT 70

Read Product Ads Contracts Filter Ads With Contracts cts Lasp Operation 71

Read Counter 1 Counter 2 Counter 3 s acts Filter Ads With Contracts Counter 1 Counter 2 Counter 3 Counter 1 Counter 2 Counter 3 tion ained CRDT Counter 1 Counter 2 Counter 3 ined CRDT 72

Rovio Ad Counter Increment Read 50,000 Remove Rovio Ads Rovio Ad Counter Union Ads Product Ads Contrac Riot Ad Counter Riot Ads Contracts Lasp Operat 74

0 Remove Rovio Ads Read Counter 1 Counter 2 Counter 3 Union Ads Product Ads Contracts Filter Ads With Contracts Counter 1 Counter 2 Counter 3 Counter 1 Counter 2 Counter 3 Riot Ads Contracts Lasp Operation User-Maintained CRDT Lasp-Maintained CRDT Counter 1 Counter 2 Counter 3 75

INFORMATION FLOW: MONOTONIC METADATA TO PREVENT DUPLICATE PROPAGATION 76

AD COUNTER EXAMPLE CODE 77

%% @doc Client process; standard recursive looping server. client(id, AdsWithContracts, PreviousValue) -> receive view_ad -> %% Get current ad list. {ok, {_, _, AdList0}} = lasp:read(adswithcontracts, PreviousValue), AdList = riak_dt_orset:value(adlist0), case length(adlist) of 0 -> %% No advertisements left to display; ignore %% message. client(id, AdsWithContracts, AdList0); _ -> %% Select a random advertisement from the list of %% active advertisements. {#ad{counter=ad}, _} = lists:nth( random:uniform(length(adlist)), AdList), %% Increment it. {ok, _} = lasp:update(ad, increment, Id), lager:info("incremented ad counter: ~p", [Ad]), end. end client(id, AdsWithContracts, AdList0)

%% @doc Server functions for the advertisement counter. After 5 views, %% disable the advertisement. %% server({#ad{counter=counter}=ad, _}, Ads) -> %% Blocking threshold read for 5 advertisement impressions. {ok, _} = lasp:read(counter, 5), %% Remove the advertisement. {ok, _} = lasp:update(ads, {remove, Ad}, Ad), lager:info("removing ad: ~p", [Ad]).

%% Generate a series of unique identifiers. RovioAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("rovio Ad Identifiers are: ~p", [RovioAdIds]), TriforkAdIds = lists:map(fun(_) -> druuid:v4() end, lists:seq(1, 10)), lager:info("trifork Ad Identifiers are: ~p", [TriforkAdIds]), Ids = RovioAdIds ++ TriforkAdIds, lager:info("ad Identifiers are: ~p", [Ids]), %% Generate Rovio's advertisements. {ok, RovioAds} = lasp:declare(?set), lists:map(fun(id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?counter), %% Add it to the advertisement set. {ok, _} = lasp:update(rovioads, {add, #ad{id=id, counter=counterid}}, undefined) end, RovioAdIds), %% Generate Trifork's advertisements. {ok, TriforkAds} = lasp:declare(?set), lists:map(fun(id) -> %% Generate a G-Counter. {ok, CounterId} = lasp:declare(?counter), %% Add it to the advertisement set. {ok, _} = lasp:update(triforkads, {add, #ad{id=id, counter=counterid}}, undefined) end, TriforkAdIds), %% Union ads. {ok, Ads} = lasp:declare(?set), ok = lasp:union(rovioads, TriforkAds, Ads), %% For each identifier, generate a contract. {ok, Contracts} = lasp:declare(?set), lists:map(fun(id) -> {ok, _} = lasp:update(contracts, {add, #contract{id=id}}, undefined) end, Ids), %% Compute the Cartesian product of both ads and contracts. {ok, AdsContracts} = lasp:declare(?set), ok = lasp:product(ads, Contracts, AdsContracts), %% Filter items by join on item it. {ok, AdsWithContracts} = lasp:declare(?set), FilterFun = fun({#ad{id=id1}, #contract{id=id2}}) -> Id1 =:= Id2 end, ok = lasp:filter(adscontracts, FilterFun, AdsWithContracts), %% Launch a series of client processes, each of which is responsible %% for displaying a particular advertisement. %% Generate a OR-set for tracking clients. {ok, Clients} = lasp:declare(?set), %% Each client takes the full list of ads when it starts, and reads %% from the variable store. lists:map(fun(id) -> ClientPid = spawn_link(?module, client, [Id, AdsWithContracts, undefined]), {ok, _} = lasp:update(clients, {add, ClientPid}, undefined) end, lists:seq(1,5)), %% Launch a server process for each advertisement, which will block %% until the advertisement should be disabled. %% Create a OR-set for the server list. {ok, Servers} = lasp:declare(?set), %% Get the current advertisement list. {ok, {_, _, AdList0}} = lasp:read(adswithcontracts), AdList = riak_dt_orset:value(adlist0), %% For each advertisement, launch one server for tracking it's %% impressions and wait to disable. lists:map(fun(ad) -> ServerPid = spawn_link(?module, server, [Ad, Ads]), {ok, _} = lasp:update(servers, {add, ServerPid}, undefined) end, AdList),

AD COUNTER DISTRIBUTION 85

DISTRIBUTION BOUNDARIES: ARBITRARY ALLOWS COMPOSITION OF ENTIRE SYSTEM 90

MATERIALIZED VIEWS 91

MATERIALIZED VIEWS: INCREMENTALLY UPDATING, DELTA PROPAGATION SCHEMA BASED OR DYNAMIC; 2I IN RIAK 92

DATABASE AS STREAM OF UPDATES 93

K 1 K 2 K 3 K 1 K 2 Riak 94

RS 1 K 1 K 2 K 3 K 1 K 2 Riak RS 2 RS 3 95

K 1 K 1 RS 1 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 96

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 97

COMPOSE PROGRAMS 98

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 Lasp Stream Processor Template 99

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 Lasp Stream Processor Template Lasp All Objects Program 100

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 Lasp Stream Processor Template Lasp All Objects Program Lasp Over 65 Filter 101

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 Lasp Stream Processor Template Lasp All Objects Program Lasp Over 65 Filter Lasp Over 80 Filter 102

K 1 K 1 N 1 N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 2 K 2 K 3 RS 3 Lasp Stream Processor Template Lasp All Objects Program Lasp Over 65 Filter Lasp Over 80 Filter Lasp Named Chris Filter (L) 103

DISTRIBUTE APPLICATIONS 104

K 1 K 1 N 1 L N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 3 L RS 3 L K 2 K 2 Lasp Named Chris Filter (L) 105

N 2 L MERGE L N 3 L RESULTS N 1 (SEC) 106

L L RS 1 RS 2 SUM RESULTS 107

K 1 K 1 N 1 L N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 3 L RS 3 L K 2 K 2 Execution of Lasp Named Chris Filter (L) 108

CACHE: CACHE RESULTS OF MERGE SPECULATIVELY EXECUTE BASED ON DIVERGENCE 109

INTERNET OF THINGS 110

K 1 K 1 N 1 L N 2 N 3 K 1 K 2 K 3 K 1 K 2 Riak RS 2 K 3 L RS 3 L K 2 K 2 Execution of Lasp Named Chris Filter (L) 111

U 1 U 1 L S 1 K 1 K 2 K 3 K 1 K 2 IoT Network S 2 U 2 U 2 L U 3 S 3 L Execution of Lasp Temperature Sensor > 90 F 112

U 1 U 1 L S 1 IoT Network S 2 L U 2 U 2 U 3 S 3 L Execution of Lasp Temperature Sensor > 90 F 113

IOT: EXECUTE AT THE EDGE WRITE CODE THINKING ABOUT ALL DATA 114

RELATED WORK 115

RELATED WORK DISTRIBUTED OZ 116

RELATED WORK DERFLOW L 117

RELATED WORK BLOOM L 118

RELATED WORK LVARS 119

RELATED WORK D-STREAMS 120

RELATED WORK SUMMINGBIRD 121

FUTURE WORK 122

FUTURE WORK INVARIANT PRESERVATION 123

FUTURE WORK CAUSAL+ CONSISTENCY 124

FUTURE WORK ORSWOT OPTIMIZATION 125

FUTURE WORK DELTA STATE-CRDTs 126

FUTURE WORK OPERATION-BASED CRDTs 127

FUTURE WORK DEFORESTATION 128

SOURCE GITHUB.COM/CMEIKLEJOHN/LASP 129

ERLANG WORKSHOP 2014 DERFLOW DISTRIBUTED DETERMINISTIC DATAFLOW PROGRAMMING FOR ERLANG 130

PAPOC / EUROSYS 2015 LASP A LANGUAGE FOR DISTRIBUTED, EVENTUALLY CONSISTENT COMPUTATIONS WITH CRDTs 131

RICON 2015 NOVEMBER 4-6, 2015 SAN FRANCISCO, CA 132

SYNCFREE SYNCFREE IS A EUROPEAN RESEARCH PROJECT TAKING PLACE FOR 3 YEARS, STARING OCTOBER 2013, AND IS FUNDED BY THE EUROPEAN UNION, GRANT AGREEMENT N 609551. 133

Questions? Please remember to evaluate via the GOTO Guide App