Graph: composable production systems in Clojure. Jason Wolfe Strange Loop 12

Transcription

1 Grah: comosable roduction systems in Clojure Jason Wolfe Strange Loo 12

2 Motivation Interesting software has: many comonents comlex web of deendencies Develoers want: simle, factored code easy testability tools for monitoring and debugging

3 Grah Grah is a simle, declarative way to exress system comosition A Grah is just a ma of functions that can deend on revious oututs Grahs are easy to create, reason about, test, and build uon i x {:x (fnk [i]...) :y (fnk [j x]...) :z (fnk [x y]...)} z j y inut {:i 1 :j 2} outut {:x 2 :y 5 :z 12}

4 Outline Prismatic Design Goals Grah: secs and comilation {:x (fnk [i]...) :y (fnk [j x]...) :z (fnk [x y]...)} Alications newsfeed generation roduction services resonse resonse

5 Prismatic Personalized, interest-based newsfeeds Build crawlers, toic models, grah analysis, story clustering,... Backend 99.9% Clojure Personalized ranked feeds in real-time (~200ms) getrismatic.com

6 Prismatic s roduction API service >100 comonents storage systems caches & indices ranking algorithms ec2-keys index snashots doc index Coordinate in intricate dance to serve feeds fast SQL feed-builder to news handlers server Relentlessly refactored env log store observer Still dozens of to-level comonents in comlex deendency network service-name ubsub logger service-info Parameters Remote Storage Caches, Indices Fns, Other Thread Pools udate index

7 The feed builder user query 20+ stes from query to ersonalized ranking, 20+ arameters Not a simle ieline resonse

8 The feed builder user query 20+ stes from query to ersonalized ranking, 20+ arameters Not a simle ieline > 10 feed tyes w/ slightly different stes, configurations resonse resonse

9 The feed builder user query 20+ stes from query to ersonalized ranking, 20+ arameters Not a simle ieline > 10 feed tyes w/ slightly different stes, configurations Suort for early stoing resonse resonse

10 Theme: comlexity of comosition Previous imlementations: defns with huge lets Unwieldy for large systems with comlex or olymorhic deendencies Hard to test, debug, and monitor resonse resonse

11 The monster let Tens of arameters, not comositional Mocks/olymorhic flow difficult Ad hoc monitoring & shutdown logic er item Core issue: structure of (de)comosition is locked u in an oaque function (defn start [{:keys [a,z]}] (let [s1 (store a...) s2 (store b...) db (sql-db c) t2 (cron s2 db...)... srv (server...)] (fn shutdown [] (.sto srv)... (.flush s1))))

12 Prismatic software engineering hilosohy Fine-grained, comosable abstractions (FCA) Libraries >> Frameworks Strive for simlicity, work with the language Grah is a FCA for comosition

13 Goal: declarative Declarative secifications fix monster let Exlicitly list comonents, deendencies Enable abstractions over comonents, reasoning about comosition Not new: Pregel, Dryad, Storm,...

14 Goal: simle Distill this idea to its simlest, most idiomatic exression a Grah sec is just a (Clojure) ma no XML files or interface hell Grahs are ordinary data maniulate them for free --> unexected alications It is better to have 100 functions oerate on one data structure than 10 functions on 10 data structures. - Alan Perlis

15 From let to Grah m xs n m2 (defn stats [{:keys [xs]}] (let [n (count xs) m (/ (sum xs) n) m2 (/ (sum sq xs) n) v (- m2 (* m m))] {:n n :m m :m2 m2 :v v})) v {:n (fn k [xs] (count xs)) :m (fn k [xs n] (/ (sum xs) n)) :m2 (fn k [xs n] (/ (sum sq xs) n)) :v (fn k [m m2] (- m2 (* m m)))}

16 Bring on the fnk fnk = keyword function Similar to {:keys []} destructuring nicer ot. arg. suort asserts that keys exist metadata about args Quite useful in itself Only macros in Grah (defnk foo [x y [s 1]] (+ x (* y s))) (= 8 (foo {:x 2 :y 3 :s 2})) (= 5 (foo {:x 2 :y 3})) (thrown? Ex. (foo {:x 2})) (= (meta foo) {:req-ks #{:x :y}} :ot-ks #{:s})

17 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:n (fnk [xs] (count xs)) :m (fnk [xs n] (/ (sum xs) n)) :m2 (fnk [xs n] (/ (sum sq xs) n)) :v (fnk [m m2] (- m2 (* m m)))} m m2 v

18 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:xs [ ]} {:n (fnk [xs] (count xs)) :m (fnk [xs n] (/ (sum xs) n)) :m2 (fnk [xs n] (/ (sum sq xs) n)) :v (fnk [m m2] (- m2 (* m m)))} m m2 v

19 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:xs [ ]} {:n 4(fnk [xs] (count xs)) :m (fnk [xs n] (/ (sum xs) n)) :m2 (fnk [xs n] (/ (sum sq xs) n)) :v (fnk [m m2] (- m2 (* m m)))} m m2 v

20 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:xs [ ]} {:n 4(fnk [xs] (count xs)) :m 3(fnk [xs n] (/ (sum xs) n)) :m2 (fnk [xs n] (/ (sum sq xs) n)) :v (fnk [m m2] (- m2 (* m m)))} m m2 v

21 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:xs [ ]} {:n 4(fnk [xs] (count xs)) :m 3(fnk [xs n] (/ (sum xs) n)) :m (fnk [xs n] (/ (sum sq xs) n)) :v (fnk [m m2] (- m2 (* m m)))} m m2 v

22 A Grah Secification A Grah is just a ma from keywords to fnks Required keys of each fnk secify grah relationshis Entire grah secifies a fnk to ma of results xs n {:xs [ ]} {:n 4(fnk [xs] (count xs)) :m 3(fnk [xs n] (/ (sum xs) n)) :m (fnk [xs n] (/ (sum sq xs) n)) :v 3.5 (fnk [m m2] (- m2 (* m m)))} m m2 v

23 Comiling Grahs Comile grah to fnk that returns ma of oututs (def g {:n (fnk [xs]...) :m (fnk [xs n]...) :m2 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile g)) (= (stats {:xs [ ]}) {:n 4 :m 3 :m :v 3.5)

24 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked (def g {:n (fnk [xs]...) :m (fnk [xs n]...) :m2 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile g)) (= (thrown? (stats {:xs [ ]}) {:n (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

25 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma (def g {:n (fnk [xs]...) :m (fnk [xs n]...) :m2 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile (lazy-comile g)) g)) (thrown? (= (stats (:m (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

26 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma can auto-arallelize (def g {:n (fnk [xs]...) :m (fnk [xs n]...) :m2 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile (lazy-comile (ar-comile g)) g)) (thrown? (= (stats (:m (:v (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) 3.5) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

27 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma can auto-arallelize (def g {:n 2(fnk [xs]...) :m (fnk [xs n]...) :m2 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile (lazy-comile (ar-comile g)) g)) (thrown? (= (stats (:m (:v (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) 3.5) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

28 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma can auto-arallelize (def g {:n 2(fnk [xs]...) :m 3(fnk [xs n]...) :m2 13 (fnk [xs n]...) :v (fnk [m m2]...)}) (def stats (comile (lazy-comile (ar-comile g)) g)) (thrown? (= (stats (:m (:v (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) 3.5) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

29 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma can auto-arallelize (def g {:n 2(fnk [xs]...) :m 3(fnk [xs n]...) :m2 13 (fnk [xs n]...) :v 4(fnk [m m2]...)}) (def stats (comile (lazy-comile (ar-comile g)) g)) (thrown? (= (stats (:m (:v (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) 3.5) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

30 Comiling Grahs Comile grah to fnk that returns ma of oututs error checked can return lazy ma can auto-arallelize With more tooling, also comile grahs to roduction services Could comile to crossmachine toologies,... (def g {:n 2(fnk [xs]...) :m 3(fnk [xs n]...) :m2 13 (fnk [xs n]...) :v 4(fnk [m m2]...)}) (def stats (comile (lazy-comile (ar-comile g)) g)) (thrown? (= (stats (:m (:v (stats {:xs {:xs [1 2 [13 5]})) 6]}) {:n 3) 3.5) (Ex. 4 missing :m 3 :xs ) (stats :m {:x:v 1})) 3.5)

31 Before: feed builder Real-time ersonally ranked feeds 100-line fn exressed core comosition logic, ~20 arams several nested lets, escae hatches Comonent olymorhism (10 flavors of feeds) kludge of cases ball of multimethods rotocols + hacks resonse resonse

32 Feed builder in Grah Default arameters Grah with holes catures shared logic x y (def default-arams {:alha :hasers :stun}) resonse (def artial-grah {:query (fnk...)... :y (fnk [a x]..)... :res (fnk...)})

33 Feed builder in Grah Each feed tye secifies udated arameters missing/new grah nodes To make feed fn, just merge in udates comile resulting grah (def default-arams..) (def artial-grah..) (def toic-feed (comile-feed-fn {:alha 0.2} {:x (fnk...) :q (fnk...)})) (defn comile-feed-fn [arams nodes] (let [ (merge default-arams arams) g (comile (merge artial-grah nodes))] (fn feed [req] (g (merge req)))))

34 After: feed builder Simler, cleaner code Polymorhism is trivial (def toic-feed (comile-feed-fn {:alha 0.2} {:x (fnk...) :q (fnk...)})) (def home-feed (comile-feed-fn {:alha 0.4} {:x (fnk...) :r (fnk...) :s (fnk...)}))

35 After: feed builder Simler, cleaner code Polymorhism is trivial Early stoing for free via lazy comilation tt resonse

36 After: feed builder Simler, cleaner code Polymorhism is trivial Early stoing for free via lazy comilation tt (let [h (home-feed req)] (:tt h)) resonse

37 After: feed builder Simler, cleaner code Polymorhism is trivial Early stoing for free via lazy comilation v tt (let [h (home-feed req)] [(:tt h) (:v h)]) resonse

38 Also: easy to analyze Detect mis-wirings at grah comile time ositional constructor N N N N N N N N N N Avoid wrong # of args errors, arg ordering bugs N N N N N N N N Visualize grahs in 5 loc N N (defn edges [grah] (for [[k f] grah :let [{:keys [req-ks ot-ks]} (meta f)] arent (concat req-ks ot-ks)] [arent k]))

39 Also: easy to monitor Add monitoring and error reorting by maing over fnks Since a Grah is a Ma, can just use ma-vals node n avg ms errors :fetch :rank :client (defn observe-grah [g] (into {} (for [[k f] g] [k (with-meta (fn [m] (let [v (f m)] (rint k m v) v)) (meta f))])))

40 Examle 2: roduction API service (def ai-service (service {:service-name ai :backend-ort :server-threads 100} {:store1 (instance store {:tye :s3...}) :memo (fnk [store1] {:resource...})... :ai-server (...)}))

41 Service definitions Service definition = resource grah arameter ma + Crane reads arams for rovisioning, deloyment Grah = service code arameters are args cron jobs, handlers at leaves (def ai-service (service {:service-name ai :backend-ort :server-threads 100} {:store1 (instance store {:tye :s3...}) :memo (fnk [store1] {:resource...})... :ai-server (...)}))

42 Service definitions Service definition = resource grah arameter ma + Crane reads arams for rovisioning, deloyment Grah = service code arameters are args cron jobs, handlers at leaves (def ai-service (service {:service-name ai ec2-keys doc index index snashots :backend-ort :server-threads feed-builder 100} handlers server SQL {:store1 (instance store {:tye :s3...}) :memo observer log store (fnk [store1] service-name {:resource ubsub...}) logger service-info... Parameters Remote Storage Caches, Indices Fns, Other Thread Pools :ai-server (...)})) env udate index to news

43 Service built-ins arameters Parameters and grah nodes available by convention Interface with deloyment, other services, dashboard Smartly reconfigure with env -- test/staging/rod {:env :rod :instance-id i-123abc :ec2-keys... } resources {:nameserver... :observer... :ubsub... }

44 Nodes build Resources Resource = comonent e.g., database, cache, fn Plus metadata for shutdown, handlers,... Reresent as a ma Library of resources that work with builtins data stores rocessing queues recurring tasks... (defnk refreshing-atom [f eriod] (let [a (atom (f)) e (Exec/newExec)] (.schedatfixedrate e #(reset! a (f)) eriod) {:res a :shutdown #(.sd e)}))

45 Starting and Stoing Transform resource grah to ordinary grah ma over leaves, ull out :resource assoc new :shutdown key Run grah to start service, get clean shutdown hook (defn start-service [sec] ((->> (:grah sec) resource-transform comile) (:arameters sec))) (def ai (start-service ai-service)) ((:shutdown ai))

46 Sub-Comonents ec2-keys index snashots doc index feed-builder to news SQL handlers server env log store observer udate index service-name service-info logger ubsub Parameters Remote Storage Caches, Indices Fns, Other Thread Pools

47 Sub-Comonents ec2-keys index snashots doc index feed-builder to news SQL handlers server env log store observer udate index service-name service-info logger ubsub Parameters Remote Storage Caches, Indices Fns, Other Thread Pools

48 Sub-Comonents ec2-keys index snashots doc index feed-builder to news SQL handlers server env log store observer udate index service-name service-info logger ubsub Parameters Remote Storage Caches, Indices Fns, Other Thread Pools

49 Sub-Comonents Nodes can themselves be Grahs just nested mas Package comonents as sub-grahs Sub-grahs are transarent debugging monitoring imerfect abstractions (def write-back-cache {:store (instance store...) :write-queue (instance queue...) :eriodic-rune (instance task...)})

50 Easy system testing Old xxx-line lets were imossible to test With grah, just merge in mock node fnks no elaborate mocks objects or redefs automatic, safe shutdown (deftest home-feed-systest (test-service (assoc ai-service :doc-index (fnk [] {:res fake-idx}) :get-user (fnk [] {:res (constantly me)})) (is (= (titles (slur url)) [ doc1 doc2 ]))))

51 Summary Grah = way exress comlex comositions declaratively simly Widely alicable Simler code, better tooling Hoe to oen source soon (we re hiring!) resonse resonse