Using Haskell at Bdellium

Transcription

1 Using Haskell at Bdellium Parallelism, Lenses, Pipes and Industry Report

2 A note about me (Fredrik Olsen) Studied Engineering Physics at LTH Started working for Bdellium in 2011

3 We help business grow Developing tools that deliver new services Retirement Plan Rails Haskell Optimizer Fund Rating Security Clustering Data-Driven Decision Making

4 Why functional programming?

5 Why we like Haskell at Bdellium Expressive type system, allows descriptions of data Very good support for parallelism and concurrency Easier to get correct software, implementations look more like math Developer happiness and easy hiring Good community (IRC, Mailinglist, Stackoverflow, Reddit)

6 Parallelism and concurrency

7 Parallelism and concurrency Parallel and Concurrent Programming in Haskell by Simon Marlow Both parallelism and concurrency is obviously growing more important all the time. There are a lot of different ways to do both parallelism and concurrency in Haskell.

8 Parallelism and concurrency Multiple cores for performance Multiple threads for modularity of interaction Parallelism Concurrency Pure, Deterministic Effectful

9 Parallelism Eval Monad, rpar and rseq Strategies Par Monad Data Parallel Programming with Repa GPU Programming with Accelerate Concurrency Threads and MVars Software Transactional Memory Distributed Concurrency with Cloud Haskell (like Erlang)

10 Parallelism - rpar and rseq as building blocks do a <- rpar a b <- rpar b return (a,b ) do a <- rpar a b <- rseq b return (a,b ) Example 1 a do a <- rpar a b <- rseq b rseq a return (a,b ) do a <- rpar a b <- rpar b rseq a rseq b return (a,b ) return b

11 Parallelism - rpar and rseq as building blocks do a <- rpar a b <- rpar b return (a,b ) do a <- rpar a b <- rseq b return (a,b ) do a <- rpar a b <- rseq b rseq a return (a,b ) do a <- rpar a b <- rpar b rseq a rseq b return (a,b ) Example 2 b return a

12 Parallelism - rpar and rseq as building blocks do a <- rpar a b <- rpar b return (a,b ) do a <- rpar a b <- rseq b return (a,b ) do a <- rpar a b <- rseq b rseq a return (a,b ) do a <- rpar a b <- rpar b rseq a rseq b return (a,b ) Example 3 b a return

13 Parallelism - rpar and rseq as building blocks do a <- rpar a b <- rpar b return (a,b ) do a <- rpar a b <- rseq b return (a,b ) do a <- rpar a b <- rseq b rseq a return (a,b ) do a <- rpar a b <- rpar b rseq a rseq b return (a,b ) Example 4 b a return

14 Parallelism - Control.Parallel.Strategies Tuple Strategies seqpair, parpair, General Traversals seqtraverse, partraverse List strategies seqlist, parlist, parmap, parlistchunk, Possible (and not very hard) to build own strategies

15 Parallelism - Control.Parallel.Strategies Sequential version -- Create forecasts for every particular participant / design combination. assignforecasts :: Inputs -> [Participant] assignforecasts ins = map create (ins^.participants) Parallel version -- Create forecasts for every particular participant / design combination. assignforecasts :: Inputs -> [Participant] assignforecasts ins = map create (ins^.participants) `using` parlist rdeepseq

16 Parallelism - Par monad Using rpar, rseq and strategies might quickly lead to having to learn evaluation order, garbage collection and a wealth of other subjects in order to increase performance. Par monad aims to simplify the process and almost lets you write code that looks a bit like imperative code. Just like Strategies defines common tasks in the library, such as parmap etc fork :: Par () -> Par () data IVar a -- instance Eq new :: Par (IVar a) put :: NFData a => IVar a -> a -> Par () get :: IVar a -> Par a

17 Parallelism - Par monad runpar $ do v1 <- new v2 <- new fork $ put v1 (f x) fork $ put v2 (g x) get v1 get v2 return (v1 + v2) We don t have to think about when something is evaluated. Put is fully strict put :: NFData a => IVar a -> a -> Par () Computation can be seen as a data flow graph. Nodes without dependencies between them can be evaluated in parallel.

18 Parallelism - Repa and Accelerate Libraries for automatic parallelisation. Repa uses unboxed vectors, great for any computation that can be expressed as matrices. Accelerate very similar to Repa but is a DSL for GPU programming. Compiles part of code to CUDA for sending to GPU.

19 Concurrency - The building blocks forkio :: IO () -> IO ThreadId newemptymvar :: IO (MVar a) takemvar :: MVar a -> IO a putmvar :: MVar a -> a -> IO () MVars are single-value communication channels. A box than can be full or empty. Example stolen from Simon Marlow do m1 <- newemptymvar m2 <- newemptymvar forkio $ do r <- geturl " putmvar m1 r forkio $ do r <- geturl " putmvar m2 r r1 <- takemvar m1 r2 <- takemvar m2 return (r1,r2)

20 Concurrency - Control.Concurrent.Chan newchan :: IO (Chan a) writechan :: Chan a -> a -> IO () readchan :: Chan a -> IO a Abstraction on top of MVars to provide full communication channels. We re still in low-level message-passing. Can be hard to get right.

21 Concurrency - Control.Monad.STM data TVar a -- abstract newtvar :: STM (TVar a) readtvar :: TVar a -> STM a writetvar :: TVar a -> a -> STM () Provides atomic actions, transactions, that are composable Does not block, but does provide a way to simulate blocking, namely to retry transactions until some condition is met. (Does not busy-wait but waits until some change has been made to the var) We can get channels with STM as well.

22 Concurrency - Control.Distributed.Process (aka Cloud Haskell) Distributed concurrency, a.k.a. Erlang in Haskell Includes process management and communication methods. (With local or Azure backends, more coming) Automatic node discovery. Has successfully seen some real-world applications with big server farms.

23 Libraries are what makes a language viable

24 The two most useful libraries we use Lens by Edward Kmett Pipes by Gabriel Gonzalez hackage.haskell.org/package/lens hackage.haskell.org/package/pipes

25 A brief introduction to Lenses Courtesy of Cartesian Closed Comic

26 Why do we want lenses? A lens provides access into the middle of a data structure A lens is a first-class value, meaning you can pass it around to functions just as any other value. Lenses are composable, so we can take several lenses and make a lens out of them. Awesome images are stolen from Simon Peyton Jones talk about Lenses as Haskell exchange. Go watch his talk if you want a more thorough introduction.

27 Why do we want lenses? How do you get / set values in these data structures? data Person = Person { name :: String, addr :: Address, salary :: Int } data Address = Address { road :: String, city :: String, postcode :: String } getname :: Person -> String getname p = name p get :: (s -> a) -> s -> a get property structure = property structure getcity :: Person -> String getcity p = city (addr p) getdeep :: (s1 -> s2) -> (s2 -> a) -> s1 -> a getdeep prop1 prop2 structure = prop2 (prop1 structure) >>> getdeep addr city == get (addr.city)

28 Why do we want lenses? How do you get / set values in these data structures? data Person = Person { name :: String, addr :: Address, salary :: Int } data Address = Address { road :: String, city :: String, postcode :: String } setname :: String -> Person -> Person setname n p = p { name = n } setpostcode :: String -> Person -> Person setpostcode pc p = p { addr = addr p { postcode = pc }} -- UGLY set :: (s -> a) -> a -> s -> s set prop n p = p { prop = n } LensExample.hs:15:20: `prop' is not a (visible) constructor field name

29 With Control.Lens we can simplify setpostcode, setpostcode', setpostcode'' :: String -> Person -> Person setpostcode pc p = set (addr.postcode) pc p setpostcode' = set (addr.postcode) setpostcode'' pc p = p & addr.postcode.~ pc getpostcode :: Person -> String getpostcode p = p^.addr.postcode

30 How we use Lenses in practice We have a lot of nested data structures so we make heavy use of the accessor methods. p^.assets p^.forecast.retiresat p^.forecast.current.gapanalysis p^.forecast.optimized.projectedfunding.income.percent Lenses make traversals trivial -- sum of all unconstsalary records sumof (accs.traverse.unconstsalary) paccs -- maximum forecastsalary in the first 36 months maximumof (accs.taking 36 traverse.forecastsalary) paccs -- a list of the RRR for each participant under the current plan design ps^..folded.forecast.current.projectedfunding.income.percent

31 A brief introduction to Pipes One of many libraries for stream programming in Haskell. Dealing with long-running, complex or Lazy IO in Haskell can be very difficult. Pipes provides a clean and simple API to provide effectful, streaming, and composable programming.

32 Why do we want pipes? Lazy IO is especially hard to get right withfile "hello.txt" ReadMode (hgetcontents >=> print) >>> "Hello world" withfile "hello.txt" ReadMode hgetcontents >>= print >>> "" IO is difficult to decompose and re-use. Most attempts result in some sort of pipeline, whether explicit or not.

33 Example of decomposing echo program main = do eof <- iseof unless eof $ do str <- getline putstrln (transform str) main transform :: String -> String transform = reverse

34 Example of decomposing echo program main = do eof <- iseof unless eof $ do str <- getline putstrln (transform str) main transform :: String -> String transform = reverse main = readdata transform printer readdata :: (String -> String) -> (String -> IO ()) -> IO () readdata t p = do eof <- iseof unless eof $ do str <- getline p $ t str readdata t p printer :: String -> IO () printer s = putstrln s transform :: String -> String transform = reverse Still very tightly coupled and readdata handles a lot of logic not specific to reading. We can improve, but it s a lot of work

35 The Pipes way - the types document and constrain main = runeffect mainpipeline mainpipeline :: Effect IO () mainpipeline = readdata >-> transform >-> printer readdata :: Producer String IO () readdata = forever $ do eof <- lift iseof unless eof $ do str <- lift getline yield str transform :: Monad m => Pipe String String m () transform = forever $ do str <- await yield $ reverse str printer :: Consumer String IO () printer = forever $ do str <- await lift $ putstrln str

36 Why do we want pipes? Simplifies construction of modular data pipelines, whether your application streams or not. Solves the Lazy IO problem. Comes with batteries included; a lot of default producers, consumers and pipes out of the box. Producers - stdinln, fromhandle Consumers - stdoutln, tohandle Pipes - take n, takewhile, drop n, scanl Folds - all, null, last

37 How we use Pipes in practice -- Run the pipeline. runrsa :: RunInformation -> IO () runrsa runinfo = runeffect $ rsapipeline runinfo -- Assemble the RSA pipeline. rsapipeline :: RunInformation -> Effect IO () rsapipeline runinfo = fetchrawinputs runinfo >-> preprocessinputs >-> addforecasts >-> createandwritereport runinfo -- Grab raw inputs from the database. fetchrawinputs :: RunInformation -> Producer RawInputs IO () -- Preprocess raw inputs and yield them downstream. preprocessinputs :: Monad m => Pipe RawInputs Inputs m () -- Augment inputs with participant forecasts and yield them downstream. addforecasts :: Monad m => Pipe Inputs Inputs m () -- Consume inputs, create a report document, and write it to the database. createandwritereport :: RunInformation -> Consumer Inputs IO ()

38 A word on the spread of Haskell Facebook uses Haskell in the Haxl project, a DSL to allow front-end developers write business logic that performs complex parallell computations in the back-end. Silk is a platform for sharing collections about anything, from your favorite coffee places to modernist painters to 3D printers Pioneering better e-learning. The Nordic market leading solution for e-signing of documents. Bump use a Haskell-based server, Angel, for process supervisor for all their backend systems, and for other infrastructure tasks.

39 Well-Typed - Consultancy AT&T - Network security Bank of America - Data processing Barclays Capital - An EDSL to specify exotic equity derivatives Ericsson AB - An EDSL for digital signal processing algorithms Functor AB - Static analysis Galois - Projects under contract for corporations and government clients in the demanding application areas of security, information assurance and cryptography Standard Chartered - Large group using Haskell for all aspects of its wholesale banking business Tsuru Capital - Operating an automated options trading system written in Haskell

40 Thank you for your time and attention