Concurrency: what, why, how

Transcription

1 Concurrency: what, why, how Oleg Batrashev February 10, 2014

2 what this course is about? additional experience with try out some language concepts and techniques Grading java.util.concurrent.future? agents, software transactional memory (STM) ~5 lab assignments: 20% (1 Java + 4 Clojure) home assignments: 2 15% (2 Clojure) declarative /agents... STM/distributed program... exam: 50% (5-10 questions/exercises for 1-2 hours) Languages: Clojure, Oz, Haskell, Erlang, Scala, Go Concurrency: what, why, how 2 / 35

3 Lecture about everything and nothing in particular Explain basic idea (pseudo) vs. List reasons for using Present briefly different classifications approaches models and languages Concurrency: what, why, how 3 / 35

4 Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) Concurrency: what, why, how 4 / 35

5 Basic idea Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) intuitively simultaneous execution of instructions (with CPU pipelines) actions (functions within a program) programs (distributed application) what is simultaneous? physically at the same time? nearly at the same time? 2 threads on a one single-core CPU? Concurrency: what, why, how 5 / 35

6 Dependency Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) If 2 actions do not need result of each other (independent) do not interfere otherwise e.g. do not write to the same variable/file then the order of their execution does not matter: a = c+d b = c+e may be run in parallel (concurrent) excessive optimism (about action dependency/interference) leads to troubles are a and e aliased? Language semantics do matter! Concurrency: what, why, how 6 / 35

7 Some terms (1) Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) Not a rule, just to extend the understanding of. Parallel - execute simultaneously - the order of execution does not matter From wikipedia: Parallel computing is a form of computation in which many calculations are carried out simultaneously. computing is a form of computing in which programs are designed as collections of interacting computational processes that may be executed in parallel. sometimes referred to as pseudoparallel. Concurrency: what, why, how 7 / 35

8 Some terms (2) Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) Haskell community variants. Parallel - deterministic data crunching simultaneous execution of the same type tasks - non-deterministic execution of unrelated communicating processes From Chapter 24. and multicore : A concurrent program needs to perform several possibly unrelated tasks at the same time. In contrast, a parallel program solves a single problem. Concurrency: what, why, how 8 / 35

9 Some terms (3) Basic idea Dependency Some terms (1) Some terms (2) Some terms (3) Programming Clojure, 5.1 State, Concurrency, Parallelism, and Locking A concurrent program models more than one thing happening simultaneously. A parallel program takes an operation that could be sequential and chooses to break it into separate pieces that can execute concurrently to speed overall execution. Concurrency: what, why, how 9 / 35

10 List of reasons Some real life analogies Faster programs Hiding latency Better structure Concurrency: what, why, how 10 / 35

11 List of reasons List of reasons Some real life analogies Faster programs Hiding latency Better structure Faster programs running on several cores/cpus/computers More responsive programs GUI interface hiding disk/network latency Programs with natural distributed programs (client-server, etc) Fault tolerant programs using redundancy Better structured programs Concurrency: what, why, how 11 / 35

12 Some real life analogies List of reasons Some real life analogies Faster programs Hiding latency Better structure Speed of a process With 1 axe one friend can chop wood and the other collect it (conveier) With 2 axes both friends can chop wood in parallel Hiding latency When we turn on a kettle we do not wait until it boils e.g. we go and take out cups from cupboard then return to the kettle Better structured doing ironing and cooking concurrently is messy assign to different people Concurrency: what, why, how 12 / 35

13 Faster programs List of reasons Some real life analogies Faster programs Hiding latency Better structure Calculate elements of an array in parallel Perform calculations on several processors/nodes Serving youtube videos from multiple servers End of Moore s law The number of transistors that can be placed inexpensively on an integrated circuit has increased exponentially, doubling approximately every two years Every new laptop comes with (at least) dual core technology usually stuck with 50% CPU usage Concurrency: what, why, how 13 / 35

14 Hiding latency List of reasons Some real life analogies Faster programs Hiding latency Better structure disk/network take time either if a thread blocks while waiting for HTTP response, no other action can happen no other network request not even UI events work asynchronously leads to sliced programs and pyramids of doom dedicated thread asking for synchronization troubles.. leads to race conditions Concurrency: what, why, how 14 / 35

15 Better structure List of reasons Some real life analogies Faster programs Hiding latency Better structure Assign different threads to unrelated tasks (if reasonable) Data sharing server vertically, one thread per request typical web processing (excl. database access) horizontally (conveier) dedicated thread(s) for reading requests dedicated thread(s) for searching data new thread for sending data Mixing tasks of all threads in one thread asynchronous behavior structural nightmare Concurrency: what, why, how 15 / 35

16 Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Concurrency: what, why, how 16 / 35

17 Task and data Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Task : different operations concurrently calculate g and h in f(g(x), h(y)) concurrently threads in the same program several programs running on the same computer Data : same operation for different data (SIMD) loop operations: forall i=1..n do a[i]=a[i]+1 vectorised operations: MMX, SSE, etc A program may benefit from both! Concurrency: what, why, how 17 / 35

18 Coarse and fine grained Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Ratio of computation and communication coarse-grain parallel programs compute most of the time e.g distribute data, calculate, collect result (Google MapReduce) fine-grain parallel programs communicate frequently lots of dependencies between distributed data medium-grained DOUG: lots of computation interchange with lots of communication Concurrency: what, why, how 18 / 35

19 High and low level Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Different granularity (unit of ) instruction-level conveiers and pipelines in CPU; MMX expression level run expression in separate thread function level process level Source of confusion: this sometimes referred as fine/coarse grained. Question of terminology? However, it is possible to have two processes that communicate very frequently.. Concurrency: what, why, how 19 / 35

20 (1) Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Models and languages for Parallel Computation; David B. Skillicorn, Domenico Talia; 1998 Parallelism explicit (hints for possible ) Loops: forall i in 1..N do a[i]=i Fortran 90 matrix sum: C=A+B Decomposition explicit (specify parallel pieces) Mapping explicit (map pieces to processors) Communication explicit (specify sends/recvs) Synchronization explicit (handle details of message-passing) Concurrency: what, why, how 20 / 35

21 (2) Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Possibilities 1. nothing explicit, (OBJ, P3L) 2. explicit, decomposition Loops - Fortran variants, Id, APL, NESL 3. decomposition explicit, mapping (BSP, LogP) 4. mapping explicit, communication (Linda) 5. communication explicit, synchronization Actors, smalltalk 6. everything explicit PVM, MPI, fork Concurrency: what, why, how 21 / 35

22 Formalizations Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model How to desribe (concurrent) computations? operational semantics describe operations in Virtual Machine (VM) Oz way reasoning for a programmer denotational semantics describe algebraic rules concurrent lambda calculus, Pi calculus, CSP, Petri nets, DDA (Data Dependency Algebra) reasoning for a matematician axiomatic semantics describe logical rules TLA (Temporal Logic of Actions) reasoning for a machine (a prover) Concurrency: what, why, how 22 / 35

23 By application areas Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model Scientific computing High-Performance Computing (HPC) High-Throughput Computing (HTC) Distributed applications clients, servers P2P telephone stations (Erlang PL) Desktop applications responsive user interfaces utilizing multiple cores Concurrency: what, why, how 23 / 35

24 By computation model Task and data Coarse and fine grained High and low level (1) (2) Formalizations By application areas By computation model What style of is supported? Declarative concurrent model (pure) functional logical Message-passing model synchronous, asynchronous, RPC active objects, passive objects Shared-state (shared memory) model locks transactions Concurrency: what, why, how 24 / 35

25 Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Concurrency: what, why, how 25 / 35

26 Why language? Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Why not just library? cleaner syntax safer semantics forces usage patterns control over compilation process In 198x there were hundreds PLs for concurrent, now there are thousands. the following slides describe some languages Concurrency: what, why, how 26 / 35

27 Oz Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB roots in logic dataflow variables (logical variables with suspension) multiparadigm (advertises different styles of ) functional object oriented constraint (logic) explicit task (thread statement) explicit and communication (through dataflow variables) for distributed and desktop Concurrency: what, why, how 27 / 35

28 Erlang Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Ericsson project from ~1990 for telecom applications Concurrency handle thousands of phone calls robustness, distribution processes with message-passing (actors) focus on fault tolerance Concurrency: what, why, how 28 / 35

29 Scala Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB 2008 year hot topic interoperable with Java (runs on JVM) syntax similar to Java mix of object oriented and functional static typing, type inference, type parameters,... Concurrency in general quite sophisticated task processes with message-passing (actors) not main focus, thus not mature Akka library for actors Concurrency: what, why, how 29 / 35

30 Clojure Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB 2008 year hot topic targets the Java Virtual Machine Lisp syntax functional, macro Concurrency 4 reference models (atoms,vars,refs,agents) task reactive Agent system software transactional memory Concurrency: what, why, how 30 / 35

31 High-Performance Fortran since 1993, extension of Fortran 90 Concurrency data A lot of extensions but no success Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Concurrency: what, why, how 31 / 35

32 NESL Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB since 1995 available only on rare platforms a way to handle nested data Concurrency sparse matrice storage in quicksort algorithm nested data Concurrency: what, why, how 32 / 35

33 and Parallel Haskell Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Parallel Haskell with par and pseq deterministic speculative execution Haskell with forkio locks, monitors, etc synchronization variables MVars STM (software transactional memory) with atomically and more: mhaskell Data Parallel Haskell with parallel arrays NDP (nested data ) Concurrency: what, why, how 33 / 35

34 Intel TBB Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Intel Thread Building Blocks recent C++ library Concurrency task Concurrency: what, why, how 34 / 35

35 ArBB Why language? Oz Erlang Scala Clojure High-Performance Fortran NESL and Parallel Haskell Intel TBB ArBB Intel Array Building Blocks compiler beta version Concurrency immutable data (declarative model) (nested) data Concurrency: what, why, how 35 / 35