List of lectures. Lecture content. Concurrent computing. TDDA69 Data and Program Structure Concurrent Computing Cyrille Berger

Size: px

Start display at page:

Download "List of lectures. Lecture content. Concurrent computing. TDDA69 Data and Program Structure Concurrent Computing Cyrille Berger"

Mariah McBride
5 years ago
Views:

List of lectures TDDA69 Data and Program Structure Concurrent Computing Cyrille Berger 1Introduction and Functional Programming 2Imperative Programming and Data Structures 3Parsing 4Evaluation

/ 65 Lecture content Parallel Programming Multithreaded Programming The States Problems and Solutions Atomic actions Language and Interpreter Design Considerations Single Instruction, Multiple

1 List of lectures TDDA69 Data and Program Structure Concurrent Computing Cyrille Berger 1Introduction and Functional Programming 2Imperative Programming and Data Structures 3Parsing 4Evaluation 5Object Oriented Programming 6Macros and decorators 7Virtual Machines and Bytecode 8Garbage Collection and Native Code 9Concurrent Computing 10Declarative Programming 11Logic Programming 12Summary 2 / 65 Lecture content Parallel Programming Multithreaded Programming The States Problems and Solutions Atomic actions Language and Interpreter Design Considerations Single Instruction, Multiple Threads Programming Distributed programming Message Passing MapReduce Concurrent computing In concurrent computing several computations are executed at the same time In parallel computing all computations units have access to shared memory (for instance in a single process) In distributed computing computations units communicate through messages passing 3 / 65 4 / 65

2 Benefits of concurrent computing Disadvantages of concurrent computing Faster computation Responsiveness Interactive applications can be performing two tasks at the same time: rendering, spell checking... Availability of services Load balancing between servers Controllability Tasks can be suspended, resumed and stopped. 5 / 65 Concurrency is hard to implement properly Safety Easy to corrupt memory Deadlock Tasks can wait indefinitely for each other Non-deterministic Not always faster! The memory bandwidth and CPU cache is limited 6 / 65 Concurrent computing programming Stream Programming in Functional Programming Four basic approach to computing: Sequencial programming: no concurrency Declarative concurrency: streams in a functional language Message passing: with active objects, used in distributed computing Atomic actions: on a shared memory, used in parallel computing No global state Functions only act on their input, they are reentrant Functions can then be executed in parallel As long as they do not depend on the output of an other function 7 / 65 8 / 65

Parallel Programming Parallel Programming In parallel computing several computations are executed at the same time and have access to

elements) are processed in parallel SIMT: Single Instruction, Multiple Threads The same instruction is executed by multiple threads

different threads SIMD, SIMT, SMT (2/2) SIMD: PUSH [1, 2, 3, 4] PUSH [4, 5, 6, 7] VEC_ADD_4 SIMT: execute([1, 2, 3, 4], [4, 5, 6, 7],

3 Parallel Programming Parallel Programming In parallel computing several computations are executed at the same time and have access to shared memory Unit Unit Unit Memory 10 SIMD, SIMT, SMT (1/2) SIMD: Single Instruction, Multiple Data Elements of a short vector (4 to 8 elements) are processed in parallel SIMT: Single Instruction, Multiple Threads The same instruction is executed by multiple threads (from 128 to 3048 or more in the future) SMT: Simultaneous Multithreading General purpose, different instructions are executed by different threads SIMD, SIMT, SMT (2/2) SIMD: PUSH [1, 2, 3, 4] PUSH [4, 5, 6, 7] VEC_ADD_4 SIMT: execute([1, 2, 3, 4], [4, 5, 6, 7], lambda a,b,ti: a[ti]=a[ti] + max(b[ti], 5)) SMT: a = [1, 2, 3, 4] b = [4, 5, 6, 7]... Thread.new(lambda : a = a + b) Thread.new(lambda : c = c * b) 11 12

4 Why the need for the different models? Flexibility: SMT > SIMT > SIMD Less flexibility give higher performance Unless the lack of flexibility prevent to accomplish the task Performance: SIMD > SIMT > SMT Multithreaded Programming 13 Singlethreaded vs Multithreaded Multithreaded Programming Mode Start with a single root thread Fork: to create concurently executing threads Join: to synchronize threads Threads communicate through shared memory Threads execute assynchronously They may or may not execute on different processors sub 0 sub 0 main... main... main sub n sub n 15 16

5 A multithreaded example thread1 = new Thread( /* do some computation */ thread2 = new Thread( /* do some computation */ thread1.start(); thread2.start(); thread1.join(); thread2.join(); The States Problems and Solutions 17 Global States and multi-threading Example: var a = 0; thread1 = new Thread( a = a + 1; thread2 = new Thread( a = a + 1; thread1.start(); thread2.start(); What is the value of a? This is called a (data) race condition Atomic actions 19

6 Atomic operations Mutex An operation is said to be atomic, if it appears to happen instantaneously read/write, swap, fetch-and-add... test-and-set: set a value and return the old one To implement a lock: while(test-and-set(lock, 1) == 1) To unlock: lock = 0 Mutex is the short of Mutual exclusion It is a technique to prevent two threads to access a shared resource at the same time Example: var a = 0; var m = new Mutex(); thread1 = new Thread( m.lock(); a = a + 1; m.unlock(); thread2 = new Thread( m.lock(); a = a + 1; m.unlock(); thread1.start(); thread2.start(); Now a= Dependency Condition variable Example: var a = 1; var m = new Mutex(); thread1 = new Thread( m.lock(); a = a + 1; m.unlock(); thread2 = new Thread( m.lock(); a = a * 3; m.unlock(); thread1.start(); thread2.start(); What is the value of a? 4 or 6? A Condition variable is a set of threads waiting for a certain condition Example: var a = 1; var m = new Mutex(); var cv = new ConditionVariable(); thread1 = new Thread( m.lock(); a = a + 1; cv.notify(); m.unlock(); thread2 = new Thread( cv.wait(); m.lock(); a = a * 3; m.unlock(); thread1.start(); thread2.start(); a =

7 Deadlock Advantages of atomic actions What might happen: var a = 0; var b = 2; var ma = new Mutex(); var mb = new Mutex(); thread1 = new Thread( ma.lock(); mb.lock(); b = b - 1; a = a - 1; ma.unlock(); mb.unlock(); thread2 = new Thread( mb.lock(); ma.lock(); b = b - 1; a = a + b; mb.unlock(); ma.unlock(); thread1.start(); thread2.start(); thread1 waits for mb, thread2 waits for ma Very efficient Less overhead, faster than message passing Disadvantages of atomic actions Blocking Meaning some threads have to wait Small overhead Deadlock A low-priority thread can block a high priority thread A common source of programming errors Language and Interpreter Design Considerations 27

8 Common mistakes Forget to unlock a mutex Race condition Deadlocks Granularity issues: too much locking will kill the performance Forget to unlock a mutex Most programming language have, either: A guard object that will unlock a mutex upon destruction A synchronization statement some_rlock = threading.rlock() with some_rlock: print("some_rlock is locked while this executes") Race condition Can we detect potential race condition during compilation? In the rust programming language Objects are owned by a specific thread Types can be marked with Send trait indicate that the object can be moved between threads Types can be marked with Sync trait indicate that the object can be accessed by multiple threads safely Safe Shared Mutable State in rust (1/3) let mut data = vec![1, 2, 3]; for i in 0..3 thread::spawn(move data[i] += 1; Gives an error: "capture of moved value: `data`" 31 32

9 Safe Shared Mutable State in rust (2/3) Safe Shared Mutable State in rust (3/3) let mut data = Arc::new(vec![1, 2, 3]); for i in 0..3 let data = data.clone(); thread::spawn(move data[i] += 1; Arc add reference counting, is movable and syncable Gives error: cannot borrow immutable borrowed content as mutable let data = Arc::new(Mutex::new(vec![1, 2, 3])); for i in 0..3 let data = data.clone(); thread::spawn(move let mut data = data.lock().unwrap(); data[i] += 1; It now compiles and it works Single Instruction, Multiple Threads Programming Single Instruction, Multiple Threads Programming With SIMT, the same instructions is executed by multiple threads on different registers 36

10 Single instruction, multiple flow paths (1/2) Single instruction, multiple flow paths (1/2) Using a masking system, it is possible to support if/ else block Threads are always executing the instruction of both part of the if/else blocks data = [-2, 0, 1, -1, 2], data2 = [...] function f(thread_id, data, data2) if(data[thread_id] < 0) data[thread_id] = data[thread_id]-data2[thread_id]; else if(data[thread_id] > 0) data[thread_id] = data[thread_id]+data2[thread_id]; Benefits: Multiple flows are needed in many algorithms Drawbacks: Only one flow path is executed at a time, non running threads must wait Randomize memory access Elements of a vector are not accessed sequentially Programming Language Design for SIMT OpenCL, CUDA are the most common Very low level, C/C++-derivative General purpose programming language are not suitable Some work has been done to be able to write in Python and run on a GPU with float32[:], float32[:]], target='gpu') def add_matrix(a, B, C): A[cuda.threadIdx.x] = B[cuda.threadIdx.x] + C[cuda.threadIdx.x] with limitation on standard function that can be called Distributed programming 39

11 Distributed Programming (1/4) Distributed programming (2/4) In distributed computing several computations are executed at the same time and communicate through messages passing Unit Unit Unit Memory Memory Memory A distributed computing application consists of multiple programs running on multiple computers that together coordinate to perform some task. Computation is performed in parallel by many computers. Information can be restricted to certain computers. Redundancy and geographic diversity improve reliability Distributed programming (3/4) Distributed programming (4/4) Characteristics of distributed computing: Computers are independent they do not share memory. Coordination is enabled by messages passed across a network. Individual programs have differentiating roles. Distributed computing for largescale data processing: Databases respond to queries over a network. Data sets can be partitioned across multiple machines

Message Passing Message Passing Messages are (usually) passed through sockets Messages are exchanged synchronously or asynchronously Communication can be centralized or peer-to-peer 46 Python's

Every 100 interpreter ticks True multithreading is not possible with CPython 47 CPython can only interpret one single thread at a given time Single-threaded are faster (no need to lock in memory

12 Message Passing Message Passing Messages are (usually) passed through sockets Messages are exchanged synchronously or asynchronously Communication can be centralized or peer-to-peer 46 Python's Global Interpreter Lock (1/2) Python's Global Interpreter Lock (2/2) CPython can only interpret one single thread at a given time The lock is released, when: The current thread is blocking for I/O Every 100 interpreter ticks True multithreading is not possible with CPython 47 CPython can only interpret one single thread at a given time Single-threaded are faster (no need to lock in memory management) Many C-library used as extensions are not thread safe To eliminate the GIL Python developers have the following requirements: Simplicity Do actually improve performance Backward compatible Prompt and ordered destruction 48

13 Python's Multiprocessing module The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads It implements transparent message passing, allowing to exchange Python objects between processes 49 Python's Message Passing (1/2) Example of message passing from multiprocessing import Process def f(name): print 'hello', name if name == ' main ': p = Process(target=f, args=('bob',)) p.start() p.join() Output hello bob 50 Python's Message Passing (2/2) Example of message passing with pipes from multiprocessing import Process, Pipe def f(conn): conn.send([42, None, 'hello']) conn.close() if name == ' main ': parent_conn, child_conn = Pipe() p = Process(target=f, args=(child_conn,)) p.start() print parent_conn.recv() p.join() Output [42, None, 'hello'] Transparent message passing is possible thanks to serialization 51 Serialization A serialized object is an object represented as a sequence of bytes that includes the object s data, its type and the types of data stored in the object. 52

14 pickle In Python, serialization is done with the pickle module It can serialize user-defined classes The class definition must be available before deserialization Works with different version of Python By default, use an ASCII format It can serialize: Basic types: booleans, numbers, strings Containers: tuples, lists, sets and dictionnary (of pickable objects) Top level functions and classes (only the name) Objects where dict or getstate() are pickable Example: pickle.loads(pickle.dumps(10)) Shared memory Memory can be shared between Python process with a Value or Array. from multiprocessing import Process, Value, Array def f(n, a): n.value = for i in range(len(a)): a[i] = -a[i] if name == ' main ': num = Value('d', 0.0) arr = Array('i', range(10)) p = Process(target=f, args=(num, arr)) p.start() p.join() print num.value print arr[:] And of course, you would need to use Mutex to avoid race condition Big Data Processing (1/2) MapReduce MapReduce is a framework for batch processing of big data. Framework: A system used by programmers to build applications Batch processing: All the data is available at the outset, and results are not used until processing completes Big data: Used to describe data sets so large and comprehensive that they can reveal facts about a whole population, usually from statistical analysis 56

Big Data Processing (2/2) The MapReduce idea: Data sets are too big to be analyzed by one machine Using multiple machines has the same complications, regardless of the application/ analysis Pure

15 Big Data Processing (2/2) The MapReduce idea: Data sets are too big to be analyzed by one machine Using multiple machines has the same complications, regardless of the application/ analysis Pure functions enable an abstraction barrier between data processing logic and coordinating a distributed application MapReduce Evaluation Model (1/2) Map phase: Apply a mapper function to all inputs, emitting intermediate key-value pairs The mapper takes an iterable value containing inputs, such as lines of text The mapper yields zero or more key-value pairs for each input MapReduce Evaluation Model (2/2) MapReduce Execution Model (1/2) Reduce phase: For each intermediate key, apply a reducer function to accumulate all values associated with that key The reducer takes an iterable value containing intermediate key-value pairs All pairs with the same key appear consecutively The reducer yields zero or more values, each associated with that intermediate key 59 60

16 MapReduce Execution Model (2/2) MapReduce example From a 1.1 billion people database (facebook?), we want to know the average number of friends per age In SQL: SELECT age, AVG(friends) FROM users GROUP BY age In MapReduce: the total set of users in splitted different users_set function map(users_set) for(user in users_set) send(user.age, user.friends.size); The keys are shuffled and assigned to reducers function reduce(age, friends): var r = 0; for(friend in friends) r += friend; send(age, r / friends.size); MapReduce Assumptions Constraints on the mapper and reducer: The mapper must be equivalent to applying a deterministic pure function to each input independently The reducer must be equivalent to applying a deterministic pure function to the sequence of values for each key Benefits of functional programming: When a program contains only pure functions, call expressions can be evaluated in any order, lazily, and in parallel Referential transparency: a call expression can be replaced by its value (or vis versa) without changing the program In MapReduce, these functional programming ideas allow: Consistent results, however computation is partitioned Re-computation and caching of results, as needed MapReduce Benefits Fault tolerance: A machine or hard drive might crash The MapReduce framework automatically re-runs failed tasks Speed: Some machine might be slow because it's overloaded The framework can run multiple copies of a task and keep the result of the one that finishes first Network locality: Data transfer is expensive The framework tries to schedule map tasks on the machines that hold the data to be processed Monitoring: Will my job finish before dinner?!? The framework provides a web-based interface describing jobs 63 64

17 Summary Parallel programming Multi-threading and how to help reduce programmer error Distributed programming and MapReduce 65 / 65

61A Lecture 36. Wednesday, November 30

61A Lecture 36. Wednesday, November 30 61A Lecture 36 Wednesday, November 30 Project 4 Contest Gallery Prizes will be awarded for the winning entry in each of the following categories. Featherweight. At most 128 words of Logo, not including