Threads: either under- or over-utilised

Transcription

1 Threads: either under- or over-utilised Underutilised: limited by creation speed of work Cannot exploit all the CPUs even though there is more work Overutilised: losing performance due to context switches There is overhead when switching between OS threads Each thread needs to warm up cache again Increases memory pressure Worst case: continual slow-down The cost of creating threads is partially borne by kernel User code may slow down more than kernel code under load Number of workers slowly goes up; completion rate goes down HPCE / dt1/ 215 / 7.1

2 Solving under-utilisation template<class TI, class TF> void parallel_for(const TI &begin, const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; std::thread left( // Spawn the left thread in parallel ); [&]() parallel_for(begin, mid, f); // Perform the right segment on our thread parallel_for(mid, end, f); // wait for the left to finish left.join(); HPCE / dt1/ 215 / 7.2

3 Creation of work using trees Tree starts on one thread [..4) template<class TI, class TF> void parallel_for( const TI &begin,const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; std::thread left( [&]() parallel_for(begin, mid, f); ); parallel_for(mid, end, f); left.join(); HPCE / dt1/ 215 / 7.3

4 Creation of work using trees Tree starts on one thread Create thread to branch [..4) spawn [2..4) [..2) template<class TI, class TF> void parallel_for( const TI &begin,const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; std::thread left( [&]() parallel_for(begin, mid, f); ); parallel_for(mid, end, f); left.join(); HPCE / dt1/ 215 / 7.4

5 Creation of work using trees Tree starts on one thread Create thread to branch [..4) spawn [2..4) spawn [..2) spawn [3..4) [2..3) [1..2) [..1) template<class TI, class TF> void parallel_for( const TI &begin,const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; std::thread left( [&]() parallel_for(begin, mid, f); ); parallel_for(mid, end, f); left.join(); HPCE / dt1/ 215 / 7.5

6 Creation of work using trees Tree starts on one thread Create thread to branch [..4) spawn Execute function at leaves [2..4) spawn [..2) spawn [3..4) [2..3) [1..2) [..1) template<class TI, class TF> void parallel_for( const TI &begin,const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; f(3) f(2) f(1) f() std::thread left( [&]() parallel_for(begin, mid, f); ); parallel_for(mid, end, f); left.join(); HPCE / dt1/ 215 / 7.6

7 Creation of work using trees Tree starts on one thread Create thread to branch Execute function at leaves Join back up to the root [..4) [2..4) [3..4) spawn spawn [..2) spawn [2..3) [1..2) [..1) template<class TI, class TF> void parallel_for( const TI &begin,const TI &end, const TF &f) if(begin+1 == end) f(begin); else TI mid=(begin+end)/2; f(3) f(2) f(1) f() [3..4) [2..3) [1..2) [..1) join join [2..4) join [..2) std::thread left( [&]() parallel_for(begin, mid, f); ); parallel_for(mid, end, f); left.join(); [..4) HPCE / dt1/ 215 / 7.7

8 Properties of fork / join trees Recursively creating trees of work is very efficient We are not limited to one thread creating all tasks Exponential rather than linear growth of threads with time Problem solved? Growth of threads is exponential with time Can put significant pressure on the OS thread scheduler Context switching 1s of threads is very inefficient Each thread requires significant resources Need kernel handles, stack, thread-info block,... Can t allocate more than a few thousand threads per process HPCE / dt1/ 215 / 7.8

9 Re-examining the goals What we want is parallel_for: Iterations may execute in parallel std::thread gives us something different: The new thread will execute in parallel Our thread based strategy is too eager to go parallel We want to go just parallel enough, then stay serial HPCE / dt1/ 215 / 7.9

10 Tasks versus threads A task is a chunk of work that can be executed A task may execute in parallel with other tasks A task will eventually be executed, but no guarantee on when Tasks are scheduled and executed by a run-time (TBB) Maintain a list of tasks which are ready to run Have one thread per CPU for running tasks If a thread is idle, assign a task from the ready queue to it No limit on number of tasks which are ready to run (OS is still responsible for mapping threads to CPUs) TBB has a number of high-level ways to use tasks But there is a single low-level underlying task primitive HPCE / dt1/ 215 / 7.1

11 Overview of task groups A task group collects together a number of child tasks The task creating the group is called the parent One or more child tasks are created and run() by the parent Child tasks may execute in parallel Parent task must wait() for all child tasks before returning HPCE / dt1/ 215 / 7.11

12 parallel_for using tbb::task_group #include "tbb/task_group.h" template<class TI, class TF> void parallel_for(const TI &begin, const TI &end, const TF &f) if(begin+1 == end) f(begin); else auto left=[&]() parallel_for(begin, (begin+end)/2, f); auto right=[&]() parallel_for((begin+end)/2, end, f); // Spawn the two tasks in a group tbb::task_group group; group.run(left); group.run(right); group.wait(); // Wait for both to finish HPCE / dt1/ 215 / 7.12

13 Overview of task groups A task group collects together a number of child tasks The task creating the group is called the parent One or more child tasks are created and run() by the parent Child tasks may execute in parallel Parent task must wait() for all child tasks before returning Some important differences between tasks and threads Threads must execute in parallel A thread may continue after its creator exits Threads must be joined individually HPCE / dt1/ 215 / 7.13

14 More patterns: tbb::parallel_invoke template<typename Func, typename Func1> void parallel_invoke(const Func& f, const Func1& f1); template<typename Func, typename Func1, typename Func2> void parallel_invoke(const Func& f, const Func1& f1, const Func2& f2); Takes two or more functions and may run in parallel Overloaded for different numbers of arguments No overload for 1 argument for obvious reasons Interface is very clean, but also quite simple Decision about number of tasks is completely static You can t add more tasks once some starts No choice about when to synchronise with tasks HPCE / dt1/ 215 / 7.14

15 parallel_invoke using task_group parallel_invoke can be implement using task_group task_group supports a super-set of the functionality template<typename Fc, typename Fc1, typename Fc2> void parallel_invoke(const Fc& f, const Fc1& f1, const Fc2& f2) tbb::task_group group; group.run(f); group.run(f1); group.run(f2); group.wait(); HPCE / dt1/ 215 / 7.15

16 Can t do task_group using parallel_invoke task_group is intrinsically dynamic Decide how much work to add at run-time Can add work even while tasks are running in the group void my_function(int n, float *x) tbb::task_group group; for(unsigned i=;i<n;i++) if(x[i]==) group.run([=]() f(i); ); else group.run([=]() g(x[i]); ); group.wait(); HPCE / dt1/ 215 / 7.16

17 The underlying primitive: tbb::task TBB has a basic primitive called tbb::task This is the raw unit of scheduling understood by the lib. Other high-level wrappers create tasks internally The TBB run-time takes tasks and schedules them to a CPU Tasks are very flexible, with a lot of power Can express complicate dependency graphs Build non-local synchronisation and barriers With power comes responsibility They allow you to make mistakes Possible (though not likely) to mess up the TBB run-time Better to create wrappers on top that hide tasks parallel_for, parallel_invoke, task_group, parallel_reduce,... HPCE / dt1/ 215 / 7.17

18 class : public tbb::task int start, end; ; (int _start, int _end) start=_start; end=_end; tbb::task * execute() if(cond()) set_ref_count(3); void (int start, int end) if(cond()) tbb::task_group group; group.run([=]()(start,(start+end)/2); ); group.run([=]()((start+end)/2,end); ); DoSomethingFirst(); group.wait(); DoSomethingElse(); &t1=*new(allocate_child()) (start,(start+end)/2); spawn(t1); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse(); void CreateTasks(int start, int end) &root=*new(allocate_root()) (start,end); tbb::task::spawn_root_and_wait(); HPCE / dt1/ 215 / 7.18

19 Life-cycle of a task Life-cycle of task due to interaction between task and runtime Individual task calls spawn, wait_for_all (sync), return TBB run-time will keep track of a task s children (dependencies)

20 Scheduling through reference counts Each task has a reference count and a task The reference count identifies whether a task is blocked If the reference count is zero then the task could be run But only if it has been given to the task scheduler Legal to create a task and not give it to the scheduler Note the difference: reference count vs C++ reference Successor task identifies the task blocked by this task Generally the task is the creator, or parent When a task completes it decrements the count of its HPCE / dt1/ 215 / 7.2

21 tbb::task Created - void CreateTasks(int start, int end) &root=*new(allocate_root()) (start,end); tbb::task::spawn_root_and_wait();

22 tbb::task Running - void CreateTasks(int start, int end) &root=*new(allocate_root()) (start,end); tbb::task::spawn_root_and_wait();

23 tbb::task Running - tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

24 tbb::task Running - 3 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

25 tbb::task Running - tbb::task Created 3 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

26 tbb::task Running - tbb::task Created tbb::task Created 3 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

27 tbb::task Running - tbb::task Ready tbb::task Ready 3 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

28 tbb::task Running - tbb::task Running tbb::task Ready 3 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

29 tbb::task Blocked - tbb::task Running tbb::task Ready 2 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

30 tbb::task Blocked - tbb::task Running tbb::task Ready 2 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

31 tbb::task Blocked - tbb::task Running tbb::task Running 2 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

32 tbb::task Blocked - tbb::task Running tbb::task Finished 1 tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

33 tbb::task Running - tbb::task Finished tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

34 tbb::task Running - tbb::task * ::execute() if(cond()) set_ref_count(3); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

35 tbb::task Finished - void CreateTasks(int start, int end) &root=*new(allocate_root()) (start,end); tbb::task::spawn_root_and_wait();

36 Managing reference counts What happens if we get the reference count wrong? Finishing task calls decrement_ref_count on Automatically returns task to scheduler if count becomes zero HPCE / dt1/ 215 / 7.36

37 tbb::task Running - 1 tbb::task * ::execute() if(cond()) set_ref_count(1); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

38 tbb::task Running - tbb::task Finished tbb::task Ready tbb::task * ::execute() if(cond()) set_ref_count(1); &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

39 tbb::task Running - tbb::task Running tbb::task * ::execute() if(cond()) &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); set_ref_count(3); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

40 tbb::task Running - tbb::task Finished tbb::task Ready -1 tbb::task * ::execute() if(cond()) &t1=*new(allocate_child()) (start,(start+end)/2); &t2=*new(allocate_child()) ((start+end)/2, end); spawn(t1); spawn(t2); set_ref_count(3); DoSomethingFirst(); wait_for_all(); DoSomethingElse();

41 Some help is available TBB library comes in two forms: debug and release release library does no error checking all about speed debug library will check reference counts at many points Choose library version at compilation and link stages Debug: #define TBB_USE_DEBUG=1 when compiling On microsoft compilers it will automatically link the correct library On other compilers use -ltbb vs -ltbb_debug Usually maintain different release and debug settings Debug: /DTBB_USE_DEBUG=1 /MDd Release: /DNDEBUG=1 /O2 Can setup in Visual Studio or in a makefile Generally: try to avoid raw tasks if possible HPCE / dt1/ 215 / 7.41

42 Many design patterns are built on tasks Iteration in various forms parallel_for, parallel_for_each Reduction and accumulation parallel_reduce Data-dependent looping and queue processing parallel_do Support for heterogeneous tasks parallel_invoke, task_group Heterogeneous tasks and token-based data-flow parallel_pipeline Goal: turn design patterns in concrete functions HPCE / dt1/ 215 / 7.42