Shared memory parallel computing. Intel Threading Building Blocks

Transcription

1 Shared memory parallel computing Intel Threading Building Blocks

2 Introduction & history Threading Building Blocks (TBB) cross platform C++ template lib for task-based shared memory parallel programming v1.0 created in 2006 one year after Intel Pentium D v4.4 is currently the most recent version

3 Introduction & history Although Intel proprietary, it s worth looking at because: it has an Open Source licensed version it s supported, maintained & documented on a wide range of platforms and operating systems (Windows, *NIX, Android) it uses some interesting & important concepts (ideas from Cilk, task-based parallelism, work-stealing scheduler,...) it actually works pretty well

4 Introduction & history Must be copyconstructible! Implicit barrier 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 5 using namespace std; 6 using namespace tbb; 7 8 /** 9 * Functor that represents the task being parallelized 10 */ 11 struct Func { 12 void operator()(const blocked_range<size_t>& range) const { 13 cout << "Hello world!" << endl; 14 } 15 }; int main(int argc, char* argv[]) { 18 // Define the task 19 Func f; 20 // Define the range to work on 21 blocked_range<size_t> rng(0, 10); 22 // Run task in parallel 23 parallel_for(rng, f); return 0; 26 } size_t is used to represent the maximum size of any object (including arrays) in the particular implementation. It is used as the return type of the sizeof operator. Must not modify Func since it can be copied! Half open interval [0,10) i.e. values [0,...,9]

5 Introduction & history 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 5 using namespace std; 6 using namespace tbb; 7 8 int main(int argc, char* argv[]) { 9 // Define the range to work on 10 blocked_range<size_t> rng(0, 10); 11 // Run task in parallel 12 parallel_for(rng, [](const blocked_range<size_t>& range){ 13 cout << "Hello world!" << endl; 14 }); return 0; 17 }

6 Upcoming topics Common parallel algorithms in TBB Less technical Synchronisation Concurrent containers Work-stealing task scheduler Memory allocation issues Practical remarks More technical

7 Common parallel algorithms parallel_for 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 5 using namespace std; 6 using namespace tbb; 7 8 class Work { 9 public: 10 Work(double* data) : m_data(data) {} 11 void operator()(const blocked_range<size_t>& r) const { 12 for (size_t idx = r.begin(); idx!= r.end(); ++idx) { 13 // Do some work on data[idx] 14 m_data[idx] = 2 * m_data[idx]; 15 } 16 } 17 private: 18 double* m_data; 19 }; int main(int argc, char* argv[]) { 22 // Define the data set to work on 23 const int n = 1000; 24 double* data = new double[n]; 25 const int grain_size = 10; 26 blocked_range<size_t> data_rng(0, n, grain_size); 27 // Do work in parallel 28 parallel_for(data_rng, Work(data)); 29 // Clean up 30 delete[] data; 31 return 0; 32 } To implement the concept of a parallel_for body, the class Work must implement: Work::Work(const Work&); Work::~Work(); void Work::operator()(Range& r) const;

8 Common parallel algorithms parallel_for 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 5 using namespace std; 6 using namespace tbb; 7 8 class Work { 9 public: 10 Work(double* data) : m_data(data) {} 11 void operator()(const blocked_range<size_t>& r) const { 12 for (size_t idx = r.begin(); idx!= r.end(); ++idx) { 13 // Do some work on data[idx] 14 m_data[idx] = 2 * m_data[idx]; 15 } 16 } 17 private: 18 double* m_data; 19 }; int main(int argc, char* argv[]) { 22 // Define the data set to work on 23 const int n = 1000; 24 double* data = new double[n]; 25 const int grain_size = 10; 26 blocked_range<size_t> data_rng(0, n, grain_size); 27 // Do work in parallel 28 parallel_for(data_rng, Work(data)); 29 // Clean up 30 delete[] data; 31 return 0; 32 } There s also a blocked_range2d blocked_range3d or your_own_range_concept

9 1 #include <iostream> 2 #include Common <cstdlib> parallel algorithms 3 #include <tbb/parallel_reduce.h> 4 #include <tbb/blocked_range.h> 5 6 using namespace std; parallel_reduce (imperative) 7 using namespace tbb; 8 9 struct Sum { 10 Sum(double* data) : m_data(data), m_sum(0.0) {} // Constructor 11 Sum(Sum& that, split) : m_data(that.m_data), m_sum(0.0) {} // Split constructor 12 // Local part of reduction (NOTE: might be called more than once!!!) 13 void operator()(const blocked_range<size_t>& r) { 14 for (size_t idx = r.begin(); idx!= r.end(); ++idx) { 15 m_sum += m_data[idx]; 16 } 17 } 18 void join(const Sum& that) { m_sum += that.m_sum; } // Join the work of that into this 19 double* m_data; // Pointer to array to reduce 20 double m_sum; // Local sum 21 }; int main(int argc, char* argv[]) { 24 if (argc!= 2) { 25 cout << argv[0] << " n" << endl; 26 return 1; 27 } 28 // Define the data set to work on 29 const int n = atoi(argv[1]); 30 double* data = new double[n]; 31 for (int idx = 0; idx < n; ++idx) { 32 data[idx] = 1 + idx; 33 } // Run the reduction task in parallel 36 Sum sum(data); 37 parallel_reduce(blocked_range<size_t>(0, n), sum); cout << "sum_{i=1}^{" << n << "}(i) = " << sum.m_sum << endl; // Clean up 42 delete[] data; 43 return 0; 44 } To implement the concept of a parallel_reduce body, the class Sum must implement: Sum::Sum(Sum& that, split); Sum::~Sum(); void Sum::operator()(Range& r); void Sum::join(Sum& that); split is a dummy argument defined in the library to distinguish split ctor from regular copy ctor

10 Common parallel algorithms parallel_reduce (functional) See website for an example!

11 Common parallel algorithms 3 #include <tbb/pipeline.h> 1 #include <iostream> 2 #include <algorithm> 4 #include <tbb/concurrent_queue.h> 5 6 tbb::concurrent_queue<std::string*> bag; 7 tbb::concurrent_queue<std::string*> shelf; 8 9 struct ApplyButter : public tbb::filter { 10 ApplyButter() : tbb::filter(tbb::filter::parallel) {} 11 void* operator()(void* sandwich) { 12 static_cast<std::string*>(sandwich)->append(" with butter"); 13 return sandwich; 14 } 15 }; struct AddCheese : public tbb::filter { 18 AddCheese() : tbb::filter(tbb::filter::parallel) {} 19 void* operator()(void* sandwich) { 20 static_cast<std::string*>(sandwich)->append(" and cheese"); 21 return sandwich; 22 } 23 }; struct TakeSandwichFromBag : public tbb::filter { 26 TakeSandwichFromBag() : tbb::filter(tbb::filter::parallel) {} 27 void* operator()(void* dummy) { 28 std::string* sandwich; 29 if (bag.try_pop(sandwich)) { 30 return sandwich; 31 } else { 32 return NULL; 33 } 34 } 35 }; struct PutSandwichOnShelf : public tbb::filter { 38 PutSandwichOnShelf() : tbb::filter(tbb::filter::parallel) {} 39 void* operator()(void* sandwich) { 40 shelf.push(static_cast<std::string*>(sandwich)); 41 return NULL; 42 } 43 }; pipeline A pipeline represents pipelined application of a series of filters to a stream of items. 45 int main(int argc, char* argv[]) { 46 for (int idx = 0; idx < 10; ++idx) 47 bag.push(new std::string("sandwich")); TakeSandwichFromBag tsfb; 50 ApplyButter ab; 51 AddCheese ac; 52 PutSandwichOnShelf psos; tbb::pipeline sandwich_machine; 55 sandwich_machine.add_filter(tsfb); 56 sandwich_machine.add_filter(ab); 57 sandwich_machine.add_filter(ac); 58 sandwich_machine.add_filter(psos); 59 sandwich_machine.run(4); 60 sandwich_machine.clear(); return 0; 63 }

12 Common parallel algorithms parallel_do For parallel iteration over iteration spaces without an a priori known size. Does NOT scale well but might give a speedup in some cases. parallel_scan parallel_sort Guess what?

13 Synchronisation mutex 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 #include <tbb/mutex.h> 5 6 using namespace std; 7 using namespace tbb; 8 9 int main(int argc, char* argv[]) { 10 mutex m; parallel_for(blocked_range<size_t>(0, 10), [&](blocked_range<size_t>& r){ 13 m.lock(); 14 cout << "Doing some critical work on [" << r.begin() << ", " << r.end() << "]" << endl; 15 m.unlock(); 16 }); return 0; 19 }

14 Synchronisation mutex::scoped_lock 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 #include <tbb/mutex.h> 5 6 using namespace std; 7 using namespace tbb; 8 9 int main(int argc, char* argv[]) { 10 mutex m; parallel_for(blocked_range<size_t>(0, 10), [&](blocked_range<size_t>& r){ 13 mutex::scoped_lock my_lock(m); 14 cout << "Doing some critical work on [" << r.begin() << ", " << r.end() << "]" << endl; 15 }); return 0; 18 } Named object A temporary would release the lock immediately!

15 Synchronisation spin_mutex 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 #include <tbb/spin_mutex.h> 5 6 using namespace std; 7 using namespace tbb; 8 9 int main(int argc, char* argv[]) { 10 spin_mutex m; parallel_for(blocked_range<size_t>(0, 10), [&](blocked_range<size_t>& r){ 13 spin_mutex::scoped_lock my_lock(m); 14 cout << "Doing some critical work on [" << r.begin() << ", " << r.end() << "]" << endl; 15 }); return 0; 18 }

16 Synchronisation null_mutex 1 #include <iostream> 2 #include <tbb/parallel_for.h> 3 #include <tbb/blocked_range.h> 4 #include <tbb/null_mutex.h> 5 6 using namespace std; 7 using namespace tbb; 8 9 int main(int argc, char* argv[]) { 10 null_mutex m; parallel_for(blocked_range<size_t>(0, 10), [&](blocked_range<size_t>& r){ 13 null_mutex::scoped_lock my_lock(m); 14 cout << "Doing some critical work on [" << r.begin() << ", " << r.end() << "]" << endl; 15 }); return 0; 18 } DDDoDooioiininngngg g s ssosoomommemee e c ccrcrririitittitiiciccacaalall l w wwowoororrkrkk k o oononn 2,,, D 86o3]]i] n gdd Doosoiioinnmnggeg sscsoorommimeete i ccccrrariilitt tiiwiccocaarallkl wwowoonorr rkk[k 1 oo,onn 3,,, D 97o4]]i] n gd osionmge scormiet iccrailt iwcoarlk woonr k[ 4o,n 5[]9, 10]

17 1 #include <cstdlib> 2 #include <iostream> 3 #include <tbb/atomic.h> 4 #include <tbb/parallel_for.h> 5 #include <tbb/blocked_range.h> 6 7 using namespace std; 8 using namespace tbb; 9 10 int main(int argc, char* argv[]) { 11 if (argc!= 2) { 12 cout << argv[0] << " n" << endl; 13 return 1; 14 } 15 int n = atoi(argv[1]); atomic<int> sum_atomic; 18 sum_atomic = 0; 19 int sum = 0; 20 Synchronisation atomic<t> 21 parallel_for(blocked_range<size_t>(0, n), [&](blocked_range<size_t>& r){ 22 sum_atomic += 1; 23 }); parallel_for(blocked_range<size_t>(0, n), [&](blocked_range<size_t>& r){ 26 sum += 1; 27 }); cout << "sum = " << sum << endl; 30 cout << "sum_atomic = " << sum_atomic << endl; return 0; 33 } Example result: $./ex_atomic 100 sum = 95 sum_atomic = 100

18 Concurrent containers Typical STL containers are NOT thread-safe! (i.e.: watch out for simultaneous access to a shared container from different threads) TBB has a few that ARE thread-safe: Consider using them! (They are probably much more efficient than mutexing your own containers) concurrent_hash_map concurrent_unordered_map concurrent_unordered_set concurrent_queue concurrent_bounded_queue concurrent_priority_queue concurrent_vector

19 Work-stealing task scheduler Algorithms / concepts provide a high-level logical view of parallelism Low-level task creation is also possible

20 Work-stealing task scheduler 1 #include <ctime> 2 #include <cstdlib> 3 #include <iostream> 4 #include <tbb/task.h> 5 6 struct TaskA : public tbb::task { 7 tbb::task* execute() { 8 std::cout << " Doing the laundry" << std::endl; 9 sleep(2); 10 std::cout << " Laundry is done" << std::endl; 11 return 0; 12 } 13 }; struct TaskB : public tbb::task { 16 tbb::task* execute() { 17 std::cout << " Doing the dishes" << std::endl; 18 sleep(1); 19 std::cout << " Dishes are clean" << std::endl; 20 return 0; 21 } 22 }; struct TaskRoot : public tbb::task { 25 tbb::task* execute() { 26 std::cout << "Doing household tasks" << std::endl; 27 TaskA* t_a = new(tbb::task::allocate_child()) TaskA; 28 TaskB* t_b = new(tbb::task::allocate_child()) TaskB; 29 set_ref_count(3); // Number of children we spawn (including the wait) 30 spawn(*t_a); 31 spawn(*t_b); 32 wait_for_all(); 33 std::cout << "All tasks finished; let's go to the pub" << std::endl; 34 return 0; 35 } 36 }; int main(int argc, char* argv[]) { 39 TaskRoot* t_r = new(tbb::task::allocate_root()) TaskRoot; 40 tbb::task::spawn_root_and_wait(*t_r); 41 return 0; 42 } Doing household tasks Doing the dishes Doing the laundry Dishes are clean Laundry is done All tasks finished; let's go to the pub

21 1 #include <ctime> 2 #include <cstdlib> Work-stealing task scheduler 3 #include <iostream> 4 #include <tbb/task.h> 5 #include <tbb/task_group.h> 6 7 void householdtasks() { 8 std::cout << "Doing household tasks" << std::endl; 9 10 // Create group of child tasks and spawn them 11 tbb::task_group g; 12 g.run([](){ 13 std::cout << " Doing the laundry" << std::endl; 14 sleep(2); 15 std::cout << " Laundry is done" << std::endl; Much easier to work 16 }); 17 g.run([](){ 18 std::cout << " Doing the dishes" << std::endl; with! 19 sleep(1); 20 std::cout << " Dishes are clean" << std::endl; 21 }); 22 // Wait for children to finish 23 g.wait(); std::cout << "All tasks finished; let's go to the pub" << std::endl; 26 } int main(int argc, char* argv[]) { 29 // Spawn root task and wait 30 tbb::task_group g; 31 g.run_and_wait(&householdtasks); return 0; 34 }

22 Work-stealing task scheduler A tree of parent / child tasks is created recursively The scheduler tries to evaluate this tree with a balance of depth-first and breadth-first evaluation. Each thread has a double-ended queue of tasks:... which is used as a stack: New tasks spawned by a thread are pushed on its stack. Ready for task: pop task from stack (hot cache!)... which is used as a queue: When local stack is empty: steal oldest task from another thread

23 Work-stealing task scheduler The task scheduler's fundamental strategy is: "breadth-first theft and depth-first work". The breadth-first theft rule raises parallelism sufficiently to keep threads busy. The depth-first work rule keeps each thread operating efficiently once it has sufficient work to do. (hot cache...)

24 Work-stealing task scheduler This is a pretty scalable strategy: There is no global dispatch i.e.: no global task-list that would require locking Idle threads steal work from busy threads i.e.: dynamic load balancing All high level algorithms use this scheduler. However, there are cases where it makes more sense to keep work a little bit more local to the thread... Affinity!

25 Work-stealing task scheduler A partitioner decides how the iteration range of a parallel_for/reduce is split up over the threads. (default)

26 Common parallel algorithms 2 tbb::flow::graph Fully introduced in TBB 4 Build dataflow algorithms that can be parallelized automatically Well-suited for streaming data apps Connect nodes with edges See website for an example!

27 Common parallel algorithms 2 tbb::flow::graph

28 Memory allocation issues Some issues with regular memory allocation Scalability: many threads / cores can compete for memory allocation at the same time False sharing: cores working on neighbouring data can invalidate each other s caches Memory distance: non-numa-aware allocation results in non-optimal location of data vs code

29 Memory allocation issues scalable_allocator<t> 1 #include <tbb/scalable_allocator.h>... 6 // Allocator object with own memory pool 7 tbb::scalable_allocator<double> alloc; // Allocate memory for n doubles 13 double* data = alloc.allocate(n); // Free memory 39 alloc.deallocate(data, n); This template can improve the performance of programs that rapidly allocate and free memory

30 Memory allocation issues cache_aligned_allocator<t> 1 #include <tbb/cache_aligned_allocator.h>... 6 // Allocator object with own memory pool 7 tbb::cache_aligned_allocator<double> alloc; // Allocate memory for n doubles 13 double* data = alloc.allocate(n); // Free memory 39 alloc.deallocate(data, n); Two objects allocated by cache_aligned_allocator are guaranteed to not have false sharing. In other cases, the use of (thread-)local variables can help.

31 Memory allocation issues NUMA As of now TBB does not offer NUMA-aware memory allocation. Although, affinity scheduling might help in some cases! You can try to play around with the Linux tool numactl and libnuma

32 Practical remarks Timing: 1 #include <tbb/tick_count.h>... 6 tick_count t_start = tick_count::now(); 7 // Go to restroom... 8 tick_count t_stop = tick_count::now(); 9 double time_spent_in_restroom = (t_stop - t_start).seconds(); Debugging TBB programs: Define: -DTBB_USE_DEBUG=1 Link against debug versions of the library: libtbb_debug.so & libtbbmalloc_debug.so

33 Practical remarks Number of threads: 1 #include <tbb/task_scheduler_init.h> // Force task_scheduler initialization with n threads 14 tbb::task_scheduler_init t_sched(n);...

34 README! "Tutorial" is a pretty extensive document of 90 pages; easy read but VERY useful!!! Reference is the real API manual The Foundations for Scalable Multi-Core Software in Intel TBB Collection of short articles with use-cases for parallel programming challenges Not specifically for TBB