Parallel Execution with OpenMP

Transcription

1 Parallel Execution with OpenMP Advanced Statistical Programming Camp Jonathan Olmsted (Q-APS) Day 4: May 30th, 2014 PM Session ASPC Parallel Execution with OpenMP Day 4 AM 1 / 28

2 Outline 1 OpenMP 2 Use and Mis-Use 3 Examples ASPC Parallel Execution with OpenMP Day 4 AM 2 / 28

3 What is OpenMP? OpenMP is an API for one of several C++ parallelization techniques. Restricted to a single machine, like our use of a socket cluster at the R level. Unlike the socket cluster, though, it will not run everywhere. Requires special compile-time instructions. Users annotate the section of their code that should be parallelized. We will only focus on parallel for loops (the simplest). ASPC Parallel Execution with OpenMP Day 4 AM 3 / 28

4 Outline 1 OpenMP 2 Use and Mis-Use 3 Examples ASPC Parallel Execution with OpenMP Day 4 AM 4 / 28

5 Header File # include <RcppArmadillo.h> // [[Rcpp::depends(RcppArmadillo)]] # include <omp.h> using namespace Rcpp ; We now have an additional dependency to build against. ASPC Parallel Execution with OpenMP Day 4 AM 5 / 28

6 Compiler/Linking Options library("rcpp") Sys.setenv("PKG_CXXFLAGS" = "-fopenmp") Sys.setenv("PKG_LIBS" = "-fopenmp") We set environmental variables so that the annotations are read and OpenMP is used. ASPC Parallel Execution with OpenMP Day 4 AM 6 / 28

7 Printing Output Sequentially // [[Rcpp::export()]] void omp1 () { for (int i = 0 ; i < 10 ; i++) { Rcout << " " << i << " " ; Rcout << std::endl ; Rcout sequentially is helpful. ASPC Parallel Execution with OpenMP Day 4 AM 7 / 28

8 sourcecpp("omp_functions.cpp") omp1() ## omp1() ## ASPC Parallel Execution with OpenMP Day 4 AM 8 / 28

9 Printing Output in Parallel // [[Rcpp::export()]] void omp2 (int t = 1) { omp_set_num_threads(t) ; # pragma omp parallel for for (int i = 0 ; i < 10 ; i++) { Rcout << " " << i << " " ; Rcout << std::endl ; Rcout in parallel is less helpful and will cause problems. Do not do this with your code! ASPC Parallel Execution with OpenMP Day 4 AM 9 / 28

10 omp2(4) ## omp2(4) ## There is no guaranteed order in which iterations are processed. ASPC Parallel Execution with OpenMP Day 4 AM 10 / 28

11 Iterations Must be Independent // [[Rcpp::export()]] NumericVector omp3 (NumericVector x, int t = 1 ) { omp_set_num_threads(t) ; NumericVector y(x.length()) ; # pragma omp parallel for for (int n = 0 ; n < x.length() ; n++) { y(n) = pow(x(n), 2); return(y) ; Typical usage where each iteration s task can depend on preliminary data, but iterations do not interact with each other. ASPC Parallel Execution with OpenMP Day 4 AM 11 / 28

12 omp3(9) ## [1] 81 vec <- rnorm(1e6) library(microbenchmark) ASPC Parallel Execution with OpenMP Day 4 AM 12 / 28

13 microbenchmark(c0 = vec ^ 2, c1 = omp3(vec, 1), c2 = omp3(vec, 2), c3 = omp3(vec, 3), c4 = omp3(vec, 4), c8 = omp3(vec, 8), c12 = omp3(vec, 12), c24 = omp3(vec, 24), c96 = omp3(vec, 96), c192 = omp3(vec, 96), times = 20 ) ## Unit: milliseconds ## expr min lq median uq max neval ## c ## c ## c ## c ## c ## c ## c ## c ## c ## c The best number of threads will be problem-specific. The sweet spot here is around 8. ASPC Parallel Execution with OpenMP Day 4 AM 13 / 28

14 RcppArmadillo and OpenMP // [[Rcpp::export()]] bool omp4 (arma::mat x, int iter, int t = 1 ) { omp_set_num_threads(t) ; # pragma omp parallel for for (int i = 0 ; i < iter ; i++) { arma::mat y = x.row(i).t() * x.row(i) ; return(true) ; As long as there is no cross-iteration dependence, you can do most operations in parallel. However, the ideal degree of parallelism always depends on the problem. ASPC Parallel Execution with OpenMP Day 4 AM 14 / 28

15 mx <- matrix(rnorm(5e2 * 5e2), nrow = 5e2) microbenchmark(omp4(mx, 96, 1), omp4(mx, 96, 2), omp4(mx, 96, 4), omp4(mx, 96, 12), omp4(mx, 96, 24), times = 10 ) ## Unit: milliseconds ## expr min lq median uq max neval ## omp4(mx, 96, 1) ## omp4(mx, 96, 2) ## omp4(mx, 96, 4) ## omp4(mx, 96, 12) ## omp4(mx, 96, 24) ASPC Parallel Execution with OpenMP Day 4 AM 15 / 28

16 Scalability Depends on Demands // [[Rcpp::export()]] bool omp5 (int d, int iter, int t = 1 ) { omp_set_num_threads(t) ; # pragma omp parallel for for (int i = 0 ; i < iter ; i++) { arma::mat x(d, d) ; x.randn() ; chol(x.t() * x) ; return(true) ; ASPC Parallel Execution with OpenMP Day 4 AM 16 / 28

17 microbenchmark(omp5(10, 20, 1), omp5(10, 20, 4), omp5(600, 20, 1), omp5(600, 20, 4), times = 20 ) ## Unit: microseconds ## expr min lq median uq ## omp5(10, 20, 1) ## omp5(10, 20, 4) ## omp5(600, 20, 1) ## omp5(600, 20, 4) ## max neval ## ## ## ## ASPC Parallel Execution with OpenMP Day 4 AM 17 / 28

18 R s RNGs are Not Reproducible in Parallel // [[Rcpp::export()]] NumericVector omp6 () { int N = 100 ; NumericVector x(n) ; omp_set_num_threads(4) ; #pragma omp parallel for for (int t = 0 ; t < N ; t++) { x(t) = R::rnorm(0, 1) ; return(x) ; Never use R s RNG functions in parallel. ASPC Parallel Execution with OpenMP Day 4 AM 18 / 28

19 set.seed(1) mean(omp6()) ## [1] set.seed(1) mean(omp6()) ## [1] ASPC Parallel Execution with OpenMP Day 4 AM 19 / 28

20 Outline 1 OpenMP 2 Use and Mis-Use 3 Examples ASPC Parallel Execution with OpenMP Day 4 AM 20 / 28

21 Parallel Pairwise Distance sourcecpp("pwdomp.cpp") dfcounties <- read.csv("counties.csv") mcoord <- cbind(dfcounties$latitude, dfcounties$longitude ) ASPC Parallel Execution with OpenMP Day 4 AM 21 / 28

22 Rcpp::NumericMatrix calcpwdomp (Rcpp::NumericMatrix x, int nthread = 1 ) { omp_set_num_threads(nthread) ; int nrows = x.nrow() ; int ncols = x.nrow() ; Rcpp::NumericMatrix out(nrows, ncols) ; double rad = ; const double pi = ; #pragma omp parallel for for(int arow = 0; arow < nrows; arow++) { for(int acol = 0; acol < ncols; acol++) { double phi1 = x(arow, 0) * pi / 180 ; double phi2 = x(acol, 0) * pi / 180 ; double lambda1 = x(arow, 1) * pi / 180 ; double lambda2 = x(acol, 1) * pi / 180; double q1 = 2 * rad ; double q2 = pow(sin((phi1 - phi2) / 2), 2) ; double q3 = pow(sin((lambda1 - lambda2) / 2), 2) ; double q4 = cos(phi1) * cos(phi2) ; out(arow, acol) = q1 * asin(sqrt(q2 + q4 * q3)) ; return(out) ; ASPC Parallel Execution with OpenMP Day 4 AM 22 / 28

23 dftimes <- microbenchmark(n1 = calcpwdomp(mcoord, 1), n2 = calcpwdomp(mcoord, 2), n3 = calcpwdomp(mcoord, 3), n4 = calcpwdomp(mcoord, 4), n6 = calcpwdomp(mcoord, 6), n8 = calcpwdomp(mcoord, 8), n12 = calcpwdomp(mcoord, 12), times = 10 ) ASPC Parallel Execution with OpenMP Day 4 AM 23 / 28

24 ASPC Parallel Execution with OpenMP Day 4 AM 24 / 28

25 Instead of #pragma omp parallel for for(int arow = 0; arow < nrows; arow++) { for(int acol = 0; acol < ncols; acol++) { create a function with for(int arow = 0; arow < nrows; arow++) { #pragma omp parallel for for(int acol = 0; acol < ncols; acol++) { Time them. Which is faster? ASPC Parallel Execution with OpenMP Day 4 AM 25 / 28

26 Parallel E-Step in EM Probit sourcecpp("em_probit_omp.cpp") library(zelig) data(turnout) fit0 <- glm(vote ~ income + educate + age, data = turnout, family = binomial(link = "probit") ) y <- matrix(turnout$vote) X <- model.matrix(fit0) ASPC Parallel Execution with OpenMP Day 4 AM 26 / 28

27 List em_probitomp (arma::mat y, arma::mat X, int maxit = 10, int nthr = 1 ) { omp_set_num_threads(nthr) ; int N = y.n_rows ; int K = X.n_cols ; arma::mat beta(k, 1) ; beta.fill(0.0) ; arma::mat eystar(n, 1) ; eystar.fill(0) ; for (int it = 0 ; it < maxit ; it++) { arma::mat mu = X * beta ; #pragma omp parallel for for (int n = 0 ; n < N ; n++) { if (y(n, 0) == 1) { eystar(n, 0) = mu(n, 0) + f(mu(n, 0)) ; if (y(n, 0) == 0) { eystar(n, 0) = mu(n, 0) - g(mu(n, 0)) ; beta = (X.t() * X).i() * X.t() * eystar ; List ret ; ret["n"] = N ; ret["k"] = K ; ret["beta"] = beta ; return(ret) ; ASPC Parallel Execution with OpenMP Day 4 AM 27 / 28

28 microbenchmark( a = {em_probitomp(y = y,x = X, maxit = 100,nthr = 1), b = {em_probitomp(y = y,x = X, maxit = 100,nthr = 4), c = {em_probitomp(y = y,x = X, maxit = 100,nthr = 16), times = 20 ) ## Unit: milliseconds ## expr min lq median uq max neval ## a ## b ## c ASPC Parallel Execution with OpenMP Day 4 AM 28 / 28