Intel Xeon Phi Coprocessor

Transcription

1 Architecture Advanced Workshop Memory Session Speaking: Shannon Cepeda Intel,, Cilk,, Pentium, VTune and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries 1

2 Objective This module will: Review the explicit and implicit memory models Demonstrate advanced usages of these models, including asynchronous offload and buffering Recommend performance best practices for the memory hierarchy 2

3 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 3

4 Storage Basics Per-core caches reference info: Type Size Ways Set conflict Location L1I (instr) 32KB 4 8KB on-core L1D (data) 32KB 8 4KB on-core L2 (unified) 512KB 8 64KB connected via core/ring interface Memory: 8 memory controllers, each supporting 2 32-bit channels GDDR5 channels theoretical peak of 5.5GT/s Practical peak BW limited by ring and other factors 4

5 More Storage Basics Per-core TLBs reference info: Type Entries Page Size Coverage L1 Instruction 32 4KB 128KB L1 Data 64 4KB 256KB 32 64KB 2MB 8 2MB 16MB L2 64 4KB, 64KB, or 2MB Up to 128MB Note: Operating system support for 64K pages may not yet be available 5

6 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 6

7 Quick Review of Explicit Memory Model pa Host 2 Copy over 4 Copy back Intel Xeon Phi Coprocessor 1 Allocate 3 5 Read/Modify Free You (the programmer) explicitly control data and function movement between the Host and Target(s) Data are copied (not shared) Must be bitwise copy-able (pointers NOT relocated) Supported for Fortran, C, and C++ 7

8 Data persistence between offloads is managed by user code Global data marked attribute ((target(mic))) Will exist on both the host and target systems Copied between host and target when referenced Name targets target(mic) runtime picks the card target(mic:2) explicitly name the logical card number Use in/out/inout/nocopy clauses to specify data transfer and direction Use alloc_if(0/1), free_if(0/1) to manage memory on the implicitly- or explicitly named device 8

9 Data persistence in the real world requires a stable anchor point (1 of 3) Allocate persisted data on heap Create structure to track persistent data, pass it as a parameter between functions struct pcache { int *p1; }; #define ASIZE 128 main(int argc, char **argv) { struct pcache *share = start_job(); continue_job(share);... } Declaration of struct with link to dynamic data Struct with pointer passed between functions 9

10 Data persistence in the real world requires a stable anchor point (2 of 3) struct pcache * Dynamically-allocated start_job() pointer structure... { struct pcache *mycache = (struct pcache *)malloc(sizeof(pcache)); int *A = mycache->p1 = (int *) malloc(asize * sizeof(int)); for (i=0;i < ASIZE; ++i) { A[i] = i; } #pragma offload target(mic:0) in (A: length(128) free_if(0)) { for (i=0;i < ASIZE; ++i) { A[i] += A[i] + 1; } } return mycache; }...and the dynamic array it points to 10

11 Data persistence in the real world requires a stable anchor point (3 of 3) void continue_job(struct pcache *mine) { int i; } Dereference pointer to dynamic data in struct that was passed in int *A = mine->p1; #pragma offload target(mic:0) in (A: length(0) alloc_if(0) free_if(0)) { for (i=0;i < ASIZE; ++i) { A[i] += A[i] + 1; } } 11

12 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 12

13 Quick Review: Implicit Memory Model Same address range C/C++ executable Offload code Host Memory Host Intel Xeon Phi Coprocessor Xeon Phi Coprocessor Memory User code declares data objects to be shared Data allocated at same address on host and target Modified data is copied at synchronization points Allows sharing of complex data structures No data marshaling necessary Supported only for C and C++ Can t propagate exceptions from to CPU 13

14 Asynchronous Offload Example: Finding the area of an annular ring Use Monte Carlo integration (random dots, in or out) Find the areas of the outer and inner rings and subtract Write one area function as shared and run concurrently Find area 2 on the coprocessor and area 1 on the host Outer radius r2 Inner radius r1 14

15 Asynchronous Offload FindArea function _Cilk_shared float FindArea(float r) { float x, y, area; unsigned int seed=(unsigned int) cilkrts_get_worker_number()+clock(); unsigned int seed2=(unsigned int) cilkrts_get_worker_number()+clock()+2; cilk::reducer_opadd<int> inside(0); cilk_for (int i=0;i<20000;i++) { x = (float) rand_r(&seed)/rand_max; y = (float) rand_r(&seed2)/rand_max; x = 2.0 * x - 1.0; y = 2.0 * y - 1.0; if (x * x + y * y < r * r) } { } inside++; area=4.0*inside.get_value()/ ; return area; } Function is implicitly shared: Instances created for each of target and host Private variables for each function instance Intel Cilk Plus reducer and cilk_for available in both instances No _Cilk_shared variables in this example 15

16 Asynchronous Offload - Main int main() { // Get input, do error checking AreaLg = _Cilk_spawn _Cilk_offload FindArea(r2); Launch offload to find area for ring with radius r2 AreaSm = FindArea(r1); While the target execution proceeds, use the host to find area for ring with radius r1 cilk_sync; Wait for offloaded area function to complete float MontyArea = AreaLg - AreaSm; float pi = ; float RefArea= pi * (r2 * r2 - r1 * r1); float Accuracy = 100 * (1- fabs(montyarea-refarea)/refarea); printf("area 1=%lf, Area 2=%lf \n", AreaLg, AreaSm); printf("donut Area =%lf, Accuracy = %lf %\n", MontyArea, Accuracy); } 16

17 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 17

18 Double Buffering Example Overlap computation and communication Generalizes to data domain decomposition Host Target Pre-work data block Iteration 0 data block data block process Iteration 1 data block data block process Iteration n data block data block process Iteration n+1 data block data block process Last Iteration data block process 18

19 Double Buffering Example - Main int main(int argc, char* argv[]) { int i; double st_time, end_time; double sync_tm, async_in_tm; // Allocate & initialize in1, res1, // in2, res2 on the host Allocate arrays on target: alloc_if(1) Retain for duration of sample: free_if(0) in1 and in2 represent 2 separate buffers. #pragma offload_transfer target(mic:0) in(cnt) \ nocopy(in1, res1, in2, res2 : length(cnt) \ alloc_if(1) free_if(0) ) do_async_in(); // Validate results and print timings Free target allocations: free_if(1) #pragma offload_transfer target(mic:0) \ nocopy(in1, res1, in2, res2 : length(cnt) \ alloc_if(0) free_if(1) ) 19

20 Double Buffering do_asynch_in, evens void do_async_in() { float lsum; int i; lsum = 0.0f; Begin an initial transfer of the first dataset for the target to work on. Transfer begins immediately, is non-blocking, and will signal when complete. #pragma offload_transfer target(mic:0) \ in(in1 : length(cnt) alloc_if(0) free_if(0) ) signal(in1) for (i=0; i < iter; i++) { if (i%2 == 0) { For even loop iterations (except the final), first start another non-blocking transfer of the next dataset. #pragma offload_transfer target(mic:0) if(i!=iter-1) \ in(in2 : length(cnt) alloc_if(0) free_if(0) ) \ signal(in2) #pragma offload target(mic:0) nocopy(in1) wait(in1) \ out(res1 : length(cnt) alloc_if(0) free_if(0) ) { compute(in1, res1); } lsum = lsum + sum_array(res1); } While that transfer progresses, process the previous dataset through the compute() function, first waiting for its transfer to complete, then return a result. Execution on the host waits for the function to return. 20

21 Double Buffering do_asynch_in, odds else { #pragma offload_transfer target(mic:0) if(i!=iter-1) \ in(in1 : length(cnt) alloc_if(0) free_if(0) ) \ signal(in1) For odd iterations (except the final), work on the other buffer. Start another nonblocking transfer. #pragma offload target(mic:0) nocopy(in2) wait(in2) \ out(res2 : length(cnt) alloc_if(0) free_if(0) ) { compute13(in2, res2); } lsum = lsum + sum_array(res2); } } async_in_sum = lsum / (float) iter; } Offload the compute function, and host waits for the result. Repeat, alternating between the buffers (in1 & in2). 21

22 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 22

23 Paging Use of an L2 TLB provides significantly more memory coverage Consider using large pages if: Your code is accessing >=16MB memory range Your code features semi-random data accesses or many different streams You have a large, heavily accessed data structure On native apps, use mmap to get large pages for dynamically allocated data For offloaded data, set the environment variable MIC_USE_2MB_BUFFERS=size The hugetlbfs library can also be used without modifying code 23

24 Standard Paging Technique with mmap Reserve 2MB pages in the kernel: Remember this leaves less memory available for other programs! Example (reserves 128 2MB pages): echo 128 > /proc/sys/vm/nr_hugepages Use mmap with flags to request dynamic allocation of 2MB pages: #include <sys/mman.h> void *p; size_t s = 64*2*1024*1024; // 128MB p = mmap(0, s, PROT_READ PROT_WRITE, MAP_ANONYMOUS MAP_PRIVATE MAP_HUGETLB, -1, 0); if (p == MAP_FAILED) perror( mmap failed ); 24

25 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 25

26 Effective blocking: understand data reuse within the recurrence of the kernel Look at this 7-point stencil code for (i = 0; i < niter; i++) { for (z = 0; z < nz; z++) for (y = 0; y < ny; y++) for (x = 0; x < nx; x++) f2[z,y,x] = cc*f1[z,y,x] + cw*f1[z,y,x-1] + ce*f1[z,y,x+1] + cn*f1[z,y-1,x] + cs*f1[z,y+1,x] + cb*f1[z-1,y,x] + ct*f1[z+1,y,x] temp = f2; f2 = f1; f1 = temp; }...and at the recurrence of data use (intermediate optimization steps will be covered in the lab) Used with permission and modified from original sources: View original license at Note modified sources above are provided under the license given on slide

27 Z,Y 1,4 1,5 1,6 1,7 1,8 1,9 1,10 1,11 Examination of the recurrence of data use reveals opportunities to exploit cache locality 2,4 2,5 2,6 2,7 2,8 2,9 2,10 2,11 3,4 3,5 3,6 3,7 3,8 X f2[z=2,y=5,:] =... f2[z=2,y=6,:] =... f2[z=2,y=7,:] =... f2[z=3,y=5,:] =... f2[z=3,y=6,:] =... f2[z=3,y=7,:] =... NX=256, the array looks like this Each box represents a cache line in X The X values have spatial locality The Ys on each Z-plane are adjacent o Not blocking in X means prefetch spillover starts next Y row for free Iterations reuse 3 rows of X in next iterations of Y and Z Tiling in Y can preserve cache lines long enough for reuse in Z 27

28 Tiling in Y helps data locality #define YBF 13 #pragma omp for collapse(2) for (int yy = 0; yy < ny; yy += YBF) { //block loop for (int z = 0; z < nz; z++) { int ymax = (yy+ybf < ny)? (yy+ybf) : ny; for (int y = yy; y < ymax; y++) { // tile loop Interchanging loops sets up the Y-axis for tiling Used with permission and modified from original sources: View original license at Note modified sources above are provided under the license given on slide

29 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 29

30 Prefetching Defaults includes hardware prefetchers Hardware and software prefetching can be used together Currently by default with O2 and above the Intel Compiler generates: Maximal loop prefetches (no bounds checks) o 1 prefetch to L2 followed by another prefetch to L1 No straight-line prefetches 30

31 Some Prefetching Strategies Use different first-level (vprefetch1) and secondlevel prefetch (vprefetch0) distances to fine-tune your application performance -opt-prefetch-distance=n1[,n2] Useful values to try for n1: 0,4,8,16,32,64 Useful values to try for n2: 0,1,2,4,8 Can also use prefetch pragmas to do this on a perloop basis 31

32 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 32

33 Best practice: Use 4K Memory Alignment Unaligned memory can significantly affect performance and bandwidth for offloading For data that will be offloaded, align memory allocations on host to 4K if possible Static memory o Allocated by compiler/linker o Add attribute ((aligned(n))) in front of variable declaration o Applies to global/local static variables as well as stack/auto variables Dynamic memory o Allocated by language runtime o Use mm_aligned_malloc(size, alignment_bytes) o Example: buf = (char*) _mm_malloc(bufsize, 4096); o Pair it with mm_aligned_free() 33

34 Outline Memory Basics Data Access Semantics Explicit Memory Model Implicit Memory Model Performance Tuning Overlapping communication and computation Page management Data locality Prefetching Alignment Stride 1 34

35 Avoid set conflicts Reference Level 1 cache holds 8 instances of 64B cache lines that are a multiple of 32KB/8=4KB away Level 2 cache holds 8 instances of 64B cache lines that are a multiple of 512KB/8=64KB away Because LRU is imperfect, it may act like it has <8 sets Recipe for identifying a bottleneck High cache miss rate, even though working set < cache capacity References or (array dimensions*element size) are a multiple of 4K (L1) or 64K (L2) apart Remedy Pad array dimensions 35

36 Best practice: Avoid Scatter / Gather Strided or indirect accesses can result in scatter/gather (vscatter/vgather) instructions Example from sparse matrix vector multiply implementation, for a matrix stored in Compressed Sparse Row format: for (int i = 0; i < nrows; i++ ) { } y[i] = 0.0; for (int j = row_ptr[i]; j < row_ptr[i+1]; j++) { } y[i] += val[j]*x[col_ind[j]]; Gathers can align data for vectorization, but with their paired scatters can consume many cycles Avoid if possible One strategy: Convert ArraysOfStructures -> StructuresOfArrays 36

37 Summary Use asynchronous data transfer and double buffering offloads to overlap communications with computation Optimizing memory use on the Intel Xeon Phi architecture target relies on understanding access patterns Many old tricks still apply: peeling, collapsing, unrolling and vectorization can all benefit performance 37

38 Resources Intel Xeon Phi Architecture Developer site: Intel Compiler methodology for Intel Xeon Phi Architecture: 38

39 39

40 Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPETY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Sandy Bridge and other code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to Intel, Core, Xeon, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright 2012 Intel Corporation. Hyper-Threading Technology: Requires an Intel HT Technology enabled system, check with your PC manufacturer. Performance will vary depending on the specific hardware and software used. Not available on all Intel Core processors. For more information including details on which processors support HT Technology, visit Intel 64 architecture: Requires a system with a 64-bit enabled processor, chipset, BIOS and software. Performance will vary depending on the specific hardware and software you use. Consult your PC manufacturer for more information. For more information, visit Intel Turbo Boost Technology: Requires a system with Intel Turbo Boost Technology capability. Consult your PC manufacturer. Performance varies depending on hardware, software and system configuration. For more information, visit 40

41 Source banner and attribution for modified Physis sources used through pages Physis original sources are available here: Copyright (c) <2012>, Intel Corporation All Rights Reserved. The source code, information and material ("Material") contained herein is owned by Intel Corporation or its suppliers or licensors, and title to such Material remains with Intel Corporation or its suppliers or licensors. The Material contains proprietary information of Intel or its suppliers and licensors. The Material is protected by worldwide copyright laws and treaty provisions. No part of the Material may be used, copied, reproduced, modified, published, uploaded, posted, transmitted, distributed or disclosed in any way without Intel's prior express written permission. No license under any patent, copyright or other intellectual property rights in the Material is granted to or conferred upon you, either expressly, by implication, inducement, estoppel or otherwise. Any license under such intellectual property rights must be express and approved by Intel in writing. Copyright (c) , Naoya Maruyama All rights reserved. * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of RIKEN AICS nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Unless otherwise agreed by Intel in writing, you may not remove or alter this notice or any other notice embedded in Materials by Intel or Intel s suppliers or licensors in any way. 41

42 Intel compilers, associated libraries and associated development tools may include or utilize options that optimize for instruction sets that are available in both Intel and non-intel microprocessors (for example SIMD instruction sets), but do not optimize equally for non-intel microprocessors. In addition, certain compiler options for Intel compilers, including some that are not specific to Intel micro-architecture, are reserved for Intel microprocessors. For a detailed description of Intel compiler options, including the instruction sets and specific microprocessors they implicate, please refer to the Intel Compiler User and Reference Guides under Compiler Options." Many library routines that are part of Intel compiler products are more highly optimized for Intel microprocessors than for other microprocessors. While the compilers and libraries in Intel compiler products offer optimizations for both Intel and Intelcompatible microprocessors, depending on the options you select, your code and other factors, you likely will get extra performance on Intel microprocessors. Intel compilers, associated libraries and associated development tools may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include Intel Streaming SIMD Extensions 2 (Intel SSE2), Intel Streaming SIMD Extensions 3 (Intel SSE3), and Supplemental Streaming SIMD Extensions 3 (Intel SSSE3) instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. While Intel believes our compilers and libraries are excellent choices to assist in obtaining the best performance on Intel and non-intel microprocessors, Intel recommends that you evaluate other compilers and libraries to determine which best meet your requirements. We hope to win your business by striving to offer the best performance of any compiler or library; please let us know if you find we do not. revision #