2^48 - keine Angst vor großen Datensätzen in MATLAB

Transcription

1 2^48 - keine Angst vor großen Datensätzen in MATLAB 9. July 2014 Rainer Mümmler Application Engineering Group 2014 The MathWorks, Inc. 1

2 Challenges with Large Data Sets Out of memory Running out of address space Slow processing Data too large to be efficiently managed between RAM and virtual memory Lots of data to process Gaining insight Large data visualization Modeling with no equation and lots of predictors 3

3 Agenda Available system memory Memory usage in MATLAB Techniques for processing large data sets 4

4 System Memory System Memory = RAM + Swap/Page on Disk Virtual Memory Process sees contiguous block of memory Memory actually divided between RAM and disk (swap/page file) OS maps virtual address to physical address General guidelines: Add RAM, possibly swap space If thrashing, consider alternative approaches Virtual Memory (per process) RAM Disk 5

5 Memory and Your Operating System 32-bit operating systems 4GB of addressable memory per process Part of it is reserved by the OS, leaving the application < 4GB 64-bit operating systems In theory, can address 18 Exabytes of memory Determined by OS and processor Essentially limited by the amount of RAM and disk available on the computer Memory per Process Reserved by Operating System Available for Process Use 64-bit OS, if possible 6

6 Agenda Available system memory Memory usage in MATLAB Techniques for processing large data sets 7

7 Memory Management in MATLAB Preallocate arrays Large matrices first Clear variables when no longer needed Check memory available (Windows only) >> memory Control contiguous memory with startup switch (Windows only) C:\matlab shield medium MATLAB Process Address Space x1 = zeros(25,1) x4 x2 = zeros(100,1) zeros(50,1) x3 = zeros(25,1) x2 = zeros(50,1) x4 = zeros(100,1) Allocated x1 = zeros(25,1) Allocated x3 = zeros(25,1) 8

8 Data Copies Function calls Data is copy-on-write (lazy-copy) Passed by reference into the function a not copied function y = foo(x,a,b) y = a * x + b; end a is copied function y = foo(x,a,b) a(1) = a(1) + 12; y = a * x + b; end If not modified, no copy is made If modified, a temporary copy is made 9

9 Data Copies In-Place Optimizations MATLAB performs calculations in-place when: Output variable name is the same as input variable name Performing element-wise computation y = 2*x + 3; x = 2*x + 3; not in-place in-place 10

10 Techniques for Minimizing Data Copies In-place operations, if possible Nested functions Share the workspace of all outer functions Avoids making temporary copies of input arguments For objects, consider handle classes Copy of a handle object refers to the same object as the original handle 11

11 Using Appropriate Data Storage Numerical data types Floating point for math (e.g. linear algebra) Integers where appropriate (e.g. images) Cells and structures Sparse arrays Categorical arrays 12

12 How does MATLAB store data? Container overhead* Data Data Data Fieldname (64) d Header (112) Cell Header (112) dcell Header (112) Element Header (112) dstruct Header (112) d = [1 2] dcell ={[1 2]} dstruct.d = [1 2] * Using values for 64-bit MATLAB 13

13 Sparse Matrices Require less memory and are faster When to use sparse? < 1/2 dense on 64-bit (double precision) < 2/3 dense on 32-bit (double precision) Functions that support sparse matrices >> help sparfun Blog Post: Creating Sparse Finite Element Matrices 14

14 Agenda Available system memory Memory usage in MATLAB Techniques for processing large data sets 17

15 Processing Large Data Sets Break your large data into separate pieces and process independently Partial reading and writing of files Built-in functionality for block-processing System Objects for stream processing (signals, videos) Use the whole dataset at once Single array across memory of multiple machines 18

16 Reading in Part of a Dataset from Files ASCII file Import Tool, textscan MAT file Load and save part of a variable using the matfile Binary file Read and write directly to/from file using memmapfile Maps address space to file Databases (with Database Toolbox) ODBC and JDBC-compliant (e.g. Oracle, MySQL, Microsoft, SQL Server) Database Explorer App 19

17 Summary Examples: Reading in Part of a Dataset from Files ASCII file Import Tool, textscan MAT file Load and save part of a variable using the matfile Binary file Read and write directly to/from file using memmapfile Maps address space to file Only read/write parts of datasets, and not the whole file 20

18 Block Processing Images Available from Image Processing Toolbox blockproc automatically divides an image into blocks for processing Reduces memory usage Read and write block directly from image file Processes arbitrarily large images 21

19 Batch processing Load the entire file and process it all at once Memory Source Batch Processing Algorithm Stream processing Load a frame and process it before moving on to the next frame MATLAB Memory Stream Source Stream Processing 22

20 System Objects Available from DSP System Toolbox Communications System Toolbox Computer Vision System Toolbox Phased Array System Toolbox A class of MATLAB objects that support streaming workflows Simplifies data access for streaming applications Manages flow of data from files or network Handles data indexing and buffering Contain algorithms to work with streaming data Manages algorithm state Available for Signal Processing, Communications, Video Processing, and Phased Array Applications 23

21 Processing Large Data Sets Break your large data into separate pieces and process independently Partial reading and writing of files Built-in functionality for block-processing System Objects for stream processing Use the whole dataset at once Single array across memory of multiple machines 24

22 Distributing Large Data Available from Parallel Computing Toolbox MATLAB Distributed Computing Server MATLAB Desktop (Client) Worker Worker Worker Worker Remotely Manipulate Array from Client Distributed Array Lives on the Workers 25

23 Regular MATLAB code Using Distributed Arrays 26

24 Investigation: Distributed Calculations Effect of number of computers on execution time Time (s) N 1 node, multithreaded 2 nodes, 32W Distributed 4 nodes, 64W Processor: Intel Xeon E cores, 60 GB RAM per compute node 10 Gigabit Ethernet 27

25 Sample of Other Technical Resources MATLAB documentation User s Guide Programming Fundamentals Software Development Memory Usage The Art of MATLAB, Loren Shure s blog blogs.mathworks.com/loren/ Memory Management Guides MATLAB Answers 30

26 31