6 BLAS (Basic Linear Algebra Subroutines)

Transcription

1 161 BLAS 6.1 Motivation 6 BLAS (Basic Linear Algebra Subroutines) 6.1 Motivation How to optimise programs that use a lot of linear algebra operations? Efficiency depends on but also on: processor speed number of arithmetic operations the speed of memory references Hierarchical memory structure:

2 162 BLAS 6.1 Motivation fast, small, expensive Registers cache RAM HardDisk slow, large, cheap Tape, CS, DVD, Network storage Where Useful are arithmetic operations onlyperformed the topon of thepicture? hierarchy before operations: what data needs is datatomovement be moved up; direction? after operations the data is moved down the hierarchy information movement faster on top What As a rule: is faster, speed speed of arithmetic of arithmetic operations operations faster orthan data data movement?

3 163 BLAS 6.1 Motivation Consider an arbitrary algorithm. Denote: f flops (# arithmetic operations: +, -, /) m # memory references Introduce q = f /m. Why this number is important? t f time spent on 1 flop t m time for memory access, Then calculation time is: f t f + m t m = f t f (1 + 1 q (t m t f )) In general, t m t f and therefore, total time is reflecting processor speed only if q is large or small?

4 164 BLAS 6.1 Motivation Example. Gauss elimination method for each i key operations: A(i + 1 : n,i) = A(i + 1 : n,i)/a(i,i), (12) A(i + 1 : n,i + 1 : n) = A(i + 1 : n,i + 1 : n) A(i + 1 : n,i) A(i,i + 1 : n). (13) Operation (12) represents following general operation: y = ax + y, x,y R n, a R (14) Operation (14) called saxpy (sum of a times x plus y) (13) represents: A = A vw T, A R n n, v,w R n (15) (15) matrix A rank-1 update, (Matrix vw T has rank 1, because but Why? each row of it is a multiple of vector w)

5 165 BLAS 6.1 Motivation Operation (14) analysis: Operation (15) analysis: m = 3n + 1 memory references: 2n + 1 reads * vectors x, y * scalar a n writes * new y m = n 2 + 2n memory references: n 2 + 2n reads n 2 writes Computations f = 2n 2 flops = q = 1+O(1/n) 1 with large n Computations take f = 2n flops = q = 2/3 + O(1/n) 2/3 for large n (14) 1st order operation (O(n) flops); (15) 2nd order operation (O(n 2 ) flops). Note that coefficient q is O(1) in both cases

6 166 BLAS 6.1 Motivation Faster results in case of 3rd order operations (O(n 3 ) operations with O(n 2 ) memory references). For example, matrix multiplication: C = AB +C, A, B, C R n n. (16) Here m = 4n 2 and f = n 2 (2n 1) + n 2 = 2n 3 (check it!) = q = n/2 if n. This operation can give processor work near peak performance, with good algorithm scheduling!

7 167 BLAS 6.2 BLAS implementations 6.2 BLAS implementations BLAS standard library for simple 1st, 2nd and 3rd order operations BLAS freeware, available for example from netlib ( netlib.org/blas/) Processor vendors often supply their own implementation BLAS ATLAS implementation ATLAS ( sourceforge.net/) self-optimising code Example of using BLAS (fortran90): LU factorisation using BLAS3 operations ( SC/konspekt/Naited/lu1blas3.f90.html) main program for testing different BLAS levels ( ~eero/sc/konspekt/naited/testblas3.f90.html)

8 168 Optimisation 6.2 BLAS implementations 7 Optimisation methods Minimising (or maximising) a (possibly multivariate) function F, F: A R, where A - subset of the Eucledian space R n Objective function F called differently in different cases: loss function cost function (minimization) indirect utility function (minimization) utility function (maximization) an energy function energy functional

9 169 Optimisation 6.2 BLAS implementations Classification number of independent controls (dimensionality) one variable several variables (two to ten) dozens hundreds tens of thousands an important special class discrete optimiziation large number of controls of boolean or integer values requires combinatorial methods calculus of variations infinitely many controls

10 170 Optimisation 6.2 BLAS implementations cost of evaluation of the function (realistic goals and methods are different) from milliseconds of computer time too costly measurements in natural experiments sharpness of the maximum The presence of constraints and their character Stochastic optimization - noise in measurements or uncertainty of some factors Structural optimization Multi-objective optimization (e.g. Light and strong) multi-modal optimization (many good solutions)

11 171 Optimisation 6.2 BLAS implementations How much knowledge there is about F? not much a lot of knowledge Type of x Discrete Combinatorial search Algorithmic Brute-force Stepwise MCMC Population-based... Continous Numerical methods Gradient Descent, (Quasi-)Newton, population-based Analytic... Here short insight only into the unconstrained optimization of problems with a few independent controls

12 172 Optimisation 7.1 Gradient descent methods 7.1 Gradient descent methods First order optimization algorithm finding local minimum special case of steepest descent method if F(x) - differentiable at x multivariate function = vector F(x) directs towards smaller values (at least locally) = for small enough α: F(x α F(x)) F(x) Method: make a sequence x 0, x 1,x 2,... such that x n+1 = x n α n F(x n )

13 173 Optimisation 7.1 Gradient descent methods Algorithm 1. Compute the new direction x n = F(x n ) 2. Choose the stepsize α n according to line search or backtracking 3. Update x n+1 = x n + α n x n

14 174 Optimisation 7.1 Gradient descent methods Exact line search Minimise F along the ray x + α x α 0 : α = argmin s 0 F(x + s x) used when the cost of the minimization problem low compared to the cost of computing the search direction itself In some special cases the minimizer can be found analytically and in others it can be computed efficiently

15 175 Optimisation 7.1 Gradient descent methods Backtracking line search The minimiser α chosen approximately to minimise F along the ray {x + α x α 0} or even reduce F enough given a descent direction x for F at x domf, β (0,0.5), γ (0,1). α := 1 while F(x + α x) > F(x) + βα F(x) T x, α := γα