Reliable Adaptive Algorithms for Integration, Interpolation and Optimization

Size: px

Start display at page:

Download "Reliable Adaptive Algorithms for Integration, Interpolation and Optimization"

Martin Clifton Waters
5 years ago
Views:

Reliable Adaptive Algorithms for Integration, Interpolation and Optimization Fred J. Hickernell Department of Applied Mathematics, Illinois Institute of Technology hickernell@iit.

1 Reliable Adaptive Algorithms for Integration, Interpolation and Optimization Fred J. Hickernell Department of Applied Mathematics, Illinois Institute of Technology mypages.iit.edu/~hickernell Joint work with Sou-Cheng Choi (NORC & IIT) Yuhan Ding (IIT) Lan Jiang (IIT) Lluís Antoni Jiménez Rugama (IIT) Art Owen (Stanford) Xin Tong (UIC) Yizhi Zhang (IIT) Xuan Zhou (J. P. Morgan) Supported by NSF-DMS-5392, Thank you for your kind invitation to visit! U Illinois Chicago, October 5, 25 /8

2 Why We Want Adaptive Algorithms % An easy function to integrate... numerically f sin(x.^2); tic, I = integral(f,,), toc I = Elapsed time is.53 seconds. f(x) x % A hard function to integrte... numerically f2 2*sin((2*x).^2); tic, I2 = integral(f2,,), toc I2 = Elapsed time is.6246 seconds. f2(x) x Adaptive algorithms exert enough computational effort to get the correct answer, but not much more effort than necessary. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

3 Why Adaptive Algorithms Fail f fluky integral data - f spiky chebfun data -.5 f cheb fminbnd data fminbnd location of minimum approximating f spiky with Chebfun minimizing f spiky with fminbnd ż f fluky (x) dx = integral(ffluky,,)= meanmc_clt(y,.,...) aims to estimate E(Y) with an absolute error of. with 99% confidence. The answer is wrong half the time. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

4 Why Adaptive Algorithms Fail f spiky chebfun data -.5 f cheb fminbnd data fminbnd location of minimum approximating f spiky with Chebfun minimizing f spiky with fminbnd Chebfun (Hale et al., 25) and fminbnd stop sampling when they think that they have enough information, but miss the spike Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

5 Why Adaptive Algorithms Fail.5 f fluky integral data ż f fluky (x) dx = integral(ffluky,,)= integral approximates its error by the difference of two different quadratures, which might coincidentally coincide Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

6 Why Adaptive Algorithms Fail meanmc_clt(y,.,...) aims to estimate E(Y) with an absolute error of. with 99% confidence. The answer is wrong half the time. Insufficient () samples are taken to estimate var(y) because Y has a heavy-tailed distribution (large Probability density function for Y -5 - kurtosis) y Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

7 Why Adaptive Algorithms Fail Adaptive algorithms fail due to one or both of the following: We do not use enough samples to start. Our error bounds are faulty. Fred J. Hickernell, Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

8 Why Adaptive Algorithms Fail Adaptive algorithms fail due to one or both of the following: We do not use enough samples to start. How many samples is enough? Our error bounds are faulty. What are the alternatives? Fred J. Hickernell, Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

9 Making the Trapezoidal Rule Adaptive The trapezoidal rule for approximating ż f (x) dx is T n (f ) = [f () + 2f (/n) + + 2f ( /n) + f ()]. 2n.5 The quadrature error is ż f (x) dx T n (f ) ď Var(f ) 8n 2. Without knowing Var(f ), what is n that makes this error ď ε a? f(x).5 T n (f) =.86 int. = x Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 4/8

10 Making the Trapezoidal Rule Adaptive The quadrature error is ż f (x) dx T n (f ) ď Var(f ) 8n 2 for n =? without knowing Var(f ). want ď ε a, (ERR) f(x).5.5 T n(f) =.86 int. =.84 We assume that f is not too spiky, i.e., Var(f ) ď τ f f () + f (), (CONE) x and note that f f () + f () ď nÿ f () f () f (i/n) f ((i )/n) n + Var(f ) 2n i= This implies that (ERR) is satisfied for f satisfying (CONE), and n satisfying τ nÿ f () f () 4n(2n τ) f (i/n) f ((i )/n) n ď ε a i= Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 5/8

11 integral_g in GAIL This rigorously justified, data-based error bound is employed in the adaptive algorithm integral_g, part of the Guaranteed Automatic Integration Library (GAIL) (Choi et al., 23 25)..5 f fluky integral data Clancy et al. (24) contains details and shows that the computational cost of integral_g is ) O (bτ Var(f )/ε a operations. This cost is asymptotically optimal among all possible algorithms. H. et al. (25+) remove the factor of τ ż f fluky (x) dx = integral(ffluky,,)= integral_g(ffluky,,)= Don t approximate error by the difference of two quadratures (Lyness, 983). Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 6/8

12 General Recipe for Justified Adaptive Algorithms Error bound ż Trapezoidal Rule f (x) dx T n (f ) ď Var(f ) 8n 2 Function Approximation, Optimization, etc. sol(f ) appxn (f ) c f ď convex sol and appx n n p Cone condition Var(f ) ď τ f f () + f () f ď τ f For f «A(f ) f f () + f () ď A(f ) f () + f () + Var(f ) 2n f ď A(f ) + c 2 f n q so f ď τ A(f ) + c 2τ f n τ A(f ) so f ď c 2 τ/n q Databased error bound τ A(f ) f () + f () 4n(2n τ) c τ A(f ) n p ( c 2 τ/n q ) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 7/8

13 funappx_g and funmin_g in GAIL -.5 GAIL includes rigorously justified, adaptive algorithms for function approximation and function minimization (Choi et al., 23 25). - f spiky f -.5 cheb f GAIL fminbnd location of minimum funmin g location of minimum The error bounds employed ensure that enough samples are taken. Spiky functions are handled well provided that τ is large enough. See Clancy et al. (24) and Tong (24) for details. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 8/8

14 Making Monte Carlo Adaptive Monte Carlo algorithms approximate µ = E(Y) = average: ^µ n = n nÿ Y i, i= Y, Y 2,... IID Y when the Y i can be generated, but the integral is too hard. The Central Limit Theorem (CLT) says that [ P µ ^µ n ď 2.58^σ ]? Ç 99%, lomon n but this is not rigorous. ε a Probability density function for Y ż y ϱ(y) dy by a sample y Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 9/8

15 Making Monte Carlo Adaptive µ = E(Y) «^µ n = n nÿ Y i, i= Y, Y 2,... IID Y To obtain a guaranteed error bound we need to assume a bound on the kurtosis of Y: kurt(y) := E[(Y µ) 4 ]/ var(y) 2 ď κ max. (CONE) Cantelli s inequality implies a bound on the true variance in terms of the sample variance: P[C 2^σ 2 ě var(y)] ě 99.5%, where ^σ 2 = n σ n σ ÿ i= (Y i ^µ nσ ) 2, where C ą and n σ depend on κ max. The Berry-Esseen Inequality (a finite sample version of the CLT) determines the sample size, n, needed for the sample mean: Φ (?nε a /(C^σ) ) loooooooooomoooooooooon + n (? loooooooooooomoooooooooooon nε a /(C^σ), κ max ) ď.25. CLT part Berry-Esseen extra part Compute the sample mean, ^µ n, of an independent sample of size n. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 /8

16 meanmc_g in GAIL Probability density function for Y y meanmc_clt(y,...) fails half the time meanmc_g(y,...) succeeds 99% of the time GAIL includes rigorously justified, algorithm meanmc_g for computing µ = E(Y) to the desired error tolerance (Choi et al., 23 25; H. et al., 24). Heavy-tailed random variables are handled well provided that κ max is large enough. One may use meanmc_g to evaluate ż ż g(x) dx = f (x) ϱ(x)dx = E[f (X)], R d R d where X ϱ for some probability density function ϱ. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 /8

17 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

18 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

19 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

20 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

21 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

22 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

23 Making Quasi-Monte Carlo Adaptive ż Quasi-Monte Carlo (QMC) algorithms approximate µ = f (x) dx by a [,] d sample average of correlated, evenly place points. The evenness improves the convergence to the answer (Dick et al., 24) Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 2/8

24 cubsobol_g and cublattice_g in GAIL Data-driven error bounds for quasi-monte Carlo are obtained via discrete Fourier transforms of the function values (H. and Jiménez Rugama, 25+; Jiménez Rugama and H., 25+): Fourier Walsh for Sobol sampling, and Fourier sine/cosine for lattice sampling. Substantial speedups can by using QMC sampling rather than IID sampling. % Option price class object % based on user defined input MCopt = optprice(inp); % meanmc_g to compute the price tic MCPrice = genoptprice(mcopt) toc % Results MCgenoptPrice =.2 Elapsed time is seconds. % Now use Sobol' sampling... with a PCA construction % After changing the... parameters in inp QMCopt = optprice(inp) tic QMCPrice = genoptprice(qmcopt) toc % Results QMCgenoptPrice =.2 Elapsed time is.8 seconds. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 3/8

Relative errors can be handled. More problems are waiting to be tackled: algorithms with higher order convergence, multi-level Monte Carlo,.

25 Summary Adaptive algorithms are valuable because they determine the computational effort required based on the problem difficulty. Most adaptive algorithms are flawed. By moving from error analysis based on of input (functions, random variables, etc.) to, we can justify adaptive algorithms. Relative errors can be handled. More problems are waiting to be tackled: algorithms with higher order convergence, multi-level Monte Carlo,... Applied and computational mathematicians should learn to write code well, not only prove theorems about their algorithms. Fred J. Hickernell, Reliable Adaptive Algorithms U Illinois Chicago, /5/5 4/8

26 Raising the Bar for Numerical Algorithms Implemented Widely available in a common language Has a convenient user interface Tested for bugs Coded efficiently Stopping criterion is data-based Provides an answer within a user-specified error tolerance Runs in a reasonable amount of time on available hardware All inputs and parameters can be specified explicitly Can be written in pseudo-code Implementable Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 5/8

27 Raising the Bar for Numerical Algorithms Justified Justifiable The stopping criterion to reach a user-specified error tolerance is guaranteed to succeed This rate of convergence is asymptotically optimal with respect to all possible algorithms The rate of convergence as the computational effort tends to infinity is known The error can be bounded in terms of information about the inputs and parameters Has a theoretical basis Provides a reasonable answer for a wide range of problems Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 6/8

28 Thank you Fred J. Hickernell, Reliable Adaptive Algorithms U Illinois Chicago, /5/5 7/8

29 References Choi, S.-C. T., Y. Ding, F. J. H., L. Jiang, Ll. A. Jiménez Rugama, X. Tong, Y. Zhang, and X. Zhou GAIL: Guaranteed Automatic Integration Library (versions. 2.). Clancy, N., Y. Ding, C. Hamilton, F. J. H., and Y. Zhang. 24. The cost of deterministic, adaptive, automatic algorithms: Cones, not balls, J. Complexity 3, Cools, R. and D. Nuyens (eds.) 25+. Monte Carlo and quasi-monte Carlo methods 24, Springer-Verlag, Berlin. Dick, J., F. Kuo, and I. H. Sloan. 24. High dimensional integration the Quasi-Monte Carlo way, Acta Numer., 53. Hale, N., L. N. Trefethen, and T. A. Driscoll. 25. Chebfun version 5.2. H., F. J., L. Jiang, Y. Liu, and A. B. Owen. 24. Guaranteed conservative fixed width confidence intervals via Monte Carlo sampling, Monte Carlo and quasi-monte Carlo methods 22, pp H., F. J. and Ll. A. Jiménez Rugama Reliable adaptive cubature using digital sequences, Monte Carlo and quasi-monte Carlo methods 24. to appear, arxiv:4.865 [math.na]. H., F. J., M. Razo, and S. Yun Reliable adaptive numerical integration. submitted for publication. Jiménez Rugama, Ll. A. and F. J. H Adaptive multidimensional integration based on rank- lattices, Monte Carlo and quasi-monte Carlo methods 24. to appear, arxiv: Lyness, J. N When not to use an automatic quadrature routine, SIAM Rev. 25, Tong, X. 24. A guaranteed, adaptive, automatic algorithm for univariate function minimization, Master s Thesis. Fred J. Hickernell, hickernell@iit.edu Reliable Adaptive Algorithms U Illinois Chicago, /5/5 8/8

Constructing Trustworthy Automatic Numerical Algorithms

Constructing Trustworthy Automatic Numerical Algorithms Fred J. Hickernell Department of Applied Mathematics, Illinois Institute of Technology hickernell@iit.edu www.iit.edu/~hickernell Joint work with