Large-scale workflows for wave-equation based inversion in Julia

Transcription

1 Large-scale workflows for wave-equation based inversion in Julia Philipp A. Witte, Mathias Louboutin and Felix J. Herrmann SLIM University of British Columbia

2 Motivation Use Geophysics to understand the earth obtain information about the earth from surface seismic experiments source receivers model

3 Motivation Use Geophysics to understand the earth invert for subsurface parameters, e.g. velocity, density, porosity Depth [z] Velocity [km/s] Lateral position [km].5

4 Motivation Use Geophysics to understand the earth image geological interfaces (perturbations of earth parameters) Depth [z] Lateral position [km] 5 2 Perturbation [km/s] 3

5 Motivation Formulate inverse problems and use numerical optimization invert for velocity nonlinear least squares optimization problem minimize m 2 A(m) q d 2 2 (Virieux and Operto, 29) image the subsurface linear least squares optimization problem minimize m 2 J m d 2 2 (Dong et al., 22) Need: access to objective function values and gradients matrices A(m) J,, or actions of these matrices on vectors

6 Motivation Challenges: problem sizes are huge: - seismic surveys consist of tens of thousands of individual experiments - model wave propagation over thousands of time steps in large domains - typical size of modeling matrix: A(m) 2 R n n,n=e6 model physical system data is irregular, has coordinates and meta data inverse problems are difficult to solve (ill- posed, non- convex, non- unique) software to solve seismic inverse problems: - needs to be fast and handle large amounts of data - often tailored towards a specific application - difficult to maintain and modify, often slow adaptation of new concepts - iterative imaging algorithms rely on using all the data in each iteration

7 Motivation Our goal: flexible framework for solving seismic inverse problems abstract matrices and vectors to easily formulate algorithms data containers for irregular seismic data robust parallel framework that scales on HPC environments/the cloud interface Devito (finite difference DSL) to solve PDEs framework that can handle actual 3D industry- sized problems - lower computational costs using stochastic optimization

8 The forward problem Solve (acoustic) wave equation for a given model continuous 2 u r2 u = s m: model parameters (slowness squared) u : wave fields s : source wave fields discretize and rewrite as linear system A u = s A is lower triangular, solve with forward elimination

9 The forward problem Model surface recorded seismic data d = P r F P > s q F := A(m) P r : receiver restriction P > s q : source injection : source wavelet P > s q P r u amplitude time [s]

10 The forward problem Model surface recorded seismic data (2 experiments) d = P r F P > s q F P > s d P r q = * * * d 2 q 2 P r2 F 2 P > s2 can barely store d (e ) cannot store P > s q (e6 ) cannot form F, P r, P s explicitly (e6 e6)

11 The forward problem Adapt idea of matrix- free linear operators SPOT - a linear operator toolbox in Matlab (van den Berg and Friedlander, 22) operators look and behave like explicit matrices matrix knows how to apply itself to vectors forward, adjoint matrix- matrix, matrix- vector products etc.

12 The forward problem Implement abstract linear operators and vectors in Julia JOLI - Julia Operator Library d P r F P > s q = * * * d 2 q 2 P r2 F 2 P > s2

13 The forward problem Solve forward wave equation: Solve adjoint wave equation: JIT compilation

14 Software design What happens after executing the modeling command? check that dimensions are correct check that geometries match call serial/parallel modeling function set function arguments with parameters from linear operators

15 Software design linear operators, data containers, objective functions Julia parallel modeling function parallelization: distribute sources, data serial modeling function interface to Devito (Python) call code return results set up PDEs, discretization generate code + JIT compilation generate code solve PDE Python C

16 Software design linear operators, data containers, objective functions Julia parallel modeling function parallelization: distribute sources, data completely automatic: no manual discretization no implementation of long stencils optimal stencils for current hardware interface to Devito (Python) serial modeling function call code return results set up PDEs, discretization generate code + JIT compilation generate code solve PDE C

17 The inverse problem Inverse problem invert model for given data how does a perturbation in the model relate to a perturbation in the wave field? u = @m A(m) q := J (Jacobian) (Virieux and Operto, 29) including source/receiver projections: J = P A(m) P > s q

18 The inverse problem Depth [km]. -. J m Time [s] m Lateral position [km] Perturbation w.r.t background model d Receiver number Surface recorded seismic data

19 The inverse problem Depth [km]. J m m modeling d Time [s] Lateral position [km] Receiver number Depth [km].5 > J d c m Lateral position [km] -4 imaging Time [s] d Receiver number 2 4

20 Linear inverse problems: seismic imaging Seismic imaging as a linear least squares problem (Dong et al., 22) objective function: minimize m 2 J m d 2 2 (J 2 R m n,m>e,n>e9) easy to implement a solver using JOLI operators:

21 Seismic least squares imaging Seismic imaging as a linear least squares problem J is ill- conditioned and very large, can only afford s of iterations, need to save ü[t] (~TB) reduce computational cost using techniques from stochastic optimization linearized Bregman method (Yin, 2) minimize m subject to: A x x + 2 x 2 2 b 2 apple : thresholding parameter : noise level designed for compressive sensing problems with A 2 R m n,m<<n related to sparse (block- ) Kaczmarz solver for problems w/ any A 2 R m n (Lorenz et al., 24)

22 Seismic least squares imaging Linearized Bregman method for least squares imaging: minimize m subject to: J m C m + 2 C m 2 2 d 2 apple C: Curvelet transform : thresholding parameter : noise level in each iteration possible to work w/ subset of rows of J, d J x = d J sub x = d sub only subsets of experiments in each iteration

23 Seismic least squares imaging Linearized Bregman method for least squares imaging: Algorithm (Lorenz et al., 24). for k =,, 2. z k+ = z k t k J > r(k) (J r(k)x k d r(k) ) max(, Ĵ r(k) x k ) b r(k) 2 3. x k+ = C S (Cz k+ ) 4. end for r(k) is sequence of subsampled experiments (cyclic or random) instead of touching full data set in each iteration, only touch every data sample one or twice calculate gradients on a few large nodes (TB RAM) rather than many small ones

24 Seismic least squares imaging Example with model from the introduction: large 2D model ( 4e6 grid points) surface seismic experiments ( 2.5e9 data points) 2 iterations w/ experiments per iteration (4 PDE solves)

25 Nonlinear inverse problems Objective functions for nonlinear inverse problems functions that spin off function values and gradients can be passed to black- box optimization libraries (e.g. NLopt, JuMP) Example: invert for velocity model (full waveform inversion) nonlinear least squares problem gradient given by minimize m 2 P r A(m) P > s q d 2 2 g = J > P r A(m) P s > q d

26 Nonlinear inverse problems Some final words about scaling: so far only large 2D example ( 4e6 model parameters, 2.5e9 data points) large 3D problems have 2e8 model parameters, e2 data points (or more) For large- scale 3D inverse problems: bounded by RAM, need to store the forward wavefields for all experiments with stochastic optimization methods (linearized Bregman etc.), reduce number of experiments per iteration 3D becomes feasible, if working with few, but large nodes (RAM > TB)

27 Conclusions Julia framework for seismic modeling and inversion modular software structure matrix- free linear operators and seismic data containers efficient and fast PDE solves through Devito scales to very large and realistic problem sizes parallel framework, resilience to hardware failures easy to formulate algorithms, objective functions + gradients, etc. easy to interface optimization libraries

28 Outlook In the future we plan to: test our framework in the cloud add IO functions for seismic data (automatic set up of data containers + geometry) application to 3D seismic field data sets (requires nodes with ~ TB RAM)

29 Acknowledgements This research was carried out as part of the SINBAD project with the support of the member organizahons of the SINBAD Consorhum.