Parallel 3-D simulation of seismic wave propagation in heterogeneous anisotropic media: a grid method approach

Transcription

1 Geophys. J. Int. (6) 165, doi: /j X x arallel 3-D simulation of seismic wave propagation in heterogeneous anisotropic media: a grid method approach Hongwei Gao and Jianfeng Zhang Institute of Geology and Geophysics, tate Key Laboratory of Lithosphere Evolution, Chinese Academy of ciences, Beijing 19, China. Zhangjf@mail.igcas.ac.cn Accepted 6 January 6. Received 5 November 1; in original form 5 May 19 UMMARY This paper presents a parallel numerical technique for modelling wave propagation in 3-D heterogeneous anisotropic media. The scheme is developed by following a so-called 3-D grid method of the elastic-isotropic case. The proposed parallel algorithm needs small data exchanges between subdomains in contrast to that developed based on other numerical techniques; therefore, it is more suitable for a C-Cluster. The algorithm is implemented on a mesh of mixed tetrahedrons and parallelepipedons, thus providing an accurate description of arbitrary 3-D surface and interface topographies and an easy generation of a non-uniform, unstructured mesh. The unstructured mesh means that the proposed algorithm can reduce the memory requirement by flexibly assigning small grid spacing in regions with low velocities and larger grid spacing in regions with higher velocities. Like the 3-D grid method, the resulting anisotropic scheme naturally satisfies the free-surface boundary conditions of arbitrary surface topography. As a result, the near-surface scattering effects can be more accurately modelled. The proposed scheme can handle a general anisotropy without any interpolations. In this paper, the transversely isotropic medium with a tilted symmetry axis, as typically caused by a system of parallel cracks or fine layers, is discussed in detail. A paraxial absorbing boundary condition in a 3-D general anisotropic case is also proposed. Comparisons with analytical solutions demonstrate the accuracy of the parallel algorithm. Computed 3-D radiation patterns illustrate shear-wave splitting, as predicted by the theory. We show the generality and flexibility of the algorithm by modelling wave propagation in an anisotropic half-space with a hemispherical crater on the surface and in mixed isotropic/anisotropic models with horizontal and inclined interfaces. Key words: numerical techniques, seismic anisotropy, seismic modelling, synthetic seismograms, seismic wave propagation, topography. GJI eismology 1 INTRODUCTION Geological media often exhibit seismically anisotropic, resulting, for example, from fine-scale layering, the presence of oriented microcracks or fractures, or the preferred orientation of non-spherical grains or anisotropic minerals. The numerical simulation of seismic wave propagation in heterogeneous anisotropic media is, therefore, important to improve our understanding of complicated wave propagation phenomena. ince in-plan and out-of-plan motions are coupled for the transversely isotropic medium with a tilted symmetry axis, 3-D simulation seems necessary in anisotropic media. Wave propagation in stratified anisotropic media can be modelled using the reflectivity method (Booth & Crampin 1983; Fryer & Frazer 1987). However, the reflectivity method has become much more time consuming than that in the isotropic case due to the fact that Corresponding author. the eigenvalues and eigenvectors to construct the propagator must be found numerically in a general anisotropic medium (Fryer & Frazer 1984). For a weak heterogeneous and weak anisotropic medium, one can use the asymptotic ray theory (Gajewski & sencik 1987) and other similar approaches as WKBJ (ingh & Chapman 1988) and Gaussian beams (Hanyga 1986). However, like the reflectivity method, the ray theory approach also becomes more complicated and time consuming in the anisotropic case. For the general case of a heterogeneous, anisotropic medium one needs to turn to the fully numerical techniques. Numerical techniques such as the finite-difference (e.g. Igel et al. 1995; aenger & Bohlen 4), pseudospectral (e.g. Carcione et al. 199), spectral element (e.g. Komatitsch et al. ), and finiteelement methods (e.g. Bao et al. 1998) have been extended to anisotropic media by replacing the elastic isotropic stiffness tensor with an anisotropic one. However, the use of a global basis in the pseudospectral scheme leads to inaccuracies for models with strong heterogeneity or sharp boundaries (Komatitsch et al. ). C 6 The Authors 875 on 18 November 17

2 876 H. Gao and J. Zhang Regarding the finite-difference method, the standard staggered grid scheme (Igel et al. 1995) needs some interpolations of components of the strain tensor for handling general anisotropic media, whereas the rotated staggered grid scheme (aenger & Bohlen 4) can avoid them. As in the isotropic case, it is not convenient for the finite-difference and pseudospectral schemes to implement the freesurface boundary conditions of arbitrary surface topography. The spectral element and finite-element methods satisfy the free-surface boundary conditions naturally. It can also model the 3-D surface and interface topographies by using curved or piecewise elements. However, the classical finite-element method comes with a high computational cost; and a small (spectral) element that is smaller than that required by the numerical dispersion is needed in the description of the surface (or interface) topography with a high curvature, as for instance in the description of a hemispherical crater (Komatitsch & Tromp 1999). The simulation of seismic wave propagation in realistic 3-D heterogeneous anisotropic media using the fully numerical techniques leads to the huge memory requirement and computational cost as the dominant frequencies increase (e.g. in seismic exploration). Although the memory requirement can be reduced by using the higherorder schemes such as the pseudospectral and spectral element methods, a parallel computation based on a partition of the computational domain seems necessary. When extending the numerical techniques into a parallel algorithm, we prefer small data exchanges between subdomains so that a high speed-up rate for using a large number of processors can be achieved, thus giving a new point to assess the current 3-D modelling schemes. mall data exchanges become crucial when the parallel algorithm is implemented on a class of cheap parallel computers, that is, C-Cluster, owing to the use of a large number of processors and the relatively slow communication speed between them. ince the spatial derivatives in the pseudospectral scheme are calculated using the full data along one spatial direction, the parallel algorithm developed by adapting the pseudospectral scheme needs a large amount of communication between processors, especially when the partition of the computational domain occurs in three spatial directions, instead of one spatial direction as in Furumura et al. (1998). As a result, the resulting parallel algorithm (e.g. Furumura et al. 1998) will be decelerated a lot on a C-Cluster. In contrast, the spectral element method in the absence of an explicit stiffness matrix (Komatitsch et al. 3) and the second-order finite-difference scheme can achieve high speed-up rates when being implemented on a parallel computer. A numerical technique that can model wave propagation in both -D and 3-D heterogeneous isotropic media including surface topography has been proposed by Zhang & Liu (1999, ). There it was called the grid method to distinguish it from the finite-element or finite-difference method. The grid method is flexible in incorporating -D and 3-D surface and interface topographies with a natural satisfaction of the free-surface boundary conditions. The -D grid method has been extended to account for wave propagation in media with high velocity contrasts (Zhang 4a), in fractured media (Zhang 5), and in mixed elastic/poroelastic models (Zhang 1) as well as wave propagation across fluid solid boundaries (Zhang 4b). Although both the grid method and the finite volume method (e.g. Dormy & Tarantola 1995) are developed based on the divergence theorem, the different numerical implementations for evaluating the surface and volume integrals mean that the two schemes exhibit different numerical qualities; for example, the grid method satisfies the free-surface conditions of a complex geometrical elastic boundary naturally (Zhang & Liu 1999, ), but the implementation of the free-surface conditions requires further study in the elastic finite volume method (Dormy & Tarantola 1995). In this paper, the 3-D grid method is extended to account for wave propagation in 3-D heterogeneous anisotropic media as well as a parallel implementation on a C-Cluster. The 3-D anisotropic grid scheme is developed by following the integral approach of the 3-D elastic momentum equations for a discretization of tetrahedrons and parallelepipedons. The resulting algorithm can, therefore, be implemented on a mesh of mixed tetrahedral and parallelepiped grid cells, thus providing an accurate description of arbitrary 3-D surface and interface topographies and an easy generation of a non-uniform, unstructured mesh. The unstructured mesh means that the algorithm can significantly reduce the memory requirement by flexibly assigning small grid spacing in regions with low velocities and larger grid spacing in regions with higher velocities. The proposed scheme is of a single-mesh algorithm. In contrast to the schemes based on the dual lattices (e.g. Dormy & Tarantola 1995), the discretization for the heterogeneous structures can be easily implemented by making the medium within each grid cell homogeneous. Like the 3-D grid method, the proposed anisotropic scheme also naturally satisfies the free-surface boundary conditions of arbitrary surface topography. As a result, the near-surface scattering effects can be more accurately modelled. Moreover, the algorithm architecture, that is, how the spatial derivatives at a node are calculated, of the 3-D anisotropic grid scheme leads to small data exchanges between subdomains in its parallel simulation, regardless of 1-D or 3-D partition of the computational domain. A high speed-up rate on a C-Cluster with a large number of processors can then be achieved. The problem is formulated in terms of three displacement components at each node. As a result, no interpolations as those in the standard staggered grid finite-difference scheme are needed for the algorithm to handle general anisotropy. The radiation conditions for simulating a semi-infinite medium deserve to be a challenging task for numerically modelling seismic wave propagation in 3-D anisotropic media. Available methods include the perfectly matched layer (ML), as discussed in a -D anisotropic case (Collino & Tsogka 1), and the absorbing boundary conditions, as developed by combining several first-order absorbing boundary conditions designed for the scale wave equation in isotropic cases (Higdon 1991; eng & Toksoz 1995). For an easy implementation, a paraxial absorbing boundary condition in a 3-D general anisotropic case is developed and incorporated into the proposed parallel algorithm. Unlike the classical paraxial conditions (e.g. Clayton & Engquist 1977), the paraxial approximations in the proposed method under 3-D general anisotropic cases are obtained by an analytical or numerical eigen-decomposition. This paper is set up as follows: first anisotropic wave equations are discussed, especially for the transversely isotropic medium with a tilted symmetry axis; then a 3-D anisotropic grid scheme is proposed. Next a paraxial absorbing boundary condition for 3-D general anisotropic media is given; after this the parallel numerical implementation of the 3-D anisotropic grid scheme is presented. Finally a few numerical examples are presented to illustrate the parallel algorithm on a C-Cluster. ANIOTROIC WAVE EQUATION The 3-D elastic wave equations in anisotropic media, in the absence of body source, can be expressed in terms of displacements and stresses as ρ u i t = τ ij x j, (1) C 6 The Authors, GJI, 165, on 18 November 17

3 τ ij = 1 c ijkl ( uk x l + u ) l, () x k arallel 3-D modelling for anisotropic wave propagation 877 X 3 (Z) where i, j, k, l = 1,, 3, we assume the summation convention for repeated indices; and ρ is the density, x i are the Cartesian coordinate components with the x 3 -axis pointing vertical downwards, u i are the components of the displacement, τ ij are the Cartesian components of the stress tensor, and c ijkl are the components of a fourth-order elastic stiffness tensor. The term c ijkl satisfies the (Green) symmetry conditions, that is, c ijkl = c jikl = c ijlk = c klij, so only 1 independent constants are needed to describe the constitutive relation for a general anisotropic medium. However, the anisotropy with its 1 independent elastic constants is unnecessarily complicated for seismology. It is unlikely that seismic data will ever have the resolution to measure 1 elastic constants. When we think of anisotropic earth models, we usually think in terms of anisotropy from fractures or sequences of thin layers, such as sand and shale. The anisotropy caused by stratified sedimentary (choenberg 1983) and parallel fractures (Crampin 1984) more often exhibits a transversely isotropic, and the orientation of fractures or fine layers determines the direction of the symmetry axis. As a result, the elastic stiffness tensor c ijkl can be described by five modified Lamé parameters and two azimuth angles of the symmetry axis. rovided that a new coordinate system (x 1, x, x 3 ) is set up so that its one coordinate axis, for example, the x 3-axis, coincides with the symmetry axis of a transversely isotropic medium, the elastic stiffness tensor expressed in this new coordinate system becomes simply. Its contracted (matrix) version reads λ 1 + μ 1 λ 1 η λ 1 λ 1 + μ 1 η η η λ + μ, (3) μ μ μ 1 where λ 1,μ 1,λ,μ and η are the modified Lamé parameters. Here subscripts 1 and indicate the perpendicular and parallel directions to the plane of isotropy. Moreover, with θ denoting the angle of the symmetry axis with respect to the x 3 -axis and φ the angle of the projection of the symmetry axis on the x 1 x -plane with respect to the x 1 -axis, as illustrated in Fig. 1, the transformation matrix for the coordinate system (x 1, x, x 3 )to(x 1, x, x 3 ) is expressed as cos φ cos θ cos θ sin φ sin θ T = sin φ cos φ. (4) cos φ sin θ sin φ sin θ cos θ The elastic stiffness tensor c mnpq, expressed in the coordinate system (x 1, x, x 3 ) with the x 3 -axis pointing vertical downwards, can then be obtained as follows (Ting 1996): c mnpq = mi nj pk ql c o ijkl, i, j, k, l, = 1,, 3, (5) where we assume the summation convention for repeated indices; and mi, nj, pk and ql denote the elements of the transformation matrix T, as expressed in eq. (4), and c o ijkl denote the components of the stiffness tensor, as expressed in eq. (3). The stiffness tensor c mnpq, described by five modified Lamé parameters and two azimuth angles, is related to a transversely isotropic medium with a tilted symmetry axis, as typically caused by a system of parallel cracks or fine layers. X 1 (X) φ θ X 3 (ymmetric axis) X (Y) Figure 1. Illustration of the azimuth angles of the symmetry axis. 3 3-D ANIOTROIC GRID CHEME The 3-D (isotropic) grid method is developed based on an integral approach of the 3-D elastic momentum equations for a discretization of tetrahedrons and parallelepipedons. This integral approach, that is, a weak form of equilibrium equations, is the key for the 3-D (isotropic) grid method to exhibit many desirable qualities, such as the flexibility in incorporating 3-D surface and interface topographies and natural satisfaction of the free-surface boundary conditions. This paper will follow this integral approach to derive the 3-D anisotropic grid scheme. Unlike the isotropic grid scheme in Zhang & Liu (), which employs three displacements and three velocities in a staggered time level, the anisotropic grid scheme is developed by formulating the problem in terms of three displacement components (of two time steps) at each node. By following the discussion in Zhang & Liu (), the integral approach of the 3-D elastic momentum equations, that is, eq. (1), for an inner node under a discretization of mixed tetrahedral and parallelepiped grid cells is obtained as (see Appendix A) ( / M u ) i t = m t 3 3 l=1 τ l ij m ( p c j )l + l=1 τ l ij ( ) e j, (6) l where m t denotes the number of the tetrahedral grid cells around node, m p denotes the number of the parallelepiped grid cells around node, ( u i / t ) are the second-order time derivatives of the displacement components at node, τ l ij are the stresses inside the lth tetrahedral or parallelepiped grid cell around node, m p is a mass parameter, and (cj ) l and (ej ) l are, respectively, the geometrical coefficients for the lth tetrahedral and parallelepiped grid cells around node, as expressed in eqs (A3) and (A4). Furthermore, the integral approach of the 3-D elastic momentum equations for a surface node B under a discretization of tetrahedral grid cells is given by (see Appendix A) ( 3 3 ( / M B u ) i t = m t B l=1 τ l ij ( ) c B j + l T τ ij n j )ds, (7) where T is the surface topography, n j are the direction cosines of the outward-directed normals to the surface; and other variables have the same meanings as those in eq. (6). C 6 The Authors, GJI, 165, on 18 November 17

4 878 H. Gao and J. Zhang The second terms on the right-hand side of eq. (7) are equivalent to zero for the surface in the absence of source, that is, a free surface, and equivalent to the known loads with a surface source. The former means that the free-surface boundary conditions can be implemented by simply omitting the second terms on the righthand side of eq. (7). The first terms are same in eqs (6) and (7). Therefore, the free-surface boundary conditions can be naturally satisfied by using eq. (6) for nodes at both the internal domain and surface. Moreover, we can easily incorporate the surface source by adding values to the resulting second-order time derivatives of the displacement components at a surface node, as expressed in eq. (7). Although eqs (6) and (7) are derived based on the dual lattices, as used in the standard staggered grid finite-difference and finite volume schemes, it is found that only tetrahedral and parallelepiped grid cells, that is, a single mesh, are involved in eqs (6) and (7). As a result, the only requirement for the numerical discretization of heterogeneous structures is that the medium within each grid cell should be homogeneous. The proposed scheme takes stresses τ l ij inside each grid cell as intermediate variables that do not need to be stored. These stresses can be solved using the displacements at the nodes of the grid cell. By following the finite element (Zienkiewicz & Taylor 1989) and 3-D grid (Zhang & Liu ) methods, we obtain the first-order spatial derivatives of the displacement components inside a typical tetrahedral grid cell as u i = 1 4 c j x j V u i, (8) =1 and inside a typical parallelepiped grid cell as u i = 1 8 e j x j W u i, (9) =1 where cj and ej are the same as (cj ) l and (ej ) l used in eqs (6) and (7) for denoting each of the four or eight nodes of the tetrahedral or parallelepiped grid cell, and u i denote the displacement components at each node, V and W denote the volumes of tetrahedron and parallelepipedon, respectively. ince the three displacement components are available at each node, no interpolations of the displacement components as in the staggered grid finite-difference scheme (e.g. Igel et al. 1995) are needed for solving stresses τ l ij inside each grid cell in the presence of a general anisotropic medium (i.e. most of the components of the elastic stiffness tensor are non-zero). The stresses can be solved by substituting the displacement components into eqs (8) and (9) and then into eq. (). The computation is launched by assuming the displacement fields at time levels t and t t. Calculating the stresses inside each grid cell using the displacement field at time level t and then substituting them into eq. (6), we obtain the second-order time derivatives of the displacement components at time t at each node. Then the displacement fields can be updated from time levels t and t t to time levels t + t and t using the standard centre difference scheme in time as follows: u t+ t i = u t i ut t i + t ( u i / t ) t. (1) ince stresses will never be used again after being substituted into eq. (6), they do not need to be stored. Only the three displacement components of each node at two time steps need to be stored for the proposed scheme. From the foregoing discussion, it is clear that the proposed 3-D anisotropic grid scheme involves two computational steps, that is, the computation inside each grid cell related to eqs (), (8) and (9) and computation to collect the contribution to the time derivatives of the displacement components at each node from each grid cell related to eq. (6). Moreover, the latter can be accomplished by an accumulation looping through all grid cells, as discussed in the following. This algorithm architecture is, therefore, suitable for the parallel simulation based on a partition of the computational domain. We propose a stability criterion for the 3-D anisotropic grid scheme by adapting that presented in Zhang & Liu () without a rigorous analysis in the anisotropic case. For the tetrahedral grid cell, the stability criterion is given by t 3h/α, (11) and for the parallelepiped grid cell, the stability criterion reads t 3h/3α, (1) where t is the time interval, h is the minimum edge length of tetrahedral or parallelepiped grid cells, and α is the maximum quasi-wave velocity. For the mesh of mixed tetrahedral and parallelepiped grid cells, eq. (1) can serve as a stability criterion to determine proper temporal and spatial intervals. 4 ABORBING BOUNDARY CONDITION Only two types of boundary conditions have to be considered for modelling seismic wave propagation in complex 3-D heterogeneous structures: the free-surface conditions of surface topography, and the radiation conditions for simulating a semi-infinite medium. The implementation of the free-surface boundary conditions, especially in the presence of 3-D surface topography, is difficult for the conventional finite-difference schemes even in an isotropic case. Owing to the use of the integral approach of the 3-D elastic momentum equations, the proposed 3-D anisotropic grid scheme naturally satisfies the free-surface boundary conditions for a 3-D complex geometrical surface. However, the radiation conditions deserve to be a challenging task for modelling seismic wave propagation in 3-D anisotropic structures. The ML method (Collino & Tsogka 1) has a good behaviour in -D anisotropic cases and it is possible to extend the ML to 3-D anisotropic media. However, for an easy implementation, this paper proposes a paraxial absorbing boundary condition for the 3-D general anisotropic cases. The paraxial condition is developed by defining a preferred orientation of wave propagation and assuming plane wave propagation in the preferred orientation in the vicinity of the boundary. Instead of analytically studying the dispersion relation as in the classical paraxial conditions (e.g. Clayton & Engquist 1977), the paraxial approximations in this 3-D anisotropic case are obtained by an analytical or numerical eigen-decomposition. This leads to a flexible paraxial approximation. rovided that the medium is homogeneous in the vicinity of the artificial boundary, substituting eq. () into eq. (1) leads to ρ u i t = 1 c ijkl ( u k x j x l + u l x j x k ). (13) Let a new coordinate system (y 1, y, y 3 ) have one axis, that is, the y 1 -axis, to coincide with the predefined preferred orientation of wave propagation in the vicinity of the artificial boundary. The assumption, that is, plane wave propagation in the preferred orientation, yields / y = / y 3 =. With T kl denoting the elements of the transformation matrix for the coordinate system (y 1, y, y 3 ) C 6 The Authors, GJI, 165, on 18 November 17

5 arallel 3-D modelling for anisotropic wave propagation 879 to (x 1, x, x 3 ), we have = T j1 T l1, (14) x j x l y1 where T kl are determined by two azimuth angles as in eq. (4). ubstituting eq. (14) into eq. (13) yields ρ u i = 1 t c ijklt j1 (T l1 + T k1 ) (u k + u l ), i, j, k, l = 1,, 3, y1 (15) where we assume the summation convention for repeated indices. The matrix version of eq. (15) reads U t = Q U, (16) y1 where U T = {u 1, u, u 3 }. To eigendecompose matrix Q into Q = 1 V, we can rewrite eq. (16) as U t = V U, (17) y1 where V is a diagonal matrix whose elements denote the propagation velocities of the quasi-, quasi-v and quasi-h waves in the preferred orientation in the vicinity of the artificial boundary. Let {U} T ={u 1, u, u 3 }. The analytical solutions of eq. (17) give ū i (y 1 v i t, y, y 3, t t) = ū i (y 1, y, y 3, t), i = 1,, 3, (18) where v i are the diagonal elements of matrix V. Eq. (18) represents the outgoing nature of wavefield. With T kl denoting the inversion of T kl, that is, the components of the transformation matrix for the coordinate system (x 1, x, x 3 )to(y 1, y, y 3 ), eq. (18) can be expressed in the coordinate system (x 1, x, x 3 )as u i (x 1 T 11 v i t, x T 1 v i t, x 3 T 31 v i t, t t) = u i (x 1, x, x 3, t), i = 1,, 3. (19) Thus, the u i at a boundary node (x 1, x, x 3 ) at time t can be obtained by picking up u i at an internal point (x 1 T 11 v i t, x T 1 v i t, x 3 T 31 v i t) at time t t. We then solve u i at the boundary node using U = 1 {u 1, u, u 3 } T. Usually, interpolations are needed for getting u i at internal point (x 1 T 11 v i t, x T 1 v i t, x 3 T 31 v i t) from u i at the neighbouring nodes. Here u i can be obtained by U at each node. The absorbing boundary conditions in a general anisotropic medium can thus be implemented by simulating the outgoing nature of wavefield, as shown in eqs (18) or (19). The complexity resulting from implementing the proposed paraxial absorbing boundary conditions in a general anisotropic medium is all restricted to the eigen-decomposition of matrix Q. This eigendecomposition can be performed analytically or numerically in advance. The related interpolation coefficients for obtaining u i at point (x 1 T 11 v i t, x T 1 v i t, x 3 T 31 v i t) from u i at the neighbouring nodes can also be solved and stored in a table in advance. As a result, low computational cost is introduced in implementing the proposed absorbing boundary conditions in a general anisotropic medium. ince the artificial boundaries are usually put a little far away the target zone of interest and the media in the vicinity of the artificial boundary are normally less complicated, the predefined preferred orientations of wave propagation on the artificial boundaries can be approximately estimated by tracing the ray paths from the source to the boundaries. 5 ARALLEL NUMERICAL IMLEMENTATION The parallel computation is implemented based on a partition of the computational domain. The smaller the data exchanges between subdomains, the higher the speed-up rate of the parallel algorithm. Two aspects determine the data exchanges between subdomains. The first is the algorithm architecture, that is, how the spatial derivatives at a node are calculated as well as how many neighbouring nodes are involved in this calculation, and the second is the partition of the computational domain, that is, the computational domain is sliced in one spatial direction, two spatial directions, or three spatial directions. It is clear that the number of contacting nodes between subdomains on a 1-D partition is much more than that of a 3-D partition. The data exchanges are crucial when the parallel algorithm is implemented on a C-Cluster with the use of a large number of processors due to the relatively slow communication speed between those processors. A 3-D parallel algorithm on a C-Cluster, therefore, should be developed by adapting a numerical technique that has a localized calculation of the spatial derivatives and is flexible in the partition of the computational domain. It is found from eqs (6), (8) and (9) that the second-order time derivatives of the displacement components at each node are solved only using the displacements at the closing nodes. Moreover, the computation is performed by first a calculation within each grid cell and then a simple accumulation, as shown in eq. (6), rather than by first picking up all the relevant displacement components and then substituting them into a difference or multiplication formulation. Therefore, the 3-D anisotropic grid scheme is of a localized calculation and of flexible in the partition of the computational domain. As a result, the parallel algorithm developed by adapting the 3-D anisotropic grid scheme yields small data exchanges between subdomains as compared with that developed based on current numerical techniques. The following is a detailed description of its parallel numerical implementation: (1) the computational domain is parted in three spatial directions to yield a lot of approximately equal volume subdomains by following the numerical mesh. () We store the three displacement components at two time steps of each subdomain in the related processor and assign other three memory units to each node of this subdomain in the processor. (3) The computation within each grid cell is performed simultaneously on all processors. First compute the spatial derivatives u i / x j by eqs (8) or (9) using the displacements at time t; then solve stresses τ ij by substituting these spatial derivatives into eq. (); after this, compute fi = 3 τ ijc j for taking each of the four nodes of the tetrahedral grid cell and fi = 3 τ ije j for taking each of the eight nodes of the parallelepiped grid cell; finally add fi (i = 1,, 3) to the three memory units corresponding to the node represented by of this grid cell. Here cj and ej are the geometric coefficients of each grid cell, which are the same as (cj ) l, etc., and (ej ) l, etc., expressed in eqs (A3) and (A4). (4) For nodes on the contacting boundaries of subdomains, data exchanges between processors are needed. First send the accumulation results at those nodes, that is, the values stored in the related memory units, to the corresponding processors; then receive the accumulation results of those boundary nodes in other subdomains; after this add the received results to the memory units corresponding to those nodes on each processor simultaneously. Finally we can simultaneously update the displacement fields using eq. (1) on all processors. C 6 The Authors, GJI, 165, on 18 November 17

6 88 H. Gao and J. Zhang It is seen that only step (4) involves the data exchanges between subdomains, which are only related to the nodes on the contacting boundaries of subdomains. If we perform computation of step (3) first for the grid cells which have nodes on the contacting boundaries and then for other internal grid cells, the data changes can be simultaneously completed during the computation of step (3). Therefore, the proposed parallel algorithm can achieve a high speed-up rate when being implemented on a C-Cluster with the use of a large number of processors. As in Zhang & Liu (), we need to store only one volume and 1 geometric coefficients for congruent tetrahedral or parallelepiped grid cells in each subdomain. Hence, no much memory requirement arises from using a non-uniform, unstructured mesh. Contrarily, the unstructured mesh leads to a significant reduction of the memory requirement by assigning small grid spacing in regions with low velocities and larger grid spacing in regions with higher velocities. 6 NUMERICAL EXAMLE 6.1 Analytic comparison The accuracy of the proposed numerical technique is tested by comparing the numerical results with the analytical solutions of the Lamb s problem for a transversely isotropic medium with a vertical (VTI) and horizontal (HTI) symmetry axes. A vertical Gaussian point source, containing frequencies up to 1 Hz, is loaded at the free surface of a 3-D half-space. The comparisons are performed by first transforming the 3-D numerical results into a line-source response by carrying out an integration along the receiver line (Wapenaar et al. 199) and then comparing the resulting results with the -D Lamb s analytical solutions. Note that the line-source response of the 3-D HTI medium represents a numerical solution of the -D isotropic medium when the source-line coincides with the symmetry axis. It should be mentioned that Carcione et al. (199) presented an analytical comparison of the point-source response in a 3-D VTI medium in the absence of the free surface. The VTI and HTI media have the same modified Lamé parameters and density, as shown in Table 1. The numerical mesh used is made up of mixed tetrahedrons and right cubes with the total nodes of for all computations. 3 subdomains are obtained by slicing in three spatial directions and the parallel simulation is then performed on a 3-node C-Cluster. Here, the tetrahedral grid cells are generated through splitting each right cube into five tetrahedrons. This represents an inhomogeneous grid cell case. We define a Cartesian coordinate system (x, y, z) with the z-axis pointing vertically downwards. Fig. shows the comparison between the resulting numerical and -D analytical y-direction components of the displacement for the VTI medium. The integration is performed along the receiver line on the free surface parallel to the x-direction with a normal distance of m away the point source. This represents a -D result of m away the source. The VTI medium has a quasi--wave velocity of 1414 ms 1 and a quasi-wave velocity of 3688 ms 1 in the horizontal plane. The used spatial and temporal intervals are m and ms, respectively. Thus, the number of nodes per minimum quasi--wave wavelength is 7.1 for the analytic comparison of the VTI medium. Fig. 3 shows the same Table 1. Elastic constants of the VTI and HTI media. λ 1 (Ga) μ 1 (Ga) λ (Ga) μ (Ga) η(ga) ρ(kg m 3 ) Y component of Displacement 1 x Lamb VTI Modelling x=m Rayleigh wave Direct wave.5 1 Time (s) 1.5 Figure. Comparison between numerical and analytical y-direction components of the displacement for the VTI medium. Lamb s result is an analytical solution of the -D Lamb s problem of the VTI medium. The modelling result is the line-source response of the 3-D VTI medium obtained by the superposition of the 3-D point-source responses. A good agreement is observed. Y component of Displacement 15 x Lamb HTI Modelling x=m Rayleigh wave Direct wave Time (s) Figure 3. ame comparison as Fig. for the HTI medium. Lamb s result is an analytical solution of the -D Lamb s problem of the isotropic medium. The modelling result is the line-source response of the 3-D HTI medium obtained by the superposition of the point-source responses. This represents a -D isotropic solution because the source line coincides with the symmetry axis. A good agreement is observed. comparison for the HTI medium. The used spatial and temporal intervals are the same as those for the VTI medium. The HTI medium has a quasi--wave velocity of 1414 ms 1 and a quasi--wave velocity of 3688 ms 1 in the y-direction and a quasi--wave velocity of 1556 ms 1 and a quasi--wave velocity of 4195 ms 1 in the x-direction. Thus, the number of nodes per minimum quasi--wave wavelength is 7.1 for the analytic comparison of the HTI medium. In spite of the errors resulting from the transformation of the pointsource response into the line-source one, numerical and analytical results agree well in Figs and 3. The surface waves propagate without dispersion. These comparisons demonstrate the accuracy of the proposed 3-D anisotropic grid scheme and the corresponding parallel algorithm. C 6 The Authors, GJI, 165, on 18 November 17

7 arallel 3-D modelling for anisotropic wave propagation 881 Y component of Displacement 1 x x=m VTI Modelling Lamb Rayleigh wave Direct wave Time (s) 4 8 Y-axis (m) X-axis (m) Figure 4. Effect of an insufficient number of nodes per Rayleigh wavelength. The display is the same comparison as Fig. except that the spatial spacing of nodes has been changed from to 4 m. The dispersion of the Rayleigh wave is apparently. To assess the numerical dispersion, we repeat the same experiment as for the VTI medium with larger spatial and temporal intervals of 4 m and 4 ms. Fig. 4 shows the same comparison as Fig. under a spatial spacing of 4 m. The dispersion of the Rayleigh wave in Fig. 4 is apparently because of an insufficient number of nodes per minimum Rayleigh wavelength, that is, 3.7 nodes per minimum Rayleigh wavelength (with a Rayleigh wave velocity of 1483 ms 1 in the VTI medium). The number of nodes per minimum direct wavelength is 9. in this experiment. The direct wave propagates without dispersion in Fig. 4, as one should expect. We can, therefore, conclude that the lengths of the grid cell edges should normally not exceed one-fifth the minimum wavelength to model elastic wave propagation accurately. 6. Radiation pattern 3-D snapshot of the x-direction component of the displacement at.4 s propagation time is shown in Fig. 5 for wave propagation in a homogeneous transversely isotropic medium with a tilted symmetry axis of θ = 3 and φ = 4 (as illustrated in Fig. 1). The five modified Lamé parameters and density of the homogeneous medium is listed in Table. This represents an elliptical anisotropic case. The Ricker wavelet point source is located at the centre of the 3-D space with a peak frequency of Hz. The shear-wave splitting due to the difference between the velocities of H-waves propagating parallel and perpendicular to the symmetry axis can be observed clearly in Fig. 5, as predicated by the theory. The numerical model is composed of nodes with an even spatial spacing of 4 m. The time step is.5 ms. The numerical mesh is made up of tetrahedral grid cells, which is generated through splitting each right cube into five tetrahedrons. Note that the different tetrahedral grid cells are used under this discretization. 3 equal volume subdomains with nodes are obtained by slicing in three spatial directions. The parallel simulation is then performed on a 3-node C-Cluster. Fig. 6 shows a -D slice of the snapshot of the displacement at.3 s propagation time for wave propagation in a general anisotropic 1 Figure 5. napshot of the x-direction component of the displacement in a homogeneous transversely isotropic medium with a tilted symmetry axis. The display is a point-source response at a propagation time of.4 s. The medium represents an elliptical anisotropy case. The shear-wave splitting can be observed, as predicated by the theory. Table. Elastic constants of the elliptical anisotropic medium. λ 1 (Ga) μ 1 (Ga) λ (Ga) μ (Ga) η(ga) ρ(kg m 3 ) medium, that is, Rochelle salt, as discussed in Wang & Achenbach (1996). The non-zero elastic constants of Rochelle salt are c 11 =.5, c = 3.81, c 33 = 3.71, c 1 = 1.41, c 13 = 1.16, c 3 = 1.46, c 44 = 1.34, c 55 =.3, and c 66 =.98. Here c ij define the contracted matrix notation for c ijkl, and the unit is Ga. The computed wave fronts resembles those predicated by the theory in the line-source case (Wang & Achenbach 1996). The shear-wave splitting can also -5 5 X-axis (m) -5 5 Figure 6. xz-plane slice (passing through the source) of the snapshot of the z-direction component of the displacement for wave propagation in Rochelle salt at a propagation time of.3 s. More complicated wave fronts are observed in this general anisotropic medium. C 6 The Authors, GJI, 165, on 18 November 17

8 88 H. Gao and J. Zhang be observed. The Ricker wavelet point source is located at the centre of the 3-D space with a peak frequency of 1 Hz. Again, the numerical mesh is generated through splitting each right cube into five tetrahedrons but the edge of the right cube is changed to m. The parallel simulation is performed on a 3-node C-Cluster. The time step used is.5 ms. ) (45 o, 135 o ) (36 o, 9 o ) (45 o, 45 o ) Absorbing boundary Following the computation of the radiation pattern in the transversely isotropic medium, as shown for.4 s propagation time in Fig. 5, we test the proposed paraxial absorbing boundary conditions by continuing this computation until.8 s. Fig. 7 shows 3-D snapshots of the x-direction component of the displacement at.5 and ) 1 (36 o,18 o ) ( o, o ) (36 o, o ) Y-axis (m) 4.5s 8 1 X-axis (m) (45 o,135 o ) (36 o,9 o ) (45 o,45 o ) Figure 8. redefined preferred orientations for a typical -D absorbing boundary in term of two azimuth angles. The angles in each region represent the predefined preferred orientations for wavefield impinging on this region. The definitions of the azimuth angles see Fig Y-axis (m) 4.8s 8 1 X-axis (m) Figure 7. Wave propagation across absorbing boundaries. The display is the same as that in Fig. 5 except that the propagation times become.5 and.8 s. The absorbing boundaries are positioned at the minimum x and the maximum y and z. Note that the quasi- and splitting quasi- waves propagate through the boundaries almost without reflections..8 s propagation times. The predefined preferred orientations for a typical -D absorbing boundary are illustrated in term of (θ, φ) in Fig. 8. Here, the two angles denote the azimuth angles as illustrated in Fig. 1. It is observed from Fig. 7 that the quasi- wave and splitting quasi- waves propagate through the absorbing boundary almost without reflections. The proposed paraxial absorbing boundary conditions behave well even for wavefield with a high incident angle. 6.4 calability test To assess the efficiency of the parallel computation, we again apply the proposed parallel algorithm to a homogeneous medium with the physical parameters same as those in Table. The structured mesh is considered first, for which the numerical model is composed of nodes with an even spatial spacing of 5 m. Here, the mesh is generated through splitting each right cube into five tetrahedrons. This means that the mesh is constructed by inhomogeneous tetrahedral grid cells. The model is then subdivided into eight equal volume subdomains with both 1-D and 3-D partitions of the computational domain. The computation times with and without parallel processing as well as under the 1-D and 3-D partitions for performing 3 time steps are listed in Table 3. Here, the computation time without the parallel processing, that is, the computation time of the whole model at a single CU, is simply estimated using eight time computation times of a subdomain (due to the fact that the computational cost of the 3-D anisotropic grid scheme is linearly related to the number of grid cells). The speed-up rate under the 3-D partition is higher, up to 94.3 per cent, as one should expect. Next, we consider the unstructured mesh. The unstructured mesh is generated by first introducing random shifts in the three spatial directions to each node of the structured mesh and then cancelling the grid cells with too short an edge (e.g. less than 3 m) and splitting C 6 The Authors, GJI, 165, on 18 November 17

9 arallel 3-D modelling for anisotropic wave propagation 883 Table 3. Computation times with and without parallel processing for 3 time steps. tructured mesh Unstructured mesh 1-D partition 3-D partition Without parallelization 1-D partition 3-D partition Without parallelization 6 min 158 min 119 min 7 min 159 min 1 min 4 x tructured mesh Unstructured mesh Z component of Displacement 1 Figure 9. Local mesh in the vicinity of the contacting boundaries of two subdomains under the unstructured mesh. The width between the contacting boundaries is created for visualization. the grid cells with too long an edge (e.g. greater than 9 m). Again, two kinds of partitions of the computational domain, that is, 1-D and 3-D partitions, are tested. A local mesh in the vicinity of the contacting boundary of two subdomains is shown in Fig. 9. It is seen that the boundary exhibits are obviously irregular. Owing to using tetrahedral grid cells and allowing a curved, irregular contacting boundary, we still can make the volume of each subdomain and the number of the nodes on each contacting boundary approximately equal. This is very helpful to gain a higher efficiency of the parallel computation. The computation times corresponding to those of the structured mesh for 3 time steps are also listed in Table 3. A high speed-up rate, 94.3 per cent under the 3-D partition, is still achieved in this unstructured mesh. It is found from Table 3 that the unstructured mesh does not degrade the computational efficiency of the proposed parallel algorithm. The comparison of the z-direction component of the displacement at the same position between the results using the structured and unstructured meshes is shown in Fig. 1. The receiver station is just 5 m below the source. Two results agree well. This demonstrates that small errors arise when the proposed parallel algorithm uses an unstructured mesh, only if the mesh is generated in the absence of the grid cells with too uneven edges. 6.5 Isotropic/anisotropic interface This example presents wave propagation across an isotropic/ anisotropic interface. Both horizontal and inclined interfaces are considered. The isotropic medium has a -wave velocity of 3 ms 1,an-wave velocity of 14 ms 1 and a density of 3 kg m 3. The five modified Lamé parameters and density of the anisotropic (transversely isotropic) medium are the same as those listed in Table. The symmetry axis of the transversely isotropic medium is chosen to coincide with the normals to the interfaces for both mod Time (s) Figure 1. Comparison between results using the structured and unstructured meshes. The plots are the z-direction component of the displacement for the receiver just 5 m below the source. A good agreement is observed. els. This means that we can make the inclined interface model equivalent to the horizontal one by appropriate rotations. As a result, the performance of the proposed numerical technique in handling an irregular interface can be assessed by comparing the two resulting results. Fig. 11 shows a 3-D snapshot of the x-direction component of the displacement for the inclined interface model at.7 s propagation times, and Fig. 1 shows the yz-plane slice (passing through the source) of the snapshot of the z-direction component of the displacement for the horizontal interface model at.63 s propagation time. The inclined and horizontal interfaces are represented by the white lines in Figs 11 and 1. The same Ricker wavelet pressure source, with a peak frequency of Hz, is positioned at the isotropic medium for both models, as represented by point. Fig. 13 further shows a comparison between the z-direction component of the displacement of the horizontal interface model with the component of the displacement normal to the inclined interface of the inclined interface model at point R (as shown in Fig. 1) and the equivalent point in Fig. 11 (an inner node determined by the rotations). The snapshot in Fig. 11 shows clear wave fronts of the direct wave and reflected and waves in the isotropic medium, and the transmitted quasi- and quasi- waves in the anisotropic medium. Fig. 1 shows the similar wave fronts in a -D slice. Two results agree well in Fig. 13. This demonstrates that the proposed numerical algorithm handles the irregular interface as well as the transversely isotropic medium with a tilted symmetry axis correctly. The numerical model is composed of nodes with an even spatial spacing of 5 m for the horizontal interface model and 5. m for the inclined one. The temporal interval used is both.7 ms. Both numerical meshes are made up of tetrahedral grid cells, which are generated through splitting each right cube into five tetrahedrons. However, special effort is needed to make the facets of C 6 The Authors, GJI, 165, on 18 November 17

10 884 H. Gao and J. Zhang 5 1 Y-axis(m) X-axis(m) Component erpendicular to Interface 3 x Horizontal Inclined Time (s) 15 Figure 11. napshot of the x-direction component of the displacement for the inclined interface model at.7 s propagation time. The white lines shown in the snapshot denote the inclined interface. The upper-right medium containing source is the isotropic medium. The direct wave and reflected and waves in the isotropic medium, and the transmitted quasi- and quasi- waves in the anisotropic medium can be seen. Figure 13. Comparison between results of the horizontal and inclined interface models. The plots are the z-direction component of the displacement for the horizontal interface model and the component of the displacement normal to the inclined interface for the inclined interface model. A good agreement is observed. Y-axis (m) R Figure 14. Local mesh in the vicinity of the hemispherical crater. The unstructured mesh in this vicinity is constructed by the tetrahedral grid cells with the minimum edge of 6.56 m. Note that the edges of grid cells are more or less constant. Figure 1. yz-plane slice (passing through the source) of the snapshot of the z-direction component of the displacement at.63 s propagation time for the horizontal interface model. The white line denotes the horizontal interface, point the source, and point R the receiver for the comparison in time history between the horizontal and inclined interface models. the tetrahedrons follow the inclined interface in the generation of the numerical mesh of the inclined interface model. 3 subdomains are obtained by slicing in three spatial directions and the computation is then performed on a 3-node C-Cluster. 6.6 Hemispherical crater To test the proposed scheme s effectiveness in accounting for the 3-D surface topography, we present an example of wave propaga- tion in an anisotropic half-space with a hemispherical crater on the free surface. The radius of the hemispherical crater is 15 m. The anisotropic half-space is a transversely isotropic medium with a tilted symmetry axis of θ = 3 and φ = 4. Its five modified Lamé parameters and density are the same as those listed in Table. A vertical Ricker wavelet point source, with a peak frequency of 1 Hz, is positioned at the free surface with a distance away the centre of the hemispherical crater of 15 m. The numerical mesh is made up of tetrahedral grid cells in the vicinity of the hemispherical crater and surface and right cube grid cells in other regions. An unstructured mesh is used in the vicinity of the hemispherical crater, as illustrated in Fig. 14. The mesh accurately models the 3-D surface topography and the edges of grid cells are more or less constant. Moreover, no extra points are introduced for the correct description of the hemispherical crater. The minimum edge of all the tetrahedral grid cells is 6.56 m and the edge of the right cube grid cells is 7.5 m. The used time step is 1. ms. The number of total C 6 The Authors, GJI, 165, on 18 November 17

11 arallel 3-D modelling for anisotropic wave propagation Y-axis (m) 5 X-axis (m) The diffracted waves resulting from the hemispherical crater can be seen clearly. Also, the shear-wave splitting in the internal domain is observed. The absorbing boundaries are positioned at the minimum x and the maximum y and z. The good behaviour of the proposed paraxial absorbing boundary conditions is again demonstrated by the observation that no reflections result from those boundaries. Owing to its shorter wavelength, the diffraction of Rayleigh wave is much stronger than that of the direct wave when propagating through the hemispherical crater, as seen in Fig Y-axis (m) 5 X-axis (m) Figure 15. napshots of the z-direction component of the displacement at.65 (top) and 1.3 s (bottom) propagation times for the hemispherical crater model. The xy-plane denotes the free surface, the white lines the hemispherical crater on the free surface, and point the source. Note the diffracted waves resulting from the hemispherical crater. The shear-wave splitting can also be observed. nodes is approximately subdomains are obtained by a 3-D partition of the computational domain. The subdomains that contain tetrahedral grid cells should have smaller volume because the computational cost for a right cube grid cell is much less than that for five tetrahedral grid cells. The computation is performed on a 3-node C-Cluster. 3-D snapshots of the z-direction component of the displacement at.65 and 1.3 s propagation times are shown in Fig. 15. Here, the xy-plane denotes the free surface, the white lines the hemispherical crater on the free surface, and point the source. The snapshots show very clear wave fronts of the Rayleigh wave, the direct wave on the surface, and the quasi- and quasi- waves in the internal domain. 7 CONCLUION We have presented a parallel algorithm for modelling seismic wave propagation in 3-D heterogeneous anisotropic media including surface topography. This is an extension of the 3-D grid method to account for anisotropy and parallel simulation. The proposed anisotropic scheme is developed based on the integral approach of the 3-D elastic momentum equations for a discretization of mixed tetrahedral and parallelepiped grid cells. Owing to the special algorithm architecture in the proposed anisotropic scheme, that is, calculations are performed within each grid cell together with a simple accumulation to its nodes, the resulting parallel algorithm needs small data exchanges between subdomains with a flexible partition of the computational domain. The proposed parallel algorithm is, therefore, more suitable to be implemented on a C-Cluster with the use of a large number of processors. The proposed scheme can accurately model the 3-D surface and interface topographies using an unstructured mesh. Almost no extra nodes are introduced for a correct description of the surface and interface topographies. Furthermore, the scheme naturally satisfies the free-surface boundary conditions of arbitrary 3-D surface topography. A paraxial absorbing boundary conditions suited for a general 3-D anisotropic medium is also developed and incorporated into the proposed parallel algorithm. The proposed parallel algorithm can serve as a powerful tool for the study of wave propagation phenomena in 3-D heterogeneous anisotropic media, especially the near-surface scattering effects. Comparisons with the analytical solutions validate the algorithm. The scalability tests under both structured and unstructured meshes illustrate the efficiency of the parallel computation. The numerical simulations on a C-Cluster for radiation patterns, wave propagation across isotropic/anisotropic interfaces and surface wave propagation through a hemispherical crater demonstrate its high qualities. ACKNOWLEDGMENT Thanks to the National Natural cience Fund for Distinguished Young cholars of China (under grant 451) and National Natural cience Fund of China (under grant ) who supported this work. This work is also supported by the National Basic Research rogram of China (under grant 5CB414). The comments of two anonymous reviewers helped improve the manuscript. REFERENCE Bao, H., Bielak, J., Ghattas, O., Kallivokas, L.F., O Hallaron, D. R., hewchuk, J.R. & Xu, J., Large-scale simulation of elastic wave propagation in heterogeneous media on parallel computers. Comp. Meth. appl. Mech. Eng., 15, Booth, D.C. & Crampin,., The anisotropic reflectivity technique: theory. Geophys. J. Roy. Astr. oc., 7, C 6 The Authors, GJI, 165, on 18 November 17