Multiscale simulations of Langmuir cells and submesoscale eddies using XSEDE resources

Transcription

1 Multiscale simulations of Langmuir cells and submesoscale eddies using XSEDE resources Luke P. Van Roekel Northland College 1411 Ellis Avenue Ashland, Wisconsin Peter E. Hamlington Department of Aerospace Engineering Sciences University of Colorado Boulder, CO Baylor Fox-Kemper Cooperative Institute for Research in Environmental Sciences University of Colorado Boulder, CO ABSTRACT A proper treatment of upper ocean mixing is an essential part of accurate climate modeling. This problem is difficult because the upper ocean is home to many competing processes. Vertical turbulent mixing acts to unstratify the water column, while lateral submesoscale eddies attempt to stratify the column. Langmuir turbulence, which often dominates the vertical mixing, is driven by an interaction of the wind stress and surface wave (Stokes) drift, while the submesoscale eddies are driven by lateral density and velocity changes. Taken together, these processes span a large range of spatial and temporal scales. They have been studied separately via theory and modeling. It has been demonstrated that the way these scales are represented in climate models has a nontrivial impact on the global climate system. The largest impact is on upper ocean processes, which filter air-sea interactions. This interaction is especially interesting, because it is the interface between nonhydrostatic and hydrostatic, quasigeostrophic and ageostrophic, and small-scale and large-scale ocean dynamics. Previous studies have resulted in parameterizations for Langmuir turbulence and submesoscale fluxes, but these parameterizations assume that there is no interaction between these important processes. In this work we have utilized a large XSEDE allocation (9 million SUs) to perform multiscale simulations that encompass the Langmuir scale (O(10-100m)) and submesoscale eddies (O(1-10km)). One simulation includes a Stokes drift, and hence Langmuir turbulence, while the other does not. To adequately represent such disparate spatial scales is a challenge in numerous regards. Numerical prediction algorithms must balance efficiency, scalability, and accuracy. These simulations also present a large challenge for data storage and transfer. However, the results of these simula- This author will present Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. XSEDE12, July , Chicago, Illinois, USA Copyright 2012 ACM /12/07...$ tions will influence climate modeling for decades. Categories and Subject Descriptors J.2 [Physical Sciences and Engineering]: Earth and atmospheric sciences General Terms Experimentation, Algorithms, Performance Keywords Upper ocean dynamics, mixing, Langmuir turbulence, Submesoscale eddies, Multiscale simulations 1. INTRODUCTION Coupled ocean-atmosphere simulations conducted for the most recent report from the Intergovernmental Panel on Climate Change (IPCC) are only able to resolve processes occurring at scales larger than the mesoscale, O(100km). Any processes that have smaller length scales must be parameterized. While not all processes below the mesoscale contribute significantly to the large-scale climate, there are indications that some have a significant impact on the dynamics and structure of the oceanic mixed layer. The ocean surface (mixed) layer, O(100m) deep, is the intermediary for fluxes of momentum and tracers between the ocean and atmosphere. This suggests that incorrect representation of these small-scale processes that occur in the upper ocean will have an adverse impact on IPCC class simulations. The features simulated in this research span a range of scales from tens of meters to about 10 km. While this range of scales may seem small, the behavior at the ends of these scales is very different. At the smallest scales, Langmuir turbulence (LT) dominates with its three-dimensional, non-hydrostatic motions. On the opposite end of our scales of interest, submesoscale fronts and instabilities dominate. Such submesoscale phenomena have vertical motions that are much smaller than their horizontal counterparts. A number of studies have considered LT and submesoscale fronts and instabilities (including their impact on the largescale climate) separately [34, 7], but none have yet to examine the possible interactions between Langmuir turbulence and submesoscale features. Langmuir turbulence is created when vertical vorticity anomalies are tilted into the horizontal by wave induced (Stokes) drift, whereas submesoscale

2 Figure 1: Windrows evident from streaks of orange oil in the visual spectrum after the Deep Water Horizon oil spill. Note the size of the boats for scale, the estimated windrow spacing is m. Image courtesy of npr.org features could be thought of as vertical vorticity anomalies themselves. This suggests that some type of interaction should be expected. There are many important questions to be answered about this interaction. A few would include: (1) Do submesoscale eddies alter LT? (2) Does vertical mixing by Langmuir turbulence affect the formation of submesoscale eddies? (3) What are the relative contributions of LT and submesoscale processes in the structure and transport of momentum and tracers within the mixed layers. A detailed examination of these questions is considered in [11]. Here we focus on numerical details with a brief discussion of preliminary results. In Section 2, the basic physics of the two phenomenon considered in the simulations are discussed. Section 3 discusses the planning and implementation of these grand challenge simulations. This includes the algorithms used and the benefits of XSEDE resources and software. Finally, Section 4 includes a discussion of the basic results of the simulations. 2. PHYSICS OF LANGMUIR CELLS AND SUBMESOSCALE PHENOMENA 2.1 Langmuir turbulence and Langmuir cells Langmuir cells (LC) are small overturning cells ( m wide) that form in the upper ocean. Often, windrows of surface debris form at the convergence zones of these overturning cells (Figure 1). After the Deep Water Horizon oil spill, streaks of orange oil were evident in the visual spectrum. The spacing of the Langmuir cells can be estimated by comparing them to the ships evident in the photograph. Figure 1 shows a m spacicng of the LC. LC are formed when the Stokes drift converts vertical vorticity, which is often created by gradients in the surface (wind-driven) current, into horizontal vorticity. This is evident in Figure 2. Small variations in the wind and hence in the transfer or momentum between the atmosphere and ocean lead to a small jet (local maximum of current). This gradient in currents leads to two vertical vorticity patches on either side of the peak. The Stokes drift, which in this Figure 2: Schematic illustrating the formation of Langmuir cells. The red arrows represent surface currents. The blue arrow is the surface Stokes drift. The wind is assumed to point in the direction of the Stokes drift and currents. case is assumed to be in the same direction as the wind, tilts the vertical vorticity into the horizontal. These horizontal vortex tubes are LC. This mechanism suggests that when the Stokes drift does not project in the direction of the wind stress, LC will not form. Recent results [33] suggest that the misalignment between the wind and waves is an important consideration. When data is projected into the direction of LC, many important scalings are recovered. Instead of assuming that the wind and waves point in the same direction, one could assume that the wind and waves project into a third direction, that of the LC. In this framework, a similar mechanism as discussed above can proceed. In the multiscale simulations considered, there is a separation between wind and waves. Recent literature has offered conflicting views on the role of LT in setting the depth of the mixed layer. Some [22, 13, 3] show rapid deepening in the presence of LC, while others [36] do not. A recent study [35], has shown that the Community Climate System Model (CCSM) is sensitive to a very simple LC parameterization. The global mean mixed layer depth (MLD) in CCSM, with the LC parameterization, deepened substantially ( 10%). In particular, the simulated Southern ocean MLD bias dramatically improved. However, it is important to note that this parameterization is very sensitive and will benefit from future improvements. One goal of the simulations conducted here is to better represent the effects of LC mixing in many different environments to improve current parameterizations. Despite this large sensitivity, recent results [35, 20] strongly suggest that LT is often an important process in upper ocean dynamics. 2.2 Submesoscale processes The surface layer of the ocean is often assumed to be fully mixed in the vertical and horizontal directions. However, the weak upper ocean horizontal temperature (buoyancy) gradient supports instabilities. Imagine that a storm (e.g. a hurricane) has recently passed over a section of the ocean. The

3 might influence one another. As previously mentioned, LC are the result of wave drift tilting vertical vorticity anomalies into the horizontal and mixed layer eddies (MLEs) are vertical vorticity anomalies themselves! This suggests that the existence of MLEs should influence the development of LC. On the other hand, the growth of MLEs is very sensitive to the velocity shear in which they grow, and LC and the Stokes drift shear that helped create them alter the near surface shear. Understanding these interactions will help guide future parameterizations and determine whether the MLE and LT parameterizations should balance, coexist, cancel, etc. 3. NUMERICS Figure 3: Photo of the Deep water horizon oil spill taken from the International Space Station on May 4, Numerous frontal instabilities (mesoscale and submesoscale) are evident as a reduced glitter in the figure. The size of the Mississippi delta ( 25 km) is given for reference. The largest eddies (mesoscale) are approximately equivalent in size to the delta. The submesoscale are the smaller ( 5 km) eddies. Image courtesy of NASA Earth Observatory resulting temperature field is vertically homogeneous in the upper ocean, but horizontal gradients exist. Soon after the passage of the storm, restratification (slumping of the vertical isotherms) begins. This could be accomplished simply by incident surface solar heating. In addition to thermal restratification, dynamical processes can slump the isotherms. This begins as a simple response to gravity. After the slumping begins, rotation (i.e. the Coriolis force) becomes important and geostrophic adjustment occurs. This is not the end of the story, since it has been shown [2, 9, 8] that these geostrophically balanced fronts can become baroclinically unstable to submesoscale eddies (see Figure 3). Here, many instabilities are evident as reduced glitter near the Mississippi delta after the Deep Water Horizon oil spill. In this figure, the largest eddies, which are approximately equal in size to the Mississippi delta are mesoscale (approximately 25 km). The smaller eddies in Figure 3 are submesoscale (approximately 5 km across). These smaller eddies do the bulk of the restratification. Despite this evidence, processes at the submesoscales have been assumed to be of secondary importance. A number of studies [24, 10, 31, 6] have shown, however, that submesoscale eddies are often of primary importance in the mixed layer budgets of momentum and tracers. Inclusion of a parameterization of the restratification due to submesoscale eddies has been shown to reduce the deep bias in climate model predictions of mixed layer depth [7]. 2.3 Coupling Langmuir turbulence and submesoscale eddies In essence, our interest is an examination of the balance between the destratifying effects of LT and the restratifying effects of submesoscale (or more generally mixed layer) eddies. As both processes can exist in the same portion of the ocean, this raises the question of how these two processes 3.1 Simulation design The multiscale simulations have been carried out on Kraken, a Cray XT5 with 112,896 compute cores. Even though Kraken is ideally suited for these large multiscale simulations and we were given a large allocation, we wanted to make the most of every cpuhr. Thus, many steps were taken to choose the most useful forcing parameters and domain size. A double front configuration was chosen (Figure 4). In the domain, which we assume is doubly periodic, the fronts have to be spaced far enough apart so that there will be no interaction between them during the simulation (10-12 days). The wind stress was chosen to have a slight cross front component (30 o ) 1. The wind stress is rotated 30 o counter-clockwise from the x-direction. When a cross frontal wind is included there are two resulting effects. First, the wind will directly advect the fronts. Second, the Ekman transport will cause one front to become unstable (cold water advected on top of warm) and the other to be restratified. The wind stress must be chosen such that the two fronts do not interfere with each other until the submesoscales become sufficiently large (approximately days). However, the wind stress is not the only parameter that will influence the time it takes for the two fronts to interfere with each other. The fastest growing submesoscale mode has length, time, and velocity scales [25, 10] L s = 2π k s τ(k s) = = 2πU f Ri 5/2, (1) 1 + Ri, (2) f U = M 2 H, (3) f Ri N 2 U 2 z. (4) 1 The total (Lagrangian) current can be written as u L = u + u S, where u is the Eulerian current and u S is the Stokes drift. Initial theoretical work suggested that the Lagrangian wind should obey a Lagrangian thermal wind balance (i.e. u LT z ˆk b, where b is the buoyancy), however, recent theoretical work has suggested that this approximation is only valid for weak values of Stokes drift. Further, this choice does set these simulations apart from recent work by Skyllingstad and colleagues [21]

4 y u * us Figure 4: Schematic illustrating the surface conditions of the multiscale simulations. The red box is warm temperature and the blue boxes are cold. The domain will be assumed to be doubly periodic. The wind (green arrow) is 30 o counter-clockwise from the x-direction. When a Stokes drift is included (blue arrow), it is in the direction of the wind stress. Using these characteristic scales, it is evident that the mixed layer depth, H, horizontal buoyancy gradient, M 2 (2f) 2, and vertical stratification, N 2 b, all set the z time that will elapse before the two fronts interfere with each other. Prior to making a final selection of parameters, another criterion is applied. In our multiscale simulations, the buoyancy fluxes due to the submesoscale instabilities should be approximately equal to the Langmuir scale fluxes. A relation can be derived by considering previous scalings for the Langmuir scale [15] and submesoscale [9] fluxes. Setting these two fluxes equal yields u 2 S = (1.875 M ) 4 H 2 1/ (5) u 2 fn 2 u If M 2, H, N 2, and u can be chosen in some way, equation 5 can be used to determine the strength of the wave induced drift that would result in LC nearly balancing submesoscale eddies. To choose the initial parameters, a large number of single-scale simulations were conducted with the MITgcm. In these simulations, the submesoscales were resolved, but the Langmuir scales were ignored. The values of M 2, N 2, and H were chosen such that a sufficient number of eddies (7-10) fit in a 20 kilometer box throughout the single-scale simulation. The wind stress had to be chosen such that the submesoscales on each front would not interfere with each other for at least days. Different values of M 2 and N 2 are theoretically possible. A specific value was chosen by forcing these values to be close to observed (i.e. realistic) values. The resulting parameters from these simulations are listed in Table 1. In this table, L x, L y, L z are the domain sizes in the x-, y-, and z-directions respectively and N x, N y, and N z are the number of points in each direction. These values have x been chosen to at least minimally resolve Langmuir cells. The simulations considered here have a horizontal resolution of 4.85 m and a vertical resolution of 1.25 m. 3.2 Numerical algorithms The simulations are performed with a numerical model that solves the Craik-Leibovich equations with the Boussinesq assumption. The equations are [12] u t + (ω + f ˆk) u L = π + bˆk, (6) b z + u L b = 0, (7) u = 0, (8) where ω u is the Eulerian vorticity, f is the Coriolis parameter, π p + 1 u 2 L 2 is the generalized pressure per unit density [16], and u L is the Lagrangian velocity. As done under the Boussinesq approximation, the buoyancy is defined as b g(ρ ρ o)/ρ o. Salinity is not considered. The only difference between this equation set and standard Boussinesq equation sets is the presence of a Stokes velocity in the advection and generalized pressure term. The Stokes drift is assumed to be horizontally uniform and decays superexponentially with depth [5]. The NCAR large eddy simulation (LES) model [17, 27, 16, 26] is used to solve equations (6) - (8). The pressure field required to make the flow divergence-free is computed as the solution of the discrete Poisson equation. This is derived by applying a low-pass spatial filter to the equations of motion. Thus, all fields can be decomposed into a resolved and a subgrid component. The subgrid scale temperature and momentum fluxes are cast in terms of an eddy diffusivity and viscosity that are proportional to the square root of the subgrid turbulence kinetic energy. The subgrid closure has been described in great detail by previous authors [4, 17, 18, 27]. The equations (7) - (8) are stepped in time using a third order Runge Kutta discretization. The vertical discretization uses a second order finite difference scheme for momentum and a third order monotone scheme for tracers. The horizontal derivatives are computed spectrally. The vertical flux of potential temperature is determined via a nearly monotone second-order scheme [1]. This prevents unrealizable oscillations in regions of strong vertical temperature gradients. The time stepping is accomplished with a thirdorder Runge-Kutta scheme with fixed Courant number [23, 28]. This model was chosen due to its high degree of scalability. Computing horizontal derivatives with fast fourier transforms (FFTs) and solving the Poisson equation for pressure involves a large degree of MPI communication, and hence special consideration. The parallelization of the NCAR LES model is based on three important criteria. First, the twodimensional domain decomposition should be accomplished with MPI only. Second, pseudospectral differencing in x-y planes using FFTs should be preserved. Finally, incompressible Boussinesq conditions must be maintained. Satisfaction of criterion one is a major advantage and allows numerical simulations on meshes of or more [19]. The domain can be split in y and z slabs. Each processor is then responsible for a three dimensional brick. Communication between these slabs involves transposes and the exchanges of ghost points.

5 Table 1: Parameters for the multiscale simulations L x(km), L y(km), L z(m) M 2 (s 2 ) N 2 (s 2 ) H(m) u (ms 1 ) u s(ms 1 ) f(s 1 ) N x, N y, N z 20, 20, , 4125, 128 The NCAR LES model has been designed such that the transpose operation only requires local communication; that is, communication is between neighboring processors only. Meridional derivatives (i.e. f/ y) are computed by performing a transpose from x to y, then the derivative is computed in spectral space. The result is then returned to the proper orientation and physical space. All FFTs are carried out via FFTPACK (version 5). Details have been thoroughly described in [30]. In this model, no global MPI communication is required. This requires more communication, but messages are smaller. This allows for improved scalings, especially when there are many more horizontal points than vertical points [29]. The strong and weak scaling has been assessed on machines similar to Kraken [29]. For these simulations, only the strong scaling is assessed. For up to 3000 processors a near linear scaling is found. For a doubling of processors a speed-up of approximately 1.7 is realized. Thus the NCAR LES model is well suited to these multiscale simulations. The NCAR LES model outputs seven prognostic variables. This implies that 3D output volumes will create GB output files. This presents two challenges: movement and storage. The former problem has proven much more tricky than the latter, which was solved by use of hpss at NICS. Standard scp transfer rates between XSEDE resources and our home institution averaged about MBPS. This implies that transfetring a single output file takes more than 24 hours to complete. During each 24 hour wall clock period, four three-dimensional volumes are generated. Using scp would be impossible. Fortunately, gridftp, and globusconnect in particular, have essentially removed this bottle neck. Using gridftp transfer, files are transferred at rates that exceeded 10 times that of scp. These rates were increased further when files were transferred from Kraken to Nautilus for initial processing and analysis. Analysis of the 3D volumes required additional consideration. Often, array sizes in fortran are limited to 4 GB. Each individual variable in our output volume far exceeds this maximum. Therefore, special routines had to be developed that utilize MPI to accomplish the analysis. Code has been written to plot subsections of the 3D volume (horizontal and vertical slices). The code can also compute average profiles. To attempt to separate the Langmuir scale from the submesoscale in our results, two types of FFTs are applied. Despite the presence of a front in the y-direction, we utilize a two-dimensional, spherically symmetric, FFT at every level and output time. To account for inhomogeneity in the y-direction, a one-dimensional (x-direction) FFT is also carried out. To separate the two scales, a critical wave number is chosen. 4. GENERAL RESULTS The wind blowing across the temperature fronts at 30 relative to the x-axis will have opposite effects on each front. Note that the figures discussed in this section are transposed relative to Figure 4. The x-axis now runs vertically. The origin is the top left corner. The wind blows from top left Figure 5: Horizontal slices of temperature (Kelvin). (a) A simulation without Stokes drift (day 11.4) (b) A simulation with Stokes drift (day 11.1). (c) As in (a) but zoomed into the black box region of (a). (d) As in (c) but for the run with Stokes drift. to bottom right. This causes an Ekman transport directed toward the bottom left. On this side cold water will be advected over warm water. Here we expect strong instabilities and growth of submesoscale features [32, 14]. On the other front we expect minimal eddy growth. This is clearly shown in Figure 5, which show the surface temperature field for the shear only (Figure 5a) and Stokes (Figure 5b) simulations. These figures are transposed relative to Figure 4. At first glance, it appears that the temperature fields are similar. On closer inspection, however, when a wave-induced drift is included, the variance of temperature at smaller scales is slightly reduced. This is more evident when zooming in tight (Figure 5(c) and (d)). There seems to be slightly more detail in the simulation without Stokes drift. Despite the slightly reduced variability in small scale temperature, the eddies appear more coherent in the run with waves included (compare Figures 5(a) and (b)). Vertical velocity fields near the depth of maximum vertical velocity variance show a dramatic increase in small scale variability when waves are considered (Figure 6(b)). When focusing in tighter, as shown in Figures 6(c) and (d), upward and downward patches of vertical velocity show the prevalence of Langmuir cells. Note that, in these figures, the boxes contain a portion of the submesoscale front. The

6 Figure 6: As in Figure 5 but for vertical velocity. strong region of blue (downward motion) cutting through Figure 6(d) is not directly a result of Langmuir cells. There are regions near the submesoscale activity of very small vertical velocity in Figure 6(a) that are not evident in 6(b). It appears that submesoscale eddy activity has a strong impact on small scale vertical velocity. However, it appears that LT counters this influence. Since the Stokes drift decays superexponentially with depth, we expect (and have found) that suppression of small scale vertical velocity does occur sufficiently far below the surface in the simulation with Stokes drift (not shown). In order to to separate the impacts of the larger, submesoscale, features and those due to LT, low-pass and high-pass filtered fields are examined. Figure 7 shows the filtered vertical velocity for the shear only (Figure 7(a) and (c)) and Stokes drift (Figure 7(b) and (d)) simulations. In these figures, the critical wave length is chosen to be 1km. Thus, the low-pass filtered data contains features larger than a kilometer. There appears to be a slightly larger energy content at submesoscales in the run with shear only (Figure 7(a)). This suggests that Langmuir turbulence might have an impact on the submesoscales. As discussed previously, there is a slight reduction of temperature variance at small scales. This is now easily seen through the scale decomposition (compare Figures 7(c) and (d). 5. CONCLUSIONS AND FUTURE DIRECTIONS Using XSEDE resources we have conducted two of the first ever multiscale simulations that resolve Langmuir turbulence and submesoscale eddies. Attempting such simulations is truly a grand-challenge. The NCAR LES model was an important choice due to its scalability, especially in cases Figure 7: Low- (a)-(b) and High- (c)-(d) pass filtered temperature for the runs with no Stokes drift (a) and (c), and with Stokes drift (b) and (d). In these plots, the critical wave length is set to 1km. where there are far more points in the horizontal direction relative to the vertical. The code is also highly portable and very few SUs had to be consumed in porting the code. The size of Kraken allowed for fairly quick throughput (limited queue wait times) and gridftp proved to be essential for data analysis. Initial experiences at another supercomputing center without gridftp in place illustrated how important this tool is to these simulations. Data storage issues, which initially seemed to be a very big issue, have not proved to be difficult. The analysis software we have developed (locally and on Nautilus) allow for parallel processing of data. Our initial results are tantalizing. It appears that there are indeed interactions between Langmuir turbulence and submesoscale eddies. For example, it appears that LT slightly suppresses small scale temperature variability near the surface. Submesoscale activity suppresses vertical velocity near the unstable front, but Langmuir turbulence counteracts this influence. It also appears that Langmuir turbulence decreases larger scale energy (vertical velocity). These initial results lead to more questions than answers. Detailed structures near the front have not been compared to those far from the front. There are a large number of instability parameters to examine. For example, the simulation containing a Stokes drift may contain instabilities that are not evident in runs without LT. It is also possible that flow quantities are mixed differently between the two simulations. Momentum (water currents) may be thoroughly mixed to one depth in the run with Stokes drift and to a different depth in the run neglecting Stokes drift. The simulations shown here include a small surface cooling. However, the scaling argument that set the buoyancy

7 flux from Langmuir turbulence to that due to submesoscale eddies (equation 5) assumes no surface buoyancy flux. This will have an impact on the relative strengths of LT and submesoscale eddy fluxes. Understanding the interaction between these two processes is essential to an improved simulation, and understanding of, the large-scale climate system. A subset of these questions are being addressed in other publications [11]. These questions are just the tip of the iceberg. The results of these (and the following related) simulations will impact ocean, and hence climate, modeling for years. 6. ACKNOWLEDGMENTS The authors wish to acknowledge the support of NSF CMG grant A large portion of this research was supported by an allocation of advanced computing resources provided by the National Science Foundation. The computations were performed on Kraken and Nautilus. We also acknowledge high-performance computing support provided by NCAR s Computational Information Systems Laboratory and resources provided by NSF-MRI Grant CNS , with additional support from the University of Colorado. 7. REFERENCES [1] A. K. Beets and B. Koren. Large-eddy simulation with accurate implicit subgrid-scale diffusion. Dept. numerical mathematics report nm-r9601, Utrecht University, The Netherlands, [2] G. Boccaletti, R. Ferrari, and B. Fox-Kemper. Mixed layer instabilities and restratification. Journal of Physical Oceanography, 37(9): , [3] E. A. D Asaro. Turbulent vertical kinetic energy in the ocean mixed layer. Journal of Physical Oceanography, 31(12): , Dec [4] J. W. Deardorff. Stratocumulus-Capped Mixed Layers Derived From A 3-Dimensional Model. Boundary-Layer Meteorology, 18(4): , [5] M. A. Donelan, J. Hamilton, and W. H. Hui. Directional Spectra of Wind-Generated Waves. Philosophical Transactions of the Royal Society of London Series A-Mathematical Physical and Engineering Sciences, 315(1534): , [6] R. Ferrari and D. L. Rudnick. Thermohaline variability in the upper ocean. Journal of Geophysical Research-Oceans, 105(C7): , July [7] B. Fox-Kemper, G. Danabasoglu, R. Ferrari, S. M. Griffies, R. W. Hallberg, M. M. Holland, M. E. Maltrud, S. Peacock, and B. L. Samuels. Parameterization of mixed layer eddies. III: Implementation and impact in global ocean climate simulations. OCEAN MODELLING, 39(1-2, SI):61 78, [8] B. Fox-Kemper and R. Ferrari. Parameterization of mixed layer eddies. Part II: Prognosis and impact. Journal of Physical Oceanography, 38(6): , [9] B. Fox-Kemper, R. Ferrari, and R. Hallberg. Parameterization of mixed layer eddies. Part I: Theory and diagnosis. Journal of Physical Oceanography, 38(6): , [10] T. W. N. Haine and J. C. Marshall. Gravitational, Symmetric and Baroclinic Instability of the Ocean Mixed Layer. Journal of Physical Oceanography, 28: , [11] P. E. Hamlington, L. P. Van Roekel, and B. Fox-Kemper. Langmuir-Submesoscale Interactions : Multiscale Simulations with the Craik-Leibovich Equations. Journal of Geophysical Research-Oceans, In prep. [12] D. D. Holm. The ideal Craik-Leibovich equations. PHYSICA D, 98(2-4): , Nov [13] M. Li and C. Garrett. Is Langmuir Circulation driven by surface-waves or surface cooling? Journal of Physical Oceanography, 25:64 76, [14] A. Mahadevan, A. Tandon, and R. Ferrari. Rapid changes in mixed layer stratification driven by submesoscale instabilities and winds. J. Geophys. Res., 115:C03017, [15] J. C. McWilliams and P. P. Sullivan. Vertical Mixing by Langmuir Circulations. Spill & Science Technology Bulletin, 6: , [16] J. C. McWilliams, P. P. Sullivan, and C.-H. Moeng. Langmuir turbulence in the ocean. Journal of Fluid Mechanics, 334:1 30, [17] C. H. Moeng. A Large-Eddy-Simulation Model For The Study Of Planetary Boundary-Layer Turbulence. Journal of the Atmospheric Sciences, 41(13): , [18] C. H. Moeng and J. C. Wyngaard. Spectral-Analysis Of Large-Eddy Simulations Of The Convective Boundary-Layer. Journal of the Atmospheric Sciences, 45(23): , Dec [19] D. Pekurovsky, P. K. Yeung, D. Donzis, W. Pfeiffer, and G. Chukkapalli. Scalability of a pseudospectral DNS turbulence code with 2D domain decomposition on Power4+/Federation and Blue Gene systems. In ScicomP12 and SP-XXL, Boulder, CO, [20] F. Qiao, Y. Yuan, T. Ezer, C. Xia, Y. Yang, X. Lü, and Z. Song. A three-dimensional surface wave ocean circulation coupled model and its initial testing. Ocean Dynamics, 60: , [21] E. D. Skyllingstad and R. M. Samelson. Large-Eddy Simulations of Baroclinic Instability and Turbulent Mixing, [22] J. A. Smith. Evolution of Langmuir circulation during a storm. Journal of Geophysical Research-Oceans, 103: , [23] P. R. Spalart, R. D. Moser, and M. M. Rogers. Spectral Methods For The Navier-Stokes Equations With One Infinite And 2 Periodic Directions. Journal of Computational Physics, 96(2): , Oct [24] M. Spall. Baroclinic jets in confluent flow. Journal of Physical Oceanography, 27: , [25] P. H. Stone. On Non-Geostrophic Baroclinic Stability. Journal of the Atmospheric Sciences, 23: , [26] P. P. Sullivan, J. C. McWilliams, and W. K. Melville. The oceanic boundary layer driven by wave breaking with stochastic variability. Part 1. Direct numerical simulations. Journal of Fluid Mechanics, 507: , [27] P. P. Sullivan, J. C. McWilliams, and C. H. Moeng. A Subgrid-Scale Model For Large-Eddy Simulation Of

8 Planetary Boundary-Layer Flows. Boundary-Layer Meteorology, 71(3): , Nov [28] P. P. Sullivan, J. C. McWilliams, and C. H. Moeng. A grid nesting method for large-eddy simulation of planetary boundary-layer flows. Boundary-Layer Meteorology, 80(1-2): , July [29] P. P. Sullivan and E. G. Patton. A highly parallel algorithm for turbulence simulations in planetary boundary layers: Results with meshes up to {1024ˆ3}. In 18th Conference on Boundary Layer and Turbulence, Stockholm, Sweden, [30] P. P. Sullivan and E. G. Patton. The Effect of Mesh Resolution on Convective Boundary Layer Statistics and Structures Generated by Large-Eddy Simulation. Journal of the Atmospheric Sciences, 68(10): , [31] L. N. Thomas. Destruction of potential vorticity by winds. Journal of Physical Oceanography, 35: , [32] L. N. Thomas and C. M. Lee. Intensification of ocean fronts by down-front winds. 35: , [33] L. P. Van Roekel, B. Fox-Kemper, P. P. Sullivan, P. E. Hamlington, and S. R. Haney. The form and orientation of Langmuir Cells for misaligned winds and waves. Journal of Geophysical Research Oceans, In press, [34] A. Webb, B. Fox-Kemper, E. Baldwin-Stevens, G. Danabasoglu, B. Hamlington, W. G. Large, and S. Peacock. Global Climate Model Sensitivity to Estimated Langmuir Mixing. Ocean Modelling, [35] A. A. Webb, B. Fox-Kemper, S. Peacock, and W. R. Large. Global Model Sensitivity to Parameterizing Langmuir Circulation [36] R. A. Weller and J. F. Price. Langmuir circulation within the oceanic mixed layer. Deep Sea Research, 35: , 1988.