DIVA: updates and new potential for improving SDC data products

Transcription

1 DIVA: updates and new potential for improving SDC data products Alexander Barth, Charles Troupin, Sylvain Watelet, Aida Alvera-Azcárate, and Jean-Marie Beckers GHER, University of Liège, Belgium GeoHydrodynamics and Envionment Research 1 / 26

2 High-resolution coastal gridded data Context: coastal data products related to eutrophication Sample dataset from Scheldt estuary Most common parameter is dissolved oxygen concentration oxygen concentration is linked to eutrophication oversupply of nutrients overgrowth of plants and algae bacterial degradation consumption of oxygen 2 / 26

3 Scheld dataset ODV files, individual observation 68 unique position time range: from to no information about depth, but very shallow area anyway Overview from the CDI interface (not all dots corresponds to oxygen) 3 / 26

4 Land-sea mask Land-sea mask is tricky at these scales: disconected water bodies (e.g. Veerse Meer disconnected from the North Sea) land-sea mask based on sea-level 0 (however the actual land-sea mask depend on tides) Use of EMODNET bathymetry 4 / 26

5 Number of data at every unique location, their mean and standard deviation 5 / 26

6 Sample results Test analysis at 12 arc seconds resolution (about 360 m, 1/300 degree) Single 2D analysis combing all time instances Anamorphosis transform to avoid negative values Use of of the Julia version of DIVA (divand) 6 / 26

7 Original data (mean) vs diva anaysis 7 / 26

8 Phosphate Similar test with phosphate from World Ocean Database 8 / 26

9 What is DIVA? DIVA: Data Interpolating Variational Analysis Objective: derive a gridded climatology from in situ observations The variational inverse methods aim to derive a continuous field which is: close to the observations (it should not necessarily pass through all observations because observations have errors) "smooth" Observations Analysis 9 / 26

10 DIVA updates We aim to fully rewrite DIVA in Julia (divand.jl) Julia: good trade-off between efficiency of a compiled language and flexibility of a dynamic language Facilitate the installation: Use Jupyter notebooks fully configured environment for divand.jl Docker container allows one to easily replicate these environments Continue to maintain the Fortran version of DIVA for ODV 10 / 26

11 Matlab Linear Algebra Python Modern programming concepts Data frames + Compilation to machine code + Performance approaching C + Multiple dispatch + Type system + Lisp-like macros and Metaprogramming 11 / 26

12 Jupyter notebooks Integrated web environment Computing Interactive Julia, Python, R,... Visualization Documentation High-quality type setting and equations (Latex) Export to HTML and PDF (among others) Easy to share, on e.g. nbviewer.jupyter.org and github.com Facilitate reproducibility and peerreview (of DIVA climatologies in particular) Significant community around Jupyter notebooks Also involvement of players outside of the scientific community (Google, Microsoft with Azure ML) Jupyter notebooks: single user 12 / 26

13 Jupyter architecture Python https/ websocket ZMQ... Datasets 13 / 26

14 Jupyterhub architecture Jupyterhub: multiple users Authentication Python Isolated environments 14 / 26

15 Test installation Test installation deployed in OpenStack at CINECA: Docker containers, preinstalled with Julia and various Julia packages: Plotting library (PyPlot) and a more specialized library for ocean data ZMQ DIVAnd... Julia packages are precompiled Integration in SeaDataCloud authentication: Implementation CAS authentication Marine ID can be used to login into jupyterhub EUDATs B2Access might be considered as an alternative Transfer files via WebDAV in Julia: Either transparently mounted or using explicit download and upload requests 15 / 26

16 Example notebook 16 / 26

17 Example notebook 17 / 26

18 Example notebook 18 / 26

19 Example notebook 19 / 26

20 Example notebook 20 / 26

21 21 / 26

22 New products: Currents DIVA applied to currents Red vector: hypothetical measurement Black vectors: analyzed field 22 / 26

23 New products: Currents Coastline as a boundary condition 23 / 26

24 New products: Currents Low horizontal divergence of currents 24 / 26

25 Analyzed HF radar currents The H2020 call: emphasis on coastal data HF radars measure radial surface currents In collaboration with SOCIB (Emma Reyes, Jaime Hernandez Lasheras, Baptiste Mourre, Joaquín Tintoé) 25 / 26

26 Conclusions New approach to generate DIVA climatologies using a cloud computing infrastructure Template of jupyter notebooks will be provided which NODC can adapt Improve the consistency between product Facilitate reproducibility Jupyter notebook is not a software specific to SeaDataCloud Users might already be familiar with Jupyter notebooks But if not, learning to work with Jupyter notebooks can also be useful in other contexts Jupyterhub: Docker allows to provide a standardized computing environment to all users The jupyter notebook can be used to fully document the generation of the climatology DIVA is adapted to work with ocean currents 26 / 26