Big data for big river science: data intensive tools, techniques, and projects at the USGS/Columbia Environmental Research Center

Transcription

1 Big data for big river science: data intensive tools, techniques, and projects at the USGS/Columbia Environmental Research Center Ed Bulliner U.S. Geological Survey, Columbia Environmental Research Center

2 Goals of Presentation How are the data available to us different than the past? What different approaches are needed to analyze these data? What questions are we asking and answering that we could not before? Big river science four examples How does this relate to NRDAR/ecological restoration?

3 Big Data What is big data? Emerging field Several definitions volume, variety, variability Do we work with big data or lots of data Is that distinction important? Regardless of semantics, increasing scale and complexity of problems and necessary data What do increasing amounts of data mean for science and scientists? How do we get the most value from the data available to us? Why is this important?

4 Data Intensive Science Paradigm shift in how we do science Can ask (and answer) new kinds of questions New tools and techniques

5 Traditional versus Data- Intensive Analyses Where do we see dataintensive science? Within river science? Within USGS/government? Why now? (what s different?) Data availability Data resolution Computational power What are the different tools and approaches currently used?

6 Tools for Data-Intensive Analyses Data storage Increased hard drive space Databases Data manipulation Scripting languages Web scraping/data munging / data mining Modeling Scripting languages Modeling packages Data visualization

7 Python Web Queries OS Operations Database Integration Statistics IDL Data Visualization ArcGIS & ArcPY General purpose scripting language Lots of modules Free * Tools for: Data management Data filtering/cleaning Scientific computing Geospatial analyses Plotting Collaborating

8 Pretty cool, but what can we use it for?

9 Question: Where do riverine sandbars exist and how do they change over time?

10 Create database of rivers and flows Mask active channel within overlap of rivers and landsat images Integrate Landast metadata with corresponding discharge data through relational database Query imagery by discharge/date Automated download and analysis of imagery timeseries of sandbars

11 Identified areas of persistent sand Investigated flows where sand was exposed Examined spatial variation Used metrics of exposure to help model success of Least Tern nests

12 Main Points Scripts and databases allow for automated downloading and linking of multiple data types Too much data for manual analysis Python can be used to batch-process images across programs without manual intervention Scripted tools can be used to directly query, plot, and perform statistics on image data

13 Question: What information can we synthesize from a 400+ day archive of field measurements?

14 EXPLANATION Velocity, in cubic meters per second Velocities and depths measured along regular transects Lateral, longitudinal, and vertical variability Water column Velocity ensemble Velocity bin River bottom 4-beam depths fast slow

15 ADCP and single-beam survey dates, locations and discharges EXPLANATION Flow percentile Low <25% Medium 25-75% High >75%

16 Compiled over 32,000 individual cross-sections from Joined dataset to river mile and gage to allow dischargespecific queries Can group data by location along river and varying discharge levels to compare

17 Ongoing restoration question: how does habitat (velocity) compare in river chutes versus main channel Chutes = restoration 37 field days where measurements in chutes were taken incidentally or deliberately Can use geospatial tools and scripts to come up with relevant comparisons

18 Measurement archive in lieu of hydrodynamic model sturgeon spawning locations?

19 Main Points Scripts and databases allow for efficient querying and cleaning of archived datasets Python can be used to quickly and interactively summarize datasets by specific groupings Existing data can be repurposed and integrated with new data for value-added analyses using scripting

20 Question: How can we better visualize field measurements of channel velocity and bathymetry?

21 Measurements of velocity collected along regular transects Python used to interpolate data into structured grid (3d matrix)

22 Paraview

23 Can visualize flowlines around structures (biology) Identified bias in field measurements?

24 33 million+ data points! Noticed systematic bias Collaborating with ILWSC

25 Main Points Python scripts allow for interpolation and visualization of field data Using open-source (free) tools along with Python allows for replication of abilities from more expensive software New insights can be gained from visualizing data in different ways

26 Question: How can we better characterize inundation patterns along the Missouri River?

27 Hydrodynamic (HEC- RAS) model provided by USACE describing water surface elevations at cross sections over time Used scripting to extend cross sections across floodplain for Missouri River

28 Merged LIDAR and channel data provides high-resolution characterization of floodplain elevation Spatial interpolations of water elevation Calculations of inundation depths

29 Inundation return interval statistics

30 n dates n dates Base Time unit series for calculations: of rasters, 1 per 1 date, day water for 29,892 depth modeled raster grid days (30m) for 1 area

31 z Stack over time x y Structured 3-dimensional matrix of data x and y are geospatial coordinates (raster dims) z is time coordinate (29,892 days) Water depth for each x,y,z

32 Time Data structured as hierarchical data format (hdf) on disk to allow computationally efficient slicing in time domain Setting inundation threshold allows for identification of inundated periods per pixel

33 n years n years Can aggregate data by year Evaluate inundation status by criteria (such as longest consecutive inundated period during growing season) Summarize metrics across all modeled years

34 Main Points Python scripts allow for dealing with data too big for one computer Processing across virtual machines Processing large files Time-series analyses on large datasets are useful for answering management questions Computational models are a useful supplement to field data

35 Data Intensive Restoration? There have been many attempts at ecological restoration Meta-analysis of restoration success is nothing new What data are available to us in USGS/DOI that might lend itself to these approaches? What data are needed by people implementing NRDAR restoration? How can NRDAR projects contribute useful information?

36 NRDAR Case Map and Document Library

37 Conclusions As scientists, we work in an expanding world of big data We can t analyze data by ourselves need tools Sharing data is important Ongoing projects are just beginning to utilize the scope of available datasets and capabilities of tools like Python What existing data is not fully utilized? Think big Add value

38 Questions?