- PDF Free Download

Size: px

Start display at page:

Download ""

Alexandra Holt
6 years ago
Views:

2 A Tidal Wave of Scientific Data

3 Experimental Science Theoretical Science Newton s Laws, Maxwell s Equations Computational Science Simulation of complex phenomena Data-Intensive Science. a a 2 4 G 3 c a 2 2 captured by instruments generated by simulations generated by sensor networks

5 An edited collection of 26 short technical essays, divided into 4 sections

The Problem for the e-scientist Experiments &

Literature facts Answers Other Archives facts Data

organize it How to reorganize it How to share with

Query and Vis tools Building and executing models

6 The Problem for the e-scientist Experiments & Instruments facts Simulations facts Questions Literature facts Answers Other Archives facts Data ingest Managing a petabyte Common schema How to organize it How to reorganize it How to share with others (With thanks to Jim Gray) The Generic Problems Query and Vis tools Building and executing models Integrating data and Literature Documenting experiments Curation and long-term preservation

Forecasting Integration Analysis Reporting Done poorly, but a few notable counter-examples Distribution Aggregation Quality assurance Collation Monitoring

8 Forecasting Integration Analysis Reporting Done poorly, but a few notable counter-examples Distribution Aggregation Quality assurance Collation Monitoring Done poorly to moderately, not easy to find Sometimes done well, generally discoverable and available, but could be improved (I. Zaslavsky & CSIRO, BOM, WMO)

9 Environmental Ecosystem Knowledge Action Inform

10 Environmental Ecosystem Knowledge Action Analysis Data Insight Inform Decide Communicate Implement Publish

12 Data Variety The Spice of Life Manual Measurement Automated Measurement Sample Collection Typing Historical Photographs Satellite Aircraft Surveys Model Output Counting Relatively Ubiquitous Motes

14 Source Data (Swath format) Reprojected Data (Sinusoidal format)

followed by (non-repeatable and tedious) preprocessing True science ready data set but concerns about misuse, misunderstanding particularly for

15 Why Make this Distinction? Provenance and trust widely varies Data acquisition, early processing, and reporting ranges from a large government agency to individual scientists. Smaller data often passed around in ; big data downloads can take days (if at all) Data sharing concerns and patterns vary Open access followed by (non-repeatable and tedious) preprocessing True science ready data set but concerns about misuse, misunderstanding particularly for hard won data. Computational tools differ. Not everyone can get an account at a supercomputer center Very large computations require engineering (error handling) Space and time aren t always simple dimensions KB PB TB GB Complex shared detector Complex and Heavy process by experts Simple instrument (if any) Science happens when PBs, TBs, GBs, and KBs can be be mashed up up simply simply Ad hoc observations and models

Source Metadata Request Queue AzureMODIS Service Web Role

.. Data Collection Stage Reprojection Queue Reprojection

Results Catharine van Ingen (Microsoft Research), Jie Li,

16 Source Metadata Request Queue AzureMODIS Service Web Role Portal Scientist Source Imagery Download Sites... Data Collection Stage Reprojection Queue Reprojection Stage Reduction Queue Analysis/Reduction Stage Scientific Results Catharine van Ingen (Microsoft Research), Jie Li, Marty Humphreys (UVA), Youngryel Ryu (UCB), Deb Agarwal (BWC/LBL)

17 Pilot study R1 : million observations Engineering success Collaborators: Humberto da Rocha (USP) Andreas Terzis (JHU) Juliana Salles, Rob Fatland (MSR) Brito Cruz (FAPESP) Continuation R2 : Science and engineering objectives Solve the carbon balance problem Build an interoperable data system

Common Problems with Data To use data from different sources o o o o Non-standard formats, scales, and units Lack of data quality control Lack of metadata Difficult to repurpose data for different

19 Common Problems with Data To use data from different sources o o o o Non-standard formats, scales, and units Lack of data quality control Lack of metadata Difficult to repurpose data for different (my) tools Data Sources To share data o Lack of incentive (no credit) SQL data data XML o Need extra resources and tools Hidden problems, seldom addressed o Versioning Data Cube CSV (data) o Provenance o Curation

20 Current State of Data Ecosystem Applications Android iphone Windows Phone WebOS Data Sources Excel Java Silverlight.NET AJAX PHP MATLAB SQL data data XML (data) Cloud Service HPC Cluster DB Sever Data Server Data Cube CSV Web server

21 Advance data discoverability, accessibility, and consumability maps PivotViewer A Web protocol for querying and updating data provides a way to unlock your data and free it from data silos does this by building upon Web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores. In Open Source/Specifications Promise SQL Spatial Marketplace An application of a set of internet standards: HTTP, Atom (RFC 4287), AtomPub (RFC 5023), REST semantics Existing standards + easy data access API Adding Geospatial data support Feedback from the Community encouraged It allows you to form URLs based on what you know about the underlying data

23 NodeXL Network graph visualization Binary and source code:

24 NodeXL Network Overview Discovery and Exploration add-in for Excel 2007/2010 A minimal network can illustrate the ways different locations have different values for centrality and degree

26 interactive exploration cinematic narrative

28 speech recognition technologies used to crack audio files Indexing automatic transcripts as text does not work real enterprise automatic transcription accuracy is only 50-80% % accuracy improvement over indexing automatic transcripts index word alternatives robust to recognizer errors index timing navigate to exact point in video No need to invest in H/W infrastructure

29 ScienceCinema Use NLP and Bing Search to expand word dictionary Enables discovery of speech content 1,000 hours of AV content currently available NEW

Seamless Rich Social Media Virtual Sky Web application

sets and one-click contextual access Easy access and

for extensibility Excel Add-in for easy data

30 Seamless Rich Social Media Virtual Sky Web application for science and education Goals Integration of data sets and one-click contextual access Easy access and use Tours for sharing information/insights Updates API for extensibility Excel Add-in for easy data integration We invite you to experience it!

design to the human needs and interaction models Revolutionize the way we think about

32 Natural User Interfaces (NUI) Rethinking ways in which people will interact with computers/technologies of the future Re-evaluating everything from their (non-) physical design to the human needs and interaction models Revolutionize the way we think about technology and what it can do on our behalf

33 NUI Kinect SDK and WWT

escience in the Cloud Dan Fay Director Earth, Energy and Environment

escience in the Cloud Dan Fay Director Earth, Energy and Environment dan.fay@microsoft.com New ways to analyze and communicate data EOS Article: Mountain Hydrology, Snow Color, and the Fourth Paradigm