escience in the Cloud Dan Fay Director Earth, Energy and Environment

Size: px

Start display at page:

Download "escience in the Cloud Dan Fay Director Earth, Energy and Environment"

Johnathan Crawford
5 years ago
Views:

1 escience in the Cloud Dan Fay Director Earth, Energy and Environment

2 New ways to analyze and communicate data

3 EOS Article: Mountain Hydrology, Snow Color, and the Fourth Paradigm by Jeff Dozier

(I. Zaslavsky & CSIRO, BOM, WMO) Information about water is more useful as we climb the value ladder Forecasting Integration Analysis Reporting Done poorly, but a few notable

4 (I. Zaslavsky & CSIRO, BOM, WMO) Information about water is more useful as we climb the value ladder Forecasting Integration Analysis Reporting Done poorly, but a few notable counter-examples Distribution Aggregation Quality assurance Collation Monitoring Done poorly to moderately, not easy to find Sometimes done well, generally discoverable and available, but could be improved

5 The Ecological Data Flood We re living in a perfect storm of remote sensing, cheap ground-based sensors, internet data access, and commodity computing Yet deriving and extracting the variables needed for science remains problematic Specialized knowledge for algorithms, internal file formats, data cleaning, etc, etc Finding the right needle across the distributed heterogeneous and very rapidly growing haystacks

6 Data Variety The Spice of Life Manual Measurement Automated Measurement Sample Collection Typing Historical Photographs Aircraft Surveys Model Output Counting Relatively Ubiquitous Motes

7 Data Integration Challenges Regular rasters, points, and spatial features Time series and intermittent Vocabulary meanings (ontology) Sparse in time, duration, or location Science variable derivation Gaps Spatial/temporal harmonization

Smaller data often passed around in email; big data downloads can take days (if at all) Data sharing concerns and patterns vary Open access followed by

8 Why Make this Distinction? PB Provenance and trust widely varies Data acquisition, early processing, and reporting ranges from a large government agency to individual scientists. Smaller data often passed around in ; big data downloads can take days (if at all) Data sharing concerns and patterns vary Open access followed by (non-repeatable and tedious) pre-processing True science ready data set but concerns about misuse, misunderstanding particularly for hard won data. Computational tools differ. Not everyone can get an account at a supercomputer center Very large computations require engineering (error handling) Space and time aren t always simple dimensions KB TB GB Complex shared detector Simple instrument (if any) Science Science happens happens when when PBs, PBs, TBs, TBs, GBs, GBs, and KBs and can KBs be can mashed be mashed up simply up simply Complex and Heavy process by experts Ad hoc observations and models

AzureMODIS Azure Service for Remote Sensing Geoscience Source Imagery Download Sites Science pipeline for download, initial processing, and

Dramatically lowers resource and complexity barriers to use satellite imagery for terrestrial hydrology and geoscience.

NET, Java, or MatLab) On-demand scalability beyond local desktop or cluster In use now to compute 10 year continental scale water balance for

Per year: 500 GB (~60K files) upload of 9 different source imagery products from 15 different locations 400 GB reprojected harmonized imagery

9 AzureMODIS Azure Service for Remote Sensing Geoscience Source Imagery Download Sites Science pipeline for download, initial processing, and reduction of satellite imagery. Developed by MSR, UVa, UCB. Dramatically lowers resource and complexity barriers to use satellite imagery for terrestrial hydrology and geoscience. Common imagery location determination and upload from diverse sources Optional scientist-provided reduction algorithm (.NET, Java, or MatLab) On-demand scalability beyond local desktop or cluster In use now to compute 10 year continental scale water balance for North America. Per year: 500 GB (~60K files) upload of 9 different source imagery products from 15 different locations 400 GB reprojected harmonized imagery consuming ~3500 cpu hours 5 GB reduced science result leveraging reported field data aggregates consuming ~60 cpu hour Source Metadata Request Queue AzureMODIS Service Web Role Portal Scientist... Data Collection Stage Reprojection Stage Reprojection Queue Analysis/Reduction Stage Reduction Queue Scientific Results Catharine van Ingen (Microsoft Research), Jie Li, Marty Humphreys (UVA), Youngryel Ryu (UCB), Deb Agarwal (BWC/LBL)

17 WRF Service Architecture Azure Table storage Database Azure Storage Front Front end end Front end Partition Layer Stream Layer

Syntax hi-lite Intellisense Find / Browse Python Tools for VS: Project mgmt Solutions Deployment Built-in REPL IPython REPL Interactive Parallel Computing Inline graphics Profiler A/B

21 Syntax hi-lite Intellisense Find / Browse Python Tools for VS: Project mgmt Solutions Deployment Built-in REPL IPython REPL Interactive Parallel Computing Inline graphics Profiler A/B comparions Parallel Watch view Parallel Stack view CPython IronPython Or any interpreter Python Debugger.Net Debugger Remote Debugging HPC Support F5 MPI debugging Batch or Interactive 21

22 Technical / Scientific Computing PTVS + Numpy + SciPy is a productive T.C. Workbench HPC / MPI w support for cluster debugging Inline Graphics in REPL IPython notebook: Python in browser: any OS/ any browser Experimental:.Net versions of Numpy + SciPy Pyvot: A live bridge between PTVS and Excel 22

23 Python on Azure: IPython notebook Python IDE in browser Any Browser Any OS Backed by Python engine on Azure Windows or Linux Key features Intellisense, completion, Inline graphics Markdown Executable Document IPython REPL also built-into PTVS

24 Windows Server and Linux Flexible Workload Support Virtual Private Networking

31 Resources Azure for Students Python on Azure Windows Azure Python SDK Windows Azure How to use Service Management from Python Weatherservice.cloudapp.net Blogs on Big data and big compute Data Science in a box with IPython Notebook Virtual Machine Depots Azure for Educators Apply for Grant for Azure use in courses Faculty Connection Windows Azure University Faculties If interested send with Subject line: AzureU@Microsoft.com Include your contact information

escience in the Cloud: A MODIS Satellite Data Reprojection and Reduction Pipeline in the Windows

escience in the Cloud: A MODIS Satellite Data Reprojection and Reduction Pipeline in the Windows Jie Li1, Deb Agarwal2, Azure Marty Platform Humphrey1, Keith Jackson2, Catharine van Ingen3, Youngryel Ryu4