Open Software Development for CHPS: Towards a software API

Transcription

1 Open Software Development for CHPS: Towards a software Chris Brunner, CHPS Project Manager & Andy Rost, Director NOHRSC Office of Hydrologic Development (OHD) National Weather Service (NWS) FEWS User Days Nov 2010

2 2 Background = Application Programming Interface NWS culture: developing custom systems for operations 13 River Forecast Centers (RFCs): function largely independently of each other except where basins meet 13 RFCs: unique customers, unique forecasting challenges National applications often undergo numerous changes to meet all needs; some RFCs don t ever use them Numerous locally-developed applications dot the landscape CHPS: a paradigm change for RFCs: Current/legacy NWS River Forecast System (RFS) was custom; FEWS is akin to COTS Exception: IFD was custom-developed by Deltares for RFCs

3 3 Upcoming NWS requirements E.g., Calibration, Verification, Hydrologic Ensemble Forecast Service (HEFS), Archive Database All will require at least one of: Access to forecast data stored in operational FEWS database Access to archived data stored in Archive database User interfaces incorporating identical look and feel of FEWS (dock-able windows, thumbnails, etc.) Concern: how to develop without dependency on (small) Deltares development team?

4 Implementation options for upcoming software development 4 NWS continues to code everything independently Potentially a lot of duplication NWS software would need to keep up with FEWS feature changes Open source (FEWS master code, NWS-provided changes) Deltares must retain architectural and design philosophies Application Programming Interface () Well-defined division of code ownership Good compromise solution

5 Prototyping a GUI : the Graphics Generator 5 Purpose: allow users to pre-define characteristics of graphics products via a user interface which mimics the look and feel of the FEWS user interfaces Problem: many desired features were not available even via configurations Approach: extend existing FEWS PI-service to pass information (e.g., time series data) back to the calling application as XML string

6 6 Graphics Generator Example 1 Exceedance probability plots displaying multiple empirical cumulative distribution functions (blue line) and a fitted parametric distributions (red-line)

7 7 Graphics Generator Example 2 Spaghetti plot specifying the color to use associated with each ensemble member index (in GraphGen, the colors are assigned according to an algorithm and can be overridden by the user)

8 8 Graphics Generator Example 3 A product showing numerous added features, including reverse rendering and time of day of first tick mark

9 Prototyping a GUI : the Graphics Generator 9 Advantages Separation of code bases (Deltares vs. NWS), which makes debugging problems easier Disadvantages: XML parsing is time consuming slow processing time Approx 75% of processing time is due to XML parsing Need two identical code bases for parsing XML returned by the PI-service must always keep software changes synchronized Charting tools provided by Deltares sometimes inadequate Conclusion: has potential, but need to apply more thought

10 10 What Comes Next Questionnaire, to gather early NWS requirements Assign resources NWS, Deltares Preliminary Design NWS, Deltares First versions of code Deltares (FEWS) Office of Hydrologic Development (NWS) Learn, iterate over time: re-code Document early, update often Are others interested in contributing? E.g., BOM expressed interest during 2009 visit to USA Would need to discuss how to collaborate effectively

11 In the Meantime CHPS Era Archive/Data Backup

12 (field) data Current Messy Situation at the NWS acquisition SHEF SHEF ICP EVS flat files (grids) SHEF-dec. IHFS (ops DB) ofs-de FEWS fc DB raw SHEF-dec. RAX archive RDB proc.shef-dec. SHEF NWSRFS SHEF MPE NAVX XSETS products

13 Current Situation (FEWS Archive) FEWS archive: All data related to forecast bundled in one zip file Forecast and associated inputs can be retrieved in stand alone FEWS FEWS fc DB meta data RDB Data not directly accessible zip files archive-server Recommendation: This is a intermediatesolution only (as short as possible)

14 Where Do We Plan To Go? HEFS Verif Calb

15 Key Differences with Existing Archive System Data flows through the operational system to feed the archives Eliminates quality control nightmare Minimizes multiple decode pathways Provide for a more integrated archive 2 way data flow to support usage in operations and analysis Scalable, flexible and extensible architecture Hardware & database schemas

16 Gaps in Current Requirements: Meta-data What do we really need? What do we want? What about current archived data with little meta-data? Modifiers in CHPS Implemented significantly differently than NWSRFS Some decisions need to be made on how we want to handle modified time series

17 CHPS Time Series Example: Original Time Series Modifiers Time Series Used

18 Scheduled Objectives: Finalize conceptual design - Done Determine hardware requirements Dec 2010 Data requirements finalization Feb 2011 Build prototype system Sept 2011 Test system built at NOHRSC Operational field test at NWRFC Provide initial DB schema for review Sep 2011 Field deployment - ~2012

19 FEWS Archive and s FEWS DB Records Data IO Archive Process Archive Data I/O Archive DB Fully relational Records Archive New Apps: HEFS Calib Verify GUI FEWS GUI Files Data IO Insert/ Restore Process Event Trigger Archive Files Service Socket Archive Active Client/Server Data Exchanges Reformat, xform, etc on the fly FTP Site FTP Passive Data Publishing and/or Subscription External Decision Support Systems

20 Questions? Comments? FEWS DB Records Data IO Archive Process Archive Data I/O Archive DB Fully relational Records Archive New Apps: HEFS Calib Verify GUI FEWS GUI Files Data IO Insert/ Restore Process Event Trigger Archive Files Service Socket Archive Active Client/Server Data Exchanges Reformat, xform, etc on the fly FTP Site FTP Passive Data Publishing and/or Subscription External Decision Support Systems