Earth Science Data Delivery Reducing the Gaps Dave Hartzell, NOAA Affiliate david.hartzell@noaa.gov TIP 2013 1
Agenda Introduction, Mission Data, data-centers, Networks Current Future Challenges 2
Introduction Goal: deliver an overview of NOAA data challenges from a networking perspective. Data in this case is: Sensor data Input to weather models, e.g. observational Output from weather models, e.g. predictions Disaster recovery, archiving and dissemination 3
NOAA s Mission Science, Service, and Stewardship. To understand and predict changes in climate, weather, oceans, and coasts, To share that knowledge and information with others, and To conserve and manage coastal and marine ecosystems and resources (http://www.noaa.gov/about-noaa.html) 4
Where the data rests or originates at NOAA Archives at National Data Centers: Comprehensive Large-array Data Stewardship System (CLASS, Boulder and Asheville, NC) Real-time delivery from: National Weather Service Satellite Program (NESDIS) Web Operations Center (WOC) clusters R&D High Performance Computing: Supercomputers and archives All the above are examples many more exist 5
Current NOAA Networking Major data centers serviced by regional Network Ops Centers (NOCs) for I1,I2: Boulder, CO Silver Spring, MD (Wash DC area) Seattle Ft. Worth N-Wave for agency wide-area connectivity 6
What is N-Wave? N-Wave is an operational network that supports both science and research activities Current users of N-Wave are: NOAA s R&D HPCS NESDIS / CLASS Hawaii Pacific Region Center WOC Built in partnership with Internet2 and RONs and GRNOC Traditional carrier-grade IP+MPLS backbone 7
N-Wave Backbone 8
Current N-Wave Connections 9
Boulder NOC Utilization Example Boulder NOC network I1, I2 & RON network utilization. (Blue = outgoing, green = incoming Includes traffic for NGDC, NWS FO, WOC, etc.) NOAA Boulder Network 10
N-Wave Total Monthly Volumes 2.5E+15 2.0E+15 1.5E+15 RDHPCS 1.0E+15 OAR CLASS 500.0E+12 000.0E+0 11
Moore s Law: Future Network Drivers Larger supercomputers = bigger data, higher resolution models, etc. New Sensor Systems GOES-R JPSS Doppler radar Agency consolidation and cost-savings pursuit Install systems and data centers where they are lower cost to operate 12
Futures (2) Larger supercomputers = higher resolution modeling, faster turn-around time on processing High Resolution Rapid Refresh (HRRR) model. http://ruc.noaa.gov/hrrr/ 13
Futures GOES-R Next-generation Geo-stationary Observational Satellite See: http://goes-r.gov For more information 14
GOES-R Data Flows Higher-resolution data vs. existing GOES Additional data products Higher demand? Internal (to NOAA ground systems) data rates: ~1.3 Gbps Outgoing to users: ~Gbps? Always subject to change 15
NPP / JPSS NOAA / NASA Next Generation Polar Orbiting System Suomi NPP Preparatory mission for JPSS (http://www.jpss.noaa.gov/) 16
NPP / JPSS Data Flows Low earth orbit constellation will have multiple spacecraft starting in 2016 (with JPSS-1) Predicted raw data volumes, per spacecraft per day: ~200-300 Gbytes PRODUCT data flows could be on the order of 10s of Tbytes per day to hundreds of users - TBD Real-time delivery requirements to AND from: DoD International Partners Weather models Suomi NPP 17
Challenges (gaps) How will we move all of this data? Today: Existing I1, I2 services are adequate Gratuitous (and abusive) use of private point-to-point lines Future: N-Wave Leverage existing Regional, National and International R&E network partners Stop using the private p2p lines too expensive, International and not enough current capacity Multicast delivery for common products is being investigated The cloud??? 18
Challenges (2) Change the thought processes: Starting thinking about shared networks Ethernet, not SONET No more point-to-points Hard to fit into modern US Gov t security models anyway Technology IPv6 Firewalls, science DMZs, Jumbo frames, multicast, etc. 19
Conclusion As new sensors, systems and models come online, NOAA needs to find faster, better and cheaper ways to move data. This is a problem starting NOW to 2-3 years out. We are looking at these futures, and are trying to solve them now. Link - GFDL SST 20