Air Force Data Reference Architecture and Platform

Transcription

1 Headquarters U.S. Air Force Air Force Data Referece Architecture ad Platform Ms. Jackie Murray 11 Oct

2 AF Data Challeges Large umber of legacy systems with umerous poit-to-poit iterfaces that are expesive to implemet ad maitai Atiquated, brittle architectures ihibit legacy system ehacemets required to make data accessible May legacy systems employ data jails that prevet timely data access that supports the aalytics eeds of Fuctioal commuities The time is ow to employ ew, commercially available data services that will chage the ladscape of how data eables missio effectiveess 2

3 Air Force Data Goals Esure the Air Force ca leverage data as a strategic asset i every facet of military operatios ad support fuctios Eable ad empower Airme to use data as a critical strategic asset whe ad where it is eeded to accelerate iformed decisios at speed ad scale Trasform the Air Force to a data-drive orgaizatio 3

4 Achievig the AF Strategy Architecture & CDO Platform Key Teats Agile Approach Solutio Priorities Cloud-Native Services First Ope Source Software Secod COTS Products Third -- Miimize Licesed Software No Proprietary Solutios Vedor Agostic Must ot be reliat o oe compay Product Agostic Must allow for access to a variety of tools best suited to meet idividual missio demads Architect for easy adoptio of ew tools ad services while miimizig impact to the uderlyig data Utilize with o-prem or off-prem services Decouplig of compute ad storage Miimizes RMF impact 4

5 Deliverig Data Through the AF Data Platform ü Iitial Istace this year Support AF Aalytics Eterprise Iformatio Model to Discover & Access Authoritative Data Sources Based o AF Data Services Referece Architecture Iform Policy, Stadards & Goverace for Data Owers Policy & Stadards Data quality, Data taggig, Metadata maagemet, Data loss prevetio, Igest methods, ADS registratio, Iformatio asset defiitio, Data dictioary maagemet, Chage data capture

6 Puttig Data i the Hads of the Operators Eterprise Iformatio Model / AF CDO Data Catalog ADS Data Dictioary Services Registry Search Discovery Iformatio Asset Catalog Data Dictioary Data elemet level glossary (ratioalizatio of Eterprise vocabulary) Iformatio Asset Catalog Aggregated data elemets desigated as AF data assets ADS Registry Listig of origiatig sources for the data elemets ad iformatio assets Services Registry Descriptio of Eterprise Services utilizig cataloged iformatio assets Operatioal Metrics Operatioal Portal Data Catalog + Operatioal Metrics + Authorizatio & Busiess Rules = Eterprise Iformatio Model 6

7 Eablig the Missio with Data Aalytics INGEST CURATION WRANGLING ANALYTICS VISUALIZATION a) Data is iducted ito the lake ad igest pipelies are created usig Apache Kylo ad Apache NiFi b) Data is profiled, idexed ad loaded ito Apache Hive table c) Data is added to the Catalog a) Data cleasig ad validatio policies are set a optimized target format is selected b) Create additioal busiess metadata ad tags for searchig c) A feed schedule is fially established to check for ew files d) Profile details a) Usig R i a Zeppeli otebook key colums are imported b) A Spark data frame is created c) Libraries are imported, colum ames ad data types are properly assiged d) Additioal wraglig removes ivalid sequeces ad maipulates the data ito a tidy format a) The R script cotiues by calculatig the mea ad stadard deviatio of each NIIN by FY QTR b) A fial table of statistics is aggregated, collected, ad verified i preparatio for plottig a) Additioal wraglig shapes the stats table ito proper format for plottig b) Codig calls libraries to display high ed publishable graphics from the statistical aalysis 7

8 QUESTIONS 8