Overview of the EMF Refresher Webinar Series. EMF Resources

Transcription

1 Overview of the EMF Refresher Webinar Series Introduction to the EMF Working with Data in the EMF viewing & editing Inventory Data Analysis and Reporting 1 EMF User's Guide EMF Resources EPA's CoST website CoST User's Guide t/2.5.1/cost_usersguide_ _final.pdf 2 1

2 Roadmap for Today's Webinar What is the Emissions Modeling Framework (EMF)? What can the EMF do? What challenges can the EMF help solve? How is data stored/managed in the EMF? How do I get started in the EMF? What can EMF tell me about the data? How can I view and export data? 3 What is the EMF? Software package built to help in emissions work Manage emissions data files Track changes to data files Create summaries and reports from inventories Perform QA tasks on inventories Create controlled and/or future-year inventories 4 2

3 How the EMF Works The EMF client application runs on your local computer The client application communicates with the remote EMF server The remote EMF server stores the emissions data and does the number crunching 5 What Can the EMF Do? Analyze emissions inventory data; for example: Create inventory summary reports by source characteristics like county, SCC, plant ID Compare inventory data from one year to another year Check inventory records for bad values like unknown or missing SCC codes, or negative emissions values 6 3

4 What Can the EMF Do? The EMF includes the Control Strategy Tool (CoST) - covered in separate webinars Estimate emissions reductions and costs associated with control strategies Apply control measures to create controlled inventories Apply growth factors to create future year inventories growth factors EPA or regional Customize inventory temporalization to your analysis 7 What Can the EMF Do? The EMF can manage the inputs and settings needed to run emissions modeling tools like SMOKE SMOKE converts emissions inventory data into air quality model inputs by applying allocation factors We will not be covering this topic in this webinar series. 8 4

5 What Problems Can the EMF Help Solve? How do we organize all the various files that are used? How can everyone see the data that's available and find the data that they need? How do we track changes? Who made the change, what was changed, and why was it changed. When updates are made, how do we make sure we use the latest version of the data? 9 What Problems Can the EMF Help Solve? How do we share information with others? How do we ensure that created output is reproducible? How do we keep track of the analysis and QA that's been done for a particular file? 10 5

6 Poll: Which of these is not something the EMF can do? a) Keep track of all the inventory files for a project b) Create a summary of an inventory by county and SCC code c) Project an inventory from 2011 to 2030 d) Create a summer weekday inventory from an annual inventory 11 How is Data Stored in the EMF? Data files are imported into the EMF to create datasets During import, the EMF reads the raw data from the file into the EMF database When you import a file, you tell the EMF what dataset type to use The EMF knows the format of the data file based on the dataset type Column names and what kind of data (string, integer, decimal, etc.) is in each column Data to read must exactly match the dataset type column definition 12 6

7 Kinds of Data Managed by the EMF Emissions inventories Measured or estimated emissions by source Emissions data can have different temporal resolutions like annual, monthly, daily, hourly, or seasonal Data can be base year emissions, future year emissions, and/or controlled emissions The EMF can read several emissions inventory files in different formats Knows about the different types of emissions sources like on-road mobile, nonroad, nonpoint, and point 13 Inventory Dataset Types in the EMF Flat File 2010 [Nonpoint Point Activity] Inventories containing annual or month-specific emissions in different FF10 formats Flat File 2010 [Nonpoint Point] Daily Day-specific inventories in FF10 format ORL [Fire Nonpoint Nonroad Onroad Point] Inventory Annual inventories in different ORL formats 14 7

8 Kinds of Data Managed by the EMF Allocation factors used to calculate more finely detailed estimates from the original emissions inventory data; for example: Gridding surrogates: factors to allocate county-level emissions to grid cells Speciation profiles: factors to split emissions by pollutant into chemical mechanism species Temporal profiles: factors to estimate hourly emissions based on the original temporal resolution of the emissions inventory data 15 Kinds of Data Managed by the EMF Growth factors used to project a base year inventory to a future year Control measures to model control technologies Effectiveness information for the control technology (i.e. control efficiency, rule penetration) Factors to calculate the engineering costs associated with implementing particular measures 16 8

9 Kinds of Data Managed by the EMF Reference data, including: State and county codes and names for reporting emissions summarized by geographic area County latitude and longitude coordinates that can be included in output so data can be visualized using GIS tools Names for codes like SCC, SIC, MACT, NAICS, etc. 17 Poll: We talked about several kinds of data. Which didn't we discuss? a) Seasonal point source emissions b) Hourly temperature data c) Factors to estimate gridded emissions from county-level emissions d) Control measure efficiencies 18 9

10 Starting Up the EMF Each time you launch the EMF client you will be prompted to login with your username and password The first time you start the EMF client you'll need to create a username and password and enter your profile information 19 Main EMF Window 20 10

11 EMF Windows The Status window will display information about tasks the EMF is running The Downloads window will be used when downloading data or reports from the EMF 21 The Dataset Manager The Dataset Manager is the main interface for listing the datasets that have been imported into the EMF and finding the dataset you want to work with To open the Dataset Manager, open the Manage menu at the top of the main EMF window and select Datasets Initially, no datasets are listed; for now, we'll use the Show Datasets of Type pull-down menu to select All 22 11

12 The Dataset Manager 23 The Dataset Manager The Dataset Manager displays a summary of information about each dataset aka Dataset Properties The Dataset Manager lets you select and work with datasets In the homework, you'll explore a few datasets and get familiar with the EMF user interface 24 12

13 Viewing the Properties of a Dataset Select a dataset by checking the checkbox in the Select column next to the dataset name Click the Edit Properties button to open the Dataset Properties window The Summary tab will be displayed (see next slide) 25 Dataset Properties: Summary Tab Name: nonroad_caps_2005v2_jul_orl_nc Description: header of inventory file Project: assigned when dataset was imported Creator: user who created the dataset Dataset Type Time Period Start and End: determined from file header Temporal Resolution: determined from dataset type Sector: set by dataset type Region: assigned by user Country: determined from file header 26 13

14 Dataset Properties: Summary Tab Status: imported from file vs. created by EMF Dates last modified, accessed, and created Intended Use: public vs. private Default Version: will discuss in a moment 27 Poll: Which of these properties is not stored for datasets? a) Date the dataset was created b) The user who created the dataset c) Temporal resolution d) Spatial resolution e) Geographic region the dataset covers 28 14

15 Viewing vs. Editing Viewing an item: looking at the item but can't change its information Editing an item: can change information about the item Only one user can edit a dataset at a time but multiple users can view a dataset You must edit a dataset in order to change anything about it including adding or running QA steps (we'll discuss more later) 29 How to Find Datasets Use the Show Datasets of Type pull-down menu in the Dataset Manager This pull-down menu does have an All option to see all the datasets in the system 30 15

16 How to Find Datasets If you have a lot of datasets of a given type, use the Name Contains field You can type in a string to match the names of datasets For example, typing 2020 will match all datasets whose name contains 2020 You can use wildcards in this field: pt*2020 matches datasets whose name contains pt followed by 2020 at some point ptnonipm_2020cc_nc_sc_va_09nov2009_v0 31 How to Find Datasets Advanced button displays the Advanced Dataset Search window Match datasets by name, description, creator, project Find datasets by QA step or data values Datasets must match all criteria specified (boolean AND) 32 16

17 How to Find Datasets Another option is to filter the list of datasets in the Dataset Manager Click the Filter toolbar button to display the Filter Rows dialog Create a filter to only show rows where the Type column contains the word Inventory ; note that filter criteria are case-sensitive 33 How to Find Datasets Edit your user profile to limit the list of dataset types that the EMF shows you In the main EMF window, open the Manage menu and select My Profile to see the Edit User window In the Edit User window, you'll see a list of Visible Dataset Types and a list of Hidden Dataset Types Select visible dataset types and click the Hide button to move the selected types to the hidden list 34 17

18 How to Find Datasets 35 Poll: How can you find datasets by name? a) Advanced dataset search b) Create criteria to filter the rows displayed in the Dataset Manager c) Enter a search term in the Name Contains field in the Dataset Manager d) All of the above 36 18

19 Dataset Properties: Data Tab Provides access to the raw data associated with the dataset Shows information about each version of a dataset Click View to view the raw data (will be covered in more detail later) 37 Dataset Properties: Keywords Tab Keywords store additional metadata about datasets The keywords pass info to the EMF to be used for certain processing like including prefixes or suffixes when exporting a dataset or setting up indexes of the raw data 38 19

20 Dataset Properties: Notes Tab Shows notes that have been made about the dataset Notes can be questions, answers, comments, etc. Create a new note by clicking the Add button 39 Creating a Dataset Note Enter a summary of your note Select the note type and enter the details If your note references a previous note (like answering a previous question), select the referenced note Select which version of the dataset your note refers to 40 20

21 Dataset Properties: Revisions Displays information about revisions that have been made to the raw data in the dataset Revisions are automatically created when the raw data is edited 41 Dataset Properties: History Displays a record of when the dataset was exported for use in modeling runs 42 21

22 Dataset Properties: Sources Displays information about where the data associated with this dataset came from (i.e. the original file that was imported) Also shows which table the raw data is stored in in the database 43 Dataset Properties: QA Steps Shows a list of QA steps that have been created for this dataset QA steps represent different actions performed like summarizing the data by county and pollutant or creating a list of unique SCCs in the data Allows you to add, edit, copy, and delete QA steps We will discuss QA steps in the next webinar 44 22

23 Poll: True or False? 1) You can't view a dataset while someone else is editing it. 2) Every dataset in the EMF uses at least one table in the database to store its raw data. 3) Dataset notes are associated with a particular version of a dataset. 45 Dataset Actions Refresh: checks with the EMF server and gets the latest information for the dataset Other users can be editing datasets while you're working so you may need to refresh to see their latest changes Save: save the changes you've made to a dataset Export: export the raw data to a file (next slide) Close: close the Dataset Properties window; if you have unsaved changes, you'll be asked if you want to discard them 46 23

24 Exporting Datasets Exporting a dataset creates a file containing the data The exported file format is determined by the dataset's type, i.e. an FF10 Nonpoint Inventory dataset will be exported in FF10 Nonpoint format In the Dataset Manager window, select the datasets you would like to export and click the Export button to display the Export customization window 47 Exporting Datasets 48 24

25 Exporting Datasets Check the Download files to local machine? Checkbox Creates file on the EMF server that will be automatically downloaded to your computer The file name is generated from the dataset name You can enter a file name prefix to add to the file name If the dataset has multiple versions, you can choose which version you would like to export 49 Exporting Datasets Choose datasets to export Download files to local machine? CHECK Optional File Name Prefix Choose Version Filter to limit rows exported 50 25

26 Exporting Datasets In the Export dialog, click the Export button to begin the export Check the Status window for messages: Completed export of SmokeFlatFile_NONPOINT_ csv to C:\EMF_State_Data\downloads\StateEMF\SmokeFlatFile_NON POINT_ _csv_27sep2014_v0 in seconds. The file will start downloading momentarily, see the Download Manager for the download status. No. of records in database: 9,389,860; Exported: 60, Exporting Datasets In the main EMF window, open the Window menu and select Downloads to open the Download Manager Once the progress bar reaches 100%, your download is complete Right click the filename and choose Open Containing Folder to open the folder where the file was downloaded 52 26

27 Exporting Datasets Local Machine Download TIP: Re-name with.csv extension to easily open as a spreadsheet 53 Exporting Datasets Opening as a Spreadsheet 54 27

28 Coming up Homework: links to be provided in follow-up EMF Refresher, Part 2: October 21, 1:30 3:00 How do I QA and summarize inventory data? What is a QA step? Creating a QA step: new and from templates Comparing datasets Exporting data in KMZ format to view in Google Earth 55 Preview of EMF Refresher, Part 2 QA step results: What are the statewide totals by pollutant? For each pollutant, which counties and SCCs? EMF bonus SCC descriptions 56 28

29 Acknowledgement Institute for the Environment from University of North Carolina at Chapel Hill 57 29