Starting a Data Analysis

Size: px

Start display at page:

Download "Starting a Data Analysis"

Sophie Jackson
5 years ago
Views:

1 03/20/07 PHY310: Statistical Data Analysis 1 PHY310: Lecture 17 Starting a Data Analysis Road Map Your Analysis Log Exploring the Data Reading the input file (and making sure it's right) Taking a first look at each variable What's important when looking at data.

2 03/20/07 PHY310: Statistical Data Analysis 2 Your Analysis Log Remember, this is a paper notebook, or an electronic file Your choice: It must be easy for you to use. What goes in an analysis log Every little thing! Start every session with the date so you can reconstruct your thinking Don't try to make a log into a polished document. It is a collection of notes to yourself Keep it in chronological order For you own sake of understanding what you did, don' t try to dress it up It's also a matter of scientific integrity Mistakes are OK, but correct them! When you find a mistake, note it, but make the correction in the entry for the current date Don't erase mistakes, you might find out you were right in the first place. You will use the log to write a paper. Put enough information in so you don't need to repeat work.

3 03/20/07 PHY310: Statistical Data Analysis 3 Starting the Analysis Log Start by describing where the data can be found Describe the variables available in the analysis Be sure note the units! Analysis Log 03/15/07 Start the analysis. Copy data file from eventdata.txt to eventdata txt Field definitions: event energy in GeV (oops... It's MeV 3/19/07) invariant mass in MeV/c^2 momentum in MeV/c...

4 03/20/07 PHY310: Statistical Data Analysis 4 The Data The data is in eventdata.txt Data Exposure: 1490 days Simulated Exposure: days Variable Definitions All Events: Energy (MeV): This is the energy of the event assuming particles are γ-rays Invariant mass (MeV/c^2): The invariable mass assuming γ-rays Momentum (MeV/c): The momentum assuming γ-rays Decay times (ns): The time to the first muon decay candidate (zero if none) Direction cosine: The cosine of the zenith angle for the event direction Track1 type: The log likelihood difference for γ-rays muon. Positive is γ-ray Track1 energy (MeV): The energy of the track assuming it is a γ-ray Track2 type: The log likelihood difference (as above) Track2 energy (MeV): The energy of the track assuming it is a γ-ray Event type: 0) Data 1) General neutrino event 2) neutral current single π prod. Sum of MC type 1 and type 2 should match the data.

5 03/20/07 PHY310: Statistical Data Analysis 5 The Data (Continued) Variable Definitions Only MC: Neutrino type: The Particle Data Group keeps an official index of numbers to identify various particles 12) Electron Neutrino -12) Electron Anti Neutrino 14) Muon Neutrino -14) Muon Anti Neutrino Neutrino energy (GeV): Note the units are GeV! Neutrino direction cosine: The direction the neutrino was going True π momentum (MeV/c): If an event is a single π, then this is the momentum of the π. Only valid for MC, and only for events from neutral current single π production. This goes in your analysis log

6 03/20/07 PHY310: Statistical Data Analysis 6 Look at the Data File mcgrew@boxer:~$ head eventdata txt 14 evis invmass mom decay zenith track1 energy1 track2 energy2 type nutype nuenergy nuzenith nupizero Do the fields and values match with what you are expecting based on what the fields are suppose to be? Write a routine to read the data into variables so that you can process it It's a good idea to make it into a subroutine since that way you can copy it into scripts to make it easy to read data.

7 03/20/07 PHY310: Statistical Data Analysis 7 Reading and Analyzing the Data We are looking at 14 variables, so the code to read and store the data in memory should be very organized. Possible solutions: Individual arrays with the same index This is the fortran solution. Passing values for an event to a routine Passing the event index Pros: Very simple programing Cons: Code gets messy very quickly Array with one structure This is the C solution (not C++!) Pass values as a pointer to an event. Pros: Code is easier to read (one handle per event) Cons: Must use a struct.

8 03/20/07 PHY310: Statistical Data Analysis 8 Defining the Event Structure in Memory The Fortran Way const int gmaxevents = 50000; int gnevents = 0; float genergy[gmaxevents]; float ginvmass[gmaxevents]; float gmomentum[gmaxevents]; float gdecay[gmaxevents]; float gdirection[gmaxevents]; float gtrack1[gmaxevents]; float genergy1[gmaxevents]; float gtrack2[gmaxevents]; float genergy2[gmaxevents]; int gtype[gmaxevents]; int gnutype[gmaxevents]; float gnuenergy[gmaxevents]; float gnudirection[gmaxevents]; float gnupizero[gmaxevents]; The C Way struct eventdata { float energy; float invmass; float momentum; float decay; float direction; float track1; float energy1; float track2; float energy2; int type; int nutype; float nuenergy; float nudirection; float nupizero; }; const int gmaxevents=50000; int gnevents = 0; struct eventdata geventdata[gmaxevents];

9 03/20/07 PHY310: Statistical Data Analysis 9 Looping the Events #include "readevents.c" void countevents() { readevents(); int dataevents = 0; int npi0events = 0; int pi0events = 0; for (struct eventdata* ev = geventdata; ev < &geventdata[gnevents]; ++ev) { if (ev->type == 0) ++dataevents; if (ev->type == 1) ++npi0events; if (ev->type == 2) ++pi0events; } } std::cout << "Data Events: " << dataevents << std::endl; std::cout << "MC Events: " << npi0events + pi0events << std::endl; std::cout << " Non-pi0: " << npi0events << std::endl; std::cout << " pi0: " << pi0events << std::endl;

10 03/20/07 PHY310: Statistical Data Analysis 10 Filling in readevents.c void readevents(char* inputfile = "eventdata txt") { std::ifstream input(inputfile); gnevents = 0; int fields; input >> fields; if (fields!= 14) { return; } std::string fieldnames; for (int i=0; i<fields; ++i) input >> fieldnames; } struct eventdatastruct line; for (;;) { input >> geventdata[gnevents].energy >> geventdata[gnevents].invmass >> geventdata[gnevents].momentum >> geventdata[gnevents].decay >> geventdata[gnevents].direction >> geventdata[gnevents].track1 >> geventdata[gnevents].energy1 >> geventdata[gnevents].track2 >> geventdata[gnevents].energy2 >> geventdata[gnevents].type >> geventdata[gnevents].nutype >> geventdata[gnevents].nuenergy >> geventdata[gnevents].nudirection >> geventdata[gnevents].nupizero; if (!input.good()) break; ++ gnevents; } std::cout << "Read " << gnevents << " events" << std::endl;

11 03/20/07 PHY310: Statistical Data Analysis 11 A First Look at the Data The first step in a data analysis is to look at the most important variables Make sure that you understand where each variable came from What is the definition? How is it related to measurements in the detector? Do you understand the rough distribution of each variable? Are there any surprising features? Does the variable distribution match with your intuition? Are there any effects that you should be looking out for? e.g. neutrino oscillations? How do you expect the variables to be correlated to each other? None? Some? A lot? Start with the simple variables (i.e. start with direct measurements).

12 03/20/07 PHY310: Statistical Data Analysis 12 Event Type An integer field indicating where the event came from The numbers of each event type 1739 Data Events (type 0) Simulated Events (type 1 and type 2) Non NC 1 π events (type 1) 5885 NC 1 π events (type 2) Number of simulated events scaled to the data exposure Expected 1900±8.8 events in 1489 days Discrepancy between number of measured and number of expected data events. 161±43 events 3.8 sigma Probability 5% 91.5% Normalization Factor (data/simulation)

13 03/20/07 PHY310: Statistical Data Analysis 13 Digression: Systematic Uncertainty Notice there is a mild disagreement between The predicted event rate (1900) The measured event rate (1739) Is this a real difference? Maybe, the χ² probability is ~5% Why might the numbers be different? Systematic Uncertainty

14 03/20/07 PHY310: Statistical Data Analysis 14 What is Systematic Error Unknown factors that affect the measurement. Two important types: Unknown factors due to random processes Unknown factors due to uncertain knowledge Mistakes are not an uncertainty Keep mistakes from happening Take very good notes on how you do an analysis. Make internal cross checks Don't present a result until you are ready to defend it. Incorrect results found before publication are learning experiences Incorrect results found after publication are mistakes Mistakes do happen Correct problems as quickly as possible Don't minimize the effect, don't hide the mistake

15 03/20/07 PHY310: Statistical Data Analysis 15 Systematic Uncertainty: Type 1 Unknown factors from other random processes Often the result of a calibration Conversion from ADC digital measurement to charge Calibration of a thermometer Calibration of a clock Sometimes the result of an input measurement The atmospheric neutrino flux depends on the measured cosmic ray flux hitting the earth. The value of the uncertainty is usually available in the calibration documentation Can think of this uncertainty as the experimental error on measurements made by somebody else. Always be very explicit about where you get your calibration values.

16 03/20/07 PHY310: Statistical Data Analysis 16 Systematic Uncertainty: Type 2 Unknown factors due to uncertain knowledge Approximations Theoretical Inputs Fit nicely in the Bayesian Framework These are not classical statistical quantities Can't have an ensemble of best estimates What is a good Classical Statistician suppose to do? Try to keep statistical and systematic errors separated. No particularly objective way to include... Usually punt and treat as a random uncertainty It's not random! Make a S.W.A.G. at how well the factor is known. Be as intellectually honest as possible Don't overestimate you knowledge It's just as bad to underestimate what you know

17 03/20/07 PHY310: Statistical Data Analysis 17 Practical Systematic Uncertainty Systematic Uncertainty tries to capture your lack of knowledge We want to be able to treat our ignorance statistically! Systematic Uncertainty is not the difference between your expectation and observation Not based on your measured quantity But can be based on data measured in your detector. e.g. Comparison of your expectation with calibration data. Start the systematic uncertainty analysis early, and update it continually while you work

18 03/20/07 PHY310: Statistical Data Analysis 18 Back to the Normalization Uncertainty Data vs MC Normailization Observed 1739 two ring events Expected 1900 two ring events Contributing uncertainties Atmospheric Neutrino Flux Published uncertainty: 15% Depends on neutrino energy Need to look at the expected distribution of parent neutrinos. Neutrino Cross Section Should look at which interactions are contributing to the data Typical cross section uncertainty 15%-20% Detector Efficiency Can't be estimated from our data, but typically 5-10% for SK Overall normalization uncertainty: ~26% Data and MC are definitely within the systematic uncertainty

19 03/20/07 PHY310: Statistical Data Analysis 19 The Event Energy The mean energies are ~8% different (~2% probability this is a fluctuation)

20 03/20/07 PHY310: Statistical Data Analysis 20 Scale Data and MC to Same Exposure Data above expectation Data below expectation Need to quantify the probability that the data and MC came from the same P.D.F. Use Pearson's Statistic

21 03/20/07 PHY310: Statistical Data Analysis 21 Pearson's χ² Test 2 = N x i t i 2 1 t i For large samples, this follows the expected χ² distribution for N degrees of freedom Need to scale theory to the data exposure If the data sample size is small, then this will not follow the usual χ², and you need to generated the PDF using MC

22 03/20/07 PHY310: Statistical Data Analysis 22 Scale Data and MC to Same Exposure Data above expectation Data below expectation Pearson's χ² = 70.6 for 40 d.o.f. (probability ~0.2%) This is a clear difference between data and MC. Need to find explanation

23 03/20/07 PHY310: Statistical Data Analysis 23 First Analysis Homework Get the data from my web site It's linked from the homework assignment Write a function to read the data There is an example on the web site Reproduce the event counts found in this lecture (1739 data MC) Reproduce the energy distribution figures. There is an example script for plotting the event energy. Look at the distributions for: Decay Signal Time Event Direction Divide the MC into events from electron neutrinos and muon neutrinos For the MC only: Plot the neutrino energies for electron and muon neutrinos Plot the neutrino directions for the electron and muon neutrinos. The End

Performance of the GlueX Detector Systems

Performance of the GlueX Detector Systems GlueX-doc-2775 Gluex Collaboration August 215 Abstract This document summarizes the status of calibration and performance of the GlueX detector as of summer 215.