AIDA JAS Massimiliano Turri, SLAC Massimiliano Turri, SLAC INFN-LNF, 7 July 2003

Transcription

1 AIDA JAS Massimiliano Turri, SLAC

2 AIDA Abstract Interfaces for Data Analysis

3 AIDA Outline Abstract Interfaces for Data Analysis Introduction Features - Advantages History Organization - Developers Status Interfaces Overview Example Tools and Users Conclusions

4 AIDA Introduction Abstract Interfaces for Data Analysis The goal of the AIDA project is to define abstract interfaces for common physics analysis objects, such as histograms, ntuples, fitters. The adoption of these interfaces makes it easier for developers and users to select and use different tools without having to learn new interfaces or change their code. In addition it is possible to exchange data (objects) between AIDA compliant applications through a standard XML format.

5 AIDA Features Abstract Interfaces for Data Analysis Interfaces start with I Define only pure virtual methods for analysis objects AIDA defines behavior, analysis tool provides implementation Object factories independence of user code from AIDA implementation easy to switch implementations IAnalysisFactory af = IAnalysisFactory.create();

6 AIDA Advantages Abstract Interfaces for Data Analysis Use same code with any AIDA-compliant analysis tool. User code (e.g. GEANT4) A I D A Analysis tool 1 Analysis tool 2 Used for experiment s code, e.g. reconstruction or individual user s analysis User needs to learn only one set of interfaces. Interoperability of analysis tools.

7 AIDA Speaking of GRID Abstract Interfaces for Data Analysis GRID-land GRID Node AIDA GRID Node Analysis Objects AIDA Distributed data analysis: GRID enabled AIDA implementation. No need for user to change code. Analysis Objects Many places within the GRID where abstract interfaces could be adopted. AIDA working group may be a good model to follow

8 AIDA History Abstract Interfaces for Data Analysis Initial idea formed at HepVis-99 workshop at Orsay. Informal AIDA discussions at CERN in 2000 AIDA workshops: January Paris/Orsay April Boston (preceding HepVis 2001) June 2002 CERN July CERN Informal meetings e.g. during Geant4 meetings and V.C.

9 AIDA Organization and Developers Abstract Interfaces for Data Analysis Open source project No formal collaboration/author list currently an informal CERN-LAL-SLAC effort Interfaces are designed by discussion and (eventual) consensus Takes some time, but result is well though out and robust Past and present contributors: Guy Barrand, Pavel Binko, Grzegorz Chwajol, Mark Dönszelmann, Wolfgang Hoschek, Tony Johnson, Emmanuel Medernach, Dino Ferrero Merlino, Lorenzo Moneta, Jakub Moscicki, Ioannis Papadopoulos, Andreas Pfeiffer, Max Sang, Victor Serbo, Max Turri

10 AIDA Status Abstract Interfaces for Data Analysis AIDA Version 3.0 (Oct 2002) second end-user release. AIDA 3.2 coming out soon AIDA 2.2 first end-user release Interfaces available in C++ and Java automatically generated with AID ( easy to add more languages General information, relevant links Tutorial, users guide, examples Downloads and web-browsable source code

11 AIDA Abstract Interfaces for Data Analysis Interfaces Overview

12 AIDA IHistogram Abstract Interfaces for Data Analysis Histograms fixed and variable width binning filling access to overall statistics (entries, mean, rms) bin information (center, entries, height, error) arithmetic (add, multiply, divide) projections slices

13 AIDA ICloud Abstract Interfaces for Data Analysis Histograms Clouds unbinned histograms scatter plots auto-conversion to histograms

14 AIDA IProfile Abstract Interfaces for Data Analysis Histograms Clouds Profiles fill 1D and 2D profile plots access to bin statistics (height, mean, rms) access to overall statistics (entries, mean rms) scaling

15 AIDA IDataPointSet Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets sets of n-dimensional points with errors add, remove, get points scaling of values and errors full mathematical arithmetic (+-*/) simple error propagation

16 AIDA ITuple Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets Ntuples Evaluators Filters fill and retrieve data support folder-like structure projections to histograms, clouds and profiles of evaluated quantities with filtering chaining

17 AIDA IPlotter Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets Ntuples Evaluators Filters Plotting create plot area, regions control styles (title, text, markers, lines etc.)

18 AIDA IFitter Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets Ntuples Evaluators Filters Plotting Fitter fitting to all the data storage types IFitResult result = fitter.fit(data, function) change fit method (c 2,max. Likelihood, etc) change optimizer (Minuit, etc) control parameters (bounds, fix, step, etc.) set constraints create scans and contours use function s analytical gradient

19 AIDA IFunction Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets Ntuples Evaluators Filters Plotting Fitter Functions create scripted or built-in functions access/change parameter s values evaluate function and its gradient support of PDFs (normalized functions over a range)

20 AIDA ITree Abstract Interfaces for Data Analysis Histograms Clouds Profiles Data Point Sets Ntuples Evaluators Filters Plotting Fitter Functions IO storage for analysis objects XML interchange format standard (.aida files) unix-like : ls, cp, mv, cd.

21 AIDA Example code Abstract Interfaces for Data Analysis IAnalysisFactory af = IAnalysisFactory.create(); ITree tree = af.createtreefactory().create(); IHistogramFactory histf = af.createhistogramfactory( tree ); IFitFactory fitf = af.createfitfactory(); IFitter fitter = fitf.createfitter("chi2"); IPlotter plotter = af.createplotterfactory().create("plot"); IHistogram1D gausshist = histf.createhistogram1d("gausshist", "Gaussian Histogram",100,-5,5); Random r = Random(); for (i=0; i<10000; i++) gausshist.fill(r.nextgaussian()); IFitResult chi2fit = fitter.fit(gausshist,"g"); plotter.createregions(); plotter.region(0).plot( gausshist ); plotter.region(0).plot( chi2fit.fittedfunction() ); plotter.show();

22 AIDA Tools and Users Abstract Interfaces for Data Analysis Three implementations are available within: Anaphe/Lizard (C++) Open Scientist (C++) JAIDA/JAS (Java) + AIDA-JNI 3.0 (C++) Users GEANT4: Analysis Advanced examples BaBar online (via JAS) GAUDI/ATHENA users Linear Collider users LCG is considering the adoption of AIDA in their Architectural Blueprint Massimiliano Turri, Physicist SLAC Interfaces (PI) group INFN-LNF, is working 7 July to 2003 create an AIDA implementation based on ROOT classes

23 AIDA Future Abstract Interfaces for Data Analysis PAW Root Remote Access to AIDA: Access to remote AIDA objects (through CORBA, RMI, SOAP, ) from another AIDA implementation. Access to AIDA objects through WEB Browser (with limited functionality) BaBar Online Oracle A I D A Remote AIDA Tool 1 AIDA Tool 2 -Remote AIDA Tree - WEB Access

24 AIDA Future Abstract Interfaces for Data Analysis Achieve component level interoperability e.g.: use ANAPHE histograms, fit with JAS3 fitter and plot with OpenScientist at last week s workshop we focused on developer interfaces first prototype due before October s workshop AIDA 4.0 end of 2003

25 AIDA Conclusions Abstract Interfaces for Data Analysis AIDA has a comprehensive set of interfaces for data analysis Three implementations are currently available Currently a few users but potential to become popular in HEP community AIDA needs feedback

26 JAS Mark Dönszelmann, Tony Johnson, Joseph Perl, Victor Serbo, Max Turri

27 JAS Outline Introduction to JAS JAS3 a new JAS Tour of major components of JAS3 AIDA Root and PAW file readers Tuple Explorer Java compiler and loader Scripting Languages Pnuts Record loop Spreadsheet JAS3 Architecture Status Conclusions

28 JAS Introduction to JAS JAS is a : general purpose open-source almost fully in java data analysis tool with the following features: Plotting of 1d, 2d and 3d histograms, XY plots, scatterplots etc. High quality print output. Fitting (binned or unbinned) using an extensible set of optimizers including Minuit. Export of plots in a variety of formats including PS, EPS, PDF, SVG, GIF, PNG.

29 JAS Introduction to JAS Easy to learn GUI for performing common analysis tasks (plotting, ntuple-analysis, fitting etc.). More complex analysis can be performed using a variety of scripting languages (pnuts, jython, etc.), or by writing Java analysis modules. Able to read data in a variety of formats including: aida, hbook, root, SQL databases, text-files. AIDA compliant analysis system. Built-in editor and compiler. Simple spreadsheet capabilities. Logbook for recording analysis tasks, and optionally publishing results to the web. Highly modular structure Massimiliano Turri, allows SLAC addition, replacement or removal of modules to customize application INFN-LNF, 7 July 2003 for a particular problem domain.

30 JAS JAS3 - a new JAS First version of JAS2 released in 2000 Has worked well, but some limitations Too heavily based on event loop No all analysis tasks involve looping over events No support for scripting Limited fitting capabilities Limited N-Tuple analysis capabilities Often used for online monitoring, often used outside HEP Not really designed for this use

31 JAS JAS3 - a new JAS Opportunities Realized a lot of overlap with WIRED project Created FreeHEP Java library to Pull out (re-factor) common parts of JAS, WIRED Implement new common requirements for JAS and WIRED Vector Graphics, Application Framework, Utilities Formed collaboration with developers of similar tools to create AIDA package

32 JAS JAS3 Components Tour designed to give you an overview of the capabilities of JAS3, you can try them out for yourself with the built-in Tutorial and/or examples Welcome Page, gives initial info and links to example scripts and programs Memory monitor

33 JAS Opening Files Use file menu Drag from explorer

34 JAS Graphical Interface to AIDA Histograms, Clouds, Tuples all presented in AIDA tree.aida files,.hbook files,.root files all presented as AIDA objects Drag items onto page, or use (popup) menus

35 JAS Printing Or copy/paste into Word, PowerPoint etc. Can send individual plots or full page direct to printer Or save as PS, EPS, PDF, SWF, SVG, PNG, GIF

36 JAS Java Editor, Compiler and Loader Tree shows loaded programs JAS3 allows any Java program to be loaded. This example main routine is taken directly from the AIDA manual Built-in editor for writing analysis code Built-in Java compiler

37 JAS Scripting Can also write and run scripts Console allows direct interaction with scripting language

38 JAS Pnuts Language Currently support Pnuts scripting language Complete and well documented Fast (although not as fast as compiled Java) Syntax very similar to Java Can easily call compiled Java classes from scripts best of both worlds Plan to support other languages in future In particular Python

39 JAS Record Sources Opening record (or event) based files causes the run control toolbar to appear Job controls. Also supports random access and tagged data sets (mainly for event displays)

40 JAS Tuple Explorer - Plots Works with any tuple, read from file or dynamically created Histogram Profile ScatterPlot XY Data (More appropriate for smaller data sets)

41 JAS Tuple Explorer Define Columns

42 JAS Tuple Explorer - Cuts

43 JAS Tuple Explorer - Tabulate

44 JAS Tuple Explorer Record Source To be used with record loop

45 JAS JAS3 Spreadsheet Simple spreadsheet plugin for Displaying results Calculations Simple Plots Supports reading/writing.csv files Excel files Cut/Paste with Excel etc Coming Soon Scripting interface GUI for building plots User defined functions Massimiliano Turri, Java, SLACscripting INFN-LNF, 7 July 2003

46 JAS Miscellaneous Features Save/Restore configuration User Preferences Plugin Manager Massimiliano Turri, SLAC INFN-LNF, 7 July 2003

47 JAS JAS3 Architecture Design based on Application Shell, into which many (optional) modules can be plugged All of the features we have discussed previously are implemented by one (or more) plugin modules Loose coupling between modules allows for modules to be removed without breaking remainder of program Extra modules (e.g. LCD specific modules) can be easily added Highly customizable for different application domains HEP/Astrophysics/Other DST analysis/online Monitoring/GRID analysis Experiment/User specific modules Modules can be updated independently of shell Possible to release bug fixes fast

48 JAS Status Currently released JAS3 version AIDA functionality is quite solid Compiler, Loader, Record Loop all quite recently added, Certainly still some rough edges Documentation is still quite limited Talks, tutorials, built-in example scripts and programs accessible from simple built-in web-browser If you are used to JAS2 you will find some functionality not yet ported to JAS3 Remote (client/server) access to data (this is being worked on) 3D Lego/Surface plots

49 JAS JAS 2 GRID interface (Tech-X)

50 JAS JAS and the GRID We plan to add client-server/distributed capabilities to JAS3 similar (but better) than those in JAS2 Will be based on (remote) AIDA Want to use Grid standards where they exist Work with others (PPDG-CS11,???) to define standards where they do not exist Tech-X have phase II SBIR approved and will work closely with us

51 JAS Help us make it better We have recently purchased and are now seriously using a commercial bug tracking system. Used not only for bugs, but also for tracking tasks, ideas etc. Please report any problems you find, and make suggestions for changes or improvements you would like to see.

52 JAS JAS3 Links, More Info JAS - JAS3 JAIDA AIDA FreeHEP - FreeHEP Java Libraries - WIRED