zap Documentation Release 1.0.dev86 Kurt Soto

Size: px
Start display at page:

Download "zap Documentation Release 1.0.dev86 Kurt Soto"

Transcription

1 zap Documentation Release 1.0.dev86 Kurt Soto February 03, 2016

2

3 Contents 1 Installation Requirements Steps Examples Sparse Field Case Filled Field Case Extra Functions 7 4 Command Line Interface 9 5 Interactive mode 11 i

4 ii

5 zap Documentation, Release 1.0.dev86 ZAP (the Zurich Atmosphere Purge) is a high precision sky subtraction tool which can be used as complete sky subtraction solution, or as an enhancement to previously sky-subtracted data. The method uses PCA to isolate the residual sky subtraction features and remove them from the observed datacube. Though the operation of ZAP is not dependent on perfect flatfielding of the data in a MUSE exposure, better results are obtained when these corrections are made ahead of time. Contents 1

6 zap Documentation, Release 1.0.dev86 2 Contents

7 CHAPTER 1 Installation 1.1 Requirements ZAP requires the following packages: Numpy or later Astropy v1.0 or later SciPy v or later Many linear algebra operations are performed in ZAP, so it can be beneficial to use an alternative BLAS package. In the Anaconda distribution, the default BLAS comes with Numpy linked to OpenBlas, which can amount to a 20% speedup of ZAP. 1.2 Steps Once the code is downloaded, cd into the zap directory and install via:: python setup.py install 3

8 zap Documentation, Release 1.0.dev86 4 Chapter 1. Installation

9 CHAPTER 2 Examples In its most hands-off form, ZAP can take an input fits datacube, operate on it, and output a final fits datacube: import zap zap.process('input.fits', outcubefits='output.fits') Care should be taken, however, since this case assumes a sparse field, and better results can be obtained by applying masks. The main function is zap.process: There are a number of options that can be passed to the code which we describe here: The code can handle datacubes trimmed in wavelength space. Since the code uses the correlation of segments of the emission line spectrum, it is best to trim the cube at specific wavelengths. The cube can include any connected subset of these segments. (for example Angstroms) [0, 5400] [5400, 5850] [5850, 6440] [6440, 6750] [6750, 7200] [7200, 7700] [7700, 8265] [8265, 8602] [8602, 8731] [8731, 9275] [9275, 10000] 2.1 Sparse Field Case This case specifically refers to the case where the sky can be measured in the sky frame itself, using: zap.process('input.fits', outcubefits='output.fits') In both cases, the code will create a resulting processed datacube named DATACUBE_ZAP.fits and an SVD file named ZAP_SVD.fits in the current directory. While this can work well in the case of very faint sources, masks can improve the results. For the sparse field case, a mask file can be included, which is a 2d fits image matching the spatial dimensions of the input datacube. Masks are defined to be >= 1 on astronomical sources and 0 at the position of the sky. Set this parameter with the mask keyword 5

10 zap Documentation, Release 1.0.dev86 zap.process('input.fits', outcubefits='output.fits', mask='mask.fits') 2.2 Filled Field Case This approach also can address the saturated field case and is robust in the case of strong emission lines, in this case the input is an offset sky observation. To achieve this, we calculate the SVD on an external sky frame using the function zap.svdoutput An example of running the code in this way is as follows: zap.svdoutput('offset_field_cube.fits', svdfn='zap_svd.fits', mask='mask.fits') zap.process('source_cube.fits', outcubefits='output.fits', extsvd='zap_svd.fits', cfwidthsp=50) The integration time of this frame does not need to be the same as the object exposure, but rather just a 2-3 minute exposure. Often residuals can be further reduced by changing cfwidthsp to a smaller value. However, this parameter should not be reduced to smaller than 15 pixels. 6 Chapter 2. Examples

11 CHAPTER 3 Extra Functions Aside from the main process, three functions are included that can be run outside of the entire zap process to facilitate some investigations. 7

12 zap Documentation, Release 1.0.dev86 8 Chapter 3. Extra Functions

13 CHAPTER 4 Command Line Interface ZAP can also be used from the command line: python -m zap INPUT_CUBE.fits More information use of the command line interface can be found with the command python -m zap -h 9

14 zap Documentation, Release 1.0.dev86 10 Chapter 4. Command Line Interface

15 CHAPTER 5 Interactive mode ZAP can also be used interactively from within IPython import zap zobj = zap.process('input.fits', interactive=true) The run method operates on the datacube, and retains all of the data and methods necessary to process a final data cube in a python class named zclass. You can elect to investigate the data product via the zclass, and even reprocess the cube with a different number of eigenspectra per region. A workflow may go as follows: import zap from matplotlib import pyplot as plt # allow ZAP to run the optimize routine zobj = zap.process('input.fits', optimization='normal', interactive=true) # plot the variance curves and the selection of the number of eigenspectra used zobj.plotvarcurve(5) # plot a spectrum extracted from the original cube plt.figure() plt.plot(zobj.cube[:,50:100,50:100].sum(axis=(1,2)), 'b', alpha=0.3) # plot a spectrum of the cleaned ZAP dataproduct plt.plot(zobj.cleancube[:,50:100,50:100].sum(axis=(1,2)), 'g') # choose just the first 3 spectra for all segmments zobj.reprocess(nevals=3) # plot a spectrum extracted from the original cube plt.plot(zobj.cube[:,50:100,50:100].sum(axis=(1,2)), 'b', alpha=0.3) # plot a spectrum of the cleaned ZAP dataproduct plt.plot(zobj.cleancube[:,50:100,50:100].sum(axis=(1,2))), 'g') # choose some number of modes by hand zobj.reprocess(nevals=[2,5,2,4,6,7,9,8,5]) # plot a spectrum plt.plot(zobj.cleancube[:,50:100,50:100].sum(axis=(1,2))), 'k') # Use the optimization algorithm to identify the best number of modes per segment zobj.optimize() 11

16 zap Documentation, Release 1.0.dev86 # compare to the previous versions plt.plot(zobj.cleancube[:,50:100,50:100].sum(axis=(1,2))), 'r') # identify a pixel in the dispersion axis that shows a residual feature in the original plt.figure() plt.matshow(zobj.cube[2903,:,:]) # compare this to the zap dataproduct plt.figure() plt.matshow(zobj.cleancube[2903,:,:]) # write the processed cube as a single extension fits zobj.writecube('datacube_zap.fits') # or merge the zap datacube into the original input datacube, replacing the data extension zobj.writefits(outcubefits='datacube_final_zap.fits') 12 Chapter 5. Interactive mode

Doing a li6le astronomy with. Python. Ryan Cooke (K16) These slides & examples:

Doing a li6le astronomy with. Python. Ryan Cooke (K16) These slides & examples: Doing a li6le astronomy with Python Ryan Cooke (K16) These slides & examples: www.ast.cam.ac.uk/~rcooke/python/ An aside Let s begin by installing ATLAS: (AutomaIcally Tuned Linear Algebra SoKware) > cd

More information

FIFI-LS: Basic Cube Analysis using SOSPEX

FIFI-LS: Basic Cube Analysis using SOSPEX FIFI-LS: Basic Cube Analysis using SOSPEX Date: 1 Oct 2018 Revision: - CONTENTS 1 INTRODUCTION... 1 2 INGREDIENTS... 1 3 INSPECTING THE CUBE... 3 4 COMPARING TO A REFERENCE IMAGE... 5 5 REFERENCE VELOCITY

More information

VIP Documentation. Release Carlos Alberto Gomez Gonzalez, Olivier Wertz & VORTEX team

VIP Documentation. Release Carlos Alberto Gomez Gonzalez, Olivier Wertz & VORTEX team VIP Documentation Release 0.8.9 Carlos Alberto Gomez Gonzalez, Olivier Wertz & VORTEX team Feb 17, 2018 Contents 1 Introduction 3 2 Documentation 5 3 Jupyter notebook tutorial 7 4 TL;DR setup guide 9

More information

VERY LARGE TELESCOPE 3D Visualization Tool Cookbook

VERY LARGE TELESCOPE 3D Visualization Tool Cookbook European Organisation for Astronomical Research in the Southern Hemisphere VERY LARGE TELESCOPE 3D Visualization Tool Cookbook VLT-SPE-ESO-19500-5652 Issue 1.0 10 July 2012 Prepared: Mark Westmoquette

More information

The Italian LBT spectroscopic data reduction pipeline

The Italian LBT spectroscopic data reduction pipeline LBTO 2017 Users' Meeting The Italian LBT spectroscopic data reduction pipeline Alida Marchetti INAF-IASF Milano Firenze, June 20th-23rd reduction pipeline SOME NUMBERS INAF nights 46 Effective observing

More information

ARTIFICIAL INTELLIGENCE AND PYTHON

ARTIFICIAL INTELLIGENCE AND PYTHON ARTIFICIAL INTELLIGENCE AND PYTHON DAY 1 STANLEY LIANG, LASSONDE SCHOOL OF ENGINEERING, YORK UNIVERSITY WHAT IS PYTHON An interpreted high-level programming language for general-purpose programming. Python

More information

Week Two. Arrays, packages, and writing programs

Week Two. Arrays, packages, and writing programs Week Two Arrays, packages, and writing programs Review UNIX is the OS/environment in which we work We store files in directories, and we can use commands in the terminal to navigate around, make and delete

More information

Data products. Dario Fadda (USRA) Pipeline team Bill Vacca Melanie Clarke Dario Fadda

Data products. Dario Fadda (USRA) Pipeline team Bill Vacca Melanie Clarke Dario Fadda Data products Dario Fadda (USRA) Pipeline team Bill Vacca Melanie Clarke Dario Fadda Pipeline (levels 1 à 2) The pipeline consists in a sequence of modules. For each module, files are created and read

More information

POSSUM: analysis for early science and beyond

POSSUM: analysis for early science and beyond POSSUM: analysis for early science and beyond Cormac Purcell, Bryan Gaensler and the POSSUM team 2016-06-02 ASKAP 2016 askap.org/possum Rotation 1 Rotation 2 Figure: O'Sullivan et al. 2012 POSSUM-12 versus

More information

HANDS ON DATA MINING. By Amit Somech. Workshop in Data-science, March 2016

HANDS ON DATA MINING. By Amit Somech. Workshop in Data-science, March 2016 HANDS ON DATA MINING By Amit Somech Workshop in Data-science, March 2016 AGENDA Before you start TextEditors Some Excel Recap Setting up Python environment PIP ipython Scientific computation in Python

More information

OpenMSI Arrayed Analysis Toolkit: Analyzing spatially defined samples in mass spectrometry imaging

OpenMSI Arrayed Analysis Toolkit: Analyzing spatially defined samples in mass spectrometry imaging OpenMSI Arrayed Analysis Toolkit: Analyzing spatially defined samples in mass spectrometry imaging Introduction This is the accompanying notebook of the manuscript OpenMSI Arrayed Analysis Toolkit: Analyzing

More information

Tutorial Four: Linear Regression

Tutorial Four: Linear Regression Tutorial Four: Linear Regression Imad Pasha Chris Agostino February 25, 2015 1 Introduction When looking at the results of experiments, it is critically important to be able to fit curves to scattered

More information

MATPLOTLIB. Python for computational science November 2012 CINECA.

MATPLOTLIB. Python for computational science November 2012 CINECA. MATPLOTLIB Python for computational science 19 21 November 2012 CINECA m.cestari@cineca.it Introduction (1) plotting the data gives us visual feedback in the working process Typical workflow: write a python

More information

Matplotlib Python Plotting

Matplotlib Python Plotting Matplotlib Python Plotting 1 / 6 2 / 6 3 / 6 Matplotlib Python Plotting Matplotlib is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive

More information

Python for Data Analysis

Python for Data Analysis Python for Data Analysis Wes McKinney O'REILLY 8 Beijing Cambridge Farnham Kb'ln Sebastopol Tokyo Table of Contents Preface xi 1. Preliminaries " 1 What Is This Book About? 1 Why Python for Data Analysis?

More information

Bi 1x Spring 2014: Plotting and linear regression

Bi 1x Spring 2014: Plotting and linear regression Bi 1x Spring 2014: Plotting and linear regression In this tutorial, we will learn some basics of how to plot experimental data. We will also learn how to perform linear regressions to get parameter estimates.

More information

Manual of SPCI (structural and physico-chemical interpretation) open-source software version 0.1.5

Manual of SPCI (structural and physico-chemical interpretation) open-source software version 0.1.5 Manual of SPCI (structural and physico-chemical interpretation) open-source software version 0.1.5 Version (date) Changes and comments 0.1.0 (02.02.2015) Changes from alpha version: 1. More precise default

More information

Python With Data Science

Python With Data Science Course Overview This course covers theoretical and technical aspects of using Python in Applied Data Science projects and Data Logistics use cases. Who Should Attend Data Scientists, Software Developers,

More information

1 Introduction: Download the Sample Code and Data

1 Introduction: Download the Sample Code and Data PHYS-4007/5007: Computational Physics Python Tutorial Making Plots of Spectra in Python 1 Introduction: Download the Sample Code and Data Log into your Linux account and open the web browser. Open the

More information

HESP PIPELINE v. 1.0

HESP PIPELINE v. 1.0 HESP PIPELINE v. 1.0 Installation and Usage Arun Surya 20/06/2017 1. INSTALLATION The HESP pipeline installation zip file can be downloaded from, https://www.iiap.res.in/hesp/hesp_pipeline.zip. The zip

More information

Pandas and Friends. Austin Godber Mail: Source:

Pandas and Friends. Austin Godber Mail: Source: Austin Godber Mail: godber@uberhip.com Twitter: @godber Source: http://github.com/desertpy/presentations What does it do? Pandas is a Python data analysis tool built on top of NumPy that provides a suite

More information

Introduction to Data Science. Introduction to Data Science with Python. Python Basics: Basic Syntax, Data Structures. Python Concepts (Core)

Introduction to Data Science. Introduction to Data Science with Python. Python Basics: Basic Syntax, Data Structures. Python Concepts (Core) Introduction to Data Science What is Analytics and Data Science? Overview of Data Science and Analytics Why Analytics is is becoming popular now? Application of Analytics in business Analytics Vs Data

More information

Part VI. Scientific Computing in Python. Tobias Neckel: Scripting with Bash and Python Compact Max-Planck, February 16-26,

Part VI. Scientific Computing in Python. Tobias Neckel: Scripting with Bash and Python Compact Max-Planck, February 16-26, Part VI Scientific Computing in Python Compact Course @ Max-Planck, February 16-26, 2015 81 More on Maths Module math Constants pi and e Functions that operate on int and float All return values float

More information

pvextractor Documentation

pvextractor Documentation pvextractor Documentation Release 0.0.dev289 Adam Ginsburg and Thomas Robitaille Sep 12, 2017 Contents I Extracting slices programmatically 3 1 Defining a path 5 2 Extracting a slice 7 3 Saving the slice

More information

pyregion Documentation

pyregion Documentation pyregion Documentation Release 2.1.dev336 Jae-Joon Lee Jul 02, 2018 Contents I Documentation 3 i ii Release 2.1.dev336 Date Jul 02, 2018 pyregion is a python module to parse ds9 region files. It also

More information

Python in the Copernicus Climate Change Service

Python in the Copernicus Climate Change Service Python in the Copernicus Service Gionata Biavati C D S : A n e w w a y o f s e r v i n g d a t a Copernicus Services (C3S) (https://climate.copernicus.eu) is providing the Data Store (CDS) (https://cds.climate.copernicus.eu)

More information

ESO SCIENCE DATA PRODUCTS STANDARD. Doc. No. GEN-SPE-ESO , Issue 5. Addendum. Date: 15/07/2015. Integral Field Spectroscopy: 3D Data Cubes

ESO SCIENCE DATA PRODUCTS STANDARD. Doc. No. GEN-SPE-ESO , Issue 5. Addendum. Date: 15/07/2015. Integral Field Spectroscopy: 3D Data Cubes ESO SCIENCE DATA PRODUCTS STANDARD Doc. No. GEN-SPE-ESO-33000-5335, Issue 5 Addendum Date: 15/07/2015 Integral Field Spectroscopy: 3D Data Cubes The data format being defined in this section applies to

More information

PS6-DCT-Soln-correction

PS6-DCT-Soln-correction PS6-DCT-Soln-correction Unknown Author March 18, 2014 Part I DCT: Discrete Cosine Transform DCT is a linear map A R N N such that the N real numbers x 0,..., x N 1 are transformed into the N real numbers

More information

pandas: Rich Data Analysis Tools for Quant Finance

pandas: Rich Data Analysis Tools for Quant Finance pandas: Rich Data Analysis Tools for Quant Finance Wes McKinney April 24, 2012, QWAFAFEW Boston about me MIT 07 AQR Capital: 2007-2010 Global Macro and Credit Research WES MCKINNEY pandas: 2008 - Present

More information

Spectral Classification

Spectral Classification Spectral Classification Spectral Classification Supervised versus Unsupervised Classification n Unsupervised Classes are determined by the computer. Also referred to as clustering n Supervised Classes

More information

Conda Documentation. Release latest

Conda Documentation. Release latest Conda Documentation Release latest August 09, 2015 Contents 1 Installation 3 2 Getting Started 5 3 Building Your Own Packages 7 4 Getting Help 9 5 Contributing 11 i ii Conda Documentation, Release latest

More information

Certified Data Science with Python Professional VS-1442

Certified Data Science with Python Professional VS-1442 Certified Data Science with Python Professional VS-1442 Certified Data Science with Python Professional Certified Data Science with Python Professional Certification Code VS-1442 Data science has become

More information

Multivariate Calibration Quick Guide

Multivariate Calibration Quick Guide Last Updated: 06.06.2007 Table Of Contents 1. HOW TO CREATE CALIBRATION MODELS...1 1.1. Introduction into Multivariate Calibration Modelling... 1 1.1.1. Preparing Data... 1 1.2. Step 1: Calibration Wizard

More information

Exercise of data reduction for MOIRCS Multi-Object Spectroscopy

Exercise of data reduction for MOIRCS Multi-Object Spectroscopy 1D spectrum (Goal for quick workers) Subaru Autumn School 2014, on 24-26 September 2014 Exercise of data reduction for MOIRCS Multi-Object Spectroscopy Schedule on 25 th Sep. (as a guide) 10:10-12:00 (~2

More information

5 File I/O, Plotting with Matplotlib

5 File I/O, Plotting with Matplotlib 5 File I/O, Plotting with Matplotlib Bálint Aradi Course: Scientific Programming / Wissenchaftliches Programmieren (Python) Installing some SciPy stack components We will need several Scipy components

More information

KNIME Python Integration Installation Guide. KNIME AG, Zurich, Switzerland Version 3.7 (last updated on )

KNIME Python Integration Installation Guide. KNIME AG, Zurich, Switzerland Version 3.7 (last updated on ) KNIME Python Integration Installation Guide KNIME AG, Zurich, Switzerland Version 3.7 (last updated on 2019-02-05) Table of Contents Introduction.....................................................................

More information

Case Study: Acquiring Image Data from ZTF, and Displaying and Working with Images in Python

Case Study: Acquiring Image Data from ZTF, and Displaying and Working with Images in Python Case Study: Acquiring Image Data from ZTF, and Displaying and Working with Images in Python For this exercise, we will access the IRSA database for ZTF directly and determine the time and location of a

More information

Lecture 15: High Dimensional Data Analysis, Numpy Overview

Lecture 15: High Dimensional Data Analysis, Numpy Overview Lecture 15: High Dimensional Data Analysis, Numpy Overview Chris Tralie, Duke University 3/3/2016 Announcements Mini Assignment 3 Out Tomorrow, due next Friday 3/11 11:55PM Rank Top 3 Final Project Choices

More information

CSC 1315! Data Science

CSC 1315! Data Science CSC 1315! Data Science Data Visualization Based on: Python for Data Analysis: http://hamelg.blogspot.com/2015/ Learning IPython for Interactive Computation and Visualization by C. Rossant Plotting with

More information

Science Cookbook. Practical Data. open source community experience distilled. Benjamin Bengfort. science projects in R and Python.

Science Cookbook. Practical Data. open source community experience distilled. Benjamin Bengfort. science projects in R and Python. Practical Data Science Cookbook 89 hands-on recipes to help you complete real-world data science projects in R and Python Tony Ojeda Sean Patrick Murphy Benjamin Bengfort Abhijit Dasgupta PUBLISHING open

More information

NAVIGATING UNIX. Other useful commands, with more extensive documentation, are

NAVIGATING UNIX. Other useful commands, with more extensive documentation, are 1 NAVIGATING UNIX Most scientific computing is done on a Unix based system, whether a Linux distribution such as Ubuntu, or OSX on a Mac. The terminal is the application that you will use to talk to the

More information

Part VI. Scientific Computing in Python. Alfredo Parra : Scripting with Python Compact Max-PlanckMarch 6-10,

Part VI. Scientific Computing in Python. Alfredo Parra : Scripting with Python Compact Max-PlanckMarch 6-10, Part VI Scientific Computing in Python Compact Course @ Max-PlanckMarch 6-10, 2017 63 Doing maths in Python Standard sequence types (list, tuple,... ) Can be used as arrays Can contain different types

More information

Tutorial 2 PHY409 Anadi Canepa Office, TRIUMF MOB 92 B ( )

Tutorial 2 PHY409 Anadi Canepa Office, TRIUMF MOB 92 B ( ) Tutorial 2 PHY409 Anadi Canepa canepa@triumf.ca Office, TRIUMF MOB 92 B (1-604- 222-7330) Alan Manning mannin2@phas.ubc.ca Mohammad Samani samani@physics.ubc.ca During the 1 st tutorial We learnt What

More information

Introduction to Scientific Computing with Python, part two.

Introduction to Scientific Computing with Python, part two. Introduction to Scientific Computing with Python, part two. M. Emmett Department of Mathematics University of North Carolina at Chapel Hill June 20 2012 The Zen of Python zen of python... fire up python

More information

CIS192 Python Programming

CIS192 Python Programming CIS192 Python Programming Machine Learning in Python Robert Rand University of Pennsylvania October 22, 2015 Robert Rand (University of Pennsylvania) CIS 192 October 22, 2015 1 / 18 Outline 1 Machine Learning

More information

MS6021 Scientific Computing. TOPICS: Python BASICS, INTRO to PYTHON for Scientific Computing

MS6021 Scientific Computing. TOPICS: Python BASICS, INTRO to PYTHON for Scientific Computing MS6021 Scientific Computing TOPICS: Python BASICS, INTRO to PYTHON for Scientific Computing Preliminary Notes on Python (v MatLab + other languages) When you enter Spyder (available on installing Anaconda),

More information

Introduction to Python

Introduction to Python Introduction to Python Ryan Gutenkunst Molecular and Cellular Biology University of Arizona Before we start, fire up your Amazon instance, open a terminal, and enter the command sudo apt-get install ipython

More information

L3-Python-for-Statistical-Modeling

L3-Python-for-Statistical-Modeling L3-Python-for-Statistical-Modeling October 16, 2015 1 Python modules for Statistics 1.1 NumPy NumPy is short for Numerical Python, is the foundational package for scientific computing in Python. It contains

More information

Imaging and Deconvolution

Imaging and Deconvolution Imaging and Deconvolution Urvashi Rau National Radio Astronomy Observatory, Socorro, NM, USA The van-cittert Zernike theorem Ei E V ij u, v = I l, m e sky j 2 i ul vm dldm 2D Fourier transform : Image

More information

Lab 4: Structured Programming I

Lab 4: Structured Programming I 4.1 Introduction Lab 4: Structured Programming I Lab this week is going to focus on selective structures and functions. 4.2 Resources The additional resources required for this assignment include: 0 Books:

More information

Spectroscopy techniques II. Danny Steeghs

Spectroscopy techniques II. Danny Steeghs Spectroscopy techniques II Danny Steeghs Conducting long-slit spectroscopy Science goals must come first, what are the resolution and S/N requirements? Is there a restriction on exposure time? Decide on

More information

Introduction to Remote Sensing Wednesday, September 27, 2017

Introduction to Remote Sensing Wednesday, September 27, 2017 Lab 3 (200 points) Due October 11, 2017 Multispectral Analysis of MASTER HDF Data (ENVI Classic)* Classification Methods (ENVI Classic)* SAM and SID Classification (ENVI Classic) Decision Tree Classification

More information

PACS Data Reduction Guide: Spectroscopy. Issue user Version 15.0 March 2017

PACS Data Reduction Guide: Spectroscopy. Issue user Version 15.0 March 2017 PACS Data Reduction Guide: Spectroscopy Issue user Version 15.0 March 2017 PACS Data Reduction Guide: Spectroscopy Table of Contents 1. PACS Spectroscopy Launch Pad I... 1 1.1. Introduction... 1 1.1.1.

More information

Use case: mapping sparse spatial data with TOPCAT

Use case: mapping sparse spatial data with TOPCAT Use case: mapping sparse spatial data with TOPCAT This use case describes a workflow related to large hyperspectral datasets. In this example you will use data from the VIRTIS/Rosetta experiment and study

More information

COSC 490 Computational Topology

COSC 490 Computational Topology COSC 490 Computational Topology Dr. Joe Anderson Fall 2018 Salisbury University Course Structure Weeks 1-2: Python and Basic Data Processing Python commonly used in industry & academia Weeks 3-6: Group

More information

Lab on MODIS Cloud spectral properties, Cloud Mask, NDVI and Fire Detection

Lab on MODIS Cloud spectral properties, Cloud Mask, NDVI and Fire Detection MODIS and AIRS Workshop 5 April 2006 Pretoria, South Africa 5/2/2006 10:54 AM LAB 2 Lab on MODIS Cloud spectral properties, Cloud Mask, NDVI and Fire Detection This Lab was prepared to provide practical

More information

STREAMING ALGORITHMS. Tamás Budavári / Johns Hopkins University ANALYSIS OF ASTRONOMY IMAGES & CATALOGS 10/26/2015

STREAMING ALGORITHMS. Tamás Budavári / Johns Hopkins University ANALYSIS OF ASTRONOMY IMAGES & CATALOGS 10/26/2015 STREAMING ALGORITHMS ANALYSIS OF ASTRONOMY IMAGES & CATALOGS 10/26/2015 / Johns Hopkins University Astronomy Changed! Always been data-driven But we used to know the sources by heart! Today large collections

More information

FFT-Based Astronomical Image Registration and Stacking using GPU

FFT-Based Astronomical Image Registration and Stacking using GPU M. Aurand 4.21.2010 EE552 FFT-Based Astronomical Image Registration and Stacking using GPU The productive imaging of faint astronomical targets mandates vanishingly low noise due to the small amount of

More information

Introduction to Digital Image Processing

Introduction to Digital Image Processing Fall 2005 Image Enhancement in the Spatial Domain: Histograms, Arithmetic/Logic Operators, Basics of Spatial Filtering, Smoothing Spatial Filters Tuesday, February 7 2006, Overview (1): Before We Begin

More information

deepatari Documentation

deepatari Documentation deepatari Documentation Release Ruben Glatt July 29, 2016 Contents 1 Help 3 1.1 Installation guide............................................. 3 2 API reference 5 2.1 Experiment Classes........................................

More information

Programming for Engineers in Python

Programming for Engineers in Python Programming for Engineers in Python Autumn 2016-17 Lecture 11: NumPy & SciPy Introduction, Plotting and Data Analysis 1 Today s Plan Introduction to NumPy & SciPy Plotting Data Analysis 2 NumPy and SciPy

More information

Euler s Method with Python

Euler s Method with Python Euler s Method with Python Intro. to Differential Equations October 23, 2017 1 Euler s Method with Python 1.1 Euler s Method We first recall Euler s method for numerically approximating the solution of

More information

Video Article: Selecting multiple biomarker subsets with similarly effective binary classification performances

Video Article: Selecting multiple biomarker subsets with similarly effective binary classification performances 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Video Article: Selecting multiple biomarker subsets with similarly effective binary classification performances AUTHORS AND AFFILIATIONS:

More information

Nonlinear curve-fitting example

Nonlinear curve-fitting example Nonlinear curve-fitting example Implementation of curve-fitting in Python. Compare with results of Mathematica for same data sets: see pythontest.nb. In [1]: import scipy as sp from scipy.optimize import

More information

ADC Figure of Merit. Introduction:

ADC Figure of Merit. Introduction: ADC Figure of Merit Introduction: The improved performance LRIS in spectroscopic mode with the ADC (over the no-adc case) has many factors. Concentrating solely on spectral throughput, those factors that

More information

MDM 4UI: Unit 8 Day 2: Regression and Correlation

MDM 4UI: Unit 8 Day 2: Regression and Correlation MDM 4UI: Unit 8 Day 2: Regression and Correlation Regression: The process of fitting a line or a curve to a set of data. Coefficient of Correlation(r): This is a value between and allows statisticians

More information

NumPy and SciPy. Lab Objective: Create and manipulate NumPy arrays and learn features available in NumPy and SciPy.

NumPy and SciPy. Lab Objective: Create and manipulate NumPy arrays and learn features available in NumPy and SciPy. Lab 2 NumPy and SciPy Lab Objective: Create and manipulate NumPy arrays and learn features available in NumPy and SciPy. Introduction NumPy and SciPy 1 are the two Python libraries most used for scientific

More information

PyEmir Documentation. Release Sergio Pascual, Nicolás Cardiel

PyEmir Documentation. Release Sergio Pascual, Nicolás Cardiel PyEmir Documentation Release 0.11 Sergio Pascual, Nicolás Cardiel Dec 11, 2018 Contents 1 PyEmir User Guide 3 2 PyEmir MOS Tutorial 47 3 PyEmir Reference 101 4 Glossary 105 Python Module Index 107 i ii

More information

Using IPython on Windows HPC Server 2008

Using IPython on Windows HPC Server 2008 Using IPython on Windows HPC Server 2008 Release 1.1.0: An Afternoon Hack Brian E. Granger November 05, 2013 Contents 1 Getting started with Windows HPC Server 2008 1 1.1 Introduction..........................................

More information

Southern African Large Telescope

Southern African Large Telescope Southern African Large Telescope Title: Author(s): MIDAS automatic pipeline for HRS data Alexei Kniazev Doc. number: HRS0000006 Version: 1.0 Date: November 10, 2016 Keywords: HRS, Pipeline Approved: Petri

More information

HW Assignment 3 (Due by 9:00am on Mar 6)

HW Assignment 3 (Due by 9:00am on Mar 6) HW Assignment 3 (Due by 9:00am on Mar 6) 1 Theory (150 points) 1. [Tied Weights, 50 points] Write down the gradient computation for a (non-linear) auto-encoder with tied weights i.e., W (2) = (W (1) )

More information

Catapult Open. The Open Data Cube (ODC) A tool to increase the value and impact of global Earth observation satellite data

Catapult Open. The Open Data Cube (ODC) A tool to increase the value and impact of global Earth observation satellite data The Open Data Cube (ODC) A tool to increase the value and impact of global Earth observation satellite data SATELLITE APPLICATIONS CATAPULT Our Mission // To innovate for a better world, empowered by satellites.

More information

JWST Pipeline & Data Products

JWST Pipeline & Data Products JWST Pipeline & Data Products Stage 1: Ramps-to-Slopes Karl D. Gordon JWST Calibration WG Lead Space Telescope Sci. Inst. Baltimore, MD, USA Stage 2: Calibrated Slopes Stage 3: Ensemble Processing 18 May

More information

JWST Pipeline & Data Products

JWST Pipeline & Data Products JWST Pipeline & Data Products Stage 1: Ramps-to-Slopes Karl D. Gordon JWST Calibration WG Lead Space Telescope Sci. Inst. Baltimore, MD, USA Stage 2: Calibrated Slopes Stage 3: Ensemble Processing Star

More information

Using the Matplotlib Library in Python 3

Using the Matplotlib Library in Python 3 Using the Matplotlib Library in Python 3 Matplotlib is a Python 2D plotting library that produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.

More information

AMS209 Final Project: Linear Equations System Solver

AMS209 Final Project: Linear Equations System Solver AMS209 Final Project: Linear Equations System Solver Rene Gutierrez Marquez 1 UCSC 1 December 7, 2016 Abstract In this project an implementation of a solver of a system of linear equations is implemented.

More information

Ch.1 Introduction. Why Machine Learning (ML)? manual designing of rules requires knowing how humans do it.

Ch.1 Introduction. Why Machine Learning (ML)? manual designing of rules requires knowing how humans do it. Ch.1 Introduction Syllabus, prerequisites Notation: Means pencil-and-paper QUIZ Means coding QUIZ Code respository for our text: https://github.com/amueller/introduction_to_ml_with_python Why Machine Learning

More information

PINGSoft 2: an IDL Integral Field Spectroscopy Software

PINGSoft 2: an IDL Integral Field Spectroscopy Software arxiv:1211.0277v1 [astro-ph.im] 1 Nov 2012 PINGSoft 2: an IDL Integral Field Spectroscopy Software F. Fabián Rosales-Ortega Departamento de Física Teórica, Universidad Autónoma de Madrid, Spain Instituto

More information

MeerKAT Data Architecture. Simon Ratcliffe

MeerKAT Data Architecture. Simon Ratcliffe MeerKAT Data Architecture Simon Ratcliffe MeerKAT Signal Path MeerKAT Data Rates Online System The online system receives raw visibilities from the correlator at a sufficiently high dump rate to facilitate

More information

ipywidgets_demo July 17, Interactive widgets for the Jupyter notebook (ipywidgets)

ipywidgets_demo July 17, Interactive widgets for the Jupyter notebook (ipywidgets) ipywidgets_demo July 17, 2017 1 Interactive widgets for the Jupyter notebook (ipywidgets) Maarten Breddels - Kapteyn Astronomical Institute / RuG - Groningen Material on github https://github.com/maartenbreddels/ewass-2017

More information

Status of PSF Reconstruction at Lick

Status of PSF Reconstruction at Lick Status of PSF Reconstruction at Lick Mike Fitzgerald Workshop on AO PSF Reconstruction May 10-12, 2004 Quick Outline Recap Lick AO system's features Reconstruction approach Implementation issues Calibration

More information

PACS Data Reduction Guide. issue: user, Version: 9 Nov 2011

PACS Data Reduction Guide. issue: user, Version: 9 Nov 2011 PACS Data Reduction Guide issue: user, Version: 9 Nov 2011 PACS Data Reduction Guide Table of Contents 1. PACS Launch Pads... 1 1.1. Introduction... 1 1.2. PACS Data Launch Pad... 1 1.2.1. A quick on terminology...

More information

Visual Analytics Tools for the Global Change Assessment Model. Michael Steptoe, Ross Maciejewski, & Robert Link Arizona State University

Visual Analytics Tools for the Global Change Assessment Model. Michael Steptoe, Ross Maciejewski, & Robert Link Arizona State University Visual Analytics Tools for the Global Change Assessment Model Michael Steptoe, Ross Maciejewski, & Robert Link Arizona State University GCAM Simulation When exploring the impact of various conditions or

More information

Connecting ArcGIS with R and Conda. Shaun Walbridge

Connecting ArcGIS with R and Conda. Shaun Walbridge Connecting ArcGIS with R and Conda Shaun Walbridge https://github.com/sc w/nyc-r-ws High Quality PDF ArcGIS Today: R and Conda Conda Introduction Optional demo R and the R-ArcGIS Bridge Introduction Demo

More information

solving polynomial systems in the cloud with phc

solving polynomial systems in the cloud with phc solving polynomial systems in the cloud with phc Jan Verschelde University of Illinois at Chicago Department of Mathematics, Statistics, and Computer Science http://www.math.uic.edu/ jan jan@math.uic.edu

More information

Facial Recognition Using Eigenfaces

Facial Recognition Using Eigenfaces Lab 11 Facial Recognition Using Eigenfaces Load the Data Lab Objective: Use the singular value decomposition to implement a simple facial recognition system. Suppose we have a large database containing

More information

The SciPy Stack. Jay Summet

The SciPy Stack. Jay Summet The SciPy Stack Jay Summet May 1, 2014 Outline Numpy - Arrays, Linear Algebra, Vector Ops MatPlotLib - Data Plotting SciPy - Optimization, Scientific functions TITLE OF PRESENTATION 2 What is Numpy? 3rd

More information

CME 193: Introduction to Scientific Python Lecture 1: Introduction

CME 193: Introduction to Scientific Python Lecture 1: Introduction CME 193: Introduction to Scientific Python Lecture 1: Introduction Nolan Skochdopole stanford.edu/class/cme193 1: Introduction 1-1 Contents Administration Introduction Basics Variables Control statements

More information

G012 Scattered Ground-roll Attenuation for 2D Land Data Using Seismic Interferometry

G012 Scattered Ground-roll Attenuation for 2D Land Data Using Seismic Interferometry G012 Scattered Ground-roll Attenuation for 2D Land Data Using Seismic Interferometry D.F. Halliday* (Schlumberger Cambridge Research), P.J. Bilsby (WesternGeco), J. Quigley (WesternGeco) & E. Kragh (Schlumberger

More information

LECTURE 22. Numerical and Scientific Computing Part 2

LECTURE 22. Numerical and Scientific Computing Part 2 LECTURE 22 Numerical and Scientific Computing Part 2 MATPLOTLIB We re going to continue our discussion of scientific computing with matplotlib. Matplotlib is an incredibly powerful (and beautiful!) 2-D

More information

Advanced Python on Abel. Dmytro Karpenko Research Infrastructure Services group Department for Scientific Computing USIT, UiO

Advanced Python on Abel. Dmytro Karpenko Research Infrastructure Services group Department for Scientific Computing USIT, UiO Advanced Python on Abel Dmytro Karpenko Research Infrastructure Services group Department for Scientific Computing USIT, UiO Support for large, multi-dimensional arrays and matrices, and a large library

More information

Data Science and Machine Learning Essentials

Data Science and Machine Learning Essentials Data Science and Machine Learning Essentials Lab 3C Evaluating Models in Azure ML By Stephen Elston and Graeme Malcolm Overview In this lab, you will learn how to evaluate and improve the performance of

More information

TelFit Documentation. Release Kevin Gullikson

TelFit Documentation. Release Kevin Gullikson TelFit Documentation Release 1.3.0 Kevin Gullikson June 23, 2015 Contents 1 Introduction to Telluric Modeling with TelFit 3 2 Installation 5 3 TelFit Tutorial 7 3.1 Generating a Telluric Model with TelFit.................................

More information

Visualisation in python (with Matplotlib)

Visualisation in python (with Matplotlib) Visualisation in python (with Matplotlib) Thanks to all contributors: Ag Stephens, Stephen Pascoe. Introducing Matplotlib Matplotlib is a python 2D plotting library which produces publication quality figures

More information

Spectral Extraction of Extended Sources Using Wavelet Interpolation

Spectral Extraction of Extended Sources Using Wavelet Interpolation The 2005 HST Calibration Workshop Space Telescope Science Institute, 2005 A. M. Koekemoer, P. Goudfrooij, and L. L. Dressel, eds. Spectral Extraction of Extended Sources Using Wavelet Interpolation Paul

More information

PACS Spectrometer Simulation and the Extended to Point Correction

PACS Spectrometer Simulation and the Extended to Point Correction PACS Spectrometer Simulation and the Extended to Point Correction Jeroen de Jong February 11, 2016 Abstract This technical note describes simulating a PACS observation with a model source and its application

More information

AstroDendro Documentation

AstroDendro Documentation AstroDendro Documentation Release 0.0.0 Thomas Robitaille, Braden McDonald, Chris Beaumont, Erik Roso August 02, 2013 CONTENTS 1 About dendrograms 3 2 Documentation 5 2.1 Installing astrodendro........................................

More information

The STScI STIS Pipeline VII: Extraction of 1-D Spectra

The STScI STIS Pipeline VII: Extraction of 1-D Spectra STIS Instrument Science Report 97-02 The STScI STIS Pipeline VII: Extraction of 1-D Spectra Steve Hulbert, Phil Hodge, and Ivo Busko February 1997 ABSTRACT This report discusses the extraction of 1-D spectra

More information

ME30_Lab1_18JUL18. August 29, ME 30 Lab 1 - Introduction to Anaconda, JupyterLab, and Python

ME30_Lab1_18JUL18. August 29, ME 30 Lab 1 - Introduction to Anaconda, JupyterLab, and Python ME30_Lab1_18JUL18 August 29, 2018 1 ME 30 Lab 1 - Introduction to Anaconda, JupyterLab, and Python ME 30 ReDev Team 2018-07-18 Description and Summary: This lab introduces Anaconda, JupyterLab, and Python.

More information

PyPlot. The plotting library must be imported, and we will assume in these examples an import statement similar to those for numpy and math as

PyPlot. The plotting library must be imported, and we will assume in these examples an import statement similar to those for numpy and math as Geog 271 Geographic Data Analysis Fall 2015 PyPlot Graphicscanbeproducedin Pythonviaavarietyofpackages. We willuseapythonplotting package that is part of MatPlotLib, for which documentation can be found

More information