cosmic rays problem EUCLID VIS simulation M. Brescia - Data Mining

Transcription

1 Data Mining

2 CNN Strong Lensing Systems with magnified and distorted images of background objects, due to the deflection of light by massive foreground objects (lenses), can be used as astrophysical tools to probe mass distributions, magnify distant objects and measure fundamental cosmological parameters. Main problems with strong gravitational lenses is their rarity and false positive: For modern wide-field optical surveys which observe tens of millions of galaxies over many thousands square degrees, one expects to find only several hundred such systems. While strong lenses are relatively easy to spot by eye, machine learning has thus far been unsuccesful in reliably identifying strong lenses and distinguishing them from image artifacts (cosmic rays) and other false positive. But maybe CNN could help!! 2

3 cosmic rays problem EUCLID VIS simulation 3

4 cosmic rays problem EUCLID VIS simulation 4

5 cosmic rays problem EUCLID NISP simulation Slitless spectroscopy NIR spectral images 5

6 cosmic rays problem EUCLID SIR simulation Slitless spectroscopy Almost all spectra are contaminated by cosmic rays 6

7 cosmic rays problem EUCLID SIR simulation Slitless spectroscopy By splitting red and blue grisms, resulting spectrum is less affected in % 7

8 CNN Strong Lensing Do we have enough data to reasonably train and test a CNN? Can we get around this by artificially inflating the data, e.g. by adding rotated images? How do citizen-scientists do compared with this automated system? Can we use the results of citizenscientists to train the CNN? By using features extracted from a CNN trained on galaxy morphology how well does it perform when determining the presence of strong gravitational lenses? 8

9 CNN Strong Lensing SPACE WARPS ( is a web service that enables the discovery of strong gravitational lenses in wide-field imaging surveys by large numbers of people. Carefully produced, color composite images are displayed to volunteers via a classification interface which records their estimates of the positions of candidate lensed features. Simulated lenses and non-lenses are inserted into the image stream at random intervals; this training set is used to give the volunteers feedback on their performance, and to estimate a dynamically-updated probability for any given image to contain a lens. Base dataset has 24,177 images from stage 1, of which 5159 simulated lenses, and 1876 images from stage 2, of which 151 simulated lenses. There are 9030 classifications that stage 2 users made of images in the CFHTLS survey where it is unknown whether they contain a lens or not. From these classifications, a list of approximately 40 candidate objects have been found which will soon receive spectroscopic follow-up. 9

10 CNN Strong Lensing ML classifier The used Convolutional Neural Network (CNN) is built from the ground up using THEANO. It is based on a five-layer architecture consisting of two convolution/max dropout layers, a fully connected layer, and a softmax layer. Here the CNN is used as a feature extractor, and take images from the fully connected layer. Thus it transforms a 3x96x96 image into a 500 feature vector. Then it trains these feature vectors on various classifiers (Random Forest, Support Vector Machine, Softmax) and evaluate results against a test set. 10

11 CNN Strong Lensing The used Convolutional Neural Network (CNN) is built from the ground up using THEANO. It is based on a five-layer architecture consisting of two convolution/max dropout layers, a fully connected layer, and a softmax layer. The results of training it on 8,000 randomly selected images from the SPACE WARPS data set for 35 epochs is shown, where is plotted the ROC Curve obtained from model s predictions on the training set and a test set composed of another (non-overlapping) randomly selected 8,000 images from the SPACE WARPS data set. This result should be easily improved on by training on more images, by adding data augmentation and by increasing CNN complexity. 11

12 CNN Strong Lensing ROC of SPACE WARPS system and different linear classifiers trained on feature vectors extracted from a CNN originally used to determine galaxy morphologies Softmax classifications perform best on the test dataset, however all the feature vectors perform worse than the users themselves User classification by eyes 12

13 Future CNN - Astroparticles In the past decade, our understanding of the very-high-energy (VHE; 100 GeV < Eγ < 100 TeV) gamma-ray sky has greatly progressed by the use of stereoscopic imaging atmospheric Cherenkov telescopes (IACTs). They record images of an extensive air shower induced by an incident VHE gamma-ray photon or a cosmic-ray (CR) particle. The air-shower images are then analyzed to reconstruct the information of the incident photons or the CR particles, the latter of which form a substantial background. The ability to separate gamma rays from CR particles is important, as it is directly related to the sensitivity of the instrument. 13

14 Future CNN - Astroparticles Few geometric image parameters, e.g. width and length, of an air-shower image are shown to be effective in discriminating gamma rays from CR background. These parameters exploit our knowledge of the well-understood air shower physics, specifically the fact that a CR shower typically produces a wider image as it generally carries a larger transverse momentum as a result of hadronic interactions. The long established analysis method is to apply box cuts to the image parameters (Hillas 1985). It is simple and effective, therefore has been the standard analysis method for two decades. However, the box-cuts method mentioned above is complicated by several types of analysis methods. Some of these advanced methods exploit more details in the air showers or their images, e.g. 3-D reconstruction of air showers (Lemoine-Goumard et al. 2006), adding timing structure (Aliu et al. 2009), fitting a 2-D Gaussian to shower images (Christiansen 2012), using a 3-D maximum likelihood analysis (Cardenzana 2015), and matching image templates (Vincent 2015). The image parameters in VHE gamma-ray data analysis greatly reduce the dimension of the data using domain knowledge. However, the use of fitted parameters inevitably leads to information loss. CNN models work directly on raw images of VHE gamma-ray events, exploiting pixel-level information in the data. But this technique is still under investigation by H2020 ASTERICS PROJECT 14

15 Future CNN - Astroparticles Alternative Procedure (Feng et al. 2017, IAU Proceedings, in press) 15

16 Future CNN - Astroparticles Alternative Procedure (Feng et al. 2017, IAU Proceedings, in press) Vi ricordate il metodo per la smoke detection con camere non infrarosse, basato sull analisi dinamica della variazione della standard deviation?... 16

17 Future CNN - Astroparticles gamma-ray photon Cosmic-ray photon 17

18 DNN 18

19 DNN 19

20 DNN 20

21 DNN 21

22 DNN - lezione 4 22

23 DNN 23

24 DNN 24

25 DNN - What is Caffe? Open framework, models, and worked examples for deep learning 2 years old 1,000+ citations, 150+ contributors, >1 pull request / day average focus has been vision, but branching out speech + text Pure C++ / CUDA library for deep learning Command line, Python, MATLAB interfaces Fast, well-tested code Tools, reference models, demos, and recipes Seamless switch between CPU and GPU Prototype Train Deploy 25

26 DNN - Caffe All in a day s work with Caffe Prototype Train Deploy 26

27 DNN - TensorFlow TensorFlow is an open source python software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well. 27

28 DNN - Theano Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano features: tight integration with NumPy Use numpy.ndarray in Theano-compiled functions. transparent use of a GPU Perform data-intensive calculations up to 140x faster than with CPU.(float32 only) efficient symbolic differentiation Theano does your derivatives for function with one or many inputs. speed and stability optimizations Get the right answer for log(1+x) even when x is really tiny. dynamic C code generation Evaluate expressions faster. extensive unit-testing and self-verification Detect and diagnose many types of errors. 28

29 import theano import theano.tensor as T import numpy as np A NN in 28 lines of Theano X = theano.shared(value=np.asarray([[1, 0], [0, 0], [0, 1], [1, 1]]), name='x') y = theano.shared(value=np.asarray([[1], [0], [1], [0]]), name='y') rng = np.random.randomstate(1234) LEARNING_RATE = 0.01 def layer(n_in, n_out): return theano.shared(value=np.asarray(rng.uniform(low=-1.0, high=1.0, size=(n_in, n_out)), dtype=theano.config.floatx), name='w', borrow=true) W1 = layer(2, 3) W2 = layer(3, 1) output = T.nnet.sigmoid(T.dot(T.nnet.sigmoid(T.dot(X, W1)), W2)) cost = T.sum((y - output) ** 2) updates = [(W1, W1 - LEARNING_RATE * T.grad(cost, W1)), (W2, W2 - LEARNING_RATE * T.grad(cost, W2))] train = theano.function(inputs=[], outputs=[], updates=updates) test = theano.function(inputs=[], outputs=[output]) for i in range(60000): if (i+1) % == 0: print(i+1) train() print(test()) 29

30 DNN - Theano X = theano.shared(value=np.asarray([[1, 0], [0, 0], [0, 1], [1, 1]]), name='x') y = theano.shared(value=np.asarray([[1], [0], [1], [0]]), name='y') rng = np.random.randomstate(1234) LEARNING_RATE = 0.01 Here, we re creating shared variables X and y, representing our inputs and outputs, respectively. Shared variables are like global variables in a programming language; they are shared between functions, such as the functions train and test later on. We also initialize a random number generator rng and define a learning rate. def layer(n_in, n_out): return theano.shared(value=np.asarray(rng.uniform(low=-1.0, high=1.0, size=(n_in, n_out)), dtype=theano.config.floatx), name='w', borrow=true)&amp;lt;/pre&amp;gt; W1 = layer(2, 3) W2 = layer(3, 1) Here, we define a function which creates and returns a matrix of random numbers between -1.0 and 1.0, whose size we specify. The matrix is also a shared variable. We use this function to create the weights W1 and W2 for our network. 30

31 DNN - Theano output = T.nnet.sigmoid(T.dot(T.nnet.sigmoid(T.dot(X, W1)), W2)) cost = T.sum((y - output) ** 2) updates = [(W1, W1 - LEARNING_RATE * T.grad(cost, W1)), (W2, W2 - LEARNING_RATE * T.grad(cost, W2))] We finally get into constructing the network. Theano usefully includes the sigmoid function, which is used as the network s activation function. We multiply the input vector X by the first weight matrix and apply the activation function; we then take this output and multiply it by the second weight matrix before again applying the activation function. 31

32 DNN - Torch Torch is a scientific computing framework with wide support for machine learning algorithms that puts GPUs first. It is easy to use and efficient, thanks to an easy and fast scripting language, LuaJIT, and an underlying C/CUDA implementation. A summary of core features: a powerful N-dimensional array lots of routines for indexing, slicing, transposing,... amazing interface to C linear algebra routines neural network, and energy-based models numeric optimization routines Fast and efficient GPU support Embeddable, with ports to ios, Android and FPGA backends The goal of Torch is to have maximum flexibility and speed in building your scientific algorithms while making the process extremely simple. 32

33 DNN Nvidia resources The NVIDIA Deep Learning SDK provides high-performance tools and libraries to power innovative GPU-accelerated machine learning applications in the cloud, data centers, workstations, and embedded platforms 33

34 cudnn Nvidia resources cudnn is thread safe, and offers a context-based API that allows for easy multithreading and (optional) interoperability with CUDA streams. This allows the developer to explicitly control the library setup when using multiple host threads and multiple GPUs, and ensure that a particular GPU device is always used in a particular host thread 34

35 cudnn Nvidia resources cudnn is integrated into the development branch of the CAFFE neural network toolkit. It is expected to be part of the official CAFFE 1.0 release. In CAFFE, a DNN is completely defined and implemented via text-based configuration files. With CAFFE you define each of the layers of your neural network, specifying the type of the layer (eg. data, convolutional, or fully connected) and the layers that provide its input. There is a very similar configuration file to define how to initialize the parameters of your network and how many iterations to train it for and so on. The following is a slightly simplified example of a CAFFE neural network definition configuration with one data layer and two convolutional layers. layers { name: MyData type: DATA top: data top: label } layers { name: Conv1 type: CONVOLUTION bottom: MyData top: Conv1 convolution_param { num_output: 96 kernel_size: 11 stride: 4 } } layers { name: Conv2 type: CONVOLUTION bottom: Conv1 top: Conv2 convolution_param { num_output: 256 kernel_size: 5 } } 35

36 cudnn Nvidia resources NVIDIA CUDA Deep Neural Network library (cudnn) is a GPU-accelerated library of primitives for deep neural networks. cudnn provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cudnn is part of the NVIDIA Deep Learning SDK. Deep learning researchers and framework developers worldwide rely on cudnn for highperformance GPU acceleration. It allows them to focus on training neural networks and developing software applications rather than spending time on low-level GPU performance tuning. cudnn accelerates deep learning frameworks, including Caffe, TensorFlow, Theano, Torch. The cudnn library is targeted at developers of DNN frameworks (eg. CAFFE, Torch). However it is easy to use directly and you do not need to know CUDA in order to use it. The next example code shows how to allocate storage for an input batch of images and a convolutional filter in cudnn, and how to run the batch in the forward direction through a convolutional layer. 36

37 cudnn Nvidia resources The calls to cudnnsettensor4ddescriptor() and cudnnsetfilterdescriptor() define the input to this convolutional layer and filter parameters, respectively. 37

38 cudnn Nvidia resources The call to cudnnsetconvolutiondescriptor initializes the convolution descriptor for this layer using the descriptors from the previous two calls and some layer-specific information such as padding and striding parameters. The following call, cudnngetoutputtensor4ddim(), calculates the dimensions of the convolution s output for you. 38

39 cudnn Nvidia resources The next calls simply configure and allocate storage for the output of this layer, and then cudnnconvolutionforward() performs the NVIDIA-tuned convolution. 39