assessment of its interpretable features

Transcription

1 Black-box model explained through an assessment of its interpretable features Francesco Ventura, Tania Cerquitelli, Francesco Giacalone

2 Introduction In the last few years algorithms have been widely exploited in many practical use cases influencing various aspects of our life Even more complex machine learning algorithms support the data-driven decision-making process They can support the user in the decision making process They can make decision automatically It is very difficult to ascertain why and how they produce a certain output wrongdoing is possible

3 Introduction The most efficient machine learning algorithms operate essentially as black boxes E.g. deep neural networks Neural network architectures present a natural propensity to opacity in terms of understanding data processing and prediction The need for algorithmic transparency becomes even more prominent Focus on image classification through Deep Convolutional Neural Networks

4 EBAnO (Explaining BlAck-box model) To put existing, effective, and efficient DCNN to practical use cases. To provide explanations about the outcome produced through a deep convolutional neural network To define an explanation as the influence analysis of an interpretable set of features Oriented to black-box models the user is agnostic about the model the explanation takes into account the model

5 EBAnO (Explaining BlAck-box model) Given an input image It defines a set of interpretable features Interpretable features as set of correlated pixel The user is able to understand the semantic meaning of the set of pixels It exploits an iterative process of feature perturbations and classification It produces a prediction-local transparency report Specifically for a class of interest It produces two quality indexes Influence Relation (IR) Influence Relation Precision (IRP) It produces a visual report

6 EBAnO Process

7 Interpretable feature extraction Definition of a set of interpretable features to correctly explain the forecasting/classification of a black-box model It exploit Hypercolumns analysis of the DCNN Given a black box CNN composed of many layers and an image 1. Process the image with a CNN and get its representation through the most representative convolutional layers. 2. Extract the hypercolumns for each pixel of the image 3. A matrix of vectors is generated where each cell of the matrix represents the hypercolumn of an input pixel 4. Clustering analysis of the hypercolumns exploiting the k-means algorithm

8 Interpretable feature extraction Interpretable features Groups of pixels highly correlated within the black-box-model Hypercolumns are used to identify correlated portions of the image instead of well defined objects Interpretable features should be neither too specific nor too general a single pixel of an image is both totally trifling and completely opaque portions of image defined by a set of correlated pixels are more effective. The output of the cluster analysis produces k groups of correlated pixels corresponding to k interpretable features.

9 Explanation process For the original image EBAnO calculates the set of probability values for each membership class For each interpretable feature it starts an iterative process of 1. blur perturbation of the original image in correspondence of the given feature 2. Prediction of the set of probability values for each membership class of the perturbed image exploiting the black-box model

10 Transparency report Given a class of interest (e.g. the real class of the image) EBAnO computes two indices to explain the black-box model behaviour: The IR index (Influence Relation ) measures the local influence of the input feature with respect to the class of interest. The IRP index (Influence Relation Precision ) measures the inter-class influence of input features EBAnO produces a visual report Qualitative representation of the influence of each feature over the classification process Based on IR index

11 Transparency report: IR The IR index is calculated for each feature of an image It represents the ratio between the probability of the original image to belong to the class of interest and the corresponding probability obtained after the perturbation of the image. It ranges in [0,inf) IR values higher then 1 represent a positive influence of the feature IR values lower then 1 represent a negative influence over the prediction of the class of interest

12 Transparency report: IRP The IRP index is calculated for each feature of an image It is computed as the ratio between the IR value for the target class and the weighted average of IR for the whole set of predicted classes the weights correspond to the probabilities of the predicted membership class for the original image For IRP values lower than 1 the input feature not only has an impact on the predicted class but also on all the others For IRP value higher than 1, the importance of the input feature for the real class is significant with respect to the whole set of predicted classes. IRP represents the ability of an input feature to uniquely represent the class of the original image

13 Experimental results Preliminary implementation Developed in python Exploiting Keras with Tensorflow backend VGG16 as black-box model CNN for image classification Pre-Trained on imagenet 16 layers (convolutional and fully connected) 1000 classes 85 images 75 belong to different classes 10 belong to class pizza Extraction of 10 interpretable features for each image Hypercolumns extracted from last 10 convolutional layers

14 Experimental results: original prediction Class P(C) % hand_blower washbasin soap_dispenser toilet_seat 8.77 toilet_tissue 7.35 mouse 5.10

15 Experimental results: feature extraction

16 Experimental results: mouse class IR and IRP Feature Perturbation P(c)% IR IRP

17 Experimental results: mouse class visual report

18 Experimental results: IRP evaluation P(c=pizza)% = 3.71%

19 Influence Relation Influence Relation Precision

20 Influence Relation Feature 9 Influence Relation Precision

21 Conclusion and future works EBAnO is a preliminary project It is able to provide interpretable explanation for the image classification task through DCNN Analysis improvements Identify global influence explanations Make indexes and reports more user friendly Extend the analysis to different models E.G. VGG_19, Inception_v3, Resnet_v2 Support of different types of unstructured data E.G. Textual analysis

22 Thank you! Francesco Ventura, Tania Cerquitelli, Francesco Giacalone