Interpreting Deep Learning Models

Abstract

Cognitive Science and model of brain has always been great contributors to the Deep Learning. Based on how our cognition works, we have been able to create models that can achieve human level classification accuracy. Due to its nature and our curiosity we have been developing ways to understand, interpret and evaluate Deep Learning models. However, most of these interpretations are either completely metric or visual assumptions. These methods do not explain what and how a neural network is thinking. On the other hand, there are many Cognitive Science methods developed to understand and interpret what brain is thinking. One of such a method would be Multivariate Pattern Analysis of fMRI images. It is a method to learn neural contribution to stimuli, then interpreting stimuli through learned neural activity. In this project, influenced by the deep connection between Neural Networks development and Cognitive Science, we are proposing a method to interpret what a Neural Network is going to predict. We create a method to apply Multivariate Pattern Analysis on Neural Networks. To achieve this, we first create a Neural Network trained on CIFAR-10 public dataset. We train the model until we have a satisfactory result of around 70%. We train Support Vector Machines for binary classification of each class in CIFAR-10 dataset. The Support Vector Machines do not require to have a good accuracy as they may not be able to converge due to number of features. Then we use Recursive Feature Elimination discussed in to eliminate the noncontributing neurons. The neurons are eliminated until there are 80, 40, 20, 10 neurons left. At this point we have a good accuracy for binary classification which is generally around 80%. After eliminating the neurons, we create a new classifier based on the combination of binary classifier results for the test data and we try to interpret what the neural network is thinking. The results are compared to PCA and ICA and achieved higher or comparable results overall. Based on the neuron we also created a visualization of neural connection based on classes they contribute to.