Exploring data utilisation in neural networks

Abstract

The generalisation ability of deep neural networks differs somewhat from that of more traditional models. Speciőcally, large networks that have the ability to łmemorisež the training data can still generalise well, regardless of regularisation. This generalisation ability is still not completely understood. As part of a larger approach to studying the generalisation ability of neural networks by measuring the utilisation of training data, we aim to őnd a method of measuring mutual information and redundancy, speciőcally in the context of information processing in neural networks. To this end, we study various existing redundancy estimators, and, using these as inspiration, we develop a new, nearest-neighbours-based redundancy estimator that can be used with discrete-continuous mixture distributions. We evaluate this new estimator on synthetically generated data and compare its behaviour to that of existing estimators. As a demonstration of the use of this estimator in neural network analysis, we calculate various node-based mutual information estimates in fully connected, feedforward networks trained for classiőcation. Our demonstration reveals interesting regularities and differences between networks with different generalisation characteristics. Overall, we implement and evaluate several redundancy estimators and show that node-based redundancy estimates can be used to analyse neural networks.