Exploring the effect of clustered hidden representations on generalisation in DNNs

Abstract

Different approaches have been used to better understand the generalisation ability of deep neural networks (DNNs), that is, their ability to perform well on unseen data. An approach that has only been studied to a limited extent to date, is the way in which hidden representations form implicit clusters within DNNs. Each such cluster is either formed by representations that have similar values, or by some features for some samples that have similar representation values.

We build upon previous work related to clustering in DNNs by exploring the characteristics of representation clusters. Specifically, we investigate the forming of sample-feature clusters between individual and combinations of nodes throughout the network layers. We aim to find a relationship between generalisation and the characteristics of the clusters formed. We investigate these factors on DNNs with different generalisation abilities in order to support or disprove our theoretical intuition on the role of cluster forming in DNNs.

Using existing clustering methods such as k-means, we find that models with better generalisation tend to form purer clusters than those with poorer generalisation. We explore biclustering as a means of identifying sample-feature clusters and find that biclustered representations are also predictive of generalisation. This time, however, models with purer biclusters at the convolutional layers tend to generalise more poorly. We mitigate the complexity introduced by the high dimensionality of representations by using autoencoders to compress the representations. We find that these compressed representations tend to be predictive of generalisation, with purer clusters again predictive of better generalisation. Combined, these findings provide a novel perspective on the effect of clustered representations on the generalisation ability of DNNs.