Lightweight Neural Networks from PCA & LDA Based Distilled Dense Neural Networks

This paper presents two methods for building lightweight neural networks with similar accuracy than heavyweight ones with the advantage to be less greedy in memory and computing resources. So it can be implemented in edge and IoT devices. The presented distillation methods are respectively based on Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). The two methods basically rely on the successive dimension reduction of a given dense neural network (teacher) hidden features, and the learning of a \textit{smaller} neural network (student) which solves the initial learning problem along with a mapping problem to the reduced successive features spaces. The presented methods are compared to baselines --learning the student networks from scratch--, and we show that the additional mapping problem significantly improves the performance (accuracy, memory and computing resources) of the student networks.