Training Deep Predictive Coding Networks

Abstract

Energy-based models trained with equilibrium propagation are neural networks that perform inference through an iterative energy minimization process. Previous studies have demonstrated the effectiveness of this class of models in shallow architectures, but their performance degrades significantly as the depth increases to more than five/seven layers. In this study, we use models trained using the predictive coding energy to show that the reason behind this degradation is due to (1) errors between layers during weight updating, and (2) predictions from the previous layer not being effective in guiding updates in deeper layers. We address this by introducing both a novel weight update mechanism that reduces error accumulation in deeper layers, and a method to optimize the distribution of energy among layers during the `relaxation phase'. Empirically, we show that our methods largely improve both training and test accuracy across networks with more than seven layers, achieving comparable results to equivalent models trained with backpropagation. These initial findings suggest that a better understanding of the relaxation phase is important to train models using equilibrium propagation at scale, and open new possibilities for their application in complex tasks.