DEMON: IMPROVED NEURAL NETWORK TRAINING WITH MOMENTUM DECAY

Momentum is a widely used technique for gradient-based optimizers in deep learning. Here, we propose a decaying momentum (Demon) hyperparameter rule. We conduct large-scale empirical analysis of momentum decay methods for modern neural network optimization and compare to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, Demon achieves Top-1 and Top-3 performance in 39% and 85% of cases, respectively, almost doubling the second-placed cosine learning rate schedule at 17% and 60%, respectively. Demon consistently outperforms other widely-used schedulers including, but not limited to, the learning rate step schedule, linear schedule, OneCycle schedule, and exponential schedule. Compared with the widely-used learning rate step schedule, Demon is less sensitive to parameter tuning, which is critical to training neural networks in practice. Results are demonstrated across a variety of settings and architectures, including image classification models, generative models, and language models. Demon is easy to implement, requires no additional tuning, and incurs almost no extra computational overhead compared to the vanilla counterparts. Code is readily available.