Continual Learning in Vision Transformer

Deep Neural Networks (DNNs) are widely used state-of-the-art approaches for the massive audio signal, language and vision tasks. However, we still lack sufficient understanding of the theoretical issues inside. Based on the equivalence between deep residual networks (ResNets) and the Euler forward scheme, we present several ways to construct DNNs (ResNet as the example) in advanced numerical and linear multi-step schemes. Furthermore, we show that ResNets with various advanced schemes have better accuracy, convergence, robustness and an explainable rank against the property of these schemes. Finally, previous works are summarized and discussed; sufficient experiments are provided. Improvements in several aspects are noticeable and theoretical.