Delay differential neural networks

Anumasa, S. and Srijith, P.K. (2021) Delay differential neural networks. In: 6th International Conference on Machine Learning Technologies, 23 April 2021 through 25 April 2021, Virtual, Online.

Text
Delay_Differential_Neural_Networks.pdf
Download (713kB)

Abstract

Neural ordinary differential equations (NODEs) treat computation of intermediate feature vectors as trajectories of ordinary differential equation parameterized by a neural network. In this paper, we propose a novel model, delay differential neural networks (DDNN), inspired by delay differential equations (DDEs). The proposed model considers the derivative of the hidden feature vector as a function of the current feature vector and past feature vectors (history) unlike only the current feature vector in the case of NODE. The function is modelled as a neural network and consequently, it leads to continuous depth alternatives to recent ResNet variants. For training DDNNs, we discuss a memory-efficient adjoint method for computing gradients and back-propagate through the network. DDNN improves the data efficiency of NODE by further reducing the number of parameters without affecting the generalization performance. Experiments conducted on real-world image classification datasets such as cifar10 and cifar100 to show the effectiveness of the proposed model. Â© 2021 ACM.

[error in script]

IITH Creators: