Employing Attention Mechanism for Compute-efficient Design of Compact DNNs

Saini, Rajat and Mittal, Sparsh (2019) Employing Attention Mechanism for Compute-efficient Design of Compact DNNs. Masters thesis, Indian institute of technology Hyderabad.

Text
Thesis_Mtech_CS_5545.pdf - Submitted Version
Restricted to Repository staff only until June 2020.
Download (2MB) | Request a copy

Abstract

Convolutional neural networks (CNNs) are now ubiquitous in artificial intelligence domains due to their unprecedented performance. However, the increasing computational cost and the model size prohibit the deployment of CNNs on resource-constrained devices. Compared to the large models, the high-performance compact CNNs such as MobileNet-V1 and MobileNet-V2 have reduced the computations to a great extent. However, both these compact CNNs incur significant computation overhead due to the computationally-inefficient learning of cross channel information. In this work, we propose a novel ultra-lightweight and compute-efficient “grouped attention block” mechanism (GrAB) which reduces the computations by using compute-efficient learning of cross channel information and hence, making CNNs more compact with negligible performance loss. Our proposed block (GrAB) is end-to-end trainable and can be easily incorporated in the existing compact CNNs while exercising better trade-off between number of computations and performance. Notably, our work is the first attempt which aims to reduce the computational cost in compact CNNs, with negligible performance loss, using attention mechanism. We perform our experiments with MobileNet-V1 and MobileNet-V2 on ImageNet-1K dataset and also on the fine-grained datasets, specifically Stanford Dogs and Caltech-UCSD Birds 200. We show the scalability and effectiveness of our blocks through extensive ablation study

[error in script]

IITH Creators: