Visual Question Answering Using Multimodal Transformers for Post Disaster Assessment

Payyappilly, Leanda J. (2023) Visual Question Answering Using Multimodal Transformers for Post Disaster Assessment. Masters thesis, Indian Institute of Technology Hyderabad.

Text
CE20MTECH12003.pdf - Submitted Version
Restricted to Registered users only until 29 June 2028.
Download (73MB) | Request a copy

Abstract

Visual Question Answering (VQA) is a challenging multimodal task in the field of artificial intelligence with the goal of answering a question that is related to an image. It is a complex task that requires understanding of both natural language processing (NLP) and computer vision. VQA has its applications in various fields such medicine, video surveillance, for visually impaired personals, etc. The proposed work tries to extend the application of VQA towards post disaster assessment. In the aftermath of natural disasters such as floods, hurricanes, and earthquakes, VQA can play a critical role in aiding disaster relief efforts by fully grasping the situation from post disaster images and offering question- related responses. As of now, research in this field is still progressing and there is a need for developing good quality datasets as well as accurate algorithms need to be created. This work focuses on creating a VQA model using multimodal transformer fusion technique combining the different pretrained text and image transformer models available. The model performance is analyzed using the FloodNet dataset, a High-Resolution Aerial Imagery Dataset for Post Flood Scene Understanding. Also, an attempt is made to train the model on the Ida-BD dataset, a high-resolution satellite imagery for building damage assessment from Hurricane Ida.

[error in script]

IITH Creators: