Hello, I am

Koushik Ahmed A Machine Learning Engineer Medium-Towards Data Science blogger

Model Idea

This uses a classical CNN-LSTM model like shown below, where Image features and language features are computed separately and combined together and a multi-layer perceptron is trained on the combined features.

Similar models have been presented at following links, this work takes ideas from them.

1. https://ai.googleblog.com/2023/07/modular-visual-question-answering-via.html
2. https://github.com/avisingh599/visual-qa
3. https://github.com/VT-vision-lab/VQA_LSTM_CNN

Pretrained VGG Net (VGG-16)

While VGG Net is not the best CNN model for image features, GoogLeNet (winner 2014) and ResNet (winner 2015) have superior classification scores, but VGG Net is very versatile, simple, relatively small and more importantly portable to use.

For reference here is the VGG 16 performance on ILSVRC-2012

Model Insight:

We have trained our model with only 9% of ms-coco datapoints where train - 40,000 samples with 10,000 of validation samples and 10,000 test samples.Datasamples are splited with stratified techniques.Custom Image augmentation applied to enhence model capacity of image feature understanding.Though initialize class weight to reduce bias of models it down-performs.We built custom datagenerator to split the performance on cpu based computatation.Before final layer,added 2 bidirectional LSTM layer on top of Attention layer to keep recurrence active.

Image-size:(224 X 224 X 3)

Batch size:32

Word Embedding::256

Text max-length: 100

Optimizer: Adam (.0001 with RPF=.3)

Learning rate decay: default

Reduce on plateau factor:0.3

Current model performance:

Challenges:

One of the challenges with working with ms-coco datasets is the large class imbalance present in the datasets. Ideally, we would train our model using an evenly balanced dataset so that all the class attribute of training cases would contribute equally to the loss.If we use a normal cross-entropy loss function with a highly unbalanced dataset, as we are seeing here, then the algorithm will be incentivized to prioritize the majority class, since it contributes more to the loss.

The custom weighted loss method for each training case will be ..

applying weighted labels within each class would have the same aggregate contribution to the loss function. Recall that for the multi-class loss, we add up the average loss for each individual class. Note that we also want to add a small value, epsilon, to the predicted values before taking their logs. This is simply to avoid a numerical error that would otherwise occur if the predicted value happens to be zero.

Update coming soon....