Skip to content
Jaehyun Jun
한국어
Back to publications

International Conferences

Bilinear Attention Networks

A multimodal attention architecture that directly models interactions between image regions and question words, achieving state-of-the-art results on VQA and visual grounding benchmarks.

Published
2018
Venue
NeurIPS 2018
Authors
Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang
  • Visual Question Answering
  • Attention
  • Multimodal
Bilinear attention maps of a two-glimpse BAN (paper Figure 3). Each group pairs the image with its log-scaled attention map. The six most salient boxes from the first attention map are drawn on both images for comparison; the model answers brown.
Bilinear attention maps of a two-glimpse BAN (paper Figure 3). Each group pairs the image with its log-scaled attention map. The six most salient boxes from the first attention map are drawn on both images for comparison; the model answers brown.

Overview

Bilinear Attention Networks (BAN) addresses a central limitation of multimodal attention: image regions and question words need to interact directly, but modeling every pair naively is computationally expensive. BAN represents those pairwise interactions with a single bilinear attention distribution and makes the formulation practical through low-rank bilinear pooling.

The work was published at NeurIPS 2018 and evaluated across both visual question answering and visual grounding tasks.

My Contributions

  • Co-authored the research that formulated bilinear attention over pairs of visual and linguistic channels.
  • Contributed to the architecture that combines low-rank bilinear pooling with residual learning across multiple attention glimpses.
  • Participated in experimental validation on VQA 2.0, Flickr30k Entities, and the VQA Challenge 2018.

Technical Approach

  • Pairwise multimodal attention — Models interactions between every image region and question word directly instead of separating attention by modality.
  • Low-rank bilinear pooling — Compresses the otherwise expensive bilinear interaction into a trainable and computationally practical representation.
  • Residual multi-glimpse reasoning — Combines multiple attention maps without the parameter growth caused by naïvely stacking independent modules.

Results

  • Established new state-of-the-art results on VQA 2.0 and Flickr30k Entities at publication time.
  • Achieved 1st place in the single-model track and 2nd place in the ensemble track at the VQA Challenge 2018.
  • Demonstrated that direct pairwise attention can improve both question answering and visual grounding within one general architecture.

Why It Matters

BAN made fine-grained interaction between vision and language practical at scale. The architecture showed that attention does not need to treat each modality independently: explicitly modeling their pairwise relationships can provide a stronger shared representation for multimodal reasoning.