What is Adversarial VQA?

Adversarial VQA is a new VQA benchmark that is collected with Human-And-Model-in-the-Loop for evaluating the robustness of state-of-the-art VQA systems.

  • 2 datasets: AdVQA (in-domain) and AVQA (out-of-domain)
  • Collected in single round or multiple rounds
  • 81,253 images (COCO/Conceptual Captions 3M/Fakeddit/VCR)
  • 1.9 human-verified adversarial questions on average per image
  • 10 ground truth human-written answers per verified question

Dataset

Details on downloading the latest dataset may be found on the download webpage.

  • August 2021: Full release (v1.0)

AdVQA (In-domain)
  • Collected in single round
  • 41,807 COCO images
    (only for val/test)
  • 46,807 questions
  • 468,070 human-written answers
AVQA (Out-of-domain)
  • Collected with 3 rounds
  • 40,637 images from Conceptual Captions/
    Fakeddit/VCR (for train/val/test)
  • 104,410 verified questions, 73,075 unverified questions
  • 1,044,100 human-written answers for verified questions,
    73,075 VQA model answers for unverified questions

Papers

Human-Adversarial Visual Question Answering [Paper] [BibTeX]




Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models (ICCV 2021) [Paper] [BibTeX]





Contact

Have any questions or suggestions? Feel free to reach us at [email protected]!