VANTAGE-Bench

Introduction

We introduce VANTAGE-Bench, a benchmark for evaluating vision-language models on fixed-camera operational video. Instead of internet video, egocentric clips, or broadcast footage, it focuses on real-world scenes from warehouses, intersections, and public spaces where the camera remains fixed and the model must reason from a stable vantage point.

VANTAGE-Bench contains 35,027 expert-curated annotations across 3,346 media assets. It spans three operational domains and four reasoning pillars across both image and video. The benchmark stands out for its domain relevance, modality breadth, task diversity, and evaluation novelty, culminating in the first quantitative single-object tracking benchmark for VLMs in fixed-camera operational footage.

Task taxonomy and distribution of VANTAGE-Bench across three operational domains and four reasoning pillars.

The architecture of VANTAGE-Bench is driven by the fundamental disconnect between how VLMs are currently evaluated and how operational video systems actually work. Existing video benchmarks rely on a cinematic prior, with dynamic and human-centric framing, and reduce evaluation to a single format, typically multiple choice. VANTAGE-Bench addresses both limitations. It uses footage from fixed-infrastructure cameras that remove internet-video priors and extends evaluation beyond multiple choice to include generative dense captioning, precise coordinate prediction, and continuous spatio-temporal tracking. These are tasks that cannot be solved by selecting from a predefined set of answers.

Leaderboard

Zero-shot evaluation of frontier and open-weight models across all four reasoning pillars. All models evaluated under identical conditions — greedy decoding, no chain-of-thought.

Bold = best in column  ·  ✓ verified = independently confirmed  ·  Updated 2026-05-29

Infrastructure AI

Infrastructure AI is a sub-domain of Physical AI focused on fixed-infrastructure cameras such as CCTV networks, elevated sensors, and wide-angle lenses. These systems are deployed for safety monitoring, access control, traffic understanding, and operational logging. Unlike Embodied AI, which centers on moving agents navigating environments, Infrastructure AI operates from a persistent, stationary vantage point.

VANTAGE-Bench is purpose-built for this setting. Its footage, drawn from warehouses, roads, and public spaces, comes from cameras that remain fixed. Models must reason from a wide-area, static perspective rather than from edited, human-centered video.

Frontier models are largely trained on internet-crawled data and are therefore biased toward standard photographic perspectives. Under fixed-camera conditions, this prior breaks down. Models must instead rely on spatial-temporal reasoning. A model can achieve state-of-the-art performance on internet video while still exhibiting dangerous performance deficits in the environments that matter most.

Operational Domains

VANTAGE-Bench evaluates models across three structurally distinct deployment environments. Each domain requires different reasoning capabilities and exposes different failure modes in current VLMs.

W

Warehouse

Dense logistics environments with structured layouts, repeated objects, and human-robot interaction. Footage from elevated fixed cameras captures forklift operations, pallet movements, worker activity, and access control.

Forklift trackingPallet localizationWorker detectionAccess controlRobot profiles

T

Transportation

Roadside and intersection monitoring with multi-vehicle scenes. Fixed roadside cameras capture traffic flow, pedestrian crossings, and safety-critical events.

Vehicle detectionCollision verificationTraffic flowPedestrian crossing

SS

Smart Spaces

Unstructured public and semi-public environments with ambiguous human behavior. Models must reason about access control, tailgating, and crowd dynamics without structured interaction cues.

Crowd safetyAccess controlActivity detectionTailgating

Dataset

Available on Hugging Face: nvidia/PhysicalAI-VANTAGE-Bench →

VANTAGE-Bench was built around footage that is genuinely hard to source — specifically fixed-infrastructure cameras in real operational environments. Annotations were produced by trained professionals using domain-specific guidelines rather than crowdsourcing, and each annotation was reviewed by a secondary expert before inclusion.

Task Taxonomy & Metrics

VANTAGE-Bench is organized around four reasoning pillars. Each pillar targets a capability that current VLMs handle well in internet-video settings, but struggle with when the camera stops moving.

Benchmark Comparison

How does VANTAGE-Bench fit into the existing evaluation landscape?

Most VLM benchmarks focus on a single reasoning dimension and typically cover only one modality. VANTAGE-Bench is the first benchmark to jointly evaluate all four reasoning dimensions across both image and video in a single suite.

Table 1 — Scope comparison

What this shows: how VANTAGE-Bench compares to the benchmarks it most directly relates to in terms of scale, modality coverage, and task diversity.

PL = programmatically generated labels from human-verified annotations Table 1: VANTAGE-Bench is the only benchmark spanning all four reasoning dimensions across both image and video modalities.

Does VANTAGE-Bench actually measure something different?

Yes, and by a significant margin. The table below compares the same model (Qwen3-VL-8B) on published scores on standard consumer-centric benchmarks against its scores on equivalent tasks in VANTAGE-Bench.

Table 2 — The performance gap

Qwen3-VL-8B zero-shot scores on the standard reference benchmark for each task vs. its score on the equivalent VANTAGE-Bench task. A negative gap means the model performs worse on VANTAGE-Bench. All scores are scaled 0–100.

Table 2: Performance gap for Qwen3-VL-8B across tasks. Negative values indicate degradation on VANTAGE-Bench relative to the standard reference benchmark for that task. The largest drops occur in Event Verification (−29.96) and 2D Pointing (−23.56), confirming that fixed-camera footage breaks priors that models rely on in consumer-centric settings. Single Object Tracking has no reference score because no prior VLM tracking benchmark exists. * Qwen3-VL-8B scored 31.44 AUC on SOT; best overall is Gemini 3.1 Pro at 64.35 AUC.

Citation

If you use VANTAGE-Bench, the leaderboard, or its evaluation resources in your research, please cite:

@misc{vantagebench2026,
  title        = {VANTAGE-Bench: A Benchmark for Vision-Language Models on Fixed-Camera Infrastructure AI},
  author       = {{VANTAGE-Bench Team}},
  year         = {2026},
  howpublished = {\url{https://github.com/Clemson-Capstone/VANTAGE-Bench}},
  note         = {Benchmark, dataset, evaluation framework, and public leaderboard. Leaderboard: https://huggingface.co/spaces/clemson-computing/VANTAGE-Bench-Leaderboard}
}

How to Evaluate

To evaluate your model on VANTAGE-Bench, use the official evaluation harness. The harness handles data loading, prompt formatting, and inference, and exports predictions in the required LLaVA submission format automatically.

Once inference is complete, archive your prediction files and submit through the submission portal. Our server scores predictions against held-out ground truth. Results are emailed to you after evaluation.

Full setup instructions, task-specific configurations, and format documentation are in the GitHub repository. Per-task prediction schemas are documented on the submission page →