Spatio-temporal Reasoning Benchmark

UAV-DualCog: A Dual-Cognition Benchmark for Grounded UAV Agent-World Reasoning with MLLMs

Like Liu¹Zhengzheng Xu²Hongzhe Li²Lin Wang³Dian Shao^2,†

School of Software, Northwestern Polytechnical University, Xi'an, China
Unmanned System Research Institute and School of Artificial Intelligence, Northwestern Polytechnical University, Xi'an, China
Embodied Perception and Interaction Lab (EmPACT Lab), School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

Overview

Why UAV dual cognition is necessary

Multimodal large language models have made strong progress on general image understanding, video understanding, and visual question answering, yet systematic evaluation remains limited for UAV embodied intelligence. Unlike ground-view settings, a UAV moves continuously in open 3D space and must reason not only about external targets, obstacles, and reachable directions, but also about its own position, current flight behavior, and future viewpoint changes. We therefore treat Agent-State Reasoning and World-State Reasoning as two complementary cognitive prerequisites for aerial embodied intelligence, rather than as isolated downstream tasks.

UAV-DualCog is designed around that dual-cognition view. At the capability level, it organizes the benchmark into Agent-State and World-State task lines; at the observation level, it spans both image and video settings. The benchmark is supported by a full automated toolchain that starts from semantic point clouds and landmark assets, then builds multi-view image QA, hierarchical flight-behavior video QA, and unified experiment outputs. With the current stable release, UAV-DualCog covers 12 AirSim scenes, 512 valid landmarks, 4,096 image QA samples, and 2,048 video QA samples, and exposes clear gaps between semantic answering, spatial grounding, behavior recognition, and temporal localization.

Reading guide: start from this homepage overview, then proceed in order through Benchmark, Construction, Evaluation, Leaderboard, Analysis, Sim-to-Real, and Training. The Usage page is the primary entry point for reproduction.

Benchmark Task definition, prompt design, examples, statistics, landmark assets, and flight behavior library. Construction Stage-by-stage pipeline, review checkpoints, intermediate artifacts, and toolchain logic. Sim-to-Real Three 3DGS reconstructions from real aerial imagery, 30 reviewed landmarks, 400 image tasks, and paired results for 16 models. Evaluation Metric design, evaluation protocol, prompt templates, and experiment setup. Leaderboard Image, video, capability-wise, and paired dual-cognition tables with ranking cues. Analysis Capability gaps, cross-modality trends, representative models, and benchmark insights. Training UAV-DualCog-Train construction, SFT and GRPO settings, and optimization results. Usage Download channels, codebase entry points, benchmark commands, and reproduction guidance.

Benchmark Snapshot

Benchmark Statistics

The current release contains 12 scenes, 512 released landmarks, 4,096 image samples, and 2,048 video samples, while the broader asset pool already spans 18 scenarios, 746 valid landmarks, and 166 fine-grained subcategories for future benchmark expansion.

Released Scenes

512

Released Landmarks

4096

Image Samples

2048

Video Samples

4840

Source Images

5h 27m 13s

Video Duration

Open Full Statistics

Benchmark statistics figure — **Benchmark Statistics.** Released task balance, landmark distribution, and video-side trajectory statistics in the benchmark.

Task Definitions

The released task matrix summarizes the six benchmark tasks, their modality, cognition axis, and structured output interface.

Task	Modality	Cognition	Output
Landmark-relative Position Reasoning Infer the UAV position relative to the target landmark from a landmark-centric reference view and a current egocentric observation.	image	Agent-State Reasoning	option + bbox
Action-conditioned Future Observation Prediction Predict which candidate image is the correct future observation after a described orbit action and localize the landmark in that selected image.	image	Agent-State Reasoning	option + bbox
Self-relative Position Reasoning Judge where the target landmark lies relative to the UAV's current forward direction and ground it in the same observation.	image	World-State Reasoning	option + bbox
Goal-conditioned Action Decision Choose which direction the UAV should move to approach the target landmark and ground the landmark in the current observation.	image	World-State Reasoning	option + bbox
Flight Behavior Recognition and Temporal Localization Recognize the UAV's own flight behaviors from first-person video and localize the corresponding temporal intervals.	video	Agent-State Reasoning	behavior option(s) + intervals
Landmark Visibility Counting and Interval Reasoning Count landmark appearances in flight video and localize every visible interval of the target landmark.	video	World-State Reasoning	count + intervals

Open Full Task Matrix

Task Examples

This homepage example browser mirrors the benchmark-page explorer so readers can inspect representative image and video tasks immediately after the task matrix.

Landmark-relative Position Reasoning

env_7_27_237_self_shared_4way_000149_where

Image 1 shows the Front facade of light gray mid-rise building with red tiled roof and dormer windows in the landmark-centric coordinate frame. Based on that reference, what is your position relative to the landmark in image 2? Select one option and return the normalized bounding box of the landmark in image 2. Options: A. Right B. Back C. Left D. Front

Original Question

Open-ended Question

2 images: 1 reference view and 1 query observation.

Reference Image

Query Observation

Ground Truth

Option: Option A (Right)

BBox: [0.398, 0.384, 0.562, 0.674]

Claude Opus 4.8

Option: A ✓

BBox: [0.270, 0.490, 0.620, 0.860] ✕ IoU=0.205 latency=46396 ms

GPT-6 Astra

Option: B ✕

BBox: [0.281, 0.703, 0.673, 1.000] ✕ IoU=0.000 latency=15076 ms

GPT-5.6 Sol

Option: B ✕

BBox: [0.266, 0.546, 0.645, 0.996] ✕ IoU=0.107 latency=113246 ms

Gemini 3.8 Flash

Option: C ✕

BBox: [0.398, 0.382, 0.564, 0.686] ✓ IoU=0.942 latency=5180 ms

Gemini 3.1 Pro Preview

Option: D ✕

BBox: [0.310, 0.380, 0.560, 0.800] ✕ IoU=0.444 latency=32703 ms

Muse Spark 1.3 Contributor

Option: - ✕

BBox: - ✕ IoU=0.000

Qwen3.7-Plus

Option: D ✕

BBox: [0.340, 0.450, 0.620, 0.850] ✕ IoU=0.299 latency=2548 ms

GLM-5.3-Flash

Option: B ✕

BBox: [0.330, 0.380, 0.620, 0.850] ✕ IoU=0.349 latency=7095 ms

Kimi K2.6

Option: B ✕

BBox: [0.297, 0.448, 0.562, 0.783] ✕ IoU=0.373 latency=4920 ms

MiMo-V2.5

Option: A ✓

BBox: [0.550, 0.100, 0.750, 0.400] ✕ IoU=0.002 latency=2534 ms

Qwen3.8-27B

Option: C ✕

BBox: [0.300, 0.400, 0.600, 0.800] ✕ IoU=0.366 latency=3241 ms

InternVL 3.5-30B-A3B

Option: C ✕

BBox: [0.420, 0.450, 0.580, 0.550] ✕ IoU=0.288 latency=48156 ms

SpaceOm

Option: C ✕

BBox: [0.600, 0.300, 0.800, 0.500] ✕ IoU=0.000 latency=653 ms

SpaceR

Option: C ✕

BBox: [0.100, 0.200, 0.300, 0.400] ✕ IoU=0.000 latency=853 ms

SpaceThinker

Option: C ✕

BBox: [0.000, 0.200, 0.200, 0.400] ✕ IoU=0.000 latency=436 ms

ViLaSR

Option: D ✕

BBox: [1.000, 1.000, 1.000, 1.000] ✕ IoU=0.000 latency=1223 ms

Prediction Summary. This Landmark-relative Position Reasoning question uses GT Option A (Right) and BBox [0.398, 0.384, 0.562, 0.674]. Across 16 available outputs, option prediction succeeds for 2/16 and spatial grounding for 1/16; both succeed for 0/16 (none). Semantic-only cases are Claude Opus 4.8 and MiMo-V2.5, evidence-only cases are Gemini 3.8 Flash, and 13/16 fail both. This exposes whether reference-frame reasoning and localization of the shared landmark succeed together.

Leaderboard

Current leading models

Top-3 views summarize modality, capability, and paired coordination results. The Dual columns use the same matched Agent–Landmark records and definitions as the paper tables. The full-width row resolves Dual Cognition into position, action–observation, and behavior–visibility pairings, followed by the cross-media aggregate.

Image

#	Model	Acc	Dual Sem.	Dual Evi.
1	Gemini 3.8 Flash	66.6%	44.4%	34.3%
2	Muse Spark 1.3 Contributor	54.0%	28.1%	4.5%
3	Gemini 3.1 Pro Preview	52.6%	27.1%	30.1%

Video

#	Model	Acc	Dual Sem.	Dual Evi.
1	Gemini 3.8 Flash	47.1%	6.9%	8.2%
2	Gemini 3 Flash	46.2%	11.4%	12.6%
3	Muse Spark 1.3 Contributor	45.4%	12.6%	16.2%

Agent-State

#	Model	Acc	Dual Sem.	Dual Evi.
1	Gemini 3.8 Flash	57.5%	12.0%	13.2%
2	Gemini 3.1 Pro Preview	46.9%	2.2%	3.8%
3	Gemini 3 Flash	44.2%	15.7%	3.4%

World-State

#	Model	Acc	Dual Sem.	Dual Evi.
1	Gemini 3.8 Flash	59.1%	12.0%	13.2%
2	Muse Spark 1.3 Contributor	58.8%	17.4%	7.1%
3	Qwen3.6-Flash	57.2%	13.4%	6.6%

Dual Cognition

#	Model	Position		Action–Observation		Behavior–Visibility		Overall
#	Model	Sem.	Evi.	Sem.	Evi.	Sem.	Evi.	Sem.	Evi.
1	Muse Spark 1.3 Contributor	24.7%	6.7%	32.6%	3.4%	12.6%	16.2%	17.4%	7.1%
2	Gemini 3 Flash	25.6%	1.9%	24.3%	2.1%	11.4%	12.6%	15.7%	3.4%
3	Qwen3.6-27B	17.5%	13.0%	19.3%	4.1%	13.4%	7.7%	15.5%	6.9%

Open Full Leaderboard

Dual-Cognition Coordination

Paired semantic and evidence profiles across three coordination demands

Six axes report paired co-success for reciprocal position reasoning, action–observation association, and behavior–visibility consistency in matched Agent–Landmark contexts.

Coordination remains profile-dependent

The six axes organize three paired demands under the same Agent–Landmark context. Relative-position coordination tests reciprocal UAV- and landmark-centric judgments; action–observation coordination links a commanded motion to its expected view and the action required for a target view; and behavior–visibility coordination couples UAV motion with landmark appearance over time.

Semantic and evidence co-success are reported independently for each pair. A broad, balanced polygon indicates higher co-success across the measured relations, whereas a narrow spike exposes task-specific strength. Gemini 3.8 Flash is strongest on the image pairings, while the compact video axes are led by different model families.

Exact values, aligned-record counts, and the complete model table are reported in the paired audit.

Open Full Dual-Cognition Table

Analysis

Dual-cognition analysis reveals complementary spatial and temporal capabilities

Dual cognition is the benchmark's main target, with images and videos probing complementary spatial and temporal demands. The analysis compares single-task results with Dual Semantic Acc and Dual Evidence Acc, then examines cross-media associations and the contributions of Agent-State and World-State Reasoning.

World-State semantic judgments are often stronger than Agent-State judgments, while grounding and paired success follow distinct model rankings. Human validation, diagnostic cases, open-ended questions, and reasoning-mode comparisons connect these quantitative patterns to the demands of coordinate-frame reasoning, action-observation association, and behavior-visibility consistency in UAV tasks.

Open Full Analysis

About

Citation

Please cite our paper when using UAV-DualCog, its datasets, or the accompanying code in your research. The BibTeX entry is provided below.

@misc{liu2026uavdualcogdualcognitionbenchmarkgrounded,
      title={UAV-DualCog: A Dual-Cognition Benchmark for Grounded UAV Agent-World Reasoning with MLLMs},
      author={Like Liu and Zhengzheng Xu and Hongzhe Li and Lin Wang and Dian Shao},
      year={2026},
      eprint={2607.16193},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.16193},
}

Contact

We welcome questions and feedback about UAV-DualCog, including dataset access, evaluation protocols, and implementation, as well as opportunities for research collaboration.

Like Liu · like.liu@mail.nwpu.edu.cn