1326
Registered aerial views
Sim-to-Real Extension
Sim-to-Real 3DGS Validation
A three-scene GauUscene pilot tests whether UAV-DualCog construction and paired evaluation transfer from AirSim to 3D Gaussian Splatting reconstructions built from real aerial imagery and metric geometry.
Scope & Scene Library
Real aerial appearance and metric geometry across three reconstructed scenes
LFLS, SMBU, and Upper Campus provide directly loadable CityGaussianV2 reconstructions together with GauUscene V2.1 camera poses and LiDAR. The pilot evaluates the construction and paired assessment protocol on rendered observations from real aerial scenes, combining reconstructed appearance with metric geometry.
30
Reviewed landmarks
400
8-way image questions
16
Reported models
The library follows the same visual audit pattern as the Construction page. An oblique RGB overview rendered from the 3DGS scene shows reconstructed appearance; top-down and oblique LiDAR point-cloud views show scene geometry without semantic category overlays; and a local aerial view shows one task landmark.
1 / 3
Scene scope. The final set contains 11 LFLS, 10 SMBU, and 9 Upper Campus landmarks. All 30 landmarks are reviewed across multiple registered views before task generation.
Platform Extension & Construction
The common bridge preserves the benchmark contract across reconstructed scenes
Step 1
Metric scene ingestion
CityGaussianV2 provides the reconstructed appearance. GauUscene V2.1 contributes metric COLMAP cameras and LiDAR geometry. A scene-specific similarity transform aligns camera poses, LiDAR points, and Gaussian coordinates without retraining the released reconstruction.
Step 2
Common bridge adaptation
The 3DGS adapter implements the same pose-driven capture interface used by the benchmark pipeline. It exposes RGB, depth, alpha, camera projection, and geometry queries so downstream construction code remains independent of the rendering backend.
Step 3
Candidate landmark mining
We sample 200 spatially distributed registered views per scene, segment visually distinctive objects, lift masks through metric depth, and fuse observations in 3D. Spatial deduplication produces 100 candidate landmarks per scene for review.
Step 4
Multiview review
Candidates are checked for visual clarity, cross-view identity, category coverage, minimum view support, projected-box quality, and canonical orientation. The retained 30 landmarks span buildings, vegetation, infrastructure, and urban facilities.
Step 5
Task generation
The shared DCCS image module reuses registered observations and geometry-derived relations to generate four high-difficulty 8-way tasks. Landmark use is balanced where feasible, and every sample follows the same answer-plus-BBox output contract as the AirSim benchmark.
Step 6
Release audit
The final 400 questions are checked for valid media, nondegenerate boxes, option consistency, landmark coverage, and deterministic pair identities. The pilot is reported in its own tables to preserve its scene composition, difficulty, and sampling protocol.
Task Examples
Sim-to-Real Task Examples
Each scene contributes one verified example per task. As in the Benchmark examples, the answer toggle overlays the ground-truth and selected model boxes directly on the query observation, alongside semantic and evidence outcomes from representative proprietary and open-weight models.
Landmark-relative Position Reasoning
gauu_lfls_gauu_lfls_candidate_0038_self_shared_8way_010005_where
Image 1 shows the Front-Right facade of dense green deciduous tree cluster with irregular shape located in urban gap in the landmark-centric coordinate frame. Based on that reference, what is your position relative to the landmark in image 2? Select one option and return the normalized bounding box of the landmark in image 2. Options: A. Back B. Left C. Back-Left D. Front-Left
Original Question
-
Open-ended Question
-
dense green deciduous tree cluster with irregular shape located in urban gap
Reference landmark view
Query observation
Ground Truth
Option: C
BBox: [0.478, 0.427, 0.549, 0.511]
Option: C ✓
BBox: [0.430, 0.420, 0.550, 0.540] ✕ IoU = 0.413
Option: A ✕
BBox: [0.272, 0.455, 0.459, 0.652] ✕ IoU = 0.000
Option: C ✓
BBox: [0.320, 0.660, 0.590, 0.810] ✕ IoU = 0.000
Option: D ✕
BBox: [0.260, 0.840, 0.325, 0.920] ✕ IoU = 0.000
Option: A ✕
BBox: [0.510, 0.430, 0.620, 0.520] ✕ IoU = 0.250
Option: B ✕
BBox: [0.420, 0.400, 0.560, 0.580] ✕ IoU = 0.236
Option: C ✓
BBox: [0.482, 0.535, 0.576, 0.651] ✕ IoU = 0.000
Option: D ✕
BBox: [0.380, 0.480, 0.480, 0.580] ✕ IoU = 0.003
Option: B ✕
BBox: [0.460, 0.420, 0.580, 0.540] ✕ IoU = 0.413
Option: A ✕
BBox: [0.460, 0.390, 0.560, 0.500] ✕ IoU = 0.438
Prediction Summary. For the dense green deciduous tree cluster with irregular shape located in urban gap in this Landmark-relative Position Reasoning question, the target is GT option C and BBox [0.478, 0.427, 0.549, 0.511]. Across 10 available outputs, option prediction succeeds for 3/10 and landmark localization at IoU >= 0.5 for 0/10; both succeed for 0/10 (none). Semantic-only cases are Gemini 3.8 Flash, GPT-5.6 Sol, and Kimi K2.6, evidence-only cases are none, and 7/10 fail both. The split outcomes show that reference-frame reasoning and localization of the same reconstructed landmark do not yet reliably coincide.
Single-Task Evaluation
Four-task results for the reported model set
The reported evaluation covers 16 models and four tasks with 100 8-way questions each. Missing or invalid responses count as failures. BBox Acc uses IoU >= 0.5; mIoU averages all samples. Superscript T denotes low-effort reasoning, with Muse Spark using its minimum available setting.
#1 within group#2 within group#3 within group
| Model | Landmark-relative Position | Future Observation | Self-relative Position | Action Decision | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | BBox@50 | mIoU | Acc | BBox@50 | mIoU | Acc | BBox@50 | mIoU | Acc | BBox@50 | mIoU | |
| Proprietary Models | ||||||||||||
| Gemini 3.8 FlashT | 43.0% | 76.0% | 68.7% | 49.0% | 79.0% | 70.5% | 32.0% | 81.0% | 70.3% | 34.0% | 87.0% | 72.5% |
| Gemini 3 Flash | 33.0% | 61.0% | 57.0% | 32.0% | 55.0% | 49.5% | 50.0% | 68.0% | 58.9% | 28.0% | 76.0% | 61.9% |
| GPT-6 AstraT | 28.0% | 20.0% | 20.7% | 39.0% | 11.0% | 16.8% | 44.0% | 28.0% | 29.9% | 34.0% | 27.0% | 26.5% |
| GPT-5.6 SolT | 32.0% | 19.0% | 19.8% | 33.0% | 8.0% | 13.9% | 46.0% | 14.0% | 23.8% | 31.0% | 25.0% | 29.6% |
| GPT-5.6 TerraT | 32.0% | 19.0% | 19.0% | 31.0% | 9.0% | 16.3% | 41.0% | 23.0% | 26.5% | 35.0% | 25.0% | 27.1% |
| GPT-5.5 | 29.0% | 19.0% | 20.5% | 37.0% | 10.0% | 15.3% | 37.0% | 21.0% | 25.6% | 36.0% | 16.0% | 21.5% |
| Muse Spark 1.3 ContributorT | 31.0% | 21.0% | 23.5% | 37.0% | 17.0% | 18.8% | 51.0% | 23.0% | 27.6% | 49.0% | 22.0% | 26.3% |
| Qwen3.7-Plus | 23.0% | 26.0% | 27.7% | 25.0% | 15.0% | 19.7% | 45.0% | 28.0% | 29.9% | 37.0% | 35.0% | 30.4% |
| Qwen3.6-Plus | 23.0% | 27.0% | 26.7% | 27.0% | 10.0% | 17.9% | 46.0% | 30.0% | 28.3% | 48.0% | 28.0% | 28.9% |
| Open-weights Models | ||||||||||||
| GLM-5.3-FlashT | 28.0% | 33.0% | 33.4% | 36.0% | 31.0% | 32.2% | 57.0% | 42.0% | 38.2% | 48.0% | 47.0% | 39.4% |
| Kimi K2.6 | 24.0% | 29.0% | 27.3% | 31.0% | 21.0% | 22.2% | 39.0% | 43.0% | 38.8% | 34.0% | 37.0% | 35.3% |
| Kimi K2.5 | 28.0% | 25.0% | 24.1% | 27.0% | 12.0% | 15.9% | 38.0% | 39.0% | 33.3% | 23.0% | 39.0% | 33.5% |
| MiMo-V2.5 | 27.0% | 35.0% | 34.0% | 33.0% | 29.0% | 33.6% | 46.0% | 46.0% | 38.2% | 40.0% | 45.0% | 39.7% |
| Qwen3.8-27B | 28.0% | 36.0% | 34.8% | 30.0% | 22.0% | 23.5% | 44.0% | 51.0% | 42.4% | 41.0% | 50.0% | 40.2% |
| Qwen3.6-27B | 26.0% | 36.0% | 34.5% | 31.0% | 11.0% | 15.2% | 53.0% | 46.0% | 38.4% | 50.0% | 44.0% | 37.6% |
| Qwen3.5-27B | 19.0% | 13.0% | 24.0% | 27.0% | 4.0% | 13.3% | 45.0% | 24.0% | 27.3% | 40.0% | 28.0% | 27.8% |
Single-task performance on real-scene reconstructions. Scores aggregate all 400 questions across the three scenes. Colors identify the three highest distinct values within each model group; the result snapshot records run provenance and sample coverage.
Paired Dual Cognition
Position and action-observation coordination
Position pairs align Landmark-relative and Self-relative Position; action-observation pairs align Future Observation and Action Decision. Each relation contains 66 deterministic pairs matched within the same scene, landmark, and difficulty stratum. Overall is their harmonic mean.
#1 within group#2 within group#3 within group
| Model | Relative Position | Action-Observation | Overall | |||
|---|---|---|---|---|---|---|
| Dual Sem. | Dual Evi. | Dual Sem. | Dual Evi. | Dual Sem. | Dual Evi. | |
| Proprietary Models | ||||||
| Gemini 3.8 FlashT | 15.15% | 66.67% | 13.64% | 72.73% | 14.35% | 69.57% |
| Gemini 3 Flash | 13.64% | 50.00% | 10.61% | 53.03% | 11.93% | 51.47% |
| GPT-6 AstraT | 12.12% | 7.58% | 7.58% | 9.09% | 9.32% | 8.26% |
| GPT-5.6 SolT | 15.15% | 6.06% | 9.09% | 6.06% | 11.36% | 6.06% |
| GPT-5.6 TerraT | 12.12% | 6.06% | 4.55% | 7.58% | 6.61% | 6.73% |
| GPT-5.5 | 13.64% | 4.55% | 7.58% | 1.52% | 9.74% | 2.27% |
| Muse Spark 1.3 ContributorT | 13.64% | 10.61% | 10.61% | 6.06% | 11.93% | 7.71% |
| Qwen3.7-Plus | 12.12% | 16.67% | 7.58% | 9.09% | 9.32% | 11.76% |
| Qwen3.6-Plus | 10.61% | 15.15% | 7.58% | 6.06% | 8.84% | 8.66% |
| Open-weights Models | ||||||
| GLM-5.3-FlashT | 13.64% | 28.79% | 13.64% | 24.24% | 13.64% | 26.32% |
| Kimi K2.6 | 10.61% | 21.21% | 13.64% | 3.03% | 11.93% | 5.30% |
| Kimi K2.5 | 15.15% | 21.21% | 9.09% | 4.55% | 11.36% | 7.49% |
| MiMo-V2.5 | 16.67% | 30.30% | 10.61% | 21.21% | 12.96% | 24.96% |
| Qwen3.8-27B | 15.15% | 27.27% | 7.58% | 22.73% | 10.10% | 24.79% |
| Qwen3.6-27B | 15.15% | 28.79% | 10.61% | 7.58% | 12.48% | 11.99% |
| Qwen3.5-27B | 6.06% | 6.06% | 13.64% | 0.00% | 8.39% | 0.00% |
Paired coordination on real-scene reconstructions. Dual Semantic and Dual Evidence independently require both paired semantic decisions or both paired boxes to be correct. Each relation uses 66 matched-context pairs; Overall harmonically combines the two relation scores.
Analysis
Real-scene reconstruction preserves the benchmark's main capability separation
Task accuracy and paired coordination
GLM-5.3-Flash leads four-task macro semantic accuracy at 42.25% and reaches 13.64% Overall Dual Semantic Acc; Gemini 3.8 Flash reaches 14.35% paired semantics. The paired evaluation complements individual-task accuracy by directly measuring success on both Agent-State and World-State questions in matched contexts.
Semantics and localization remain distinct
Gemini 3.8 Flash leads Overall Dual Evidence Acc (69.57%) but reaches only 14.35% Overall Dual Semantic Acc. The gap echoes the AirSim benchmark: recognizing the relation and localizing the shared landmark are complementary capabilities.
A common protocol across scene representations
The pilot extends the SUSP and DCCS construction workflow to 3DGS reconstructions while preserving task and evaluation contracts. Reproducing the broad semantic-grounding separation across these rendered real-scene observations motivates further evaluation on physical UAV platforms in diverse environments.