Evaluation Protocol

Protocol

Semantic correctness and localization quality are evaluated separately

Image tasks combine a semantic decision with target-landmark localization; video tasks combine behavior or visibility predictions with temporal intervals. Scoring these outputs independently reveals how answer correctness relates to spatial and temporal grounding.

Compatibility Audit

Deterministic compatibility rules are applied uniformly by model run

The canonical protocol remains strict. The reported tables additionally use one shared, deterministic compatibility and rescoring pipeline. This pipeline applies to Deterministic compatibility parsing for image outputs, selected-candidate BBox scoring for Future Observation, and class-aware behavior interval matching for video outputs. Coordinate scale is resolved at the model-run level rather than guessed independently for each prediction. Both the original and compatible results are retained for traceability.

The audit averages the four image-task BBox Acc@0.5 values for each model with complete stored outputs. Only models exhibiting positive gains (+pp) under the deterministic compatibility parser are shown above. Image semantic scores are unchanged. Video results are separately rescored with class-aware, one-to-one temporal matching as specified below.

Both original and compatible parsing results are retained; no individual sample is repaired selectively.

BBox Reference Convention

Every score is evaluated in a defined image coordinate domain

Existing multi-view landmark assets use reviewed boxes enclosing projected visible landmark points. Newly captured environment views use the axis-aligned envelope of in-frustum 3D oriented-box corners; these are geometric projection boxes and need not equal visible-mask tight boxes. Fully in-frame views are preferred, accepted partial projections are clipped to the image boundary, and degenerate or fully out-of-frame boxes are rejected before export.

For Future Observation, the predicted BBox is compared with the landmark box in the model-selected candidate image. Invalid candidate identifiers or invalid boxes receive zero localization credit. Option accuracy is still scored against the correct future view, so the two reported metrics remain independent.

Class-aware Temporal Matching

Behavior identity remains bound to its predicted interval

Composite semantics use exact set accuracy, whereas atomic semantics use label-set F1. Temporal predictions are greedily matched one-to-one only within the same behavior class at tIoU ≥ 0.5. Unmatched and extra predicted classes count as false positives. Visibility intervals use the same one-to-one threshold matching, while visible-count accuracy is scored independently. For video Dual Evidence, visibility and every applicable behavior level must reach sample-level F1@0.5 of at least 0.5.

Matched-context Dual Metrics

Coordination is measured as paired Agent-State and World-State co-success

Image questions are paired within the same scene, Agent–Landmark identity, and difficulty stratum for the position and action–observation task families; paired images can use different UAV poses. Video questions are paired by source trajectory. Dual Semantic Accuracy requires both semantic outputs to succeed, while Dual Evidence Accuracy independently requires both localization outputs to succeed. This matched-context design tests coordinated capability over coordinate-frame reasoning, action–visual association, and behavior–visibility localization through observable task outcomes.

Suffix Handling

Instant/Thinking suffixes are interpreted per model family

The evaluation runtime strips -Instant, -Thinking, or -Reasoning to resolve the base route in common_api_runtime.yaml. It then applies family-compatible request controls (for example enable_thinking, reasoning, or chat_template_kwargs) according to provider capabilities. This keeps the routing contract stable across local deployment and API invocation while preserving a unified experiment naming convention.