Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?

Zihao Zhang; Haochen Tian; Tianyu Li; Changhui Jing; Jingliang He; Naisheng Ye; Ziyuan Pu; Zhenjie Yang

Visual Representation Alignment / October 2026

Do Better Visual Representations
Always Lead to Better
End-to-End Autonomous Driving?

Visual representation alignment improves driving.
Its benefits depend on the target—and the supervision.

Zihao Zhang^1,2,3 Haochen Tian² Tianyu Li² Changhui Jing² Jingliang He³ Naisheng Ye² Ziyuan Pu^1,‡ Zhenjie Yang^2,‡

¹ Southeast University

² OpenDriveLab at The University of Hong Kong

³ Shenzhen Loop Area Institute

^‡ Corresponding authors.Primary contact: [email protected]

Read the paper arXiv GitHub

ViRA-Diffusion92.3 EPDMSNAVSIM v2 · navtest

Challenging scenarios+16.4 ptsRap* with ViRA · navhard

Training-time alignment0 added costAt inference · original planner retained

01The study

From visual representations
to driving performance.

A controlled look at what foundation models bring to end-to-end planners—and when their differences matter.

ViRA overview: training-time representation alignment, perception improvements, and findings across planners, VFM targets and auxiliary supervision — FIG. 01One interface, three empirical findings. ViRA aligns planner features with a frozen visual foundation model during training. The deployed planner architecture and inference cost stay unchanged.

Abstract

Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance. To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged.

Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation. (2) The choice of VFM target matters for planning performance, and alignment to a different VFM can further benefit planners with pre-trained VFM encoders. (3) Auxiliary perception supervision reduces sensitivity to VFM target selection, narrowing the EPDMS spread across five targets from 2.7 to 0.5 points and potentially compensating for less effective VFM targets.

Guided by these findings, we develop ViRA-Diffusion, a diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points. The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving.

02Driving gains

Different planners.
Consistent overall gains.

Regression, diffusion and scoring planners all benefit from VFM-guided alignment in our evaluations. The gains extend to zero-shot closed-loop simulation.

NAVSIM v2 · navtest

Camera-only evaluation · standard driving scenarios

Non-reactive simulation

NAVSIM v2 navtest performance
Method	NC ↑	DAC ↑	DDC ↑	TLC ↑	EP ↑	TTC ↑	LK ↑	HC ↑	EC ↑	EPDMS ↑
Human Agent	100	100	99.8	100	97.8	100	100	98.1	90.1	97.9
Perception-based
TransFuser	97.1	89.6	99.0	99.8	97.9	95.9	95.9	98.3	87.3	83.6
TransFuser-ViRA	97.7	94.1	99.5	99.8	98.2	96.7	96.8	98.3	87.9	89.0+5.4
DiffusionDrive	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	84.5
DiffusionDrive-ViRA	98.2	96.5	99.6	99.8	98.2	97.4	97.4	98.3	88.0	91.9+7.4
Perception-free
Rap^*	91.6	88.8	93.5	99.8	87.2	92.5	90.9	97.2	56.7	72.8
Rap^*-ViRA	97.1	96.9	99.2	99.9	87.5	96.9	96.2	97.7	48.1	83.7+10.9

All methods use camera inputs without LiDAR. Rap* is our reimplementation with a different backbone and without the original data augmentation. DiffusionDrive-ViRA (91.9) and ViRA-Diffusion (92.3) are distinct configurations.

NAVSIM v2 · navhard

Challenging scenarios · full S1 and S2 sub-metrics

Non-reactive simulation

NAVSIM v2 navhard performance, combined EPDMS across stages
Method	Stage	NC ↑	DAC ↑	DDC ↑	TLC ↑	EP ↑	TTC ↑	LK ↑	HC ↑	EC ↑	EPDMS ↑
PDM-Closed	S1	94.4	98.8	100	99.5	100	93.5	99.3	87.7	36.0	56.6
PDM-Closed	S2	90.5	90.6	95.4	98.4	100	98.4	74.2	91.9	29.7	56.6
Perception-based
TransFuser	S1	96.4	73.8	95.6	99.3	94.8	95.6	88.0	97.6	82.2	27.6
TransFuser	S2	82.6	62.8	77.9	98.2	91.0	79.9	40.9	97.5	80.8	27.6
TransFuser-ViRA	S1	97.4	80.0	98.7	99.3	98.0	95.6	95.8	97.8	81.3	31.7+4.1
TransFuser-ViRA	S2	81.2	73.9	86.0	98.0	97.0	78.3	47.3	96.1	71.1	31.7+4.1
DiffusionDrive	S1	96.8	86.0	98.8	99.3	84.0	95.8	96.7	97.6	79.6	30.5
DiffusionDrive	S2	80.1	72.8	84.4	98.4	85.9	76.6	46.4	96.3	72.8	30.5
DiffusionDrive-ViRA	S1	96.4	87.8	99.2	99.3	98.5	95.3	96.4	97.6	78.7	35.8+5.3
DiffusionDrive-ViRA	S2	80.5	74.1	86.4	98.4	97.5	77.2	47.6	95.8	71.3	35.8+5.3
Perception-free
Rap^*	S1	89.4	71.8	82.3	99.6	86.6	89.6	85.8	95.8	47.1	32.8
Rap^*	S2	90.6	77.6	85.6	98.8	71.4	88.9	50.8	98.4	68.9	32.8
Rap^*-ViRA	S1	96.7	90.7	97.2	100.0	85.2	96.2	94.2	96.4	42.2	49.2+16.4
Rap^*-ViRA	S2	95.5	87.6	94.9	99.6	69.3	94.5	55.3	98.7	68.3	49.2+16.4

EPDMS is a combined score across stages, shown once per method. PDM-Closed uses ground-truth symbolic inputs; the other methods use camera inputs only. Rap* uses our modified backbone and omits the original data augmentation.

HUGSIM · zero-shot

Trained on NAVSIM · evaluated without further training

Closed-loop simulation

Zero-shot HUGSIM performance across four difficulties
Method	Easy ↑		Medium ↑		Hard ↑		Extreme ↑		Overall ↑
Method	RC	HD-Score	RC	HD-Score	RC	HD-Score	RC	HD-Score	RC	HD-Score
Perception-based
TransFuser	66.8	56.7	28.1	11.6	24.5	9.2	26.2	10.9	36.4	22.1
TransFuser-ViRA	72.7	62.6	31.9	15.3	27.8	11.4	25.5	9.9	39.5	24.8+2.7
DiffusionDrive	68.9	56.8	29.8	12.4	25.9	9.7	25.7	9.8	37.6	22.2
DiffusionDrive-ViRA	77.5	69.3	37.7	20.7	30.0	12.0	27.9	12.3	43.3	28.6+6.4
Perception-free
Rap^*	11.8	10.6	11.1	9.9	7.8	6.3	4.1	2.3	8.7	7.3
Rap^*-ViRA	18.1	15.9	16.7	14.6	10.5	8.2	4.3	2.3	12.4	10.3+3.0

RC = route completion. HD-Score weights route completion by per-step safety and comfort scores. Overall HD-Score improves for all three planners; gains are not uniform at every difficulty. Rap* uses our modified backbone and omits the original data augmentation.

Scroll the table horizontally to explore all metrics ↔

t-SNE comparison of TransFuser and Rap* representations before and after ViRA alongside the reference VFM — FIG. 02The t-SNE visualization suggests a shift of planner representations toward the reference VFM.

Planning performance compared with inference latency and parameter counts; alignment preserves deployed planner cost — FIG. 03Alignment improves planning while preserving each planner’s inference cost. Bubble area represents model parameters.

03What matters

The target matters.
So does the supervision.

The benefits of visual representations depend on how the planner is trained. Two experiments reveal why target selection deserves a closer look.

Finding 02 / Target selection

Different foundations,
different driving gains.

VFM targets yield different planning gains for the same planner. Even a planner with a pre-trained VFM encoder can benefit from alignment to another target.

83.8→86.3

Rap* with a DINOv3 encoder
+ VGGT alignment · NAVSIM v2 navtest EPDMS

Different visual foundation model targets produce different planning performance across the evaluated planners — FIG. 04Comparing VFM targets across planner architectures and supervision settings.

Finding 03 / Auxiliary supervision

Supervision changes
target sensitivity.

For TransFuser, auxiliary perception supervision narrows the EPDMS spread across the same five VFM targets. Without it, the choice of target has a larger effect on planning.

2.7Without auxiliary supervision

→

0.5With auxiliary supervision

EPDMS spread across five targets

TransFuser ablations compare perception metrics and planning gains with and without auxiliary perception supervision across five VFM targets — FIG. 05Matched TransFuser experiments show reduced sensitivity to VFM target selection with auxiliary perception supervision.

Putting the findings into practice

ViRA-Diffusion

A diffusion planner aligned to DINOv3, trained without auxiliary perception supervision. It exceeds the recent methods in our paper’s comparison by at least 1.9 EPDMS points on NAVSIM v2 navtest.

Explore the configuration

92.3EPDMSNAVSIM v2 · navtest

04See the difference

Better perception.
Better planned trajectories.

Qualitative comparisons on NAVSIM and zero-shot driving rollouts in HUGSIM complement the quantitative results.

Qualitative NAVSIM comparisons showing perception predictions, visual attention and planned trajectories before and after ViRA — FIG. 06NAVSIM. Qualitative examples of perception and planning with ViRA.

HUGSIM in motion

Zero-shot · closed-loop simulation

HUGSIM closed-loop driving comparison, rollout 1 — Planner comparisons in a photorealistic closed-loop environment.

HUGSIM closed-loop driving comparison, rollout 2 — NAVSIM-trained planners evaluated without HUGSIM fine-tuning.

05Build on this work

Citation

If you find this work useful for your research, please consider citing our paper.

BibTeX / arXiv:2610.09695

@article{zhang2026vira,
  title   = {Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?},
  author  = {Zhang, Zihao and Tian, Haochen and Li, Tianyu and Jing, Changhui and He, Jingliang and Ye, Naisheng and Pu, Ziyuan and Yang, Zhenjie},
  journal = {arXiv preprint arXiv:2610.09695},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.09695}
}