NativeMEM

Long-horizon visual memory for VLA robot policies

Native Memory Compression for Long-Horizon Robotic Manipulation

NativeMEM gives pretrained Vision-Language-Action policies long-term, real-time visual memory by compressing each historical frame-view observation into a single native memory token.

Demos arXiv Code
Ziye Wang1 Modi Shi2 Chaojun Ni3 Jiazhi Yang4 Mengdi Li5 Zhizhong Su5 Tianwei Lin5 Hongyang Li1†
1 The University of Hong Kong 2 Beihang University 3 Peking University 4 The Chinese University of Hong Kong 5 Horizon Robotics
† Corresponding author

At a glance

Native memory without an external memory module

NativeMEM turns long-horizon robot history into native visual memory: no external planner, no separate memory module, and no language-only notes. By compressing each historical frame-view into one memory token, the policy can retain minute-level visual context while staying compatible with the pretrained VLA token sequence.

1 token per historical frame
160+ frames history memory capacity
100 demos for finetuning
98.7% real-world average success
NativeMEM keeps full visual history coverage with one-token memory compression while preserving low latency.
01

Native memory tokens

Each historical frame-view observation is compressed into one token in the VLA's native visual token space.

02

Action-supervised compression

The tokenizer is trained through the frozen VLA's original action objective, aligning memory with control rather than reconstruction.

03

Real-time long horizon

The policy can attend to minute-level histories while preserving low latency and modest GPU memory usage.

Method

Compress history into the same token space as the VLA

Stage 1

Learn the memory tokenizer

A memory encoder initialized from the VLA vision encoder uses a learnable query to aggregate each frame-view into one summary token. The VLA stays frozen, so gradients force the memory branch to encode action-relevant evidence.

Stage 2

Finetune with cached memory

After tokenizer training, memory tokens are cached for task demonstrations and appended to the standard VLA input sequence during task-specific finetuning.

Deployment

Update memory in real time

The tokenizer runs independently from action prediction, maintaining a compact queue that the policy consumes whenever an action is requested.

Robot demonstrations

Memory-dependent manipulation

Memory-dependent Task

Similar current observations can require different actions depending on prior events, so the policy must remember interaction history.

Click Buttons
Follow a specified sequence without repeating previous clicks.
Put Back Block
Recall where the block started after moving it to the center.
Grocery Checkout Scanning
Track scanned items and avoid duplicates or misses.

Click Buttons

Put Back Block

Grocery Checkout Scanning (Unseen)

*Unseen: not included during first-stage memory-tokenizer training.

NativeMEM

π0.5

Mem-0

MEM-short

NativeMEM

π0.5

Mem-0

MEM-short

Results

Large gains on simulation and real-robot memory tasks

Simulation 32.4% to 84.0%

Average success exceeds the strongest listed memory baseline.

Real world 34.7% to 98.7%

Average final-stage success exceeds the strongest listed baseline.

Data efficiency 20% data

10 demonstrations reach or exceed other models trained with 50.

Simulation success rates (%)

Baselines NativeMEM

Click Buttons

0π0.5 7X-VLA 12MemER 0Mem-0 4HAMLET* 0MEM-short* 94NativeMEM

Click Buttons (Hard)

0π0.5 12X-VLA 8MemER 1Mem-0 17HAMLET* 39MEM-short* 88NativeMEM

Swap Blocks

24π0.5 16X-VLA 18MemER 67Mem-0 11HAMLET* 4MEM-short* 94NativeMEM

Put Back Block

11π0.5 18X-VLA 12MemER 90Mem-0 3HAMLET* 15MEM-short* 100NativeMEM

Observe and Pickup

9π0.5 9X-VLA 2MemER 4Mem-0 10HAMLET* 6MEM-short* 44NativeMEM

Average

8.8π0.5 12.4X-VLA 10.4MemER 32.4Mem-0 9.0HAMLET* 12.8MEM-short* 84.0NativeMEM

Real-world success rates (%)

Baselines NativeMEM

Click Buttons

2π0.5 16π0.5 + RTC 0Mem-0 40MEM-short* 96NativeMEM

Put Back Block

14π0.5 24π0.5 + RTC 0Mem-0 0MEM-short* 100NativeMEM

Grocery Checkout Scanning

28π0.5 64π0.5 + RTC 0Mem-0 52MEM-short* 100NativeMEM

Average

14.7π0.5 34.7π0.5 + RTC 0.0Mem-0 30.7MEM-short* 98.7NativeMEM

Runtime and memory scaling

Measured on RTX 5090 32GB across increasing visual history frame budgets.

MEM NativeMEM
Latency (ms)
100 1000 5 20 100 500 2000 5000 Frame budget Latency (ms)
Peak VRAM (GB)
16 24 32 32 GB cap 5 20 100 500 2000 5000 Frame budget Peak VRAM (GB) MEM OOM

Qualitative analysis

Interpretable memory activates native VLA priors

Spatial attention of the NativeMEM memory tokenizer over historical observations

Spatial attention

NativeMEM concentrates memory tokens on task-relevant objects, robot endpoints, and workspace landmarks. Even on tasks unseen during memory-tokenizer pretraining, it preserves task-relevant information in a general way without being distracted by background pixels.

Temporal attention map with memory snapshot time on the x-axis and action prediction time on the y-axis

Temporal attention

Each heatmap shows action-to-memory lookup: the x-axis is the timestamp of each historical memory, the y-axis is the action prediction timestep, and brighter cells mark stronger attention. Reading across a row shows which memories support one action prediction; reading down a column shows how long a particular memory remains useful.

NativeMEM attends to task-relevant moments rather than simply the latest frame. In Click Buttons, high-attention memories follow the three button-click stages. In Put Back Block, attention concentrates on the pickup moment, which tells the policy the original pad where the block should be returned.

Case study

Recovery from failure

At 5s, the third item is not grasped, yet the policy actively retries, a behavior absent from demonstrations. At 16s, it has executed four scan motions but only three succeeded. Instead of overfitting to the action count, NativeMEM understands the real history and performs a fifth scan for the missed item, relying on pretrained VLA priors beyond the demonstrated trajectories.

Citation

BibTeX

@article{wang2026nativemem,
  title={NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation},
  author={Wang, Ziye and Shi, Modi and Ni, Chaojun and Yang, Jiazhi and Li, Mengdi and Su, Zhizhong and Lin, Tianwei and Li, Hongyang},
  journal={arXiv preprint arXiv:2607.06678},
  year={2026}
}