Long horizon, contact rich manipulation is inherently partially observable. This is as a
single visual observation rarely captures a robot's full action context, including prior
attempts, interactions, or progress. Consequently, standard visuomotor policies or
vision-language-action models are prone to struggle in such tasks due to a lack of memory.
To address this, we introduce Compressed Action Memory
Policy (CAMP) based on the insight that a robot's own action history serves as a
highly informative, self-supervised signal, enabling the policy to learn a robust, compact
history representation. In our approach, we train a memory module to maintain a compressed
representation of past actions, forcing it to encode a latent behavioral memory of all the
robot's past interactions that can then be used to better contextualize future actions.
This allows our approach to implicitly track generalized task progress and learn from
failed attempts without any additional supervision, or external oversight. We evaluate CAMP
across four real-robot setups and two novel simulation benchmarks: Memory-T-Bench and
Memory-Manip-Bench. By demonstrating substantial gains over state-of-the-art baselines,
CAMP is, to our knowledge, the first policy to demonstrate substantial success on
contact-rich partially observable manipulation tasks purely through learned memory.
