Understanding Minimal-Time Attacks on Reinforcement Learning Agents

Reinforcement learning (RL) agents are known to be vulnerable to small perturbations to observations that can disproportionately degrade performance. An active area of research seeks to understand the extent to which these agents are vulnerable to stealthy attacks by targeting a limited number of time steps associated with “critical moments” of the agent. However, these attacks suffer from two major shortcomings: (i) they assume an often unrealistic threat model as they target disparate time steps across the agent’s trajectory, requiring the attacker to manipulate the agent at arbitrary times, and (ii) these attacks often require complex and prohibitively time-consuming methods for crafting the adversarial attack in order to degrade the agent’s performance within short bursts. In this work, we propose a more restrictive threat model for these agents where the attacker can only attack for a small number of consecutive steps. Based on this threat model, we propose SequΛ¹, an attack that targets the agent sequentially within a short window. In developing SequΛ, we leverage key ideas behind critical-moment approaches in our design, and we discover situations in which assumptions behind attacking at sporadic, key moments fail to adequately degrade the test-time performance of the agent. We evaluate SequΛ on Mujoco environments of robotics simulations with continuous state- and action-spaces and show that agent performance can be drastically degraded with attacks during brief consecutive windows. Our results show that short-duration critical window attacks created by SequΛ are as effective as their non-consecutive counterparts and can be deployed in more realistic settings.