ICML 2024 · PMLR 235 · 2024

Langevin Policy

Langevin Policy for Safe Reinforcement Learning

Fenghao Lei · Long Yang · Shiting Wen · Zhixiong Huang · Zhiwang Zhang · Chaoyi Pang

Safety is not a single optimum

Safe reinforcement learning must seek reward without crossing a defined cost limit. Langevin Policy frames that balance as a distribution to sample from, rather than a point to optimize once.

Reward attracts the policy toward useful actions, cost suppresses unsafe regions, and a controlled policy term prevents each update from moving arbitrarily far.

The constrained MuJoCo and Safety Gym evaluation tasks

Sample carefully, execute directly

Short Langevin chains refine actions with reward and cost critics. Their outputs become targets for an actor that executes in one forward pass.

Sampling serves as the policy-improvement teacher; the actor distills that improvement into a practical deployable policy.

The sampler improves the policy during training; the actor preserves one-pass execution at deployment.

Reward under constraint

LAC is evaluated on four constrained MuJoCo tasks and four randomized Safety Gym tasks. It remains within the reported cost limit across all eight while retaining strong constrained return.

8 / 8

tasks within the reported cost limit

2.5M

environment steps per evaluation

benchmark families

Langevin Policy Figure 2: reward and cost curves on MuJoCo tasks

Langevin Policy Figure 3: reward and cost curves on Safety Gym tasks

The deployed actor approximates the sampled policy, and the Gaussian generator limits expressiveness; both define clear directions for stronger generative policies.

A safe policy should not merely avoid failure. It should preserve the freedom to act well within the boundary.

Read the paper

Abstract and citation

Existing safe reinforcement learning methods are predominantly optimization-based, leaving the potential of sampling-based safe policies underexplored. This work formulates a Langevin Policy that directly infers actions through a numerical solver for continuous-time Langevin dynamics. It further proposes Langevin Actor-Critic (LAC), which accumulates transitions induced by the Langevin Policy and reproduces them with a generator for efficient inference. Experiments on MuJoCo and Safety Gym demonstrate an effective balance between reward and safety constraints.

@inproceedings{lei2024langevin,
  title={Langevin Policy for Safe Reinforcement Learning},
  author={Lei, Fenghao and Yang, Long and Wen, Shiting and Huang, Zhixiong and Zhang, Zhiwang and Pang, Chaoyi},
  booktitle={Proceedings of the 41st International Conference on Machine Learning},
  pages={27174--27190},
  year={2024},
  volume={235},
  series={Proceedings of Machine Learning Research}
}