Langevin Policy for Safe Reinforcement Learning

Safe reinforcement learning must maximize reward while keeping cumulative cost below a defined limit. Langevin Policy approaches this as a sampling problem: desirable actions should become more likely, unsafe actions less likely, without changing the policy uncontrollably.

Sampling for safe improvement

The energy function combines reward advantage, cost advantage and the current policy probability. Reward favors useful actions, the cost term suppresses unsafe ones, and the policy term keeps each update controlled.

Langevin Policy Figure 1: constrained MuJoCo and Safety Gym evaluation tasks

Langevin Policy energy function, Equation 8

Making sampling practical

LAC starts each short Langevin chain from an action proposed by the current actor, rather than unrelated noise. This makes refinement efficient enough to serve as a training signal.

Actor proposal

The current policy supplies an informed initial action, avoiding a long sampling chain from unrelated Gaussian noise.

Energy-guided refinement

Reward and cost critics reshape the sample together: useful regions attract it, while unsafe regions are suppressed without removing exploration.

Policy distillation

The refined action becomes a learning target for an actor that executes in one pass; sampling improves the policy during training without slowing deployment.

LAC uses the Langevin sampler as a policy-improvement teacher and the actor as its efficient deployable approximation.

The refined action becomes a target for the actor. The sampler acts as a policy-improvement teacher, while the actor distills that result into a policy that executes in one forward pass.

Reward and safety

The evaluation covers four constrained MuJoCo tasks and four randomized Safety Gym tasks. Every method trains for 2.5 million environment steps.

4 / 4

MuJoCo tasks within the reported cost limit

4 / 4

Safety Gym tasks within the reported cost limit

2.5M

environment steps in each evaluation

Langevin Policy Figure 2: reward and cost learning curves on MuJoCo tasks

Langevin Policy Figure 3: reward and cost learning curves on Safety Gym tasks

Scope. The deployed actor approximates the sampled policy, and the Gaussian generator limits expressiveness; the reported evaluation is simulation-based.

Langevin Policy reframes safe policy improvement as sampling from a reward- and constraint-aware energy landscape. LAC then turns that theoretically motivated but expensive sampler into a practical actor-critic algorithm by using short chains, conservative critics and policy distillation.