**Keywords:** reinforcement learning, nonnormality, transient
amplification, action smoothing, Koopman surrogate, Lyapunov
certificate, safe control

# Introduction

High-return continuous-control policies can still produce jittery
commands, dense constraint-margin losses, and frequent safety-layer
corrections. The usual response is to penalize action variance, action
differences, or actuator roughness. These penalties are useful, but they
do not explain why the same magnitude of residual action noise can be
benign in one closed loop and dangerous in another.

This paper argues that the central issue is not whether actions are
smooth in Euclidean norm. The central issue is whether residual action
covariance is injected into directions that a nonnormal closed-loop
surrogate strongly amplifies. The action residual is a source; the
closed-loop dynamics form an amplifier; state covariance, safety-margin
loss, posterior Bellman-Lyapunov residuals, and projection burden are
downstream consequences.

The goal is therefore a problem-first diagnostic/control framework
rather than a broad safe-RL benchmark or a global nonlinear safety
theorem. The concrete response framework, certificate-guided residual
variance control (CG-RVC), is used only as one implementation of this
source-amplifier view. It combines local surrogate fitting, Lyapunov
weighting, residual covariance control, smoothing, rate limiting, and
safety projection.

# Nonnormal Closed-Loop Amplifier

Consider a local lifted surrogate
``` math
z_{t+1}=Kz_t+\Phi_u r_t+w_t ,
\label{eq:local}
```
where $`z_t=\psi(x_t)`$ is a lifted state, $`r_t`$ is a residual action
or command fluctuation, $`\Phi_u`$ is the input channel, and $`w_t`$
captures disturbances and model error. For white source covariance
$`\Sigma_r`$ and disturbance covariance $`W`$, the local stationary
covariance satisfies
``` math
\Sigma_z=\sum_{i=0}^{\infty}K^i(\Phi_u\Sigma_r\Phi_u^\top+W)(K^i)^\top .
\label{eq:covprop}
```
Equation <a href="#eq:covprop" data-reference-type="eqref"
data-reference="eq:covprop">[eq:covprop]</a> is the basic separation:
action variance is not state variance itself; it becomes state variance
only after propagation through the closed-loop amplifier.

Spectral-radius stability, $`\rho(K)<1`$, is not enough to rule out
short-horizon energy growth. If $`K=V\Lambda V^{-1}`$ is highly
nonnormal, eigenvectors can be ill-conditioned even when eigenvalues lie
inside the unit disk. Diagnostics such as $`\kappa(V)`$, peak transient
gain $`G_{\mathrm{peak}}=\max_{t\le H}\|K^t\|_2`$, resolvent norms, and
Kreiss-like proxies are therefore relevant to reinforcement-learning
evaluation.

<figure id="fig:source_amp">
<img src="source_amplifier_layers.png" />
<figcaption>Source-amplifier view. Residual actions, exploration noise,
execution noise, and disturbances enter the input channel <span
class="math inline"><em>Φ</em><sub><em>u</em></sub></span> and are
propagated by a nonnormal closed-loop amplifier <span
class="math inline"><em>K</em></span>. The downstream consequences are
state covariance, safety-margin loss, posterior Bellman-Lyapunov
residuals, and projection burden.</figcaption>
</figure>

# Why Ordinary Smoothing Is Not Enough

A scalar action-variance penalty,
``` math
R_{\mathrm{var}}=\mathrm{tr}(\Sigma_u),
```
measures total source energy but ignores input direction. A temporal
roughness penalty,
``` math
R_{\Delta u}=\mathbb{E}\|u_t-u_{t-1}\|^2,
```
suppresses fast changes but does not determine whether the remaining
residual covariance aligns with high-amplification directions.

Given a local Lyapunov metric $`P`$ with measured contraction factor
$`\gamma_P`$,
``` math
K^\top P K \preceq \gamma_P^2P,
```
the residual input directions can be weighted by
``` math
M=\Phi_u^\top P\Phi_u .
```
The corresponding P-weighted action Lyapunov energy (P-ALE) is
``` math
R_{\mathrm{P\text{-}ALE}}=
\frac{\mathrm{tr}(M\Sigma_r)}{1-\gamma_P^2}.
\label{eq:pale}
```
Smoothness is temporal regularity. P-ALE is amplifier-aware residual
covariance control. They are related, but not equivalent.

<div id="tab:diagnostics">

| Layer | Quantity | Interpretation |
|:---|:---|:---|
| Source | $`\mathrm{tr}(\Sigma_r)`$, applied variance, diff MSE | How much residual or executed action jitter is injected. |
| Amplifier | $`\rho(K)`$, $`\kappa(V)`$, $`G_{\mathrm{peak}}`$, resolvent norm | Whether stable-looking dynamics can transiently amplify sources. |
| Certificate | $`P`$, $`\gamma_P`$, $`M=\Phi_u^\top P\Phi_u`$ | Which action directions are high-energy under the local certificate. |
| Consequence | state covariance trace, dense violation, B-L residual | Whether source jitter becomes dispersion or safety-margin loss. |
| Execution | projection trigger rate, normalized correction | How much repair the safety layer must perform. |

Diagnostic quantities for a source-amplifier paper.

</div>

# Minimal Certificate-Guided Response

The response framework has five steps. First, fit a local surrogate
$`z_{t+1}=K_0z_t+\Phi_u u_t+w_t`$ using local linearization, EDMDc-style
regression, or a replay-buffer local model. Second, construct a nominal
or projected closed-loop matrix, for instance
$`K_{\mathrm{raw}}=K_0-\Phi_u L`$ followed by
$`K_{\mathrm{proj}}=\Pi_\beta(K_{\mathrm{raw}})`$. Third, solve a
Lyapunov equation or inequality to obtain $`P`$. Fourth, add the P-ALE
penalty in Eq. <a href="#eq:pale" data-reference-type="eqref"
data-reference="eq:pale">[eq:pale]</a> to the actor objective. Fifth,
apply execution-layer shaping through smoothing, rate limiting, and
safety projection.

The spectral target $`\beta`$ is not a P-metric contraction guarantee.
Bounds and diagnostics should use the measured $`\gamma_P`$. Moreover,
smoothing and projection make the execution layer nonlinear; the
certificate is therefore a local design and diagnostic object rather
than a global nonlinear plant certificate.

If the residual policy is state-dependent, the residual is not purely
exogenous. A local diagnostic should estimate
``` math
K_\theta=K+\Phi_u J_r ,
\label{eq:ktheta}
```
where $`J_r`$ is a finite-difference actor-Jacobian proxy. When $`J_r`$
is large, the feedback component must be folded into the amplifier.

# Evidence

The evidence should separate source, amplifier, and safety repair.
Figure <a href="#fig:ci" data-reference-type="ref"
data-reference="fig:ci">2</a> combines two controlled interventions.
CI-1 fixes eigenvalues, spectral radius, input channel, and source
covariance while changing eigenvector geometry. In the archived result,
$`\kappa(V)`$ grows from about $`1.0`$ to $`20.05`$, state covariance
trace grows from about $`0.37`$ to $`15.66`$, and peak transient gain
grows from about $`1.0`$ to $`5.55`$. This supports the statement that
nonnormal geometry can independently amplify a fixed source in a
controlled matrix family.

CI-2 fixes the strongly nonnormal amplifier and changes the source. In
the archived result, the filtered source lowers action variance from
$`0.03992`$ to $`0.00317`$ and state covariance trace from $`15.849`$ to
$`12.393`$, while structural peak gain stays fixed near $`5.55`$. This
is source-side evidence, but the filtered input is colored; it should be
interpreted through an augmented colored-input model.

<figure id="fig:ci">
<figure>
<img src="fig1_ci1_amplifier_isolation.png" />
<figcaption>CI-1</figcaption>
</figure>
<figure>
<img src="fig2_ci2_source_intervention.png" />
<figcaption>CI-2</figcaption>
</figure>
<figcaption>Controlled interventions. CI-1 manipulates nonnormal
amplifier geometry while fixing source statistics. CI-2 manipulates
source statistics while fixing the amplifier.</figcaption>
</figure>

<div id="tab:ablation">

| Method            |  RMSE |   Var. |  Viol. | B-L max |
|:------------------|------:|-------:|-------:|--------:|
| Direct TD3        | 0.537 | 0.1138 | 0.9389 |    92.4 |
| TD3 + Var$`(u)`$  | 0.612 | 0.1026 | 0.9278 |   161.9 |
| TD3 + action diff | 0.565 | 0.1118 | 0.9389 |   109.1 |
| Projection only   | 0.505 | 0.1033 | 0.1222 |    82.3 |
| Full response     | 0.284 | 0.0060 | 0.0000 |    5.24 |

Compressed high-noise 2D constrained-RL evidence. Values are mean
summaries from the archived matched ablation.

</div>

Table <a href="#tab:ablation" data-reference-type="ref"
data-reference="tab:ablation">2</a> shows why ordinary smoothing is
incomplete in the high-noise constrained-RL task. Scalar action variance
and action-difference penalties do not remove dense violations or
posterior residuals. Projection alone lowers violations but leaves high
applied variance and residuals. The full response combines source
shaping and execution repair.

Projection burden gives an operational downstream diagnostic. In the
archived 2D high-noise run, normalized correction is about $`0.09362`$
for projection-only, $`0.05426`$ for P-ALE plus projection, and
$`0.01093`$ for the full method. A residual-feedback diagnostic gives
another caveat: the finite-difference actor-Jacobian proxy is about
$`16.23`$ for Direct TD3 and $`0.274`$ for the full method. This
supports, but does not prove, the source-side interpretation.

# Limitations

The claim level is deliberately narrow. First, all certificate claims
are local-surrogate-level statements, not global nonlinear safety
guarantees. Second, P-ALE reduces certificate-weighted residual
covariance entering the amplifier; it does not directly reduce
nonnormality of $`K`$. Third, if residual feedback is strong, the
source-only interpretation becomes diagnostic unless the feedback is
folded into Eq. <a href="#eq:ktheta" data-reference-type="eqref"
data-reference="eq:ktheta">[eq:ktheta]</a>. Fourth, projection depends
on local feasibility and cannot create feasibility. Fifth, Full3D
quadrotor and ControlGym results are simulation-level relevance checks,
not hardware validation or broad safe-RL benchmark dominance.

# Conclusion

Low-variance safe reinforcement learning should report both the source
and the amplifier. The relevant object is not only $`\mathrm{Var}(u)`$
or $`\|u_t-u_{t-1}\|^2`$, but the full chain from $`\Sigma_r`$ through
$`\Phi_u`$, $`K`$, $`P`$, and $`M`$ into state covariance, safety-margin
loss, posterior Bellman-Lyapunov residuals, and projection burden. This
source-amplifier framing explains why ordinary smoothing can be
insufficient and why certificate-weighted residual covariance is a
natural diagnostic/control response for nonnormal RL closed loops.
