# Paper Plan

Working title:

**Nonnormality in Reinforcement Learning Closed Loops: From Action Smoothing to Source-Amplifier Diagnostics**

## One-Sentence Contribution

We show that residual action variance in continuous-control RL should be evaluated through a nonnormal closed-loop amplifier, and we provide certificate-weighted diagnostics and controls for this source-amplifier mechanism.

## Main Argument

Action jitter is a source, not the final risk. A local closed-loop surrogate can amplify residual actions, exploration noise, execution noise, and disturbances into downstream consequences. These consequences include state covariance, safety-margin loss, projection burden, and posterior Bellman-Lyapunov residuals.

## Six-Page Structure

| Page | Section | Function | Key Content |
|---:|---|---|---|
| 1 | Introduction / Problem Statement | Move from action smoothing to nonnormal amplification | High-return but jittery control, stable but unsafe rollouts, source-amplifier thesis, compact contributions. |
| 2 | Nonnormality as a Closed-Loop Amplifier | Give the minimal mathematical model | `z_{t+1}=Kz_t+Phi_u r_t+w_t`, covariance propagation, `rho(K)` vs transient gain. |
| 3 | Why Action Smoothing Is Not Enough | Critique scalar and temporal smoothing | Compare `tr(Sigma_u)`, `E||u_t-u_{t-1}||^2`, and `tr(Phi_u^T P Phi_u Sigma_r)`. |
| 4 | A Minimal Certificate-Guided Response | Present CG-RVC as a response framework | Surrogate fitting, stable projection, Lyapunov certificate, P-ALE, smoothing, rate limiting, safety projection. |
| 5 | Experiments: Separating Source, Amplifier, and Safety Repair | Show targeted evidence | CI-1 amplifier scan, CI-2 source intervention, 2D constrained RL ablation, projection burden, actor-Jacobian diagnostic. |
| 6 | Limitations and Failure Modes | Keep claims defensible | Locality, P-ALE does not reduce nonnormality, feedback changes amplifier, projection limits, simulation-only evidence. |

## Core Equations

1. Local source-amplifier surrogate:

```tex
z_{t+1}=Kz_t+\Phi_u r_t+w_t .
```

2. Covariance propagation:

```tex
\Sigma_z=\sum_{i=0}^{\infty}K^i(\Phi_u\Sigma_r\Phi_u^\top+W)(K^i)^\top .
```

3. Lyapunov certificate:

```tex
K^\top P K \preceq \gamma_P^2 P .
```

4. Amplifier-aware residual metric:

```tex
M=\Phi_u^\top P\Phi_u,\qquad
R_{\mathrm{P\text{-}ALE}}=\frac{\mathrm{tr}(M\Sigma_r)}{1-\gamma_P^2}.
```

5. Residual-feedback caveat:

```tex
K_\theta = K+\Phi_u J_r .
```

## Figure and Table Plan

| Item | Purpose | Required Content |
|---|---|---|
| Figure 1 | Source-amplifier diagram | Sources, input channel, nonnormal amplifier, consequences, projection burden, `M=Phi_u^T P Phi_u`. |
| Figure 2 | Controlled interventions | CI-1: `kappa(V)` or `G_peak` vs state covariance; CI-2: white, filtered, variance-matched colored, spectrum-matched scaled source. |
| Figure 3 | 2D RL ablation | Applied variance, dense violation, Bellman-Lyapunov max, projection correction. |
| Table 1 | Concept diagnostics | Source, amplifier, nonnormality, certificate, weighted source, consequence. |
| Table 2 | Main ablation summary | Direct TD3, `Var(u)`, action-diff, projection only, full response framework. |

## Experiment Matrix

| Experiment | Manipulated Variable | Fixed Variables | Answered Question |
|---|---|---|---|
| CI-1 amplifier scan | Eigenvector geometry / nonnormality parameter | Eigenvalues, spectral radius, input channel, source covariance | Can the same source covariance cause larger state covariance under stronger nonnormal amplification? |
| CI-2 source intervention | Source filtering and source energy | Physical amplifier and input channel | Does changing the source reduce downstream covariance, and can colored-input caveats be separated? |
| CI-3 direction-aware source test | Principal direction of `Sigma_r` at fixed trace | `K`, `Phi_u`, total source trace | Why is scalar action variance insufficient? |
| 2D constrained RL | Regularization and execution-layer components | Environment, seeds, noise level, nominal controller | Which parts reduce source variance, violations, B-L residuals, and projection burden? |
| Projection burden | Projection-only vs P-ALE plus projection vs full framework | Same safety layer and environment | Does source shaping reduce the amount of safety repair? |
| Actor-feedback diagnostic | Residual policy Jacobian magnitude | Local surrogate and evaluation state set | When does the residual actor become part of the amplifier rather than an exogenous source? |

## Required Wording Constraints

- Use "source-amplifier diagnostics and control" instead of "solves safe RL."
- Use "local surrogate certificate" instead of "global nonlinear certificate."
- Use "reduces certificate-weighted residual covariance" instead of "reduces nonnormality."
- Use "simulation-level evidence" instead of "deployment validation."

