%Version 3.1 December 2024
% See section 11 of the User Manual for version history
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%                                                                 %%
%% Please do not use \input{...} to include other tex files.       %%
%% Submit your LaTeX manuscript as one .tex document.              %%
%%                                                                 %%
%% All additional figures and files should be attached             %%
%% separately and not embedded in the \TeX\ document itself.       %%
%%                                                                 %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%\documentclass[referee,sn-basic]{sn-jnl}% referee option is meant for double line spacing

%%=======================================================%%
%% to print line numbers in the margin use lineno option %%
%%=======================================================%%

%%\documentclass[lineno,pdflatex,sn-basic]{sn-jnl}% Basic Springer Nature Reference Style/Chemistry Reference Style

%%=========================================================================================%%
%% the documentclass is set to pdflatex as default. You can delete it if not appropriate.  %%
%%=========================================================================================%%

%%\documentclass[sn-basic]{sn-jnl}% Basic Springer Nature Reference Style/Chemistry Reference Style

%%Note: the following reference styles support Namedate and Numbered referencing. By default the style follows the most common style. To switch between the options you can add or remove �Numbered� in the optional parenthesis. 
%%The option is available for: sn-basic.bst, sn-chicago.bst%  
 
%%\documentclass[pdflatex,sn-nature]{sn-jnl}% Style for submissions to Nature Portfolio journals
%%\documentclass[pdflatex,sn-basic]{sn-jnl}% Basic Springer Nature Reference Style/Chemistry Reference Style
\documentclass[pdflatex,sn-mathphys-num]{sn-jnl}% Math and Physical Sciences Numbered Reference Style
%%\documentclass[pdflatex,sn-mathphys-ay]{sn-jnl}% Math and Physical Sciences Author Year Reference Style
%%\documentclass[pdflatex,sn-aps]{sn-jnl}% American Physical Society (APS) Reference Style
%%\documentclass[pdflatex,sn-vancouver-num]{sn-jnl}% Vancouver Numbered Reference Style
%%\documentclass[pdflatex,sn-vancouver-ay]{sn-jnl}% Vancouver Author Year Reference Style
%%\documentclass[pdflatex,sn-apa]{sn-jnl}% APA Reference Style
%%\documentclass[pdflatex,sn-chicago]{sn-jnl}% Chicago-based Humanities Reference Style

%%%% Standard Packages
%%<additional latex packages if required can be included here>

\usepackage{graphicx}%
\usepackage{multirow}%
\usepackage{amsmath,amssymb,amsfonts}%
\usepackage{amsthm}%
\usepackage{mathrsfs}%
\usepackage[title]{appendix}%
\usepackage{xcolor}%
\usepackage{textcomp}%
\usepackage{manyfoot}%
\usepackage{booktabs}%
\usepackage{algorithm}%
\usepackage{algorithmicx}%
\usepackage{algpseudocode}%
\usepackage{listings}%
\usepackage{tikz}
\usetikzlibrary{shapes.geometric, arrows.meta, positioning, calc}
%%%%

%%%%%=============================================================================%%%%
%%%%  Remarks: This template is provided to aid authors with the preparation
%%%%  of original research articles intended for submission to journals published 
%%%%  by Springer Nature. The guidance has been prepared in partnership with 
%%%%  production teams to conform to Springer Nature technical requirements. 
%%%%  Editorial and presentation requirements differ among journal portfolios and 
%%%%  research disciplines. You may find sections in this template are irrelevant 
%%%%  to your work and are empowered to omit any such section if allowed by the 
%%%%  journal you intend to submit to. The submission guidelines and policies 
%%%%  of the journal take precedence. A detailed User Manual is available in the 
%%%%  template package for technical guidance.
%%%%%=============================================================================%%%%

%% as per the requirement new theorem styles can be included as shown below
\theoremstyle{thmstyleone}%
\newtheorem{theorem}{Theorem}%  meant for continuous numbers
%%\newtheorem{theorem}{Theorem}[section]% meant for sectionwise numbers
%% optional argument [theorem] produces theorem numbering sequence instead of independent numbers for Proposition
\newtheorem{proposition}[theorem]{Proposition}% 
%%\newtheorem{proposition}{Proposition}% to get separate numbers for theorem and proposition etc.

\theoremstyle{thmstyletwo}%
\newtheorem{example}{Example}%
\newtheorem{remark}{Remark}%

\theoremstyle{thmstylethree}%
\newtheorem{definition}{Definition}%

\raggedbottom
%%\unnumbered% uncomment this for unnumbered level heads

\begin{document}

\title[Article Title]{On the Design of Reward Function for Reinforcement Learning–Based Path Following Control Using the TD3 Algorithm}

%%=============================================================%%
%% GivenName	-> \fnm{Joergen W.}
%% Particle	-> \spfx{van der} -> surname prefix
%% FamilyName	-> \sur{Ploeg}
%% Suffix	-> \sfx{IV}
%% \author*[1,2]{\fnm{Joergen W.} \spfx{van der} \sur{Ploeg} 
%%  \sfx{IV}}\email{iauthor@gmail.com}
%%=============================================================%%

\author*[1]{\fnm{Mahdi} \sur{Khodaparast}}\email{mahdi97@email.kntu.ac.ir}

\author[1]{\fnm{Jafar} \sur{Roshanian}}\email{roshanian@kntu.ac.ir}
%\equalcont{These authors contributed equally to this work.}

%\author[1,2]{\fnm{Third} \sur{Author}}\email{iiiauthor@gmail.com}
%\equalcont{These authors contributed equally to this work.}

\affil[1]{\orgdiv{Intelligent Control Systems Institute (ICSI)}, \orgname{K.N. Toosi University of Technology}, \orgaddress{\city{Tehran}, \country{Iran}}}

%\affil[2]{\orgdiv{Department}, \orgname{Organization}, \orgaddress{\street{Street}, \city{City}, \postcode{10587}, \state{State}, \country{Country}}}

%\affil[3]{\orgdiv{Department}, \orgname{Organization}, \orgaddress{\street{Street}, \city{City}, \postcode{610101}, \state{State}, \country{Country}}}

%%==================================%%
%% Sample for unstructured abstract %%
%%==================================%%

\abstract{Reward design is one of the most influential factors in determining the performance of reinforcement learning (RL) algorithms in continuous-control tasks. This paper presents a comparative study of three reward formulations for training a TD3 agent to perform three-dimensional path following using a particle kinematic system modeled as double-integrator dynamics. The first reward is a linear formulation that assigns proportional penalties based on normalized position and velocity tracking errors. The second reward employs an exponential decay based on the same error terms to emphasize high-precision tracking and reduce sensitivity to smaller deviations. The third reward, a Piecewise Quadratic Reward (PQR), combines quadratic penalties on position and velocity errors with a positive discrete progress bonus, forming a hybrid structure that balances guidance and corrective feedback. All three reward functions are evaluated across three representative trajectories—vertical ascent, straight-line cruise, and helical motion—and analyzed with respect to convergence behavior, tracking accuracy, and control smoothness. The results show that the exponential reward produces more stable trajectories with higher tracking accuracy leading to smoother actions in complex paths, while the PQR formulation penalize the agent heavily leading to less stable learning. These findings provide practical insights into how reward shaping influences the behavior of TD3-based path-following agents and offer guidelines for designing effective reward functions in continuous-control RL applications.}



\keywords{Reinforcement Learning, TD3 Algorithm, Reward Function Design, Path Following Control, Trajectory Tracking, Continuous Control, Autonomous Aerial Robots}

%%\pacs[JEL Classification]{D8, H51}

%%\pacs[MSC Classification]{35A01, 65L10, 65L12, 65L20, 65L70}

\maketitle

\section{Introduction}\label{sec1}

Reinforcement learning (RL) has become a powerful framework for continuous-control 
problems in robotics, enabling autonomous agents to handle nonlinear dynamics, 
high-dimensional state spaces, and uncertain operating conditions without relying 
on explicit analytical models. Among such applications, three-dimensional 
(3D) path following remains a fundamental challenge where a control policy 
must track a prescribed trajectory with high precision while producing smooth, 
dynamically feasible commands. Although RL provides a general mechanism for 
learning such behaviors through trial-and-error interaction, the performance of 
RL agents in continuous-control tasks is highly sensitive to the design of the 
reward function. The reward is the sole channel through which the designer conveys 
task objectives to the agent, and even subtle modifications in its structure can 
substantially influence convergence speed, tracking accuracy, control smoothness,
and long-term stability.

Despite the central importance of reward design, RL-based robotic control studies
often rely on ad-hoc or heuristic reward structures without systematic 
justification. Many works combine weighted penalties on position and velocity 
tracking errors \cite{yuan2021reinforcement, srinivasan2020learning} or adopt 
quadratic tracking penalties as default choices \cite{tang2020reinforcement, 
li2021quadrotor}, while others introduce handcrafted shaping terms based on 
domain intuition \cite{mehr2021end, abdolhosseini2019deep}. These approaches 
can yield functional controllers, but they obscure the influence of reward 
formulation on the learned policy. Foundational studies on reward shaping—such as 
sparse versus dense rewards \cite{ng1999policy} and potential-based 
transformations \cite{devlin2012dynamic}—address reward structure in general 
but do not compare alternative functional forms for continuous tracking tasks.

More recent studies emphasize reward design but remain limited in scope. Several 
works examine reward construction in simplified settings such as 2D navigation or 
single-axis control \cite{kalashnikov2018qt, bellemare2016unifying}. In 3D 
control contexts, UAV trajectory tracking studies typically evaluate a single 
reward formulation without analyzing how different structures affect performance 
\cite{lee2021deep, hwangbo2017control}, while navigation- and exploration-oriented 
RL research \cite{liu2022deep, tai2017virtual} focuses on high-level behaviors 
rather than precise path tracking. Efforts in underwater and aerial robotics 
demonstrate custom mixed rewards for collision avoidance or path following 
\cite{rubí2021quadrotor_path_following, havenstrom2020auv_drl}, and Cao 
et al.\ \cite{cao2024_deep_mobile_path_following} explore reward shaping for mobile 
robots. However, these works do not conduct controlled comparisons of fundamentally 
different reward formulations.

Several recent publications highlight the continued importance of reward function 
selection across domains. Welde et al.\ \cite{welde2024symmetry_tracking} and 
Liu et al.\ \cite{liu2025trajectory} show that reward shaping accelerates 
convergence and improves robustness for complex 3D tracking tasks. In maritime 
and marine robotics, comparative studies investigate reward function impacts on 
collision avoidance and trajectory control \cite{krautwig2025navigating, 
suyama2024comparison, wang2024reinforcement}. Likewise, iterative reward shaping 
has been applied to prevent overshoot in nonlinear aerodynamic systems 
\cite{hu2023iterative}, and stand-off tracking formulations have been developed 
for aerial robotics applications \cite{jung2023reward}. Broader investigations into reward selection under 
limited feedback \cite{chaudhari2025rewards} and the use of fused sparse and 
shaping rewards in SAC-based mobile robot navigation \cite{bakar2023fusion} 
reinforce the need for structured evaluation methodologies. Yet none of these works 
perform a side-by-side analysis of multiple reward classes under identical 
algorithmic and experimental conditions.

To the best of our knowledge, no prior study—particularly within robotic 
path-following control—has provided a controlled experimental comparison of 
multiple reward structures while isolating their effects from algorithmic, 
architectural, or modeling differences.

This paper addresses this gap by providing a systematic comparison of three reward 
structures for 3D path following using the Twin Delayed Deep Deterministic Policy 
Gradient (TD3) algorithm \cite{fujimoto2018addressing}. TD3 is particularly well 
suited for trajectory control due to three properties: (1) strong performance in 
continuous action spaces matching the continuous acceleration commands required 
for tracking; (2) deterministic policy outputs ensuring reproducible, precise 
reference following; and (3) reduced Q-value overestimation through clipped double 
critics, stabilizing training in the presence of rapidly varying errors.

The agent controls a particle kinematic model with 3D double-integrator dynamics—a 
canonical representation that captures the core difficulty of continuous trajectory 
tracking while remaining computationally tractable for RL training. The agent 
receives current position, velocity, and reference trajectory information as 
inputs, and outputs continuous acceleration commands. To challenge each reward 
function across varying dynamical conditions, we evaluate three representative 
trajectories: vertical ascent, straight-line cruise, and helical motion. These 
trajectories represent different levels of difficulty, from single-axis altitude 
tracking that includes a challenging vertical-to-horizontal transition, through 
sustained long-duration cruise tracking, to fully coupled 3D maneuvers with 
continuously varying curvature.

We study three widely used reward formulations that capture fundamentally 
different approaches to penalizing tracking errors:

\begin{enumerate}
\item \textbf{Linear Reward:} proportional penalties on normalized tracking 
errors, producing uniform sensitivity across error magnitudes.
\item \textbf{Exponential Reward:} an exponentially decaying formulation that 
emphasizes high accuracy near the reference while sharply reducing reward for 
large deviations.
\item \textbf{Piecewise Quadratic Reward (PQR):} quadratic error penalties 
combined with a progress-reward term, balancing corrective feedback with 
forward-motion incentives.
\end{enumerate}

These reward structures differ in gradient characteristics and error sensitivity, 
leading to distinct behaviors during optimization. Linear rewards treat all 
deviations with equal urgency; exponential rewards adaptively scale penalties; and 
PQR rewards introduce nonlinear growth in error terms, potentially offering stronger 
corrective signals but also introducing non-smooth reward landscapes. Because these 
differences can significantly influence learning dynamics, a controlled and 
systematic comparison is essential for practitioners seeking to design simple and 
effective reward functions for robotic control.

Each reward function is evaluated under identical training conditions—same 
hyperparameters, architectures, and exploration parameters—ensuring that 
performance differences arise solely from reward structure rather than confounding 
algorithmic choices. Evaluation metrics include success rate, root-mean-square 
tracking errors (RMSE), and control smoothness.

The contributions of this work are as follows:
\begin{itemize}
\item A systematic, controlled comparison of three reward structures for TD3-based
3D path following, providing insight into how reward design affects learning behavior.
\item A unified experimental methodology that isolates the impact of reward 
formulation and enables fair, interpretable comparisons across trajectories of 
varying complexity.
\item Practical guidelines for reward design in continuous-control robotic 
applications, based on empirical findings regarding tracking accuracy, convergence, 
learning stability, and control smoothness.
\end{itemize}

The remainder of this paper is organized as follows. 
Section~\ref{sec2} defines the path-following problem, summarizes the system 
model and control framework, and formulates the task as a Markov Decision 
Process (MDP). Section~\ref{sec3} details the reward functions.  
Section~\ref{sec4} describes the experimental setup.  
Section~\ref{sec5} presents the simulation results and comparative analysis.  
Sections~\ref{sec:conclusion} and~\ref{sec:future_work} provide the conclusions 
and outline directions for future research, respectively.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Problem Formulation}\label{sec2}

This section defines the path-following problem as a reinforcement learning task and describes the system dynamics, reference trajectories, and the TD3 algorithm used to train the control policy.


\subsection{Path Following vs Trajectory Tracking}
The core difference between trajectory tracking and path following is the role of time, as shown in Figure~\ref{fig:TTvsPF}: path following treats the route geometry as primary (with flexible timing) \cite{rubi2020survey}, whereas trajectory tracking enforces time-dependent waypoints and tightly couples the robot’s motion to a time profile \cite{li2021auv}. This leads to different controller designs and trade-offs. Path followers often achieve smoother convergence and better disturbance rejection \cite{walters2014online} (useful for survey missions or slow vehicles), whereas trajectory trackers enforce exact timing and speed (necessary for synchronized or high-precision tasks). Each approach has its use cases and challenges, and many studies contrast their performance in various domains \cite{he2025review}. The recently provided work on laser beam steering along 3D paths explicitly frames beam steering as a path-following problem and contrasts it with trajectory tracking, giving concrete examples where timing flexibility simplifies controller objectives and improves robustness in 3D guidance tasks\cite{tamadazte2018laser}.

\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{TTvsPF.pdf}
\caption{Trajectory tracking versus path following.}
\label{fig:TTvsPF}
\end{figure}



Although the reference trajectories in this study are defined as explicit functions of time, the designed reward functions do not enforce time synchronization. As a result, the agent is not required to match the temporal evolution of the reference, but only to minimize the spatial tracking error. This makes the control objective functionally equivalent to path following rather than strict trajectory tracking, consistent with standard definitions in the literature. For this reason, the notions \textit{trajectory tracking} and \textit{path following} are used interchangeably throughout this work. Moreover, because the reward design is formulated solely in terms of general tracking errors, the same framework can incorporate additional error terms—such as timing constraints—without modification. Consequently, the methodology and results presented here are applicable to both path-following and trajectory-tracking control problems, and may serve as a foundation for future research in either domain.



\subsection{System Dynamics}

The control plant is modeled as a three-dimensional double-integrator system~\cite{lewis2013cooperative}. This canonical representation captures the fundamental challenges of continuous trajectory tracking while remaining computationally tractable for reinforcement learning. The system evolves according to the following equations:

\begin{equation}
\ddot{\mathbf{p}} = \mathbf{a}
\label{eq:dynamics}
\end{equation}

where $\mathbf{p} = [x, y, z]^T \in \mathbb{R}^3$ represents the position vector in three-dimensional space, and $\mathbf{a} = [a_x, a_y, a_z]^T \in \mathbb{R}^3$ denotes the acceleration command (control input) provided by the RL agent. The dynamics can be expressed in state-space form as:

\begin{equation}
\frac{d}{dt}\begin{bmatrix} \mathbf{p} \\ \dot{\mathbf{p}} \end{bmatrix} = \begin{bmatrix} \mathbf{0}_{3\times3} & \mathbf{I}_{3\times3} \\ \mathbf{0}_{3\times3} & \mathbf{0}_{3\times3} \end{bmatrix}\begin{bmatrix} \mathbf{p} \\ \dot{\mathbf{p}} \end{bmatrix} + \begin{bmatrix} \mathbf{0}_{3\times3} \\ \mathbf{I}_{3\times3} \end{bmatrix}\mathbf{a}
\label{eq:state_space}
\end{equation}

where $\mathbf{I}_{3\times3}$ is the $3\times3$ identity matrix and $\mathbf{0}_{3\times3}$ is the $3\times3$ zero matrix. The discrete-time update for simulation is computed using forward Euler integration with sampling time $\Delta t = 0.01$ s:

\begin{align}
\mathbf{p}(t+\Delta t) &= \mathbf{p}(t) + \dot{\mathbf{p}}(t)\Delta t \label{eq:position_update}\\
\dot{\mathbf{p}}(t+\Delta t) &= \dot{\mathbf{p}}(t) + \mathbf{a}(t)\Delta t \label{eq:velocity_update}
\end{align}

The acceleration command is bounded componentwise to ensure physically realizable control actions: $\mathbf{a} \in [-3, 3]^3$ m/s$^2$. This constraint reflects typical acceleration limits for autonomous systems and promotes smooth, dynamically feasible trajectories.

\subsection{Reference Trajectories}

To evaluate the performance and robustness of each reward formulation across diverse dynamical conditions, three representative reference trajectories are considered. Each trajectory is designed as a complete, standalone path that tests distinct aspects of the control policy.

\subsubsection{Trajectory 1: Vertical Ascent with Cruise}

The first trajectory consists of a vertical climb from the origin to a specified altitude, followed by a brief horizontal cruise phase. This trajectory tests the agent's ability to handle single-axis dominant motion with a transition to multi-axis tracking. The reference path is defined as:

\begin{equation}
\mathbf{p}_d(t) = \begin{cases}
\begin{bmatrix} 0 \\ 0 \\ \frac{h_{\text{max}}t}{t_{\text{climb}}} \end{bmatrix}, & 0 \leq t \leq t_{\text{climb}} \\[10pt]
\begin{bmatrix} v_c(t-t_{\text{climb}}) \\ 0 \\ h_{\text{max}} \end{bmatrix}, & t_{\text{climb}} < t \leq t_{\text{max}}
\end{cases}
\label{eq:traj_vertical}
\end{equation}

where $h_{\text{max}} = 5$ m is the target altitude, $t_{\text{climb}} = 5$ s is the climb duration, $v_c = 0.8$ m/s is the cruise speed, and $t_{\text{max}} = 7$ s is the total trajectory duration. The inclusion of the cruise phase addresses the practical challenge of transitioning from vertical to horizontal motion, which is particularly relevant for aerial robotic systems. This trajectory contains 701 data points at sampling interval $\Delta t = 0.01$ s.

\subsubsection{Trajectory 2: Straight-Line Cruise}

The second trajectory consists of a straight-line cruise at constant altitude,
primarily a single-axis tracking task, but one that tests the agent’s ability
to maintain stable long-duration tracking while preventing drift in the lateral
and vertical axes:

\begin{equation}
\mathbf{p}_d(t) = \begin{bmatrix} v_c t \\ 0 \\ h_0 \end{bmatrix}, \quad 0 \leq t \leq t_{\text{max}}
\label{eq:traj_cruise}
\end{equation}

where $v_c = 0.8$ m/s is the cruise speed, $h_0 = 5$ m is the constant altitude, and $t_{\text{max}} = 75$ s is the total trajectory duration. This trajectory emphasizes sustained tracking performance and altitude stability. The trajectory contains 7501 data points.

\subsubsection{Trajectory 3: Helical Maneuver}

The third trajectory consists of a three-dimensional helical maneuver, representing the most demanding tracking scenario with continuously varying curvature and direction:

\begin{equation}
\mathbf{p}_d(t) = \begin{bmatrix} x_0 + R\cos(\omega t) \\ y_0 + R\sin(\omega t) \\ h_0 + v_z t \end{bmatrix}, \quad 0 \leq t \leq t_{\text{max}}
\label{eq:traj_helical}
\end{equation}

where $R = 4$ m is the helix radius, $\omega = \frac{\pi}{6}$ rad/s is the angular velocity, $v_z = 0.4$ m/s is the vertical speed, $(x_0, y_0, h_0) = (66, -4, 13)$ m is the arbitrary initial horizontal position and altitude, and $t_{\text{max}} = 30$ s is the total trajectory duration. This trajectory requires simultaneous control in all three spatial dimensions. The trajectory contains 3001 data points.


\subsection{Markov Decision Process Formulation}

The path-following task is formulated as a continuous-control Markov Decision Process (MDP) defined by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$, where:

\subsubsection{State Space ($\mathcal{S}$)}

The state vector at time step $t$ contains the agent's current position, velocity, and the corresponding reference trajectory information:

\begin{equation}
s_t = [x, \dot{x}, y, \dot{y}, z, \dot{z}, x_d, \dot{x}_d, y_d, \dot{y}_d, z_d, \dot{z}_d]^T \in \mathbb{R}^{12}
\label{eq:state}
\end{equation}

where $(x, y, z)$ and $(\dot{x}, \dot{y}, \dot{z})$ represent the agent's actual position and velocity, and $(x_d, y_d, z_d)$ and $(\dot{x}_d, \dot{y}_d, \dot{z}_d)$ denote the desired position and velocity from the reference trajectory at the current time step. This state representation provides the agent with full observability of tracking errors in both position and velocity domains, which are essential for accurate trajectory following.

\subsubsection{Action Space ($\mathcal{A}$)}

The action space consists of continuous three-dimensional acceleration commands:

\begin{equation}
a_t = [a_x, a_y, a_z]^T \in \mathcal{A} = [-3, 3]^3 \subset \mathbb{R}^3
\label{eq:action}
\end{equation}

where each component is bounded to $[-3, 3]$ m/s$^2$. These bounds ensure that commanded accelerations remain within physically realizable limits and promote control smoothness.

\subsubsection{Transition Dynamics ($\mathcal{P}$)}

The state transition function $\mathcal{P}: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ is deterministic and follows the double-integrator dynamics described in Equations~(\ref{eq:position_update}) and (\ref{eq:velocity_update}). The reference trajectory portion of the state is updated by advancing to the next time step in the precomputed trajectory data.

\subsubsection{Reward Function ($\mathcal{R}$)}

The reward function $\mathcal{R}: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ quantifies the desirability of the agent's behavior at each time step. Three distinct reward formulations—Linear, Exponential, and Piecewise Quadratic—are systematically compared in this study. The detailed mathematical specifications of these reward functions are provided in Section~\ref{sec3}.

\subsubsection{Discount Factor ($\gamma$)}

A discount factor of $\gamma = 0.99$ is used to balance immediate and future rewards, encouraging the agent to consider long-term tracking performance over the entire trajectory.

\subsubsection{Episode Termination}

An episode terminates under any of the following conditions:
\begin{itemize}
\item The agent successfully completes the reference trajectory (reaches the final time step).
\item The position or velocity error magnitude exceeds a threshold of 3 m or 3 m/s, indicating loss of tracking.
\item The altitude falls below $-0.5$ m (ground collision avoidance).
\end{itemize}

Early termination due to large tracking errors accelerates training by preventing the agent from exploring unproductive regions of the state space.

\subsection{Twin Delayed Deep Deterministic Policy Gradient (TD3)}

The TD3 algorithm is employed to learn a deterministic policy $\pi_\phi: \mathcal{S} \to \mathcal{A}$ that maps states to continuous acceleration commands. TD3 is particularly well-suited for this application due to three key properties: (1) native support for continuous action spaces, matching the continuous acceleration commands required for trajectory tracking; (2) deterministic policy outputs, which are essential for reproducible and precise reference following; and (3) reduced Q-value overestimation through clipped double critics, which stabilizes training when the agent encounters rapidly varying tracking errors.

TD3 extends the Deep Deterministic Policy Gradient (DDPG) algorithm with three critical modifications to improve stability and performance:

\subsubsection{Clipped Double Q-Learning}

TD3 maintains two independent critic networks, $Q_{\theta_1}$ and $Q_{\theta_2}$, and uses the minimum of their estimates to compute target values:

\begin{equation}
y = r + \gamma \min_{i=1,2} Q_{\theta_i'}(s', \pi_{\phi'}(s'))
\label{eq:target_value}
\end{equation}

where $\theta_1'$ and $\theta_2'$ denote the parameters of the target critic networks. This approach mitigates the overestimation bias common in value-based methods by providing a conservative Q-value estimate.

\subsubsection{Target Policy Smoothing}

Noise is added to the target policy action to regularize the Q-value estimates and prevent overfitting to narrow peaks in the value function:

\begin{equation}
a' = \pi_{\phi'}(s') + \epsilon, \quad \epsilon \sim \text{clip}(\mathcal{N}(0, \tilde{\sigma}), -c, c)
\label{eq:target_smoothing}
\end{equation}

where $\tilde{\sigma}$ controls the noise magnitude and $c$ is the clipping range. This smoothing encourages the critic to learn more robust Q-value estimates in the vicinity of the target action.

\subsubsection{Delayed Policy Updates}

The actor network and target networks are updated less frequently than the critic networks (typically every $d=2$ critic updates). This delayed updating reduces the coupling between actor and critic, allowing the critic to stabilize before being used to update the policy:

\begin{align}
\phi &\leftarrow \phi + \alpha_\pi \nabla_\phi \mathbb{E}_s[Q_{\theta_1}(s, \pi_\phi(s))] \label{eq:actor_update}\\
\theta_i' &\leftarrow \tau \theta_i + (1-\tau)\theta_i', \quad i \in \{1,2\} \label{eq:critic_target_update}\\
\phi' &\leftarrow \tau \phi + (1-\tau)\phi' \label{eq:actor_target_update}
\end{align}

where $\alpha_\pi$ is the actor learning rate and $\tau$ is the soft update coefficient. The expectation in Equation~(\ref{eq:actor_update}) is approximated using mini-batch samples from the replay buffer.

\subsubsection{Training Procedure}

The complete TD3 training procedure is summarized in Algorithm~\ref{alg:td3} and illustrated in the flowchart in Figure~\ref{fig:td3_flowchart}. At each time step, the agent selects an action by adding exploration noise to the deterministic policy output. The resulting transition $(s_t, a_t, r_t, s_{t+1})$ is stored in an experience replay buffer $\mathcal{B}$. When sufficient experience has been collected, mini-batches are sampled from $\mathcal{B}$ to update the critic and actor networks using stochastic gradient descent.

% Placeholder for TD3 pseudocode
\begin{algorithm}[h]
\caption{TD3 Algorithm for Path Following}\label{alg:td3}
\begin{algorithmic}[1]
\State \textbf{Initialize:} Actor $\pi_\phi$, Critics $Q_{\theta_1}, Q_{\theta_2}$, Target networks $\pi_{\phi'}, Q_{\theta_1'}, Q_{\theta_2'}$
\State \textbf{Initialize:} Replay buffer $\mathcal{B} \leftarrow \emptyset$
\For{episode $= 1$ to $N_{\text{episodes}}$}
    \State Observe initial state $s_1$
    \For{$t = 1$ to $T_{\text{max}}$}
        \State Select action with exploration noise: $a_t = \pi_\phi(s_t) + \epsilon$, $\epsilon \sim \mathcal{N}(0, \sigma)$
        \State Execute $a_t$, observe reward $r_t$ and next state $s_{t+1}$
        \State Store transition $(s_t, a_t, r_t, s_{t+1})$ in $\mathcal{B}$
        \If{ready to update}
            \State Sample mini-batch $\{(s_i, a_i, r_i, s_i')\}_{i=1}^N$ from $\mathcal{B}$
            \State Compute target actions: $a_i' = \pi_{\phi'}(s_i') + \text{clip}(\epsilon, -c, c)$, $\epsilon \sim \mathcal{N}(0, \tilde{\sigma})$
            \State Compute target Q-values: $y_i = r_i + \gamma \min_{j=1,2} Q_{\theta_j'}(s_i', a_i')$
            \State Update critics: $\theta_j \leftarrow \theta_j - \alpha_Q \nabla_{\theta_j} \frac{1}{N}\sum_i (Q_{\theta_j}(s_i, a_i) - y_i)^2$
            \If{$t \mod d = 0$}
                \State Update actor: $\phi \leftarrow \phi + \alpha_\pi \nabla_\phi \frac{1}{N}\sum_i Q_{\theta_1}(s_i, \pi_\phi(s_i))$
                \State Update target networks:
                \State \quad $\theta_j' \leftarrow \tau \theta_j + (1-\tau)\theta_j'$ for $j \in \{1,2\}$
                \State \quad $\phi' \leftarrow \tau \phi + (1-\tau)\phi'$
            \EndIf
        \EndIf
    \EndFor
\EndFor
\end{algorithmic}
\end{algorithm}

% =========================================================
% ********* TD3 Algorithm Flowchart - TikZ Code **********
% =========================================================
\begin{figure}[htbp]
\centering
\includegraphics[width=0.6\textwidth]
{flowchart.pdf}
\caption{Flowchart of the TD3 algorithm for trajectory tracking control. 
The algorithm alternates between interaction with the environment 
(action selection and execution) and learning (critic and actor updates from replay buffer samples).}
\label{fig:td3_flowchart}
\end{figure}


% =========================================================
% ******** TD3 ARCHITECTURE DIAGRAM - TikZ Code *********
% =========================================================
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{TD3_architecture.pdf}
\caption{Architecture of the TD3 agent showing the actor network, twin critic networks, experience replay buffer, and target networks. The agent receives state observations from the environment, outputs continuous acceleration commands, and updates both actor and critic networks using sampled transitions from the replay buffer.}
\label{fig:td3_architecture}
\end{figure}


The key distinction of this work lies not in modifications to the TD3 algorithm itself, but rather in the systematic comparison of reward function designs. All hyperparameters, network architectures, and training procedures are held constant across experiments to isolate the effect of reward formulation on learning performance.

% ===========================================================
% SECTION 3: REWARD FUNCTIONS
% ===========================================================

\section{Reward Functions}\label{sec3}

The reward function is the primary mechanism through which task objectives are communicated to the RL agent. In trajectory tracking tasks, the reward structure directly determines what behaviors the agent learns to prioritize—whether precise position tracking, velocity matching, control smoothness, or a combination thereof. This section presents three reward formulations that embody fundamentally 
different approaches to tracking error: rewards that decrease linearly 
with error, rewards that decay exponentially with increasing error, 
and quadratic penalties combined with forward progress incentives.


All three reward functions operate on the same error signals but differ in how they transform these errors into scalar reward values. The error signals are defined as:

\begin{align}
\mathbf{e}_{\text{pos}} &= \mathbf{p}_d - \mathbf{p} = [x_d - x, y_d - y, z_d - z]^T \label{eq:position_error}\\
\mathbf{e}_{\text{vel}} &= \dot{\mathbf{p}}_d - \dot{\mathbf{p}} = [\dot{x}_d - \dot{x}, \dot{y}_d - \dot{y}, \dot{z}_d - \dot{z}]^T \label{eq:velocity_error}
\end{align}

where $\mathbf{p}_d$ and $\dot{\mathbf{p}}_d$ denote the desired position and velocity from the reference trajectory, and $\mathbf{p}$ and $\dot{\mathbf{p}}$ represent the agent's actual position and velocity. The magnitudes of these error vectors are computed as:

\begin{align}
\|\mathbf{e}_{\text{pos}}\| &= \sqrt{(x_d - x)^2 + (y_d - y)^2 + (z_d - z)^2} \label{eq:position_error_magnitude}\\
\|\mathbf{e}_{\text{vel}}\| &= \sqrt{(\dot{x}_d - \dot{x})^2 + (\dot{y}_d - \dot{y})^2 + (\dot{z}_d - \dot{z})^2} \label{eq:velocity_error_magnitude}
\end{align}

These error magnitudes serve as inputs to the reward functions described below.

\subsection{Linear Reward Function}

The linear reward function implements a proportional penalty structure where the reward decreases linearly with increasing tracking error. This formulation treats all magnitudes of error with uniform sensitivity, providing consistent gradient information across the entire error range. The linear reward is defined as:

\begin{equation}
R_{\text{linear}} = R_{\text{pos}} + R_{\text{vel}}
\label{eq:linear_reward}
\end{equation}

where the position and velocity reward components are given by:

\begin{align}
R_{\text{pos}} &= w_p \left(1 - \frac{\|\mathbf{e}_{\text{pos}}\|}{s_{\text{pos}}}\right) \label{eq:linear_position}\\
R_{\text{vel}} &= w_v \left(1 - \frac{\|\mathbf{e}_{\text{vel}}\|}{s_{\text{vel}}}\right) \label{eq:linear_velocity}
\end{align}

The parameters $w_p$ and $w_v$ are weighting coefficients that determine the relative importance of position and velocity tracking, while $s_{\text{pos}}$ and $s_{\text{vel}}$ are normalization scales(maximum allowed error magnitudes) that map error magnitudes to the range $[0, 1]$. For this study, the following parameter values are used:

\begin{itemize}
\item Position weight: $w_p = 0.5$
\item Velocity weight: $w_v = 0.5$
\item Position scale: $s_{\text{pos}} = 3.0$ m
\item Velocity scale: $s_{\text{vel}} = 3.0$ m/s
\end{itemize}

The linear structure ensures that the reward gradient with respect to error is constant, meaning the agent receives the same marginal reward for reducing error from 2 m to 1 m as it does for reducing error from 1 m to 0 m. This property can be advantageous when exploration is needed across a wide range of error magnitudes, but may provide insufficient incentive for achieving high-precision tracking near the reference.

\subsection{Exponential Reward Function}

The exponential reward function applies an exponentially decaying reward based on tracking error magnitude. This formulation emphasizes high-precision tracking by assigning disproportionately higher rewards for small errors while remaining tolerant of larger deviations. The exponential reward is defined as:

\begin{equation}
R_{\text{exp}} = R_{\text{pos}} + R_{\text{vel}}
\label{eq:exponential_reward}
\end{equation}

where the position and velocity components are:

\begin{align}
R_{\text{pos}} &= w_p \cdot e^{-\sigma \|\mathbf{e}_{\text{pos}}\|} \label{eq:exp_position}\\
R_{\text{vel}} &= w_v \cdot e^{-\sigma \|\mathbf{e}_{\text{vel}}\|} \label{eq:exp_velocity}
\end{align}

The parameter $\sigma$ controls the decay rate of the exponential function—higher values of $\sigma$ produce steeper gradients near zero error, emphasizing precision, while lower values distribute reward more evenly across error magnitudes. For this study, the parameters are set as:

\begin{itemize}
\item Position weight: $w_p = 0.5$
\item Velocity weight: $w_v = 0.5$
\item Decay coefficient: $\sigma = 3.0$
\end{itemize}

The exponential structure creates a highly nonlinear reward landscape. Near the reference ($\|\mathbf{e}_{\text{pos}}\| \approx 0$), the reward gradient is steep, providing strong incentive for precision. As error increases, the gradient diminishes, meaning the agent 
receives a weaker marginal signal to reduce error when tracking 
performance is already poor. This characteristic can accelerate convergence to high-accuracy policies but may also introduce challenges during early training when large errors are common and the gradient signal becomes weak.

\subsection{Piecewise Quadratic Reward (PQR) Function}

The Piecewise Quadratic Reward combines continuous quadratic error penalties with a discrete positive progress bonus, forming a hybrid structure that balances corrective feedback with forward motion incentives. Unlike the previous formulations, PQR includes both negative (penalty) and positive (bonus) components. The total reward is:

\begin{equation}
R_{\text{PQR}} = P_{\text{pos}} + P_{\text{vel}} + R_{\text{progress}}
\label{eq:pqr_reward}
\end{equation}

where the position and velocity penalties are quadratic functions of error magnitude:

\begin{align}
P_{\text{pos}} &= -w_p \cdot \|\mathbf{e}_{\text{pos}}\|^2 \label{eq:pqr_position}\\
P_{\text{vel}} &= -w_v \cdot \|\mathbf{e}_{\text{vel}}\|^2 \label{eq:pqr_velocity}
\end{align}

and the progress reward is a constant positive term:

\begin{equation}
R_{\text{progress}} = +0.1
\label{eq:pqr_progress}
\end{equation}

The parameter values for PQR are:

\begin{itemize}
\item Position penalty weight: $w_p = 0.5$
\item Velocity penalty weight: $w_v = 0.5$
\item Progress bonus: $R_{\text{progress}} = 0.1$
\end{itemize}

The quadratic penalty structure produces error gradients that scale linearly with error magnitude—small errors receive small penalties, while large errors are heavily penalized. This nonlinear growth can provide strong corrective signals when tracking performance is poor, potentially accelerating early-stage learning. However, the steep penalties for large errors may also introduce discontinuities in the value function, making exploration more challenging.

The progress bonus serves a distinct purpose: it encourages the agent to maintain forward motion along the trajectory even when tracking errors are present. Without this term, the agent might adopt conservative behaviors such as stopping or slowing down to minimize error, which would fail to complete the trajectory. The constant positive reward ensures that continued progress is always beneficial, provided errors remain bounded.


A critical design consideration in PQR is the relative scale of the progress reward compared to the quadratic tracking penalties. Because the total reward is obtained by summing both terms, an excessively large progress bonus can mask the severity of tracking errors. For example, if the quadratic penalty reaches $P_{\text{pos}} + P_{\text{vel}} = -1.5$ while the progress reward is set to $+1.0$, the net reward becomes $-0.5$. Although this still reflects a negative outcome, the magnitude of the penalty is greatly attenuated. As a result, the agent may fail to recognize poor tracking performance and instead learn that advancing along the path is more valuable than reducing error. This imbalance biases the policy toward ``progress-dominant'' behavior, where the agent prioritizes moving forward even when deviating substantially from the desired trajectory. To prevent this effect, the progress reward must remain sufficiently small relative to the maximum expected quadratic penalties, ensuring that the optimizer receives informative gradients about tracking quality while still being encouraged to complete the trajectory.


\subsection{Comparative Properties}

The three reward functions exhibit fundamentally different characteristics in how they shape the learning process:

\begin{itemize}
\item \textbf{Gradient structure:} Linear rewards provide constant gradients; exponential rewards produce steep gradients near zero error and shallow gradients for large errors; PQR produces gradients that increase linearly with error magnitude.

\item \textbf{Error sensitivity:} Linear rewards treat all error magnitudes uniformly; exponential rewards emphasize precision near the reference; PQR penalizes large errors most severely.

\item \textbf{Exploration incentives:} Linear rewards encourage broad exploration; exponential rewards may inhibit exploration due to weak gradients at large errors; PQR's progress bonus actively encourages forward motion.

\item \textbf{Convergence behavior:} Linear rewards may converge slowly but stably; exponential rewards may converge quickly to high-precision policies; PQR may accelerate early learning but introduce training instability due to steep penalties.
\end{itemize}

An additional distinction concerns the presence of progress reward. In the PQR formulation, progress is encoded explicitly through a discrete bonus term, whereas the linear and exponential rewards do not require such an additive component. Because both linear and exponential rewards accumulate over time in proportion to instantaneous tracking accuracy, they inherently provide a progress reward: maintaining low error directly increases cumulative reward. This intentional difference allows the experiments in Section~\ref{sec5} to contrast explicit discrete progress shaping with the implicit continuous progress shaping present in smooth reward formulations, where all three formulations are evaluated under identical conditions to isolate the impact of reward design on path following performance.


\begin{figure}[htbp]
\centering
\includegraphics[width=0.4\textwidth]{reward_comparison.pdf}
\caption{Comparison of the three reward functions as a function of error magnitude. The linear reward decreases proportionally with error, the exponential reward is highest at zero error and decays rapidly with increasing error, and the PQR combines quadratic penalties with a constant progress bonus.}
\label{fig:reward_comparison}
\end{figure}






Beyond these structural distinctions, the selection of these three reward formulations was also motivated by their relevance in the existing literature. Linear and exponential rewards represent smooth, continuously shaped paradigms commonly adopted in trajectory-tracking problems, whereas the PQR formulation reflects a class of reward designs that combine discrete progress terms with negative squared-error penalties. Several prior works employ such discrete or negative-error rewards without direct comparison to smooth alternatives; therefore, evaluating all three under identical conditions provides a systematic basis for understanding the practical consequences of using discrete versus continuous reward structures.




% ============================================
% SECTION 4: EXPERIMENTAL SETUP - REVISED
% ============================================

\section{Experimental Setup}\label{sec4}

This section summarizes the implementation details of the TD3 agent, including the network architecture, hyperparameters, training configuration, and evaluation metrics. All experiments were conducted under identical conditions for the three reward functions to ensure a fair comparison.

\subsection{Network Architecture}

The TD3 agent consists of an actor network and two critic networks. All networks use fully connected feedforward architectures with ReLU activations in the hidden layers.

The actor $\pi_\phi: \mathbb{R}^{12} \to \mathbb{R}^3$ maps the 12-dimensional state vector (reference position, reference velocity, current position, current velocity) to bounded acceleration commands. The architecture is:
\[
256 \; \text{ReLU} \rightarrow 256 \; \text{ReLU} \rightarrow \tanh,
\]
where the final output is scaled elementwise to the range $[-3, 3]$~m/s$^2$.

Each critic network $Q_{\theta_i}: \mathbb{R}^{15} \to \mathbb{R}$ receives the concatenated state–action pair and uses the same structure:
\[
256 \; \text{ReLU} \rightarrow 256 \; \text{ReLU} \rightarrow 1.
\]

Target networks replicate the online networks and are updated via soft updates with $\tau = 0.005$.


\subsection{Hyperparameters}

Table~\ref{tab:hyperparameters} summarizes the hyperparameters used for training the TD3 agent. These values were selected based on preliminary experiments and remain constant across all three reward functions to isolate the effect of reward structure on learning performance.

\begin{table}[htbp]
\centering
\caption{TD3 training hyperparameters.}
\label{tab:hyperparameters}
\begin{tabular}{lc}
\toprule
\textbf{Hyperparameter} & \textbf{Value} \\
\midrule
Action bounds ($a$) (m/s$^2$) & $[-3, 3]$ \\
Discount factor ($\gamma$) & 0.99 \\
\midrule
\multicolumn{2}{l}{\textit{Exploration Noise}} \\
\quad Initial standard deviation ($\sigma$) & 0.1 \\
\quad Minimum standard deviation & 0.05 \\
\quad Decay rate & $10^{-5}$ \\
\quad Clipping range & $[-a, a]$ \\
\midrule
\multicolumn{2}{l}{\textit{Target Policy Noise}} \\
\quad Initial standard deviation ($\tilde{\sigma}$) & 0.2 \\
\quad Minimum standard deviation & 0.03 \\
\quad Decay rate & $5 \times 10^{-4}$ \\
\quad Clipping range & $[-a/2, a/2]$ \\
\midrule
\multicolumn{2}{l}{\textit{Optimization}} \\
\quad Actor learning rate ($\alpha_\pi$) & $10^{-4}$ \\
\quad Critic 1 learning rate ($\alpha_{Q_1}$) & $3 \times 10^{-4}$ \\
\quad Critic 2 learning rate ($\alpha_{Q_2}$) & $3 \times 10^{-4}$ \\
\quad Target update coefficient ($\tau$) & 0.005 \\
\quad Policy update frequency ($d$) & 2 \\
\midrule
\multicolumn{2}{l}{\textit{Experience Replay}} \\
\quad Buffer capacity & $10^6$ \\
\quad Mini-batch size & 256 \\
\botrule
\end{tabular}
\end{table}




% ============================================
% SECTION 5: RESULTS AND ANALYSIS
% ============================================

\section{Results and Analysis}\label{sec5}

This section presents a comprehensive evaluation of the TD3 agent trained under the three reward formulations introduced earlier: the linear reward, the exponential reward, and the piecewise quadratic reward (PQR). The analysis spans three reference trajectories of increasing complexity—vertical ascent, straight-line cruise, and helical motion. For each trajectory, we examine the training convergence behavior, the final tracking performance, and the reward distributions from the last training episode. Quantitative performance metrics, including RMSE, maximum errors, and control smoothness, provide additional insight into the characteristics of each reward function.

It is important to emphasize that the success rate computed in this section is \emph{reward-function-dependent}. Each reward function has a different maximum achievable reward value and a different mapping between tracking error and reward magnitude. Therefore, a higher success rate does not necessarily imply superior tracking performance. In fact, the exponential reward deliberately penalizes deviations more aggressively, producing lower success rates even when its actual trajectory tracking accuracy is significantly higher. Conversely, the linear and PQR rewards may yield higher success rates despite poorer tracking quality due to their more liberal reward assignment. This distinction is essential when interpreting the results that follow.

% ============================================
% 5.1 Training Convergence Across Reward Functions
% ============================================

\subsection{Training Convergence Across Reward Functions}

Figures~\ref{fig:train_traj1}--\ref{fig:train_traj3} illustrate the learning curves for all three trajectories and reward functions. Each plot contains both the episode reward and the episode length over the entire training horizon.

\paragraph{Trajectory 1 (Vertical Ascent)}
The linear and PQR rewards exhibited faster convergence in terms of episode reward magnitude; however, this does \emph{not} reflect superior tracking. The exponential reward resulted in consistently lower reward values because it penalizes deviations more heavily, but it converged to a more stable and accurate policy, which becomes evident in later tracking and RMSE evaluations.

\paragraph{Trajectory 2 (Straight-Line Cruise)}
Training the cruise trajectory requires maintaining accurate tracking over an extended time horizon. All reward functions eventually stabilize; however, the exponential reward demonstrates smoother convergence and fewer reward fluctuations, indicating a more reliable gradient signal during policy learning.

\paragraph{Trajectory 3 (Helical Maneuver)}
The helical trajectory represents the most challenging scenario due to continuous curvature changes. All reward functions show slower and noisier convergence. Notably, the PQR reward achieves a high final success rate, yet tracking accuracy is significantly poorer, reinforcing that success rate alone cannot measure control quality.

% ---------- Figures: Training Convergence ----------

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj1_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj1_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj1_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Training convergence for Trajectory 1 (Vertical Ascent).}
\label{fig:train_traj1}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj2_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj2_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj2_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Training convergence for Trajectory 2 (Straight-Line Cruise).}
\label{fig:train_traj2}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj3_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj3_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{train_traj3_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Training convergence for Trajectory 3 (Helical Maneuver).}
\label{fig:train_traj3}
\end{figure}

% ============================================
% 5.2 Trajectory Tracking Performance
% ============================================

\subsection{Trajectory Tracking Performance}

Tracking performance for all three trajectories is shown in Figures~\ref{fig:path_traj1}--\ref{fig:path_traj3}. These visualizations reveal key behavioral differences between the reward functions.

\paragraph{Linear reward}
Consistently achieves moderate tracking performance but tends to permit larger deviations due to its mild gradient near the origin.

\paragraph{Exponential reward}
Produces the most precise tracking across all trajectories. Its steep exponential penalty strongly discourages tracking errors, resulting in tighter path following.

\paragraph{PQR reward}
Often yields unstable or overly cautious behavior, especially for complex trajectories. Although it may assign high final episode rewards (and thus high success rates), its actual tracking quality is inferior, particularly visible in the helical trajectory.

% ---------- Figures: Path Following ----------

\begin{figure}[htbp]
\centering
\begin{minipage}{0.29\textwidth}
    \includegraphics[width=\textwidth]{path_traj1_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{path_traj1_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.34\textwidth}
    \includegraphics[width=\textwidth]{path_traj1_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Tracking performance for Trajectory 1 (Vertical Ascent).}
\label{fig:path_traj1}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{path_traj2_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{path_traj2_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{path_traj2_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Tracking performance for Trajectory 2 (Straight-Line Cruise).}
\label{fig:path_traj2}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.30\textwidth}
    \includegraphics[width=\textwidth]{path_traj3_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{path_traj3_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.34\textwidth}
    \includegraphics[width=\textwidth]{path_traj3_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Tracking performance for Trajectory 3 (Helical Maneuver).}
\label{fig:path_traj3}
\end{figure}

% ============================================
% 5.3 Reward Distribution and Success Rates
% ============================================

\subsection{Reward Distribution and Success Rates}

Figures~\ref{fig:reward_dist_traj1}--\ref{fig:reward_dist_traj3} show the reward distribution of the final training episode for all reward functions.

The exponential reward exhibits a sharply concentrated distribution around small errors, consistent with accurate tracking. The linear reward produces a broader distribution due to its mild penalty. The PQR distribution is often irregular or heavily skewed, consistent with the instability observed in tracking.

Table~\ref{tab:success_rates} summarizes success rates and final episode rewards. As stated earlier, \emph{success rate should not be interpreted as a direct indicator of tracking accuracy}. For example:

- PQR yields the highest success rate in Trajectory 1 and 2  
- Yet its tracking is demonstrably poorer (Tables~\ref{tab:metrics_traj1}--\ref{tab:metrics_traj3})  
- Exponential reward yields the \emph{lowest} success rate  
- Yet it consistently delivers the \emph{best tracking accuracy}  

This distinction arises because success rate is normalized by the reward function’s own maximum achievable score.

% ---------- Figures: Reward Distributions ----------

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj1_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj1_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj1_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Reward distribution for the final episode of Trajectory 1.}
\label{fig:reward_dist_traj1}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj2_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj2_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj2_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Reward distribution for the final episode of Trajectory 2.}
\label{fig:reward_dist_traj2}
\end{figure}

\begin{figure}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj3_linear.pdf}
    \small{(a) Linear}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj3_exponential.pdf}
    \small{(b) Exponential}
\end{minipage}
\hfill
\begin{minipage}{0.32\textwidth}
    \includegraphics[width=\textwidth]{reward_dist_traj3_PQR.pdf}
    \small{(c) PQR}
\end{minipage}
\caption{Reward distribution for the final episode of Trajectory 3.}
\label{fig:reward_dist_traj3}
\end{figure}

% ---------- Table: Success Rates ----------

\begin{table}[htbp]
\centering
\caption{Success rates and reward statistics across all trajectories and reward functions.}
\label{tab:success_rates}
\begin{tabular}{p{6cm}ccc}
\toprule
\textbf{Metric} & \textbf{Linear} & \textbf{Exponential} & \textbf{PQR} \\
\midrule
Single-step reward range & [0,1] & [0,1] & [-8.9, 0.1] \\
\midrule
\textbf{Trajectory 1 (Vertical Ascent)} & & & \\
\multicolumn{4}{l}{\textit{Time steps:} 700} \\
\quad Maximum ideally achievable episode reward & 700 & 700 & 70 \\
\quad Final episode reward & 653.30 & 565.26 & 13.02 \\
\quad Success rate & 93.34\% & 80.75\% & 99.10\% \\
\midrule
\textbf{Trajectory 2 (Straight-Line Cruise)} & & & \\
\multicolumn{4}{l}{\textit{Time steps:} 7500} \\
\quad Maximum ideally achievable episode reward & 7500 & 7500 & 750 \\
\quad Final episode reward & 7016.77 & 6836.48 & 283.71 \\
\quad Success rate & 93.56\% & 91.15\% & 99.31\% \\
\midrule
\textbf{Trajectory 3 (Helical Maneuver)} & & & \\
\multicolumn{4}{l}{\textit{Time steps:} 3000} \\
\quad Maximum ideally achievable episode reward & 3000 & 3000 & 300 \\
\quad Final episode reward & 2680.61 & 2931.76 & -367.35 \\
\quad Success rate & 89.35\% & 97.73\% & 97.53\% \\
\botrule
\end{tabular}
\end{table}

% ============================================
% 5.4 Quantitative Tracking Metrics
% ============================================

\subsection{Quantitative Tracking Metrics}

Tables~\ref{tab:metrics_traj1}--\ref{tab:metrics_traj3} present detailed RMSE and maximum error values for position and velocity, as well as action smoothness.

Across all trajectories, the exponential reward achieves:

- the lowest position RMSE  
- the lowest velocity RMSE  
- the lowest maximum errors  
- competitive or superior control smoothness  

The linear reward performs reasonably well but less accurately, while the PQR reward suffers from substantial errors, confirming the visual findings from the path tracking figures.

% ---------- Tables: Metrics ----------

\begin{table}[htbp]
\centering
\caption{Performance metrics for Trajectory 1 (Vertical Ascent).}
\label{tab:metrics_traj1}
\begin{tabular}{p{4cm}ccc}
\toprule
\textbf{Metric} & \textbf{Linear} & \textbf{Exponential} & \textbf{PQR} \\
\midrule
\multicolumn{4}{l}{\textit{Position RMSE (m)}} \\
$\text{RMSE}_x$ & 0.0744 & \textbf{0.0377} & 0.1403 \\
$\text{RMSE}_y$ & 0.1311 & \textbf{0.0308} & 0.1607 \\
$\text{RMSE}_z$ & 0.1416 & \textbf{0.0801} & 0.1630 \\
$\text{RMSE}_{\text{pos}}$ & 0.2068 & \textbf{0.0937} & 0.2685 \\
\midrule
\multicolumn{4}{l}{\textit{Velocity RMSE (m/s)}} \\
$\text{RMSE}_{v_x}$ & 0.1316 & \textbf{0.0812} & 0.1857 \\
$\text{RMSE}_{v_y}$ & 0.1058 & \textbf{0.0229} & 0.1898 \\
$\text{RMSE}_{v_z}$ & 0.1913 & 0.1638 & \textbf{0.1421} \\
$\text{RMSE}_{\text{vel}}$ & 0.2552 & \textbf{0.1842} & 0.3012 \\
\midrule
\multicolumn{4}{l}{\textit{Maximum Errors}} \\
$e_{\max,\text{pos}}$ (m) & 0.3456 & \textbf{0.1576} & 0.7299 \\
$e_{\max,\text{vel}}$ (m/s) & 1.1547 & 1.1037 & \textbf{0.7387} \\
\midrule
\multicolumn{4}{l}{\textit{Control Effort Smoothness}} \\
$\sigma_{\text{actions}}$ (m/s$^{2}$) & 0.4922 & 1.3598 & \textbf{0.3394} \\
\botrule
\end{tabular}
\end{table}

\begin{table}[htbp]
\centering
\caption{Performance metrics for Trajectory 2 (Straight-Line Cruise).}
\label{tab:metrics_traj2}
\begin{tabular}{p{4cm}ccc}
\toprule
\textbf{Metric} & \textbf{Linear} & \textbf{Exponential} & \textbf{PQR} \\
\midrule
\multicolumn{4}{l}{\textit{Position RMSE (m)}} \\
$\text{RMSE}_x$ & 0.2577 & \textbf{0.0370} & 0.2001 \\
$\text{RMSE}_y$ & 0.1932 & \textbf{0.0374} & 0.2279 \\
$\text{RMSE}_z$ & 0.1442 & \textbf{0.0325} & 0.1454 \\
$\text{RMSE}_{\text{pos}}$ & 0.3529 & \textbf{0.0618} & 0.3363 \\
\midrule
\multicolumn{4}{l}{\textit{Velocity RMSE (m/s)}} \\
$\text{RMSE}_{v_x}$ & 0.0573 & \textbf{0.0311} & 0.0529 \\
$\text{RMSE}_{v_y}$ & 0.0328 & \textbf{0.0135} & 0.0752 \\
$\text{RMSE}_{v_z}$ & 0.0466 & \textbf{0.0092} & 0.0527 \\
$\text{RMSE}_{\text{vel}}$ & 0.0808 & \textbf{0.0351} & 0.1059 \\
\midrule
\multicolumn{4}{l}{\textit{Maximum Errors}} \\
$e_{\max,\text{pos}}$ (m) & 0.6623 & \textbf{0.1472} & 0.9308 \\
$e_{\max,\text{vel}}$ (m/s) & 0.3696 & 0.8058 & \textbf{0.3481} \\
\midrule
\multicolumn{4}{l}{\textit{Control Effort Smoothness}} \\
$\sigma_{\text{actions}}$ (m/s$^{2}$) & 0.1135 & 0.1834 & \textbf{0.0531} \\
\botrule
\end{tabular}
\end{table}

\begin{table}[htbp]
\centering
\caption{Performance metrics for Trajectory 3 (Helical Maneuver).}
\label{tab:metrics_traj3}
\begin{tabular}{p{4cm}ccc}
\toprule
\textbf{Metric} & \textbf{Linear} & \textbf{Exponential} & \textbf{PQR} \\
\midrule
\multicolumn{4}{l}{\textit{Position RMSE (m)}} \\
$\text{RMSE}_x$ & 0.3113 & \textbf{0.0651} & 0.3669 \\
$\text{RMSE}_y$ & 0.2297 & \textbf{0.0518} & 0.4430 \\
$\text{RMSE}_z$ & 0.2105 & \textbf{0.0395} & 0.2114 \\
$\text{RMSE}_{\text{pos}}$ & 0.4404 & \textbf{0.0921} & 0.6128 \\
\midrule
\multicolumn{4}{l}{\textit{Velocity RMSE (m/s)}} \\
$\text{RMSE}_{v_x}$ & 0.1891 & \textbf{0.0413} & 0.0875 \\
$\text{RMSE}_{v_y}$ & 0.1779 & \textbf{0.0359} & 0.2347 \\
$\text{RMSE}_{v_z}$ & 0.1819 & \textbf{0.0209} & 0.0815 \\
$\text{RMSE}_{\text{vel}}$ & 0.3170 & \textbf{0.0586} & 0.2634 \\
\midrule
\multicolumn{4}{l}{\textit{Maximum Errors}} \\
$e_{\max,\text{pos}}$ (m) & 0.8959 & \textbf{0.1567} & 1.1140 \\
$e_{\max,\text{vel}}$ (m/s) & 1.1423 & \textbf{0.1561} & 0.4895 \\
\midrule
\multicolumn{4}{l}{\textit{Control Effort Smoothness}} \\
$\sigma_{\text{actions}}$ (m/s$^{2}$) & 0.1639 & \textbf{0.0659} & 0.1183 \\
\botrule
\end{tabular}
\end{table}

% ============================================
% End Section 5
% ============================================







\section{Conclusion}\label{sec:conclusion}

This study investigated the influence of reward function design on the performance of a TD3 agent in three–dimensional path–following control. Three representative reward structures—linear, exponential, and piecewise quadratic (PQR)—were evaluated under identical training conditions across vertical, cruise, and helical trajectories. Performance was assessed using position and velocity RMSE, reward statistics, and control smoothness.

The results show that the \textbf{exponential reward function consistently delivered the highest tracking accuracy} across all trajectories. Relative to the linear reward, it achieved a reduction of approximately \textbf{55--83\%} in position RMSE and \textbf{28--82\%} in velocity RMSE. Compared to the PQR reward, the exponential formulation reduced position RMSE by \textbf{73--76\%} and velocity RMSE by \textbf{58--80\%}. These improvements highlight the benefits of smooth, rapidly decaying error penalization and the strong gradient information provided by the exponential structure.

While the exponential reward produced lower accumulated episode returns than the linear and PQR rewards, this outcome reflects its stricter penalization of imprecise tracking rather than inferior control performance. The linear reward yielded stable yet moderately accurate tracking, whereas the PQR reward exhibited the weakest performance due to non-smooth transitions in its structure that adversely affected policy optimization.

The experimental findings confirm that \textbf{reward shaping is a central determinant of learning efficiency and tracking quality in reinforcement-learning-based control}. By isolating reward formulations under a unified algorithmic and training framework, this work provides clear evidence that small differences in reward structure can lead to substantial performance disparities.

The control plant considered in this study was a three–dimensional double-integrator model, selected deliberately to isolate the effect of reward design without confounding factors from higher-order nonlinear dynamics. The insights derived from the comparative evaluation, however, are not limited to this simplified representation. The fundamental error–sensitivity properties that distinguish the reward formulations are expected to generalize to more complex robotic platforms, including full quadrotor and underwater-vehicle dynamics.

Overall, the exponential reward emerged as the most effective formulation among those tested, offering superior accuracy, smoother control inputs, and consistent performance across trajectories of varying complexity. These results provide practical guidance for practitioners seeking principled reward-design strategies in continuous-control reinforcement-learning applications.



\section{Future Work}\label{sec:future_work}

This study focused on reward-function evaluation using a 3D double-integrator model, which isolates the effect of reward design without the confounding influence of complex vehicle dynamics. Future work may extend the analysis to higher-order and nonlinear systems such as full quadrotor dynamics, underwater vehicles, or other robotic systems to assess whether the observed trends in reward sensitivity generalize to more realistic dynamic regimes.

Another natural direction is to incorporate external disturbances, actuator limits, wind fields, and sensor noise to evaluate the robustness of each reward formulation under real-world uncertainties. Integrating safety layers—such as control barrier functions or model predictive safety filters—could further ensure constraint satisfaction during training and deployment.

The methodology may also be expanded to multi-agent scenarios, including cooperative formation flight, leader–follower tracking, and distributed path following. Finally, reward-discovery techniques such as inverse reinforcement learning, automatic reward shaping, or meta-learned reward structures represent promising avenues for designing more principled and task-adaptive reward functions in continuous-control domains.






\section*{Declarations}

\subsection*{Funding}
The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

\subsection*{Competing Interests}
The authors have no relevant financial or non-financial interests to disclose.

\subsection*{Author Contributions}
All authors contributed to the conceptual development, methodology design, experimental implementation, data analysis, and preparation of the manuscript. 
The first author conducted the simulations, developed the reinforcement learning framework, and performed the quantitative evaluations. The co-authors provided supervision, guidance on research design, critical feedback on analysis, and revisions to the manuscript. All authors have read and approved the final version of the manuscript.

\subsection*{Ethical Approval}
The authors state that this research complies with applicable ethical standards.

\subsection*{Data Availability Statement}
The data supporting the findings of this study are available from the corresponding author upon reasonable request.




\bibliography{sn-bibliography}% common bib file

%% if required, the content of .bbl file can be included here once bbl is generated


\end{document}
