Non-Terrestrial Networks (NTNs) have become the quintessence of sixth-generation (6G) wireless networks, providing everywhere global coverage through connectivity with satellite, aerial, and terrestrial infrastructures. Among them, Unmanned Aerial Vehicles (UAVs) possess a unique value because they are easy to deploy quickly, highly maneuverable, and can establish line-of-sight links with ground users in time-varying or coverage-constrained scenarios [1]–[3]. Being part of NTN architectures, UAVs are expected to facilitate mission-critical services such as emergency communication, remote sensing, and rural broadband access, particularly in the case that they are backhauled using Low Earth Orbit (LEO) satellites [4], [5]. However, achieving predictable and energy-efficient UAV operation within NTNs is faced with a variety of challenges. UAVs must navigate 3D space intelligently to maintain optimal communication connections in the face of constrained onboard energy, user mobility, and wireless dynamic environments [6]. Traditional rule-based control or static optimization methods are bound to fail in such scenarios because they cannot adapt quickly to time-varying network conditions and user demands [7]. To surmount these constraints, recent research works have explored the possibility of applying Deep Reinforcement Learning (DRL) for autonomous UAV decision-making. DRL enables agents to learn the best policy by taking advantage of trial-and-error interactions with the environment without models [8]. Among the different DRL algorithms, Proximal Policy Optimization (PPO) is noted for its performance vs. stability trade-off and has specifically proven beneficial for continuous control problems such as UAV navigation and resource allocation [9], [10]. While effort has been made with DRL for UAV trajectory planning optimization [11], power control [12], or task scheduling [13] individually, few studies have focused on the integrated learning framework that can jointly optimize mobility and transmission power in NTN environments. Moreover, integrating satellite backhaul dynamics into learning was extremely under-explored. This paper introduces a novel PPO-based DRL framework to enable a UAV to learn a trajectory and transmission power control policy simultaneously in a LEO-assisted NTN system. The UAV agent is trained in a tailor-made simulation environment, which mimics user positions, SNR conditions, and energy consumption. A tailor-made reward function is utilized to guide the agent towards optimal communication performance and energy efficiency. Our test results confirm the agent's ability to learn stable, optimal policies that maximize SNR transmission and conserve energy compared to baseline approaches. Through solving the common optimization problem under a realistic NTN setting, this paper proposes a realistic and intelligent solution for the future 6G UAV deployment.

The accelerated evolution of UAVs in 6G Non-Terrestrial Networks (NTNs) has prompted an upsurge of research exploring intelligent flight control and communication optimization methods. Although initial solutions depended mostly on deterministic optimization or rule-based control, current research has shifted toward learning-based frameworks that can provide flexibility under dynamic and partially observable settings [13]. Among the most promising is Deep Reinforcement Learning (DRL), particularly for optimization of complex trajectory problems. For example, Wang et al. [14] applied DRL to suggest 3D navigation policies with respect to user distribution and link quality in UAV-based 6G access networks. Likewise, Gu et al. [15] employed a multi-agent DRL model to co-learn UAV trajectory and Reconfigurable Intelligent Surface (RIS) configuration on THz bands. These works confirm the applicability of DRL in managing UAV mobility in dense and hybrid NTN scenarios. Power optimization has also been explored in addition to mobility. A DRL-injected framework was put forward in [16] for managing UAV transmit power in relation to residual battery energy and communication QoS requirements. On the contrary, Smith and Kumar [17] used a PPO-based method for energy harvesting UAVs with prolonged lifetime without degrading SNR. These works show growing care about sustainable UAV use. Furthermore, radio resource and bandwidth management feasibility is demonstrated in DRL. In [18], joint trajectory and spectrum optimization algorithm for UAVs in ISAC systems was presented by the authors, while [19] included beam steering with backhaul-constrained backhaul. Liu et al. [20] addressed real-time bandwidth slicing in satellite-attached UAV swarms using DRL with latency minimization and throughput balancing as the priority. Under uncertainty and limited feedback in NTNs, transfer learning and meta-learning approaches have become the go-to. Ince et al. [21] promoted a few-shot offline DRL framework to empower UAVs to generalize to novel territory and tasks with extremely few training samples. Concurrently, Zhang et al. [22] proposed a reinforcement learning methodology with quick adaptation of policy for mission-critical use cases. Resilience and trust are also key design challenges in NTN. Blockchain-based DRL was introduced in [23] to provide secure UAV-based data transmission in vehicle networks. The system provides decentralized learning with immutable logging, which facilitates easier integrity across several agents. Multi-agent reinforcement learning (MARL) has played a key role in swarm and cooperative UAV systems. As discussed in [24], coordinated swarming using DRL achieves real-time task redistribution and collision avoidance. Moreover, [25] investigated the use of distributed PPO agents for cooperative area coverage with impressive scalability in multi-UAV operations. RIS-assisted UAV systems have also been investigated in the studies. Joint design of UAV position and RIS reconfiguration was discussed in [26], while [27] has suggested a dual-timescale learning framework that is adaptive to the channel state and population of users. These works emphasize the significance of cooperation between UAV-RIS for interference minimization and optimization of spectral efficiency. There are certain surveys and systematic reviews that have outlined these developments. Wu et al. [28] indicated the combination of DRL and AI for UAV communications, challenges in generalization, security, and reward shaping. Similarly, Li et al. [29] provided a taxonomy of RIS-assisted UAV communication methods, such as optimization objectives, learning paradigms, and deployment models. A survey on autonomous UAV control algorithms by Zeng et al. [30] followed the evolution of DRL algorithms and revealed open problems in joint power-trajectory scheduling, particularly under the consideration of backhaul constraints. Irrespective of these advances, most evaluation decouples mobility or power control and tends to ignore joint optimization in satellite-connected NTN cases. More importantly, they are based on very naive channel models or neglect energy-SNR tradeoffs that are crucial in practical deployments. To bridge this gap, our research introduces one PPO-based DRL framework learning the trajectory and transmission power policies jointly in a real NTN simulation environment. The agent can learn responding to spatial dynamics and communication requirements cumulatively using reward shaping and environmental cues and shows a scalable and energy-efficient solution for future 6G networks.

This section outlines the complete configuration utilized to train and test a UAV agent to optimize autonomous trajectory and transmission power within an NTN environment with Deep Reinforcement Learning (DRL).

It consists of three main parts: a customized-designed simulated environment, a Proximal Policy Optimization (PPO)-based deep reinforcement learning (DRL) agent, and a reward mechanism playing the role of UAV trajectory management and transmission power control simultaneously. The simulated environment mimics UAV-user dynamics, returning state observations and implementing actions chosen by the agent. The PPO agent is learned to discover an optimal policy from actions to states by maximizing the total reward. The mechanism of the reward balances the agent's decision of the action with respect to communication performance and energy consumption. In this configuration, the UAV is thought to be having a continuous high-rate backhaul link via a Low Earth Orbit (LEO) satellite connection in line with future 6G Non-Terrestrial Network (NTN) infrastructure. This satellite link enables the UAV to gather information from the global network and transmit it to ground users in real time, hence acting as a mobile airborne base station. This setup is particularly useful to extend wireless coverage in sparsely covered or infrastructurally poor areas.

The global dynamics of the system are illustrated in Fig. 1, which illustrates the communication between the UAV, the user, the learning agent, the satellite link, and the reward engine.

World with two dimensions in which a UAV can move continuously along the x–y plane. A single ground user is placed randomly in the environment for each episode. The UAV's world observation space consists of six continuous features: current location coordinates and transmission capacity of the UAV and position and communication request of the user. This setting is constructed based on the OpenAI Gym setting to be interfaced with the common reinforcement learning interfaces. Additionally, it follows the path of movement of the UAV per episode and utilizes it for the evaluation as well as the visualization of the derived learned policy. The reward mechanism is in synchronization with the environment to assist the UAV in optimizing not only its path but also power consumption. The general system workflow, i.e., the communication between UAV, user, agent, and reward calculation engine, is illustrated in Fig. 1.

The agent would learn to execute three continuous actions in each time step: horizontal movement (Δx), vertical movement (Δy), and transmission power. The transmission power is a one-dimensional continuous scalar number ranging from 0 to 1 and indicates the energy consumption of the UAV when communicating. These three control variables together form the action vector, which is given as at=[Δx,Δy,P_t], where at denotes the agent's action in time step t. This continuous action space allows the UAV to have immense control over its path and transmission plan, making it easier for it to learn and improve in the NTN environment.

At each time step, the UAV agent receives a state vector that represents the current environment conditions. This state includes the UAV’s current position in two-dimensional space (x,y), its transmission power level, the user’s position (xu,yu), and the user’s communication demand. Together, these variables form a 6-dimensional continuous state vector that is fed into the learning agent to inform its decision-making process.

The reward function is constructed so that high-quality communication and energy efficiency are promoted at the same time. In particular, the reward RRR at any given time step is computed on the basis of the signal-to-noise ratio (SNR) as well as the transmission power employed by the UAV. Algebraically, it can be represented as:

Where d is the distance between the UAV and the user, and α, β are reward weights controlling the trade-off.

We employ Stable-Baselines3's Proximal Policy Optimization (PPO) because it is stable in continuous action space. Our agent is trained to 100,000 timesteps with the following setup:

Certain performance measures are used in identifying the performance of the suggested DRL-based UAV control system. To begin with, a learning curve is plotted by computing the cumulative reward gained at every step of training that reflects the learning ability of the agent and convergence towards the best policy. Second, UAV path plots are plotted to determine flight path of agents through evaluation episodes and compute UAV movement in the world from user position. Third, histograms of action distributions are plotted to analyze frequency and direction of selected actions, i.e., movement directions and power levels. Lastly, overall performance of the agent is tested by computing average reward over 10 independent test episodes. Together, these factors paint a complete picture of the agent's learning curve, action direction, and decision-making capacity in dynamic NTN environments.

To ensure the efficacy of the proposed DRL-based UAV control system, we conducted a set of training and test experiments. Results validate the agent learning process, policy behavior, and adaptation within the simulated NTN environment. Three distinguishable visualizations were attained: reward learning curve, UAV trajectory plot, and action distribution histogram. Each of them is discussed in the subsequent subsections.

The learning curve, shown in Fig. 2, provides the cumulative reward earned for each training step for 100,000 timesteps. The agent starts with erratic action with fluctuating rewards, typical of initial DRL training. Upon further training, the smoothed reward curve shows a steady increase, informing us that the agent is learning progressively at its actions. The curve eventually stabilizes as it becomes flat, showing that there is convergence to a loose optimal policy. The increasing and eventually stable reward trend reflects the agent's power efficient SNR improvement capability in the long term.

Motion movement of the test UAV is depicted in Fig. 3, where the 2D path of the UAV from any initial position to a specified user position is plotted. The path shows a structured and guided pattern of movement, as proof that the trained agent can move in the space with a goal-directed type of movement. The trajectory begins from exploratory behavior and then drifts toward the user's position and arrives there. The green box denotes the beginning, the red cross the user's location, and the orange diamond the destination of the UAV. The output confirms that the agent has learned to identify the spatial relationship of itself in relation to the user and therefore fly in energy-efficient and goal-directed manner.

To gain better insights into the learned policy of the agent, we analyzed action distribution across course-of-evaluation episodes. Histograms for horizontal displacement (Δx), vertical displacement (Δy), and power transfer are shown in Fig. 4. The histogram for Δx indicates the agent is moving nearly entirely small quantities to the right, and the histogram for Δy indicates downward movement strongly—both in agreement with the agent attempting to move toward the user. The transmission power histogram indicates the agent is choosing high transmission power values with high frequency, which indicates that it is trying to maximize SNR. There is also a trade-off in terms of energy consumption, which in the future can be overcome by adjusting the power penalty term in the reward function. These distributions together guarantee that the agent has passed the random action stage and arrived at a stable, goal-seeking policy.

To estimate policy performance quantitatively, we performed 10 independent episodes of evaluation and computed the average total reward over an episode. The results show that the agent consistently performs high values of reward, with its average converging to approximately 870. This confirms that the learned policy generalizes well for different user locations and is also resilient to environmental changes. The agent demonstrates efficacy and stability in finding the trade-off between communication performance and mobility.

To further ensure learning stability, Fig. 5 shows the cumulative reward curve across 100 training episodes. The curve is smooth and continuously increasing, indicating that the agent consistently receives greater cumulative rewards as it continues training. This pattern confirms that the policy learned does not suffer from reward collapse or oscillation and the UAV increasingly possesses continuous high-quality communication and energy-conserving mobility. The cumulative reward graph provides an overall vision of performance consistency in aggregate.

In addition to the visual results, a qualitative summary of system performance is offered in the form of a key set of evaluation metrics provided in Table I. The metrics capture the learning efficiency of the agent, behavior of actions, power usage, and convergence stability in training and testing environments. The metrics indicate the UAV agent's ability to learn an energy-aware, goal-oriented policy that generalizes over different scenarios. The results confirm the stability and robustness of the suggested DRL algorithm in the NTN context.

Table I — Performance Summary of DRL-Based UAV Control Framework

Taking advantage of the promising results of this work, some useful lines of future research are envisioned. To start with, expanding the single-UAV deployment of this work to multi-UAV cooperative scenarios can bring scalability and resilience to large-scale deployments of NTNs. Multi-agent reinforcement learning (MARL) techniques can be explored to manage UAV swarms for cooperative coverage, load balancing, and interference management for this end. Second, the existing model today relies on static user positions; future research should consider user mobility patterns and develop dynamic policies that can sense dynamically the real-time variation of user distribution. Online learning or meta-reinforcement learning can be implemented to enable the UAV agent to learn more generalized reactions to novel environments or rapidly changing situations. Third, integration with RIS and beamforming technology can provide further gains in energy efficiency and link quality. Moreover, use of federated learning together with DRL can allow UAVs to learn collaboratively with no data privacy compromise and low overhead in communication. Finally, experimentation in the real world and validation via testbeds or digital twin testbeds need to be carried out to bridge the loop from simulation to actual deployment regarding ensuring reliability, latency performance, and security compliance for operating 6G NTN systems.

In this article, a Proximal Policy Optimization (PPO)-based deep reinforcement learning (DRL) system has been presented for co-managed UAV trajectory control and transmission power control of Non-Terrestrial Networks (NTNs). By a custom-designed simulation environment and an optimally designed reward function, the UAV has learned to fly autonomously and provide energy-efficient communication under various scenarios. The system learned to achieve informed trade-offs between SNR gain and energy efficiency and adapt reasonably at user positions without prior knowledge of the environment. Simulation results, comprising action distributions, trajectory visualizations, and reward convergence, validate the performance and stability of the learned policy. This solution confirms the viability of employing AI-powered UAVs in next-generation 6G networks to improve coverage and energy and spectral efficiency in challenging environments. More research can be directed at extending this framework to multi-UAV networks, with user mobility taken into account, or merging it with RIS-assisted communication and federated learning to further enhance the scalability and robustness of networks.