The structure of the OAM transformer and its working principle are illustrated in Fig. 2a. It consists of three diffractive layers, operating at 0.3 THz (corresponding to a wavelength of 1 mm). Each layer is spatially encoded with 60 × 60 diffractive neurons, with a pixel size of 1 mm, and the diffractive layers are cascaded along the propagation direction with an interval length of 60 mm, modulating the phase of light as it propagates through them at a resolution of 1 mm. These layers are connected by the diffraction process, often mathematically described using the Rayleigh-Sommerfeld (RS) diffraction theory. According to this, each diffractive neuron of these layers can be considered as a secondary spherical wave source with different phases, and its radiation weight factor is:

where λ is the wavelength, represents the distance between the 2 points and , on the rear and front planes of the diffraction path. Then, these radiated waves combine to generate a new field, known as the diffracted field.

The OAM phase modulator works as a phase mask to spatially modulate the Gaussian THz incident waves (see more details in the model of Methods), with the same spatial size as the diffractive layers. In this way, it can generate the input OAM beam in front of the first diffractive layer. The separation between the OAM phase modulator and the first diffractive layer is 60 mm. Behind the last diffractive layer is the output plane, where the output OAM field can be detected, with the field of view (FoV) being 36 mm. The distance between the last diffractive layer and the output plane is also 60 mm.

The OAM transformer can convert a set of input OAM modes to another set of output OAM modes. For example, here we design an OAM transformer to implement OAM multiplication (shown in Fig. 2b). A set of input modes l_in = {-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5} is transformed to another set of output modes l_out = {-10, -8, -6, -4, -2, 0, 2, 4, 6, 8, 10}, satisfying the multiplication relation: l_out = 2 × l_in. The output modes exhibit high efficiency and Pearson correlation coefficient (PCC) values (see definitions in the training of Methods), suggesting good mode transformation performance.

To physically achieve the OAM transformer, we employ an AI-based framework to optimize the phase parameters of diffractive layers, which utilizes automatic gradient descent and error backpropagation algorithms (see more details in the training of Methods). The first step in establishing this framework is to define the loss function, which can mathematically describe the relationship between the output and the target THz fields (see the ground truth of output OAM modes with various TC values in Note I of the Supplementary Information). This function should minimize the difference between the two, while maintaining a relatively high output efficiency (see the detailed definition of loss functions in the training of Methods). The second step is mathematically describing the connection between input and output fields. We already know that the RS diffraction theory can achieve this. Still, practically, we calculate this using the angular spectrum method (ASM, see more details in the model of Methods), which is mathematically equivalent to the RS diffraction theory [33]. In this way, the entire optimization framework is established, and the final updated phases of the diffractive layers are depicted in Fig. 2c. Notably, using this framework, the OAM modes can be designed and transformed arbitrarily. Another OAM transformer that performs OAM division is demonstrated in Note II of the Supplementary Information.

The phase profiles of the OAM phase modulators with various TC values are shown in Fig. 2d (see the mathematical definition in the model of Methods), which can generate the input OAM beams for the OAM transformer, as illustrated in Fig. 2e. The intensity and phase distributions clearly show the features of OAM modes. These input OAM beams then pass through the diffractive layers and form the converted OAM modes on the FoV of the output plane (shown in Fig. 2f). The characteristics of output OAM modes are clearly observed, with a high average PCC value of 0.963 between the output modes and their target output.

The schematic of the OAM filter can also refer to Fig. 2a, but the function of the OAM filter is different from that of the OAM transformer. As shown in Fig. 3a, for a set of input OAM modes l_in = {-6, -5, -4, -3, -2, -1, 1, 2, 3, 4, 5, 6}, several modes l_in1 = {-5, -3, -1, 2, 4, 6} can pass through the OAM filter (with high efficiency and PCC values), while other modes l_in2 = {-6, -4, -2, 1, 3, 5} are filtered (with low efficiency and PCC values). We note that the passed or filtered modes can also be arbitrarily designed using the optimization framework. To realize the filtering feature, the loss function should be well-defined during the training process. The loss function of the passed modes is the same as that of the OAM transformer. By contrast, the loss function of the filtered modes can minimize the correlation between the output modes and their corresponding ground truth, while simultaneously maintaining a low output efficiency, thus achieving the desired filtering results (see the detailed loss functions in the training of Methods). As the loss function is defined, the entire OAM filter is well-designed, and the final optimized phase profiles are demonstrated in Fig. 3b.

The phase profiles of OAM phase modulators are shown in Fig. 3c, along with their corresponding input OAM beams in Fig. 3d. The output OAM modes on the FoV of the output plane are depicted in Fig. 3e. For the passed modes, the feature of OAM with different TC values is clearly observed, with a high average PCC value of 0.983. In contrast, the filter OAM beams cannot maintain the OAM feature (with a low average PCC value of merely 0.023). The phase distributions are irregular, and the output intensities are very weak. As a result, the function of the OAM filter is validated.

Although the function of the OAM processor has been demonstrated through numerical simulation, a THz experiment is still required to verify it. The setup of the THz field scanning experiment is demonstrated in Fig. 4a, along with its corresponding schematic diagram in Fig. 4b. Using this experimental setup, the THz field distributions at the output plane (including both intensity and phase) at 0.3 THz within the FoV can be well measured. More details can be found in the experimental setup of Methods.

The optimized phase profiles and their corresponding fabricated samples of the OAM transformer and filter are shown in Figs. 4c and 4d. To reduce the complexity of the experiment, such as the alignment of the fabricated diffractive layers, we use only 2 layers to build the experimental DONN, instead of 3 layers in the simulation. During the training process of the experimental model, a vaccination strategy is also incorporated to help resist the potential misalignment of these fabricated samples (see more details in the model of Methods). The fabricated OAM phase modulators and their encoded phase profiles are depicted in Fig. 4e. All the samples are manufactured using the 3D printing technique with a high-temperature resin material. The height of each unit cell in the samples varies, as encoded by the required phase profiles (see more details in the sample fabrication of Methods).

The performance analysis of the experimental OAM transformer is shown in Fig. 5a. The 3 input OAM modes l_mod = {-2, 0, 2}, also denoting 3 multiplexed channels CH 1, 2, and 3, are converted to 3 altered output OAM modes l_out = {-1, 0, 1}, with high output efficiency and PCC values. The simulated and measured field distributions are demonstrated in Figs. 5c and 5e. For all 3 output modes, the OAM profiles are clearly demonstrated, possessing a high average PCC value of 0.971 (simulated) and 0.761 (measured). Thus, it experimentally demonstrates a good OAM transformation performance.

For the experimental OAM filter, the performance analysis is demonstrated in Fig. 5b. The 4 input OAM modes l_mod = {-2, -1, 1, 2} can represent 4 multiplexed channels: CH 1, CH 2, CH 3, and CH 4. The CH 1 and CH 2 (l_mod1 = {-2, -1}) are filtered, with low output efficiency and PCC values. By contrast, the CH 3 and CH 4 (l_mod2 = {1, 2}) can pass through the experimental OAM filter, possessing high efficiency and PCC values. Specifically, the output simulated and measured field distributions are shown in Figs. 5d and 5f. For CH 1 and CH 2, the output fields exhibit chaotic behavior with a low intensity, indicating the OAM modes are filtered (with average simulated and measured PCC values of 0.031 and 0.164, respectively). By contrast, for CH 3 and CH 4, the OAM profiles remain, retaining their high intensity. It suggests that these OAM modes are reserved, with a high average PCC value of 0.982 (simulated) and 0.763 (measured). These results experimentally verify the OAM filtering function.

Next, we employ the THz wireless communication experiment to demonstrate the OAM processor’s applications in THz MDM wireless links. The photograph of the experimental setup is shown in Fig. 6a, along with the corresponding schematic diagram in Fig. 6b. This experimental setup can be applied to validate the wireless communication performance of the OAM processors. In this setup, the wireless signal is modulated using the 16-state quadrature amplitude modulation (16QAM) format, with a symbol rate of 15 GBaud, thereby achieving a communication speed of 60 Gbps per channel. More details can be found in the experimental setup of Methods.

The wireless communication performance of the OAM transformer is shown in Fig. 7a. We first demonstrate the demodulated field distributions, including both simulated and measured results. The experimental setup for the field distribution measurement is also depicted in Figs. 5a and 5b; however, an OAM phase demodulator is incorporated between the last diffractive layer and the Rx probe, with the same 60 mm interval distance. We note that these OAM phase demodulators are identical to the modulators with corresponding TC values (see Fig. 4e). The mode feature of the received demodulated field depends on the mode of the output field (see Figs. 5e and 5f) and the OAM phase demodulator, described as:

where l_Rx, l_out, and l_demod are the TC values of the demodulated field, output field, and the OAM phase demodulator. To establish normal wireless links, l_Rx should be 0, i.e., the fundamental mode. We observe the fundamental mode on the Rx plane for all 3 channels of the OAM transformer, characterized by a bright spot with a uniform phase profile in the center of the demodulated field. Then, we utilize the communication experiment setup, as shown in Fig. 6, to evaluate the communication performance of the OAM transformer. The communication performance is reflected in a constellation diagram and an eye diagram (quadrature). Data scatters in these diagrams can be distinguished, with error vector magnitude (EVM) values of less than 13.5%, fulfilling the requirement for 5G communication [34].

The OAM filter’s communication performance is demonstrated in Fig. 7b. As expected, for the filtered channels (CH 1 and 2), no regular fields are detected on the Rx plane, and the intensities are very weak. The data scatters in the constellation and eye diagrams cannot be distinguished, with high EVM values exceeding 37%, which suggests that the wireless links have not been established. By contrast, for the reserved channels (CH 3 and CH 4), the fundamental mode is observed on the Rx plane. The data scatters in the constellation and eye diagrams can be clearly distinguished, with low EVM values of under 13.5%, indicating good wireless communication performance. Therefore, the different communication performance between filtered and reserved channels verifies the filtering function enabled by the OAM filter in THz MDM wireless links.

We select 3 channels of the OAM transformer and filter (i.e., OAM transformer’s CH 1, OAM filter’s CH 2, and CH 4) to demonstrate the channel isolation property experimentally. Each channel is tested by OAM phase demodulators used in its corresponding and adjacent channels. The testing results of the OAM transformer’s CH1 are shown in Fig. 8a. Only for the fundamental mode (l_Rx = 0, i.e., l_demod = -l_out = 1), the communication performance is satisfactory. For other higher Rx modes (see Eq. 2), the transmitted data cannot be well received, so the channel is isolated. This is attributed to the weaker intensities in the center of the Rx plane, where the Rx horn is placed. The vortex field shape of OAM beams exactly causes such intensity distributions. This feature makes OAM beams suitable for MDM wireless links, since each channel can only be demodulated by a certain OAM demodulator. For the OAM filter, the testing results of CH 2 are shown in Fig. 8b. Regardless of the different OAM phase demodulators, wireless links cannot be established, since the OAM fields have been filtered, which cannot support any wireless communication. By contrast, the OAM mode is reserved in CH 4 of the OAM filter, which is also tested using various OAM phase demodulators (shown in Fig. 8c). As expected, only the fundamental mode (with l_demod = -2) on the Rx plane can be successfully detected by the Rx horn, while the data carried by other higher Rx modes cannot be well received. Therefore, we conclude that only one corresponding OAM demodulator can be used in a given channel to establish a wireless link with good communication quality (except for the filtered channels). In other words, the same OAM demodulator cannot be employed in different channels. Such a characteristic reflects the channel isolation property, as each channel can only normally work with its own one-to-one match of the OAM demodulator. The OAM demodulators used in other channels cannot establish a high-quality wireless link for this channel (i.e., being isolated).

The input Gaussian beam has a uniform phase profile and an amplitude profile that follows the Gaussian function, mathematically described as:

where (ρ, θ) denotes the location of the polar coordinate system, E₀ is the maximum amplitude that is set to 1, and ω₀ = 30 mm represents the beam waist radius of the Gaussian beam.

The OAM phase modulators can generate l-th-order OAM beams with axicon phase profiles:

where l denotes the mode value (i.e., TC value) of the OAM beam, λ = 1 mm is the working wavelength, and β = 5° is the convergence angle. Specifically, these generated OAM beams are Bessel-like beams carrying various modes of OAM, with the merit of being theoretically diffraction-free during propagation. Thus, we adopt these axicon phase profiles to construct the OAM modulators.

The DONN model is composed of cascaded diffractive layers with transmittance of:

where ϕ (x, y) is the phase of diffractive neurons, and (x, y) is the location of the Cartesian coordinate system. We assume that these layers only modulate the phase of the THz field, i.e., ignoring the material loss. The propagation of THz waves between the layers is modeled using ASM. According to this, a THz field u (x, y) after propagating a distance d along the optical axis z can be expressed as:

where () represents the 2D Fourier (inverse Fourier) transform and H (f_x, f_y; d) is the free-space propagation transfer function, which is defined as:

where f_x (f_y) represents the spatial frequency along the x- (y-) direction. In our numerical simulation, the 2D Fourier (inverse Fourier) transforms are implemented by the Fast Fourier Transform (FFT) algorithm.

For the experimental model, we employ a vaccination strategy to help mitigate possible misalignment of OAM modulators and diffractive surfaces. These inaccuracies are regarded as random 2D displacements (D_x,y) on the x-y plane, following the uniform (U) random distribution:

where p = 1 mm is the pixel size of the OAM modulators and diffractive surfaces. These random displacements are incorporated in the model training stage to help resist potential misalignment. In this way, after training with this strategy, the alignment tolerance is ± 0.5 mm, i.e., = 0.5 mm.

The phase parameters of the DONNs are optimized by minimizing the loss function, defined as:

where O is the output complex field, O_gt is the ground truth of the output field (see details in Note I of the Supplementary Information), I is the input Gaussian field modulated by the OAM phase modulator (see Equations 3 and 4), and α_i (i = 1, 2, 3) are empirical scaling factors of these loss items. is the mean squared error (MSE) between the output complex fields and their ground truths:

where denotes the absolute value of a complex number, N_x = N_y = 36 is the pixel number of the FoV, and σ is the normalization factor:

is the exponential of the negative output efficiency:

where η is the output efficiency:

is the absolute value of the complex PCC between the output complex fields and their ground truths:

where and denote the mean value and complex conjugate of a complex matrix. Notably, all the PCC values in the main text refer to the absolute value of the corresponding complex PCC, which are calculated using Eq. 14. is the output efficiency:

Thus, the loss function is well-defined, comprising 4 items. The first 2 items of the loss function are used to produce the expected output OAM modes (with low MSE values and high output efficiency), while the last 2 items are applied to block the OAM modes that need to be filtered (with low PCC values and low output efficiency).

For the model of the OAM transformer, only the first 2 items of the loss function are used, with α₁ = 0.008 in the simulation model and α₁ = 0.003 in the experiment model. For the model of the OAM filter, the first 2 items of the loss function are used for the passed OAM modes, while the last 2 items are employed for the filtered OAM modes. The scaling factors are empirically selected as α₁ = α₂ = α₃ = 0.012 in the simulation model and α₁ = α₂ = α₃ = 0.005 in the experiment model.

The DONN models are trained using Python (version 3.9.18) with the machine learning framework PyTorch (version 2.0.1, Meta Platform Inc.). The Root Mean Square Propagation (RMSProp) method is chosen as the optimizer, with all hyperparameters set to their default values. Using the optimizer, PyTorch can perform automatic gradient descent calculation, thus achieving the optimization target defined by the loss functions. The training process is conducted on a PC with a 13th Gen Intel(R) Core(TM) i9-13900H CPU (Intel Inc.), an NVIDIA GeForce RTX 4060 Laptop GPU (Nvidia Inc.), and 64 GB of RAM.

The field scanning experimental setup is described first (see Figs. 4a and 4b). A radio frequency (RF) signal at 16.6667 GHz produced by a vector network analyzer (VNA, Keysight N5227B) is multiplied 18 times by a frequency extension module of the VNA (VNAX, OML WR3.4, Oleson Microwave Labs Inc.) to generate a THz signal at 0.3 THz (y-polarized). A dielectric collimating lens (90 mm diameter) is used to collimate the THz waves radiated from the Tx horn (WR3.4), to generate the Gaussian plane wave incidence, which then passes through the OAM phase modulator and the diffractive layers. The distance between the horn and the lens is 180 mm, which is exactly the focal length of the lens. The Rx probe is connected to another VNAX via a waveguide (WR3.4), placed on a 3D translation stage. The translation stage is controlled by a MATLAB (MathWorks Inc.) program to perform the planar scanning covering the FoV, with a step size of 1 mm. In this way, the VNA records the S-parameter (S21) at each location, thus obtaining the corresponding field distributions.

Next, we describe the wireless communication experimental setup (see Fig. 6). An arbitrary waveform generator (AWG, Keysight M8199A), connected to a clock generator (CG, Keysight M8008A), is used to produce the intermediate frequency (IF) signal. The CG is used to provide the clock and synchronization signal for the AWG. The IF signal is modulated by a pseudo-random binary sequence (PRBS) using the 16QAM format, with a symbol rate of 15 GBaud. Therefore, the communication speed is 15 × log₂16 = 60 Gbps (for each channel). An analog signal generator (ASG, Keysight AP5021A) is used to produce the local oscillator (LO) signal. The compact up-converter (CCU, VDI WR3.4, Virginia Diodes Inc.) and down-converter (CCD, VDI WR3.4, Virginia Diodes Inc.) can realize frequency conversion and mixing, with a relation:

where f_THz = 0.3 THz is the frequency of the THz waves, f_IF = 40 GHz is the frequency of the IF signal, N = 6 is the multiplication factor, and f_LO = 43.3333 GHz is the frequency of the LO signal. The THz waves are radiated from the Tx horn (WR3.4) of the CCU, collimated by the lens (with a focal length of 180 mm), and then pass through the OAM phase modulator, diffractive layers, and the demodulator, finally being detected by the Rx horn (WR3.4) of the CCD. Then an oscilloscope (OSC, Keysight UXR0404AP) is used to save and observe the received IF signal, which is generated from the CCD. A personal computer (PC) is employed to control the modulated data in the AWG and display the received data in the OSC. The received data is processed by software (Keysight PathWave Vector Signal Analysis, 89600 VSA Software) on the PC, including signal equalization. Then, the VSA software displays the testing results based on the processed data, including constellation diagrams, eye diagrams, and EVM values.

All the experimental samples (OAM modulators and diffractive surfaces) are fabricated using a 3D printer (Form 2, Formlabs Inc.) based on the stereolithography (SLA) technique. The printer provides the laser with a spot size of 85 µm and a layer thickness of 25 µm, realizing the printing resolutions of 85 µm in the transverse (x- and y-directions) and 25 µm in the axial (z-direction). The photo-curable high-temperature resin (refractive index of 1.6311 + j0.0245 measured at 0.3 THz) is selected as the 3D printing material due to its high resistance to deformation and scratching. A 1 mm thick substrate is first printed for each sample as a support, on which all unit cells are then fabricated. The shape of the unit cells is a square pillar with different heights, whose cross-section is 1 × 1 mm², and the pixel size is also 1 mm. The unit cells with various heights are printed using the resin material to match the required phase profiles of the samples (see Figs. 4c, 4d, and 4e), with a relation:

where n = 1.6311 is the real part of the refractive index of the resin, h is the height of a unit cell. Therefore, the phase profiles are matched by tuning h. The phase modulation is realized by changing the propagation length along the z-direction, so this phase modulation method is polarization-independent. In this way, the 3D-printed samples are encoded with the required phase profiles. Finally, before using, the fabricated samples are cleaned with alcohol by the ultrasonic method. Additionally, all the supports and frames of the experimental parts are fabricated using a second 3D printer (E2, Raise3D Inc.) with polylactic acid (PLA) material.