Fractional-Order Control Method Based on Twin-Delayed Deep Deterministic Policy Gradient Algorithm

Abstract

In this paper, a fractional-order control method based on the twin-delayed deep deterministic policy gradient (TD3) algorithm in reinforcement learning is proposed. A fractional-order disturbance observer is designed to estimate the disturbances, and the radial basis function network is selected to approximate system uncertainties in the system. Then, a fractional-order sliding-mode controller is constructed to control the system, and the parameters of the controller are tuned using the TD3 algorithm, which can optimize the control effect. The results show that the fractional-order control method based on the TD3 algorithm can not only improve the closed-loop system performance under different operating conditions but also enhance the signal tracking capability.

1. Introduction

Fractional-order calculus stands out for its flexible description of the behavior of nonlocal and non-Markovian dynamics, providing a rich mathematical tool for modeling complex systems. Its main applications include the modeling of nonlinear and nonsmooth system dynamics [1], the simulation of multiscale complex systems [2], the analysis of non-Markovian processes [3], as well as in the fields of signal processing, control systems and financial modeling [4]. In the field of control, fractional-order control has been combined with many traditional control schemes, such as fractional-order PID control [5], fractional-order robust control [6], and fractional-order sliding-mode control [7]; these fractional-order control methods have been developed in depth at the theoretical level to form a sound theoretical system and have achieved extensive and powerful results in practical applications.

As a traditional control strategy, fractional-order sliding-mode control is able to cope with the nonlinearity and uncertainty of the system more flexibly [8,9,10]. However, with the increase in system complexity and nonlinearity, choosing the optimal values of controller parameters often becomes a great difficulty [11]. Faced with this problem, different scholars have used a variety of optimization algorithms over the years, such as early adaptive control [12], the particle swarm algorithm [13], the genetic algorithm [14], and the wolf-pack algorithm [15]. These algorithms have had some success, but as technology advances, they struggle to handle increasingly complex problems.

In recent years, with the continuous development of artificial intelligence technology, more and more advanced algorithms have been applied to the control field [16]. Among them, reinforcement learning algorithms based on neural networks are increasingly becoming a focus of research because their flexibility and adaptability make them more adaptable to complex environments [17,18,19,20,21,22,23]. Reinforcement learning efficiently masters system dynamics by learning through the interaction of an agent with the environment [24]. This approach transforms the system control problem into a process in which the agent learns the optimal control strategy through continuous trial and error [25], and it can also learn without prior knowledge of the system model [26,27], which offers a flexible approach for real-time controller parameter optimization. Specifically, under the framework of reinforcement learning, the intelligent body adjusts the control strategy according to feedback signals by interacting with the environment and gradually optimizes the controller parameters [28,29]. This learning approach is more adaptive, especially in the face of a high-order system complexity, significant nonlinearities, and rapid changes in dynamic characteristics, where traditional static parameter optimization methods may appear to be inadequate [30]. Therefore, reinforcement learning provides a more intelligent solution for controller parameter optimization, which is expected to show more significant performance improvements in practical applications.

In this paper, a fractional-order control method based on TD3 reinforcement learning is proposed to optimize the parameters. A fractional-order disturbance observer is designed for estimating the disturbance signals present in the system, while an RBF network is designed for approximating the possible uncertainties in the system, and finally, a fractional-order sliding-mode controller is designed for controlling the system. For the parameter optimization part, it is performed by the TD3 reinforcement learning algorithm, which consists of six deep neural network networks designed to inhibit the bootstrap phenomenon in the reinforcement learning algorithm [31], so that the output of the network can converge to the optimal solution quickly. On this theoretical basis, a valve-controlled hydraulic system is selected for design and simulation in Matlab/Simulink to verify the effectiveness of the method proposed in this paper, and at the same time, in order to reflect the advantages and disadvantages of the proposed method, a series of comparative experiments are designed.

In summary, the main work of this paper can be succinctly summarized in the following three areas:

i.

The fractional-order disturbance observer is designed to estimate the system disturbance signal, and the RBF network is selected to approximate the uncertainties of the system; then, a fractional-order sliding-mode controller is designed to control the system according to the estimated value of the disturbance observer;

ii.

The TD3 algorithm is introduced to optimize the parameters of the controller, and an improved loss function is designed to improve the learning performance of the algorithm, accelerate the convergence of the network output, and optimize the control effect;

iii.

It is verified through simulation that not only the control effect of the proposed method is better than the selected comparison control method, but that its also has a great robustness and generalization capability.

This paper is organized as follows. Section 2 introduces the proposed fractional-order control method, including the system state equation generalization and the designed fractional-order disturbance observer with an RBF network structure, based on which the fractional-order sliding-mode controller is designed. Section 3 introduces the fundamentals of the TD3 algorithm [32] and defines the reward function and loss function used. Section 4 demonstrates the stability of the designed method to verify its theoretical correctness. In Section 5, the proposed method is simulated in Matlab/Simulink using a valve-controlled hydraulic system as a model, and a series of control simulations are performed to verify the practical feasibility of the proposed method. Finally, conclusions and future work are presented.

2. Design of Fractional-Order Control Method

In this section, the fractional-order control part of the proposed method is accomplished for a system generalization including a fractional-order disturbance observer, an RBF network, and a fractional-order sliding-mode controller.

For a common system, the equation of state can be expressed as

$$\left\{\begin{array}{c}\dot{x}=Ax+Bu+f\left(x\right)+d\hfill \\ y=Cx\hfill \end{array}\right.$$

(1)

where x is the system state, and $x\in R^n$; A is the system parameter matrix, $A\in R^{n\times n}$, B is the control matrix, and $B\in R^{n\times q}$; u is the control signal, $u\in R^q$, $C\in R^{p\times n}$ is the output matrix, $d\in R^{n\times 1}$ is the disturbance signal, and $f\left(x\right)\in R^{n\times 1}$ is the uncertainty of system.

In this paper, the Riemann–Liouville fractional-order calculus formula is defined as [33]:

$$D^{\alpha }g\left(x\right)=\frac{1}{\mathrm{\Gamma }\left(n-\alpha \right)}\frac{d^n}{x^n}{\int }_a^x\frac{g\left(t\right)}{{\left(x-t\right)}^{\alpha +1-n}}dt$$

(2)

where $\alpha$ represents the differential, $0<\alpha <1$, $\mathrm{\Gamma }\left(·\right)$ represents the Gamma function, n represents the smallest integer larger than $\alpha$, usually taken as 1, and a, x represents the lower and upper limits of the integral.

For the disturbance signal, the fractional-order disturbance observer is given by

$$\left\{\begin{array}{c}{D}^{\lambda }z=-L\left(D^{\lambda -1}z+L{D}^{\lambda -1}x\right)-L\left(A{D}^{\lambda -1}x+B{D}^{\lambda -1}u+D^{\lambda -1}\widehat{f}\left(x\right)\right)\hfill \\ \widehat{d}=z+Lx\hfill \end{array}\right.$$

(3)

where z is the disturbance observer auxiliary vector, L is the gain matrix, $-L$ is the Hurwitz matrix with n distinct eigenvalues, $\widehat{d}$ is the estimate of the disturbance, and $\widehat{f}\left(x\right)$ is an estimate of the model uncertainty.

Theorem 1.

Assume that the disturbance is bounded and its derivative is also bounded, which means that $∥d\left(t\right)∥\le \varsigma$, $∥\dot{d}\left(t\right)∥\le \zeta$. Define the estimation error of the disturbance $e_d\left(t\right)=d\left(t\right)-\widehat{d}\left(t\right)$. Based on the structure of the designed disturbance observer, the estimation error of the disturbance is bounded when the system uncertainty estimation error $\tilde{f}\left(x\right)$ is bounded, which means that $∥e_d\left(t\right)∥\le \xi$, where ξ is a very small constant greater than zero.

Proof. 

Define $M=-L$; then, M is a Hurwitz matrix with n distinct eigenvalues, so there exists an invertible matrix X such that $X^{-1}MX=diag\left({\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\right)$. Therefore, there exists a positive constant $\sigma$ such that $∥e^{Mt}∥\le \sigma e^{{\lambda }_{max}\left(M\right)t}$, where $\sigma =∥X^{-1}∥∥X∥$ [34]. Differentiating the estimation error $e_d\left(t\right)$ for the disturbance yields the following equation: $\begin{array}{cc}\frac{d\left(e_d\left(t\right)\right)}{dt}\hfill & =\dot{d}\left(t\right)-\dot{\widehat{d}}\left(t\right)=\dot{d}\left(t\right)-\frac{d}{dt}\left(z+Lx\right)\hfill \\ & =\dot{d}\left(t\right)+L\left(z+Lx\right)+L\left(Ax+Bu+\widehat{f}\left(x\right)\right)-L\left(Ax+Bu+f\left(x\right)+d\right)\hfill \\ & =\dot{d}\left(t\right)+L\widehat{d}\left(t\right)-Ld\left(t\right)+L\widehat{f}\left(x\right)-Lf\left(x\right)\hfill \\ & =\dot{d}\left(t\right)-L{e}_d\left(t\right)-L\tilde{f}\left(x\right)\hfill \end{array}$

It can be obtained that ${\dot{e}}_d\left(t\right)=\dot{d}\left(t\right)+M{e}_d\left(t\right)+M\tilde{f}\left(x\right)$. The subsequent proof is given in the stability analysis on $e_d\left(t\right)$.   □

For the model uncertainty that may exist in the system, a radial basis function (RBF) network is used for approximation. The network structure of an RBF has the ability for high-speed learning, the ability to approximate a nonlinear function, which improves the model’s fitting ability, and it can adapt to a variety of complex input–output mapping, so it is very flexible in practical applications. Moreover, compared with other types of neural networks, the RBF network structure is relatively simple, which makes it easy to realize and adjust [35]. The structure of the RBF network used in this paper is shown in Figure 1.

The input–output relationship for each layer can be expressed as an input layer, ${\theta }_i=x_i,\left(i=1,2,\cdots ,n\right)$, an implicit layer, $h_j=e^{ne{t}_j},ne{t}_j=-{\displaystyle \sum _{j=1}^m}\frac{{∥{\theta }_j-c_j∥}^2}{b_j^2},\left(j=1,2,\cdots m\right)$, and an output layer, $Y_i={W_i}^{T}H={\omega }_{i1}{h}_1+{\omega }_{i2}{h}_2+\cdots +{\omega }_{im}{h}_m$, where $∥·∥$ is the Euclidean paradigm, H is an implicit layer function vector $H={\left[h_1{h}_2\cdots h_m\right]}^T$, $c_j$ is the center vector, $b_j$ is the width of the radial basis function, and $W_i$ is the ith set of column vectors of the weight matrix W from the implicit layer to the output layer, which means that $W={\left[\begin{array}{cccc}{\omega }_{11}& {\omega }_{12}& \cdots & {\omega }_{1m}\\ {\omega }_{21}& {\omega }_{22}& \cdots & {\omega }_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ {\omega }_{n1}& {\omega }_{n2}& \cdots & {\omega }_{nm}\end{array}\right]}^T$, where i is the number of neurons in the input and output layer, and j is the number of neurons in the hidden layer.

Defining the optimal weights vector as $W^{*}$, the system uncertainty part function $f\left(x\right)$ can be expressed as [36]

$$f\left(x\right)={W^{*}}^{T}H+\epsilon$$

(4)

where $\epsilon ={\left[\begin{array}{cccc}{\epsilon }_1& {\epsilon }_2& \cdots & {\epsilon }_n\end{array}\right]}^T$ is the smallest approximation error of the RBF network, $H=\aleph \left(x,c,b\right)$, $\aleph \left(·\right)$ is the arithmetic function from the input layer to the output of the implicit layer; since the system uncertainty $f\left(x\right)$ is bounded, there exists an upper bound $W_{max}$ for $W^{*}$, which means that $∥W^{*}∥\le W_{max}$.

Defining the estimate of the RBF network for the system uncertainty $f\left(x\right)$ as $\widehat{f}\left(x\right)={\widehat{W}}^{T}H$, the estimation error can be written as [36]

$$\begin{array}{c}\tilde{f}\left(x\right)=f\left(x\right)-\widehat{f}\left(x\right)\hfill \\ \begin{array}{cc}& \end{array}={W^{*}}^{T}H-{\widehat{W}}^{T}H+\epsilon \hfill \\ \begin{array}{cc}& \end{array}=\left({W^{*}}^T-{\widehat{W}}^T\right)H+\epsilon \hfill \\ \begin{array}{cc}& \end{array}={\tilde{W}}^{T}H+\epsilon \hfill \end{array}$$

(5)

where $\epsilon$ satisfies $∥\epsilon ∥\le {\epsilon }_{max}$, with ${\epsilon }_{max}$ a bounded constant.

Based on the fractional-order disturbance observer’s estimate of the disturbance signal $\widehat{d}$ and the RBF network’s estimate of the system’s uncertainty $\widehat{f}\left(x\right)$, the sliding-mode surface of the fractional-order sliding-mode controller is given by

$$s=C_{1}e+C_2{D}^{\lambda -1}e$$

(6)

where e is the system tracking error, $e=y_d-y$, determined by the desired output $y_d$ and the actual output y of the system, $C_1$ and $C_2$ are positive real numbers, and $0<\lambda <1$ is the order of the fractional-order differentiation.

Deriving the sliding-mode surface and taking the sliding-mode convergence law as $-ks-k_s\mathrm{sgn}\left(s\right)$, where $k_s$ and k are the sliding-mode convergence law parameters and $k_s,k>0$, and substituting into (1), the control law can be obtained by

$$u={\left(CB\right)}^{-1}\left({\dot{y}}_d+\frac{C_2}{C_1}{D}^{\lambda }e+\frac{ks+k_s\mathrm{sgn}\left(s\right)}{C_1}\right)-B^{-1}Ax-B^{-1}d-B^{-1}f\left(x\right)$$

(7)

Substituting the disturbance signal estimator of (3) with the system uncertainty estimator $\widehat{f}\left(x\right)$ yields the following final control law:

$$u={\left(CB\right)}^{-1}\left({\dot{y}}_d+\frac{C_2}{C_1}{D}^{\lambda }e+\frac{ks+k_s\mathrm{sgn}\left(s\right)}{C_1}\right)-B^{-1}Ax-B^{-1}\widehat{d}-B^{-1}\widehat{f}\left(x\right)$$

(8)

3. TD3 Algorithm Based on Fractional-Order Control Method

In this section, a TD3 algorithm based on the designed fractional-order control method is proposed, and we design the reward function and loss function of the TD3 algorithm based on the proposed fractional-order control method, which in turn makes it effective for the optimization of the parameters of the fractional-order control method.

As a kind of reinforcement learning algorithm, the basic principle of the TD3 algorithm is also composed of a critic and an actor, two kinds of networks of the agent and the environment for the interactive operation. More specifically, Figure 2 shows the agent interacting with the environment to obtain the state $s_t$ and rewards $r_t$, the output of the action $a_t$ at moment t, and the update of the two kinds of networks within the agent is based on the loss function.

In this paper, the state of the environment $s_t$ is set as the error signal for each state of the system, which means that $s_t={\left[\begin{array}{cccc}{e}_1\left(t\right)& e_2\left(t\right)& \cdots & e_n\left(t\right)\end{array}\right]}^T$, and the reward signal $r_t$ is set to the negative of the weighted sum of the absolute values of the error signals for each state of the system, which means $r_t=-{\displaystyle \sum _{i=1}^n}{\eta }_i\left|e_i\left(t\right)\right|$, where ${\eta }_i>0$ denotes the weight of the ith error signal in the reward function, which needs to be set according to the actual situation; the action $a_t$ of the actor network’s output are the parameters of the fractional-order controller which should be optimized.

In the reinforcement learning algorithm, the Bellman equation is defined as [37]

$$U_t=r_t+\gamma U_{t+1}$$

(9)

where $r_t$ denotes the reward acquired by the intelligence in the environment, $U_t={\displaystyle \sum _{k=1}^n}{\gamma }^{k-1}{r}_{t+k-1}$ is a value function that represents the sum of the rewards that the intelligence expects to acquire in a continuous process after moment t, and where $0<\gamma <1$ denotes the discount rate.

For both sides of (9), the expectation based on the state S and action A at a moment can be obtained, and the value function of the action at that moment can be obtained by

$$E_{\pi }\left[U_t\left|S_t=s_t,A_t=a_t\right.\right]=E_{\pi }\left(r\left(s_t,a_t\right)\right)+\gamma E_{\pi }\left[U_{t+1}\left|S_{t+1}=s_{t+1},A_{t+1}=a_{t+1}\right.\right]$$

(10)

In the reinforcement learning algorithm, the estimation of the value function of the action is carried out through the critic network, so there is a certain bias, at which point the Bellman equation’s equal sign does not hold. Defining the difference between the left and right sides of (10) as the temporal-difference error (TD error) ${\delta }_{TD}\left(t\right)$, it can be written as [38]

$${\delta }_{TD}\left(t\right)=Q_{\pi }\left(s_t,a_t\left|{\varpi }_t\right.\right)-\left(r\left(s_t,a_t\right)+\gamma Q_{\pi }\left(s_{t+1},a_{t+1}\left|{\varpi }_t\right.\right)\right)$$

(11)

where ${\varpi }_t$ is the parameter of the critic network at moment t.

In order to solve the bootstrapping of the overestimation in traditional reinforcement learning algorithms, the TD3 algorithm adopts two sets of critic networks and target critic networks with identical structure and parameters; the corresponding network outputs are called Critic1 network $Q_{1,\pi }\left(s,a\left|{\varpi }_1\right.\right)$, Critic2 network $Q_{2,\pi }\left(s,a\left|{\varpi }_2\right.\right)$, Target Critic1 network $Q_{1,\pi }^{\prime }\left(s,a\left|{{\varpi }^{\prime }}_1\right.\right)$, and Target Critic2 network $Q_{2,\pi }^{\prime }\left(s,a\left|{{\varpi }^{\prime }}_2\right.\right)$. The algorithm adopts the strategy of smoothing regularization to reduce the estimation error, adding a noise signal $ϵ\sim clip\left(\mathcal{N}\left(0,\sigma \right);-c,c\right)$ to the action output from the actor network, so that the target value $y_{target}=r\left(s_t,a_t\right)+\gamma ·Q_{\pi }\left(s_{t+1},a_{t+1}\right)$ can be expressed as [31]

$$y_{target}=r\left(s_t,a_t\right)+\gamma \underset{i=1,2}{min}\left({Q^{\prime }}_{1,\pi }\left(s_{t+1},\left({a^{\prime }}_{t+1}+ϵ\right)\left|{{\varpi }^{\prime }}_j\right.\right)\right)$$

(12)

The agent in the TD3 algorithm stores each state transfer pair $\left(s,a,r,s^{\prime }\right)$ in the replay buffer during each interaction with the environment, and since in the process of learning by the intelligent body, the selection of the experience should be considered to prioritize the learning value, in this paper, a nonuniform sampling was designed as

$$P\left(X=r_i\right)=e^{\alpha \left(\left|r_i\right|\right)}{\left(\sum _{i=1}^n{e}^{\alpha \left(\left|r_i\right|\right)}\right)}^{-1}$$

(13)

where $P\left(X=r_i\right)$ denotes the probability of drawing a state transfer pair corresponding to reward $r_i$, $\alpha$ is a parameter that controls the shape of the distribution, and n denotes the total number of samples in the replay buffer.

Based on the sampling probability in (13), N samples are taken from the replay buffer and the loss function of the critic network is constructed using the mean squared error (MSE), which can be written as

$$J_{\varpi }\left(t\right)=\sum _{j=1}^2{J}_{{\varpi }_j}\left(t\right)=-N^{-1}\sum _{j=1}^2\sum _{i=1}^N{\left(Q_{j,\pi }{\left(s,a\left|\varpi \right.\right)}_i-y_{targe{t}_{j,i}}\right)}^2$$

(14)

For the actor network, in order to maximize the future expected return of the current strategy, which means that $\pi \left(a\left|\theta \right.\right)=\underset{\theta }{argmax}{E}_{s_t,a_t}\left({\displaystyle \sum _{i=t}^{\infty }}{\gamma }^{i-t}{R}_i\right)$, and in order to improve the explorability of the action space, the loss function was designed as

$$J_{\theta }\left(t\right)=-\left[N^{-1}\sum _{j=1}^N{E}_{s_{t_j},a_{t_j}}\left(\sum _{i=t_j}^{\infty }{\gamma }^{i-t_j}{R}_i\right)+\alpha H_{\theta }\left(\pi \right)\right]$$

(15)

where $\theta$ is the parameter of the actor network, $H_{\theta }\left(\pi \right)={\sum }_a\pi \left(s\left|\theta \right.\right)log\pi \left(s\left|\theta \right.\right)$ is a measure of uncertainty in the distribution of strategies, and $\alpha$ is the weight parameter for that item.

For both critic network and actor network parameters $\varpi$, $\theta$ are updated using the gradient descent method, which can be expressed as [31]

$$\begin{array}{c}{\varpi }_{t+1}={\varpi }_t-\alpha {\nabla }_{\varpi }{J}_{\varpi }\left(t\right)\hfill \\ \\ {\theta }_{t+1}={\theta }_t-\beta {\nabla }_{\theta }{J}_{\theta }\left(t\right)\hfill \\ \begin{array}{cc}& ={\theta }_t-\beta \left(N^{-1}\sum {\nabla }_a{Q}_{\pi }\left(s,a\left|\omega \right.\right){\nabla }_{\theta }\pi \left(s\left|\theta \right.\right)+\alpha {\nabla }_{\theta }{H}_{\theta }\left(\pi \right)\right)\end{array}\hfill \end{array}$$

(16)

where $\alpha$ and $\beta$ are the learning rate of the critic network and actor network.

For the target network parameter, a soft update is performed by introducing the update shift $\tau$ of the network parameter, which means [31]:

$$\begin{array}{c}{{\varpi }^{\prime }}_{i,t+1}=\tau {\varpi }_{i,t+1}+\left(1-\tau \right){{\varpi }^{\prime }}_{i,t}\begin{array}{cc}& \left(i=1,2\right)\end{array}\hfill \\ {{\theta }^{\prime }}_{t+1}=\tau {\theta }_{t+1}+\left(1-\tau \right){{\theta }^{\prime }}_t\hfill \end{array}$$

(17)

Finally, the TD3 algorithm uses lagged updating for the actor network, which means that the actor network is updated once when the critic network is updated multiple times. Based on the above description, a structure of a fractional-order control method based on the TD3 algorithm is proposed, and the corresponding pseudocode is shown in Algorithm 1.

Algorithm 1 FOCS pseudocode based on the TD3 algorithm

Initialize critic networks $Q_{1,\pi }\left(s,a\left|{\varpi }_1\right.\right)$ and $Q_{2,\pi }\left(s,a\left|{\varpi }_2\right.\right)$ and actor network ${\pi }_{\theta }$ with

random parameters ${\varpi }_1$, ${\varpi }_2$, $\theta$

Initialize target networks ${\varpi }_1^{\prime }\leftarrow {\varpi }_1$, ${\varpi }_2^{\prime }\leftarrow {\varpi }_2$, ${\theta }^{\prime }\leftarrow \theta$

Initialize replay buffer $\mathcal{B}$

for $t=t_0$ to $t=t_f$ do

Output action based on current parameters $a_t\sim {\pi }_{\theta }\left(s_t\right)+ϵ$, $ϵ\sim clip\left(\mathcal{N}\left(0,\sigma \right);-c,c\right)$,

Take $a_t$ for FOCS, observe the error $e\left(t\right)$, calculate r, and observe new state $s_{t+1}$

Store the transfer pair $\left(s,a,r,s^{\prime }\right)$ in $\mathcal{B}$

if $n\ge N$ $\setminus \setminus$ n is the number of transfer pair in replay buffer $\mathcal{B}$.

Sample N transfer pairs and calculate the loss function of critic network $J_{\varpi }\left(t\right)$

Update $\varpi$ by ${\varpi }_{t+1}\leftarrow {\varpi }_t-\alpha {\nabla }_{\varpi }{J}_{\varpi }\left(t\right)$

if Critic network after d updates

Calculate the loss function of critic network $J_{\theta }\left(t\right)$

Update $\theta$ by ${\theta }_{t+1}\leftarrow {\theta }_t-\beta {\nabla }_{\theta }{J}_{\theta }\left(t\right)$

Update target network by

${{\varpi }^{\prime }}_{t+1}\leftarrow \tau {\varpi }_{t+1}+\left(1-\tau \right){{\varpi }^{\prime }}_t$

${{\theta }^{\prime }}_{t+1}\leftarrow \tau {\theta }_{t+1}+\left(1-\tau \right){{\theta }^{\prime }}_t$

end if

end if

end for

4. Stability Analysis

In this section, the stability of the proposed control method is substantiated through a detailed proof, emphasizing the theoretical feasibility of the proposed approach.

For the proposed fractional-order disturbance observer, the Lyapunov function is defined as

$$V_d=\frac{1}{2}{e}_d^T{e}_d$$

(18)

The derivation of (18) and substitution into (1), (3), and (5) yield

$$\begin{array}{cc}{\dot{V}}_d\hfill & =e_d^T{\dot{e}}_d=e_d^T\left(\dot{d}-\dot{\widehat{d}}\right)=e_d^T\left(\dot{d}-\left(\dot{z}+L\dot{x}\right)\right)\hfill \\ \hfill & =e_d^T\left(\dot{d}-\left(D^{-\lambda +1}{D}^{\lambda }z+L\left(Ax+Bu+f\left(x\right)+d\right)\right)\right)\hfill \\ \hfill & =e_d^T\left(\dot{d}-L{e}_d-L\tilde{f}\left(x\right)\right)=e_d^{T}M{e}_d+e_d^{T}M\tilde{f}\left(x\right)+e_d^T\dot{d}\hfill \\ \hfill & \le {\lambda }_{max}\left(M\right)e_d^T{e}_d+{\lambda }_{max}\left(M\right)e_d^T\left({\tilde{W}}^{T}H\right)+{\lambda }_{max}\left(M\right)e_d^T\epsilon +\frac{1}{2}\left(e_d^T{e}_d+{\dot{d}}^T\dot{d}\right)\hfill \\ \hfill & \le \left(2{\lambda }_{max}\left(M\right)+\frac{1}{2}\right)e_d^T{e}_d+\frac{{\lambda }_{max}\left(M\right)}{2}tr\left(H^{T}H\right)tr\left({\tilde{W}}^T\tilde{W}\right)+\frac{1}{2}\left({\lambda }_{max}\left(M\right){\epsilon }^T\epsilon +{\dot{d}}^T\dot{d}\right)\hfill \\ \hfill & \le \left(2{\lambda }_{max}\left(M\right)+\frac{1}{2}\right)e_d^T{e}_d+\frac{{\lambda }_{max}\left(M\right)m}{2}tr\left({\tilde{W}}^T\tilde{W}\right)+\frac{1}{2}\left({\lambda }_{max}\left(M\right){\epsilon }_{max}^2+{\zeta }^2\right)\hfill \end{array}$$

(19)

where ${\lambda }_{max}\left(M\right)$ is the largest eigenvalue of M which is a Hurwitz matrix so it can be obtained that ${\lambda }_{max}\left(M\right)<0$. In order to make the estimation error of the fractional-order disturbance observer for the disturbance signal stable, M should satisfy ${\lambda }_{max}\left(M\right)<-1/4$.

In order to keep the approximation error of the designed RBF network for the system uncertainty within a certain range, the adaptive law of the RBF network weights and the implicit layer function update law are taken as

$$\begin{array}{c}\dot{\widehat{W}}={\eta }^{-1}\left(-\mathcal{K}\widehat{W}-C_{1}H{s}^{T}C\right)\hfill \end{array}$$

(20)

where $\eta \in R^{m\times m}$ is the designed positive-definite diagonal matrix, $\mathcal{K}>0$ are the designed parameters of the adaptive law, s and $C_1$ are the sliding-mode surface and the parameter which was designed in (6), and H is the implicit layer function vector.

For the proposed RBF network, the Lyapunov function is defined as

$$V_f=\frac{1}{2}tr\left({\tilde{W}}^T\eta \tilde{W}\right)$$

(21)

Derive equation (21), and substitute into the RBF network’s adaptive law (20):

$$\begin{array}{c}{\dot{V}}_f=tr\left({\tilde{W}}^T\eta \dot{\tilde{W}}\right)=tr\left[{\tilde{W}}^T\eta \left({\dot{W}}^{*}-\dot{\widehat{W}}\right)\right]\hfill \\ \begin{array}{cc}& =\end{array}-tr\left[{\tilde{W}}^T\eta \left({\eta }^{-1}\left(-\mathcal{K}\widehat{W}-C_{1}H{s}^{T}C\right)\right)\right]\hfill \\ \begin{array}{cc}& =\end{array}tr\left({\tilde{W}}^T\mathcal{K}\widehat{W}\right)+tr\left({\tilde{W}}^T{C}_{1}H{s}^{T}C\right)\hfill \\ \begin{array}{cc}& =\end{array}\mathcal{K}tr\left[{\tilde{W}}^T\left(W^{*}-\tilde{W}\right)\right]+tr\left({\tilde{W}}^T{C}_{1}H{s}^{T}C\right)\hfill \\ \begin{array}{c}\begin{array}{cc}& \le -\end{array}\frac{\mathcal{K}}{2}tr\left({\tilde{W}}^T\tilde{W}\right)+\frac{\mathcal{K}}{2}{∥W^{*}∥}^2+tr\left({\tilde{W}}^T{C}_{1}H{s}^{T}C\right)\hfill \\ \begin{array}{cc}& \le -\frac{\mathcal{K}}{2}tr\left({\tilde{W}}^T\tilde{W}\right)+\frac{\mathcal{K}}{2}{W}_{max}^2+tr\left({\tilde{W}}^T{C}_{1}H{s}^{T}C\right)\end{array}\hfill \end{array}\hfill \end{array}$$

(22)

Based on the designed sliding-mode surface function, define the Lyapunov function

$$V_s=\frac{1}{2}{s}^{T}s$$

(23)

Deriving equation (23) and substituting the system state equation shown in (1) with the control law shown in (8) yield

$$\begin{array}{c}{\dot{V}}_s=s^T\dot{s}=s^T\left(C_1\dot{e}+C_2{D}^{\lambda }e\right)=s^T\left(C_1\left({\dot{y}}_d-C\dot{x}\right)+C_2{D}^{\lambda }e\right)\hfill \\ \begin{array}{cc}& =s^T\left(C_1\left({\dot{y}}_d-C\left(Ax+Bu+f\left(x\right)+d\right)\right)+C_2{D}^{\lambda }e\right)\end{array}\hfill \\ \begin{array}{cc}& =s^T\left(C_1\right.\left({\dot{y}}_d-C\left(Ax+B\left({\left(CB\right)}^{-1}\left({\dot{y}}_d+\frac{ks+k_s\mathrm{sgn}\left(s\right)+C_2{D}^{\lambda }e}{C_1}\right)\right.\right.\right.\end{array}\hfill \\ \begin{array}{cc}& \begin{array}{cc}& \left.\left.\left.\left.-B^{-1}Ax-B^{-1}\widehat{d}-B^{-1}\widehat{f}\left(x\right)\right)+f\left(x\right)+d\right)\right)+C_2{D}^{\lambda }e\right)\end{array}\end{array}\hfill \\ \begin{array}{cc}& =\end{array}{s}^T\left(-ks-k_s\mathrm{sgn}\left(s\right)-C_{1}C\tilde{f}\left(x\right)-C_{1}C{e}_d\right)\hfill \\ \begin{array}{cc}& =-k{s}^{T}s-s^T\left(k_s\mathrm{sgn}\left(s\right)+C_{1}C\tilde{f}\left(x\right)+C_{1}C{e}_d\right)\end{array}\hfill \\ \begin{array}{cc}& \le \end{array}\left(-k+\frac{k_s}{2}+\frac{C_1∥C∥}{2}\right)s^{T}s+\frac{C_1∥C∥}{2}{e}_d^T{e}_d+\frac{k_s}{2}-s^T{C}_{1}C\left({\tilde{W}}^{T}H+\epsilon \right)\hfill \end{array}$$

(24)

To prove the overall stability of the original system with the addition of a fractional-order control method, define the Lyapunov function

$$V=V_d+V_f+V_s$$

(25)

A derivation of equation (25) and a substitution of the Lyapunov derivation of (19), (22), and (24) lead to

$$\begin{array}{c}\dot{V}=\stackrel{.}{V_d}+\stackrel{.}{V_f}+\stackrel{.}{V_s}\hfill \\ \begin{array}{c}\end{array}\le \left(2{\lambda }_{max}\left(M\right)+\frac{C_1∥C∥+1}{2}\right)e_d^T{e}_d+\left(-k+\frac{k_s}{2}+\frac{C_1∥C∥}{2}\right)s^{T}s-\left(\frac{\mathcal{K}}{2}-\frac{m{\lambda }_{max}\left(M\right)}{2}\right)tr\left({\tilde{W}}^T\tilde{W}\right)\hfill \\ \begin{array}{ccc}& & +\frac{C_1∥C∥}{2}{s}^{T}s+\frac{C_1∥C∥}{2}{\epsilon }_{max}^2+\frac{\mathcal{K}}{2}{W}_{max}^2+\frac{1}{2}\left({\zeta }^2+{\lambda }_{max}\left(M\right){\epsilon }_{max}^2\right)+\frac{k_s}{2}\end{array}\hfill \\ \begin{array}{c}\end{array}\le \left(-k+\frac{k_s}{2}+C_1∥C∥\right)s^{T}s+\left(2{\lambda }_{max}\left(M\right)+\frac{C_1∥C∥+1}{2}\right)e_d^T{e}_d\hfill \\ \begin{array}{ccc}& & -\left(\frac{\mathcal{K}}{2}-\frac{m{\lambda }_{max}\left(M\right)}{2}\right)tr\left({\tilde{W}}^T\tilde{W}\right)+\frac{1}{2}\left({\zeta }^2+{\lambda }_{max}\left(M\right){\epsilon }_{max}^2+C_1∥C∥{\epsilon }_{max}^2+\mathcal{K}{W}_{max}^2+k_s\right)\end{array}\hfill \\ \begin{array}{c}\end{array}\le -\left(k-\frac{k_s}{2}-C_1∥C∥\right)s^{T}s-\left(-2{\lambda }_{max}\left(M\right)-\frac{C_1∥C∥+1}{2}\right)e_d^T{e}_d\hfill \\ \begin{array}{cc}& \end{array}-\left(\frac{\mathcal{K}}{2}-\frac{m{\lambda }_{max}\left(M\right)}{2}\right)tr\left({\tilde{W}}^T\tilde{W}\right)+\delta \hfill \\ \begin{array}{c}\end{array}\le -\kappa V+\delta \hfill \end{array}$$

(26)

where $\kappa =min\left(\left(k-\frac{k_s}{2}-C_1∥C∥\right),\left(-2{\lambda }_{max}\left(M\right)-\frac{C_1∥C∥+1}{2}\right),\left(\frac{\mathcal{K}}{2}-\frac{m{\lambda }_{max}\left(M\right)}{2}\right)\right)$ to satisfy the system stability, the control system parameter $0<C_1<\frac{-4{\lambda }_{max}\left(M\right)-1}{∥C∥}$, $k-\frac{k_s}{2}>C_1∥C∥$, $k_s>0$ should be selected, and $\delta =\frac{1}{2}\left({\zeta }^2+{\lambda }_{max}\left(M\right){\epsilon }_{max}^2+C_1∥C∥{\epsilon }_{max}^2+\mathcal{K}{W}_{max}^2+k_s\right)$. At this point, the system is boundedly stable.

5. Simulation Result

In this section, in order to validate the effectiveness of the proposed methodology, a valve-controlled hydraulic system is selected as the object of study to design a series of simulations. Firstly, to verify the online learnability of the TD3 algorithm, three rounds of training states during the training process are selected for the comparison. Secondly, the control effect is verified for the optimized fractional-order control method of the TD3 algorithm by employing the prescribed performance fractional-order sliding-mode controller (PPC-FOSMC), an unoptimized fractional-order sliding-mode controller (FOSMC), and a sliding-mode controller (SMC). Furthermore, the antidisturbance is proved for the proposed method, and the simulation verification is carried out for different input signals and disturbance signals. The results show that the fractional-order control method based on the TD3 algorithm not only has good online learning ability and generalization ability but also has a better control effect than the traditional FOSMC, SMC, and PPC-FOSM. Finally, in order to verify that the designed method can still maintain a better control effect under noise disturbance, a Gaussian noise is selected as the disturbance signal to simulate using the online learning method.

5.1. Simulation Results of System Online Learning

A valve-controlled hydraulic system is a third-order system [39] which can be described approximately as

$$\left\{\begin{array}{c}\dot{x}=Ax+Bu+B_{d}d+f\left(x\right)\hfill \\ y=Cx\hfill \end{array}\right.$$

(27)

where $x={\left[x_v,x_s,x_a\right]}^T$ denotes the position, velocity, and acceleration, respectively, u denotes the control input, and y denotes the output of the position. The state matrix, input matrix, disturbance matrix, and output matrix $A,B,B_d$, and C are, respectively,

$$\begin{array}{c}\begin{array}{c}A=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ -R_1& -R_2& -R_3\end{array}\right],B=\left[\begin{array}{c}0\\ 0\\ b\end{array}\right],\hfill \\ \begin{array}{cccc}\begin{array}{cccc}\begin{array}{cccc}& & & \end{array}& & & \end{array}& & & \end{array}\hfill \end{array}\hfill \\ B_d={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^T,\begin{array}{c}\end{array}C=\left[\begin{array}{ccc}1& 0& 0\end{array}\right],\hfill \end{array}$$

(28)

where $R_1=\frac{4k{k}_{ce}\beta }{m{V}_t}$, $R_2=\frac{4B_p{k}_{ce}\beta }{m{V}_t}+\frac{k}{m}$, $R_3=\frac{4k_{ce}\beta }{V_t}+\frac{B_p}{m}$, $b=\frac{4k_v{k}_{sv}{k}_q\beta A_p}{m{V}_t}$, and the values of the parameters of the valve-controlled hydraulic system are shown in

Table 1. Parameters of the valve-controlled hydraulic system.

Name	Symbol	Numerical Value
Total mass of piston and load converted to piston	m	5 kg
Piston area of hydraulic cylinder	$A_p$	$0.4734$ m2
Springiness	k	1
Flow-pressure amplification factor	$k_{ce}$	$3.005\times {10}^{-13}$
Modulus of elasticity of hydraulic fluid	$\beta$	$0.005$
Total volume of hydraulic cylinder	$V_t$	$1.5$ m3
Viscous damping coefficient of piston and load	$B_p$	5
Servo amplifier gain	$k_v$	1
Servo valve gain	$k_{sv}$	$4.733\times {10}^{-3}$
Flow gain of slide valve	$k_q$	1

.

The disturbance signal during the TD3 training was set to be $d=0.5sin\left(5\pi t\right)$, the system uncertainty was $f\left(x\right)=sin\left(2x_a\right)cos\left(3x_a\right)$, and the system expected the output position signal to be $x_{pd}=sin\left(t\right)$; the training parameters are shown in

Table 2. Parameters of the TD3 algorithm.

Name	Numerical Value
Episode	1300
Episode step	100
Sampling time	$0.1$ s
Final time	10 s
Learning rate	0.001
Replay buffer	$1\times {10}^6$
Minimum batch size	128
Number of actor network layers	4
Number of critic network layers	9

.

Learning to train the system with the above parameters, the episode rewards and average rewards obtained after 1900 episodes are shown in Figure 3a, and Figure 3b–d show the state of the system for the selected episode 1, episode 500, and episode 1826. It can be noticed that in the first 420 episodes, the episode reward fluctuates a lot, and after episode 420, the episode reward and the average reward stabilize for the first time, but at that point, the network still cannot achieve a particularly good training effect. From episode 630 to episode 1300, the agent further explores the action space, and after episode 1300, the episode reward and average reward converge to more optimal values, corresponding to the system state. It can be found that the position and velocity signals of the system corresponding to episode 1 have the worst tracking performance with both large errors and desired signals, while the acceleration signals have the worst tracking performance with more noise. Compared to episode 1, episode 500’s systematic position signal can already track the desired signal at the 0.8th second but with a small error, the velocity signal can track the desired signal after 2 s but with a small noise in between, and the acceleration signal can only track the desired signal after the 4th second. Finally, compared to the previous two, episode 1826 is definitely the best performer, with the position signal tracking the desired signal in the first 0.1 s with very little error, and the velocity signal tracking the desired signal with little error after that, despite oscillating in the first 0.1 s, and with the acceleration signal tracking the desired signal after oscillating in the first 0.4 s with an acceptable error.

In order to better visually represent the training state of set 1826, the system’s individual signals and desired signals and their errors at that point in time are shown in Figure 4. It can be seen that the position, velocity, and acceleration signals of the system have good tracking performance, and the errors are all within a small range. The same is true for the estimation of disturbance signals.

5.2. Simulation Results of TD3-FOCS

For the online learning results of the valve-controlled hydraulic fractional-order control method, the training results of the 1826th episode are shown in Figure 5 with $C_1=0.001$, $C_2=290$, $k=945,k_s=43,\lambda =0.55$ selected as the controller parameters, and the parameters of FODOB and the RBF network were set to $L=157$, $\mathcal{K}=7$, and $\eta =I$, which is a single-unit diagonal matrix for the subsequent comparative simulation experiments; the simulation time was set to 100 s, and the final results are shown in Figure 6.

As can be seen from Figure 6, for the position signal, the proposed TD3-FOSMC had the fastest convergence speed (first 0.03 s) and maintained the final steady-state error in the range of (−0.001, +0.001), which was the best among the experimental results. For the velocity signal, it also had the fastest convergence speed (first 0.25 s) and maintained the final steady-state error in the range of $(-0.005,+0.005)$, which was the best performance among all the experimental methods. For acceleration signals, although the prescribed performance of the fractional-order sliding-mode controller ended up with a smaller error range, the convergence time was greater than the proposed TD3-FOSMC, which converged in the first 0.3 s. Combining all the above control effects, it can be concluded that the proposed TD3-FOSMC has a better speed and stability.

5.3. Simulation Results of Different Control Signals and Disturbance Signals

In order to demonstrate the robustness and versatility of the method, two control signals (a triangular wave composite signal and a step signal) and two disturbance signals (a strong sine wave signal disturbance and a triangular wave composite signal) were selected for the simulation verification. This allowed a more comprehensive assessment of the method’s antidisturbance performance under different control and disturbance conditions, thus demonstrating the effectiveness of the method in mitigating the effects of an external disturbance in a variety of practical situations.

As observed in Figure 7, the simulation results revealed the capability of accurately tracking the desired signal under the influence of a triangular wave composite signal and a strong sine wave signal disturbance. Specifically, the position signal achieved tracking within a remarkably short duration of 0.07 s, with very small overshoots (less than 1%) in the first 0.02 s, and the final steady-state error confined to a narrow range of (−0.001, 0.001). In the first 0.3 s, the velocity signal had large to small fluctuations in the range of (−3.7, 6); after that, it achieved synchronization with the desired signal and kept the steady-state error within the range of (−0.01, 0.01). On the other hand, the acceleration signal fluctuated in the range of (−12, 12) for the first 0.5 s and was after synchronized with the desired signal and kept the steady-state error within the range of (−0.25, 0.25). The results show that despite the large overshoot in the system response for the different input signals and disturbance signals, the system still had good stability and fast performance.

Analyzing the results shown in Figure 8, when subjected to a step signal and a strong triangular wave composite signal disturbance during the simulation, the obtained results exhibited a marginally reduced performance compared to the previous scenario. Taking the simulation time of 10 s as an example, it can be found that for the position, velocity, and acceleration signals, there were large fluctuations in the first 1 s, and their stabilization time was slowing down sequentially. For the position signal, there were fluctuations within the range of (5, 15) within the first 0.3 s; after that, it could track the desired signal, and the final steady-state error was limited to the range of (0, 0.004). The velocity signal exhibited fluctuations within the range of (−10, 10) during the first 0.5 s, after which it synchronized with the desired signal, and the final steady-state error was constrained to (−0.002, 0). For the acceleration signal, fluctuations within the range of (−20, 20) occurred during the initial 0.8 s, following which it converged to the desired value. The final steady-state error remained confined to a smaller range of (−0.0001, 0.0001). Despite a slight decrease in performance, the results underscore the method’s ability to maintain effective control even in the presence of challenging input signals and disturbance conditions.

5.4. Simulation Results under Noise Disturbance

The control performance is illustrated for the method under noise disturbance. The noise disturbance signal was selected with a mean value of zero, a variance of five, and a frequency of 10 Hz, and the simulation verification was carried out by using the online learning of the agent. The results are shown in Figure 9.

Analyzing Figure 9, it can be seen that the position signal exhibited minimal susceptibility form the noise disturbance signal, with an overshoot of less than 3% observed only within the initial 0.03 s. Subsequently, the position signal could track the desired signal, and the final steady-state error could be maintained within the range of (−0.001, 0). The velocity signal demonstrated minimal susceptibility to noise disturbances, with oscillations confined to the range of (−1.6, 4.7) within the initial 0.08 s. Following that period, the signal could adeptly track the desired trajectory, and the final steady-state error could satisfy the narrow bounds of (−0.0015, 0.0015). Comparing with the earlier online learning outcomes, the performance of the acceleration signal was degraded under the noise signal. The acceleration signal was oscillatory within the range of (−6, 6) during the initial 0.15 s. Despite these challenges, it could approximately track the desired signal. However, due to the influence of noise, it maintained a final steady-state error within the range of (−0.13, 0.13). Thus, based on the analysis above, the proposed control method could achieve the control of system within the bounded range under the noise signal.

6. Conclusions

In this paper, a fractional-order control method based on the TD3 algorithm was introduced. A fractional-order disturbance observer was designed to estimate the system’s disturbance signal, and an RBF network was selected to approximate the uncertainties in the system, and the fractional-order sliding-mode control method was also adopted to design the controller. A valve-controlled hydraulic system was simulated and validated in Matlab/Simulink using the agent online learning and the optimization parameters. Different control signals and disturbance signals were used in the optimized fractional-order control system. The results showed that the limitations of this method mainly lay in the difficulty of setting up the training environment comprehensively. Despite some shortcomings, the proposed method was generally fast and had good antidisturbance ability.