Abstract
A weight least squares algorithm is developed for rational models with outliers in this paper. Different weights are assigned for each cost function, and by calculating the derivatives of these cost functions, the parameter estimates can be estimated. Compared with the traditional least squares algorithm, the proposed algorithm can remove the bad effect caused by the outliers, thus has more accurate parameter estimates. A simulation example is proposed to validate the effectiveness of the proposed algorithm.
1. Introduction
System identification can be roughly divided into two kinds: the structure identification and the parameter estimation [1–4]. Compared to the parameter estimation, structure identification is more difficult, because one should choose the available products from numerous product pool. Once the structure is determined, parameter estimation is involved [5–7]. Recently, a lot of parameter estimation algorithms are proposed, including the least squares (LS) algorithm [8, 9], the gradient algorithm [10, 11], the particle swarm optimization algorithm, and the expectation maximization algorithm [12, 13]. Among these algorithms, the LS algorithm is to find a vector that is a local minimizer to a function that is a sum of squares; thus, it is the simplest algorithm and is widely used.
Although the LS algorithm is applied to many different kinds of nonlinear models, e.g., bilinear models [14, 15], polynomial nonlinear models [16, 17], hard nonlinear models, and rational nonlinear models [18, 19], it also brings a challenging problem: heavy computational effort caused by the inverse matrix calculation. In order to overcome this difficulty, some online algorithms are proposed. For example, Liu and Ding proposed an auxiliary model-based recursive generalized least squares algorithm for multivariate output-error autoregressive system [20], Zhu provided an implicit least squares algorithm for nonlinear rational models [21], Wang et al. developed a recursive LS (RLS) algorithm for Hammerstein models [22]. However, all the systems in the above literature have normal data, when some data of the systems are abnormal, those algorithms may be invalid.
Processes in industry usually suffer from outliers in measurement data [23–25], which make the identification of the process a challenging problem. Thus, a reliable estimation of the systems with outliers is essential to efficient process identification [26, 27]. Zhao et al. provided a VB approach for ARX models with outliers and assumed that all the variables are available and the precise interval of time-delays is known a priori [28]. Jin et al. proposed a conventional approach to nonlinear process models with outliers [29]. Both these two algorithms are off-line algorithms, which update the parameters through all the collected data, and thus have heavy computational efforts especially for large-scale systems.
In this paper, a weight LS algorithm is proposed for rational models with outliers. By introducing different weights for each cost function, the cost functions of the outliers can be neglected. Furthermore, to reduce the computational efforts, a weight recursive least squares (W-RLS) is also derived. Both these two algorithms can yield the optimal parameter estimates. Briefly, the paper is listed as follows. Section 2 introduces the rational model and the traditional LS algorithm. Section 3 develops the weight LS algorithm and the weight RLS algorithm. Section 4 provides a simulation example. Finally, Section 5 gives a conclusion of this paper.
2. Rational Model with Outliers
The rational model with outliers is written bywhere is the normal output, is the abnormal output, a normal stochastic white noise with zero mean and variance , the outliers of the rational model, whose noise to signal ratio is larger, and and are the structures of the rational model that can be expressed as
The information vectors and are the products of past inputs and past outputs , and the structures of and are known in prior, and are the unknown parameters to be estimated and can be expressed as
In application, the number of the abnormal data is far smaller than that of the normal data. Assume that we have collected input-output data, the number of the normal data is and the abnormal data is , . Without loss of generality, assume that , then the rational model is transformed into
Collect input and output data and definewhere or 2. We have
Define the cost functionthen the following LS algorithm can be obtained
Remark 1. In the LS algorithm, all the data are used to update the parameter estimates; thus, the outliers cause extremely bad influence to the estimates.
3. Weight Based Least Squares Algorithm
In order to get more accurate parameter estimates, one should weak/eliminate the bad influence caused by the outliers. Therefore, a weight least squares algorithm is proposed in this section.
3.1. Weight LS Algorithm
Rewrite the cost function aswhich can be transformed into
Clearly, each cost function plays the same role in estimating the parameters when using the traditional LS algorithm. For this reason, we can introduce different weights for each cost function, e.g., small weight for the cost function of the outliers and large weight for the cost function of the normal data.
It follows that the new cost function can be written by
Taking the derivative of with respect to and then equating it to zero give
Remark 2. Based on equations (12) and (13), we can get that each cost function has different weights, and the latest data have larger weights.
Taking the conditional expectation on both sides of (13) yieldsSince is independent of , which means thatTherefore, we haveUnfortunately, the W-LS algorithm needs to perform the inverse matrix calculation , if the order is large, and the computational effort is heavy. To overcome this difficulty, we derive a weight recursive least squares algorithm in the following subsection.
3.2. Weight Recursive Least Squares Algorithm
Define
The above two equations give rise to
Lemma 1. [30]. Assume that , , and , then the following equality holds:According to Lemma 1, can be written by
The parameter estimates by using the W-LS algorithm are rewritten by
According to equation (18), we have
Therefore, equation (21) is simplified to
The parameter estimates by using the weight recursive least squares (W-RLS) algorithm are listed as follows:
The W-RLS algorithm consists of the following iterations:(1)Let , , and give a small positive number and a positive number .(2)Let with being a column vector whose entries are all unity and , .(3)Collect and .(4)Form by equation (26).(5)Form according to equation (25).(6)Compute by equation (27).(7)Update the parameter estimation vector by equation (24).(8)Compare and : if , then terminate the procedure and obtain ; otherwise, increase by 1 and go to step 3.
3.3. Property of the W-RLS Algorithm
Lemma 2. Assume that and are both symmetric positive semidefinite matrices, then is also a symmetric positive semidefinite matrix.
Proof. Since and are symmetric positive semidefinite matrices, for a random nonzero vector , the following inequalities hold:It gives rise towhich means that is also a symmetric positive semidefinite matrix.
Lemma 3. Assume that and are both positive definite symmetric matrices. The largest eigenvalue of the matrix is , and the largest eigenvalue of the matrix is . Then the largest eigenvalue of the matrix is .
Proof. Since and are positive definite symmetric matrices, and the largest eigenvalues are and , respectively. Then we haveClearly, and are symmetric positive semidefinite matrices. We haveTherefore, the largest eigenvalue of the matrix is .
Theorem 2. For the rational model in (2), the parameter estimate by using the W-RLS algorithm is expressed by equations (24)–(27). Then we have
Proof. Rewrite as follows:and the following equation can be obtainedTaking the conditional expectation on both sides of (34) gets is Gaussian noise and satisfiesIt follows thatSinceAccording to Lemmas 2 and 3 we haveAssume that the largest eigenvalue of the matrix is , and letClearly, .
It follows thatThen the parameter estimate by using the W-RLS algorithm is asymptotic convergent.
Remark 3. Unlike the W-LS algorithm, the W-RLS algorithm avoids the inverse matrix calculation, thus has less computational efforts.
Remark 4. At the sampling instant , the W-RLS algorithm uses the data up to and including time , while the stochastic gradient algorithm only uses the current data ; thus, the W-RLS algorithm has a quicker convergence rate than that of the stochastic gradient algorithm.
4. Example
Consider the rational model proposed in [31],
Then, one can getwhere is a input sequence with zero mean and unit variance is a white noise with zero mean and variance . In simulation, we collect 1800 input-output data, and the data from are outliers. The noise to signal ratio is when , and when . The simulation data are shown in Figure 1, , it shows the noise and output data from 500 to 600 are outliers.
(a)
(b)
(c)
Apply the LS and W-LS algorithms to the rational model . The parameter estimates and their estimation errors are shown in Table 1.
Next, use the RLS and the W-RLS algorithms for this rational models , the parameter estimates and their estimation errors are shown in Table 2 and Figure 2.
Then the following findings can be obtained: (1) Table 1 shows that the W-LS algorithm is more effective than the LS algorithm; (2) Table 2 and Figure 2 demonstrate that the parameter estimates by using the W-RLS algorithm are more accurate than those by using the RLS algorithm.
5. Conclusions
This paper proposes a weight least squares algorithm for rational models with outliers. By assigning the weights for each data set, the proposed algorithm can obtain more accurate parameter estimates when compared with the traditional least squares algorithm. Furthermore, a weight recursive least squares algorithm is derived to decrease the computational efforts. The convergence analysis and simulation example show that the proposed algorithm is effective.
Data Availability
There are no data in our manuscript.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was funded by the Natural Science Foundation for Colleges and Universities in Jiangsu Province (No. 18KJB120009).