Optimal Control of Stochastic Dynamic Systems of a Random Structure with Poisson Switches and Markov Switching

Antonyuk, Svitlana V.; Byrka, Marian F.; Gorbatenko, Mykola Y.; Lukashiv, Taras O.; Malyk, Igor V.

doi:https://doi.org/10.1155/2020/9457152

Journal of Mathematics

On this page

Abstract Introduction Conclusions Data Availability Conflicts of Interest References Copyright Related Articles

Research Article | Open Access

Volume 2020 | Article ID 9457152 | https://doi.org/10.1155/2020/9457152

Optimal Control of Stochastic Dynamic Systems of a Random Structure with Poisson Switches and Markov Switching

Svitlana V. Antonyuk,¹Marian F. Byrka,²Mykola Y. Gorbatenko,³Taras O. Lukashiv,³and Igor V. Malyk¹

Academic Editor: Yongqiang Fu

Received19 Mar 2020

Revised25 May 2020

Accepted30 May 2020

Published24 Jun 2020

Abstract

The problem of synthesis of the optimal control for a stochastic dynamic system of a random structure with Poisson perturbations and Markov switching is solved. To determine the corresponding functions for Bellman functional and optimal control the system of ordinary differential equation is investigated.

1. Introduction

Systems with Markov parameters belong to an important class of systems describing processes of the rapid changes in states that occur in many cases, for example, in industry, in queuing systems [1], in ecological systems [2], in economics and finance, and in modeling of a microgrid [3]. These systems are mathematical models of the hybrid dynamic systems, in which one part of the variables changes continuously and the other part changes discretely. The most used hybrid models are models described by differential equations. This article focuses on this area of research.

Let us pay attention to the fact [4] that such abrupt changes in the systems are discrete events and are assumed to be modelled by a Markov chain taking values in a finite value set. Practical motivations as well as many theoretical results for the Markovian jump system can be found, for example, in [5–7].

The linear system of the following form is considered in [4]:

Here, is a state vector, is the Markov chain with finite number of states, are defined constant matrices for each fixed , are indefinite matrices that satisfy the specific conditions described in the following, and is a control. In this case, the problem of the synthesis of optimal control with constant inverse communication for such systems is solved.

In [2], the optimal control problem for the Ito linear stochastic system with the uncertainty of the following form is solved:

Here and in the following, , is a Wiener process.

In [6], the optimal control problem for the Ito linear stochastic system of the following form is solved:

The nonlinear case of the Ito stochastic system is considered in [4]. In this paper, the sufficient optimality conditions are obtained in the case of an infinite horizon for the system

In [7], optimality conditions were obtained in the case of an infinite horizon for a linear stochastic system with finite aftereffect and Markov parameters:

A great contribution to the development of systems with Markov parameters was introduced by Katz [8]. In this work, a Markov process can be either a Markov chain or a continuous Markov process. In addition, it is advisable to consider the case of perturbations of the impulse type, this is when, for example, the moments of time in which the discontinuities of the phase trajectories of the process are possible, and they are known in advance. For deterministic and difference systems, this situation was studied in detail in [9].

Stochastic systems of random structure according to Katz [8] with impulse Markov perturbations according to Tsarkov were considered in [10–13]. Stability in different probabilistic senses has been investigated, and the problem of optimal stabilization, solution of which is the control stabilizing the system to stochastically stable, is solved. In these papers, the absence of perturbation points was assumed, but in [14–16], the existence and uniqueness of the solution of a system of differential-difference equations with Markov parameters and switching in the presence of perturbation points were proved. Therefore, we can consider the problem of stability and optimal control for such systems.

Furthermore, in the theory of control of classical problems, there are problems of constructing the control that minimizes a certain quality functional. This problem is wider than the problem of optimal stabilization because it does not include stabilization of the solution x (t). The monograph [17] is one of the main papers in which the general theory of controllable processes and the theory of controlled stochastic differential equations are presented. Here, the classical problems of optimal stopping of random processes are considered, a rigorous proof of the derivation of Bellman’s equations [18] is presented, and applications of Bellman’s equations for constructing optimal controls are considered.

Controllable systems with finite aftereffect of the deterministic and stochastic type are considered in [19]. The paper [20] is devoted to the synthesis of optimal control of linear stochastic dynamic systems with finite aftereffect and Poisson perturbations. The general form of the Bellman equation and the Bellman functional for a stochastic dynamical system of a random structure with Markov switching was obtained in [21], which is a sufficient condition for the existence of optimal control, and the problem of synthesis of optimal control for such systems was solved in [22].

The switching times of Markov processes in [21, 22] are assumed to be known; however, in many cases, abrupt changes in the trajectories of the process at random moments are also possible. Such situations are adequately described by taking into account the term in the equation of motion, which is the integral of the Poisson measure. Stochastic systems with semi-Markov switching were introduced in [23]. In this paper, the conditions of weak convergence of semi-Markov random evolutions to the diffusion process are considered. Also, sufficient conditions for the stability of prelimited processes are discussed. In the articles [15, 16], Tsarkov et al. discussed the influence of Markov perturbation on the solutions and applied stochastic differential equations for applied problems.

This paper is devoted to the synthesis of optimal control of stochastic dynamic systems of a random structure that are under the influence of impulse Markov switching at known moments of time with allowance for Poisson perturbations that allow describing discontinuities of trajectories at random moments.

2. Main Result

Consider the stochastic random structure system given by the Ito stochastic differential equationwith Markov switchingand initial conditions

Here, is the continuous Markov process with values in the space , is the Markov chain with values in space ; is the -measurable standard Wiener process; is the centered Poisson measure [24, 25]; and the processes , and are independent [24, 25] and defined in the probability basic ().

The trajectories of process , belong to the Skorokhod space [17], control is the -measurable function from a class of admissible controls [19]; coefficients , and and function are measurable functions which satisfy the conditions for the existence and uniqueness of a solution of problems (6)–(8). An example of the existence and uniqueness conditions is the Lipschitz condition and linear growth condition [17].

Consider a sequence of functions , , and define the class of the functions:

The weak infinitesimal operator (WIO) [24]is defined on the functionals .

Here, is a strong solution of (6) on by the control , which is constructed on the interval . The optimal control problem is to find a control , from the class U that minimizes the functional of quality [19]:

For some fixed , where , and .

It is necessary to prove some auxiliary properties in order to obtain sufficient conditions of optimality. Let us consider the main statements about properties of the infinitesimal operator.

Lemma 1. Let(1)Unique solution of problems (6)–(8) exists with finite second moment for each (2)Sequence of functionals , , from class exists(3)WIO be defined for on solutions of (6)–(8)Then, and the qualityhold.

Proof. Then, the Markov process with respect to the minimal -algebra that is built on the interval , , satisfies the Dynkin formula [24]:where and .
If , then for the solution of (6)–(8), we can obtain the equalAnalogously, we should write the Dynkin formula [24] on the interval , and subtracting it from (14), we obtain (12). Lemma 1 is proved.

Lemma 2. Let(1)Conditions (1) and (2) of Lemma 1 be satisfied.(2)For , , , the equationwith boundary conditionhold, where is the WIO defined by (10).

Then, , , can be given as

Proof. Consider the solution of (6)–(8) for , which is constructed according to the initial condition.

Let us integrate (15) with respect to from to and calculate the expectation. We obtain

According to Lemma 1, the first term in (18) exists, and it is equal to increment (12):where by (16) and . So,

Substituting this equality into (18), we get the statement of Lemma 2.

Theorem 1. Let(1)Unique solution of problems (6)–(8) exists with finite second moment for each .(2)Sequence of functions , , and control , , which satisfy for all in the equation with boundary conditions exist.(3), and the inequality hold:where is WIO (10) on solutions of (6)–(8).

Then, the control is optimal, and for ,

The sequence of functions is called the cost, or the Bellman function, and equation (21) can be written as the Bellman equation:

Proof. The optimal control is an admissible control. So, there is a solution for which (21) takes the form is chosen at .
Let us integrate (26) from to , obtain the expectation, and, taking into account (22), obtainNow, let be any other admissible control from . Тhen, by condition (3), the following inequality holds: for Let us integrate (28) with respect to and obtain the expectation with fixed and initial value . Taking into account Lemmas 1 and 2 and boundary condition (22), we obtainThis is the definition of optimal control in the sense of the minimization of the functional . The theorem is proved.

2.1. General Solution of the Problem of the Optimal Control

According to [10, 11, 26], WIO (10) has the form

is a scalar product, , , ; «» is the sign of transpose, is the trace of a matrix, and is the conditional density of distribution

By assumption, .

The first equation for , , can be obtained by substituting (30) in (21). Then, the required equation takes the formwith boundary condition

The second equation for optimal control can be obtained from (32) by differentiating with respect to because , , gives a minimum of the left side of (32):

is Jacobian, which consists of elements (—similarly), and , .

Solving systems (32) and (34), even in the presence of computing technology, is quite difficult. So, it is advisable to consider a simplified view of problems (6)–(8) and (10), such as a system with a quadratic quality functional. The following paragraphs will focus on such problems.

2.2. Optimal Control of Linear Stochastic Random Structure Dynamic Systems with Markov Switching

Let us consider the problem of optimal control for the stochastic dynamic system given by the following stochastic differential equation:with Markov switchingand initial conditions

Here, are piecewise continuous integrable matrix functions of appropriate dimensions.

The optimal control problem for systems (35)–(37) is to find the control , , , from set of admissible controls such that it minimizes the functional of quality:

is a uniformly positive definite -matrix with respect to , and and are nonnegative definite matrices. To simplify, let us denote

Theorem 2. The optimal control for problems (35)–(38) has the formwhere the nonnegative definite matrix belongs to the Bellman functional:

Proof. The Bellman equation for (35)–(37) has the formwhereSubstitute (43) into (42):The expression for the optimal control can be obtained by differentiating (44) because minimizes the left side of (44):whereSo,The theorem is proved.

2.3. Construction of the Bellman Equation

Substituting (40) and (41) into (42), we obtain the following equation for :

Equating the quadratic form with respect to x and expressions that do not depend on to zero and taking into account the matrix equality , we obtain the system of differential equations for finding matrices , , , and :with boundary conditions

Thus, the following theorem holds.

Theorem 3. If the control cost is found in the form (38) for systems (35)–(37), then the system of differential equations for finding the matrices , , , and , has forms (49)–(51).

Furthermore, the problem of the existence of solutions of problems (49)–(51) must be solved. Let us use the Bellman method of iterations [18]. For simplicity, consider the interval , where , and omit all indices «» in , and . First, we define the zero approximation:where is the bounded and piecewise-continuous matrix. Let us substitute (52) into (34) and calculate the value of for the resulting equation, which corresponds to the control (42).

Then, substituting into Bellman equation (42), the control which minimizes (42) will be found. Continuing this process, one can construct a sequence of controls and functionals of the following form:where is the solution of boundary value problems (49)–(51) for .

The following estimate is valid for :

The convergence of functionals to , the convergence of controls to , and the convergence of a sequence of matrices to can be proved by using (54) [19, 20].

The following estimate is valid:

Thus, the following theorem holds.

Theorem 4. An approximate solution of the problem of synthesis of the optimal control for (35)–(38) is found using the Bellman successive approximation method for which the n-th approximation of the control and the Bellman functional for each interval , , are found from formula (53), and the error is estimated by inequality (55).

2.4. Model Example

Let us consider the scalar case with two states for Markov chains and : and . Define the following initial data for system (35):(1), , , , and (2), , , , and

Generator of the Markov process is defined as , and the transaction matrix for the Markov chain is .

Then, define , , and for (38):

The Bellman functional considers the following form: , .

System (49) for finding , , takes the form

Substituting values of , we obtain

So, the optimal control for our case has a form

The results of simulation of two trajectories of the random process are shown in Figure 1. This figure shows two trajectories of solutions with a positive (blue line) and negative (red line) initial condition . As we can see from this figure, the solutions are stabilized because the optimal control satisfies the conditions of Theorem 1 with the quadratic functional.

3. Conclusions

In this paper, the problem of synthesis of the optimal control of stochastic dynamical systems of a random structure with Poisson perturbations that are under the influence of pulse switching of the Markov chain type is solved. In the linear case, an algorithm for finding the optimal control is obtained, and its convergence is justified.

Data Availability

No data were used to support this study.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this article.

References

V. S. Korolyuk and V. V. Korolyuk, Stochastic Models of Systems, Kluwer, Boston, MA, USA, 1999.
F. Long, H. Huang, and A. Ding, “Stochastic stabilization of Ito stochastic systems with Markov jumping and linear fractional uncertainty,” Journal of Control Science and Engineering, vol. 2013, Article ID 697849, 14 pages, 2013.
View at: Publisher Site | Google Scholar
M. Bujorianu and J. Lygeros, “Toward a general theory of stochastic hybrid systems,” Lecture Notes in Control and Information Sciences (LNCIS), vol. 337, Springer, Berlin, Germany, 2006.
View at: Google Scholar
L. Hu, P. Shi, and B. Huang, “Stochastic stability and robust control for sampled-data systems with Markovian jump parameters,” Journal of Mathematical Analysis and Applications, vol. 313, no. 2, pp. 504–517, 2006.
View at: Publisher Site | Google Scholar
M. Aliyu and E. K. Boukas, “Robust H_∞ control for Markovian jump nonlinear systems,” IMA Journal of Mathematical Control and Information, vol. 17, no. 3, pp. 295–308, 2000.
View at: Publisher Site | Google Scholar
V. Dragan and T. Morozan, “The linear quadratic optimization problems for a class of linear stochastic systems with multiplicative white noise and Markovian jumping,” IEEE Translation on Automatic Control, vol. 49, no. 5, pp. 665–675, 2004.
View at: Google Scholar
K. Benjelloun, E. K. Boukas, and O. L. V. Costa, “H _∞-control for linear time-delay systems with markovian jumping parameters,” Journal of Optimization Theory and Applications, vol. 105, no. 1, pp. 73–95, 2000.
View at: Publisher Site | Google Scholar
I. Y. Katz, The Method of Lyapunov Functions in Problems of Stability and Stabilization of Systems of Random Strucure, UGAPS, Ekaterinburg, Russia, 1998.
M. L. Sverdan and Y. F. Tsarkov, Stability of Stochastic Impulse Systems, RTU, Riga, Latvia, 1994.
T. O. Lukashiv, I. V. Yurchenko, and V. K. Yasinskii, “Lyapunov function method for investigation of stability of stochastic Ito random-structure systems with impulse Markov switchings. I. General theorems on the stability of stochastic impulse systems,” Cybernetics and Systems Analysis, vol. 45, no. 2, pp. 281–290, 2009.
View at: Publisher Site | Google Scholar
T. O. Lukashiv, I. V. Yurchenko, and V. K. Yasinsky, “Lyapunov function method for investigation of stability of stochastic Ito random-structure systems with impulse Markov switchings. II. First-approximation stability of stochastic impulse systems with markov parameters,” Cybernetics and Systems Analysis, vol. 45, no. 3, pp. 464–476, 2009.
View at: Publisher Site | Google Scholar
T. O. Lukashiv, V. K. Yasinskiy, and E. V. Yasinskiy, “Stabilization of stochastic diffusive dynamical systems with impulse markov switchings and parameters. Part I. Stability of impulse stochastic systems with markov parameters,” Journal of Automation and Information Sciences, vol. 41, no. 2, pp. 1–24, 2009.
View at: Publisher Site | Google Scholar
T. O. Lukashiv, L. I. Yasinskaya, and V. K. Yasinskiy, “Stabilization of stochastic diffusive dynamical systems with impulse markov switchings and parameters. Part II. Stabilization of dynamical systems of random structure with external markov switchings,” Journal of Automation and Information Sciences, vol. 41, no. 4, pp. 26–42, 2009.
View at: Publisher Site | Google Scholar
T. Lukashiv and I. Malyk, “Existence and uniqueness of solution of stochastic dynamic systems with Markov switching and concentration points,” International Journal of Differential Equations, vol. 2017, Article ID 7958398, 5 pages, 2017.
View at: Publisher Site | Google Scholar
Y. F. Tsarkov, V. K. Yasinsky, and I. V. Malyk, “Stability in impulsive systems with Markov perturbations in averaging scheme. I. Averaging principle for impulsive Markov systems,” Cybernetics and Systems Analysis, vol. 46, no. 6, pp. 975–985, 2010.
View at: Publisher Site | Google Scholar
Y. F. Tsarkov, V. K. Yasinsky, and I. V. Malyk, “Stability in impulsive systems with Markov perturbations in averaging scheme. 3. Weak convergence of solutions,” Cybernetics and Systems Analysis, vol. 47, no. 3, pp. 442–458, 2011.
View at: Publisher Site | Google Scholar
I. I. Gikhman and A. V. Skorokhod, Controlled Stochastic Processes, Springer Verlag, New York, NY, USA, 1979.
R. Bellman, Dynamic Programming, Princeton University Press, Princeton, NJ, USA, 1972.
E. A. Andreeva, V. B. Kolmanovskii, and L. E. Shaikhet, Control of Hereditary Systems, Nauka, Moskow, Russia, 1992.
T. O. Lukashiv, L. I. Yasinskaya, and V. K. Yasinskiy, “Synthesis of the optimal control for linear stochastic dynamical systems with finite aftereffect and Poisson disturbances,” Journal of Automation and Information Sciences, vol. 40, no. 10, pp. 22–37, 2008.
View at: Google Scholar
T. O. Lukashiv and I. V. Malyk, “Sufficient optimality conditions for stochastic dynamical systems of random structure with markovian switchings,” Journal of Automation and Information Sciences, vol. 48, no. 6, pp. 60–67, 2016.
View at: Publisher Site | Google Scholar
A. Das, T. O. Lukashiv, and I. V. Malyk, “Optimal control synthesis for stochastic dynamical systems of random structure with the markovian switchings,” Journal of Automation and Information Sciences, vol. 4, no. 49, pp. 37–47, 2017.
View at: Publisher Site | Google Scholar
I. V. Malyk, “Compensating operator and weak convergence of semi-Markov process to the diffusion process without balance condition,” Journal of Applied Mathematics, vol. 2015, Article ID 563060, 7 pages, 2015.
View at: Publisher Site | Google Scholar
E. B. Dynkin, Markov Processes, Academic Press, New York, NY, USA, 1965.
B. Oksendal, Stochastic Differential Equations, Springer, New York, NY, USA, 2013.
T. Lukashiv, “One form of lyapunov operator for stochastic dynamic system with Markov parameters,” Journal of Mathematics, vol. 2016, Article ID 1694935, 5 pages, 2016.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2020 Svitlana V. Antonyuk et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies