Modeling and Optimization of 𝑀/𝐺/𝟏-Type Queueing Networks: An Efficient Sensitivity Analysis Approach

Abstract

A mathematical model for 𝑀/𝐺/1-type queueing networks with multiple user applications and limited resources is established. The goal is to develop a dynamic distributed algorithm for this model, which supports all data traffic as efficiently as possible and makes optimally fair decisions about how to minimize the network performance cost. An online policy gradient optimization algorithm based on a single sample path is provided to avoid suffering from a “curse of dimensionality”. The asymptotic convergence properties of this algorithm are proved. Numerical examples provide valuable insights for bridging mathematical theory with engineering practice.

1. Introduction

In the past decades, great efforts have been devoted to model and optimize the network-based communication systems with the increasing transmitting demands and sophisticated performance criteria. However, technical challenges abound in designing such systems due to the limited network resources and the stochastic network characteristics. It is wellstudied that queueing theory is one of the primary tools used to deal with traffic engineering problems over both wired and wireless packet networks [1–3]. Factors affecting performance of network systems, based on the models in queueing theory, include the arrival rates (or the interarrival time distributions), the service rates (or the interservice time distributions), and the queue discipline. In this paper, we will concentrate on how to optimally and efficiently allocate the service rates to all concurrent user queues in each network element according to the arrival rates, so that the the lowest possible performance cost is achieved.

Suppose that the arrival rates parameter for each user is uncontrollable, and the interarrival time parameter is exponentially distributed. Let queue discipline be first-come first-served (FCFS). Based on this model, the decision parameter is the service time, namely, the service rates allocated to each user. Without loss of generality, consider that the service times of each user are independent and identically distributed with 𝐺. Thus, the user queues are modeled as multiple concurrent 𝑀/𝐺/1 queues, for which many techniques can be found in the literature [4–7]. More importantly, the optimal resource allocation problem is then translated into a resource-constrained Markov decision problem (MDP). The objective of the MDP is to find a resource allocation policy that minimizes the overall performance cost, by observing and analyzing the system behavior information. To achieve this goal in such a mathematically tractable MDP model, some solving results were proposed in [8, 9]. However, these methods may typically suffer from a “curse of dimensionality” [10]. In addition, if no structural information about the system is gained, we cannot explicitly compare the performance cost by observing and analyzing the network system behavior under different policies. Hence, a crucial question that comes to mind is how can we achieve our goal of performance cost minimization using as little computation effort and system structure information as possible? Thinking along this direction, we propose a sensitivity-based optimization algorithm tailored for the bandwidth-constrained and backlog-constrained 𝑀/𝐺/1 queueing system.

Within the above issues addressed, we now confront a key question, namely, how to quantify the network system cost in terms of performance metrics. Note that the performance metrics may change not only as actual network scenarios vary but also as the influence factors change, that is, different network environments will lead to different definitions of performance metrics. For instance, a flexible cost function was proposed in [11], which takes into account the power, interference, backlog, and other factors in the ultrawide band (UWB) communication networks. In [12], a bandwidth related cost function was presented for wide area overlay networks. Another type of performance metrics, considering the waiting time and the energy consumption for serving jobs in an 𝑀/𝐺𝐼/1 processor sharing (PS) queue, was provided in [13]. Therefore, throughout this paper, we use the word “cost” to refer to both backlog and service rates-related performance cost criteria in a broad sense.

In summary, the contributions of this paper are as follows. Firstly, we present a distributed resource-constrained 𝑀/𝐺/1-type queueing model for supporting multiple user traffic over communication networks. By translating the multiple queues of each relay network node into continuous-time semi-Markov processes, we further formulate the network system performance cost minimization as a stochastic optimization problem in the Markov system. Secondly, within the optimization framework, we explicitly define the network system performance cost measure, based on which an online policy scheme combining performance sensitivity analysis and MDP is proposed. This scheme is cost-benefit since during the data transmission process, the performance cost is significantly reduced by choosing the optimal policy while the computational complexity is greatly decreased in that it is based on a single sample path, that is, a trajectory of each user queue. Thirdly, with the estimation of the performance gradients, a resource allocation algorithm is developed. From the performance comparison formula, we directly obtain the optimality condition. Moreover, the asymptotic convergence properties of such algorithm are proved. Last but not least, the established model can be easily extended to a more general situation when the states of the 𝑀/𝐺/1-type queueing system are partially observable, that is, partially observable Markov decision problems (POMDPs).

The rest of paper is organized as follows. Section 2 starts by presenting the 𝑀/𝐺/1-type queueing system model. Next, Section 3 proposes the cost-benefit resource allocation optimization algorithm. To evaluate the performance of the proposed algorithm, numerical examples are provided in Section 4. Finally, the paper concludes with a short discussion in Section 5.

2. System Model

In this section, we model the queueing system according to the dynamic transmission procedure of each network user. With the formulation of 𝑀/𝐺/1 user queues, firstly, we derive the steady-state Markov transition probability matrix. We then define the performance cost measure, based on which the objective function of the optimization algorithm is presented. Before digging into details, we summarize the used notations in Table 1.

Table 1 

Notations.

Consider a network modeled by a topology graph, that is, 𝒢=(𝒱,ℰ), where 𝒱 denotes the set of network elements (nodes), and ℰ represents the set of links. Note that if a link 𝜄∈ℰ, there exist a work-conserving server with time-invariant service capacity 𝐶_𝜄, which serves packets and transfers them from source element to end element of 𝜄. More precisely, consider that each element 𝑣∈𝒱 keeps a separate queue for every user traffic going through it, which is illustrated in Figure 1. For the simplicity of exposition, let 𝑁(𝑡) be the number of user queues served at any given time 𝑡≥0. Without loss of generality, consider that the capacity of each user queue 𝑖∈{1,…,𝑁(𝑡)} is upper bounded by a constant 𝐾, that is, 𝑀/𝐺/1/𝐾 queueing model with limited backlog capacity 𝐾.

Figure 1 

Simplified network structure.

Denote the service time allocated to user queue 𝑖 as a general distribution 𝐺_𝑖(𝑠,𝑡) for any given time 𝑡≥0. Since the users arrive at and depart from each network element randomly, we should allocate the service rates 𝜇_𝑖(𝑡) dynamically for all 𝑡≥0 so that the performance cost at the element is minimized.

Then, we have𝜇_𝑖1(𝑡)≡∫^∞₀𝑠d𝐺_𝑖(𝑠,𝑡)∈ℛ⁺≜(0,∞), (2.1) where 𝜇_𝑖(𝑡) represents the mean service rates of user queue 𝑖 at time 𝑡, satisfying 𝜌_𝑖≜𝜆^𝑖/𝜇_𝑖(𝑡)<1, 𝜆^𝑖 is the long-term average rate of the 𝑖th user arrivals, namely, the intensity of the Poisson arrival process. For ease of presentation, hereafter, 𝜇_𝑖(𝑡) and 𝜇_𝑖 will be used interchangeably. Let Γ_𝑖(𝑡)⊂ℛ⁺ be the feasible region of the service rates allocated to user queue 𝑖. To be more precise, we introduce some definitions to accurately modeling the 𝑖th user queue.

Definition 2.1 (semi-Markov queueing process). A semi-Markov process 𝑋^𝑖={𝑋^𝑖_𝑡,𝑡≥0} characterizes the 𝑖th user queue's behavior on the state space Φ={0,1,…,𝐾}, where 𝑋^𝑖_𝑡 represents the number of packets in the queue after the latest packet left a network element at time 𝑡≥0.

Remark 2.2. The semi-Markov kernel of 𝑋^𝑖 can be further represented as 𝑄^𝜇𝑖(𝑡)_{[(𝐾+1)×(𝐾+1)]}=𝑝^𝑖₀(𝑡)𝑝^𝑖₁(𝑡)𝑝^𝑖₂(𝑡)⋯𝑝^𝑖_𝐾−1(𝑡)𝑝^𝑖_𝐾𝑞(𝑡)_𝑖0(𝑡)𝑞^𝑖₁(𝑡)𝑞^𝑖₂(𝑡)⋯𝑞^𝑖_𝐾−1(𝑡)𝑞^𝑖_𝐾(𝑡)0𝑞^𝑖₀(𝑡)𝑞^𝑖₁(𝑡)⋯𝑞^𝑖_𝐾−2(𝑡)𝑞^𝑖_𝐾−1(𝑡)⋮⋮⋮⋱⋮⋮000⋯𝑞^𝑖₀(𝑡)𝑞^𝑖₁,(𝑡) (2.2) where 𝑞^𝑖_𝑘(𝑡)=^𝑡₀𝑒^{−𝜆𝑖𝑠(𝜆𝑖𝑠)𝑘}𝑘!d𝑝𝐺(𝑠,𝑡),𝑘∈Φ,_𝑖𝑘(𝑡)=^𝑡₀𝑞^𝑖_𝑘(𝑡−𝑠)𝜆^𝑖𝑒^{−𝜆𝑖𝑠}d𝑠,𝑘∈Φ. (2.3)

Let 𝑌^𝑖={𝑌^𝑖_𝑚;𝑚=0,1,2,…} be the embedded Markov chain of 𝑋^𝑖, where 𝑌^𝑖_𝑚 is interpreted as the number of packets in the 𝑖th user queue when the 𝑚th packet has been served. It is essential to note that 𝑌^𝑖 is positive recurrent, irreducible, and aperiodic under the condition of 𝜌_𝑖<1 since 𝑋^𝑖 is. Moreover, 𝑌^𝑖 has both the same steady-state probability vector 𝜋^𝜇𝑖=(𝜋^𝜇𝑖(0),…,𝜋^𝜇𝑖(𝐾)),𝜋^𝜇𝑖(𝑘)>0,𝑘∈Φ and the same steady-state performance cost measure (discussed later) as 𝑋^𝑖. According to [3], 𝑋^𝑖 has standard transfer probabilities 𝑝_𝑘𝑗(𝑡),𝑘,𝑗∈Φ, and for any 𝑗 with respect to 𝑘∈Φ,𝑝_𝑘𝑗(𝑡)/𝑡 converges consistently to a constant for 𝑘≠𝑗 as 𝑡→0. In this state, the transition probability matrix 𝑃^𝜇𝑖 of 𝑌^𝑖 can be derived as follows 𝑃^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]}=lim_𝑡→∞𝑄^𝜇𝑖(𝑡)_{[(𝐾+1)×(𝐾+1)]}=𝑎^𝑖₀𝑎^𝑖₁𝑎^𝑖₂⋯𝑎^𝑖_𝐾−1𝑎^𝑖_𝐾𝑎^𝑖₀𝑎^𝑖₁𝑎^𝑖₂⋯𝑎^𝑖_𝐾−1𝑎^𝑖_𝐾0𝑎^𝑖₀𝑎^𝑖₁⋯𝑎^𝑖_𝐾−2𝑎^𝑖_𝐾−1⋮⋮⋮⋱⋮⋮000⋯𝑎^𝑖₀𝑎^𝑖₁, (2.4) wherelim_𝑡→∞𝑝^𝑖_𝑘(𝑡)=lim_𝑡→∞𝑞^𝑖_𝑘(𝑡)=𝑎^𝑖_𝑘,𝑘∈Φ. (2.5)

Remark 2.3. Note that the symbol 𝑎^𝑖_𝑘 represents the probability of 𝑘 packets arriving at the time interval when a packet of the 𝑖th user queue is being served. The balance equation of each concurrent user queue 𝑖 can be further expressed as 𝜋^𝜇𝑖(𝑃^𝜇𝑖−𝐼)=0,𝜋^𝜇𝑖𝑒=1, (2.6) where 𝑒=(1,1,…,1)^𝑇 is a (𝐾+1)-dimensional column vector whose all components are 1's, and the superscript “𝑇” denotes transpose.

In principle, the performance cost measure is based on the definition of performance cost function. Note that performance cost is a commonly used term that changes its meaning with different network environments. Considering there is limited backlog space in each network element, therefore, the more backlog is occupied, the higher cost is paid. In this state, with the increasing of service rates, the backlog-related cost can be reduced accordingly. However, in many practical networks, especially considering the various wireless environments, transmission cost cannot be neglected. In general, the transmission power is considered as a convex increasing function with respect to the service rates. Thus, the design of performance cost function should trade off both the backlog related and the service rates related costs. Conceptually, we associate to each user queue a performance cost function defined as follows.

Definition 2.4 (performance cost function). Consider a general performance cost function 𝑓_𝑖∶Φ×Γ_𝑖→ℛ⁺ associated to the 𝑖th user queue, which is the sum of the backlog-related cost 𝜑₁(𝑘) and the service rates-related cost 𝜑₂(𝜇_𝑖), that is, 𝑓_𝑖(𝑘,𝜇_𝑖)=𝜑₁(𝑘)+𝜑₂(𝜇_𝑖),∀𝑘∈Φ. (2.7)

Suppose that the performance cost function 𝑓_𝑖 is differentiable with the service rates 𝜇_𝑖 on Γ_𝑖. For ease of notation, hereafter, the terms 𝑓_𝑖 and 𝑓^𝜇𝑖 are used interchangeably throughout the paper. Now, it is imperative to define the performance cost measure as our objective function for each user queue. Motivated by [3, 14], the definition is as follows.

Definition 2.5 (performance cost measure). The performance cost measure 𝜂^𝜇𝑖_𝑓 with respect to the service rates 𝜇_𝑖 for each user queue 𝑖 is denoted as 𝜂^𝜇𝑖_𝑓=𝐸_𝜋𝜇𝑖(𝑓^𝜇𝑖)=_𝑘∈Φ𝜋^𝜇𝑖(𝑘)𝑓^𝜇𝑖(𝑘)=𝜋^𝜇𝑖𝑓^𝜇𝑖, (2.8) where 𝑓^𝜇𝑖=(𝑓^𝜇𝑖(0),…,𝑓^𝜇𝑖(𝐾))^𝑇 is a (𝐾+1)-dimensional column vector, and 𝐸_𝜋𝜇𝑖 denotes the expectation with respect to the steady-state probability 𝜋^𝜇𝑖 of the semi-Markov process 𝑋^𝑖 in Definition 2.1.

Since the state space of 𝑋^𝑖 is finite, we should note that for each nonnegative bounded performance cost function, there is𝜂^𝜇𝑖_𝑓=_𝑘∈Φ𝜋^𝜇𝑖(𝑘)𝑓^𝜇𝑖(𝑘)<∞. (2.9)

Remark 2.6. Each user queue 𝑖 has been modeled as a semi-Markov queueing process 𝑋^𝑖, based on which the transition probability matrix 𝑃^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]} has been derived. Suppose that it is differentiable with service rates 𝜇_𝑖∈Γ_𝑖. Denote 𝐴^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]}=𝑃^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]}−𝐼 as an infinitesimal generator of 𝑋^𝑖 under the service rates 𝜇_𝑖, where 𝑎_𝑘𝑘 and 𝑎_𝑘𝑗 for 𝑘≠𝑗 are elements of 𝐴^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]} and satisfy 𝑎_𝑘𝑘<0 and 𝑎_𝑘𝑗≥0 for 𝑘≠𝑗, 𝑘,𝑗∈Φ. Thus, the infinitesimal generator is differentiable with respect to 𝜇_𝑖. Note that the elements of 𝐴^𝜇𝑖_{[(𝐾+1)×(𝐾+1)]} represent the transition rate of the packet number in each user queue. More importantly, the cost-benefit optimization algorithm can be further developed for all 𝑀/𝐺/1-type user queues in Section 3 by changing the service rates 𝜇_𝑖 allocated so that the corresponding infinitesimal generator of each user queue is modified.

3. Resource Allocation Algorithm

In this section, we take a fresh look at the problem of resource allocation from the perspective of system performance cost and explore a cost-benefit gradient algorithm that minimizes the performance cost for all concurrent 𝑀/𝐺/1-type queues, subject to the bandwidth constraint.

3.1. Problem Formulation and Optimality Criterion

Since we focus on the stochastic dynamic queueing system, the estimation of its statistical properties is essential. In addition, such estimation needs not only to be accurate but more importantly, to be efficient when taking into consideration the delay sensitiveness of the real-time network applications. Consider that the main tenet of perturbation analysis (PA) is that a great deal of information is contained in the sample paths of a dynamic system, beyond the usual statistics collected such as the means and variances of various variables [15]. Thus, in essence, we can estimate the performance gradient with respect to the service rates and further minimize the user queue's performance cost measure based on a single sample path with PA.

In particular, several PA approaches have been introduced in solving network problems (see, e.g., [16, 17]). However, a general approach that supports a wide range of stochastic optimization problems awaits to be proposed. A new approach was proposed in [18] to analyze a number of Markov systems based on a single sample path. Moreover, the optimization formulations for Markov [14, 19, 20], semi-Markov [21], and partially observable Markov [22] systems were proposed, and in [3], the theory has successfully been extended to evaluate the 𝑀/𝐺/1 queueing systems. The structure of PA-based queueing system is shown in Figure 2. For each feasible resource allocation policy, a set of service rates are allocated to each user queue. With each change of one user queue's service rate, a perturbation is generated on the queue's sample path, which has effect on the system performance cost.

Figure 2 

Structure of user queueing system with PA.

As illustrated in Figure 3, in an 𝑀/𝐺/1-type user queue {𝑌^𝑖_𝑚;𝑚=0,1,2,…}, such a perturbation can be regarded as a “jump” among its states 𝑘∈Φ and has effect on the performance cost 𝜂^𝜇𝑖_𝑓. Thus, we need to measure all states' effect on the performance cost 𝜂^𝜇𝑖_𝑓 before discussing the performance cost optimization. We briefly introduce a concept called performance potential that is useful in this paper [18].

Figure 3 

A perturbation of a single sample path.

Definition 3.1 (performance potential). Denote 𝑔^𝜇𝑖(𝑘) as the 𝑖th user queue's performance potential of state 𝑘∈Φ under service rates 𝜇_𝑖 with respect to the performance cost function 𝑓_𝑖. It measures the effect of state 𝑘 to the performance cost 𝜂^𝜇𝑖_𝑓 and can be written as 𝑔^𝜇𝑖(𝑘)=𝐸{^∞_𝑛=0[𝑓_𝑖(𝑌^𝑖_𝑚,𝜇_𝑖)−𝜂^𝜇𝑖_𝑓]∣𝑌^𝑖₀=𝑘}. (3.1)

Remark 3.2. The performance potential vector of the 𝑀/𝐺/1/𝐾 user queue {𝑌^𝑖_𝑚} with respect to the performance cost function 𝑓^𝜇𝑖 is denoted as 𝑔^𝜇𝑖=(𝑔^𝜇𝑖(0),…,𝑔^𝜇𝑖(𝐾))^𝑇. In essence, the performance potential vector is the solution of the Poisson equation, which has been studied remarkably in the literature [23]: 𝐴^𝜇𝑖𝑔^𝜇𝑖=−𝑓^𝜇𝑖+𝜂^𝜇𝑖_𝑓𝑒. (3.2) Furthermore, if {𝑌^𝑖_𝑚} is strongly ergodic, all of the performance potential vectors can be calculated by 𝑔^𝜇𝑖=−(𝐴^𝜇𝑖)^#𝑓^𝜇𝑖+𝑐𝑒,𝑐∈ℛ, where (𝐴^𝜇𝑖)^# is said to be the group inverse (for details, see [18]) of the {𝑌^𝑖_𝑚}'s infinitesimal generator 𝐴^𝜇𝑖 under service rates 𝜇_𝑖.

Now, we need to assign the feasible service rates region to each active user queue. It is well-studied from queueing theory that an 𝑀/𝐺/1-type user queue {𝑌^𝑖_𝑚;𝑚=0,1,2,…} is called stable if 𝜌_𝑖<1, that is, 𝜆_𝑖<𝜇_𝑖. Moreover, the network element is stable if and only if all individual user queues are stable. Suppose that the lower bound service rates of each user queue at epoch 𝑡≥0 are set to 𝜇_𝑖^min(𝑡)=𝜆_𝑖+1, and ∑^𝑁(𝑡)_𝑖=1𝜇_𝑖^min(𝑡)<𝐶_𝜄. Thus, we denote the leftover service capacity as 𝐶_𝑜(𝑡)≜𝐶_𝜄−∑^𝑁(𝑡)_𝑖=1𝜇_𝑖^min(𝑡). In this state, we further assign the 𝑖th user queue a parameter called sharing weight 𝜙_𝑖𝜆(𝑡)≜^𝑖∑^𝑁(𝑡)_𝑖=1𝜆_𝑖. (3.3) Apparently, we have ∑^𝑁(𝑡)_𝑖=1𝜙_𝑖(𝑡)=1 for any given 𝑡. Thus, at epoch 𝑡, the service rates of user queue 𝑖 are upper bounded by,𝜇_𝑖^max(𝑡)=𝜇_𝑖^min(𝑡)+𝜙_𝑖(𝑡)𝐶_𝑜(𝑡). (3.4) Therefore, the service rates policy space in Figure 2 is defined as follows.

Definition 3.3 (feasible policy space). The policy space for all user queues at epoch 𝑡≥0 can be denoted as a compact set Ω(𝑡), where Ω(𝑡)≜Γ₁(𝑡)×⋯×Γ_𝑁(𝑡)(𝑡), and Γ_𝑖(𝑡)≜[𝜇_𝑖^min(𝑡),𝜇_𝑖^max(𝑡)],𝑖∈{1⋅⋅⋅𝑁(𝑡)}.

Next, we analyze the optimal criterion for the performance cost optimization.

Theorem 3.4 (optimality criterion). A cost-benefit resource allocation policy 𝜈^∗(𝑡)≜(𝜇^∗₁(𝑡),…,𝜇^∗_𝑁(𝑡)(𝑡))∈Ω(𝑡) is optimal with each given initial policy if and only if for each user queue 𝑖, one has 𝜒_𝑖≜minimize_{𝜇𝑖(𝑡)∈Γ𝑖(𝑡)}{𝑓^{𝜇𝑖(𝑡)}+𝐴^{𝜇𝑖(𝑡)}𝑔^{𝜇∗𝑖(𝑡)}−𝑒𝜂^𝜇∗𝑖_𝑓^(𝑡)}=0,∀𝑖∈{1,…,𝑁(𝑡)}. (3.5)

Proof. Note that Γ_𝑖(𝑡) is a compact set, and 𝑓^{𝜇𝑖(𝑡)}+𝐴^{𝜇𝑖(𝑡)}𝑔^{𝜇𝑖(𝑡)} is component-wise continuous on Γ_𝑖(𝑡). Thus, there must exist at least one cost optimal service rates in Γ_𝑖(𝑡).
According to [20], we have the fact that the service rates 𝜇^∗_𝑖(𝑡) for 𝑖th user queue is cost optimal if and only if𝑓^{𝜇∗𝑖(𝑡)}+𝐴^{𝜇∗𝑖(𝑡)}𝑔^{𝜇∗𝑖(𝑡)}⪯𝑓^{𝜇𝑖(𝑡)}+𝐴^{𝜇𝑖(𝑡)}𝑔^{𝜇∗𝑖(𝑡)},∀𝜇_𝑖(𝑡)∈Γ_𝑖(𝑡), (3.6) where the symbol ⪯ denotes vector inequality or component-wise inequality in ℛ^(𝐾+1), and (𝐾+1) is the the states' number of each user queue.
Note that a better cost-benefit service rates can be searched based on the comparison of current service rate. Besides, from (3.2), we have 𝑒𝜂^𝜇∗𝑖_𝑓^(𝑡)=𝑓^{𝜇∗𝑖(𝑡)}+𝐴^{𝜇∗𝑖(𝑡)}𝑔^{𝜇∗𝑖(𝑡)}.
Thus, we can conclude that if the 𝜇^∗_𝑖(𝑡) is the cost optimal service rates, the following equation:𝜒_𝑖=minimize_{𝜇𝑖(𝑡)∈Γ𝑖(𝑡)}{𝑓^{𝜇𝑖(𝑡)}+𝐴^{𝜇𝑖(𝑡)}𝑔^{𝜇∗𝑖(𝑡)}−𝑒𝜂^𝜇∗𝑖_𝑓^(𝑡)}=0 (3.7) is established and vice versa.

3.2. Gradient-Based Policy Optimization

Now, the purpose is to develop an efficient and practical policy algorithm, which minimizes all the user queues' performance cost based on the corresponding sample paths. In essence, the objective function for each user queue in Definition 2.5 represents the time-average performance measure of the 𝑀/𝐺/1 queue. Thus, developing a global optimization algorithm for such a performance measure will greatly increase the complexity. More importantly, to some extent, it is impractical to consider both the delay sensitiveness of user applications and the dynamic changes of the network element. In this state, a fast gradient-based optimization for the stochastic system is considered. It is well studied that a gradient optimization algorithm is to find a local minimum of objective function; however, it can be fairly efficient, especially when the interval Γ_𝑖(𝑡) for each user queue 𝑖 is not very large. To begin with, the performance gradient formula for each user queue is derived as follows.

Theorem 3.5 (performance gradient). For any given resource allocation policy 𝜈(𝑡)=(𝜇₁(𝑡),…,𝜇_𝑁(𝑡)(𝑡))∈Ω(𝑡) at each event time 𝑡≥0,𝑘=1,2,…, the gradient of the performance cost 𝜂^𝜇𝑖_𝑓^(𝑡) generated by the 𝑖th user queue is obtained by ∇𝜂^𝜇𝑖_𝑓^(𝑡)=𝜋^{𝜇𝑖(𝑡)}∇𝑃^{𝜇𝑖(𝑡)}𝑔^𝜇𝑖_𝑓^(𝑡)+𝜋^{𝜇𝑖(𝑡)}∇𝑓^{𝜇𝑖(𝑡)}. (3.8)

Proof. By taking the gradient of the performance measure (2.8), we have ∇𝜂^𝜇𝑖_𝑓^(𝑡)=∇𝜋^{𝜇𝑖(𝑡)}𝑓^{𝜇𝑖(𝑡)}+𝜋^{𝜇𝑖(𝑡)}∇𝑓^{𝜇𝑖(𝑡)}. (3.9) From [14, 20], there must be particular solution to (3.2), such that 𝜂^𝜇𝑖_𝑓^(𝑡)=𝜋^{𝜇𝑖(𝑡)}𝑔^{𝜇𝑖(𝑡)}; thus, we obtain (−𝐴^{𝜇𝑖(𝑡)}+𝑒𝜋^{𝜇𝑖(𝑡)})𝑔^{𝜇𝑖(𝑡)}=𝑓^{𝜇𝑖(𝑡)}. (3.10) With 𝑃^{𝜇𝑖(𝑡)}=𝐴^{𝜇𝑖(𝑡)}+𝐼, we further have (𝐼−𝑃^{𝜇𝑖(𝑡)}+𝑒𝜋^{𝜇𝑖(𝑡)})𝑔^{𝜇𝑖(𝑡)}=𝑓^{𝜇𝑖(𝑡)}. (3.11) Left-multiply by ∇𝜋^{𝜇𝑖(𝑡)} on both sides of (3.11), it follows that ∇𝜋^{𝜇𝑖(𝑡)}𝑓^{𝜇𝑖(𝑡)}=∇𝜋^{𝜇𝑖(𝑡)}(𝐼−𝑃^{𝜇𝑖(𝑡)}+𝑒𝜋^{𝜇𝑖(𝑡)})𝑔^{𝜇𝑖(𝑡)}. (3.12) Recall that 𝜋^{𝜇𝑖(𝑡)}𝑒=1,𝑃^{𝜇𝑖(𝑡)}𝑒=𝑒,and𝜋^{𝜇𝑖(𝑡)}𝑃^{𝜇𝑖(𝑡)}=𝜋^{𝜇𝑖(𝑡)}, then we have ∇𝜋^{𝜇𝑖(𝑡)}𝑃^{𝜇𝑖(𝑡)}+𝜋^{𝜇𝑖(𝑡)}∇𝑃^{𝜇𝑖(𝑡)}=∇𝜋^{𝜇𝑖(𝑡)},∇𝜋^{𝜇𝑖(𝑡)}𝑒=0. (3.13) Hence, using (3.12), it suffices to show that ∇𝜋^{𝜇𝑖(𝑡)}𝑓^{𝜇𝑖(𝑡)}=𝜋^{𝜇𝑖(𝑡)}∇𝑃^{𝜇𝑖(𝑡)}𝑔^{𝜇𝑖(𝑡)}. (3.14) Combining (3.9) with (3.14), the result then follows.

Then, we proceed to describe the process flow of the policy gradient algorithm shown in Figure 4. The procedure of the algorithm is described in Algorithm 1. Note that in Algorithm 1,𝑠𝑝(ℎ)≜max_𝑖{ℎ(𝑖)}−min_𝑖{ℎ(𝑖)},ℎ∈ℛ^𝑛 (3.15) is defined as the span seminorm on ℛ^𝑛.

Input: An arbitrary initial policy $\nu (t)=({\mu }_1(t),\dots ,{\mu }_{N(t)}(t))\in \mathrm{\Omega }(t)$ at given $t\ge 0$. Output: The optimal policy for all user queues ${\nu }^{*}(t)=({\mu }_1^{*}(t),\dots ,{\mu }_{N(t)}^{*}(t))\in \mathrm{\Omega }(t)$.      Procedure:  ($1$) Choose $0<ϵ\ll 1$ as the stopping criterion for all user queues.  ($2$) for  queue $i=1$ to $N(t)$  do  ($3$)  repeat  ($4$)   Set iteration index $l=0$.  ($5$)   Calculate ${\pi }^{{\mu }_i^l(t)}$ and $g^{{\mu }_i^l(t)}$ by solving (2.6) and (3.2), respectively.  ($6$)   Determine the gradient such that:             $\nabla {\eta }_f^{{\mu }_i^l(t)}={\pi }^{{\mu }_i^l(t)}\nabla P^{{\mu }_i^l(t)}{g}^{{\mu }_i^l(t)}+{\pi }^{{\mu }_i^l(t)}\nabla f^{{\mu }_i^l(t)}.$  ($7$)   Do line search along the gradient, choose the right step size ${\gamma }_i^l$.  ($8$)   Update service rates ${\mu }_i^{l+1}(t):={\mu }_i^l(t)-{\gamma }_i^l\nabla {\eta }_f^{{\mu }_i^l(t)}$.  ($9$)   Set $l:=l+1$.  ($10$)  Until  $sp(f^{{\mu }_i^{l+1}(t)}+A^{{\mu }_i^{l+1}(t)}\mathrm{\hspace{0.17em}\hspace{0.17em}}{g}^{{\mu }_i^l(t)})<ϵ$  or  ${\mu }_i^{l+1}(t)\notin {\mathrm{\Gamma }}_i(t)$.  ($11$) end  for

Algorithm 1 

Policy gradient algorithm.

Figure 4 

Policy gradient algorithm procedure.

Moreover, the construction of the algorithm is presented as follows. The algorithm begins by choosing an arbitrary feasible policy for all user queues at given time 𝑡. Then, with current service rates, the corresponding performance gradient is calculated by analyzing the sample path of each user queue. Based on the line search along the gradient, the right step size is obtained. Thus, a better policy can be updated for each user queue. By iteration until the stopping criterion is met, the optimal cost-benefit resource allocation policy can finally be achieved.

Without loss of generality, suppose that every gradient iteration of each user queue leads to an improving performance cost, that is,𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)}⪯𝑓^{𝜇𝑙𝑖(𝑡)}+𝐴^{𝜇𝑙𝑖(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)},∀𝑖∈{1,…,𝑁(𝑡)}, (3.16) where 𝑙∈𝒵⁺ represents the iteration index in Algorithm 1. The convergence property of the policy gradient algorithm is evaluated as follows.

Theorem 3.6 (convergence property). Consider 𝜈^𝑙(𝑡)=(𝜇^𝑙₁(𝑡),…,𝜇^𝑙_𝑁(𝑡)(𝑡))∈Ω(𝑡),𝑙∈𝒵⁺ is a performance improving resource allocation policy sequence at each given time 𝑡≥0, then forall𝑖∈{1,…,𝑁(𝑡)}; one has:(a)lim_𝑙→∞𝑠𝑝(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}g^{𝜇𝑙𝑖(𝑡)})=0, (3.17)(b)That there must exist an optimal cost-benefit resource allocation policy, denoted as 𝜈^∗(𝑡)=(𝜇^∗₁(𝑡),…,𝜇^∗_𝑁(𝑡)(𝑡))∈Ω(𝑡), such that lim_𝑙→∞𝜇^𝑙_𝑖(𝑡)=𝜇^∗_𝑖(𝑡),lim_𝑙→∞𝜂^𝜇𝑙𝑖_𝑓^(𝑡)=𝜂^𝜇∗𝑖_𝑓^(𝑡). (3.18)

Proof. For part (a) Since 𝜈^𝑙(𝑡)=(𝜇^𝑙₁(𝑡),…,𝜇^𝑙_𝑁(𝑡)(𝑡))∈Ω(𝑡),𝑙∈𝒵⁺,𝑖∈{1,…,𝑁(𝑡)} is a performance improving resource allocation policy sequence, we can conclude that for each user queue 𝑖, {𝜂^𝜇𝑙𝑖_𝑓^(𝑡)} is a monotonously decreasing and bounded performance cost sequence, with lower bound 𝜂^𝜇∗𝑖_𝑓^(𝑡).
By using the continuity of 𝜂^𝜇𝑖_𝑓^(𝑡), we obtain a service rates 𝜇_𝑖(𝑡)∈Γ_𝑖(𝑡) for each user queue 𝑖, satisfying 𝜂^𝜇𝑖_𝑓^(𝑡)=𝑐∈ℛ⁺ and lim_𝑙→∞𝜇^𝑙_𝑖(𝑡)=𝜇_𝑖(𝑡).
Thus, it follows thatlim_𝑙→∞(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})=𝑓^{𝜇𝑖(𝑡)}+𝐴^{𝜇𝑖(𝑡)}𝑔^{𝜇𝑖(𝑡)}=𝑒𝜂^𝜇𝑖_𝑓^(𝑡), (3.19) which is equivalent to lim_𝑙→∞{(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})(𝑘)}=𝜂^𝜇𝑖_𝑓^(𝑡),∀𝑘∈Φ. (3.20)
Recall that𝑠𝑝(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})=max_𝑘∈Φ{(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})(𝑘)}−min_𝑘∈Φ{(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})(𝑘)}; (3.21) by taking limit as 𝑙→∞, we further have lim_𝑙→∞𝑠𝑝(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})=𝜂^𝜇𝑖_𝑓^(𝑡)−𝜂^𝜇𝑖_𝑓^(𝑡)=0. (3.22)
For part (b) Note that for all𝜖>0,∃𝑙₀∈𝒵⁺, such that 𝑠𝑝(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})<𝜖,holds when 𝑙>𝑙₀.
To show the second relation of the theorem, considering 𝑙>𝑙₀; it follows that𝜂^{𝜇𝑖𝑙+1}_𝑓^(𝑡)=𝜋^{𝜇𝑖𝑙+1(𝑡)}𝑓^{𝜇𝑖𝑙+1(𝑡)}=𝜋^{𝜇𝑖𝑙+1(𝑡)}(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})≤max_𝑘∈Φ{(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})(𝑘)}. (3.23)
Since in Algorithm 1, 𝜇_𝑖^𝑙+1(𝑡) is the best choice for 𝜇^𝑙_𝑖(𝑡) along the gradient by the line search, we therefore have𝜂^𝜇∗𝑖_𝑓^(𝑡)=𝜋^{𝜇∗𝑖(𝑡)}(𝑓^{𝜇∗𝑖(𝑡)}+𝐴^{𝜇∗𝑖(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})≥𝜋^{𝜇∗𝑖(𝑡)}(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})≥min_𝑘∈Φ{(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})(𝑘)}. (3.24)
Subtracting (3.23) by (3.24), for ∀𝑙>𝑙₀, it follows that0<𝜂^{𝜇𝑖𝑙+1}_𝑓^(𝑡)−𝜂^𝜇∗𝑖_𝑓^(𝑡)≤𝑠𝑝(𝑓^{𝜇𝑖𝑙+1(𝑡)}+𝐴^{𝜇𝑖𝑙+1(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)})<𝜖. (3.25)
Thus, it can be regarded that 𝜇_𝑖^𝑙+1(𝑡) is an 𝜖-optimal service rates, and we can write lim_𝑙→∞𝜂^{𝜇𝑖𝑙+1}_𝑓^(𝑡)=𝜂^𝜇∗𝑖_𝑓^(𝑡) by the arbitrariness of 𝜖.
Finally, from the fundamental fact of the uniqueness principle of limitation theory, we can conclude that 𝜂^𝜇∗𝑖_𝑓^(𝑡)=𝜂^𝜇𝑖_𝑓^(𝑡), which is equivalent to say that 𝜇_𝑖(𝑡) is an optimal cost-benefit service rates. This leads immediately to the results that lim_𝑙→∞𝜇^𝑙_𝑖(𝑡)=𝜇^∗_𝑖(𝑡) and lim_𝑙→∞𝜂^𝜇𝑙𝑖_𝑓^(𝑡)=𝜂^𝜇∗𝑖_𝑓^(𝑡).

Remark 3.7. By proving the convergence property of the policy gradient algorithm, the cost-benefit resource allocation optimization approach for the 𝑀/𝐺/1/𝐾 user queues has been proposed. In a broad sense, the service time allocated to each user queue 𝑖 has been considered as a general distribution 𝐺_𝑖. However, the mathematical expression of performance gradient is not unique according to different application scenarios.

3.3. Performance Gradient Analysis of Application Scenarios

To make the analysis more tractable, we now present two application scenarios for which the performance gradient can be explicitly derived.

Deterministic inter-service time.
This is the simplest practical case where inter-service times for each user queue 𝑖 are deterministic, that is, 𝑀/𝐷/1/𝐾. Without loss of generality, we denote each inter-service time as a constant 1/𝜇_𝑖. Consider now that 𝜌_𝑖=𝜆_𝑖/𝜇_𝑖<1. Let 𝑃^𝜇𝑖 be the transfer matrix of the embedded Markov chain. In this scenario, hence, we have that for all 𝑘∈Φ, the element 𝑎^𝑖_𝑘 of the transfer matrix is equal to, 𝑎^𝑖_𝑘=𝑒^−𝜌𝑖𝜌^𝑘_𝑖,𝑘! (3.26) implying that the element of steady-state probability vector 𝜋^𝜇𝑖(𝑘),𝑘∈Φ satisfies 𝜋^𝜇𝑖(𝑘)=1−𝜌_𝑖,if𝑘=0,(1−𝜌_𝑖)(𝑒^𝜌𝑖−1),if𝑘=1,(1−𝜌_𝑖)^𝑘_𝑗=1(−1)^𝑘−𝑗𝑒^{−𝑗𝜌𝑖}[(𝑗𝜌_𝑖)^𝑘−𝑗+(𝑘−𝑗)!(𝑗𝜌_𝑖)^{𝑘−𝑗−1}(𝑘−𝑗−1)!],otherwise. (3.27) According to (3.26), hence, we have 𝜕𝑎^𝑖_𝑘𝜕𝜇_𝑖=𝜆_𝑖𝜇²_𝑖𝑒^−𝜌𝑖=𝜌_𝑖𝜇_𝑖𝑎^𝑖_𝑘,if𝜆𝑘=0,_𝑖𝑒^−𝜌𝑖𝜇²_𝑖[𝜌^𝑘_𝑖−𝜌𝑘!^𝑘−1𝑖𝜌𝑘−1!]=^𝑖𝜇_𝑖(𝑎^𝑖_𝑘−𝑎^𝑖_𝑘−1),otherwise. (3.28)

Corollary 3.8. Considering the 𝑖th user queue in the steady state, from Theorem 3.5, the performance gradient with respect to the service rates 𝜇_𝑖 for the case of 𝑀/𝐷/1/𝐾 is𝜕𝜂^𝜇𝑖_𝑓𝜕𝜇_𝑖=𝜌_𝑖𝜇_𝑖^𝐾_𝑘=0[𝜋^𝜇𝑖(𝑘)−𝜋^𝜇𝑖(𝑘−1)]𝑔^𝜇𝑖(𝑘)+𝜋^𝜇𝑖𝜕𝑓𝜕𝜇_𝑖, (3.29) where 𝜋^𝜇𝑖=(𝜋^𝜇𝑖(0),…,𝜋^𝜇𝑖(𝐾)),𝑔^𝜇𝑖=(𝑔^𝜇𝑖(0),…,𝑔^𝜇𝑖(𝐾)), and 𝜋^𝜇𝑖(−1)=0.

Exponential inter-service time.
Having considered the straightforward scenario of deterministic service rates, we will now investigate the 𝑀/𝑀/1/𝐾 case where the inter-service times are independent and exponentially distributed, that is, memoryless. For ease of notation, we show results using the same parameters as in the first scenario. Similarly, in this case, we derive the element 𝑎^𝑖_𝑘,forall𝑘∈Φ of the transfer matrix as follows: 𝑎^𝑖_𝑘=𝜌^𝑘_𝑖(1+𝜌_𝑖)^𝑘+1; (3.30) it suffices to show that, 𝜋^𝜇𝑖(𝑘)=(1−𝜌_𝑖)𝜌^𝑘_𝑖,𝜕𝑎^𝑖_𝑘𝜕𝜇_𝑖=𝜌_𝑖−𝑘𝜇_𝑖(1+𝜌_𝑖)𝑎^𝑖_𝑘. (3.31)

Corollary 3.9. Considering the 𝑖th user queue in the steadystate, from Theorem 3.5, the performance gradient with respect to the service rates 𝜇_𝑖 for the case of 𝑀/𝑀/1/𝐾 is 𝜕𝜂^𝜇𝑖_𝑓𝜕𝜇_𝑖=𝜋^𝜇𝑖𝜕𝑓𝜕𝜇_𝑖−^𝐾_𝑘=0𝑏^𝑖_𝑘𝑔^𝜇𝑖(𝑘), (3.32) where 𝑏^𝑖_𝑘=𝜋^𝑖(0)𝑘−𝜌_𝑖1+𝜌_𝑖𝑎^𝑖_𝑘+^𝑘_𝑗=0𝜋^𝑖(𝑗+1)𝑘−𝑗−𝜌_𝑖1+𝜌_𝑖𝑎^𝑖_𝑘−𝑗. (3.33)

Remark 3.10. Note that the essential feature behind the cost-benefit resource allocation is the performance gradient corresponding to the service rates. Based on this, the policy gradient algorithm for each user queue is executed, until the performance cost of network system is minimized.

4. Performance Evaluation

In this section, we investigate the performance of the 𝑀/𝐺/1-type queueing system. Before proceeding, we first present the simulation model, namely, the sample path-based simulation scheme.

4.1. Simulation Model

To evaluate the performance of Algorithm 1 for each user queue 𝑖, it is imperative to calculate both the steady-state probability vector 𝜋^{𝜇𝑙𝑖(𝑡)} and the performance potential 𝑔^{𝜇𝑙𝑖(𝑡)} with any feasible service rates 𝜇^𝑙_𝑖(𝑡) in every iteration 𝑙=0,1,2,….

According to the Borel property [24], the steady-state probability of state 𝑘 in the embedded Markov chain 𝑌^𝑖_𝑚 has an unbiased estimate as follows:𝜋^𝜇𝑖1(𝑘)=𝜁^𝜁−1_𝑛=0𝐼_𝑘(𝑌^𝑖_𝑚)∀𝑘∈Φ, (4.1) where𝐼_𝑘(𝑌^𝑖_𝑚)≜1,𝑌^𝑖_𝑚=𝑘,0,𝑌^𝑖_𝑚≠𝑘, (4.2) and 𝜁 denotes the transfer number of the queue states and is set to 10000 in this simulation. Once the steady-state probability vector has been estimated, the estimation of the performance cost 𝜂^𝜇𝑖_𝑓 for the user queue can be derived by 𝜂^𝜇𝑖_𝑓=𝜋^𝜇𝑖𝑓^𝜇𝑖.

However, solving the Poisson equation in (3.2) for achieving 𝑔^{𝜇𝑙𝑖(𝑡)} leads to significant computational complexity, which is impractical for the online cost-benefit optimization. Therefore, a sample path-based estimation is very essential. Note that the performance potential can be estimated as𝑔^𝜇𝑖=−𝐷^𝜇𝑖_𝑓𝜋^𝜇𝑖𝑇, (4.3) where 𝐷^𝜇𝑖_𝑓=[𝑑^𝜇𝑖_𝑘𝑗]_{[(𝐾+1)×(𝐾+1)]},𝑘,𝑗∈Φ is called the realization matrix, and𝑑^𝜇𝑖_𝑘𝑗≜𝐸{^{𝐿{𝑘∣𝑗}−1}_𝑛=0[𝑓_𝑖(𝑌^𝑖_𝑚,𝜇_𝑖)−𝜂^𝜇𝑖_𝑓]∣𝑌^𝑖₀=𝑗}, (4.4) the first passage time 𝐿{𝑘∣𝑗} from state 𝑗 to state 𝑘, is expressed as 𝐿{𝑘∣𝑗}≜inf{𝑛≥0∶𝑌^𝑖_𝑚=𝑘∣𝑌^𝑖₀=𝑗}.

Thus by Theorem 3.5, the gradient corresponding to the service rates 𝜇^𝑙_𝑖(𝑡) allocated in the 𝑙th iteration for the user queue 𝑖 can be efficiently estimated by∇𝜂^𝜇𝑙𝑖_𝑓^(𝑡)=𝜋^{𝜇𝑙𝑖(𝑡)}∇𝑃^{𝜇𝑙𝑖(𝑡)}𝑔^{𝜇𝑙𝑖(𝑡)}+𝜋^{𝜇𝑙𝑖(𝑡)}∇𝑓^{𝜇𝑙𝑖(𝑡)}. (4.5)

4.2. Numerical Results

The following describes numerical examples to illustrate the analytical results derived in the previous sections. Consider a given time 𝑡₀>0 and that there are four active user queues in the network element. Here, we limit our experimental tests to the simulation parameters values that are depicted in Table 2. Note that the value of 𝜆_𝑖(𝑡₀), 𝜇_𝑖^min(𝑡₀), and 𝜇_𝑖^max(𝑡₀) can be considered as packet numbers transmitted per unit time.

Table 2 

Simulation Parameters.

Besides, the backlog capacity of every user queue is set to 100 packets. The performance cost function of the sharing user queues is considered as 𝑓_𝑖(𝑘,𝜇_𝑖)=𝑐₁𝑘+𝑐₂𝜇_𝑖,𝑖=1,2,3,4,𝑘∈Φ, where 𝑐₁,𝑐₂>0 are constants. In this experiment, they are set to 80 and 1, respectively. Moreover, the stopping criterion 𝜖 in Algorithm 1 is set to 0.001, and the link service capacity 𝐶_𝜄 is set to 200 packets per unit time. By choosing the initial resource allocation policy as the minimum service rates for all user queues, the simulation results are described in Table 3. Finally, the iteration processes of the four user queues are shown in Figure 5.

Table 3 

Optimal resource allocation policy.

(a)

(b)

(c)

(d)

Figure 5 

The iteration processes of the four user queues.

Based on the observation on Figure 5, we can conclude the following. (i)Given the the stopping criterion 𝜖=0.001 and each feasible region Γ_𝑖(𝑡₀),𝑖∈{1,…,𝑁(𝑡₀)}, all iterations for user queues can converge within 15 steps. (ii)During the algorithm iterative process, the corresponding performance cost has been reduced by 56.3%,55.7%,52.3%, and 44.5%, respectively for each user queue. (iii)The optimal service rates for each user queue (e.g., 𝜇^∗₁(𝑡₀)) may not be achieved on the boundary (i.e., 𝜇₁^min(𝑡₀) or 𝜇₁^max(𝑡₀)) of the feasible region shown in Table 2.

Remark 4.1. Note that the convergence rate of the proposed algorithm is closely related with the estimation of theperformance gradients. The convergence rate of the algorithm will grow faster with a more accurate estimation. More precisely, according to Theorem 3.5, the performance gradient can be calculated via the steady-state probability vector 𝜋^{𝜇𝑙𝑖(𝑡)} and the performance potential 𝑔^{𝜇𝑙𝑖(𝑡)} in every iteration 𝑙∈𝒵⁺. Thus, we can increase the number of queue states' transfers, thereby achieving the estimation accuracy of steady-state probability vector and performance potential, or equivalently, performance gradient. In addition, the reason for the location of the optimal service rates is that there exists a trade-off between the backlog-related and service rate-related performance cost, which can be adjusted by the definition of the performance cost function.

5. Conclusions

In this paper, performance optimization problems of communication networks with stochastic characteristics are studied. To describe this complex dynamic process of system behavior, all user queues in each network element are represented by multiple concurrent 𝑀/𝐺/1-type Markov processes such that system model is proposed. Furthermore, an efficient algorithm is developed for the optimization of system performance cost by using sensitivity analysis approach. During every iteration, the proposed algorithm estimates the derivative of the performance measure and the performance potential by analyzing a single sample path of each user queue, which implies its computational efficiency. The asymptotical convergence analysis, combined with the numerical examples, paves the way for designing cost-aware computer communications systems.

Acknowledgments

This work was supported by National Natural Science Foundation of China under Grant nos. 60904021 and 60774038 and National High-Tech Research and Development Program of China (863 Program) under Grant no. 2008AA01A317.