Modeling and Optimization of 𝑀/𝐺/𝟏-Type Queueing Networks: An Efficient Sensitivity Analysis Approach

Algorithm 1

Policy gradient algorithm.

Input: An arbitrary initial policy $\nu (t)=({\mu }_1(t),\dots ,{\mu }_{N(t)}(t))\in \mathrm{\Omega }(t)$ at given $t\ge 0$.

Output: The optimal policy for all user queues ${\nu }^{*}(t)=({\mu }_1^{*}(t),\dots ,{\mu }_{N(t)}^{*}(t))\in \mathrm{\Omega }(t)$.

     Procedure:

 ($1$) Choose $0<ϵ\ll 1$ as the stopping criterion for all user queues.

 ($2$) for  queue $i=1$ to $N(t)$  do

 ($3$)  repeat

 ($4$)   Set iteration index $l=0$.

 ($5$)   Calculate ${\pi }^{{\mu }_i^l(t)}$ and $g^{{\mu }_i^l(t)}$ by solving (2.6) and (3.2), respectively.

 ($6$)   Determine the gradient such that:

            $\nabla {\eta }_f^{{\mu }_i^l(t)}={\pi }^{{\mu }_i^l(t)}\nabla P^{{\mu }_i^l(t)}{g}^{{\mu }_i^l(t)}+{\pi }^{{\mu }_i^l(t)}\nabla f^{{\mu }_i^l(t)}.$

 ($7$)   Do line search along the gradient, choose the right step size ${\gamma }_i^l$.

 ($8$)   Update service rates ${\mu }_i^{l+1}(t):={\mu }_i^l(t)-{\gamma }_i^l\nabla {\eta }_f^{{\mu }_i^l(t)}$.

 ($9$)   Set $l:=l+1$.

 ($10$)  Until  $sp(f^{{\mu }_i^{l+1}(t)}+A^{{\mu }_i^{l+1}(t)}\mathrm{\hspace{0.17em}\hspace{0.17em}}{g}^{{\mu }_i^l(t)})<ϵ$  or  ${\mu }_i^{l+1}(t)\notin {\mathrm{\Gamma }}_i(t)$.

 ($11$) end  for