Intelligent Inventory Control via Ruminative Reinforcement Learning

Algorithm 1

RSarsa algorithm.

(L00) Initialize 𝑄(𝑠,𝑎).

(L01) Observe 𝑠.

(L02) Determine 𝑎 by policy 𝜋.

(L03) For each period,

(L04)  observe 𝑟,𝑠^′, and 𝜉;

(L05)  determine 𝑎^′ by policy 𝜋;

(L06)  calculate 𝜓=𝑟+𝛾𝑄(𝑠^′,𝑎^′)−𝑄(𝑠,𝑎);

(L07)  update 𝑄(𝑠,𝑎)←𝑄(𝑠,𝑎)+𝛼⋅𝜓;

(L08)  for each 𝑎∈𝐴(𝑠),

(L09)   calculate 𝑟,̂𝑠^′̂ with 𝑘(𝑠,𝑎,𝜉),

(L10)    determine 𝑎^′,

(L11)     calculate 𝜓=𝑟̂+𝛾𝑄(𝑠^′̂,𝑎^′)−𝑄(𝑠,𝑎),

(L12)    update 𝑄(𝑠,𝑎)←𝑄(𝑠,𝑎)+𝛼⋅𝜓

(L13)  until ruminated all 𝑎∈𝐴;

(L14)  set 𝑠←𝑠^′ and 𝑎←𝑎^′

(L15) until termination.