Optimality of Multichannel Myopic Sensing in the Presence of Sensing Error for Opportunistic Spectrum Access

Abstract

The optimization problem for the performance of opportunistic spectrum access is considered in this study. A user, with the limited sensing capacity, has opportunistic access to a communication system with multiple channels. The user can only choose several channels to sense and decides whether to access these channels based on the sensing information in each time slot. Meanwhile, the presence of sensing error is considered. A reward is obtained when the user accesses a channel. The objective is to maximize the expected (discounted or average) reward accrued over an infinite horizon. This problem can be formulated as a partially observable Markov decision process. This study shows the optimality of the simple and robust myopic policy which focuses on maximizing the immediate reward. The results show that the myopic policy is optimal in the case of practical interest.

1. Introduction

There is a significant increase in the demand for radio spectrum with the emergence of new applications and the compelling need for mobile services in recent years. This is partly due to the increasing interest of consumers in convenient and ubiquitous wireless services, and the interest has been driving the evolution of wireless networks to high speed data networks. However, ever since the 1920s, in order to avoid the serious interference in wireless services, the wireless providers have been required to apply an exclusive license from the government. Today, it is becoming very difficult to find vacant bands to either deploy new services or to enhance the existing ones with most of the spectrum being already allocated [1]. On the other hand, not every channel in every band is in use all the time; a large number of vacant spectrum holes can be discovered in the spectrum [2]. A technique for opportunistic spectrum access can effectively utilize these spectrum holes.

The spectrum sensing for detecting spectrum holes is the precondition for opportunistic spectrum access. However, the existing spectrum sensing techniques has to face one main challenge: wideband sensing which is hard to be implemented for the main reason of hardware limitations [3]. The user usually uses a tunable narrowband bandpass filter at the radio frequency (RF) front-end to sense one channel at a time due to the costliness of a wideband RF front-end. Consequently, it is a lot of time delay for detecting all channels. Meng et al. [4] study this problem with the method of compressive sensing based on the sparse observations of sensing information. Equipped with frequency selective filters, the sparse sensing information vectors of multiple channels are linearly combined and compressed. Multiple channels thus can be sensed simultaneously. However, it is used in the case of less practical interest due to the requirements of sparse observations and frequency selective filters. The basic theory of compressive sensing is given in the work of [5–8]. Some other studies focus on the reliability of sensing information. Different SNR estimations and channel fading environments are considered in [9, 10] to improve the reliability of sensing information. Chen [11] studies the optimum number of collaborative users to get the tradeoff of the reliability and the complexity. The Byzantine attacks which come from malicious users and carry false sensing data are taken into account in [12].

The studies of [13–16] also exploit a method to solve the problem of wideband sensing by estimating the information of all channels with only a small amount of sensing results. The sensing procedure is modeled as a partially observed Markov decision process (POMDP). Zhao et al. [13] propose this idea and a myopic sensing method. Wang et al. [14] exploit the impact of the rateless code. Lingcen et al. [15] modify the cost function of POMDP with the switching time.

We consider the communication system where a user has opportunistic access to multiple channels like the model of [13] but is limited to sensing and transmitting only on several channels at a given time due to its hardware limitation. Meanwhile, the presence of sensing error is also considered. We explore the problem to maximize the performance of opportunistic access when the past observations and the knowledge of the stochastic properties of these channels are given. This problem can be described as a partially observable Markov decision process since the user does not have full knowledge of the availabilities of all channels. We examine the optimality of the myopic policy for this problem in this study for the reason that the myopic policy is very simple and robust. Specially, we show that the myopic policy is optimal in the case of practical interest. Ahmad et al. [16] also study the optimality of the myopic sensing. However, we have discovered that the study of [16] is only suitable for the special case where the one-channel myopic sensing and the absence of sensing error are considered. If we consider the multichannel myopic sensing or the presence of sensing error, the study of [16] cannot hold its conclusion. The reason is that the mathematic method of the proof of [16] is very special for the case of one-channel myopic sensing, and Lemmas 2, 3, 4, and 5 of [16] cannot be improved to prove the optimality under the other conditions. We propose the proof of the optimality of multichannel myopic sensing in the presence of sensing error. Our mathematic method is rigorous and quite different from the method of [16]; we use two functions to give the proof, and the method is generally effective for such issues.

The rest of this paper is organized as follows. We formulate the problem in Section 2 and give the definition of the myopic policy in Section 3. We prove the optimality of the myopic policy in Section 4 and extend the results from the finite horizon to the infinite horizon. The numerical results of the performance comparison of the myopic policy and the optimal policy are given in Section 5, and the conclusion is drawn in Section 6.

2. System Model

Consider a spectrum consisting of 𝑁 independent and statistically identical wireless channels; each channel has two states {0(𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑),1(𝑖𝑑𝑙𝑒)}, and the state transition is given by a two-state discrete time Markov chain shown in Figure 1. It is supposed that these channels evolve according to a synchronous time slot structure indexed by 𝑡, where 𝑡=1,2,…,𝑇. Specially, the states of all channels at time 𝑡 are denoted by 𝑆(𝑡)=[𝑆₁(𝑡),…,𝑆_𝑁(𝑡)], where 𝑆_𝑛(𝑡)∈{0,1}.

Figure 1 

State transition of channels.

We consider a user seeking the spectrum holes in these channels for opportunistic access. At the beginning of the time slot 𝑡, the user selects a set 𝐴₁(𝑡) of channels to sense and a set 𝐴₂(𝑡) of channels to access, where 𝐴₂(𝑡)⊆𝐴₁(𝑡). However, due to hardware limitation, |𝐴₁(𝑡)| and |𝐴₂(𝑡)| are usually much smaller than 𝑁. The user can only sense the channels in 𝐴₁(𝑡) and does not have the states of all channels 𝑆(𝑡), when a decision is made at time 𝑡. Consequently, the spectrum is not fully observable to the users. For clarity, we use 𝑎(𝑡)={𝐴₁(𝑡),𝐴₂(𝑡)} to denote the action of the user at time 𝑡 and use ℎ_𝐴1(𝑡)(𝑡)=[ℎ_𝑖1(𝑡),…,ℎ_{𝑖1|𝐴(𝑡)|}(𝑡)] to denote the sensing information of channels, where ℎ_𝐴1(𝑡)(𝑡)∈{0,1}^{|𝐴1(𝑡)|} and 𝑖_𝑘 is the index of the channel in 𝐴₁(𝑡). Specially, we cannot guarantee the absolute reliability of the sensing information in the presence of sensing error; the detection and false alarm probabilities should be taken into account. Here, the detection probability denoted by 𝑝^𝑖𝑘_𝑑(𝑡) and the false alarm probability denoted by 𝑝^𝑖𝑘_𝑓(𝑡) [17] are the conditional probabilities that the state of the channel 𝑖_𝑘 is actually 0(𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑) when the observations are 0(𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑) and 1(𝑖𝑑𝑙𝑒), respectively.

For making the optimal decision, a sufficient statistic of this system denoted by Λ(𝑡)=[𝜆₁(𝑡),…,𝜆_𝑁(𝑡)] is given, where 𝜆_𝑛(𝑡) is the conditional probability that the channel 𝑛 is idle at the beginning of the time slot 𝑡 given all past observations and actions, and this conditional probability is called the idle conditional probability. Due to the Markovian property of the channels, the future idle conditional probability is only a function of the current idle conditional probability and the action.

Proposition 1. The relationship between Λ(𝑡) and Λ(𝑡+1) can be given by (3).

Proof. We first consider case 1 that 𝑛∈𝐴₁(𝑡) and ℎ_𝑛(𝑡)=1 to simplify the proof; the events 𝑆_𝑛(𝑡+1)=1, 𝑆_𝑛(𝑡)=1, 𝑆_𝑛(𝑡)=0, and ℎ_𝑛(𝑡)=1 are denoted by 𝐴, 𝐵₁, 𝐵₂, and 𝐶, respectively. The value of 𝜆_𝑛(𝑡+1) in case 1 can be replaced by 𝑃{𝐴∣𝐶}; then we can get 𝑃{𝐴∣𝐶}=_𝑖={1,2}𝑃𝐴,𝐵_𝑖∣𝐶=_𝑖={1,2}𝑃𝐵_𝑖,𝐶𝑃{𝐶}𝑃𝐴,𝐵_𝑖,𝐶𝑃𝐵_𝑖,𝐶=_𝑖={1,2}𝑃𝐵_𝑖∣𝐶𝑃𝐴∣𝐵_𝑖,𝐶. (1)
Since 𝐴 is independent of 𝐶 when 𝐵_𝑖 has been determined, we can get 𝑃{𝐴∣𝐶}=_𝑖={1,2}𝑃𝐵_𝑖∣𝐶𝑃𝐴∣𝐵_𝑖=𝑝₁₁1−𝑝^𝑛_𝑓(𝑡)+𝑝₀₁𝑝^𝑛_𝑓(𝑡)𝜆_𝑛()=𝑃𝑆𝑡+1^𝑛(𝑡+1)=1∣ℎ_𝑛(𝑡)=1if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑃𝑆=1_𝑛(𝑡+1)=1∣ℎ_𝑛(𝑡)=0if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑃𝑆=0_𝑛(𝑡+1)=1∣𝜆_𝑛(𝑡)if𝑛∉𝐴₁(𝑡)=𝑃𝑆_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=1×𝑃_𝑛(𝑡)=1∣ℎ_𝑛(𝑡)𝑆=1+𝑃_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=0×𝑃_𝑛(𝑡)=0∣ℎ_𝑛(𝑡)=1if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑃𝑆=1_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=1×𝑃_𝑛(𝑡)=1∣ℎ_𝑛(𝑡)𝑆=0+𝑃_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=0×𝑃_𝑛(𝑡)=0∣ℎ_𝑛(𝑡)=0if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑃𝑆=0_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=1×𝑃_𝑛(𝑡)=1∣𝜆_𝑛(𝑡𝑆)+𝑃_𝑛(𝑡+1)=1∣𝑆_𝑛(𝑡)𝑆=0×𝑃_𝑛(𝑡)=0∣𝜆_𝑛(𝑡)if𝑛∉𝐴₁(𝑡)=𝑝₁₁1−𝑝^𝑛_𝑓(𝑡)+𝑝₀₁𝑝^𝑛_𝑓(𝑡)if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑝=1₁₁1−𝑝^𝑛_𝑑(𝑡)+𝑝₀₁𝑝^𝑛_𝑑(𝑡)if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑝=0₁₁𝜆_𝑛(𝑡)+𝑝₀₁1−𝜆_𝑛(𝑡)if𝑛∉𝐴₁(𝑡). (2)(3)
The proofs of the other cases are similar.

The objective of the user is to maximize its total (discounted or average) expected reward; let 𝐽^𝜋_𝛼(Λ(1)) and 𝐽^𝜋(Λ(1)) [18] denote the rewards, respectively. Here, 𝜋 denotes the policy of the user, 𝛼 denotes the discounted factor, Λ(1) is the initial probability distribution of all channels, and 𝐸^𝜋_Λ(1) represents the mathematical expectation which is determined by the initial probability distribution Λ(1) and the policy 𝜋. Consequently, 𝐽^𝜋_𝛼(Λ(1)) and 𝐽^𝜋(Λ(1)) denote the discounted and average reward with the initial probability distribution Λ(1) and the policy 𝜋, respectively.

An optimal policy should maximize the reward of the user, and this optimization problem can be formally defined as follows: max_𝜋𝐽^𝜋_𝛼(Λ(1))=max_𝜋𝐸^𝜋_Λ(1)lim^𝑇_{𝑇→+∞𝑡=1}𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡)),max_𝜋𝐽^𝜋(Λ(1))=max_𝜋limsup_𝑇→+∞1𝑇𝐸^𝜋_Λ(1)^𝑇_𝑡=1𝑟(Λ(𝑡),𝑎(𝑡)), (4) where 𝑟(Λ(𝑡),𝑎(𝑡)) denotes the immediate reward when the user implements the action 𝑎(𝑡), and we think that each channel to be accessed brings 1 unit of reward; thus 𝑟(Λ(𝑡),𝑎(𝑡))=|𝐴₂(𝑡)|. However, there is a selection problem of 𝐴₂(𝑡). Without loss of generality and for the greedy approach, all the channels whose states are sensed as 1 in 𝐴₁(𝑡) are selected into 𝐴₂(𝑡).

Then, we can give a recursive expression of the reward function: 𝑅_𝑇(Λ(𝑇))=max_𝑎(𝑇)𝐸[𝑟(Λ(𝑇)(𝑇𝑅,𝑎))]_𝑡(Λ(𝑡))=max_𝑎(𝑡)𝐸𝑟(Λ(𝑡),𝑎(𝑡))+𝛼𝑅_𝑡+1(Λ(𝑡+1))=max_{𝑎(𝑡)𝑖∈𝐴1(𝑡)}𝜆_𝑖(𝑡)+𝛼×_{𝐴2(𝑡)⊆𝐴1(𝑡)𝑗∈𝐴2(𝑡)}𝜆_𝑗(𝑡)×_{𝑘∉𝐴2(𝑡),𝑘∈𝐴1(𝑡)}1−𝜆_𝑘(𝑡)×𝑅_𝑡+1(Λ(𝑡+1)∣𝑎(𝑡)), (5) where 𝐸[𝑟(Λ(𝑡),𝑎(𝑡))] denotes the mathematical expectation of the immediate reward in the time slot 𝑡, 𝑅_𝑡(Λ(𝑡)) denotes the maximum expected reward that is accrued from time 𝑡 to 𝑇. Specially, 𝑅₁(Λ(1))=max_𝜋𝐽^𝜋_𝛼(Λ(1)) when 𝑇→+∞. This proposition is proved in Section 4.3.

3. Myopic Policy

The myopic policy is essentially a greedy policy which maximizes the immediate expected reward in each time slot and ignores the future reward; this greedy policy has the minimal time complexity and computational complexity. The expression of the myopic policy can be given by 𝑎(𝑡)=argmax_𝑎(𝑡)𝐸[𝑟(Λ(𝑡),𝑎(𝑡))]=argmax_{𝑎(𝑡)𝑖∈𝐴1(𝑡)}𝜆_𝑖(𝑡). (6) We can discover that the channels which have the |𝐴₁(𝑡)| largest conditional probabilities in Λ(𝑡) are selected into 𝐴₁(𝑡) by the myopic policy. Consequently, the successive update of the idle conditional probability vector Λ(𝑡) can determine the action of the myopic policy at time 𝑡.

In particular, if 1−𝑝^𝑛_𝑓(𝑡) is larger than all the idle conditional probabilities and 1−𝑝^𝑛_𝑑(𝑡) is smaller than all the idle conditional probabilities for any 𝑛, the myopic policy requires only the initial condition Λ(1) but not the precise values of {Λ(2),Λ(3),…}. To give an explanation, we first simplify the expression of (3) by defining a function 𝑓(𝑥)=𝑝₁₁𝑥+𝑝₀₁(1−𝑥): 𝜆_𝑛()=𝑓𝑡+11−𝑝^𝑛_𝑓(𝑡)if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑓=11−𝑝^𝑛_𝑑(𝑡)if𝑛∈𝐴₁(𝑡),ℎ_𝑛(𝑡)𝑓𝜆=0_𝑛(𝑡)if𝑛∉𝐴₁(𝑡). (7) Due to the monotonicity of 𝑓(𝑥), 𝑓(𝑥) is a monotonically increasing function when 𝑝₁₁≥𝑝₀₁. The ordering of the idle conditional probabilities can be preserved when they are updated for the reason that 𝑓(𝜆_𝑖(𝑡))≥𝑓(𝜆_𝑗(𝑡)) if 𝜆_𝑖(𝑡)≥𝜆_𝑗(𝑡). If a channel is selected into 𝐴₁(𝑡), its idle conditional probability will become 1−𝑝^𝑛_𝑓(𝑡) when it is observed as 1, or 1−𝑝^𝑛_𝑑(𝑡) when the observation is 0. That is, the channel has the largest idle conditional probability if it is observed as 1, or the smallest idle conditional probability if it is observed as 0. The myopic policy can create a list which preserves the ordering of the idle conditional probabilities according to the initial condition Λ(1). After each update in each time slot, the channels which are not observed do not change the list, the channels which are observed as 1 are selected into 𝐴₂(𝑡) and moved to the top of the list, and the channels which are observed as 0 are moved to the bottom of the list. Consequently, the myopic policy does not require the precise values of the updated idle conditional probabilities in this case.

We have an opposite situation when 𝑝₁₁<𝑝₀₁, 𝑓(𝑥) is a monotonically decreasing function. The ordering of the idle conditional probabilities should be reversed when they are updated for the reason that 𝑓(𝜆_𝑖(𝑡))≥𝑓(𝜆_𝑗(𝑡)) if 𝜆_𝑖(𝑡)≤𝜆_𝑗(𝑡). The myopic policy also creates a list which preserve the ordering of the idle conditional probabilities. After each update in each time slot, the channels which are not observed reverse their locations, the channels which are observed as 1 are moved to the bottom of the list, and the channels which are observed as 0 are moved to the top of the list. Consequently, the myopic policy also does not require the precise values of the updated idle conditional probabilities in this case.

4. Optimality of Myopic Policy

In order to show the optimality of the myopic policy, we first define two functions which can denote the expected rewards obtained by the myopic policy and the arbitrary policy: 𝑈_𝑡(Λ)=𝑈_𝑡𝜆₁,…,𝜆_𝑁=𝜆_𝑁−|𝐴^𝑈_1|+1+⋯+𝜆_𝑁+𝛼×_{𝐴𝑈2⊆𝐴𝑈1𝑗∈𝐴𝑈2}𝜆_{𝑗𝑘∉𝐴𝑈2,𝑘∈𝐴𝑈1}1−𝜆_𝑘×𝑈_𝑡+1𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑈₂,𝑘∈𝐴^𝑈₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑈₁𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑈₂𝑈_𝑇𝜆₁,…,𝜆_𝑁=𝜆_𝑁−|𝐴^𝑈_1|+1+⋯+𝜆_𝑁, (8) where 𝑈_𝑡 denotes the expected total reward obtained by the myopic policy from time 𝑡 on. Λ denotes the sequence of the idle conditional probabilities of all channels; it is reordered to {𝜆₁,…,𝜆_𝑁} by 𝑈_𝑡 and 𝜆₁≤⋯≤𝜆_𝑁. 𝐴^𝑈₁ denotes the set {𝑁−|𝐴^𝑈₁|+1,…,𝑁} of channels which have been chosen to sense by the myopic policy. 𝐴^𝑈₂ denotes the set of channels which have been chosen to access: 𝑊_𝑡(Λ)=𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}=𝜆_𝑖1+⋯+𝜆_{𝑖𝑊1||𝐴}+𝛼×_{𝐴𝑊2⊆𝐴𝑊1𝑗∈𝐴𝑊2}𝜆_{𝑗𝑘∉𝐴𝑊2,𝑘∈𝐴𝑊1}1−𝜆_𝑘×𝑈_𝑡+1𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂𝑊_𝑇𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}=𝜆_𝑖1+⋯+𝜆_{𝑖𝑊1||𝐴}, (9) where denotes the expected total reward obtained by the arbitrary policy at time 𝑡 and the myopic policy from time 𝑡+1 on. Λ denotes the sequence of the idle conditional probabilities of all channels. The channels corresponding to its last entries {𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}} are selected into 𝐴^𝑊₁. The arbitrary set 𝐴^𝑊₁ is corresponding to the arbitrary policy. 𝐴^𝑊₁ denotes the set {𝑖₁,…,𝑖_|𝐴^𝑊_1|} of channels which have been chosen to sense by the arbitrary policy. 𝐴^𝑊₂ denotes the set of channels which have been chosen to access.

In particular, 𝑊_𝑡𝜆₁,…,𝜆_𝑁=𝑈_𝑡(Λ). (10)

Theorem 2. When 𝑇 is finite, the optimality of the myopic policy at times 1,2,…,𝑇 is equivalent to 𝑊_𝑡(Λ)≤𝑈_𝑡(Λ) (11) for any 𝐴^𝑊₁ and 𝑡=1,2,…,𝑇.

Proof. We first prove the sufficiency inductively. The myopic policy is optimal at time 𝑇 for the reasons that 𝑈_𝑇 is larger than any 𝑊_𝑇 and 𝑊_𝑇 obtained by the arbitrary policy.
Then, we suppose that the myopic policy is optimal at time 𝑡+1. We have that 𝑊_𝑡 is larger than the reward obtained by the same policy at time 𝑡 and the arbitrary policy from time 𝑡+1 due to the induction hypothesis, and the immediate reward obtained by the myopic policy is the largest. Consequently, 𝑈_𝑡 is larger than any 𝑊_𝑡. 𝑈_𝑡 is thus larger than the reward obtained by the arbitrary policy. The myopic policy is optimal at time 𝑡. The proof of the sufficiency is complete.
The proof of the necessity can also be obtained due to the optimality of the myopic policy.

Lemma 3. 𝑈_𝑡 and 𝑊_𝑡 are 𝑁 variable functions which are polynomial of order 1 for 𝑡=1,…,𝑇.

Proof. We prove this by induction over time 𝑡. 𝑈_𝑇 and 𝑊_𝑇 are polynomial of order 1 due to their definitions.
We suppose that 𝑈_𝑡+1 and 𝑊_𝑡+1 are polynomial of order 1. We have that 𝑈_𝑡+1({𝑓(1−𝑝^𝑘_𝑑)∣𝑘∉𝐴^𝑈₂,𝑘∈𝐴^𝑈₁}⋃{𝑓(𝜆_𝑖)∣𝑖∉𝐴^𝑈₁}⋃{𝑓(1−𝑝^𝑗_𝑓)∣𝑗∈𝐴^𝑈₂}) and 𝑈_𝑡+1({𝑓(1−𝑝^𝑘_𝑑)∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁}⋃{𝑓(𝜆_𝑖)∣𝑖∉𝐴^𝑊₁}⋃{𝑓(1−𝑝^𝑗_𝑓)∣𝑗∈𝐴^𝑊₂}) are polynomial of order 1 for the reason that 𝑓 is a linear function. Consequently, 𝑈_𝑡 and 𝑊_𝑡 are polynomial of order 1. The proof is complete.

4.1. The Case of 𝑝₁₁≥𝑝₀₁

Assumption 4. The transition probabilities 𝑝₁₁ and 𝑝₀₁ are such that 𝑝₁₁≥𝑝₀₁.
The function 𝑓 is monotonically increasing under Assumption 4. For any 𝑥,𝑦∈[0,1] and 𝑥≤𝑦, we have 𝑓(0)≤𝑓(𝑥)≤𝑓(𝑦)≤𝑓(1).

Assumption 5. We assume that for any discounted factor 𝛼∈[0,1] and all the idle conditional probabilities, the detection probability 𝑝_𝑑 and the false alarm probability 𝑝_𝑓 are such that 1+𝛼_{𝐴2⊆𝐴1𝑗∈𝐴2}𝜆_𝑗×_{𝑘∉𝐴2,𝑘∈𝐴1}1−𝜆_𝑘𝑝_𝑑−𝑝_𝑓−1≥0. (12) For _{𝐴2⊆𝐴1𝑗∈𝐴2}𝜆_{𝑗𝑘∉𝐴2,𝑘∈𝐴1}1−𝜆_𝑘=1, (13) we can rewrite Assumption 5 as follows 1+𝛼𝑝_𝑑−𝑝_𝑓−1≥0⟹𝑝_𝑑−𝑝_𝑓1≥1−𝛼. (14)
Assumption 5 is used to limit the reliability of the sensing information; we cannot make the optimal decision if the information is very unreliable. 𝑝_𝑑 equals 1 and 𝑝_𝑓 equals 0 in the absence of sensing error; Assumption 5 is always true.

Theorem 6. The myopic policy is optimal under Assumptions 4 and 5 when 𝑇 is finite.
To prove this theorem, one should show that 𝑈_𝑡 is larger than any 𝑊_𝑡 for 𝑡=1,…,𝑇 according to Theorem 2. One proves this inductively. Given that 𝑈_𝑖 is larger than any 𝑊_𝑖 for 𝑖=𝑡+1,…,𝑇, one wants to show that 𝑈_𝑡 is larger than any 𝑊_𝑡. This relies on a number of lemmas introduced below.

Lemma 7. For all 𝜆_𝑥≥𝜆_𝑦, 𝑥,𝑦∈𝐴^𝑊₁, one has 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_𝑥,𝜆_𝑦,…=𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_𝑦,𝜆_𝑥,…. (15)

Lemma 8. For any 𝜆_𝑗1≤⋯≤𝜆_𝑥≤𝜆_𝑦≤⋯≤𝜆_{𝑗𝑊1|𝑁−|𝐴} and 𝜆_𝑖1≤⋯≤𝜆_{𝑖𝑊1||𝐴}, one has 𝑊_𝑡…,𝜆_𝑥,𝜆_𝑦,…,𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}=𝑊_𝑡…,𝜆_𝑦,𝜆_𝑥,…,𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}. (16)

Proof. Lemmas 7 and 8 are true according to the definition of 𝑊_𝑡.

Lemma 9. One has 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,𝜆_𝑖1,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_{𝑖𝑊1||𝐴}=𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}×𝑊_𝑡𝜆_𝑗1,…,0,1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,1,0,…,𝜆_{𝑖𝑊1||𝐴}. (17)

Proof. We use LHS and RHS to denote the left-hand side and the right-hand side of the equation, respectively. We can prove that 𝑊_𝑡({𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}}) is the first-order function of 𝜆_{𝑗𝑊1|𝑁−|𝐴} and 𝜆_𝑖1 according to Lemma 3. Consequently, we can suppose that 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}=𝑎𝜆_{𝑗𝑊1|𝑁−|𝐴}+𝑏𝜆_𝑖1+𝑐𝜆_{𝑗𝑊1|𝑁−|𝐴}𝜆_𝑖1+𝑑, (18) where 𝑎, 𝑏, 𝑐, and 𝑑 are irrelevant with 𝜆_{𝑗𝑊1|𝑁−|𝐴} and 𝜆_𝑖1. Consequently, we have that LHS=𝑎𝜆_{𝑗𝑊1|𝑁−|𝐴}+𝑏𝜆_𝑖1+𝑐𝜆_{𝑗𝑊1|𝑁−|𝐴}𝜆_𝑖1−+𝑑𝑎𝜆_𝑖1+𝑏𝜆_{𝑗𝑊1|𝑁−|𝐴}+𝑐𝜆_{𝑗𝑊1|𝑁−|𝐴}𝜆_𝑖1=()𝜆+𝑑𝑎−𝑏_{𝑗𝑊1|𝑁−|𝐴}−𝜆_𝑖1,RHS=𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}×)−(=()𝜆[(𝑎0+𝑏1+𝑐0×1+𝑑𝑎1+𝑏0+𝑐0×1+𝑑)]𝑎−𝑏_{𝑗𝑊1|𝑁−|𝐴}−𝜆_𝑖1=LHS. (19)(20) The proof is complete.

Lemma 10. Consider Assumption 4. One has 𝑈_𝑡𝜆¹_𝑧,𝜆₁,…,𝜆_𝑁−1−𝑈_𝑡𝜆₁,…,𝜆_𝑖,𝜆²_𝑧,𝜆_𝑖+1,…,𝜆_𝑁−1≥𝜆¹_𝑧−𝜆²_𝑧 (21) for any 𝜆¹_𝑧≤𝜆₁≤⋯≤𝜆²_𝑧≤⋯≤𝜆_𝑁−1.

Proof. We use LHS to denote the left-hand side of the inequality. We use 𝑓^𝑛(𝑥) to denote 𝑓(𝑓(⋯𝑓(𝑥)⋯)): 𝑓^𝑛(𝑥)=𝑝₁₁−𝑝₀₁^𝑛𝑥+𝑝₁₁−𝑝₀₁^𝑛−1𝑝₀₁+⋯+𝑝₀₁𝑓^𝑛(𝑥)−𝑓^𝑛𝑦=𝑝₁₁−𝑝₀₁^𝑛𝑥−𝑦. (22) Consequently, we have 𝑓^𝑛(𝑥)−𝑓^𝑛𝑦≥⋯≥𝑓(𝑥)−𝑓𝑦≥𝑥−𝑦 (23) for any 𝑥≤𝑦. Therefore, we have LHS≥𝜆_𝑖−𝜆²_𝑧+𝜆_𝑖−1−𝜆_𝑖+⋯+𝜆¹_𝑧−𝜆₁≥𝜆¹_𝑧−𝜆²_𝑧. (24)

Lemma 11. Consider Assumptions 4 and 5. For any 𝜆_𝑗1≤⋯≤𝜆_{𝑗𝑊1|𝑁−|𝐴} and 𝜆_𝑖1≤⋯≤𝜆_{𝑖𝑊1||𝐴}, if 𝜆_{𝑗𝑊1|𝑁−|𝐴}≥𝜆_𝑖1, one has 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}≤𝑊_𝑡𝜆_𝑗1,…,𝜆_𝑖1,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_{𝑖𝑊1||𝐴}. (25)

Proof. The inequality is true at time 𝑇. We have the following equation for any time 𝑡<𝑇 according to Lemma 9: 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,𝜆_𝑖1,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_{𝑖𝑊1||𝐴}=𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}×𝑊_𝑡𝜆_𝑗1,…,0,1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,1,0,…,𝜆_{𝑖𝑊1||𝐴}. (26) Because 𝜆_{𝑗𝑊1|𝑁−|𝐴}≥𝜆_𝑖1, 𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}≤0, we use LHS to denote 𝑊_𝑡({𝜆_𝑗1,…,0,1,…,𝜆_{𝑖𝑊1||𝐴}})−𝑊_𝑡({𝜆_𝑗1,…,1,0,…,𝜆_{𝑖𝑊1||𝐴}}). According to the definition of 𝑊_𝑡, we have LHS=1+𝛼×_{𝐴𝑊2⊆𝐴𝑊1𝑗∈𝐴𝑊2−𝑖1}𝜆_𝑗×_{𝑘∉𝐴𝑊2,𝑘∈𝐴𝑊1−𝑖1}1−𝜆_𝑘×𝑈_𝑡+1𝑓1−𝑝^𝑖1_𝑓𝑓1−𝑝^𝑗_𝑓∣𝑗∉𝐴^𝑊₂−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓1−𝑝^𝑖1_𝑓−𝑈_𝑡+1𝑓1−𝑝^{𝑗𝑊1|𝑁−|𝐴}_𝑑𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓(1). (27)
According to Lemma 10, we have LHS≥1+𝛼_{𝐴𝑊2⊆𝐴𝑊1𝑗∈𝐴𝑊2−𝑖1}𝜆_𝑗×_{𝑘∉𝐴𝑊2,𝑘∈𝐴𝑊1−𝑖1}1−𝜆_𝑘×0−1−𝑝^{𝑗𝑊1|𝑁−|𝐴}_𝑑+1−𝑝^𝑖1_𝑓−1. (28) Consequently, LHS≥0 under Assumption 5: 𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}LHS≤0. (29) The proof is complete.

Now we can give the proof of Theorem 6 with Lemmas 7, 8, and 11.

Proof. 𝑊_𝑇≤𝑈_𝑇 due to their definitions. For any time 𝑡<𝑇, 𝜆_𝑗1≤⋯≤𝜆_{𝑗𝑊1|𝑁−|𝐴}, 𝜆_𝑖1≤⋯≤𝜆_{𝑖𝑊1||𝐴}, and 𝜆₁≤⋯≤𝜆_𝑁, we have 𝑊_𝑡𝜆₁,…,𝜆_𝑖−1,𝜆_𝑖+1,…,𝜆_𝑁,𝜆_𝑖≤𝑊_𝑡𝜆₁,…,𝜆_𝑖−1,𝜆_𝑖+1,…,𝜆_𝑁−1,𝜆_𝑖,𝜆_𝑁≤𝑊_𝑡𝜆₁,…,𝜆_𝑖−1,𝜆_𝑖,𝜆_𝑖+1,…,𝜆_𝑁=𝑈_𝑡𝜆₁,…𝜆_𝑁,𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}≤𝑊_𝑡…,𝜆_𝑖1−1,𝜆_𝑖1,𝜆_𝑖1+1𝜆,…,_{𝑖𝑊1||𝐴−1},𝜆_{𝑖𝑊1||𝐴},𝜆_{𝑖𝑊1||𝐴+1},…=𝑊_𝑡𝜆₁,…𝜆_𝑁=𝑈_𝑡𝜆₁,…𝜆_𝑁. (30) The inequalities are true due to Lemmas 7, 8, and 11. Consequently, 𝑊_𝑡≤𝑈_𝑡 at time 𝑡=1,…,𝑇. The myopic policy is optimal according to Theorem 2. The proof is complete.

4.2. The Case of 𝑝₁₁≤𝑝₀₁

Assumption 12. The transition probabilities 𝑝₁₁ and 𝑝₀₁ are such that 𝑝₁₁≤𝑝₀₁.
The function 𝑓 is monotonically decreasing under Assumption 12. For any 𝑥,𝑦∈[0,1] and 𝑥≤𝑦, we have 𝑓(1)≤𝑓(𝑦)≤𝑓(𝑥)≤𝑓(0).

Assumption 13. We assume that, for all the idle conditional probabilities, the discounted factor 𝛼 is such that 1−𝛼_{𝐴2⊆𝐴1𝑗∈𝐴2}𝜆_𝑗×_{𝑘∉𝐴2,𝑘∈𝐴1}1−𝜆_𝑘𝐴₁(1−𝛼)≥0. (31) Like Assumption 5, we can rewrite Assumption 13 as follows: 𝛼𝐴1−¹()11−𝛼≥0⟹𝛼≤1+𝐴₁. (32)

Theorem 14. The myopic policy is optimal under Assumptions 12 and 13 when 𝑇 is finite.

Lemma 15. Consider Assumptions 12 and 13. For any 𝜆_𝑗1≤⋯≤𝜆_{𝑗𝑊1|𝑁−|𝐴} and 𝜆_𝑖1≤⋯≤𝜆_{𝑖𝑊1||𝐴}, if 𝜆_{𝑗𝑊1|𝑁−|𝐴}≥𝜆_𝑖1, one has 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}≤𝑊_𝑡𝜆_𝑗1,…,𝜆_𝑖1,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_{𝑖𝑊1||𝐴}. (33)

Proof. The inequality is true at time 𝑇. We have the following equation for any time 𝑡<𝑇 according to Lemma 9: 𝑊_𝑡𝜆_𝑗1,…,𝜆_{𝑗𝑊1|𝑁−|𝐴},𝜆_𝑖1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,𝜆_𝑖1,𝜆_{𝑗𝑊1|𝑁−|𝐴},…,𝜆_{𝑖𝑊1||𝐴}=𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}×𝑊_𝑡𝜆_𝑗1,…,0,1,…,𝜆_{𝑖𝑊1||𝐴}−𝑊_𝑡𝜆_𝑗1,…,1,0,…,𝜆_{𝑖𝑊1||𝐴}. (34) Because 𝜆_{𝑗𝑊1|𝑁−|𝐴}≥𝜆_𝑖1, 𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}≤0, we use LHS to denote 𝑊_𝑡({𝜆_𝑗1,…,0,1,…,𝜆_{𝑖𝑊1||𝐴}})−𝑊_𝑡({𝜆_𝑗1,…,1,0,…,𝜆_{𝑖𝑊1||𝐴}}). According to the definition of 𝑊_𝑡, we have LHS=1+𝛼×_{𝐴𝑊2⊆𝐴𝑊1𝑗∈𝐴𝑊2−𝑖1}𝜆_𝑗×_{𝑘∉𝐴𝑊2,𝑘∈𝐴𝑊1−{𝑖1}}1−𝜆_𝑘×𝑈_𝑡+1𝑓1−𝑝^𝑖1_𝑓𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓(0)−𝑈_𝑡+1𝑓(1𝑓)1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓1−𝑝^{𝑗𝑊1|𝑁−|𝐴}_𝑑. (35)
We have the following inequalities due to the definition of 𝑈_𝑡: 𝑈_𝑡+1𝑓1−𝑝^𝑖1_𝑓𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓(0𝑈)≥0_𝑡+1𝑓(1)𝑓1−𝑝^𝑗_𝑓∣𝑗∈𝐴^𝑊₂−𝑖₁𝑓𝜆_𝑖∣𝑖∉𝐴^𝑊₁𝑓1−𝑝^𝑘_𝑑∣𝑘∉𝐴^𝑊₂,𝑘∈𝐴^𝑊₁−𝑖₁𝑓1−𝑝^{𝑗𝑊1|𝑁−|𝐴}_𝑑≤𝐴^𝑊₁1+𝛼¹+⋯+𝛼^𝑇−𝑡≤𝐴^𝑊₁1+𝛼¹+⋯+𝛼^+∞=𝐴^𝑊₁().1−𝛼 (36)(37)
Then, we have LHS≥1+𝛼_{𝐴𝑊2⊆𝐴𝑊1𝑗∈𝐴𝑊2−𝑖1}𝜆_𝑗×_{𝑘∉𝐴𝑊2,𝑘∈𝐴𝑊1−𝑖1}1−𝜆_𝑘×𝐴0−^𝑊1().1−𝛼 (38)
Consequently, LHS≥0 under Assumption 13. 𝜆_𝑖1−𝜆_{𝑗𝑊1|𝑁−|𝐴}LHS≤0. (39) The proof is complete.

Now we can give the proof of Theorem 14 with Lemmas 7, 8, and 15. The proof of Theorem 14 is similar with the proof of Theorem 6.

4.3. The Case of 𝑇→+∞

We discuss the optimality of the myopic policy in above subsections when 𝑇 is finite; now we consider the extensions of results when 𝑇 is infinite.

Theorem 16. If the myopic policy is optimal when 𝑇 is finite, it is optimal when 𝑇 is infinite.

Proof. max_𝜋𝐽^𝜋_𝛼(Λ(1))=max_𝜋𝐸^𝜋_Λ(1)lim^𝑇_{𝑇→+∞𝑡=1}𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))=max_𝜋lim_𝑇→+∞𝐸^𝜋_Λ(1)^𝑇_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))=max_𝜋lim_𝑇→+∞𝐽^𝜋_𝑇(Λ(1)), (40) where 𝐽^𝜋_𝑇(Λ(1)) denotes 𝐸^𝜋_Λ(1)[∑^𝑇_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))] only in this proof.
For ^𝑇_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡)(𝑡≤𝐴,𝑎))₁1+𝛼+⋯+𝛼^𝑇−1≤𝐴₁1+𝛼+⋯+𝛼^+∞=𝐴₁(),1−𝛼 (41) we can use the bounded convergence theorem to interchange lim_𝑇→+∞ and 𝐸^𝜋_Λ(1). Then, we consider the relationship of lim_𝑇→+∞ and max_𝜋, and two sequences are given as follows: Sequence 1: {𝐽^𝜋₁(Λ(1)),𝐽^𝜋₂(Λ(1)),…} Sequence 2: {max_𝜋𝐽^𝜋₁(Λ(1)),max_𝜋𝐽^𝜋₂(Λ(1)),…}.
We have the following inequalities: 𝐸^𝜋_Λ(1)^𝑇+1_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))≥𝐸^𝜋_Λ(1)^𝑇_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))⟹𝐽^𝜋_𝑇+1(Λ(1))≥𝐽^𝜋_𝑇(Λ(1))max_𝜋𝐽^𝜋_𝑇+1(Λ(1))≥𝐽^𝜋′_𝑇+1(Λ(1))≥𝐽^𝜋′_𝑇(Λ(1))⟹max_𝜋𝐽^𝜋_𝑇+1(Λ(1))≥max_𝜋𝐽^𝜋_𝑇(Λ(1)), (42) where 𝜋^′ denotes argmax_𝜋𝐽^𝜋_𝑇(Λ(1)). And for any 𝑇, we have 𝐽^𝜋_𝑇(Λ(1≤𝐴))¹(1−𝛼)max_𝜋𝐽^𝜋_𝑇(Λ(1≤𝐴))¹().1−𝛼 (43) Consequently, Sequences 1 and 2 are monotonically increasing and bounded. We can conclude that Sequences 1 and 2 have finite limits due to the monotone convergence theorem. We thus have lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))−lim_𝑇→+∞𝐽^𝜋_𝑇(Λ(1))=lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))−𝐽^𝜋_𝑇(Λ(1))≥0 (44) for any policy 𝜋 and 𝑇. Therefore, we have lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))≥max_𝜋lim_𝑇→+∞𝐽^𝜋_𝑇(Λ(1)). (45) For max_𝜋𝐽^𝜋_𝑇(Λ(1))∈{𝐽^𝜋_𝑇(Λ(1))∣𝜋}, we have lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))≤max_𝜋lim_𝑇→+∞𝐽^𝜋_𝑇(Λ(1)). (46) We can conclude that lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))=max_𝜋lim_𝑇→+∞𝐽^𝜋_𝑇(Λ(1)). (47) Consequently, we have max_𝜋𝐽^𝜋_𝛼(Λ(1))=lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))=lim_𝑇→+∞max_𝜋𝐸^𝜋_Λ(1)^𝑇_𝑡=1𝛼^𝑡−1𝑟(Λ(𝑡),𝑎(𝑡))=lim_𝑇→+∞𝑅₁(Λ(1)). (48) The myopic policy is thus optimal when 𝑇 is infinite if the myopic policy is optimal when 𝑇 is finite. 𝑅₁(Λ(1)) has been defined in Section 2.
Then, we consider the uniqueness of the optimal policy. Let 𝜋^∗ be the myopic policy, and 𝑎^∗(𝑡)={𝐴^∗₁(𝑡),𝐴^∗₂(𝑡)} is the action at time 𝑡; we have max_𝜋𝐽^𝜋_𝛼(Λ(1))=lim_𝑇→+∞max_𝜋𝐽^𝜋_𝑇(Λ(1))=lim_𝑇→+∞𝐽^𝜋∗_𝑇(Λ(1))=lim_𝑇→+∞𝐸^𝜋∗_Λ(1)^𝑇_𝑡=1𝛼^𝑡−1𝑟Λ(𝑡),𝑎^∗(𝑡)=_{𝑖∈𝐴∗1(1)}𝜆_𝑖×+𝛼_{𝐴∗2(1)⊆𝐴∗1(1)𝑗∈𝐴∗2(1)}𝜆_𝑗(1)×_{𝑘∉𝐴∗2(1),𝑘∈𝐴∗1(1)}1−𝜆_𝑘(1)×lim_𝑇→+∞𝐸^𝜋∗_Λ(2)×^𝑇_𝑡=2𝛼^𝑡−2𝑟Λ(2),𝑎^∗(2)∣𝑎^∗(1). (49) For lim_𝑇→+∞𝐸^𝜋∗_Λ(2)^𝑇_𝑡=2𝛼^𝑡−2𝑟Λ(2),𝑎^∗(2)∣𝑎^∗(1)=lim_𝑇→+∞𝐽^𝜋∗_𝑇−1Λ(2)∣𝑎^∗(1)=lim_𝑇→+∞𝐽^𝜋∗_𝑇Λ(2)∣𝑎^∗(1)=𝐽^𝜋∗_𝛼Λ(2)∣𝑎^∗(1), (50) we have 𝐽^𝜋∗_𝛼(Λ(1))=_{𝑖∈𝐴∗1(1)}𝜆_𝑖+𝛼×_{𝐴∗2(1)⊆𝐴∗1(1)𝑗∈𝐴∗2(1)}𝜆_𝑗(1)×_{𝑘∉𝐴∗2(1),𝑘∈𝐴∗1(1)}1−𝜆_𝑘(1)×𝐽^𝜋∗_𝛼Λ(2)∣𝑎^∗(1). (51) The above equation is the dynamic programming equation for the infinite horizon discounted reward problem. The uniqueness of the optimal policy can be proved due to the uniqueness of the dynamic programming solution.
The proof is complete.

Theorem 17. If the myopic policy is optimal for the discounted reward, it is optimal for the average reward.

Proof. We first consider the Blackwell optimality [19, pp. 336–341] of the optimal policy for the discounted reward. The sequence of {𝛼₁,𝛼₂,…} is given, and 𝛼₁≤𝛼₂≤⋯≤1, lim_𝑘→+∞𝛼_𝑘=1. For 1−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1≤))1−𝛼_𝑘𝐴₁1+𝛼_𝑘+𝛼²_𝑘+⋯=𝐴₁𝐽^𝜋_𝛼𝑘(Λ)−𝐽^𝜋_𝛼𝑘(Λ(1≤𝐴))0−₁1+𝛼_𝑘+𝛼²_𝑘=𝐴+⋯¹1−𝛼_𝑘, (52) we can give the definition of 𝐽(Λ(1)) and ℎ(Λ) due to the boundedness of (1−𝛼_𝑘)𝐽^𝜋_𝛼𝑘(Λ(1)) and |𝐽^𝜋_𝛼𝑘(Λ)−𝐽^𝜋_𝛼𝑘(Λ(1))|: 𝐽(Λ(1))=max_𝜋lim_𝑘→+∞1−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1ℎ(Λ)))=max_𝜋lim_𝑘→+∞𝐽^𝜋_𝛼𝑘(Λ)−𝐽^𝜋_𝛼𝑘(Λ(1)) (53) for any Λ.

Then, we can give the average cost optimality equation (ACOE) [20]. Here, we calculate the reward: 𝐽(Λ(1))+ℎ^𝜋(Λ(1))=max_𝜋lim_𝑘→+∞1−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1+𝐽))_𝜋𝛼𝑘(Λ(1))−𝐽^𝜋_𝛼𝑘(Λ(1))=max_𝜋lim_𝑘→+∞1−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1)). (54)

For 𝐽^𝜋_𝛼𝑘(Λ(1))=_{𝑖∈𝐴1(1)}𝜆_𝑖+𝛼×_{𝐴2(1)⊆𝐴1(1)𝑗∈𝐴2(1)}𝜆_𝑗(1)×_{𝑘∉𝐴2(1),𝑘∈𝐴1(1)}1−𝜆_𝑘(1)×𝐽^𝜋_𝛼(Λ(2)(1,𝐽∣𝑎))_𝜋𝛼𝑘(Λ(1))=_{𝐴2(1)⊆𝐴1(1)𝑗∈𝐴2(1)}𝜆_𝑗(1)×_{𝑘∉𝐴2(1),𝑘∈𝐴1(1)}1−𝜆_𝑘(1)𝐽^𝜋_𝛼𝑘(Λ(1)), (55) we have 𝐽(Λ(1))+ℎ^𝜋(Λ(1))=max_𝜋lim_𝑘→+∞𝐽^𝜋_𝛼𝑘(Λ(1))−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1))=max_𝜋lim_{𝑘→+∞𝑖∈𝐴1(1)}𝜆_𝑖+𝛼_𝑘×_{𝐴2(1)⊆𝐴1(1)𝑗∈𝐴2(1)}𝜆_𝑗(1×_{𝑘∉𝐴2(1),𝑘∈𝐴1})₍₁₎1−𝜆_𝑘(1×𝐽_𝜋𝛼𝑘)(Λ(2)∣𝑎(1))−𝐽^𝜋_𝛼𝑘(Λ(1)). (56)

Consequently, we have 𝐽(Λ(1))+ℎ^𝜋(Λ(1))=max_{𝜋𝑖∈𝐴1(1)}𝜆_𝑖+_{𝐴2(1)⊆𝐴1(1)𝑗∈𝐴2(1)}𝜆_𝑗(1)×_{𝑘∉𝐴2(1),𝑘∈𝐴1(1)}1−𝜆_𝑘(1)×ℎ^𝜋(Λ(2)). (57)

We thus can conclude that the stationary deterministic policy realizing the pointwise maximum on the right-hand side of the ACOE is the average optimal policy due to the boundedness of ℎ^𝜋(Λ(1)), and 𝐽(Λ(1)) is the maximum average expected reward [20, Theorems 4.1–4.3].

For 𝐽(Λ(1))=max_𝜋lim_𝑘→+∞1−𝛼_𝑘𝐽^𝜋_𝛼𝑘(Λ(1)), (58) the optimal policy for the discounted reward can maximize 𝐽(Λ(1)). Consequently, the myopic policy is optimal for the average reward when it is optimal for the discounted reward. The proof is complete.

5. Numerical Results

We consider twenty independent channels with the same bandwidth 𝐵=1 and transition probabilities {𝑝₁₁,𝑝₀₁}. The sensing capacity of the user who transmits the data on these channels is limited to 𝐶; that is, the user can sense 𝐶 channels in a sensing procedure. The transition probabilities are set as follows: 𝑝₁₁=0.8 and 𝑝₀₁=0.5 when 𝑝₁₁≥𝑝₀₁, 𝑝₁₁=0.3 and 𝑝₀₁=0.5 when 𝑝₁₁≤𝑝₀₁. The detection probability 𝑝_𝑑 equals 0.9, and the false alarm probability 𝑝_𝑓 equals 0.05. We present the numerical results to evaluate the performance of the optimal policy (OP) which is the dynamic programming solution and the myopic policy (MP).

We first use the throughput of the policies to evaluate the performance. The above subfigure of Figure 2 shows the performance comparison of OP and MP when 𝑝₁₁≥𝑝₀₁. The myopic policy is the optimal policy in this case for the reason that Assumptions 4 and 5 are met. We observe that the performance of OP is similar with MP’s. The following subfigure shows the performance comparison of OP and MP when 𝑝₁₁≤𝑝₀₁. We observe that there is a large difference between OP and MP with the growth of the sensing capacity 𝐶. The reason is that Assumption 13 is not met. For example, the curves of MP and OP at 𝛼=0.1 separate when 𝐶 equals 11 for the reason that 𝛼≥1/(1+𝐶).

Figure 2 

Performance comparison of OP and MP when 𝑝11≥𝑝01 and 𝑝11≤𝑝01.

Then, we use the collision probabilities which are the probabilities which the user accesses occupied channels to evaluate the performance. The collision probability is the key metric which measures the interference caused by the user. Figure 3 gives us similar results of Figure 2. The above subfigure shows the collision probabilities of OP and MP when 𝑝₁₁≥𝑝₀₁. We observe that they have the same collision probabilities for the reason that the myopic policy is the optimal policy. The following subfigure shows the collision probabilities of OP and MP when 𝑝₁₁≤𝑝₀₁. The myopic policies at 𝛼=0.1,0.3,0.6 have the same collision probabilities for the reason that they are the same policy. On the other hand, the optimal policies are different at different 𝛼.

Figure 3 

The comparison of collision probabilities of OP and MP when 𝑝11≥𝑝01 and 𝑝11≤𝑝01.

At last, we also give the comparison of the time complexity of OP and MP in Table 1. Since 𝑝₁₁, 𝑝₀₁, and 𝛼 do not make impact on the time complexity of the policies, we mainly consider the variation of the sensing capacity. The first column of the table is the sensing capacity. The second and third columns show the time overhead of OP and MP, respectively, and the unit is second. The time overhead of MP is nearly 0 as MP does not need to calculate any parameter. In particular, the time overhead of OP is also almost 0 when the sensing capacity is 20 for the reason that OP can directly choose the channels which are observed as 1(idle). From Table 1, we can find that the time complexity of MP is much smaller than OP’s.

Table 1 

Comparison of time complexity of OP and MP.

6. Conclusion

We show the optimality of the simple and robust myopic policy for the infinite horizon discounted and average reward criteria in the case where the stochastic evolution of channels can be modeled as the independent and identically distributed two-state Markov chains. The myopic policy is optimal when the state transitions are positively correlated and the detection probability and the false alarm probability are limited. The myopic policy is also optimal when the state transitions are negatively correlated and the discounted factor is limited.

Acknowledgments

This work was partly supported by National Natural Science Foundations of China under Grant no. 61074033 and no. 61233003, Doctoral Fund of Ministry of Education of China under Grant no. 20093402110019.