Abstract
With the increasing demands for networks, space-air-ground integrated network (SAGIN) and quality of experience (QoE) have been proposed in recent years. In order to adapt to the more complex network environment in the future, the network management based on QoE in SAGIN comes into being. In this paper, we propose a QoE driven cross-domain management architecture for SAGIN. From the perspective of systematicity, we focus on the process of the network management, including the required network functions, the distribution of different functions and the network operation strategy. Therefore, we divide the network architecture into subsystems, functional modules and network facilities, and propose the corresponding network operation strategies. In addition, we propose a network task allocation strategy based on Q-learning and apply it to the simulation environment of proposed architecture. Through simulation experiment, we demonstrate the advantages of QoE in SAGIN.
1. Introduction
In the past few years, the demand for the wireless network is growing at an alarming rate. According to a recent report by the International Telecommunication Union (ITU), the global wireless data volume in 2021 increases by nearly 60% compared to 2020, which is 200 times that of 2010 [1]. With the emergence of new network services and diverse network scenarios, people prefer to choose more autonomous and personalized network services. In order to meet the demands of end-users for network, SAGIN and QoE, which represent different development directions of network in the future, respectively, are proposed.
Focusing on the increase of network resources and coverage, SAGIN uses modern information network technology to integrate networks in different domains, including space domain, air domain and ground domain [2]. The space domain consists of geostationary earth orbit (GEO) satellites and low earth orbit (LEO) satellites, which have the characteristics of wide coverage and high transmission delay [3]. The air domain takes unmanned aerial vehicles (UAVs), airships and hot air balloons as carriers, which have the characteristics of rapid deployment, low price and flexibility [4, 5]. Ground domain consists of base stations (BSs) and wireless local area networks (WLANs), which have the characteristics of low delay, rich network resources and limited coverage [6].
Unlike SAGIN, QoE take more attention on the subjectivity of end-users [7]. QoE is defined by ITU-T as “the overall acceptability of an application or service, as qualified subjectively by the end-users”. In other words, QoE is an evaluation model, which is used to evaluate the “joy or novelty” of end-users [8, 9]. The prediction of this subjective feeling is the result of the multi-dimensional influencing factors including the communication equipment, the surrounding environment and the end-users’ mood. Compared with QoS, QoE can take more attention on personalized differences and directly obtain end-user preferences. Therefore, QoE can more accurately predict the demands of end-users.
With the development of the network technology, SAGIN will be applied in more complex network scenarios in the future, such as vehicle networks, and SAGIN needs to provide more flexible network services. Traditional models of network services will not be able to meet end-users’ demands for personalized and diverse services. Referring to traditional terrestrial wireless network [10–12], QoE would be one of possible solutions. In fact, there are a lot of studies about QoE in SAGIN in recent year. Most of studies focus on network resources allocation by minimizing cost and maximizing resource utilization to ensure end-users’ satisfaction. Literatures [13–16] improve end-users’ QoE by optimizing the allocation strategies of spectrum, cache, energy and processing capability in SAGIN, respectively. Literature [17] proposes a task offloading strategy to ensure the QoE of ultra-reliable low-latency communication (URLLC) in the Internet of vehicle (IoV) application scenario. However, these studies above do not systematically describe the required network management architecture of their strategies. In our opinion, the systematical description of a network architecture needs to explain the following issues: (i)What network facilities and network functions are needed during the implementation of QoE in SAGIN, including data collection, network control and strategic decision [18]?(ii)What network functions need to be configured and implemented for different network facilities in SAGIN?(iii)Based on these network functions, what is the network operation process?
Literature [19] proposes a framework based on artificial intelligence (AI) to maintain the QoE models of end-users. Literature [20] proposes a network management framework based on reinforcement learning (RL) to make the flight trajectory of UAVs. However, these studies only discuss the process of strategic decision.
Fortunately, there are a lot of systematic studies on the network management architecture design of QoE and SAGIN, respectively, which can be used as references for the management architecture design of QoE in SAGIN. Among them, the network management architecture design based on QoE in traditional terrestrial network mainly focuses on three aspects: QoE modeling, QoE monitoring and measurement and QoE control [21]. The network management architecture design of SAGIN mainly considers how to coordinate network management and control in different domains [18, 22, 23].
In this paper, we arm to propose a network management architecture for QoE in SAGIN, which meets the characteristics of SAGIN and the function requirements of QoE services. We focus on the systematicness of the proposed architecture, which makes our research a clear distinction from others. The key points we consider when designing the architecture include: (i)The network architecture can realize cross-domain management in SAGIN, which combines unified management and distributed control [18, 22, 23].(ii)The network architecture can realize QoE management based on cross-domain management in SAGIN, including QoE monitoring and measurement, QoE control, QoE modeling and optimization [11].(iii)The network function modules are configured to appropriate network facilities according to the capacity and spatial location of the network facilities in different domains(iv)Design the corresponding network operation strategy of proposed architecture
Furthermore, we propose a network task offloading strategy based on the proposed network management architecture to improve end-users’ QoE, which is demonstrated by simulation experiments. For clarity, the main contributions of our work are summarized as follows: (i)Propose a QoE driven cross-domain management architecture for SAGIN. The network architecture into three subsystems. Each subsystem is further refined into multiple function modules, which are configured to different facilities in SAGIN(ii)Divide the network operation strategy into online process and off-line process, and propose a near-real-time network operation strategy for some special scenarios(iii)Propose a network task allocation strategy based on Q-learning and apply it into the proposed architecture in the simulation environment. The results of the simulation experiment prove the advantages of QoE in SAGIN and the effectiveness of proposed network operation strategies
2. Related Work
With the deepening of research, people begin to consider the integration of SAGIN and QoE. A large number of studies try to promote the application of QoE in SAGIN from different aspects. We summarize some related works in recent years as shown in Table 1. Literatures [13–16] focus on the management of network resource and improve the end-users’ QoE by optimizing the allocation strategy of network resources. However, these researches focus on the optimization of algorithms and the operation process of the network is vague. Literature [19] proposes a network management framework for AI and literature [20] discusses the correlation between UAV return perception 3D trajectory, resource management and end-users’ QoE. However, their research is still the implementation of specific algorithms in SAGIN and lacks the operation process. Literature [24] proposes a operation strategy of QoE in SAGIN. But their research only involves the backhaul network. Literature [25] and literature [17] discuss the network operation process of SAGIN to realize QoE in the scenarios of Internet of thing (IoT) [25] and IoV [17].
In fact, there are a lot of researches on the network architectures of SAGIN and QoE, respectively. We try to provide references for the network architecture of QoE in SAGIN from the existing network architectures.
We refer to the design of cross-domain management architecture for SAGIN [18, 22, 23, 26]. Through cross-domain management, different domains in SAGIN can cooperate with each other, such as load balancing, resource allocation optimization and routing. Considering the mutual cooperation between different domains, literature [26] proposes the uniform management of different domains, so as to realize information sharing and make unified decisions. On this basis, in order to relieve the burden of the data center and reduce the delay of network processing, literatures [18, 22] propose to integrate distributed control and unified management: some network management functions, such as satellite handover, network topology discovery and mobile deployment, are realized by controllers in different domains. As a complement, literature [23] distributes the network functions according to the capability of the network facilities.
On the other hand, the researches on QoE network management architecture also have made a lot of progress. Literature [21] divides QoE management into three steps: QoE modeling, QoE monitoring and measurement, QoE optimization and control. Based on the monitoring of multidimensional data, literature [11] divides the data collection of QoE into several functional modules. Literature [12] construct a softwarized network management architecture to achieve more flexible network control by middle control layer. Literature [10] proposes to maintain QoE models based on big data.
In this paper, we learn from the key issues of network management architecture design of cross-domain management in SAGIN and QoE management in traditional terrestrial networks, and propose a QoE driven cross-domain management architecture for SAGIN，which can meet the functional requirements of QoE management. In addition, a corresponding network management strategy is proposed and simulated.
3. Network Management Architecture
3.1. Architectural Overview
For clarity, the architecture is divided into three subsystems: monitoring subsystem, control subsystem and management subsystem, which correspond to QoE monitoring and measurement, QoE control, QoE modeling and optimization, respectively. Each subsystem is divided into different functional modules and each functional module is configured to different network facilities, as shown in Figure 1.
3.2. Monitoring Subsystem
Monitoring subsystem is responsible for collecting the information of end-users and network states (a). Through the built-in QoE models, monitoring subsystem can predict end-users’ QoE. Monitoring subsystem uploads the collected information to control subsystem (b1, b2) and then reports to management subsystem(c).
Monitoring subsystem can be divided into two kinds of nodes: monitoring nodes and quality evaluation nodes [21]. Monitoring nodes are responsible for collecting information and uploading information to quality evaluation nodes (1a). Quality evaluation nodes are used to predict the end-users’ QoE through in-built QoE models and periodically obtain the update data of QoE models from management subsystem (f, h).
Monitoring nodes are divided into user monitoring nodes and network monitoring nodes. User monitoring nodes can be deployed on the user terminals and are mainly responsible for collecting user-side information, such as the surrounding environment of end-users. Network monitoring nodes are responsible for collecting network-side information, such as network throughput, and can be deployed in the access nodes (LEO satellites, UAVs and BSs, etc.), backhaul nodes (ground gateways, macro stations) and data center.
Quality assessment nodes should be deployed on network facilities with wide coverage, so as to collect data from more monitoring nodes. Quality assessment nodes can be deployed on LEO satellites, macro stations and high altitude platforms (HAPs).
3.3. Control Subsystem
Control subsystem is the interface between management subsystem and network facilities. On the one hand, control subsystem is responsible for the data exchange between management subsystem and network facilities. On the other hand, control subsystem is responsible for the implementation of network monitoring strategies and network management strategies. Controllers in space domain, air domain and ground domain can be deployed on GEO satellites, HAPs and macro stations, respectively. The controllers in different domains contain two kinds of function modules: transmission control modules and monitoring control modules.
Transmission control modules can adopt different control strategies to refine the management strategies into the actions of the network facilities. It should be noted that different controllers can have different control strategies.
Monitoring control modules can control the information collection of different monitoring nodes according to the network monitoring strategies, such as the monitoring frequency and the types of data collected.
In addition, the monitoring control module and the transmission control module in a controller are interconnected. The transmission control module can send a request to the monitoring control module (2a) and the monitoring control module can provide the control data of facilities to the transmission control module (2b). Controllers in the same domain can communicate with each other through the east-west interfaces.
3.4. Management Subsystem
Management subsystem is the center of the network management architecture and responsible for formulating network management strategies and network monitoring strategies. In addition, management subsystem also provides internet service providers (ISPs) with external interfaces (j). Management subsystem is mainly composed of three modules: storage modules, processing modules and application modules.
Storage modules are responsible for filtering, classifying, compressing and storing collected data.
Processing modules can retrieve data from storage modules (3a) and are responsible for updating and maintaining the QoE models of different end-users by mining the characteristics and relationships of stored data, which is the preliminary preparation for the strategy formulation of application modules.
Application modules are the “core” of the network and responsible for formulating network strategies. The input of the application modules includes stored data and real-time monitoring data (3b, 3d) and the output includes network monitoring strategies and network management strategies (d, g). ISPs can intervene in network management through the external interfaces of application modules and modify the network management strategies.
4. Network Operation Strategy
This section discusses the network management process for QoE in SAGIN. Due to the large number of end-users in SAGIN and the demand for multi-dimensional information of QoE models, the collected data will be generated at a growing rate. In order to formulate optimal network management strategies in time when the network environment changes, such as the arrival/leave of a stream or the changes of network topology, some network tasks should be carried out in advance. Therefore, we divide the management process of the network into online process and off-line process (Figure 2). Beside, a near-real-time network operation strategy is proposed to deal with the instability of management links in space domain and air domain (Figure 3).
4.1. Off-Line Process
The off-line process is responsible for processing collected information and providing data support for the online process. The off-line process mainly includes following tasks:
Periodic data collection: The monitoring nodes collect the information of end-users and network periodically according to the network monitoring strategies.
Data storage: The management subsystem stores the collected data in the storage modules;
Data mining: The processing modules maintain end-user QoE models based on the stored data.
Experience learning: The processing modules update the optimal management strategies of different network environments based on the feedback of the end-users. The result is used as empirical data in online process.
4.2. Online Process
The online process operates when the network environment changes and is arm to formulate optimal network management strategies in time. The online process mainly includes the following tasks:
Real-time data collection: Different from periodic data collection, real-time data collection collects the current state of the end-users and the networks when the network environment changes.
Preference prediction: Based on the current information and QoE models, determine the kinds of end-users and predict the end-users’ preferences.
Network management: The management subsystem formulates network management strategies based on the current information, QoE models and empirical data.
Network control: The controllers in each domain receive the network management strategies and control the action of the network facilities.
4.3. Near-Real-Time Operation Strategy
Due to the instability of network management links, the space domain and air domain cannot maintain a stable connection with management subsystem in many scenarios, such as the mobile communication of UAVs, and the transmission of the real-time information to management subsystem is difficult to ensure. Therefore, we propose a near-real-time network operation strategy to reduce the dependence of network management on real-time information. The management subsystem divides continuous time into time intervals and formulates network management strategies by collected information in each time interval. We divide the near-real-time operation strategy into four steps:
Data collection: Assume that the maximum period of disconnection between air network or space network and management subsystem is a time interval. The controllers store all collected information in the time interval and upload when connect with the management subsystem.
Environment simulation: The management subsystem maintains a network environment model and predicts the network environment in the next time interval based on the received information from the control subsystem.
Strategy set: Based on the simulation of the network environment, the management subsystem formulates a set of network management strategies, corresponding to the possible network environment in the next time interval.
Practical operation: When the management subsystem connects with the controllers, the management subsystem will send the set of network management strategy of the next time interval to the controllers. In the next time interval, the controllers only need to implement the corresponding network management strategy of the current network environment in the network management strategy set without the participation of the management subsystem.
5. A Study Case
5.1. Simulation Environment
In this section, we study a case of QoE driven cross-domain network management for SAGIN in a limited area (Figure 4). Firstly, we construct a simulation environment using Python and ns2 (Table 2). Assume that there is only one LEO satellite serving in the area at the same time and the movement and handover of LEO satellites are ignored. UAVs are simulated to hover in a fixed area as air base stations and the micro stations are simulated as the access points of the ground domain. Assume that the controllers in the space, air and ground domains are GEO satellites, control stations and macro stations, respectively, and the management subsystem is the data center. We set coverage for each access node and the coverage ratio of each domain, which is the ratio of the maximum area serviced by the domain to the total area is shown in Table 2. Therefore, the accessible nodes of the end-users in different locations could be different. In the simulation environment, the end-users within the coverage of the ground network are denser than other areas. The simulation details are described as follows:
QoE model: We design a simple QoE model represented by the satisfaction of end-users [27]. Assume that the QoE model is known and the influencing factors of QoE include cost, delay and transmission rate. The weighted product of the utility function value of influencing factors represents end-users’ QoE. In the experiment, we set up three kinds of end-users (see section 4.2).
Monitoring subsystem: Considering an ideal situation, the monitoring subsystem can collect and upload the monitoring data in real time and the category of each end-user is known.
Control subsystem: Controllers in different domains adopt a simple network resource allocation strategy to equally allocate network resources to the end-users they serve.
Management subsystem: The management subsystem adopts a task allocation strategy based on Q-Learning algorithm (see section 4.3) and manages the network by allocating network task to the networks in different domains.
5.2. QoE Model
The QoE of end-users is expressed by the satisfaction of end-users with network services and the influencing factors of end-users’ satisfaction includes cost, delay and transmission rate. Firstly, we design a utility function for each influencing factor based on utility theory [27]. (1)Cost
The cost utility function of end-useris as follows:
whereis the network cost andis the maximum network cost acceptable to end-users. (2)Delay
The delay utility function of end-useris as follows: where is the network delay, is the maximum delay of the end-user andis the sensitivity. (3)Transmission rate
The transmission rate utility function of end-user is as follows: where is the network transmission rate. and represent the maximum and minimum network transmission rate demands of end-users, respectively.
The utility functions of cost, delay and transmission rate all conform to quadratic differentiability, monotonicity, concavity and convexity. The proofs can be found in literature [27]. (4)Satisfaction
The satisfaction of end-useris expressed as: where , and are respective the weighted values corresponding to the utility functions of cost, delay and transmission rate. In the simulation experiment, we assume that these weighted values are known. The end-users in the area are expressed asand the sum satisfaction of end-users in the area is expressed as:
We set three kinds of end-users: non real-time users (NRTs), real-time users (RTs) [28] and delay sensitive users (DSs) [29] (Table 3). NRTs are the end-users requiring high transmission rate, such as downloading emails; RTs represent video users and language users, such as real-time road monitoring and network call; DSs require low latency, such as automatic driving. In the experiment, the probability of follows a Poisson distribution with parameter and the duration time offollows an exponential distribution with parameter .
5.3. Task Allocation Strategy
Describe the goal of the network task allocation strategy:
We treat each stream as a network task and take the arrival of a network task or the end of a network task as a moment. The algorithm arms to improve the sum QoE of end-users in the area by allocating different networks to optimal network tasks. Suppose that the same network task cannot change the access node and one end-user has at most one network task at the same time. Therefore, the number of the end-users serviced by the networks equals the number of allocated tasks of the network. The network state depends on the network task allocation at the previous moment and the change of network state is a Markov process. Therefore, we describe the network task allocation as Markova decision process (MDP). We define the state, action and reward of MDP and the approximately optimal strategy is obtained by Q-learning algorithm. (1)Network state
The network state is expressed as: whereis used to describe the latest network task and represent the states of the wireless communication nodes in SAGIN. is composed of the remaining bandwidth , the remaining power and the remaining end-user number of the communication node : (2)Decision time and action
In the process of network operation, the decision needs to be made only when there is a new network task arriving. The optimal communication node needs to be allocated to the end-userin the accessible communication nodes . Therefore, the action set of end-user is defined as: where is the network state and means that the network refuses the network task. (3)Reward
Adopt to indicate the long-term reward obtained in network state when taking action and to represent the end-users’ satisfaction in the network state after taking action .is expressed as: where reflects the long-term penalty caused by refusing end-user. (4)Discretization time
We discretize continuous time into a set of constant time units, which is lower than the duration time of any network state: where represents the reward within . (5)Q-learning algorithm
The Q-learning algorithm can obtain the approximate optimal strategy by continually interacting with the network and is a common method of MDP.
We define the value of state-action as , which represents the expected long-term discounted reward of network state when select action . represents the network state of next moment. We update the as follow:
Where represents the learning rate, represents the end-users’ cumulative reward value within the duration time and is the integer ratio of to :
Adopting -greedy strategy, the network randomly select action in with probability as exploration mode and selects the action with the largest with probability as development mode. The Algorithm 1 is shown below:
| ||||||||||||||||||||||||
5.4. Experiment Results
During the experiment, we first run the network simulation environment until the network environment is stable. We iterate the proposed Q-learning algorithm for 20000 steps (about 2400 s), making the of each state tend to be stable, and start timing. We take that end-users randomly access network nodes in different domains with no management as a comparison. As shown in Figure 5(a), the sum QoE of end-users in the area with both QoE management and no management remains in a stable range and QoE management increases by 30% compare with no management. According to formula (5), the sum QoE is determined by the number of end-users and end-users’ QoE in the area. As shown in Figure 5(d), the number of end-users with QoE management and no management is within the same interval. From the QoE distribution of end-users, it can be seen that end-users’ QoE in the area with QoE management shows an overall increase compared with no management (Figure 5(b)). It can be inferred that the QoE of most end-users in the area increases. Therefore, it can be inferred that the increase of sum QoE in Figure 5(a) is mainly due to the increase of most end-users’ QoE in the area. Figure 5(e) further confirms our inference that more than 80% of end-users’ QoE increases.
(a)
(b)
(c)
(d)
(e)
(f)
As shown in Figure 5(c) and Figure 5(d), there are fluctuations in the number of end-users in each domain or in the area. We believe this is due to the random appearance of end-users in the simulated environment. Interestingly, the fluctuations in the sum QoE of end-users are relatively small (Figure 5(a)). We think this is mainly because we adopt the same algorithm for network control to allocate network resources, equally allocating network resources to each end-user. Since the resources of the network are constant, the sum QoE in the area depends on the conversion efficiency of resources to end-users’ QoE, rather than the number of end-users. In addition, according to Figure 5(f), QoE management can reduce the rejection rate to end-users due to network congestion. Thus, QoE management can slightly increase the number of end-users serviced in the long time.
According to the number of end-users at a certain time in Figure 5(c) and Figure 5(d), it can be seen that when the number of end-users in the area is constant, the number of end-users served in the space domain and air domain increases by QoE management, while the number of end-users in the ground domain decreases accordingly. It shows that QoE management compromises the demands of some end-users for delay and cost, and allocates them to the space domain and air domain for more network resources. At the same time, with the reduction of end-users in the ground domain, the ground domain can provide more network resources for the end-users serviced. We believe that these changes in the number of end-users in different domains improve the end-users’ QoE in the area.
We also show the experiment results adopting the near-real-time operation strategy, as shown in Figure 6. In the experiment, the management subsystem connects with the controllers at a fixed time interval. The step of environment simulation simulates the changes of the network environment in the next time interval according to the last obtained information of network environment and construct the possible changes of the network environment into a three-layer Monte Carlo tree [30]. From Figure 6(a), when the time interval is 5 s and 10s, the sum QoE of end-users increases by about 27% and 24%, respectively, compared with no management. Taking the interval of 10s as an example, it can be observed that the near-real-time strategy achieves a similar effect to the real-time strategy and the overall QoE distribution of end-users increases (Figure 6(b)). However, when the time interval is 20s, the increase of sum QoE in the area is small, about 14%. This is because that the three-layer Monte Carl tree can only simulate up to three changes of network environment in a time interval and no management is adopted when beyond the scope of network environment simulation in management subsystem. With the increase of time interval, the probability of network environment change beyond the scope of Monte Carlo tree simulation in a time interval will increase, so as to reduce the impact of QoE management in the time interval. The scope of network environment change simulation can be increased by increasing the layers of the constructed Monte Carlo tree, but will increase the burden of data processing.
(a)
(b)
6. Conclusion
In this paper, we propose a QoE driven cross-domain network management architecture for SAGIN and systematically design the subsystems, functional modules and operation strategies of the architecture. In addition, we propose a network task allocation strategy based on Q-learning and apply it to the simulation experiment. The results of the simulation experiment prove that the QoE in SAGIN can increase the sum QoE of end-users and also demonstrate that the near-real-time network operation strategy can also achieve effective management with a short time interval.
For further research, since SAGIN will be applied to more diverse scenarios [31], such as wireless sensor network (WSN) [32, 33], we should investigate the QoE models of end-users in different scenarios and clarify the methods of obtaining and maintaining the QoE models, so as to accurately predict the demands and preferences of end-users. For different QoE models, we should discuss the methods and deployment strategies for monitoring nodes to collect data in SAGIN. In addition, the proposed architecture has potential adaptability to SDN, which can be used to realize flexible network control.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no competing interest.