Abstract
Currently, the use of 360° videos combined with virtual reality (VR) techniques in the educational context has been considered a promising field of research. However, there is still a shortage not only of educational serious games (ESG) that implement such resources but also of models that can enhance the player’s experience and evaluate their performance. This work presents a decision model for SGE with 360° videos. The model was developed from artificial intelligence (AI) techniques with the aim of enhancing the player’s learning using immersion mechanisms and pedagogical reinforcement strategies based on the player’s experience and performance evaluation. The model was integrated into the SG The Mystery of Pandora to evaluate its efficiency. The game was evaluated with a sample of 52 participants aged between 13 and 63 (, ). The results showed adjustments made by the model in the final performance of the players based on their exploration in the 360° videos. Such adjustments allowed identifying players with learning difficulties and recommending pedagogical reinforcement to enhance learning. It was also possible to verify that players would win the game without the minimum knowledge expected about the subject if the decision model was not used.
1. Introduction
Immersive virtual reality (IVR) has been considered in recent years as one of the technological innovations with great educational potential [1, 2]. It allows its users to immerse themselves in a virtual environment and interact directly with its content. In this aspect, immersion has been identified as an essential factor in the learning process [3]. It allows users to be directly involved with situations and events related to the contents worked in the learning process. One of the promising technologies to experiment with IVR is the use of 360-degree videos [1]. 360-degree videos present more realistic experiences, allowing the user to be deeply immersed in their content [4]. In addition, it is possible to create more personalized experiences with fast, simple, and accessible content, enabling a more targeted form of presentation [3].
In a similar way to IVR environments with 360-degree videos, educational serious games (ESG) have been showing significant results in several areas, such as in health, whether for training or continuing education [5, 6] or rehabilitation of patients [7]. Despite the potential of 360-degree videos in educational environments, using IVR and 360-degree videos has been little explored in ESG. Most environments that use such technologies do not implement game techniques and therefore fail to take advantage of the full potential that serious games can promote, particularly in motivating behavior change and knowledge retention.
It is worth mentioning that through ESGs, it is possible to evaluate the user’s performance in real time and verify the learning progress and its results. For Bellotti et al. [8], the performance evaluation allows for adaptation and customization of the environment to meet the user’s individual needs. Adaptation can be carried out from pedagogical reinforcement through the recommendation of specific content, the presentation of appropriate feedback, or even the change of learning styles. Performance evaluation can guide the educational process, opening paths for skills and abilities to be worked on and strengthened.
Combining the playful and challenging universe of ESGs with IVR and 360-degree videos has been seen as a very timely proposal for developing educational environments [9]. However, if on the one hand this union can provide more realistic and immersive games, favoring important aspects such as involvement and empathy, on the other hand, they come with important challenges regarding the use of 360° videos. This is because, in 360-degree videos, players are mere observers of the scene, there is no active interaction with the characters (human actors), and the players have the autonomy to explore what is happening around them but cannot intervene directly in the story that is being presented. Another fundamental concept of IVR is presence. Witmer and Singer [10] claim that presence refers to the individual’s subjective experience of being in an environment, even when physically being in another. According to Tamborini and Bowman [11], the sense of presence can be classified in three ways: spatial presence, related to the virtual environment or objects, provides the player with the sensation of being physically in the game’s virtual world; social presence is related to the way players interact with virtual social actors as if they were real; and self-presence consists of the sensation in which players treat their virtual character as if they were themselves.
In digital games, the sense of presence can be enhanced when some variables are worked on, such as reach of sensory information, which involves realism in the construction of the world (quality of graphics, three dimensionality, and sound); control of the perception of the environment, which consists of the ability to explore the scenario/map and; and ability to alter the environment physically—which consists of the ability to manipulate objects in the virtual world [12]. Enhancing the sense of presence in ESGs can provide a richer experience and, consequently, greater learning opportunities. With the use of game techniques (goals, challenges, scores, rewards, etc.), it is possible to assign a greater role to the player, allowing him to interact more actively with the environment, expanding the possibilities of greater engagement and retention of knowledge.
Considering the benefits and challenges of using ESGs, IVR, and 360-degree videos, the present work presents an intelligent model that addresses evaluative, pedagogical, and immersive aspects in serious games based on immersive virtual reality with 360-degree videos. Such aspects still need to be explored in serious games based on 360-degree videos, and a gap needs to be filled. In the works found in the literature, it was not possible to identify models that address several of these aspects related to the player’s experience in serious games based on 360° videos, mainly concerning the evaluation process. Thus, the main contributions of this work are the following: (i)Development of an intelligent model for evaluating player performance. The model does use not only the results obtained in the execution of the tasks but also the characteristics of exploration of the player in the environment. That is, it uses both the states of knowledge (score) and the behaviors (player experience) of the player for the composition of the evaluation of his performance(ii)Implementation of a recommendation mechanism and pedagogical reinforcement strategy for the player, observing their difficulties and deficiencies in the concepts presented(iii)Development of immersion mechanisms to attract and direct the player’s attention to the narrative elements in the game environment, which are important for the learning process
Applying this model to immersive ESGs with 360-degree videos is expected to enhance not only immersion but mainly player engagement in order to favor their chances of learning.
2. Serious Games with 360-Degree Videos
Dörner et al. [13] define a serious game as a digital game that was developed to entertain and achieve at least one specific objective. Machado et al. [14] understand this “specific objective” as a purpose that must exist in the game beyond entertainment, taking as an example of aiding in the educational processes of learning, awareness, and physical rehabilitation in patients, among others. Petersen and Ekambaram [15] highlight some characteristics that should be considered in SGs with educational purposes: they can be recognized for (i) providing chances to learn content not previously internalized, (ii) allowing timely feedback of student interaction during the game and presenting student performance at the end of each session, (iii) providing opportunities for reflection through virtual experiences and to repeat them to revisit the experiences; and (iv) providing a safe environment to experience contextualized learning.
Recently, increasing attention has been given to ESGs based on immersive virtual reality (IVR), as they are immersive and capable of promoting cognitive learning. ESGs based on IVR environments have the potential to create simulations of real or imaginary situations, allowing for a greater sense of player involvement and immersion. In the context of video games, immersion is understood as a phenomenon in which the player feels part of the experience as a whole [16]. The immersion is perceived similarly to the flow state described by [17]. Csikszentmihalyi [17] characterized the flow state as the condition in which people are so involved in an activity that nothing else seems to matter; the experience itself is so pleasant that people will do it even without expecting anything in return, just for pure pleasure (intrinsic motivation).
The use of immersive 360-degree videos in ESGs has become a promising proposal due to its ability to offer more realistic experiences with a greater sense of immersion and presence that, when combined with gaming techniques, has the potential to generate more immersive environments. Many authors have called the use of 360-degree videos in virtual reality (VR) environments cinematographic virtual reality.
Despite the scientific community widely uses the term cinematic virtual reality (CVR), there is still no consensus on an adequate definition. For Reyes and Dettori [18], CVR is a variety of VR closest to the cinema universe. According to Macquarrie and Steed [19], CVR is a broad term that ranges from passive 360-degree videos (without interaction) to interactive narrative videos that allow the viewer to affect the story. Generally speaking, the term is often used when cinematic experiences are produced in 360° video format and displayed from different immersive virtual reality devices or platforms, such as VR glasses (e.g., Oculus Rift, Google Cardboard, and Daydream).
In the context of digital games, ESGs traditionally focus on exploring tasks to produce educational experiences; however, this can pose a problem regarding the use of CVR in ESG due to the “passive” nature of interactivity. However, this limitation can be minimized by composing tasks with more interactive interfaces from the overlay of graphic elements (2D, 3D, and 360° video), thus enabling more interactive interfaces and increasing the player’s chances of engaging in cognitive activities.
360-degree videos have been drawing the attention of the research community for having great potential to generate highly immersive environments [20]. For Elmezeny et al. [4], 360-degree videos present a new viewing experience, allowing the audience to be deeply immersed in their content. However, [1, 4] emphasize that interactions are still restricted mainly with regard to the following points: (i) changing point of view (field of view) by the user and (ii) selection of hyperlinks integrated in the video at different times and areas, that is, loading another 360° video, replacing the current scene, and giving the sensation of transition to another virtual environment.
Argyriou et al. [9] suggest that immersive experiences with 360-degree videos in IVR must consider two fundamental aspects: (1) the design of the experience and (2) the design of the interaction. (1)Experience design: it concerns all the elements that support the development of the experience itself, that is, the media resources, the flow of the story, the elements of the scenes, and their connectivity(2)Interaction design: it complements the experience design related to navigation in the virtual environment, that is, with transition and progression in scenarios, interaction, and feedback. A strategy used in scenario transition and progression is the distribution of points of interest. Points of interest are graphic elements superimposed on the scene that attract the player’s attention [21]. Still, regarding interaction design, game techniques are useful in favoring user/player engagement in the scenes
The use of 360° video in an immersive experience requires different approaches compared to traditional videos (2D) [22]. In traditional videos, information is viewed from a single point of view, and there is control over what the user (player) sees since sequence and rhythm are established to present the information and content. In 360-degree videos, there is greater flexibility, and the user has the autonomy to choose where to explore. Brillhart [21] considers that a good strategy in these cases is using points of interest in the scene planning process. Although it seems obvious, Brillhart [21] claims that betting on POIs helps direct the user (player) to make decisions, whether to visualize information or perform actions (e.g., transition between scenes, perform tasks, and end the game). In this planning, you should consider the player’s point of view because that is what you will see first. As an example, Figure 1(a) presents the use of POIs in the work of [9], whose objective is to guide/direct the user to a certain activity. Figure 1(b) illustrates an example of the use of POIs implemented in the work of [23]; it shows a stereoscopic view of an operating room where information about one of the equipment that composes the room is presented.
(a)
(b)
Several studies have shown the immersive potential of virtual environments in learning scenarios with 360-degree videos [9, 24, 25]. However, with regard to ESGs with 360-degree videos, it is possible to identify that there is still a significant shortage. Most of the works found in the literature are gamified environments; that is, they are environments that incorporate some game elements. Thus, this section presents some applications that use 360-degree videos in different types of learning activities.
Barsom et al. [25] present a study with VR resuscitation training, a virtual environment with 360-degree videos for training in cardiopulmonary resuscitation (CPR). The application can be run on a smartphone in combination with a VR headset (e.g., Google Cardboard or Samsung Gear VR). Through video, various options of actions are presented to the player. The player must choose an option that presents a correct sequence to proceed in the training scenario. The training was carried out with 40 high school students, in which the results suggest that the IVR environment resulted in students’ greater self-confidence to perform CPR.
In the work of Argyriou et al. [9], the authors propose a conceptual framework of gamification for applications of IVR in environments with 360-degree videos. A treasure hunt game was developed with a focus on learning and exploring the cultural heritage of the city of Rethymno, Greece. It uses 360° video to convey information and presents challenges in the form of a quiz. It was evaluated with 38 participants, suggesting good levels of participant engagement and a satisfactory level of immersion.
Choi et al. [1] present a framework for content development for a mobile IVR environment that combines 360° media (images and videos) and other interactive elements. This work developed a game-based learning environment for teaching marine biology. The environment has three scenarios allowing exploration and interaction with marine life, virtual fishing, and examination of different types of fish in various water depths. The authors concluded that the test results suggest that such content in IVR improves user immersion and promotes greater engagement. Ivkovic et al. [24] present an environment with 360-degree videos to present the cultural heritage of bridges in the city of Sarajevo, Bosnia. Ninety-six participants evaluated the application, and the results pointed to a high level of immersion.
From the cited works, it was possible to observe that the contents presented in environments with IVR and 360-degree videos point to the improvement of immersion and engagement as presented in [1, 9, 24]. Engagement plays an important role in educational environments because the longer the user/player stays, the greater the learning possibilities. There was also greater retention of knowledge in environments with 360-degree videos when compared to the use of traditional videos (2D) [25]. In general, all the works mentioned combine elements of games and 360-degree videos in their implementations, which may indicate that such elements have the potential to favor both immersion and user engagement. Therefore, it is possible to glimpse the potential of ESGs with the use of 360-degree videos, allowing not only the construction of more realistic and immersive environments but with the potential to increase the engagement of their users.
Table 1 presents a list of benefits and potentialities, as well as possible difficulties encountered in using 360-degree videos in educational environments such as ESGs, which may positively and negatively impact the learning process.
These are some important points that should be observed, as they can influence player immersion and engagement.
3. Method
3.1. Decision Model Proposition for Educational Serious Games with 360-Degree Videos
The use of immersive virtual reality and, lately, 360-degree videos in ESG has been presenting themselves as a new paradigm that can provide more engaging experiences, which can contribute to improving the teaching-learning process in a more interactive and motivating way. However, despite the potential positive impacts of 360° videos on ESGs, cognitive barriers (difficulties for knowledge to be internalized) still need to be overcome, especially with regard to the role of the player. Argyriou et al. [9] and Felix et al. [6] claim that among the main barriers, we can list (i) distraction from the content (loss of focus), (ii) the time invested in carrying out the tasks, and (iii) performance evaluation strategies. Thus, it is necessary to create mechanisms to eliminate or minimize such barriers in such a way that ESGs can create conditions for players to develop appropriate knowledge about the theme presented in the game.
The present work presents an intelligent model applied to immersive ESGs with 360-degree videos. It is aimed at acting mainly under the mentioned cognitive barriers, focusing on the following aspects: (i) evaluating the player’s performance, (ii) pedagogical reinforcement strategies, and (iii) immersion. The goal is to provide greater engagement and enhance learning. The decision model is composed of three components: (i) CTT component (classical test theory), (ii) pedagogical component, and (iii) immersion component. Figure 2 illustrates a simplified view of the decision model architecture.
3.1.1. Decision Model Components
In this section, a description of the decision model components will be presented, as well as their integration and information sharing used by the decision model. To demonstrate the application of the decision model, the knowledge base (tasks, answers, weights, feedback, etc.) adapted from the work of [5] will be used, addressing the issue of domestic violence against women (DVAW).
(1) CTT Component. The CTT component is responsible for evaluating the player’s performance during the resolution of the game’s challenges; that is, it consists of verifying the accumulated score of the players during the game. The score was calculated using the classical test theory. Equation (1) presents the calculation from the CTT [32]. where represents the total observed score of the -th individual, which is the sum of the points of the -th individual, is the true score of the -th individual, and is the random error. As Pasquali [32] points out, the (true score) would be “the real magnitude of what the test wants to measure in the subject, and that would be the T itself if there were no measurement error.”
The player’s evaluation process is based on their attitudes towards the answers to the game’s challenges from the scenes with 360° video. The performance is calculated according to the alternative chosen by the player in the level challenges. The accumulated score at the level will be called “individual performance (IP)” and will be calculated according to the formula illustrated in the following equation based on the CTT. where represents the player’s performance in the level, corresponds to the degree of importance, and is the weight value of the response given by the player. Each alternative receives a specific value, and each question receives a weight, a degree of importance (DI), given by an expert in the area of the addressed content. Taking as an example the concepts used in Almeida et al. [5], Table 2 illustrates the relationship between the concepts covered in the game and their respective weights (DIs) applied to the challenges that address such concepts. For example, an item that only addresses the concept of gender has a weight (DI) of 4.0. However, if another question addresses the concepts of gender and health simultaneously, it will have greater weight (e.g., ). In this way, the game presents items with different weights according to their degree of complexity (interrelationship between theoretical concepts).
Table 3 presents the value (weight) assigned to each item (alternative) of the answer given to the challenge, with ten being the most coherent, representing a more effective intervention. Both the weights of the DIs (Table 2) and those of the alternatives (Table 3) are parameterized values which can be modified according to the specificity of the theme used in the game. The weights presented in Table 2 address items addressed in the theme of domestic violence against women.
At the end of each level, the CTT component updates the player’s performance. This information is available in the player’s context data, that is, in the data representing the player at a given moment. The context concerns the information about the player profile that can be static (e.g., age) or dynamic, such as the score, the number of challenges completed, and the exploration data. Another information calculated by the CTT component is the individual performance rate (IPR). This rate refers to the percentage of player performance relative to the maximum performance allowed in the level obtained from the following equation. where is the performance rate, represents the individual player’s performance, and is the maximum score allowed in the level. The IPR presents the player’s rate of use in relation to the contents that have been worked on at a certain level of the game. Although traditionally, this information is adopted to assess the player’s final knowledge, here, it is part of the information that is used in the assessment process carried out by the decision model. Next, the other components (pedagogical and immersion) that integrate and use the data produced by the CTT component will be presented to evaluate the player’s knowledge.
(2) Pedagogical Component. Reinforcing knowledge about the concepts used in the game’s challenges is a way to provide greater chances of internalizing such concepts. Pedagogical reinforcement can favor learning and behavior change. One way to provide this reinforcement is to assemble pedagogical strategies based on the player’s performance, mainly verifying the difficulties encountered about the contents presented. The pedagogical component is responsible for evaluating the player’s performance and suggesting, when necessary, a pedagogical reinforcement to overcome such difficulties. The pedagogical component assesses the player’s level of knowledge not only by its score (produced by the CTT component) but also by analyzing the exploration of pedagogical elements of the game that are present in the player’s experience. In this case, the evaluation is carried out under two prisms. The first is individual performance (score) in the answers to the challenges, and the second is the level of exploration of the pedagogical elements that make up the challenge: elements of exploration and player immersion.
Regarding the exploration elements, they were organized and used as follows: (i) challenge viewing time (360° video), (ii) time to visualize the question, and (iii) time to visualize the pedagogical feedback of the challenge (the reflection). Evaluating the exploration of these elements allows us to verify the player’s behavior in relation to the time spent exploring the scene. For example, not fully viewing the video that presents a particular game challenge can lead to difficulties in understanding the challenge itself and the loss of important information for learning. On the other hand, the closer to the total (ideal) time allocated for viewing the video (challenge), the more adequate the exploration will be and the greater the gain of information.
However, it is observed that the dynamics of temporal exploration can present a high degree of uncertainty; that is, there are no guarantees (certainty) that the player will always use the ideal time to explore the challenges (viewing the videos, reading the questions and answers), reading feedback, exploring the environment, etc.). Such uncertainties may impact the final assessment of the player’s performance and the knowledge obtained. In the pedagogical component, these uncertainties are dealt with through the implementation of two fuzzy systems, one responsible for inferring the player’s exploration type and the other for making the final assessment of the player.
Fuzzy logic was chosen mainly for its ability to deal with elements of uncertainties, approximate reasoning, and vague and ambiguous terms. In educational performance evaluation processes, it is common to work with imprecise terms. As human reasoning involves all these elements treated by fuzzy logic, it was decided to select a technique that could represent human reasoning and knowledge in the most realistic way. A fuzzy system can incorporate the human way of thinking into the control system. Furthermore, fuzzy linguistic rules can represent knowledge explicitly, and this knowledge can also be explicitly used in the inference process. In the case of other AI techniques, such as artificial neural networks, on the contrary, they represent both knowledge and inference implicitly. Another advantage is that fuzzy rules allow for greater communication and formalization of modeled knowledge (rule base) between designers and specialists due to the use of natural language facilitating both the understanding of existing rules and the evolution of the model from the inclusion of new rules.
Fuzzy System Exploration Type (FSE). The purpose of this fuzzy system is to verify if the player is properly exploring the elements that constitute the challenge, in this case, if the time invested in each of the elements was adequate. When explored correctly, there is a greater chance of the player getting involved with the theme and the content being covered in the challenge. Therefore, the exploration time becomes an important point for the learning verification process because if the exploration is inadequate, for example, “too fast,” the chances of understanding the conceptual elements addressed in the challenge are lower and consequently in learning. The fuzzy exploration type (FSE) system deals with the uncertainties that occur in the players’ exploration during the completion of the challenges and shows at the end how adequate it was.
The FSE input variables are obtained from the player’s exploration of the game environment that is perceived and collected by the immersion component of the decision model. The inputs are processed by the fuzzy inference engine producing an output, a fuzzy value. After the defuzzification process, the system produces a crisp value which is the final result of the FSE. Figure 3 illustrates the system flow.
A description of linguistic variables is as follows: a fuzzy linguistic variable uses linguistic terms to express imprecise or uncertain values of a concept or a variable of a given problem. The FTE has three input variables: (a) timeChallenge, (b) timeResponse, and (c) timeFeedback. They can be defined as follows:
timeChallenge: represents the time it took the player to view the video in the zone of interest where the challenge takes place. When the challenge display time is adequate, it allows the player to experience important pedagogical aspects (content, language, and expressions). In addition, it allows the experience of sensations and feelings that can contribute to learning new concepts or others already existing. The linguistic values of the timeChallenge variable are insufficient, little adequate, adequate, and long. The membership functions of each fuzzy set are illustrated in Figure 4.
Each exploration element of the challenge has a minimum time required for exploration. Table 4 exemplifies 3 (three) challenges with the minimum time necessary for exploring the elements.
Taking the challenges in Table 4 as an example, assuming that a player spent 31 seconds viewing the video of the D3 challenge, then the entry “challenge viewing time” would be 73.80%. This value is obtained in relation to the total time for a full view of the challenge (video). Table 5 illustrates the result of applying the membership functions, and it appears that the player’s exploration time has a higher degree of belonging to the linguistic term “poor adequate.”
timeQuestion: refers to the time it takes the player to read the challenge question and answers and select one of the answers. Sometimes when reading is performed at speed, the player can lose focus on what he is reading, impairing the understanding of what is being explored. This variable’s linguistic values were as follows: insufficient, little adequate, adequate, and long. Figure 5(a) illustrates membership functions.
(a) timeQuestion
(b) timeFeedback
timeFeedback: refers to the time it takes the player to read the feedback corresponding to the answer selected in the challenge. Regardless of choice, the pedagogical feedback (reflection) brings a message so that the player can reflect on his choice regarding the presented content, being able to interpret and give meaning to this content. Thus, when the player does not read or does not spend the time necessary for such reading, the learning provided by pedagogical feedback can be compromised. Analogously to the exploration timeChallenge element, the pedagogical feedback message also requires a minimum amount of time to be read. This variable’s linguistic terms and membership functions assume identical values to those presented in the timeChallenge, as seen in Figure 5(b).
Exploration: this variable concerns the type of exploration that the player achieved after exploring the elements (video, question, and feedback) that make up the game’s challenges. The better the exploration, the greater the possibilities for the player to obtain a good performance in the game and, consequently, more effective learning. The variable has the following linguistic terms: very fast, fast, poorly adequate, slow, and very slow. Figure 6 illustrates their membership functions.
Fuzzy Performance Assessment System (FPAS). The fuzzy performance evaluation system is aimed at estimating the player’s final performance considering two aspects. First, the score refers to the completion of the game’s challenges and, second, the level of exploration achieved by the player in relation to the game’s exploration elements. FPAS seeks to adjust the player’s performance by exploring the elements responsible for presenting the game’s pedagogical content.
The FPAS receives two inputs that are processed by the fuzzy inference engine and produces an output referring to the level of knowledge (insufficient, reasonable, or good). Finally, the fuzzy value is converted into a numerical value representing the player’s final performance in the game. Figure 7 presents the FPAS flow.
The pedagogical component uses the final performance of the player to verify three situations: (1)If the player can advance to the next level(2)If the player needs pedagogical reinforcement(3)If the player needs to start again (game over)
A description of linguistic variables is as follows: as illustrated in Figure 7, FPAS has two input variables that were modeled on the following variables: (1) Performance and (2) Exploration.
Performance: refers to the player’s performance rate (IPR) obtained in the level. The CTT component calculates this information after the player completes all game-level challenges. This variable’s linguistic values were the following: very bad, bad, regular, good, and very good. Figure 8 illustrates membership functions.
Exploration: refers to the type of exploration the player achieved after interacting with the exploration elements (video, question, and reflection). This variable is calculated by the fuzzy subsystem “type of exploration” and has the following linguistic values: very fast, fast, poorly adequate, slow, and very slow. Figure 9 illustrates their membership functions.
Output variable Knowledge: is the result of the FPAS inference. The system classifies the player’s knowledge after solving the challenges that were explored at the game level. For this variable, the linguistic values were defined: insufficient, reasonable, and good. Figure 10 illustrates their membership functions.
The FPAS rule base consists of 30 rules. Table 6 presents an extract of the rules used in the application.
The FPAS rule base adjusts performance (score) to the player’s exploration level. This adjustment is intended to mitigate situations in which the player’s performance does not match the time used to explore the exploration elements of the challenges. For example, analyzing a hypothetical scenario in which the player spent only 61% of the time needed to view the challenge (video), 50% to read the questions and select the answers to the challenges, and 55% to read the reflections. Note that, in this scenario, there were no good levels of exploration and that such levels may affect the player’s final performance. It is noted that in this scenario, there were not good levels of exploration and that such levels can affect the final performance of the player.
According to the rules, the faster the exploration, the less adequate it is for understanding the contents presented, impacting the player’s final performance. On the other hand, the more adequate the exploration, the greater the player’s chances of achieving a good final performance in the game.
After completing all the challenges of the level, the pedagogical component updates the player’s context with information regarding their performance and checks if there is a need to apply pedagogical reinforcement. The purpose is to present new experiences so that doubts and difficulties can be clarified and the learning process is strengthened. The pedagogical component checks if the performance is between 30% and 60%; if yes, then a strategy will be defined, suggesting a set of new challenges. If it is below 30%, the model recommends that the player restarts the level again. If it is above 60%, advance in level. These values can be modified according to the strategy you want to adopt.
After defining the number of challenges that will be recommended to the player, the pedagogical component starts the process of selecting and recommending challenges. This process is carried out as follows: (1) identification of challenges in which the player performed poorly and (2) recommendation of challenges. (1)Identification of challenges: checks which were the challenges in which the player obtained the lowest scores. This information may indicate some difficulty on the part of the player, either in understanding the themes or in the contents presented. These challenges are called target challenges, that is, challenges whose contents need to be worked on again in the form of pedagogical reinforcement(2)Challenge recommendation: selects unexplored challenges that portray situations similar to those previously experienced by the player but which are targets of pedagogical reinforcement
In this step, the most similar challenges are selected based on challenges in relation to the targets, that is, those whose player presented difficulties. It is considered here that similar challenges may present similar approaches in relation to the contents worked. The concept of similarity is used to determine how one challenge is similar to another. There are several techniques used to determine the similarity calculation between objects. However, in this work, was adopted the Euclidean distance.
Euclidean distance is calculated as the square root of the sum of squared differences between a target challenge () and an existing, unexplored challenge () on all input attributes . The distance between the two instances, and with values xi and yi for the ith attribute, can be expressed by equation (4). The attributes selected to determine the similarity between the challenges were concept(s) addressed (in content), degree of importance, number of alternatives (answers), and level.
After finding the challenge with the most significant similarity, the decision model enables the challenge(s) in the 360° environment of the game so that the player can explore and immerse in a new experience. The objective is that from this new exploration, the learning can be strengthened. For each identified target challenge, the same selection and recommendation process will be applied until there is no longer a target challenge.
(3) Immersion Component (IC). The immersion component is aimed at enhancing the player’s immersion in the challenges presented with 360-degree videos to strengthen their engagement in the game and learning of the presented theme. The immersion component is responsible for collecting data regarding the player’s experience and making them available for the other components of the decision model, used both in the initial evaluation (in the CTT component) and in the final evaluation of performance and pedagogical reinforcement (pedagogical component).
Despite the engaging visual concept, using 360-degree videos causes some concerns [33], especially regarding player interaction with elements of the virtual environment [34]. The first concern is to ensure that the viewer does not miss important narrative (interaction) elements because they are looking in the wrong direction [20]. The second is to ensure that the player can properly interact with the elements, such as checking the player’s interaction time with the environment elements.
One way to minimize these problems is to create mechanisms that attract the player’s attention and direct him to such elements. Several techniques can be worked on in this process. Some are used during the recording of 360-degree videos and others in the production of scenes that are integrated into the game from game engines. Table 7 presents some techniques used in the game.
The decision model IC seeks to implement a part of these mechanisms in the game scene, as well as use positive reinforcement (feedback) strategies to maximize player engagement. The objective is to stimulate the player to feel part of the experience as a whole, encompassing all his attention and corroborating the concept of immersion [16]. Figure 11 presents a simplified architecture of the immersion component.
In a simplified way, it can be said that the immersion component perceives player information from the environment input device (gaze) and immersion data (spatial and temporal data) and performance (number of challenges, rate of performance). The IC presents a reactive structure in which the immersion mechanics processes the perceived data from a set of production rules in order to render the environment with the scenes in 360-degree videos. Sensory rendering in the game consists of the idea of enhancing the player’s experience with the addition of visual and sound effects, which can increase their sensory and potentially emotional immersion to the narrative content.
According to Sherman and Craig [37], when immersed in the virtual environment, the user (player) feels part of this environment, passing a feeling of full involvement. Here, engagement is being addressed in the immersion component from two perspectives, sensory immersion and emotional immersion. Sensory immersion is being treated from the point of view of VR, that is, in the execution of audiovisual resources that allow the player to be focused on the game world and its stimuli, enabling greater involvement and engagement. Such resources are directional graphics, adjustable and directional sound systems, pop-up windows, text messages, and sound messages.
Emotional immersion, on the other hand, is related here as an element of satisfaction, that is, the degree of motivation or satisfaction the player feels in carrying out activities in the game world. This approach is in line with the self-determination theory (SDT) proposed by [37]. SDT primarily addresses factors that enable intrinsic motivation. O’Brien and Toms [38] claims that when there is motivation in the practice of a task, there is a tendency for this task to be more satisfactory for the individual. The feeling of satisfaction can emotionally involve the player in the “story” of the game, providing richer and more pleasurable experiences [11]. The IC uses pedagogical strategies of positive reinforcement to motivate the player to overcome the challenges of the game, strengthening their self-confidence.
The intelligence of the immersion component is implemented from a production rule system, mapping perceptions into actions. The immersion component detects (or perceives) player context information which includes the following information: spatial data, temporal data, and performance data.
Spatial context data: refers to information obtained through the player’s spatial location (-, -, and -axes) in the game scene, precisely where and what the player is looking at based on their rotation on the -axis. From this information, it is possible to record the interaction with what the player observed and correlate it with some action that the IC can apply to the game environment. Based on the player’s spatial context, for example, it is possible to verify whether certain areas and elements of exploration in a 360° scene have been explored (Figure 12). Based on this information, make some decisions relevant to the game.
(a)
(b)
(c)
Temporal context data: concerns the time spent by the player in the interaction with some exploration element located in the 360° video scene, for example, the interfaces that present the challenges and the pedagogical feedback messages for the player. The IC records the time the player spent looking at the element in question. The temporal context data are used both by the IC (e.g., message feedback) and by the pedagogical component in evaluating the player’s performance.
Performance context data: refers to the player’s performance or score regarding solving game challenges. This information is used to monitor his performance and motivate you through positive reinforcement through feedback.
Table 8 presents input data perceived by the immersion component. This data is used to build the IC knowledge base, which is represented through a set of production rules.
Table 9 presents an extract of situations modeled and represented by rules to increase player immersion and actions to monitor their exploration.
To demonstrate an action performed by the IC, let us see the example of the rule adjusting audio. This rule monitors the player’s spatial location and acts on the audio in each zone of interest of the 360° scene. Based on the location, the volume of the voices of the human actors is automatically adjusted, varying in intensity according to the exploration of the environment; that is, when the player starts to turn his head to another zone of interest, the IC adjusts the volume in both zones as illustrated in Figure 13. The purpose is to attract his attention and keep you focused on the challenge and other elements that are part of the new area (zone) that is being explored.
(a)
(b)
(c)
The approaches used in the design of the IC were designed in the particularities of the use of 360-degree videos in ESG. The concern ranges from recording 360-degree videos to integrating with game scenes. This is necessary because part of the targeting and attraction of the player’s attention could be implemented while recording 360-degree videos. Applying targeting techniques in writing the scripts of the scenes can naturally facilitate the player’s involvement without the immersion component’s need for many rules and interventions. Excessive interventions can cause the opposite effect: disengage or even break the immersion.
The IC rule base was developed not only with the objective of enhancing the player’s immersion and engagement but also to provide the opportunity to collect information from their experience, allowing a new approach to measuring their performance in relation to the presented content, taking into account how the elements that present pedagogical content are explored during the game. In addition, it makes it possible to follow the player’s exploration to attract his attention and keep him focused and motivated to explore elements that are important for the learning in question.
4. Experiments
In order to evaluate the model, it was implemented and integrated into the serious game The Mystery of Pandora [39]. The game was designed to enhance knowledge and awareness of domestic violence against women (DVAW). In the game, the challenges are presented from 360-degree videos where players must explore the videos and complete the proposed challenges. In The Mystery of Pandora [39], the player takes on the role of an investigator and needs to investigate evidence of a crime from DVAW situations portrayed in 360-degree videos. Figures 14(a) and 14(b) illustrates one of the game’s challenges.
(a)
(b)
4.1. Participants
The participant sample used includes 52 participants (, aged between 13 and 63 years, , ) composed of professionals from a public health institution (Hospital Juliano Moreira (HJM)) (), students, and professionals liberals (i.e., self-employed professional with no employment relationship) from various fields of activity (). Of these, 67.3% () were female and 32.7% () were male. As for the use of digital games, it was found that the majority 36.5% () do not play games, 28.8% () rarely play, 21.2% () play weekly, and 13.5% () play daily. It was observed that only 15.4% () of the participants had already had some experience with 3D games or with the use of virtual reality glasses. Although this percentage is small, no impediments were observed that could compromise the progress of the tests. Despite the use of digital games not being a daily activity of most respondents, it was possible to observe that there was no resistance or difficulty when applying the tests of the game that implemented the decision model.
4.2. Experimental Procedure
The research was approved according to protocol no. 13498019.7.0000.5188 of the ethics committee of the Federal University of Paraíba in Brazil. For each evaluation participant, the same protocol was adopted with the following steps: (a) a dialog exposition presenting the test team and the research objectives, (b) sterilization of equipment and materials used during the evaluation, and (c) game testing and application of questionnaires (pretest and posttest). The assessment lasted approximately 45 minutes. Each participant tested the game individually using VR goggles, headphones, and a smartphone provided by the testing team. Figures 15(a)–15(c) show moments of carrying out the game test and filling in the collection instruments at the HJM premises. Figure 16 illustrates an assessment carried out at the participant’s home.
(a)
(b)
(c)
5. Results and Discussion
The game data log files were analyzed to assess the impacts related to the model’s performance and effectiveness. Such files bring information about the player’s experience in the various occurrences and events (route choices, etc.) during the game’s execution. Tables 10 and 11 present an extract of some information obtained from the logs produced after the application of the decision model. From this information, it was possible to assess whether there was any change in the player’s final performance (score), that is, if there was a recommendation for pedagogical reinforcement, which were the main categories/concepts that he felt difficulties among other information.
The data analysis found that in 13.46% () of the evaluated cases, there was an upward adjustment in the players’ final performance compared with the same scores without the model’s performance. In this case, the players could experience the exploration elements closer to the acceptable time for understanding and/or reflecting on the concepts involved in the game’s challenges. On the other hand, in 86.54% () of the cases, there was a downward adjustment in the final grades, showing that the exploration performed was inadequate based on the rules established by the decision model.
The adjustment that the decision model makes to the final scores of the players brings with it some impacts on the final result of the game, that is, if the player managed to solve the challenges (won), if he needs to start over and explore again (Game over), or if there is a need for pedagogical reinforcement so that he can enhance his knowledge in a certain concept worked on the theme of the game. As can be seen in Figure 17, the performance of the decision model has a direct impact on performance/evaluation.
The results show that approximately 46% () would achieve enough performance to win the game. Compared with the evaluation without the application of the model, the number of players who could win the game would have a high increase of approximately 35%. Therefore, there would be a significant number (, 80%) of players who could win the game without investing the minimum time to understand the elements that are part of the challenges, and as mentioned before, such elements are important for the process of learning proposed by the SG.
Another relevant data is pedagogical reinforcement; about 27% () would need some pedagogical reinforcement. Without the model, it would not be possible to identify such a deficiency, and players could finish the game without the chance to enhance learning or even without knowledge. Of the 14 players who underwent reinforcement, only 2 (14%) failed to win the game, and 12 (86%) completed the complementary challenges. In addition, it was also possible to observe that about 32% () would have to play again, that is, revisit the challenges once again for a better understanding of the topics covered, and in this case, pedagogical reinforcement would be insufficient. On the other hand, without applying the model, this number would be greatly reduced, remaining at approximately 2% (). This means there could be a greater number of players who would win the game without the minimum condition of use or knowledge desired.
Analyzing the time invested in the pedagogical elements of exploration (viewing 360° videos, the questions and answers of the challenges, and the feedback/reflection) from Table 12, it is possible to make some reflections that corroborate the findings illustrated in Figure 17. Approximately 23% () of the players used 60% or less of the time needed to view the videos (challenges), which may compromise the understanding of the content. About 29% () for the questions and 33% () for the feedback (reflections) also used 60% or less of the required time. This data points to the number of players who failed to win the game. Therefore, without proper exploration of the elements that make up the challenges, the chances of a good performance are lower.
On the other hand, when properly exploited, there is a greater chance of victory and learning. It is observed that approximately 63% (challenges/videos), 30% (questions and answers), and 38% (feedback/reflection) of the players invested more than 90% of the suggested time for exploration. As a result, the model generated an output of about 31% of players who had a longer exposure time to pedagogical resources. This data is much closer to the number of players who managed to win the game with the performance of the model and far from the 80% (Figure 17) that only the score was considered as a form of evaluation.
Regarding pedagogical reinforcement, Figure 18 illustrates the recommendation of challenges by category made by the model. “Gender and human rights” (50%) and “human rights” (22%) were the categories with the highest incidence of recommendations. These results suggest that the players had difficulties mainly in the challenges with some violations of women’s fundamental human rights. The categories with an approach in “gender” (11%) and “gender and health” had lower percentages of recommendation, suggesting greater understanding and understanding of the challenges that work with these categories.
6. Conclusions
The use of 360-degree videos and IVR in the context of ESGs is currently presented as a promising field of research in the educational field, as it seeks to provide its users with an environment where they can experience more realistic and engaging experiences with the contents. In this approach, users can feel the sensation of being part of this story.
In this work, a decision model was proposed, modeled, and implemented for the SG with a focus on 360-degree videos. The model seeks, above all, to enhance player learning. Thus, it was designed to act on some cognitive barriers that are usually present in environments with 360-degree videos, especially distraction from the content (loss of focus); in the performance evaluation, mainly identifying the limitations; and in the recommendation of pedagogical reinforcement to overcome such limitations.
The notes described suggested that the decision model incorporated into the SG The Mystery of Pandora could contribute to player engagement. As previously presented, the mechanisms (interaction and navigation) implemented in the model considered several specificities of 360-degree videos. This fact may have potentiated player immersion and engagement, enabling good results in this evaluation. In addition, the decision model made it possible to evaluate the player’s performance from a different perspective, analyzing not only the scores of the task responses and the player’s exploration of the game environment. This way of evaluating the players allowed the creation of personalized pedagogical strategies favoring the learning process. The evaluation of the game revealed encouraging results both from the point of view of player satisfaction and learning. It suggests that the heuristics worked in the game’s decision model were successful and provided a good experience for the player.
Although the model uses elements of a behavioral approach, such as exploring the environment and the pedagogical elements displayed in the game, as well as the player’s score in solving each proposed challenge, this research did not include other elements, such as social constructivism, cognitive load, and creativity. It is possible that the insertion of these elements could improve learning. However, it remains for future studies of how and under what conditions these elements can be incorporated into the pedagogical component in order to evolve the model and consequently improve the learning process mediated by the decision model.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Consent
All the participants are accepted for this experiment, and informed consent was obtained from all participants included in the study.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported by the Brazilian National Council for Scientific and Technological Development (grants 315298/2018-9 and 305914/2021-9).