Abstract
A natural game interaction method can bring a good game experience to users, but pen-type interaction, which is one of the natural human-computer interaction methods, rarely appears in games, especially three-dimensional games. The main reason is that most pen-type interactive devices are developed for two-dimensional applications such as writing and painting. Whether the pen-based interaction can be extended from two-dimensional to three-dimensional is the key to whether the pen-based interaction can become one of the leading interactive methods of the game. The pen tablet is one of the most widely used pen-type interactive devices, and a virtual tour is an essential part of the game. Therefore, this paper takes this as a breakthrough and proposes a virtual roaming control method based on the pen tablet. First, we analyze the performance characteristics of the pen tablet and get the data information available for the pen tablet. Then, we design various virtual roaming actions such as walking, running, jumping, viewing angle rotation, view zooming, space-free displacement, acceleration, and deceleration and establish an association with the operation of the pen tablet. At the same time, the maximum likelihood function constraint was added to improve the error tolerance rate of viewing angle rotation. Finally, two games were developed to verify their effectiveness. The experimental results show that the pen tablet performs well in completing virtual roaming tasks, especially when users need to achieve similar writing and roaming tasks without switching devices back and forth. This maintains the user’s habit of writing with pen and paper and improves the interactive experience. The method is a new exploration of the application of pen-based interaction in games, which lays the foundation for future game development based on pen-based interaction.
1. Introduction
With the rapid development of virtual roaming technology, traditional human-computer interaction methods based on keyboard and mouse can no longer meet the increasingly complex and diversified requirements of human-computer interaction applications [1]. Consumers and researchers favor new interaction technologies such as pen, voice, and vision for their more natural and humanized interaction experiences. They have become the current hotspots in human-computer interaction research and application development.
Pen-based interaction is developed from the traditional paper-and-pen metaphor [2, 3]. Because of its inherent naturalness and comfort, it has become one of the most important interaction methods today, especially in writing and painting tasks. It seems very natural and convenient. Ren Xiangshi et al. [4] summarized the characteristics of pen-based interaction. Pen-based interaction simulates people's daily natural paper-and-pen working environment, helping people effortlessly handle various daily documents, event recording, and thought capture. In recent years, virtual reality has been widely used in all walks of life due to its 3I characteristics. In the design field, designers use virtual reality technology to perform high-fidelity digital modeling of various traditional building construction techniques and materials and fully record every detail, saving time, and reducing costs [5]. In the field of painting, using virtual reality technology, the painter can freely shuttle through the work. The drawing board is no longer a piece of paper or a screen but is derived into a three-dimensional space. The painter can paint light and sound with cool special effects only seen in movies. The painter can even draw inside the work, making the painting very lively and interesting [6]. In education, virtual reality technology has become a new educational means to promote the improvement of education. They used virtual reality technology to create a dynamic and realistic learning environment, such as brush writing [7] and word dictation so that students can enhance their memory through actual feelings. Compared with passive indoctrination, autonomous learning using virtual reality technology is easier for students to accept and arouse their interest in learning. However, common virtual reality interaction methods, such as handles and gestures, are difficult to achieve the best writing and drawing effects due to the limitations of equipment and technology, and the experience is far inferior to pen-based interaction. However, suppose that pen-based interaction is introduced into the virtual reality environment since it operates on a two-dimensional plane when encountering a three-dimensional task, such as virtual roaming. In that case, it is necessary to switch back to the original virtual reality interaction device. Switching devices back and forth will reduce operating efficiency and affect users’ experience.
The research motivation of this paper is to extend the pen-type interaction from two-dimensional to three-dimensional, that is, to realize the three-dimensional pen-type interaction, making pen-type exchange one of the main game interaction methods and bringing users a different game experience.
The contributions of this paper are as follows:(1)The ground roaming control method based on the pen tablet is proposed, laying the foundation for integrating more human actions in the future(2)An air roaming control method based on the pen tablet is proposed, laying the foundation for extending pen-type interaction from two-dimensional to three-dimensional(3)The successful implementation of the above two methods has laid a solid foundation for game development based on pen-based interaction(4)Since the pen-tablet device is relatively small compared to the keyboard and mouse device, it also contributes to the miniaturization of the virtual roaming device
2. Related Work
New human-computer interaction technology, especially pen-based interaction technology, has become a new hot spot in human-computer interaction due to its inherent natural and humanized interactive experience. At the same time, with the accelerated development of social industrialization, the upgrading of software and hardware technologies is also accelerating, and the mutual promotion of a range of interdisciplinary subjects has additionally promoted pen-based interaction research to emerge as a hot spot.
Richard et al. [8]designed a pen data entry system to facilitate clinicians to record information and improve work efficiency. Anne Marie Piper et al. [9]proposed a multimodal approach for treating the elderly with aphasia, which has specific help in regulating patients with aphasia. Stéphane Chatty et al. [10]applied pen-based interaction technology to video editing and replaced digital ink with video content. The results showed that the pen-based interaction method could better cultivate user creativity. Guo Junxiu et al. [11]proposed a paper-pen-based projection interaction method and designed applications such as word spelling. The experiment proved that the process is natural, comfortable, and strongly practicable. Xing Shuangqiu et al. [12]applied pen-based interaction in virtual home design and provided a reference prototype for applying pen-based interaction in home design. It shows that pen-based interaction has been widely used in various fields.
In addition to studying the application scenarios of pen-based interaction, the researchers also studied the user's behavioral habits of pen-based interaction and the performance characteristics of the pressure-sensitive pen. Juan Pablo Hourcade et al. [13]found that most older people’s touch operations are more accurate than clicking on targets, which provides practical, useable, and enjoyable use of pen-based interactive devices for the elderly to provide an excellent theoretical basis. Zhang Xiaolong et al. [14]improved the existing pen-based interaction method to adapt to children's usage habits, significantly influencing children's learning. Inkpen K et al. [15]investigated the different needs of left-handed users for pen-based interaction and found that, compared with the right-aligned scroll bar, the left-aligned scroll bar is more conducive for left-handed users to operate. Xiaojun Bi et al. [16] obtained the pen parameters of intentional scrolling for interactive purposes and accidental scrolling caused by conventional writing and drawing through two studies, which improved the error tolerance rate of pen tip input. Jibin Yin et al. [17] experimentally found that the natural pressure range used for drawing and writing was concentrated between 0.82 N and 3.16 N, and the resting force of the pen tip on the screen was between 0.78 N and 1.58 N. This provides a systematic empirical and knowledge base for the research and application of pen interaction.
This paper mainly studies the combination of pen-based interaction and virtual roaming. Therefore, besides inspecting the characteristics of pen-based interaction software and hardware, learning the virtual roaming methods suitable for pen-based interaction is also necessary. According to the report by Professors Kruij and Riecke at the IEEE VR conference, virtual roaming methods can be divided into three ways: “full-motion cues,” “some motion cues,” and “no motion cues.” Full-motion cues refer to one-to-one physical movement, meaning that the actions performed by the user in the virtual environment are almost the same as in real life. For example, Schnack et al. [18] constructed an immersive virtual reality shopping environment that mimics natural shopping behavior and allows shoppers to walk freely in a virtual store, which provides a solution for the design of virtual shopping environments for online retail platforms. Some motion cues refer to movement in place. For example, Lpba et al. [19] built a virtual environment for gait training, and users walked on a treadmill to complete virtual roaming and balancing tasks. No motion cues refer to the virtual motion. The user is not performing the corresponding movement but only operating. For example, Xue Yanping et al. [20] designed and developed a virtual roaming system for a museum of ancient architecture, controlling the virtual character by keyboard and mouse to move forward, turn, squat, etc.
Since the pen tablet controls the avatar’s movement through the pressure-sensitive pen and does not require the user to walk, it belongs to the third category, virtual roaming without motion cues. Such methods often require less equipment and are easy to popularize. In addition, in recent years, the application of drones has expanded the activity space of virtual roaming so that virtual roaming activities are no longer limited to the ground, and some flight simulation games have gradually become popular. Therefore, this paper divides the virtual roaming control based on the pen tablet into two modes, land and air, and designs and develops corresponding games for these two modes for experimental verification.
3. The Design and Implementation of Virtual Roaming Control Based on Pen Tablet
3.1. The Overall Design of Virtual Roaming Control
The input device used in this paper is Wacom Intous Pro PTH-460, which belongs to the electromagnetic induction board type. The product can be connected to a computer via a USB-Type C interface or Bluetooth. The included pressure-sensitive pen can provide information on coordinates, pressure, and tilt values, as shown in Figure 1, and can support Unity3D. The mapping relationship between the pen-tablet operation and data signals is shown in Table 1.
Referring to the literature [21, 22], this paper designs walking, running, jumping, viewing angle rotation, view zooming, and other actions for the ground roaming, and designing free displacement, acceleration and deceleration, viewing angle rotation, view zooming, and other measures for the air roaming. Furthermore, according to the characteristics of the pen tablet and the pressure-sensitive pen and the user's usage habits, the signal data of the pen tablet and the roaming action are correspondingly established, as shown in Table 2.
3.2. The Ground Roaming Control Based on the Pen Tablet
RPG games have been a mainstream game genre since the 1980s. RPG characters' action (roaming) in the scene is an integral part of game development. Walking, running, jumping, and rotating on the ground are RPG characters' most essential and vital actions. Therefore, the research starts from the ground roaming and realizes the ground roaming control based on the pen tablet. This lays the foundation for integrating more character actions in the future.
Realization of walking action: obtain the pressure value applied by the pressure-sensitive pen to the pen tablet. When the pressure value is between 0 and 2, it will enter the walking state. The walking direction is from the current position to the specified position of the pressure-sensitive pen, as shown in Figure 2.
(a)
(b)
(c)
(d)
Realization of running action: obtain the pressure value applied by the pressure-sensitive pen to the pen tablet. When the pressure value is between 2 and 10, it will enter the running state. The running direction is from the current position to the designated position of the pressure-sensitive pen.
Realization of jumping action: the jumping state corresponds to the first button on the pressure-sensitive pen or tablet, and when the first button is pressed, the jumping is performed, as shown in Figure 3.
(a)
(b)
(c)
(d)
The rotation of the viewing angle corresponds to the angle formed by the pressure-sensitive pen and the X and Y axes of the pen tablet. Moreover, the pressure-sensitive pen needs to be in the sensing area above the pen tablet. At this time, the pressure-sensitive pen is not in contact with the pen tablet, but the system can still obtain the orientation information of the pressure-sensitive pen. When the angle between the pressure-sensitive pen and the X-axis of the pen tablet is(i)greater than 90 degrees, the tilt value of the pressure-sensitive pen is between -1 and 0, and the corresponding viewing angle is rotated horizontally in the first direction(ii)equal to 90 degrees, the tilt value is 0, and the viewing angle does not rotate in the first and second directions(iii)less than 90 degrees, the tilt value is between 0 and 1, and the corresponding viewing angle is rotated horizontally to the second direction
The first and second directions correspond to the left and right directions. The first direction and the second direction are opposite each other. The horizontal leftward effect of the viewing angle is shown in Figure 4.
When the angle between the pressure-sensitive pen and the Y-axis of the pen tablet is(i)less than 90 degrees, the tilt value of the pressure-sensitive pen is between 0 and 1, corresponding to the horizontal rotation of the viewing angle in the third direction(ii)equal to 90 degrees, the tilt value is 0, and the viewing angle does not rotate in the third and fourth directions(iii)greater than 90 degrees, the tilt value is between -1 and 0, and the corresponding viewing angle is rotated horizontally to the fourth direction
The third direction and the fourth direction correspond to the upper and lower directions, and the third direction and the fourth direction are opposite to each other.
The realization of zooming in and zooming out of the viewing angle: the zooming of the view corresponds to the touch ring on the pen tablet. When the hand rotates clockwise along the touch ring, the angle of view is enlarged, and when the angle is rotated counterclockwise, the angle of view is reduced.
3.3. The Air Roaming Control Based on the Pen Tablet
The ground roaming control mentioned above is still a two-dimensional pen-type interaction because the characters move on the same plane. But in many games, character activities are no longer limited to the ground, such as flying a plane. Therefore, the realization of air roaming control based on the pen tablet can lay the foundation for extending pen-type interaction from two-dimensional to three-dimensional.
The air roaming retains the two actions of rotation and scaling of the perspective in the ground tour and includes its unique free displacement action and acceleration/deceleration actions.
The realization of free position movement requires using the tilt value, the pressure value, and the button signal of the pressure-sensitive pen. The specific method is as follows:(i)Lift the pressure-sensitive pen, but the pressure-sensitive pen is still in the sensing area, and when you press and hold the pressure-sensitive pen button, it moves upward.(ii)Press down the pressure-sensitive pen. When the pressure value is in the range of 2–10 (downward movement range), press and hold the pressure-sensitive pen button, and it will move downward.(iii)When the pressure value is between 0 and 2 (translation range), and the angle between the pressure-sensitive pen and the Y-axis of the pen tablet is less than 90 degrees (the tilt value is between 0 and 1), move to the fifth direction, as shown in Figure 5.(i)When the pressure value is in the translation range, and the angle between the pressure-sensitive pen and the Y-axis is 90 degrees (the tilt value is 0), it does not move in the fifth and sixth directions.(ii)When the pressure value is in the translation range and the angle between the pressure-sensitive pen and the Y-axis of the pen tablet is greater than 90 degrees (the tilt value is between −1 and 0), it moves in the sixth direction.(iii)When the pressure value is in the translation range and the angle between the pressure-sensitive pen and the X-axis of the pen tablet is greater than 90 degrees (the tilt value is between −1 and 0), it will move in the seventh direction.(iv)When the pressure value is in the translation range and the angle between the pressure-sensitive pen and the X-axis of the pen tablet is 90 degrees (the tilt value is 0), it does not move in the seventh and eighth directions.(v)When the pressure value is within the set translation range and the angle between the pressure-sensitive pen and the X-axis of the pen tablet is less than 90 degrees (the tilt value is between 0 and 1), it moves in the eighth direction.
The fifth, sixth, seventh, and eighth directions correspond to four directions: front, back, left, and right. The fifth direction is opposite the sixth direction. The seventh direction is opposite the eighth direction.
The air acceleration and deceleration state settings include the speed of the displacement operation in the air, corresponding to the third button and the fourth button on the pen tablet, respectively. Press the third button to accelerate, and press the fourth button to decelerate.
3.4. Set the Fault-Tolerance Value
In actual operation, if the viewing angle is rotated horizontally, the tilt value with the X-axis or Y-axis of the pen tablet cannot always be maintained at 0. Therefore, it is necessary to set a fault-tolerance threshold for the tilt value. When the tilt value is less than this threshold, the system considers the user’s misoperation.
The maximum likelihood estimation algorithm determines the fault-tolerance threshold in this paper. The specific method is as follows:(1)Given a probability distribution D and a distribution parameter , suppose its probability density function or probability aggregation function is fD, and extract a sample X1, X2,..., Xn with n values from this distribution. The sampling of n values corresponds to the tilt value P of n pressure-sensitive pens; then(2)Use the maximum likelihood estimation algorithm to estimate to find the most likely value of , that is, to find a value among all possible values to maximize the “possibility” of this sampling.(3)Because the tilt value is discontinuous, the likelihood function L() is simplified asCalculate the value of .
Because everyone’s pen holding posture, holding pen height, sitting posture, etc., cannot be the same, the fault-tolerance threshold varies from person to person, and the setting value can be adjusted adaptively according to the user’s usage habits.
4. Experiment
To verify the control effect of the tablet-based ground roaming and air roaming mentioned in the previous section, this section conducts experiments through the two games “Knowing Numbers 1–5” and “Knowing Graphics”. Use the pen tablet for manipulation and roaming. And invite 15 volunteers to participate in the test experience.
The 15 volunteers (8 males and 7 females) were 20–24 years old (mean 22.0, standard deviation 1.0), and their vision was at or corrected to an average level. To ensure the consistency of the test conditions and process, the volunteers invited to participate in the test are all computer-related college students who have knowledge or exposure to virtual reality, and they are all right-handed users.
4.1. The Ground Roaming Control Experiment Based on Pen Tablet
The experimental task can be divided into finding numbers and writing numbers.(1)Look for the digital part. The user uses the pen tablet to control the virtual roaming of the characters, looking for the numbers 1–5 scattered in the scene. The user can choose to walk, run, jump, rotate the angle of view, and zoom the view during the roaming process, as shown in Figure 6.(2)Write the number part. After the user finds the number, he can enter the digital writing module. Figure 6(f) shows that the user needs to hold a pressure-sensitive pen and write the corresponding numbers on the pen tablet according to the game prompts like traditional pen and paper writing. The game also provides brush color, brush thickness, and eraser tools.
(a)
(b)
(c)
(d)
(e)
(f)
After finding number 1 and completing the task of writing number 1, proceed to the learning of number 2, that is, find number 2 and complete the writing of number 2, and so on. Users only use the pen tablet when roaming and writing numbers without changing the device.
4.2. The Air Roaming Control Experiment Based on Pen Tablet
The experimental task can be divided into finding graphics, drawing graphics, and homing graphics.(1)Look for the graphic part, as shown in Figure 7(a). The virtual scene is scattered with football, triangle cones, tires, orange warning lights, and other familiar objects in life, representing all kinds of shapes. The user uses a pressure-sensitive pen and a pen tablet to roam the scene, looking for objects scattered in the scene.(2)Drawing graphics, as shown in Figures 7(b) and 7(c), when the user finds the graphics in the virtual scene, a prompt box for introducing the shape of the object will appear. Click the “Start” button to draw the graphics. Like writing numbers in Section 4.1, we can choose to draw with strokes of different colors and sizes, and we can also use tools such as eraser sassafras to modify the graphics.(3)As shown in Figure 7(f), the original object is not in the original position, and the user uses a pressure-sensitive pen to move the object corresponding to the image (similar to roaming in the air) to return it to its original position.
(a)
(b)
(c)
(d)
(e)
(f)
The graphics involved in this game are circles, squares, and triangles. Find the round object, draw the circle, put the round object in place, and then enter the square's learning. The square’s learning process is the same as that of the ring, and so on. The user only uses the pen tablet to operate when roaming on the ground, drawing and moving objects in three-dimensional space (the air roaming) without changing the device.
4.3. Discussion
After completing the test, the experimenter was asked to use a 7-level Likert scale [23]to evaluate the completed task subjectively, one being very dissatisfied and seven being very satisfied. The evaluation content includes ease, comfort, immersion, fun, and natural interaction. The evaluation results are shown in Tables 3 and 4.
As shown in Tables 3 and 4, it can be seen that the ground roaming control based on the pen tablet has a better user experience. It is a combination of walking (4.887), running (4.605), jumping (5.068), viewing angle rotation (4.436), and view zooming (4.683). The experience scores were much higher than the median value (3.500). Similarly, the air roaming control based on the pen tablet also improves user experience, such as viewing angle rotation. It has the lowest overall experience score in the air roaming action, but it also reached 4.27. It shows that people readily accept pen-based interaction. Among all tasks, the total experience scores of writing numbers and drawing graphics are the highest, 6.060 and 5.993, respectively, which are much higher than the experience scores of virtual roaming control tasks based on the pen tablet. It reflects that because people have developed the habit of writing with pen and paper, the advantages of pen-based interaction in writing and painting applications are more prominent.
According to the feedback of the experimenter, the virtual roaming control based on the pen tablet still has some shortcomings: it is not easy to control the pressure of the pen tablet when walking or running; when freely moving in three-dimensional space, it is challenging to balance the accuracy and speed of the displacement; when the viewing angle is rotated, the picture is easy to shake and so on.
To avoid the accidental experiment and ensure the scientificity of the experiment, the experimenter is required to do multiple experiments. As shown in Figures 8 and 9, with the increase in the number of experiments, the time spent on the virtual roaming task using the handwritten pen tablet shows an overall decreasing trend, with apparent learning effects. For example, learning number 2 is the lowest from the beginning of 1.78 s. It is reduced to 1.12 s, and the drawing of a circle drops from the highest 6.10 s to 3.72 s. The completion time of other numbers and graphics has also decreased significantly, reflecting that even users who have never used a handwritten pen tablet before can still get through training and quickly master the pen-style interactive usage to complete virtual roaming tasks more proficiently. It shows that pen-based interaction is easy to accept, which allows for the widespread promotion of pen-based interaction applications.
In addition, after the experiment is over, the experimenter is also asked to fill out a questionnaire, as shown in Table 5. This questionnaire aims to understand the user’s real feelings and thoughts about the pen-based interactive application in virtual roaming to improve the user experience and create better pen-style interactive application products. The questionnaire’s content is designed by the Google HEART user experience measurement model, including Happiness, Engagement, Adoption, Retention, and Task Success. These five dimensions determine whether the pen-based interactive application has a good user experience in virtual roaming. The score is still based on 1–7.
Figure 10 shows the questionnaire results, where Q1-Q5 represents five questions, M represents the mean, and SD represents the standard deviation.
Q1 corresponds to pleasure, the most intuitive element in the user experience. M = 5.27, SD = 0.96, reflecting that most users are satisfied with the usability and ease of using pen-based interaction in virtual roaming. According to some experimenters’ responses, more sounds, animations, and other information should be added to the application to improve the pleasure.
Q2 corresponds to participation. Participation is users' deep participation in products/services/functions. It is used to express user attitudes. Participation is usually interfered with by user habits. Most expressions do not like/like to roam in the virtual world and are not interested in using pen-based interactions in the game. M = 5.13, SD = 0.9, reflecting that most users are willing to participate in pen-based interaction, providing a prerequisite for pen-based interactive applications in virtual roaming.
Q3 corresponds to acceptance. Acceptance is an important indicator of whether a product is easy to use. How to enable users to use the product in a short period has a strong prompting effect on whether it can be widely used, M = 5.86, SD = 0.63, which reflects the relatively unified opinions on the use of pen-based interaction in virtual roaming, and people’s acceptance is high, and it is quicker to get started. It is also the original intention of the author.
Q4 corresponds to retention. Retention is the key to the long-term survival of a product. The core is not to let users get bored. M = 4.4, SD = 1.12, reflecting the different attitudes of each user. According to the feedback of experimenters, some are more inclined to use keyboard and mouse devices. It is also a common problem in new products. Remind developers and researchers to design pens. In the case of interactive products, pay attention to the timely update of functions to increase the retention rate of users.
Q5 corresponds to the task completion degree, which includes completion effect, completion efficiency, completion time, and operation error rate. The application of pen-based interaction in virtual roaming can first complete the basic virtual roaming task requirements and then highlight the advantages of short completion time and low error rate. M = 5.2, SD = 0.86. In terms of task completion, most users feel that pen-based interaction is relatively simple to complete virtual roaming tasks and can meet their needs.
In summary, pen-based interaction has a good effect on virtual roaming due to its inherent naturalness, but it needs to improve user pleasure and retention. In future work, we will pay attention to virtual roaming. While completing the task, add animation, sound, and other elements and update the function in time to improve user pleasure and stickiness.
5. Conclusion
This paper proposes a virtual roaming control method based on the pen tablet. This method can use the obtained real-time data of the pressure-sensitive pen and the pen tablet to realize the control of roaming actions, such as walking, running, jumping, viewing angle rotation, view zooming, free displacement, acceleration, and deceleration, and provides the ground roaming and the air roaming two modes. In addition, two games were designed and developed for experiments on ground roaming control and air roaming control based on the pen tablet. The experimental results show that the pen tablet performs well in completing virtual roaming tasks, especially when the user needs to complete similar writing tasks and roaming tasks in an application. Using the method in this paper does not need to switch devices back and forth, and it keeps the user’s paper. The habit of writing with a pen enhances the user’s interactive experience.
Through the user experience evaluation, we also learned that there are still some shortcomings in the virtual roaming control method based on the pen tablet; for example, the pressure value is not easy to control in walking and running operations; in free displacement, the accuracy and speed of displacement are challenging to take into account; in the perspective rotation, the screen is easy to shake; and the game design is lacking in sound animation and so on. In the future, while satisfying the roaming tasks, we should also strengthen the design details of pen interaction applications, use some emerging technologies to solve the problem of screen shaking, develop more humane products, enhance the user’s pleasure and experience, and further promote the development of pen-based interaction.
Data Availability
The data used to support the findings of this study can be obtained from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was partially supported by the National Natural Science Foundation of China, General Program, Grant no. 62171174. At the same time, the authors would like to thank the students of the Virtual Reality and Human-Computer Interaction Laboratory for their participation and valuable comments.