Abstract
For an establishment of a skill evaluation method for human support systems, development of an estimating equation of the machine operational skill is presented. Factors of the eye movement such as frequency, velocity, and moving distance of saccade were computed using the developed eye gaze measurement system, and the eye movement features were determined from these factors. The estimating equation was derived through an outlier test (to eliminate nonstandard data) and a principal component analysis (to find dominant components). Using a cooperative carrying task (cc-task) simulator, the eye movement and operational data of the machine operators were recorded, and effectiveness of the derived estimating equation was investigated. As a result, it was confirmed that the estimating equation was effective strongly against actual simple skill levels (). In addition, effects of internal condition such as fatigue and stress on the estimating equation were analyzed. Using heart rate (HR) and coefficient of variation of R-R interval (). Correlation analysis between these biosignal indexes and the estimating equation of operational skill found that the equation reflected effects of stress and fatigue, although the equation could estimate the skill level adequately.
1. Introduction
With the development of science and technology in several decades, we have had more opportunities to operate various types of machines in our daily life. In order to elicit high performance of such machines, however, the users have to strive for mastery of the operation, and much time and effort are frequently needed. With that in mind, a concept of Human Adaptive Mechatronics (HAM) [1–3], which is an intelligent mechatronics to help the mastery of the user’s operation, was presented. Under the project, various kinds of system design theories and technologies for HAM, that changes its dynamic characteristics and the supporting strategies adaptively to the status of individual user in order to enhance the performance of whole human-machine system, have been studied [4–6]. The following two main functions are required to realize HAM: quantification of skill level of users and adaptive human-assisting mechanism according to the skill level. Researches about the quantification of skills, analyses of vehicle control characteristics of drivers or pilots [7, 8], studies concerning cognitive skill for human-computer interface interaction [9], and researches on perceptual skill on video game [10] are known. On the other hand, in order to design an adaptive human-assisting mechanism, a real-time estimation of participants’ skill and its feedback to the human-assisting mechanism are required. Utilization of biological information is one of effective approach for such real-time estimation. Especially, measurement of ocular motion is adequate for analysis of vehicle drivers’ behavior and their skill [11], because the motion reflects intention and thinking of a human in real time [12]. Effectiveness of a usage of the ocular motion is widely well known, and the present authors also have been studying the related themes such as an identification of a human controller on the vehicle driving [13] and a brain monitoring analysis on oculomotor cortex for the voluntary motion skill [14].
Fatigue or stress, however, also influences the ocular motion. Utilizing this property, many studies evaluating such internal status from the measurement of eye motion are reported [15, 16]. Therefore in this paper, the following three steps which were required to establish the skill evaluation method for HAM were treated.
Step 1. Derivation of an equation to estimate a skill level of a machine operation using the measurement of eye motion.
Step 2. Analysis of the fatigue and stress during the operation.
Step 3. Investigation of an influence between the estimating equation and the fatigue and stress.
Concerning the evaluation of fatigue and stress, a method computing on the VDT (video display terminal) operation is known, where is a coefficient of variation of R-R interval which is a time sequence data of heart rate (HR) [17]. Other methods utilizing variation of HR and a chaotic property of the pulse wave are also used [18]. In reference to the previous studies, the fatigue and stress were also treated in this paper. In regard to the measurement of the eye motion, a gaze detection system which was developed in our previous study [19] was used here. As a machine operation task, a cooperative-carrying-task (cc-task) simulator was utilized [20]. These systems were integrated, and eye motion, machine operation, and biosignal of the participants were measured. Then analyses for Step 1 to Step 3 were performed.
The remainder of this paper is organized as follows. Section 2 explains the cc-task simulator and the experimental setup. In Section 3, findings about the eye movement and skill are mentioned, and development of an estimating equation of the skill level is explained (for Step 1). Section 4 explains properties of biosignal used in latter analyses. Next using the measured biosignal, adequacies of methods to evaluate fatigue and stress are verified (for Step 2). Section 5 shows the result of the correlation analysis between the obtained skill estimating equation and fatigue and stress (for Step 3). Last Section 6 is the conclusion.
2. Observation Experiment of Training Process on Cooperative-Carrying Task
2.1. Cooperative-Carrying-Task Simulator
This simulator was designed as a VDT operation of the virtual vehicle, so that the participants would not have to move their body since the large body movement is not preferable for long hours measurement of the eye motion and biosignal. In the virtual cooperation task, three participants work in the same virtual space. Each participant sits in front of each monitor and uses a joy-stick for the manipulation, as shown in Figure 1. The participant manipulates a virtual mobile robot, cooperates with the other participants, and conveys three boxes to target places. Figure 2 shows an overhead view of the virtual space. The simulator was built by a real-time computer graphics using OpenGL and ODE (open dynamics engine) library. Motions of all robots and boxes are computed according to their physical dynamics, and collision among their objects can be also simulated.
Each participant donned a head set of the gaze detection system which was developed by authors in previous studies [19, 21], as shown in Figure 3. Multivideo sources, which consist of the eye image captured by the eye camera (the top-left section in Figure 3(b)), the front camera view (the top-right), and the video source displayed to the monitor for the participant (the bottom-left), were compressed into one video signal using an image partition device. Heart rate of one participant of three was recorded using Polymate II (AP216, TEAC Corporation, Japan).
(a)
(b)
The procedure of this experiment was approved by the University’s Ethics Committee. Participants cooperated after he/she gave informed consent. Participants are 33 Japanese (31 males and 2 females, 20 years to 23 years), and eleven teams consisting of three participants executed the cc-task. Trials were repeated ten times for each team. Total 330 data files including information of eye movement and operation could be recorded. Trial time differed in each team; minimum and maximum of accumulative time of all ten trials were 78 min and 250 min, respectively. Seven persons among those 33 participants were, however, eligible for later correlation analysis between eye movement and stress and fatigue. That was because biosignal data from two members of three ones in one team were not measured from the beginning, as explained perviously. To avoid confusion to readers, data of those seven participants are treated in the following sections on ahead, and the reason of selection of those seven participants will be explained in Section 3.2. In later discussion, those seven participants are called an operators A–G.
2.2. Index of cc-Task Performance
Objective index is necessary to evaluate an ability to accomplish the cc-task in order to develop an estimating equation of the operation skill. Since all operators were asked to finish the task as soon as possible, it was expected that the trial time decreased as the trial increased. It was also expected that frequency of pause in a vehicle operation decreased since unnecessary pausing caused waste of time. Hence, the time required to finish one trial was defined as -index, and the pause time per one minutes was defined as -index. Further, since the experimenter asked operators to avoid collision against walls and other vehicles, the number of such collision was chosen as a third index and was defined as -index. Transition of these indexes are shown in Figures 4, 5, and 6, respectively. Table 1 summarizes correlation coefficients of the linear regression lines for each transition shown in these figures.
Figure 4 shows a monotonic decrease concerning -index for any case. As shown in the second row in Table 1, correlation coefficients between -index and the number of trial are all negative, and their absolute values are large as ; hence, all operators achieved perfection in terms of the task time. Figure 5 for -index indicates almost monotonic decrease, and their correlation coefficients are negative large , although there were individual differences. This means that the hesitation and waiting in the operation decreased as the trial increases, and it can be thought that the efficiency of the task performance increased. Figure 6 showing index-C may indicate the number of collision increases, but not monotonically. It is seemed that the number of trial can be used to express the degree of skill; hence, the number of the trial is treated as simple index and is used as a simple skill level as follows.
With these facts, it can be guessed that almost all operators were getting skilled up in terms of the trial time and frequency of the pause time, but they became tired, and the operational error might increase. Based on this inference, an influence of the fatigue-stress condition to the estimating equation of the machine operation skill is analyzed in later sections.
3. Development of the Estimating Equation of Skill Using Eye Motion Measurement
3.1. Eye Movement and Operational Skill
The well-known studies investigating a relation between an eye movement and a machine operation are for driving a car. Such studies elucidated the following facts.
Fact 1. As the speed of the car increases, the driver tends to concentrate to the direction of movement of the car, and he/she does not notice events that are projected to the peripheral vision on retina [22].
Fact 2. Superfluous saccadic eye movement decreases as the driver becomes an expert [23].
These facts come from the following biological properties of the human eye: a human eye has high resolution only around central visual field, and the saccadic movement is required to obtain various visual information from whole of the visual field. With this, many studies analyzed the eye saccade and fixation to unravel human cognitive process [24, 25]. In such studies, the response time of the eye movement to detect moving things (this time is described as for later discussion), the gaze duration (which is named ), and the frequency/velocity/moving distance of saccade (, , and , resp.) have been utilized as indexes of eye movement characteristics. Considering Fact 1 and related findings, we summarized them into five hypotheses using these indexes in case of the beginner and expert, as shown in Table 2.
In high-speed driving, Fact 2 implies that the expert can notice immediately a moving object since he/she can find it at the peripheral field. That is, it can be expected that the response time of the expert is smaller than that of the beginner. When the response times for expert and for beginner are denoted as and , respectively, the hypothesis is described as , as shown in Table 2. In the following, numerical subscripts attached to variables are used to distinguish each other. Again based on Fact 1, in high-speed driving, both expert and beginner tend to concentrate the line-of-sight into narrow area to the direction of movement of the car; hence, the frequency of the saccade seems low in both cases (hypothesis : ). Similarly, it is guessed that there is not actually difference between the expert and the beginner in terms of the fixation duration, and this guess was named the hypothesis . On the other hand, in low-speed driving mainly in an urban area, the response of the beginner is slower than that of expert, since the beginner does not have sufficient ability to watch something important for safety driving (hypothesis : ). In contrast, the frequency of saccade of the expert seems larger than that of beginner because the expert tries to pay attention to the driving environment as far as possible (hypothesis : ). And then the fixation time becomes short (hypothesis : ).
Based on hypotheses of , , and , the present authors analyzed correlation between the task performance in the cc-task and the aforementioned indexes of eye movements. The high-speed and low-speed ranges were specified by 50 percent of maximum velocity of the virtual vehicle dynamics since absolute speed value concerning Fact 1 was not known generally. Although this threshold was changed and analysis was executed, we could not find significant difference between beginners and experts in terms of , , and . In short, there were large individual difference, and it was concluded that the indexes of , , and were inadequate to evaluate the operational skill level [26].
Considering Fact 2 again, this fact can be interpreted as a property of the expert who can use both central visual field and peripheral vision depending on the situation. In other words, the expert utilizes characteristics of peripheral vision that can detect motion sensitively in spite of its low resolution, and he/she can perceive something moving without changing of the gaze direction. Such perception does not require a motion of the eyeball, and it differs from an ocular exploration accompanied by the saccadic eye movement; hence, it appears that this processing does not depend on the difference of the vehicle speed. Therefore, the followings are inferred in case of the expert: the moving distance of saccade is small regardless of the vehicle velocity (hypothesis ), and velocity of saccade is also small due to the short moving distance (hypothesis ). Hypotheses and are verified through the following sections.
3.2. Extraction of the Eye Movement Feature
Using an image processing algorithm presented in [26], the coordinate value corresponding to the line-of-sight was computed after detecting the position of the pupil image which was recorded by the eye camera. From the time-sequence data of the line-of-sight, the eye movement feature, which will be explained later, was derived. Although the number of operators for the experiment was 33, the number of the valid data set was of 25 operators. That is because perfect data including eye movement for all ten trials was not obtained since the recorded eye images were contaminated by shadow of eyelash or insufficient by positional error of the eye camera setting. Next, computing velocities of eye movement during all ten trials for each of the 25 operators, outlier case was checked by Smirnov-Grubbs’ outlier test using a level of significance of 0.1. Further, data including strong individual difference was eliminated by statistical testing of normality using a Lilliefors test () for each operator’s data. As a result, outlier or disnormality was found in six-operator case. Among the remained operators, seven operators were measured in terms of the biosignal. In short, the number of operators whose eye movement and biosignal could be perfectly recorded was seven. This is a reason of selection of operators A–G which were shown in previous Section 2.1.
The velocity and distance of saccade from the seven operators were investigated to check hypotheses and . Figure 7 shows a result of analysis for the saccade velocity for all operators. The graphs (a) and (b) show the transitions of the saccade velocity at high-speed driving (say ) and the other at low-speed driving , respectively. Concerning the distance of saccade, which are of high-speed driving and of low-speed driving, the results are shown in Figure 8. In both cases, strong correlations (correlation coefficients are for the saccade velocity, and for the saccade distance at the high-speed driving) were confirmed. These tendencies coincide with hypotheses and which were shown in Table 2. Therefore, it was decided that the estimating equation of operation skill was derived using , , and from data of the valid seven operators.
(a) At high-speed driving
(b) At low-speed driving
(a) At high-speed driving
(b) At low-speed driving
For development of the estimating equation based on the eye movement data, it is necessary to find a relation between scalar variable and indexes of eye movement. For this aim, a conversion from three variables (, , and ) into one scalar variable, that is named the eye movement feature, is obtained through the principal component analysis (PCA). Specifically, the factor of eye movement on th trial () is defined as follows: Using , the observation data matrix is defined as After a variance-covariance matrix was computed from , an eigenvector corresponding to th maximum eigen value is derived from . Then th principal component of th trial, , is obtained as Defining as th diagonal element in , a contributing rate of th principal component is given by Figure 9 shows the contributing rate computed by (4) using data concerning operators A–G. Since each first principal component occupied more than 90 percent in case of any operator, only the first principal component was utilized as a value of the eye movement feature finally.
Although only common characteristics among all operators were decided in this state, difference of individual property was not investigated yet; hence, the weighting vectors concerning each operator were checked. Elements , , and in the weighting vector corresponding to the first principal component are summarized in Table 3. Describing the weighting vectors of the first principal component of th operator as , the simple similarity that is defined by was computed to all combination with and . Then data of the operator D was eliminated by an outlier test against all simple similarities computed by (5). The normality of the remained data except the operator D case was confirmed using the Lilliefors test . Therefore, the mean vector was defined as average of the weighting vectors without the operator D case. Values of are described at the bottom row in Table 3. Finally the value of eye movement feature of th trial was computed as
3.3. Estimating Equation of Skill
After computing for six operators using (6) from raw data of factors of eye movement (, , and ), correlation analysis found the strong correlation between and the simple skill level which is the number of trial . This correlation analysis, however, does not care the bias which differs depending on each operator’s . Additionally, it is natural to think that the value of the index increases as an operator gets skilled up. With this, an estimating equation of skill was determined as follows by considering bias which was computed from the first and second trial data: Figure 10 shows the estimation of skill level computed using (6) and (7) from raw data of factors of eye movement (, , and ) in case of operators A–C and F-G. It was confirmed that the estimated skill value increases roughly as the number of trials increases. The correlation factors between those estimated values and the simple skill levels are as high as , and statistically high correlation was confirmed. Therefore, (7) is used in the following as the estimating equation of skill level of the cc-task operation.
4. Analysis of Fatigue and Stress
In this section, how to compute index values of fatigue and stress is mentioned. Medically, HR (heart rate) is affected by sympathetic function; the larger value means higher mental or physical load, and the smaller values means the relaxing status. HR is small when a person is bored status in menial jobs. HR is also small when the alertness level is low. It is, however, said that HR becomes large when a person chafes even if he/she is in menial jobs [27].
As another index using heartbeat, coefficient of variation of R-R interval (RRI), that is denoted as , is known to be affected by parasympathetic function. RRI is the time elapsing between two consecutive R waves in the electrocardiogram. It is said that fatigue is low (high) when shows large (small) value [28].
One of other methods to evaluate the stress level is utilization of chaotic property of the pulse wave. Specifically Lyapunov exponent of chaotic trajectory, which is computed from acceleration component of pulse wave, is used as an index of the stress and relax. The larger value means higher stress (or concentration) status [18]. In the following subsection, effectiveness of these three indexes to the cc-task operation is verified for latter analyses.
4.1. How to Compute HR, , and Lyapunov Exponent
4.1.1. HR
HR used in the present analysis was computed as follows by using mean of R-R interval per trial. Denoting this mean value by , HR is computed as Transitions of HR of operators A–G and the corresponding linear regression lines are shown in Figure 11. It is found that all operators except G show nearly monotonic decrease.
4.1.2.
Using standard deviation of RR-interval per trial (say ), is computed by Transitions of and the corresponding linear regression lines are shown in Figure 12. The figure shows that variance of is larger than that of HR and there are individual differences of an increase or decrease tendency among operators.
4.1.3. Chaotic Analysis of Mental Stress
Based on the Takens method [29], chaotic trajectory was computed from electrocardiographic waveform. Using Lyapunov exponents computed from the trajectory, level of stress of operators was investigated. The details are as follows.
In the phase of computation of the Lyapunov exponent, the Poincaré section was put in the hyperspace of the chaotic trajectory so as to intersect to the movement on the trajectory with largest speed. Next, after the intersection points of the trajectory against the Poincaré section are found, the positional vector toward the points, , is computed, where () is the number of passages through the Poincaré section. Then Lyapunov exponent is computed as where is the time interval between th passage to one. For analysis of the cc-task, which is a mean of was adopted as a Lyapunov exponent to evaluate level of stress and relaxation. Transitions of of all operators and the linear regression lines are shown in Figure 13. Variance of was larger than that of that was shown in Figure 12. From this result alone, it is difficult to find distinguishing feature at a glance.
4.2. Outlier Test and Correlation Analysis
When seeing distribution in Figures 11, 12, and 13, it is seemed that several outliers exist in those data. Therefore, outliers were eliminated by Smirnov-Grubbs’ outlier test () against Mahalanobis distances of the distribution of data points of HR, , and for each operator. After the elimination, correlation analysis was performed using all combinations between and . The details of the correlation analysis are mentioned below.
4.2.1. Stress Analysis by HR
Computing correlation factor between the number of trials and HR (Trial/HR), those factors and slopes of the regression lines are summarized in Table 4. The table shows that factors of six operators except operator G have same sign, and their absolute values are large. From this result, the data of operator G was eliminated for later analysis due to its exceptional difference of the sign. Then, since HR of all the remained operators decreased as the trial increased, we can interpret this to mean that they were getting relaxed. However, HR of several operators still continues decreasing at ten trial; hence, they might be bored status in menial jobs at that time.
4.2.2. Relation between Collision and Fatigue-Stress
Next we ascertained whether operational error such as collisions was increased due to fatigue or stress. Specifically, correlations of the following combination were investigated: -index and HR, -index and , and -index and . From these correlations analyses for operators A–F, the obtained correlation factors and slopes of the regression lines are summarized in Table 5. Sign of value of the operator F in terms of the differed from others; hence, the operator F was eliminated due to its strong individual difference. Seeing correlation factors of concerning the remained operators A–E, their signs vary widely, and the absolute values are as small as about 0.2. Relation between -index and has similar tendency. From these results, we cannot conclude that stress (from relation with HR) or fatigue (from relation with ) raised collision. This result suggests that -index is inadequate to evaluate the operational skill in this cc-task, so we decided not to use -index in later analyses.
4.2.3. Stress Analysis Using Chaotic Property
Using Lyapunov exponents of all operators except F and G (who were eliminated as exceptional cases), correlation analyses against -index and -index were performed, respectively. The results are shown in Table 6. This table shows that sign of operator D differs from others; hence, this case can be thought as additional exception. Since all correlation coefficients of of the remained operators are plus, chaotic properties are small when the trial time is small; hence, it can be said that mental stress was also small. This is a similar tendency to the result which was previously shown in Table 4. Values of correlation coefficients of shown in Table 6 were, however, small (, , ), although other values of correlation coefficients of -index/HR shown in Table 4 were large (, , ). Therefore, it can be said that HR is more adequate than Lyapunov exponent in order to evaluate mental stress in the cc-task operation.
With regard to , the tendency of operators except D is same, as shown in Table 6. Negative signs in those correlation of coefficients indicate that value in Lyapunov exponent was large when frequency of pause of vehicle manipulation was small. In other words, mental stress (or concentration) of the operator was high when the vehicle was controlled constantly. This interpretation can be accepted naturally; however, the absolute values of those coefficients are not so large ; hence, the stress analysis using chaotic property was not reliable.
5. Relation between the Estimating Equation of Skill and Stress-Fatigue
As demonstrated in Section 4, it was found that HR and were effective for evaluation of stress and fatigue, respectively. In this section, effects of these indexes to the estimating equation of skill , which was derived in Section 3, are mentioned. From factors of eye movement, the eye movement feature was computed by (6), and the estimated level of skill was obtained using (7). Correlation coefficients between and the internal status in case of operator A, B, C, and E are shown in Table 7.
With respect to HR, each HR in four operators decreases as each estimated skill level increases since signs of the correlation coefficients are all minus. That means that they were in a tense situation at first, but they got relaxed as they became skilled. In other words, the estimated skill level derived using measurement of eye movement is affected by both operational skill and condition of stress. On the other hand, correlation coefficients concerning are all minus, and decreases as they became skilled. That is, fatigue increases as they became skilled.
With this, it was confirmed that the presented equation to estimate the operational skill based on the eye-movement measurement reflects effect of stress and fatigue, although the equation can estimate the skill level adequately.
6. Conclusion
To observe learning process of the machine operation, a cooperative carrying task (cc-task) simulator system was designed, and the eye movement, electrocardiographic waveform, and log data of the machine operators were recorded using this simulator. From the recorded data, a relation between the operational skill, mental stress, and fatigue was analyzed. First, outliers data caused by nonstandard behavior and failure in the measurement were eliminated by several types of outlier tests. After extracting valid data which satisfy statistical normality, the standard estimating equation of the operational skill level based on the eye-movement data was derived. Specifically, factors of the eye movement having strong correlation with operational skill were selected, and those factors were converted into the eye movement feature value through a principal component analysis. Then the estimating equation was derived. At the decision phase for the factors of eye movement, it was confirmed that velocity and distance of saccade were adequate for them. And effectiveness of the derived equation could be confirmed since the correlation coefficients between the estimated skill value computed by the equation and actual simple skill level were sufficiently high as .
Second, biological signal such as the heart rate (HR) and the coefficient of variation of R-R interval () of operators in the cc-task work were analyzed. As a result of correlation analysis against simple skill levels, it was found that HR and were effective for evaluation of stress and fatigue, respectively. Effectiveness of other stress analysis using chaotic properties in electrocardiographic waveform was, however, not shown.
Finally, from the correlation analysis between the biological information (HR and ) and the estimating equation of the operational skill, it was confirmed that the equation reflects effect of stress and fatigue, although the equation can estimate the skill level adequately.
Acknowledgments
The present study was supported by a Grant-in-Aid for Scientific Research (A) of the Japanese Ministry of Education, Culture, Sports, Science, and Technology. Software development of the cooperative-carrying task was supported by Hiroshi Igarashi. The experiment was helped by many participants who embraced our requests kindly. The present authors appreciate their cooperation.