Abstract
The driver is one of the most important factors in road traffic. Monitoring the driver’s driving status can greatly improve the safety and road operation efficiency of urban road traffic in the case of multiple traffic modes. Fatigue has a significant impact on drivers’ safety on the road, particularly while driving in a monotonous environment for a long time. In this study, the eye movement parameters of 36 drivers were collected through the simulation experiment of a driving simulator. The pupil area and percentage of eye closure (PERCLOS) in driving scenes of the expressway and low-grade rural road were combined with the Stanford Sleepiness Scale (SSS) to determine the threshold of fatigue degree in different monotonous driving scenarios. A recognition model of different fatigue degrees of drivers is built based on the deep learning method of a long short-term memory network (LSTM) to detect the varied fatigue degrees of drivers. The result shows that the fatigue degree of drivers increases as driving time increases on both expressways and low-grade rural roads. In the same driving time, the driver felt tired faster on the expressway, and the fatigue degree was significantly higher than that on the country road. The recognition rate of the established fatigue degree recognition model for driver’s awake state, mild fatigue, moderate fatigue, and severe fatigue is 100%, 93.1%, 98.4%, and 100% respectively, and the total recognition rate can reach 97.8%, which is higher than the recognition accuracy of the traditional machine learning approach.
1. Introduction
The trend of traffic intelligence is becoming more and more visible, thanks to the rapid development of the vehicle industry and the ongoing construction and optimization of roads and road administration facilities. Despite the emergence in an endless stream of emerging technologies nowadays, their application to traditional traffic problems still needs to be addressed. The driver’s behavioral state is a key aspect of the driving process. Monitoring the driver’s status during the driving process will serve to improve the safety and efficiency of urban road traffic, as well as contribute to more coordinated regional traffic development in the case of numerous modes of transportation. Individuals, families, and the country lose a lot of money due to traffic accidents, and drowsy driving is one of the leading causes of traffic accidents and fatalities. According to statistics, roughly 600,000 people died in traffic accidents each year around the world, resulting in a direct economic loss of $12.5 billion. 57% of these accidents were related to driving fatigue. The probability of traffic accidents caused by driving fatigue accounts for more than half of the total number of accidents, which is 4 to 6 times that of ordinary driving [1]. According to a report by the National Highway Traffic Safety Administration (NHTSA), driver’s drowsiness accounted for approximately 83,000 crashes, 37,000 injuries, and 900 deaths in the United States alone [2]. At the same time, according to the American Automobile Association survey, it was found that 21% of traffic fatalities were caused by driver fatigue [3].
Drivers often show themselves in a variety of ways when they are tired, including drooping eyes, increased blinking frequency, lower focus, and so on. It is frequently accompanied by sleepiness, exhaustion, and other symptoms. As a result, when drivers are fatigued, their driving conduct poses a significant risk to traffic safety. The monotonous driving environment, long-time driving, and the driver’s physical condition are the main causes of fatigue driving.
The driver’s fatigue state is a process that gradually accumulates with the increase of driving time. Most studies now focus on the driver’s awake and fatigue states, with only a few studies categorizing the driver’s exhaustion state. At the same time, the fatigue condition of the driver varies depending on the driving environment. The goal of driving fatigue research is to identify drivers’ fatigue levels in different driving situations, define the typical index of a driver’s fatigue level, and provide a reference for developing solutions for different fatigue states.
The recognition model of the driver’s fatigue condition is constructed using the deep learning method of a long short-term memory network. The data set for training and verifying the model is the eye movement data of drivers obtained in driving simulation tests. The aim of this study is to verify the effectiveness and superiority of the deep learning method in recognizing the driver’s fatigue state.
In this study, we use a combination of subjective fatigue detection and objective fatigue detection methods through driving simulation experiments conducted on a driving simulator, which will explore the differences in driver fatigue indicators in different monotonous driving environments. The effective classification of fatigue levels is important for fatigue warning in monotonous driving environments. Moreover, the deep learning algorithm LSTM is used to establish a driver’s fatigue recognition model, which can identify the driver’s fatigue state, warn the driver of dangerous driving behavior, and improve the driver’s driving safety while driving.
2. Literature Review
The effect of the driving environment on driving fatigue has been explored in some researches. Pilcher and Huffcutt pointed out that a complex road environment and traffic conditions will make drivers feel tired more easily [4]. Mao believed that the more monotonous the road environment, the more likely it was to cause driver fatigue [5]. Thiffault and Bergeron’s research showed that the driver moved the steering wheel at a greater angle many times in the monotonous driving environment, which indicated that the driver was more cautious in the monotonous driving environment [6]. Dinges found that drivers in monotonous driving environments were less alert, especially their visual response to driving was slower [7]. However, these studies have only considered the effects of monotonous and complex driving environments on driver fatigue in isolation. In real-world driving, different monotonous driving environments have different effects on driver fatigue.
A recommended maximum continuous driving time has been proposed as a way to require drivers to take breaks during the journey. Fatigued driving is a gradual behavior, which occurs when drivers are unconscious. The individual differences of fatigue characteristics among different drivers are large, the fatigue characteristic values of drivers with the same fatigue degree are also different, and the changing trend of drivers with the same fatigue characteristic is also different [8]. Thus, it is necessary to reduce the influences caused by individual characteristics as much as possible. The previous research on fatigue driving mainly focused on how to find and identify fatigue. According to previous studies, the main methods to identify driving fatigue are as follows: (1) defining a predetermined driving duration threshold to identify fatigue driving. (2) fatigue driving is detected and recognized through many aspects of the driver or vehicle, such as physiological response, cognitive distraction, facial expression, vehicle condition, and so on. However, driver fatigue is a process that gradually accumulates over time and the fatigue characteristics of different fatigue states have different changing trends [9]. Driving fatigue should be classified to study because fatigue has a characteristic of gradual behavior. Ahlstrom et al. divided the fatigue state of drivers into three levels: awake, fatigue, and severe fatigue based on the Karolinska Sleepiness Scale (KSS) [10]. Larue et al. used the 5 minutes before driving as a reference standard and used the driver’s alertness and the number of microsleeps as evaluation indicators to divide fatigue into 4 levels [11]. Li et al. analyzed the steering wheel angle changes in three states: awake, fatigue, and extreme fatigue. They found that drivers frequently corrected the steering wheel in small increments when they were awake. When they were fatigued, they corrected the steering wheel less frequently and with larger and faster movements. They even showed no movement for a short time and then suddenly corrected the steering wheel significantly when they were extremely fatigued [12]. Zhang [13] selected the EEG Shannon entropy and sample entropy, EMG approximate entropy, and ocular wavelet time-frequency analysis indicators based on neurobiology. Then, he defined fatigue into normal, mild fatigue, mood swings, and extreme fatigue, respectively. The driver’s driving state is divided into awake and fatigue states. This two-level division of the fatigue state may be slight fatigue, but also may be serious fatigue. The subsequent fatigue warning will either be too early or too late. Both have a great impact on driving safety. Therefore, the key to driving fatigue research is to find a way to effectively segment the driver’s fatigue level and achieve effective fatigue warning for the follow up.
At present, the detection methods of driving fatigue mainly include driver’s physiological signals (such as driver’s EEG and ECG), driver’s physiological response characteristics (such as human eye movement and blink information), driver’s operation behavior (such as steering wheel rotation angle), and vehicle state information (such as using the change of vehicle trajectory and lane departure) [14]. The results obtained based on physiological signal detection are the most accurate and they can best reflect the driver’s driving fatigue characteristics. A detection method based on the EEG signal was proposed by Wang et al. to judge driving fatigue in real-time by analyzing the driver’s nervous system [15]. Due to the high detection cost and invasive detection, it has a great impact on the driver’s normal driving behavior, which is not commonly used and requires improvement. At present, the detection method based on the driver’s eye state is the most effective and convenient method to detect the fatigue of drivers. Because it is a noninvasive detection, it is more suitable for practical applications. The PERCLOS method is the most commonly used driving fatigue detection method based on the driver’s eye state. The PERCLOS value adopts the ratio of eyelid closure time to a period is used as the fatigue detection index. The greater its value, the deeper the degree of driving fatigue. The pupil diameter of people tends to decrease regularly with the deepening of fatigue [16], so the pupil area can also be used as an index parameter for fatigue judgment.
There are a lot of researches on the detection methods of driving fatigue. To improve the accuracy of fatigue detection, Xu et al. collected multisource data such as vehicle lateral position and steering wheel manipulation through driving simulation tests to calculate fatigue characteristic indexes. A decision tree model for fatigue level prediction was established by combining the driver’s subjective fatigue level and conducting a comprehensive evaluation of the fatigue level through video playback. The model had a correct prediction rate of 64.3%, a relatively low accuracy rate, and low precision [17]. Qu et al. used the evaluation method of a facial video expert and established a database of driver’s awake, fatigue, and very fatigue states, respectively, through driving simulator driving simulation experiments. Then, he selected the optimal combination of characteristic indicators by a specific algorithm after extracting the characteristic indicators of driver’s fatigue operation characteristics to establish a 3-level fatigue monitoring model for drivers based on SVM [18]. The accuracy obtained by this model can reach 87.7% when tested under simulator working conditions. Recently, driver fatigue evaluation using advanced deep learning techniques based on physiological responses has been reported by Gao et al. [19, 20]. Deep learning can greatly improve the accuracy of driver fatigue detection through its powerful information processing ability and strong robustness, which can greatly improve the ability of fatigue detection.
3. Experimental Method
The study used a driving simulator in investigating the thresholds of the fatigue level in different monotonous driving environments. Then, the fatigue level recognition model was built to identify different fatigue levels. The theoretical framework is presented in Figure 1.
3.1. Driving Simulator and Driving Scenes
The test used the high simulation driving simulator as shown in Figure 2. The motion system of the driving simulator is 3 degrees of freedom. A Volkswagen Polo car with no engine is used in the driving simulator, which includes a steering wheel, braking force feedback, electrical sensor, and sound system. The driving simulator’s functions are identical to those of a real car to ensure it is the same as a real driving situation. The driving visual scene is mainly provided by a projection system composed of four projectors, with a visual range of 250°. The effectiveness of the driving simulator passed the system test, demonstrating that the simulator’s simulation degree can fulfill research demands.
(a)
(b)
(c)
To explore the impact of different monotonous environments on drivers’ long-term fatigue driving, two types of roads are mainly set in the driving scenes. Table 1 and Figure 3 describe these two driving scenarios in detail. To restore the real driving scene as much as possible, green grass, trees, and a small number of village buildings are set on both sides of the road, and a small number of vehicles that do not affect normal driving are set on the road.
(a)
(b)
3.2. Subjects
According to previous studies, the number of subjects in the experiments was in the range of 10–38 [21–25]. The number of subjects in this study is 36, which could meet the minimum sample size requirement. Before the experiment, the gender, age, nap habits, and driving age of the 36 subjects were recorded through a questionnaire survey. The driving factor variables are summarized in Table 2. The survey results show that the subjects are mainly 22–28 years old and have been driving for 1–6 years. The subjects are mainly young and middle-aged experienced drivers, which can minimize the impact of driving experience on driving behavior. All subjects are required to hold a valid driver’s license; be in good physical condition; have no drug-taking history within 1 month before the test; not drink alcohol within 24 hours before the test; and not drink coffee, strong tea, and functional drinks within 12 hours before the test [21].
3.3. Test Process
These 36 subjects were randomly assigned to two scenarios to simulate driving situations. Scenario#1 and scenario#2 were assigned 18 personnel equally, and the process of assigning personnel ensured that the driving experience and other factors of the subjects in both scenarios were kept as equal as possible. The drivers of both driving scenarios went through the same test process, filling out the basic driver information questionnaire before entering the driving simulator. The experiment process is shown in Figure 4.
Because the drivers often feel tired in the afternoon and early morning. For experimental ease of implementation, we choose a time between 12:30 and 2:30 in the afternoon to do the test. The drivers have just finished lunch and need a midday break, which is more likely to cause sleepiness at this time. During the test, the driver is not allowed to undertake any secondary tasks, and there is no need to change lanes or switch the lights while driving. The test car uses an automatic transmission, so the driver does not need to change gears. The driver does not use mobile phones, radios, music players, or other equipment during the test. There are a small number of other traffic vehicles on the road, but they do not block the lane where the driving vehicle is located. Moreover, the driver needs to wear an eye tracker to record the eye movement data during driving.
The driving simulation experiment steps are as follows:(1)The driver familiarizes with the driving simulation system, understands relevant precautions and the purpose of the experiment, and fills in relevant information (age, driving age, nap habits, etc.).(2)The driver conducts a simulated driving test drive for 10 minutes, to enable the subject to reach a high level of fatigue within a limited time, and then conduct the formal experiment.(3)The driver wears the eye tracking device to confirm the normal transmission and storage of eye-tracking data during driving.(4)The driver’s speed is kept at about 110 km/h in scenario#1 and 60 km/h in scenario#2, and the continuous 1 h simulated driving experiment was conducted.(5)After the start of the experiment, a subjective fatigue evaluation of the driver was conducted at 0 min, 30 min, and 60 min.(6)The experiment is carried out in sequence, and the data are saved after completion of all drivers.
4. Experimental Result
4.1. Subjective Fatigue Survey
A subjective fatigue survey is one of the important means to study driving fatigue. The most widely used fatigue measurement scale is the Stanford Sleepiness Scale [26]. The SSS scale contains 1 ∼ 7 different fatigue grades (the fatigue grade is expressed by S), and the fatigue degree is deepened in turn. As shown in Table 3, S = 1 denotes complete vitality and vigor, while S = 7 indicates very tired and at the beginning of sleep. Subjects need to choose one of the seven fatigue levels to represent their current fatigue state. The advantage of the subjective evaluation table is that it is easy to operate and can be repeated. To facilitate the subsequent processing, the driver’s fatigue state was divided into four levels: awake (S = 1, 2), mild fatigue (S = 3, 4), moderate fatigue (S = 5), and severe fatigue (S = 6, 7).
According to the drivers’ subjective fatigue state, values were obtained before the experiment, 30 minutes in the experiment, and at the end of the experiment. The expressway and low-grade rural highway can be counted separately, the results are shown in Figure 5.
(a)
(b)
As shown in figure (a), all subjects remained awake at the beginning of the experiment in scenario#1. As the experiment progressed, the driver’s fatigue and sleepiness increased. When the experiment lasted for 30 min, 89% of the subjects subjectively felt mild fatigue and 11% felt moderate fatigue. When the experiment lasted for 60 min, 83% of drivers feel that they are in a state of severe fatigue, while only 17% of drivers feel moderate fatigue.
As shown in figure (b), all subjects also remained awake at the beginning of the experiment in scenario#2. When the experiment lasted for 30 min, 17 drivers felt only mild fatigue, one driver felt moderate fatigue. When the experiment lasted for 60 min, half of the subjects felt moderate fatigue and the other half of the subjects felt severe fatigue.
When the fatigue states of drivers in the two scenarios are analyzed together, as shown in Figure 6,it can be seen from the figure that in the driving process of the expressway and the rural highway, the subjective fatigue state of drivers gradually deepens with the increase of driving time. At 30 minutes of the expressway scene experiment, the drivers’ fatigue degree is slightly higher than that of the rural highway. At 60 minutes, the drivers’ fatigue degree is much higher than that of the rural highway. As can be seen from the trend line of subjective fatigue degree in the figure, from the trend line of subjective fatigue in the graph, it can be seen that the trend line rises faster and has a greater linear slope for the highway than for the rural road. This indicates that drivers who drive on the highway for a long time are more likely to feel fatigued subjectively and the rate of fatigue deepening is faster. Subjective fatigue is higher for highway drivers for the same driving time.
4.2. Eye Movement Parameter
4.2.1. Pupil Area
As a noninvasive detection method, the detection of eye movement parameters can effectively and conveniently detect the driver’s fatigue state. Generally, the pupil diameter of normal and fully rested people is about 2.5–4 mm. The pupil diameter can be an effective index to measure the driver’s fatigue. With the increase of driving time, the driver’s pupil diameter shows a regular shrinking trend [16]. Then, the driver’s pupil data collected by the eye tracker can be collected at a frequency of 60 Hz, and the average pupil area of the driver can be calculated by D-Lab software. Take all drivers every 10 minutes as a driving time section in both scenarios, calculate the average value of the stable pupil area value, and get a total of 217 sample values in 60 minutes. According to the time series, all sample points form a pupil area data scatter diagram, which is represented in Figure 7.
It can be seen from the figure that the pupil area of the driver in both scenes shows a regular decreasing trend with the increase of driving time. The sample points are sorted according to the time series and 35 sample points are taken every 10 minutes, it can be seen that the reduction trend of the driver’s pupil area in the two scenes is different. When the experiment is conducted for 25 minutes in scenario#1, there is a point with an apparent decreasing trend, indicating that there is an obvious change in the driver’s fatigue at the moment. When the experiment lasted for 40 minutes, there is a very significant downward trend, and the driver’s fatigue is much deeper at this moment than it had been previously. In scenario#2, the curve as a whole shows a regular decreasing trend. At the 98th sample point and the 136th sample point, the decreasing trend is faster than the previous ones. This serves as a reference for determining the pupil area threshold for different levels of fatigue later.
4.2.2. PERCLOS
PERCLOS refers to the time proportion of eye closure time, which has a high correlation with fatigue. According to the definition of PERCLOS, the calculation method of the PERCLOS value is shown in formula (1).where is the time when the pupil is covered longitudinally by the eyelid and is the total detection time.
In the actual application process, PERCLOS can be calculated by calculating the proportion of closed frames to the total frames [27], as shown in the following formula:where is the number of frames with eyes closed and is the number of video frames in a certain time.
The eye movement equipment is used to collect data in this experiment, and the data acquisition frequency is 60 Hz. Because the difference between the EM, P70 and P80 standards of the PERCLOS fatigue driving detection method is not significant [28], the critical judgment threshold for eye closure is set to 70%. In the detection of the closed state of human eyes, both human eyes are generally open and closed at the same time. To reduce the error, it is considered closed when one of the human eyes is closed. In both scenarios, the data were processed in a time interval of 15 min to obtain the average stable PERCLOS value for each driver during this period. All the obtained PERCLOS values were plotted according to the time series to obtain a box plot, and the results are shown in Figure 8.
(a)
(b)
(c)
(d)
As can be seen from the four-time period graphs in Figure 8, the PERCLOS values of the drivers in both scenarios increase with time. The abnormal values that appear at 15 min, 30 min, and 45 min are caused by instrumentation errors and measurement errors, so they are not argued afterward. Instead, the data obtained from the statistics of the four-time periods show that the fatigue levels of drivers in different scenarios during the same time are different. The PERCLOS values for drivers in the rural road scenario are all greater than the values for drivers in the expressway scenario. This indicates that the fatigue threshold ranges of drivers in different driving environments are different and need to be analyzed separately.
The stable PERCLOS mean value sample points of drivers were arranged according to time series, and after comparing the data, characteristics of each time in Figure 8, the PERCLOS mean value graph of drivers was obtained according to 15-minute periods, as shown in Figure 9. It is clear from the graph that the PERCLOS values of drivers show a gradual increase with increasing driving time. Whether in the expressway or rural road, the driver’s driving fatigue gradually deepens with the increase of driving time. The PERCLOS value of the driver on the expressway rises more slowly and is smaller than on the country road, indicating that the driver on the expressway feels fatigued more rapidly and at a higher level than the driver on the country road during the same driving duration.
5. Fatigue Grade Judgment Index
Although there will be individual differences in the fatigue index parameters of each individual, there is a distribution pattern within a certain range. It can be assumed that the individual fatigue index values show a normal distribution within the respective grade. Referring to the subjective fatigue state values of drivers, the pupil area and PERCLOS values after experimental analysis and processing are also used as the basis, and then the parameter changes in Figures 7, 8, and 9 are taken into account. According to the analysis results, the threshold of fatigue degree can be divided by two fatigue index parameters: pupil area and the PERCLOS value. The standard division of the fatigue grade index is shown in Table 4.
6. Driver Fatigue Identification Based on the LSTM Model
6.1. Introduction to the Long Short-Term Memory Network (LSTM)
The long and short-term memory network (LSTM) is a special type of RNN. It can learn long dependencies and avoid the gradient explosion and gradient disappearance of RNN. It was first proposed by Hochreiter and Schmidhuber [29] and has been improved and popularized by many people. Because it works very well on a variety of problems, it is now widely used. The main feature of LSTM is that its storage unit is essentially an accumulator [30] and represents the current state of memory cells, and gates can be used to protect and control the state of cells. The state of cells can generally be realized by multiple gates. At each input, when the input gate opens, the state of the cell will be remembered. When the forget gate opens, the last cell state will be forgotten. When the output gate is opened, the cell state will be transmitted to the final state . The advantage of using storage cells and gates to control information transmission is that it can prevent the gradient from disappearing rapidly. The unit structure of LSTM is shown in Figure 10.
Graves proposed the propagation implementation method of the LSTM network [31], and its equation is as follows.
Forward Pass:
Input Gates
Forget Gates
Cells
Output Gates
Cell Outputswhere represents the weight of the connection of different units, represents the network input to unit at time , and represents activation of unit at time . The subscripts , , and represent the input gate, forget gate, and output gate of the block, respectively. The peephole weights from cell to the input gate, forget gate, and output gates are denoted as , , and . is the state of cell at time. and represent the number of inputs and outputs, respectively, and denotes the number of cells in the hidden layer. , , and are all the activation functions.
Backward Pass:
The purpose of the backward pass is to calculate the gradient and thus update the parameters.
Cell Outputs
Output Gates
States
Cells
Forget Gates
Input Gateswhere is the total number of inputs to the hidden layer. represents the loss function used for training.
In the structure of LSTM, its input, output units, and cell state are one-dimensional, and multiple LSTM network structures can be connected to form a more complex structure. The formulas of LSTM in this paper are the same as that proposed by Graves[32], and its key equations are as follows:where represents sigmoid function, , , , and represent input gate, forget gate, output gate, and memory cell, respectively, represents the Hadamard product.
6.2. Model Building
The occurrence of driver fatigue is closely related to the time axis, and it is gradually deepened with the increase of time. Driver fatigue presents regular characteristics on time series, and its data characteristics belong to time series data. The LSTM model is time-dependent and has excellent performance on time series, which is very suitable for processing and predicting events in time series. Because driving fatigue has the feature of long-time driving, and the LSTM model is very good at handling long series data. Therefore, the identification of the driver’s fatigue state in this paper can be converted into a multiclassification problem for the awake, mildly fatigued, moderately fatigued, and severely fatigued driving states. The data set is based on the pupil area and PERCLOS values obtained from the experiments with drivers in the expressway scenario. Moreover, the labeled data are made according to the thresholds of different fatigue levels defined in the above. The labels for sober, mild fatigue, moderate fatigue, and severe fatigue correspond to data labels of 0, 1, 2, and 3, respectively. There are 8140 pieces of data in the data set and 2035 pieces of data are contained under each label, of which 70% are used for training and the remaining 30% are used for testing. The training set is used to establish the driver’s fatigue grade identification model and parameter optimization in the model, and the test set is used to test the generalization ability of the model.
Before the sample data enters the network model, it is necessary to reshape each sample data into a three-dimensional vector. Each sample vector contains 2 features: the fatigue feature indicators X1 (pupil area value) and X2 (PERCLOS value). Then, Min-Max normalization is used for the data to make the model fit faster and achieve the training effect better. In this paper, a one-way LSTM structure is used to construct the network model. There are two LSTM layers. After the second LSTM output layer, two fully connected layers are connected to output the classification and recognition probability. The network model structure is shown in Figure 11.
6.3. Model Training and Results
The LSTM fatigue grade recognition model is implemented in the Python language on the pytorch1.7.1 platform. The min-batch training method is used and the sample batch is 64. The cross-entropy loss function is selected as the cost function and the Adam optimizer is selected to train the network, and the learning rate lr is 0.0001. Dropout is used for LSTM layer units to prevent overfitting in training, with a value of 0.8. The dimensions of the input and output of each layer of the network are shown in Table 5.
The model operates on a hardware environment with an RTX 2060 graphics card and AMD 4800 h CPU and a window10 64 bit system with 16 GB of memory and uses a GPU to accelerate the training. When the number of network iterations is 55, the network training results perform well in the training set. When the number of iterations is increased, the network model is overfitted. The loss change rate and accuracy of the finally trained model are shown in Figures 12 and 13.
It can be seen from Figure 12 that the loss rate of the network gradually decreases and tends to be stable during the training process. Similarly, it can be seen from Figure 13 that the accuracy rate of the network model gradually increases and tends to be stable during the iterative training process.
The data set has been divided into a training set and test set before, and 30% of the data set is used for testing. Therefore, the sample of the test set is 2444. The test set is tested on the trained LSTM network to evaluate the performance of the model’s ability to identify fatigue levels according to its performance on the test set. Finally, the trained long short-term memory network model is tested on the test set of 2444 data, and the confusion matrix of recognition results is shown in Table 6 below.
It can be seen from the confusion matrix in the above table that the accuracy of awake state recognition is 100%, that of mild fatigue state is 93.1%, that of moderate fatigue state is 98.4%, that of severe fatigue state is 100%, and the total accuracy is 97.8%.
Lei classified drivers’ fatigue into four states: awake, mild fatigue, moderate fatigue, and severe fatigue [33]. He used the driving simulator and a BIOPAC multichannel physiological recorder to obtain the driver’s EEG signals. After extracting the corresponding features, a driver fatigue level recognition model was trained by using the support vector machine (SVM) method. Xu et al. divided fatigue into awake, moderate fatigue, and severe fatigue, respectively. After using a driving simulator and extracting relevant indicators, he used an ordered Logit (OL) model and an artificial neural network (ANN) model to evaluate the fatigue state[34]. The accuracy of the LSTM model used in this paper is compared with the accuracy of these models, and the results are shown in Table 7.
It can be seen from Table 7 that the deep learning approach works better than the traditional machine learning approach in the driver’s fatigue status recognition based on the driver’s physiological response signals. The long short-term memory network model showed a substantial improvement over the OL model, ANN model, and decision tree model in the recognition rate of each level of fatigue and the total recognition rate. Compared with the SVM model, the LSTM network model improved the recognition rate of moderate fatigue by 13% and the overall recognition rate by 6.7%. Therefore, the LSTM model is very effective in recognizing the fatigue level of drivers.
7. Conclusion
The era of intelligent transportation is coming, and the future must be the era of autonomous driving. However, there is still a long time for autonomous driving in the true sense. People are the most important link in a traffic system composed of people, vehicles, and roads. By monitoring the driver’s driving status, we can effectively achieve regional traffic safety and operational efficiency in multiple traffic modes, improve traffic economy, and enable the coordinated and benign development of regional traffic.
Based on the driving simulator experiment, the classification threshold of driver’s fatigue degree in two monotonous environments of the expressway and low-grade rural highway is defined from the eye movement parameters of 36 drivers and refers to the driver’s subjective fatigue state. The results show that the thresholds of different fatigue levels of them are different, and the drivers on the freeway will feel fatigued faster. Driver’s fatigue level in the expressway is deeper than that in the rural highway during the same driving time. Because fatigue is a gradual process, it needs to be analyzed and processed in stages and the driver’s fatigue state will be different in different monotonous driving environments. The deep learning method is also used to test the data samples obtained from the experiment. The results show that the total accuracy of identifying different fatigue states can reach 97.8%, which is significantly improved compared with the method based on the traditional machine learning model in the traditional literature.
There are only 38 subjects in this study, which did not fully cover the physiological signal characteristics of drivers, so more experimental samples are needed in the future. Moreover, only two common monotonous driving scenarios were considered for the study, while the real monotonous driving environment scenarios are much more than that. Although advanced driving simulators can provide driving scenarios that are highly similar to the real driving situation, making the driver’s perception and reactions highly similar to the real driving situation, some factors such as projection resolution and lighting conditions in the experiment can also affect the driver, so the real driving situation is still needed as a reference in the future study. Although this paper distinguishes different fatigue thresholds and uses the deep learning model to identify the fatigue degree, the identification after obtaining the data cannot identify the fatigue state synchronously while driving [35–44].
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The author declares that there are no conflicts of interest regarding the publication of this paper.
Authors’ Contributions
Hao Han conceived and designed the study. Kejie Li completed the entire experiment, performed data collection, and drafted manuscript preparation. Yi Li carried out mathematical modeling. All the authors reviewed the results and approved the final version of the manuscript.