Abstract
All persons in self-driving vehicle would like to receive each service. To do it, the system has to know the person’s state from emotion or stress, and to know the person’s state, it has to catch by analyzing the person’s bio-information. In this paper, we propose a system for inferring emotion using EEG, pulse, blood pressure (systolic and diastolic blood pressure) of user, and recommending color and music according to emotional state of user for a user service in self-driving vehicle. The proposed system is designed to classify the four emotional information (stability, relaxation, tension, and excitement) by using EEG data to infer and classify emotional state according to user’s stress. SVM algorithm was used to classify bio information according to stress index using brain wave data of the fuzzy control system, pulse, and blood pressure data. When 80% of data were learned according to the ratio of training data by using the SVM algorithm to classify the EEG, blood pressure, and pulse rate databased on the biometric emotion information, the highest performance of 86.1% was shown. The bio-information classification system based on the stress index proposed in this paper will help to study the interaction between human and computer (HCI) in the 4th Industrial Revolution by classifying emotional color and emotional sound according to the emotion of the user it is expected.
1. Introduction
In recent years, a new future technology called emotional artificial intelligence (AI) has emerged owing to the advancement of the fourth industrial revolution era. Particularly, AI-based emotional computing technology, which can interpret and analyze human emotions, is advancing rapidly owing to a convergence of information and communication technology (ICT) and cognitive science areas [1, 2]. Accordingly, the human–computer interface (HCI) technology is becoming increasingly important, and along with the progress made in the studies on HCI, studies are being increasingly conducted on computer reactions based on emotion inference or user intention rather than computer reactions induced by direct inputs of the user [3]. Particularly, the brain–computer interface (BCI) technology analyzes, commands, or controls electroencephalography (EEG) signals in frequency domain, which were measured at a human scalp. Various studies are conducted to provide interaction between humans and computers by converging the user’s emotion information and environment information based on the BCI technology [4, 5]. Emotion is a cognitive ability of humans and a reaction for external sensory stimulation. Humans react emotionally according to various social and cultural factors and feel various emotions accordingly. Emotion recognition technology is a means of making intelligent decisions that can enable appropriate behavior by extracting information such as facial expressions or body gestures of a user based on emotion data [6]. As such, it is extremely important for computers to have emotion recognition capability to process human emotions through learning and adaptation in order to process the interaction between humans and computers more efficiently. In modern society, mental stresses such as various work-related stresses, conflicts in interpersonal relationships, and financial problems have emerged as social problems, and efforts are actively being made to improve the quality of physical and psychological life [7]. As such, it is important to measure the emotional stress level of users by using EEG and biosignals in objective numerical values in terms of psychological state, and efforts to relieve such stresses by recognizing the subsequent physical changes are necessary. As colors and music are formed in a short time and last for a long time in memory, they can play vital roles in understanding and analyzing human emotions. Therefore, this study aims to design a system that infers emotion by using biometrics of user, such as EEG, pulse rate, and blood pressure (systolic and diastolic blood pressure), and that recommends colors and music based on the emotional state of the user, i.e., the stress index. The system proposed in this paper recognizes the emotion of the user by learning and patterning the reactions appearing according to the user’s emotional state and classifies the biometric emotional information according to the stress index. A fuzzy system was designed by using EEG data to classify the biometrics into biometric emotion information according to the stress index. Furthermore, it was designed to produce the pulse rate and blood pressure (systolic and diastolic blood pressure) data in single packets separately and to send them to the database. As such, after acquiring the biometrics, the biometric emotion information is classified according to the stress index through a support vector machine (SVM) algorithm. In general, various learning algorithms are applied to decision-support systems. However, as the biometric data used in this study consist of EEG, pulse rate, and blood pressure (systolic blood pressure and diastolic blood pressure) data, which have nonlinear data structures, the SVM algorithm was used to solve the nonlinear discrimination problem in a multilayer perceptron structure. Depending on the classified stress emotion, the data corresponding to the color and music values are classified. This paper is organized as follows. Section 2 discusses related research, and Section 3 describes the system configuration and design. Subsequently, Section 4 presents the performance evaluation and experimental results. Finally, Section 5 discusses the study’s conclusions and directions for future research.
2. Related Research
2.1. Cognitive Science
Cognitive science examines the cognitive processes of humans and animals, such as perception, language, learning, and emotions. In this field, techniques are studied for applying these cognitive processes to the development of robots, electronic products, buildings, and so on [8, 9]. Recently, as cognitive science has begun to attract attention, studies are being conducted on the detection, processing, and analyzing of various types of signals that are generated by bodily activities. In particular, studies on the structure and functions of the brain are being actively conducted. These studies utilize various biometrics for not only medical diagnoses, but also the examination of people’s cognitive and emotional states through analysis via certain algorithms.
2.2. Emotional Engineering
Emotions are high-level psychological experiences that occur in people via their senses and cognition in response to an external physical stimuli. These are often-changing psychological effects that occur in people owing to complex feelings, such as comfort, pleasure, unease, and discomfort. These are the moods and feelings that are connected to expressive behavior during emotional reactions and physiological changes. Further, they are a dynamic aspect of thought. Emotional engineering is a field that studies emotions and deals with them in practical terms. This combines areas related to human psychological reactions and their applications, such as human factor engineering, cognitive science, behavioral science, pragmatic aesthetics, and environmental psychology [10]. Changes in people’s psychological states are expressed externally in the form of feelings or emotions. In the past, people’s psychological states and emotions were mostly analyzed and used as clinical data for medical diagnoses and treatments. However, they are currently used in a variety of fields, such as buildings that incorporate psychotherapeutic art and automatic lighting that reflects emotions, cars that prevent sleepiness, and robots that understand and respond to human feelings. Researchers are studying emotionally intelligent computer technology that can understand psychological states, feelings, and emotions. Emotionally intelligent computing allows computers to recognize people’s feelings and emotions and to perform actions that are suitable to the circumstances. This entails autonomous systems that are able to perform suitable actions based on previous knowledge or the current psychological state. Recently, as wearable computing technology has evolved, it is able to measure biosignals such as electroencephalography (EEG), electrography (EMG), electrocardiogram (ECG), and galvanic skin response (GSR) more accurately, and produces improved recognition results regarding subjects’ mental states, emotions, and physiological states [11].
2.3. Biobased Emotional Cognitive Technology Trend
In general, the emotions felt by people can be distinguished by the central nervous system or autonomic nervous system reactions, i.e., information such as ECG, EEG, skin temperature (SKT), and GSR. The biosignal emotion recognition technology, that has been studied so far, can be divided into studies using statistical methods and those using machine learning methods. Studies on rule-based techniques analyze many biosignals, and extract emotional features that express emotions. Then, threshold values that correspond to rules are set for each emotion, and emotions are classified according to these thresholds. Emotion recognition methods that use machine learning are based on techniques such as neural network (NN), support vector machine (SVM), k-nearest neighbor (kNN), multi-layer perceptron (MLP), Gaussian mixture model (GMM), decision tree (DT), and Baysian network (BN) [12–14]. For biosignal learning, it is necessary to collect a large amount of feature data and perform the training required for emotion recognition. Therefore, the amount of collected feature data and its reliability have a significant effect on the subsequent performance of the recognition system. However, most emotion recognition systems that have been studied so far use data with staged and exaggerated emotions. Because it is easy to artificially acquire learning data that includes emotions, this is a major factor that degrades the performance of emotion recognition systems in real situations. Therefore, the collection of reliable biometric data is considered necessary for biosignal-based emotion recognition. However, in biosignal collection, a user’s emotional states can easily change according to the environmental conditions or the user’s psychological state, and it is very difficult to perfectly recognize a user’s emotions based on a single biosignal. Therefore, rather than emotion recognition using only one biosignal, researchers are investigating methods that use several biosignals in conjunction or use additional emotional measurement indicators, such as voice and facial expressions. Emotion recognition technology using brainwaves is one of the most actively studied fields in biotechnology. Unlike other biosignals, brainwaves show the state of the central nervous system. The alpha and beta waves that occur according to the state of brain activity are known to be related to emotion. Alpha waves generally indicate a stable or relaxed state, and these increase during positive emotions. On the other hand, beta waves increase during negative emotional states [14]. In addition to brainwaves, autonomic nervous system reactions occur when commands are received from the brain, and these indicate bodily changes when a person is surprised or encounters danger. These reactions include the heart beating rapidly, the facial muscles stiffening, the palms and back of the neck sweating, and changes in body temperature or the temperature at certain parts of the body. These bodily changes are controlled by the autonomic nervous system, and therefore emotional states can be inferred from biosignals by detecting autonomic nervous system reactions. Biosignals that can be measured following autonomic nervous system reactions include ECG, EMG, GSR, and SKT. Studies have used the heart rate variability (HRV) and heart rate (HR) to recognize emotions. A low heart rate indicates a relaxed state, and a high heart rate can indicate stress, frustration, and dissatisfaction. As such, the heart rate is often used for data analysis. Emotions can be classified by calculating the HRV from the heartbeat and extracting features. However, it is difficult to accurately recognize emotions from the heartbeat alone [15]. Emotion recognition via EMG has mainly consisted of studies on the recognition of emotions through the movement of facial muscles. The movement of facial muscles can be measured through an EMG of the face, and this can be used to measure high-stress tension and so on, and recognize emotions. In a tense/aroused state, the EMG increases, whereas in a relaxed state it decreases. However, the absolute levels of muscle tension vary according to the part of the muscle that is measured. As such, this signal requires precise measurement. GSR essentially refers to skin conductivity, which increases when sweat is present on the skin. This can be used as an indicator of stimulation or stress. The GSR amplitude increases during arousal or negative emotions. The reaction speed (latency) is quick during sensitive stimulation. The reaction sensitivity (slope) is large during sudden or sensitive stimulation. Therefore, GSR can be used as a good indicator for measuring negative emotions. SKT is different from body temperature, in that it is a temperature index for certain body parts. This is not a normal core biosignal indicator, but it can be used as a slow indicator of changes in emotional states. It is significantly affected by external environmental factors. In general, when the amplitude of the SKT signal is large, this indicates relaxed and pleasant positive emotions. When it is small, this indicates tense or uncomfortable negative emotions. Recently, there has been a sudden interest in many different fields in attempts to monitor image, voice, biometric, brainwave, and body data and extract emotions to provide emotional application services. Emotional application service technology is evolving, and exhibits significant potential for use in a variety of fields, such as entertainment, healthcare, market analysis, online education, automobiles, customer marketing, and general home use.
2.4. Stress and Bio-Information
Previous studies have used biosignals as a method of identifying stress states in individuals. Bakker et al. [16], Healey and Picard [17], and Jung and Yoon [18] have collected biosignals from workers, drivers, and senior citizens, respectively, to evaluate stress levels. Setz et al. [19], Melillo et al. [20], and Kurniawan et al. [21] administered tests that required learning capabilities to create experimental environments, and then identified stress states. However, these previous studies have been limited by the fact that they considered the identification of a posteriori states, and did not consider combinations of features that are appropriate for stress relief using emotions. In this chapter, previous studies on the identification of stress states using biosignals are listed by their purpose, the biosignals that they used, and their analysis techniques, as shown in Table 1. Previous studies have identified stress states using various combinations of biosignals such as HR, GSR, ECG, EMG, and brainwaves. Of these, the heartbeat data has been used most often to identify stress. However, because heartbeat variability can occur in a variety of contexts other than stress, this is used in combination with other data [22]. Sun et al. [23] used the heartbeat data along with GSR and accelerometer data to identify stress during physical activities. However, because GSR is more sensitive to movement than other biosignals, it is difficult to obtain accurate data in situations with a lot of movement. Kurniawan et al. performed experiments using voice data to analyze the words uttered by subjects and identify stress [21]. In studies by Setz et al. and Melillo et al., subjects were given tests that required learning capabilities to create experimental environments. In this manner, previous studies have faced difficulties in adequately measuring and analyzing user stress without using a variety of biometrics. In addition, the goals of studies on stress that have used biometrics have only been focused on measurement and analysis. As such, there has been a lack of studies on stress and emotions that are measured via biometrics. This study considers the limitations of previous work, and employs user biometrics such as brainwaves, pulse, and blood pressure (systolic and diastolic blood pressure). In addition, it proposes a system that uses biosignals to identify a user’s stress states and infer emotions, to suggest colors and sounds according to the user’s emotional state, i.e., their stress index.
3. System Configuration and Design
To infer and classify the emotional state of a user according to the user’s stress, this study designed a fuzzy system by using the EEG data after acquiring the EEG, pulse rate, and blood pressure from the sensors, and the pulse rate and blood pressure (systolic and diastolic blood pressure) data were produced in single packets separately and sent to the database. The SVM algorithm was used to classify the biometric emotion information according to the stress index by using the acquired EEG, pulse rate, and blood pressure data of the fuzzy control system. The biometric emotion information is classified into color values corresponding to the emotion based on 20 color emotion models selected through HP’s “the Meaning of Color,” and as for the music, the music pieces provided by “Samsung Idea” for music therapy were collected, classified, and used [27]. Figure 1 shows the configuration diagram of the system proposed in this paper, which performs emotional classification and recommendation based on biometrics and stress index.
This study designed a fuzzy system to classify the biometric emotion information according to a user’s stress index by using the EEG data. The stress emotion information expressed by the fuzzy control system can be divided into four types: stable, relaxed, tensed, and excited. BIOPAC MP 150 was used to measure the EEG of the user and the measurement was conducted according to the 10–20 International System of Electrode Placement [28, 29]. When conducting the measurement, sampling was performed at 256 Hz for only the EEG data that passed through a notch filter of 60 Hz in terms of hardware. The sampled data were filtered in the frequency band 0.5–50 Hz. To analyze the EEG data, the data for 60 s, which is considered the stable-state analysis section for the analysis of EEG, were used excluding each 30 s part of starting and ending. First, the EEG data, which are the data in time domain, were converted into frequency domain by using fast Fourier transform, as shown in Equation (1) in order to analyze them.
To extract an absolute size by each frequency, the data were converted into the frequency domain through the Fourier transform process. Moreover, the power spectrum analysis method was used for comparison and analysis [30]. The power spectrum analysis method is a widely used analysis method in many areas including biosignals. Depending on the expression method, it is divided into one-side and two-side. One-side shows 0 and positive frequency domain only, and two-side shows the frequency domain of {negative, 0, positive}. This study obtained the absolute values of power spectrum for the band frequencies of , , , and by using the one-side power spectrum analysis method. The Fourier inverse transform is shown in Eq. (2)
After obtaining the absolute values on both sides of Equation (2), if they are squared and added, then Equation (3) is induced. Here, it can be confirmed that the sum of square of the signal that was subjected to the Fourier transform is the same as the sum of square of the original signal. Here, the sum of Fourier transform or the sum of square of the original signal is called the total power value. In other words, it indicates that the total power value is identical in both the frequency space and the temporal space. This is called Parseval theorem [31].
The one-side spectrum analysis method that satisfies the Parseval theorem is shown in
This study extracted the absolute values of the power spectrum of theta (4–8 Hz), alpha (8–14 Hz), beta (14–30 Hz), and gamma (30–50 Hz) domains through the one-side power spectrum analysis, and used it as an input value of the fuzzy control system to classify the emotion.
To classify the emotion of the user, the fuzzy model should be constructed by quantifying the physical information that can identify the emotion of the user from the EEG data. To this end, the correlation of , , , and values, i.e., the absolute power values acquired from the user with the user emotion, was investigated to define the rules. Then, the fuzzy model was implemented as shown in Figure 2. In other words, the biometric emotion classification (stability, relaxation, tension, and excitement) was performed by applying the four fuzzy rules using the absolute power values , , , and obtained from the EEG data of the user as the input values. The fuzzy control rules are expressed as linguistic control rules of “IF Ai THEN Bi” format. To create a set of control rules, multiple rules are collected and by making an inference based on the rules, an output value is obtained. Table 2 shows the fuzzy condition for expressing the output value according to the input variable by using the EEG data.
Defuzzification refers to a process of converting a fuzzy quantity result, which is obtained from the fuzzy inference, to a representative value. In this study, the center of gravity method was used for the fuzzy inference [32, 33]. Equation (5) shows the inference output function defuzzificated by the four fuzzy rules.
To measure the pulse rate and blood pressure data of the user, this study used the ZigBee wireless sensor network. ZigBee of IEEE 802.15.4 refers to a short-range wireless communication technology that focuses on applications requiring low power consumption, low speed, and low cost [34]. A sensor module of integrating pulse rate and blood pressure sensors was used to acquire the data. Furthermore, the data were measured by using a Telos platform series as the process board, MSP430 MCU, and CC256XQFNEM. If the pulse rate and blood pressure data are produced in single packets separately and used, energy is consumed based on additional traffic and data transmission. Therefore, they were bundled in one packet and then sent to the database. Figure 3 shows the structure of sensed biometric data. MSG shows the type of biometric information and enables to isolate the type of pulse rate, i.e., systolic or diastolic blood pressure. GroupID shows the sensor information and each sensor has a GroupID. Timestamp is the time when the sensor measured the data. Reading expresses the actually measured data values in hexadecimal 2 bytes.
Various learning algorithms are used to match the biometric emotion information according to the stress index. Nevertheless, this study used the SVM algorithm, which can solve the nonlinear discrimination problem of multilayer perceptron structure because the EEG, pulse rate, and blood pressure data used in this study consist of nonlinear data structures. The goal of the SVM algorithm is to find a classification boundary by maximizing the margin between the closest observation values (support vectors) in two classes. Furthermore, even when linear separation is difficult, classification is facilitated by using a nonlinear hyperplane through a kernel function, and consequently, it is used in diverse areas such as biology, and image and text recognition. To optimize the performance level of learning data, most traditional pattern recognition methods are based on a risk minimization method. In the case of the SVM algorithm, it is based on a structural risk method to minimize the probability of incorrect classification of the data having fixed but unknown probability distribution [35]. When data cannot be linearly separated like the data of this study, i.e., when they have a pattern that cannot be completely separated because they are overlapped with each other at the linear separation boundary, a slack variable is used considering a case of incorrect classification, as shown in Eqs. (6) and (7)
However, C is a trade-off parameter and the Lagrangian function is expressed by
The above Equation (8) can be converted to Eqs. (9) and (10) by applying a Lagrangian optimization method.
If the above equations are solved, a decision function can be obtained, as shown in Eq. (11
The SVM algorithm developed for binary classification has many difficulties when solving problems having many classes in a real environment, one-against-all and one-against-one methods have been proposed. Among them, the one-against-one method consists of quantity of the SVM algorithm when k classes are inputted. Furthermore, for respective learning data that consist of data showing two groups they belong to, as the number of learning data used is small when performing the learning, the learning speed is fast [35, 36]. Therefore, this study conducted experiments by using the one-against-one method to improve the learning performance and composed the SVM algorithm as shown in Table 3.
As various kinds of emotions can be expressed according to the external environment, it is effective to predefine the emotion colors and music to be used. Accordingly, for the classification of emotion colors, this study selected 20 color emotion models as the representative elements from HP’s “the Meaning of Color.” Through HP’s color table and the emotion vocabulary matching result investigated in this study, the classification was performed based on the common emotion vocabulary, as shown in Table 4.
This study classified the colors and music corresponding to the biometric emotion information according to the stress index by using wavelength, which is a common characteristic of emotion information, color, and sound. As wavelength and frequency have an inversely proportional relationship in physics and mathematics, they can be converted to each other mathematically. Based on C (on the musical scale), the relationship with the wavelength ratio of D and E is 1 : 4/5 : 2/3. This ratio is consistent with the ratio of the respective wavelength of three primary colors, i.e., red, green, and blue: 650 nm, 520 nm, and 433 nm. As such, C, D, and E are similar to the three primary colors, which can be used to create countless colors through appropriate mixing. Therefore, if the wavelength ratio of dodecatonic scale based on the equal temperament is corresponded sequentially to the frequency of colors that can be created with the combination of the three primary colors, the colors and sounds can be linked [37].
By analyzing the color measurement values and the musical scale of music, the optimal colors and list of music corresponding to the biometric emotion information are determined according to the stress index. Music, which is an important emotion information for understanding and analyzing the emotions, can heal the emotion of the user and relieve stress. Therefore, to recommend music based on the biometric emotional state according to the stress index, this study composed a music list based on the data collected from “Samsung Idea” for musical therapy. Table 5 classifies the biometric emotion information according to the stress index and shows the classification of the corresponding colors and music.
4. Performance Evaluation and Experiment Results
In this study, the experiments were conducted by using the EEG, pulse rate, and blood pressure as input data to classify the biometric emotion information according to the stress index. Table 6 classifies the four states (stability, relaxation, tension, and excitement) based on the classification standard of biometric emotion information according to the stress index [38, 39].
This study used the SVM algorithm to design and evaluate the classification of respective emotions. The SVM algorithm is a classification algorithm that determines the discrimination boundary in order to have the largest distance between the discrimination boundary and each class. The kernel of the SVM algorithm used in this study was the radial basis function kernel; the radius of the kernel was set to 1 and the margin of the SVM algorithm was set to 1 [40, 41]. Figure 4 shows the accuracy result of classifying according to the four types of biometric emotion information through the SVM algorithm. When 80% of data were learned according to the ratios of learning data, the highest performance of 86.1% was shown. Furthermore, Table 7 shows the error matrix when 80% of data were learned. The classification results were 90.4% for stable, 83.4% for relaxed, 84.5% for tensed, and 85.7% for excited.
Furthermore, to evaluate the tracking performance of the proposed system that recommends colors and music according to the biometric emotion information and stress index, 10-fold cross validation was conducted, thereby minimizing the influence of training data and ensuring reliability. The 10-fold cross validation divides the entire data into ten equal parts and uses nine parts as training data and the remaining part as test data. The 10-fold cross validation repeats the training and testing ten times in total to evaluate the overall performance, and every time, one equal part of data used for the test is changed. Figure 5 shows an example of 10-fold cross validation.
This study divided the experimental data into evaluation data and validation data using the ratio 7 : 3. Furthermore, the system was optimized by using the leave-one-out method. Table 8 shows the average accuracy of 86.4% in the result of validation data obtained through the performance evaluation. Therefore, the performance of the recommendation system is high.
Figure 6 shows the result of implementing an application for the proposed system that recommends the colors and music according to the biometric emotion information and stress index. The emotion colors and the emotion music are classified into the corresponding items according to the biometric emotion information and stress index.
5. Conclusion
This study aimed to classify and recommend the emotion colors and the emotion music corresponding to the emotion of the user by measuring biometric information. Accordingly, after measuring the EEG, pulse rate, and blood pressure data (i.e., the biometric information), the emotion colors and emotion music are classified according to the current biometric emotion information and stress index of the user through real-time emotion analysis. To classify the EEG data into four types of biometric emotion information (stability, relaxation, tension, and excitement), a fuzzy control system was designed and the pulse rate and blood pressure data were composed in single packets separately and sent to the database. When 80% of data were learned according to the ratio of the training data by using the SVM algorithm to classify the EEG, blood pressure, and pulse rate data based on the biometric emotion information, the highest performance of 86.1% was shown. Moreover, the error matrix classification results obtained were 90.4% for stable, 83.4% for relaxed, 84.5% for tensed, and 85.7% for excited, confirming that the efficiency was high. Furthermore, when the 10-fold cross validation was performed to evaluate the tracking performance of the system that recommends the emotion colors and emotion music according to the biometric emotion information and stress index, an average accuracy of 86.4% was demonstrated, confirming that the performance of the recommendation system was high. This study proposed a biometric emotion information classification system using the stress index to classify the emotion colors and emotion music intelligently based on the emotion of the user. Therefore, it is expected to contribute to studies on HCI in the fourth industrial revolution era. In a future study, the emotions will be classified based on the situational and environmental factors as well as the biometric information of the user. Furthermore, the accuracy and efficiency of the proposed system will be enhanced by using various types of biometric information in addition to the EEG, pulse rate, and blood pressure used in this study, as measures for determining biometric emotions.
Data Availability
The BIOPAC MP 150 EEG data used to support the findings of this study have been deposited in the BrainAmp by Brain Products, Munich, Germany.
Conflicts of Interest
The authors declare that they have no conflicts of interest regarding the publication of this paper.
Acknowledgments
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIT) (No. 2019R1F1A1041186). This research was supported by the Korean MSIT (Ministry of Science and ICT), under the National Program for Excellence in SW (2017-0-00137), supervised by the IITP (Institute for Information & communications Technology Planning & Evaluation).