Abstract

The use of tactile senses in mediated communication has generated considerable research interest in past decades. Since massage is a common practice in Asian cultures, we propose to introduce massage-based interactions in mediated communication between people in a close relationship. We designed a device for distant interactive massage to be used during online conversation and we assessed its effect on interpersonal connection with eight pairs of Chinese participants in romantic relationships. All pairs were asked to engage in a conversation, either through a video call or through a massage-assisted video call. The findings showed that the use of the massage device significantly increased the perceived emotional and physical connection between the users. The results also showed a significant increase in the engagement in the massage activity, e.g., total massage time and average force per finger, from positive conversation to negative conversation, demonstrating an evidence of the interplay between audio-visual and haptic communication. Post hoc interviews showed the potential of the massage device for long-distance communication in romantic relationships as well as in parents-children relationships.

1. Introduction

In human-computer interaction (HCI) research, affective computing has found applications in online social interaction. Although most existing communication techniques exploit only two of the human senses, visual and auditory, there is a growing interest in bringing the sense of touch to computer-mediated communication, as reported in the review paper by Eid and Osman [1]. Haptic communication refers to communication via the sense of touch. At birth, touch is the most developed sensory modality and it plays fundamental role in communication throughout childhood [2]. However, after humans learn to use languages, haptic communication became mostly neglected. Still, interpersonal touch is still used in intimate relationships, as it is an important form of communication for conveying intimate emotions such as love and sympathy [3]. Indeed recent studies have shown that basic emotions, e.g., anger, fear, and happiness, can be communicated through touch only [4, 5].

In distant communication, human still predominantly rely on visual and auditory senses. New ways of communicating emotions, e.g., emoji or Animoji, are still relying on visual sense. On the other hand, geographically separated family or couples often long for physical presence and interaction [6] and touch-based communication devices are expected to mediate intimate relationships and create a sense of togetherness [7].

In this context, we aim at supporting long-distance communication through haptic interactions created by a massage-inspired device. Our study examines how the use of a vibrotactile device during a video call affects interpersonal connection and sense of togetherness of couples in long-distance communication.

2.1. Existing Mediate Haptic Communication Devices

We reviewed existing haptic communication devices and categorized them based on the type of interactions they rely on (Table 1). Since mediated physical interaction is not necessary equivalent to mediated social touch [35], we did not include devices that allow remote physical interactions that are not relevant to mediate touch between humans.

2.2. Massage-Based Interactions

Interactions through massage have the potential to be used practically in mediated communication since it is one of the interpersonal interactions that can be done continuously in a period of time without a feeling of awkwardness. Being given a massage by a known person creates a feeling of “being cared for” and “existential well-being” [36]. A hand massage as a “comforting, caring intervention” can “enhance satisfaction with care” [37]. Massage can also be seen as a way of communicating, as it enables mutual interpretation [38]. However, despite the acknowledged importance of massage for emotional well-being, massage-based interactions have never been implemented in mediated communication.

2.3. Cultural Context in Haptic Communication

Human emotion is not only biologically determined but also influenced by environment and culture [39]. While basic emotions are common across cultures, the subordinate categories of emotions are culture specific [40]. Similarly, even if haptic communication is instinct-based, cultures and context still affect the narration to touches. One obvious example is that, in Western cultures, people greet each other with hand-shaking or even cheek-touching in close friends or families, while, in Eastern cultures, people bows or use gestures without any touching at all. In mediated social touch context, it is important to consider not only the senses generated from the artificial touch, but also the touch etiquette [41]. Guidelines of this culturally aware etiquette have been suggested for the development of social robots [42]. A study about the gestural communication of emotions showed that French participants chose to perform gestures more intensely and faster than Japanese participants [43]. While most previous studies examined mediated touch in Western contexts, our paper focuses an Asian context and we aim at providing culturally grounded insights into the field of tactile mediation for distant communication.

3. Research Question

Through the introduction of a massage-based interactions in long-distance communication, we address the following research question: how does the use of a massage-based device affect interpersonal connection in long-distance communication, e.g., video call?

In the following sections, we report the development of a massage-based device and its experimental assessment with pairs of participants in a romantic relationship, in a distant communication task. We examined how participants perceived the connection with the other person, including sense of togetherness, emotional connection, and physical connection, during a video call, either with or without the massage-based device.

4. Creation of an Interactive Massage Device

4.1. Preliminary Survey

Through an online survey, we collected opinions from the general public about tactile sense distant communication.

4.1.1. Respondents

50 respondents (25 males, 25 females) aged between 18 and 27 years (M = 21.48, SD = 1.88), from Asian countries (50% from Japan, 50% from nine other Asian countries) participated in the online questionnaire.

4.1.2. Survey Results and Discussions

The questionnaire consisted of nine questions about respondents’ general uses of distant communication services and their opinions on tactile senses in distant communication. 65% of the respondents responded that they would like to be able to touch the person on the other side during video call. In the other 35%, 74% of the respondents answered, in another question, that they would like to use a device that provides a feeling of touching each other during a video call, if this device existed. 94% of the respondents thought that enabling tactile sense in distant communication would improve emotional connection. Overall, we observed that most of the respondents liked the perspective of using tactile senses in distant communication, even if they had not thought of it before.

The respondents also had to select a part of the body they would like to be given a massage to, in a distant communication scenario (Figure 1) and “shoulder” was the most popular option (38%).

4.2. Prototyping a Massage Device

We designed and made a prototype that simulates the feeling of massaging through vibrotactile stimulation.

4.2.1. Physical Prototyping of the Massage Device

We designed a device that would consist in two parts, a “sender device” for the person giving the massage and a “receiver device” for the person receiving the massage (Figures 2 and 3) and produce a vibrotactile stimulation on a shoulder. We used force sensitive resistors to collect input through the sender device and vibrotactile motors to create vibrotactile output through the receiver device. Vibration was chosen as the form of pressure output because of the simplicity of the creation and its success in communicating emotions [8, 10, 11, 21, 23, 44].

The sender device is an object shaped like a human shoulder to provide the massager with a feeling of similar grip. The receiver device consists of two human-like hands made of LEGO parts. The palm part of each hand is connected to five fingers with spherical pair allowing three degree-of-freedom movements. Each finger also contains a second joint using cylinder cylindrical bore pair allowing the finger to bend, and they are covered by rubber thumb tips. The hands parts were put on the receiver’s shoulder and straps were adjusted to keep the device in place and in contact with the shoulder.

4.2.2. Usage Process

A block diagram of the massage device is shown in Figure 4. As for the sender device, in order to detect forces from all ten fingers, ten FSR402 force sensitive resistors are used and placed in positions where ten fingertips would be. Each sensor measures the forces from each fingertip of two sender hands. Sensor value is measured as analog information. There is no force feedback to the sender. As for the receiver device, in order to simulate the touch and massage feeling on the shoulder, ten FM34F vibrotactile actuators (standard speed 13000 rpm with vibration quantity 17.6 m/S2) are used. Each actuator receives signal from the force sensitive resistor in corresponding position and results in vibration. Here, the analog information from force sensitive resistors is converted to digital information since the limitation of the vibrotactile actuators we used could either be turned off or on only. Hence, the vibrotactile actuators would start vibrating when pressed and continue until the sender stops pressing on the force sensitive resistors. In other words, the duration of pressing is conveyed by the device, but the strength of the pressing is not.

An Arduino board controls both the sender and receiver devices and Mux Shield allows enhancing the numbers of possible input and output from the board. The devices are connected with wires to prevent any technical delays of signals. For a real-life usage, the devices would be wireless and connected to the Internet.

5. Experimental Evaluation

5.1. Objective

Our objective is to study users’ perception of massage-type tactile interaction in mediated intimate communication. We specifically investigate how massage in mediated intimate communication affects the interactions and the perception of the communication, including enjoyment, sense of togetherness, emotional expression, and emotional and physical connection. The experiment examines the impact of massage in conversations, either positive and negative.

5.2. Participants

Eight couples in a romantic relationship, aged 19 to 26 (M=21.56, SD=2.15), participated in the experiment. All participants were from a Chinese cultural background and were recruited through the group of international students at [author’s university]. They have a regular practice of video calls with their family members and friends with no experience of using tactile device for communication.

5.3. Experimental Setting

The experiment was conducted in a laboratory with a pair of participants at a time. Each participant was sitting in front of a laptop and was separated from the other participant by a large panel, so they could not see each other. They were required to use sound isolating headphones to prevent hearing any sound or voice directly from each other (Figures 5 and 6).

5.4. Experimental Conditions and Tasks

In the control condition, the communication between the participants was verbal and visual only, i.e., video call through Skype. The second condition was verbal, visual and tactile, i.e., video call through Skype and the massage device. Because the prototype of the receiver device had to be strapped to the chest of the wearer, we asked male participants to act as receivers and female participants to act as senders. In both conditions, the pairs were required to perform two different tasks extracted from Suhonen et al. [17].

Task 1 (have a conversation about a happy topic). It could be something that happened in the past week or month that made you happy or excited, something you achieved that you are proud of, or something you are looking forward to, or anything else that is positive topic.

Task 2 (have a conversation about a sad or angry topic). It could be something that happened in the past week or month that made you sad or angry, something you fail or some troubles you caused, or something coming in the near future that you are afraid of or worrying about, or a complaint, or anything else that is a negative topic.

These tasks were chosen since we believe that the prototype would be used in different ways or different purposes under different types of emotion. The topic could be started either sender or receiver depending on each pair of participants. In order to counterbalance the order effect, the eight couples were arranged in different conditions and tasks (Table 2).

5.5. Experimental Procedure

In this section, we describe the experimental procedure for the first pair; for the other pairs, only the order of conditions and tasks changed. The overall experiment for each pair of participants lasted around one hour.

The experiment started with a brief explanation about tactile communication and the research objective. The participants were seated on both sides of a big panel and could not see each other. They were asked to imagine that they were sitting in different places apart from each other. Then they were asked to put on headphones and have a conversation through video call about a positive topic (happy) for three minutes. They were then asked to switch to a negative topic (sad or angry) for three minutes. Finally, they were asked to complete a questionnaire about their experience with the device.

Then they were preceded to the second condition, we showed them both parts of the massage device and instructed them how to use the device. They were provided ten minutes to test the device freely and could communicate with each other during the test session. After they were satisfied with the testing, they were asked to put the headphone on and use video call together with the provided massage device. They were asked to talk to each other with positive topic then negative topic for three minutes each. (The order of the tasks was the same as what they did in the first condition.) When both tasks were finished, they were asked to put the headphone off, and filled the same questionnaire.

After both tasks were completed, the participants were interviewed together as pair for about 20 minutes. The flow of the interview is free. We asked them to express their opinions towards the tools, their experiences towards tactile communication and their feelings towards the conversation.

5.6. Data Collection and Analysis
5.6.1. Analog Data Collected from Ten Force Sensitive Resistors

The values of ten force sensitive resistors (FSR) were recorded every 200 milliseconds, with the active threshold at 0.17 N. To convert the sensor value to force in Newton unit, the simplified circuit of one FSR is shown in Figure 7, and nine other FSR are also connecting parallel to this circuit.

First, the value from the sensor (S; range from 0 to 1023) was converted to voltage as follows:Then resistance across the sensor () is calculated from the following:Then value is compared with the official graph of FSR sensor [45], which can be approximated equation (3) when μC < 1000.

Four types of data were calculated from timestamp and sensor values:(1)Average force per finger was the average force (only when over the threshold) from each finger over three minutes.(2)Total massage times were counted from the number of time that at least one sensor was active. If the inactive duration is less than one second, it will be considered as continuation from previous active time.(3)Total massage duration was the total duration of each active time that at least one finger is activated.(4)Average numbers of fingers used were counted from the maximum number of active force sensitive resistors at each time calculated in “total massage times”.

There was a technical error on recording the sensors in one pair on positive topic task. Hence their data of both positive and negative topic was excluded from the analysis. There was no error on the vibration feedback; therefore this pair’s questionnaire and interview data are included in the analysis. Normality of the data in this section was verified using Kolmogorov-Smirnov test. Hence, paired-sample -tests were conducted for comparison.

5.6.2. Questionnaire about the Perceived Communication Experience

Participants had to assess their communication experience with video call (condition 1) or video call and massage device (condition 2) in a questionnaire that consisted in five criteria: enjoyment, sense of togetherness, emotional expression, emotional connection, and physical connection (Table 3), on a 7-point Likert scale. Here, we selected to use only self-report measures since the affective responses to haptic stimulation are shown to have significance differences in terms of perceived emotions but not in terms of biometrics measurements (e.g., electromyographic (EMG) measurement and skin conductance (SC) measurement) [46]. Kolmogorov-Smirnov test was used and showed that data was not normally distributed; hence Wilcoxon's signed rank test was conducted for comparison.

5.6.3. Comments and Opinions from the Final Interviews

All the comments were recorded and selections of comments as well as the interpretations of those comments are discussed.

6. Results

6.1. Device Usage Comparison during Positive and Negative Conversations

Paired-sample t-tests were conducted to evaluate the impact of positive and negative conversation on average force per finger, total massage times, total massage duration, and average finger used. There was a marginally significant increase in average force per finger from positive conversation (M = 0.79, SD = 0.33) to negative conversation (M = 0.90, SD = 0.63), t(69) = -1.84, p = .070 (two-tailed). The mean increase in average force per finger was -0.11 with a 95% confidence interval ranging from -0.24 to 0.01. The eta squared statistic (.05) indicated moderate effect size.

For total massage times, there was a statistically significant increase from positive conversation (M = 6.57, SD = 3.05) to negative conversation (M = 12.71, SD = 6.53), t(6) = -3.26, p = .017 (two-tailed). The mean increase in massage times was -6.14 with a 95% confidence interval ranging from 1.88 to -10.75. The eta squared statistic (.64) indicated large effect size. We found no significant difference in total massage duration for positive conversation (M = 54.51, SD = 30.91) to negative conversation (M = 35.07, SD = 12.28), t(6) = 1.68, p = .144 (two-tailed). The magnitude of the difference in the means (mean differences = 19.44, CI: -8.87 to 47.74) was large (eta squared = .32). There was a significant decrease in average numbers of finger used from positive conversation (M = 7.08, SD = 1.50) to negative conversation (M = 4.45, SD = 1.87), t(6) = 2.53, p = .045 (two-tailed). The magnitude of the difference in the means (mean differences = 2.63, CI: -0.08 to 5.18) was large (eta squared = .52).

6.2. Subjective Perception of the Communication with a Massage Device

Participants rated their experiences with the device using 7-Likert scales. A Wilcoxon signed rank test was conducted (Table 4). There was no significant difference in enjoyment, sense of togetherness, and emotional expression. On the other hand, there were statistically significant increase in emotional connection from video call only (Md = 4.50) to video call and massage (Md = 5.00), z = -2.34, p = .019 (two-tailed), with a large effect size (r = .58). For physical connection, there was a statistically significant increase from video call only (Md = 1.50) to video call and massage (Md = 5.50), z = -3.24, p = .001 (two-tailed), with a large effect size (r = .81).

6.3. Interview Comments

In this section, we report the outcomes of the interview with the participants, with a [#n-XY] code where n indicates their experiment number, X indicates whether they were using sender device (S) or receiver device (R), and Y indicates whether they were male (M) or female (F).

6.3.1. Increase of Emotional Connection

The majority of participants strongly confirmed that the massage device increases emotion sharing in distance communication. A participant commented that the device could help express something facial expressions and words cannot fully express, for example, caring, cheering up, and empathy. And the device makes her feel the presence of the other side [#8-SF]. Another participant stated that the device allowed her to release her anger and that can made her felt happier [#8-SF]. Two participants commented that when they are tired, they would like their girlfriend to send them some touch or massage them [#3-RM, #4-RM]. Another participant commented that he would like to use the device when he is missing his girlfriend [#3-RM].

6.3.2. Effects on Conversation

A participant expressed that the massage device improved the quality of their conversation, making it more meaningful and interactive. She also added that, for her, there used to be no difference between having phone call and video call when they were at a distance, but, with this massage device, video call would be more meaningful, and it would make long-distance relationship less suffering [#4-SF].

In addition, we also observed that some participants would be more likely to have informative conversations when they are communicating without the device. A participant stated that, with the device sometimes they could not focus on their topic, the device itself would become the topic of conversation occasionally. She also commented that actually that was a good thing, because they usually do not have anything important or urgent to talk about anyway, with this device you get another topic to talk about [#2-RM].

6.3.3. Comments on Sender Device

The interviews revealed a major usability issue with the lack of feedback to the sender side after a “massage message” has been sent. A participant stated that she pressed but she doesn’t know how strong the other side feel [#2-SF]. Another participant commented that she wanted mutual touch interaction and wanted to feel that she was actually pressing the shoulder of her boyfriend [#9-SF]. However, one participant gave an opposite opinion: she was happy that the massage device did not generate the same kind of feedback as real massage, and she did not need to use a lot of effort to give massage and that it made the device easier to operate since her arm does not get tired easily like real massage [#3-SF].

We believe that the massage device for communication might not necessarily have to provide the same feedback as a real massage, if the couple carries both sender and receiver devices, they can communicate and learn the feedback from each other, we suggest that the differences with a real-life massage might trigger more verbal interactions between the users and thus enrich the communication.

6.3.4. Comments on Receiver Device

One of the major opinions is that vibration is difficult to be felt as a haptic interaction directly coming from the person at the other end. A participant commented that the vibration was too weak [#6-RM]. Another participant expressed that real pressure would be a better option [#4-RM]. Two participants want the touch feeling to be softer, smoother and warmer like human touch [#5-RM, #7-RM]. One participant also stressed how important it is to be able to reflect the strength of the pressure that the other side pressed on the sender device to the receiver device, instead of a digital output [#2-RM]. One participants complained about the weight of the device that it made him tired [#3-RM].

6.3.5. Appearance of the Device

We discussed about the appearance of the device with participants, while there were no specific comments about the sender device, the majority of participants felt that the appearance of the receiver device was a bit scary, because of its human-like thumb tip. They suggested that the device should not look like human hands, but rather be robot hands [#5-RM], animal hands [#9-SF], cartoon characters hands [#9-SF], or even hidden from view [#6-RM].

7. Discussion and Implications

7.1. Massage Behavior in Positive and Negative Conversation

The results revealed that participants interacted with the massage device more actively during positive conversation, e.g., using many fingers at once, with marginally lighter force per finger. On the other hand, the massage behaviors were more focused during negative conversation, e.g., more frequent in massaging, with marginally stronger pressers. However, even if, in negative conversation, the massage was more frequently activated, the total massage duration was not significantly different from positive conversation. This could imply that the duration of each massage was shorter in negative conversation than positive conversation.

Here, due to the nature of receiver device that needs to be strapped to the chest, males are always receivers, and females are always senders in the experiment. This restricts our generalization of this part of the result, since the massage behavior observed in this paper only came from female participants, and this might not be able to represent male population. However, the massage behaviors from both positive and negative conversation showed a good agreement with Huisman [47] which observed that participants mostly used brief and forceful touches for negative emotions, e.g., anger and fear and used prolonged actions to communicate positive emotions, which their experiments consisted of majority male participants (80%). This implies that both genders should have similar trends in behavior regarding emotions.

They also commented that the device enhanced emotion sharing in the way the facial expressions or words cannot express, e.g., caring, cheering up, and empathy, including releasing unwanted emotion on the massage device and feel more relieved.

7.2. Perceived User Experience with the Massage Device

Our findings showed that participants felt significantly more physical and emotional connection when using the device, while there were marginally increases of sense of togetherness and no significant differences in enjoyment and emotional expression. This could be interpreted that the additional of haptic senses in terms of massaging made participants felt more connected physically and also emotionally. It could help them feel a little more together than without the device; however it is still far from really being together, hence only marginally significance difference. Here, it is possible that if we increase participant numbers, the result could be significantly similar to a study by Basdogan et al. that used the PHANToM, a one-point haptic device in the market, in collaborative environment context and showed significant increases in sense of togetherness [48].

On the other hand, the massage device only allows one action to perform with (massage) even if with freedom of forces or fingers, participants did not use the device directly for trying to send emotion to the other, but rather let the interaction be natural and emotions could result in subtle differences in massage behaviors. Finally, enjoyment seems to be independent from use of the massage device, but dependent on the conversation with their partner.

7.3. Design Implications

There are two types of haptic feedback: tactile feedback (temperature, pain, physical displacement of the skin) and kinesthetic feedback (force, weight) [35]. Real-life massage would normally make the massage receiver feel both types of feedback at the same time; tactile feedback: physical displacement of skin and warmth from hands, kinesthetic feedback: force and weight from hands. We created the receiver device to imitate the weight of having people’s hands on the shoulder and by using fabric together with rubber fingers; this would cause warmth from the receiver body temperature naturally after the device is worn for a short period. In real-life massage, these are considered as almost fixed values, since it depends only on the characteristic of the hands of the massage giver. Here, we simulated physical displacement of the skin using vibrotactile feedback, similar to many studies that successfully use vibrotactile arrays to represent touch, e.g., [8, 10, 11, 21, 23, 44]. We acknowledged that the force feedback is not successfully conveyed by our device, but our objective is not to make real-life massage possible in distance, but rather to complement video call with haptic feedback and interaction that are inspired by massage.

Although our sample size (N=16) was somewhat limited, we could find statistically significant differences in the results. Our sample consisted in young couples of university students because estimates suggest that up to half of university students might be in long-distance relationships at some time in their life [49]. Finer results would be found with a larger sample of participants, especially by including a wider range of age. Also, the experiment was conducted in Japan and the participants were Chinese students residing in Japan. Therefore, we believe that all participants had experienced long-distant communication with their family and might have already formed specific needs and impressions towards long-distance mediated communication. At the same time, they might not be fully representative of the general population and other potential users.

As for the scenarios of use, all participants answered that the primary targets would be couples in long-distance relationship. One in four female participants said that they would like to use the massage device with their close friends, while no male participant seemed interested in this scenario. Most participants (87.5%) suggested that they would like to use the device in long-distance communication with their parents, probably influenced by their actual current situation of being apart from their parents. To their opinion, more than the function, the fact of doing a massage for their parents would be important, as it could precede a symbolic message that they are caring for them and showing love to them. In fact, the mediated massage device has a potential to be used for elderly people, since there are numerous studies aiming to support communication for elderly people, but massage device has not been studied [50]. If a massage chair for elderly people is created, it could be a practical way to allow their family members to massage them even when they live far away to reduce their loneliness and increase their social interactions with their family. Target users for this scenario of use could be Chinese students overseas and their parents living in China, as the Chinese population of students is one of the most mobile in the world, with more than 500,000 people studying outside their home country [51]. The massage device could play an important role in supporting long-distance mediated family relationships in Chinese and other Asian cultural contexts where massage is a common practice among family members and filial piety a key virtue.

8. Conclusions

Our study investigated the use of a massage device in distance communication between people in a romantic relationship. The results demonstrated statistically significant increases in terms of emotional and physical connection, when using the massage device together with video call, as compared to using video call only. In addition, the interactions with the massage device were significantly more active during positive conversation and significantly more frequent and marginally stronger during negative conversation. We also discussed about rooms for improvement in the next potential future massage device that would allow conveying emotions between couples and family, thus to support mediated communication especially in Asian cultural context.

Data Availability

The recorded sensors data and voice recordings of the interviews (in Chinese language) used to support the findings of this study are restricted by the Ethics Committee of Tokyo Institute of Technology in order to protect participants’ privacy. Data are available from the corresponding author for researchers who meet the criteria for access to confidential data.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work was supported by the Department of Mechanical Engineering, Tokyo Institute of Technology.