Project topics and materials
Get Complete Project Material File(s) Now! »
Physiological activity in HRI
In HRI, one study (Mower et al., 2007) investigated if physiological parameters can be used in order to accurately estimate the engagement level of an individual while playing a wire puzzle game moderated by a simulated or an embodied robot. By using GSR parameters and the skin temperature, the authors reached an accuracy of 84.73% in estimating the engagement level of the participants in the study. The authors concluded that such an information could enable the robot to adapt its behavior in order to re-engage the individual it interacts with.
In (Liu et al., 2008), the authors have used the cardiac activity, heart sound, bioimpredance, electrodermal activity, electromyographic activity and temperature of six children with Autism Spectrum Disorder for affective modeling. They were able to detect three emotional states: liking, anxiety and engagement. The average accuracy is 85% for liking, 79.5% for anxiety and 84.3% for engagement. In a card game scenario developed for a humanoid robot, the authors of (Sorostinean et al., 2015), have used a thermal camera to measure the stress level of the individuals interacting with the robot. They have also found that the interaction distance between the robot and the individuals has an effect on the facial temperature. While being located in the personal space of the individuals (i.e., 0.6 – 0.7 m (Hall, 1966)), the temperature variation in the nose and perinasal regions was significantly higher than while being positioned in the social space (i.e., 1.2 – 1.3 m (Hall, 1966)). Furthermore, they have also found a significant interaction between the interaction distance and the type of gaze of the robot (i.e., direct, or averted).
By using the EEG signal of the participants in their study, the authors of (Ehrlich et al., 2014) have extracted two aspects of social engagement (i.e., intention to initiate eye contact, and the distinction between being the initiator or the responder of a gaze contact). These two measures enabled a robot to decide when and how to engage in an interaction with a human.
ME-type and cognitive performance
The morningness-eveningness type (ME-type) refers to the differences existing in individuals’ preference for the optimum time of the day to perform a given activity (Adan, 1991). Based on this distinction, an individual can be part of one of three groups: morning type, evening type, and neither or intermediate type. The most widely used assessment tool for measuring the ME-type is the Morningness Eveningness Questionnaire (Horne et al., 1976). For the development of the questionnaire, the authors have evaluated the oral temperature of the subjects and its variation throughout the day for the three groups (i.e., morning, intermediate and evening types). Further research has investigated the relationship between body temperature and efficiency. The research shows inconclusive results. Some (Fröberg, 1977) have found a negative correlation between the temperature and the omission errors for both morning and evening types. Others (Horne et al., 1980), have found a positive correlation for evening type individuals, and a negative correlation for the morning type individuals. For the study in (Horne et al., 1980), the authors have simulated a production-line inspection task and they found that morning type individuals had a better performance during the morning hours, while evening type individuals performed better in the evening hours. They also found that in the afternoon the differences between the two groups were smaller than during the morning hours. In (Song et al., 2000) the authors have investigated the relationship between MEtype, time-of-day, and the speed for processing information. They used two types of tests: inspection time test and a multidimensional aptitude battery. The authors found that morning type individuals performed significantly worse in the morning session for spatial sub-tests of the multidimensional aptitude battery. On the other hand, for the same task, evening type individuals performed significantly better in the morning hours than in the evening hours.
For a visual search task, the authors of (Natale et al., 2003) have found that morning type individuals were faster in the morning, while evening type individuals were faster in the afternoon hours.
Physiological measures for cognitive load
The response to a stimulus or a cognitive task can be measured by using certain physiological parameters (e.g., EEG, heart rate, respiration rate, GSR(Lisetti et al., 2004), facial temperature variation (Ioannou et al., 2014b), blinking, facial expressions). The measurement and the understanding of how these parameters vary are a good indicator of the internal state of an individual. Therefore, a robot that can measure these parameters in a contactless or a non-invasive way can determine the emotional state of the individual it interacts with. This could lead to the adaptation of a robot’s behavior in order to better interact with the individual. The variation of these parameters is also dependent on different personality traits, the sensory profile, or if the individual is a morning or an evening type.
Table of contents :
Acknowledgements
1 Introduction
1.1 Introduction
1.2 Motivation
1.3 Plan of the thesis
2 Related work
2.1 Introduction
2.2 Physiological activity in HRI
2.3 ME-type and cognitive performance
2.4 Physiological measures for cognitive load
2.4.1 Blinking
2.4.2 Galvanic Skin Response (GSR)
2.4.3 Facial temperature variation
2.5 Conclusion
3 Experimental Platforms
3.1 Introduction
3.2 Robotic platform: TIAGo
3.3 Robotic platform: Pepper
3.4 Sensors
3.4.1 Thermal camera
3.4.2 RGB-D sensor
3.4.3 GSR sensor
3.5 Cognitive games
3.5.1 Stroop Game
3.5.2 Matrix Task
3.6 Conclusion
4 Methodology
4.1 Introduction
4.2 Thermal data extraction and analysis:
4.2.1 Face detection
4.2.2 Facial feature point prediction
4.2.3 Thermal ROIs
4.2.4 Thermal data extraction
4.2.5 Thermal data analysis
4.3 Blinking
4.4 GSR
4.5 Questionnaires
4.5.1 Eysenck Personality Questionnaire (EPQ)
4.5.2 The Reinforcement Sensitivity Theory Personality Questionnaire (RST-PQ
4.5.3 Morningness-Eveningness Questionnaire (MEQ)
4.5.4 Adult/Adolescent Sensory Profile Questionnaire (AASP)
4.6 Conclusion and Contribution
5 RQ1: Relationship between Cognitive Performance and Physiological response
5.1 Introduction
5.2 Experimental design
5.2.1 Robotic Platform and Sensors
5.2.2 Questionnaires
5.2.3 Scenario
5.2.4 Conditions
5.2.5 Participants
5.2.6 Interaction time
5.3 Results
5.3.1 Task performance based on condition parameters
5.3.2 Task performance based on user profile
5.3.3 Physiological parameters variation based on condition parameters
5.3.4 Physiological parameters variation based on user profile
5.3.5 Results for morning individuals
5.3.6 Results for evening individuals
5.3.7 Correlation results
5.3.8 Other results
5.4 Classification results
5.5 Discussion
5.6 Conclusions
5.7 Contribution
6 RQ2: Relationship between ME-type and time of the day in relation to cognitive performance
6.1 Introduction
6.2 Experimental design
6.2.1 Experimental platform
6.2.2 Questionnaires
6.2.3 Scenario and experimental task
6.2.4 Participants
6.2.5 Interaction time
6.3 Results
6.3.1 Stroop game
6.3.2 Integer Matrix
6.3.3 Decimal Matrix
6.4 Discussion and the message to take home
6.5 Conclusion
6.6 My contribution
7 RQ3: Influence of Empathy, Emotional Intelligence and Fight/Flight system in HRI
7.1 Introduction
7.2 Experimental design
7.2.1 Robotic platform and sensors
7.2.2 Questionnaires
7.2.3 Participants
7.2.4 Scenario
7.2.5 Hypothesis
7.3 Data extraction and analysis
7.3.1 GSR
7.3.2 Facial Temperature variation
7.4 Results
7.4.1 Panas questionnaire
7.4.2 GSR
7.4.3 Facial Temperature variation
7.5 Discussion
7.6 Conclusion
7.7 Contribution
8 Assistive applications (I)
8.1 Introduction
8.2 Related work
8.3 The ENRICHME project
8.3.1 Robotic system
8.3.2 Graphical User Interface
8.4 Lessons learnt from a first interaction with the elderly
8.4.1 Scenario
8.4.2 Interaction, data recording and analysis
8.4.3 Thermal data
8.4.4 Results and comments
8.4.5 First lessons learnt
8.4.6 Discussion
8.4.7 Conclusion
8.5 Results of a 5-day interaction scenario designed for the elderly: a pilot study
8.5.1 Scenario
8.5.2 Participant description
8.5.3 Physiological data extraction and analysis
8.5.4 Hypotheses
8.5.5 Results
PANAS
Digit Cancellation
Integer Matrix Task
Stroop game
Hypothesis H1
Hypothesis H2
Other results
8.5.6 Discussion
8.6 ENRICHME project testing phase
8.6.1 RQ1: Which is the most used GUI application?
8.6.2 RQ2: Which is the most played cognitive game?
8.6.3 Performance analysis
8.6.4 Discussion
8.7 Conclusion
8.8 Contribution
9 Assistive applications (II)
9.1 Introduction
9.2 Literature review
9.2.1 Insomnia and cognitive performance
9.3 Experimental design
9.3.1 Questionnaires
9.3.2 Participants
9.3.3 Scenario
9.3.4 Data recording and analysis
9.4 Results
9.4.1 Performance results CPT Task Integer Matrix Task Stroop Task
Discussion
9.4.2 Performance results for individuals with insomnia CPT Task
Integer Matrix task
Stroop Task
9.4.3 Physiological response analysis CPT Task Integer Matrix Task Stroop Task
9.5 Discussion
9.6 Conclusions
9.7 Contribution
10 Conclusion
10.1 General considerations
10.2 Thesis summary
10.2.1 Chapter 2: Related work
10.2.2 Chapter 3: Experimental Platforms
10.2.3 Chapter 4: Methodology
10.2.4 Chapter 5: RQ1 Relationship between cognitive performance and physiological response
10.2.5 Chapter 6: RQ2 Relationship between ME-type and time of the day in relation to cognitive performance
10.2.6 Chapter 7: RQ3 Influence of empathy, emotional intelligence and fight/flight system in HRI
10.2.7 Chapter 8: Assistive applications (I)
10.2.8 Chapter 9: Assistive applications (II)
10.3 Perspectives
A List of Publications
B Morningness Eveningness Questionnaire
C Regulatory Focus Questionnaire – proverb form
D Eysenck Personality Questionnaire
E Reinforcement Sensitivity Theory Personality Questionnaire
Bibliography
