Systematic Review: Emotion Recognition Based on Electrophysiological Patterns for Emotion Regulation Detection

Abstract

The electrophysiological basis of emotion regulation (ER) has gained increased attention since efficient emotion recognition and ER allow humans to develop high emotional intelligence. However, no methodological standardization has been established yet. Therefore, this paper aims to provide a critical systematic review to identify experimental methodologies that evoke emotions and record, analyze and link electrophysiological signals with emotional experience by statistics and artificial intelligence, and lastly, define a clear application of assessing emotion processing. A total of 42 articles were selected after a search based on six scientific browsers: Web of Science, EBSCO, PubMed, Scopus, ProQuest and ScienceDirect during the first semester of 2020. Studies were included if (1) electrophysiological signals recorded on human subjects were correlated with emotional recognition and/or regulation; (2) statistical models, machine or deep learning methods based on electrophysiological signals were used to analyze data. Studies were excluded if they met one or more of the following criteria: (1) emotions were not described in terms of continuous dimensions (valence and arousal) or by discrete variables, (2) a control group or neutral state was not implemented, and (3) results were not obtained from a previous experimental paradigm that aimed to elicit emotions. There was no distinction in the selection whether the participants presented a pathological or non-pathological condition, but the condition of subjects must have been efficiently detailed for the study to be included. The risk of bias was limited by extracting and organizing information on spreadsheets and participating in discussions between the authors. However, the data size selection, such as the sample size, was not considered, leading to bias in the validity of the analysis. This systematic review is presented as a consulting source to accelerate the development of neuroengineering-based systems to regulate the trajectory of emotional experiences early on.

1. Introduction

Emotions have played a crucial evolutionary factor in the history of humanity since they allow us to adapt to the environment for surviving [1]. Emotions are a complex neural and hormonal set of interactions generated by different external and internal stimuli that control the conduct of human beings [2]. This psychophysiological process influences many aspects of our daily life, such as spontaneous decision-making, communication, and learning [3]. The lack of emotional control can lead to bad decisions along with serious consequences. Maladaptive emotional skills may lead to social conflicts, failures or even losses [4]. Over time, an inefficient emotion regulation (ER) strategy constitutes an important risk factor in developing post-traumatic stress disorder [5] or social anxiety [6].

Human beings can improve their ER process to increase their quality of life. Namely, if human beings learn how to modify the type, intensity, time course and quality of emotional responses, they will eventually be able to efficiently identify, understand, express, regulate and use their own emotions and those of others, what is well known as emotion intelligence (EI) [7]. Individuals who have developed effective ER strategies have high EI and higher quality of life. According to World Economic Forum, EI is the sixth desired skill in the top ten list of the Fourth Industrial Revolution 2020 since employees with high EI possess emotional self-awareness, emotional self-control, motivation, positive outlook, empathy, and conflict management [8]. At present, 90% of top performers in the world have higher EI than average employees. Furthermore, people with higher EI tend to have higher incomes, and in general, EI is responsible for 58% of everyone’s job performance. In addition, individuals with high EI are mentally and physically healthier. They have:

• less mood deterioration;
• less cortisol secretion in response to the stressor;
• less prolonged arousal in response to negative situations;
• are less likely to suffer from chronic arousal on physical health, including coronary heart disease, gastrointestinal disorders, asthma, psoriasis, and migraine;
• less at risk for substance-use-related health problems such as cirrhosis of the liver, pancreatitis, polyneuropathy;
• better quality and more refreshing sleep [9].

The term “EI” was popularized by the book titled “Emotional Intelligence” by Daniel Goleman, published in 1995 [10], although the pioneers who stipulated the roots of this kind of intelligence were Peter Salovey and John Mayer [11]. They defined it as the subset of social skills that allow the monitoring of emotions in order to recognize them and be able to adapt our own behavior to the social context. They considered that this ability is conformed of four branches: (1) perceiving emotions, (2) using emotions to facilitate thought, (3) understanding emotions, and (4) managing emotions in a way that enhances personal growth and social relations.

EI is multidimensional, and each skill should be analyzed with different standards and scales. Thus, the idea of creating an EI measure is a complicated task [12]. As emotional cognition and behaviors are the results of entangled central and autonomous nervous system activities, the analysis of electrophysiological correlates has been extensively used to study emotional processing [13,14,15,16,17]. Electroencephalography (EEG), electrocardiography (ECG), electrodermal activity (EDA) and electrooculography (EOG) are the most frequently used electrophysiological signals to describe objective responses during emotional cognitive processes.

Many reviews have focused on emotion recognition and classification based on electrophysiological signals. For instance, Liberati et al. selected studies that aimed to identify affective states from electrophysiological signals and emphasized their application on brain–computer interfaces (BCIs) [18]. Usually, in clinical interventions, an optimal BCI records and analyzes electrophysiological activity in real time to provide feedback to the system user, including emotional processing. Nevertheless, a lack of such application has been stressed due to low performances, and still needs to be investigated. Al-Nafjan et al. [19] reviewed published articles about emotion detection, recognition, classification, current and future trends of BCI technologies. They concluded that it is necessary to find more reliable neuro-patterns to standardize the analysis of affective states in a BCI. Also, they emphasized the use of different algorithms to test the validity of results [19]. Dryman et al. [20] conducted a systematic review about expressive suppression and cognitive reappraisal in cases of social anxiety and depression, indicating the need of long-term electrophysiological signal acquisition for a better understanding of these conditions.

As far as we are concerned, this study is the first to review the use of EI electrophysiological correlates in artificial intelligence developments. As was previously mentioned, EI has not been studied as one single concept. Therefore, this systematic review is the first attempt to categorize studies that are orientated to the EI dimensions, considering the following four key concepts: (1) emotional processing, (2) emotional identification based on electrophysiological signals, (3) ER and (4) emotional responses. The present systematic review aims to identify research gaps in the recent advances in electrophysiological pattern recognition during emotion processing to accelerate the development of neuroengineering-based systems to develop ER strategies that allow modifying and regulating the trajectory of emotional experiences early on.

2. Methods

The research was conducted in accordance with the PRISMA guidelines [21,22]. The protocol began with electronic searches of studies published in English between 2014 and 2020 in the following databases: Web of Science, EBSCO, PubMed, Scopus, ProQuest and ScienceDirect. The following keywords were used to search in all the databases: emotion, affective, recognition, regulation, electroencephalography, electrocardiography, heart rate, skin temperature, galvanic skin response, EOG and EEG. The Boolean operators AND and OR were used. Parentheses were used in ProQuest to limit search results due to the large number of studies found. The query design is detailed in

Table 1. Query design.

Index	Web of Science, EBSCO, PubMed and ScienceDirect	ProQuest
1	emotion OR affective AND recognition OR regulation AND electroencephalography	(emotion OR affective) AND (recognition OR regulation) AND electroencephalography
2	emotion OR affective AND recognition OR regulation AND electrocardiography OR heart rate	(emotion OR affective) AND (recognition OR regulation) AND (electrocardiography OR heart rate)
3	emotion OR affective AND recognition OR regulation AND skin temperature OR galvanic skin response	(emotion OR affective) AND (recognition OR regulation) AND (skin temperature) OR (galvanic skin response)
4	emotion OR affective AND recognition OR regulation AND electrooculography	(emotion OR affective) AND (recognition OR regulation) AND electrooculography
5	emotional intelligence AND electroencephalography	emotional intelligence AND electroencephalography
6	emotional intelligence AND electrocardiography	emotional intelligence AND electrocardiography
7	emotional intelligence AND skin temperature	emotional intelligence AND skin temperature
8	emotional intelligence AND galvanic skin response	emotional intelligence AND galvanic skin response
9	emotional intelligence AND electrooculography	emotional intelligence AND electrooculography

.

Pilot studies, reviews, meta-analyses, commentaries, book chapters, conference papers, master and doctoral dissertations, workshop descriptions and unpublished data were not included in this review. The eligibility criteria for paper selection are detailed hereunder, and the selection process is illustrated in Figure 1. Search within databases was carried out during the first two weeks of April 2020, and the selection process was carried out from April 2020 to July 2020.

The main purpose of this review is to comprehend the state-of-the-art experimental methodologies that analyze different dimensions of emotional intelligence. Using this information, it is possible to propose an original methodology to assess emotional intelligence as a core. For this reason, the outcomes that were sought were:

• Techniques for eliciting emotions: identify the strategies, materials, conditions, and environments that have been applied to evoke emotions.
• Quantitative and/or Qualitative Assessments: differentiate the tests based on objective and subjective measurements and highlight the necessary parameters to study EI.
• Data analysis: analyze the relation between electrophysiological signals, psychometric tests, and the evoked emotions.
• Applications: detect targets and reasons for evaluating some dimensions of emotional intelligence.
  Data outcomes were extracted at the last stage of the selection process.
  Data extraction was conducted to answer the following research questions (RQ):

  ▪

  RQ1: Which electrophysiological signals may index emotional processing?

  ▪

  RQ2: Which stimuli may be used to elicit emotions?

  ▪

  RQ3: How should psychometrics assess emotional experience?

  ▪

  RQ4: Under which conditions may emotion processing be assessed, and how does the experimental paradigm cause biases in emotional perception?

  ▪

  RQ5: How may correlates of emotion processing be applied to models of statistical analysis and artificial intelligence?

  ▪

  RQ6: What are the applications and impacts of assessing emotion processing?

Therefore, the following eligibility criteria were applied to consider studies within the review. Figure 2 details the elaboration process based on the PRISMA methodology.

2.1. Type of Studies

The literature was selected according to two inclusion criteria: (1) studies focused on the assessment of electrophysiological signals that correlate to EI: emotional stimuli processing and/or emotions recognition and/or regulation and/or empathy; (2) works centered on the classification of emotions by using machine or deep learning methods based on electrophysiological signals.

Concerning exclusion criteria, the following three key points were considered. First, studies that did not elicit emotions were discarded. Second, we excluded studies that did not mention the experimental paradigm that aimed to elicit emotions because we would not be sure about how they conducted it. Finally, studies in which emotions were not described in terms of continuous dimensions (valence and arousal) or by discrete variables (e.g., anger, fear) were excluded.

2.2. Exposures

Studies were selected if the peripheral and/or central nervous activity of participants were monitored while participants were processing emotional stimuli. Authors must have measured at least one of the following electrophysiological signals: EEG, ECG, EDA and EOG.

2.3. Participants

The scope of this review was limited to human subjects. The inclusion criteria embrace (1) studies which dealt with pathological or non-pathological participants; (2) studies in which participants suffering from any pathological medical condition received a prior adequate diagnosis following the Diagnostic and Statistical Manual of Mental Disorders (DSM) [23] or the International Statistical Classification of Diseases and Related Health Problems (ICD) [24] guidelines; and (3) studies focused on non-pathological samples were included if it was specified that participants were healthy, namely, without psychiatric illness antecedent that could impact emotion processing and electrophysiological signals. The exclusion criteria regarding those studies were not rigorously controlled in terms of demographic characteristics such as age and gender.

2.4. Comparators

Studies focused on pathological conditions were excluded if they did not consider an appropriate control group (e.g., matched healthy participants). Works centered on non-pathological samples were excluded if they did not consider an appropriate control condition (e.g., neutral stimuli processing).

2.5. Study Records

The study records were managed using Google Sheets and Mendeley software. From each selected study, the following information was obtained and compared: author, year, objective, the studied emotions, electrophysiological signals, psychometric tests, emotion induction (e.g., words, pictures, sounds), signal processing, statistical analysis, classification methods, characteristics of participants and applications for medical and nonmedical areas.

The electrophysiological correlates of emotions processing were characterized into central and peripheral neuronal activity according to emotional characteristics (e.g., pleasant, unpleasant). Results and conclusions obtained by each electrophysiological method were compiled and compared to depict a general overview of emotion processing and management by the human nervous system. In the case of data coming from publicly available databases, brief research, and description were conducted to obtain information about data recording conditions and emotions induced. Ways of eliciting and assessing emotions and EI by psychometric tests or questionnaires were synthesized to draw a general overview of the current methodology. Finally, a focus on experimental procedures was achieved (e.g., task for participants).

The main outcomes during the research were the electrophysiological signal recording techniques and their application for EI measurement, ER, recognition and classification. The secondary outcomes were experimental conditions; specific features of electrophysiological signals related to emotions (e.g., Event Related Potentials (ERP)); stimuli used in emotion induction studies (e.g., pictures from the International Affective Picture System (IAPS), sounds from International Affective Digitized Sounds (IADS)); psychometric tests implemented for emotion scaling and rating (e.g., Self-Assessment Manikin (SAM), Likert scale); electrophysiological activity databases (e.g., Database for Emotion Analysis using Physiological Signals (DEAP), SJTU Emotion EEG Dataset (SEED) used for emotion classification and artificial intelligence algorithms, such as machine learning and subsets for emotion classification).

3. Results

A total of 42 studies were selected for this review. A summary of the selection process is presented in Figure 1. The methodological concerns on electrophysiological and psychometric patterns of emotion processing are described below and have been synthetized in Figure 3. Finally,

Table 2. Main characteristics of the studies included in the review.

Study	Number of Subjects		Pathology	Elicitation Method	Emotion	Psychometric Test	Electrophysiological Signal	Statistical Analysis	Classification Method	Objective
	Healthy	Pathological
[25]	29			Picture	Sadness	SAM	EDA, EEG	ANOVA and bivariate correlation analyses		Compare the effects of three emotion regulation strategies: reappraisal, acceptance, and suppression
[26]	32 (DEAP); 15 (SEED)			Videos	Positive, negative, and calm (DEAP); positive, neutral, and negative (SEED)		EEG		Decision Tree, KNN and RF	Emotions classification according to the time variation of emotion processing
[27]	26			Videos	Positive, neutral, and negative	Ad hoc: Valence and/or arousal	EDA, ECG		SVM	Analyze autonomic control mechanisms and functional assessment of emotional responses of human
[28]	27	28	Schizophrenia	Pictures	Positive, neutral, and negative		EEG	ANOVA		Study motion processing in schizophrenia
[29]	39			Videos	Amusement, anger, fear, tenderness, and a neutral state	Ad hoc: Theory of mind	ECG, GSR	ANOVA and Bonferroni		Understand the physiology of socio-emotional processes in the cinema
[30]	28	68	Borderline personality disorder and post-traumatic stress disorder	Pictures	Negative, positive, and neutral	SAM	ECG	ANOVA and Tukey’s HSD test		Analyze HRV during a cognitive reappraisal task in female patients with borderline personality disorder
[31]	34			Pictures	Negative (mild, and high intensity), neutral,	Ad hoc: boredom and engagement	EDA	Linear Mixed-Effect Modelling		Understand the difference between the emotional response toward real and fictional pictures
[32]	44			Pictures and videos	Positive, negative, and neutral		EDA, EEG		LR, RIPPER, MLP	Validation of EEG activity as a good indicator of self-regulation
[33]	16	13	Moebius syndrome	Video	Disgust, surprise, anger, happiness and neutral		ECG	ANOVA		Study the alterations in the processing of facial expression of emotions due to congenital inability to produce facial expressions
[34]		42	Low interdependent Self construal (SC) and high interdependent SC	Pictures	Unpleasant and neutral		EEG	Random Effects Models		Asses the ability of emotion suppression
[35]	26			Pictures	High-arousing positive valence, low-arousing positive valence, high-arousing negative valence, and low-arousing negative valence		EEG	ANOVA		Explore changes in cognitive-motor performance in response to emotional stimuli
[36]	139	123	Bipolar disorder	Pictures and sounds	Disgust, erotica, fear, happiness, neutral and sadness	Ad hoc: Valence and/or arousal	ECG, EOG	ANOVA		Study modes of emotional regulation according to type of bipolar disorder
[37]	38	28	Complex post-traumatic stress disorder and complex dissociative disorders	Pictures	Unpleasant and neutral	SAM	EEG	ANOVA and t-test		Examine the effects of trauma treatment in symptoms and the neural networks involved in emotional control
[38]	69	61	Attention-deficit/hyperactivity disorder	Pictures	Happiness, fear and neutral		EEG	ANOVA		Examine ADHD-related differences in attention to emotional and neutral stimuli
[39]	30		Autism spectrum disorder	Pictures	Anger (mild and extreme), happiness and sadness		EEG	ANOVA		Research fathers of children with autism in facial emotion detection
[40]		40	Social anxiety disorder	Pictures	Negative and neutral	Ad hoc: Discomfort	EEG	ANOVA and t-test		Utilize manifold-learning to understand EEG brain dynamics associated with emotion regulation processes
[41]	31	51	Social anxiety disorder	Pictures	Negative and neutral		EEG	ANOVA and t-test		Study response to negative images in individuals with SAD and HC during emotion reactivity and reappraisal
[42]	10	10	Asperger’s syndrome	Pictures	Angry, happy, and neutral	Ad hoc: Anger and happiness	EEG	MANOVA and F-test		Investigate EEG oscillatory activity and phase-synchronization during visual recognition of emotional faces in Asperger’s syndrome patients and healthy controls
[43]		30	Drug-resistant temporal lobe epilepsy	Videos	Fear, disgust, sadness, happiness, and peacefulness		SCR	ANOVA, t-test and χ2 test		Understand the impact of biofeedback on seizure control and emotional regulation
[44]		90	Alcohol use disorder	Pictures	Positive, negative, and neutral valence	SAM	ECG	ANOVA, post hoc Bonferroni test.		Compare HF-HRV in response to emotional and neutral stimuli in two groups of alcohol use disorder abstinent patients, according to their length of abstinence
[45]	49			Words	Negative and neutral	Ad hoc: discomfort	EEG	ANOVA, t-test		Examine the neural correlates of emotional reactivity and regulation to idiographic information
[46]	36			Videos	Sadness	Ad hoc: Specific emotion	GSR, ECG	ANOVA		Study if women’s efforts and success at using cognitive reappraisal to regulate their emotions would be affected by the menstrual cycle and neuroticism levels
[47]		44	Anorexia	Pictures and Words	Neutral, happiness, sadness, fear, and angry		EEG	ANOVA and t-tests		Explore the neurophysiological correlates of emotional face perception and recognition in adolescent AN patient using ERPs
[48]	117			Videos	Amusement, sadness, and anger	Brief Differential Emotions Scale	ECG	Pearson’s and Spearman’s correlations		Asses if expression of different emotions predict different indices of physical health
[49]		50	Depression	Pictures and words	Neutral and unpleasant		EEG	ANOVA		Understand changes in emotion during controlled processing of different semantic representations
[50]	20	20	Anterior cruciate ligament reconstruction patients	Pictures	Neutral and fear	SAM	EEG and ECG	ANOVA		Identify how negative emotional stimuli affect neural processing in the brain and muscle coordination in patients after anterior cruciate ligament reconstruction
[51]	136			Videos	Fear		EEG	ANOVA		Assess emotion-regulation strategy during viewing of a fear-inducing film clip
[52]	31			Pictures and words	Neutral and negative	SAM	EEG	ANOVA		Study emotion regulation with picture and word stimuli
[53]	31			Pictures	Neutral and negative	SAM	EEG	ANOVA		Investigate how stimulus arousal affects reappraisal success
[54]	24			Pictures	Neutral, positive, and negative	SAM	EDA	Spearman’s correlation χ2 test		Compare the thermal reactivity to subjective and electrodermal responses
[55]	26	26	Schizophrenia	Pictures	Neutral and negative	SAM	EEG	ANOVA		Assess regulation of negative emotion in individuals with high schizotypal traits
[56]	96			Pictures	Neutral and negative	Ad hoc: discomfort	EEG	Pearson’s bivariate		Evaluate spontaneous emotion regulation by EEG activity
[57]	18			Pictures	Neutral and negative	SAM	EEG	ANOVA		Study the relation between distraction as an emotion regulation strategy and emotion generation
[58]	42			Words	Positive, neutral, and negative		EEG	ANOVA		Explore whether extraversion and neuroticism influence the processing of positive, neutral, and negative words.
[59]	10			Videos	Positive, neutral, and negative	Ad hoc: Valence and/or arousal	EEG		SVM	Investigate environmental psychological perception in adolescents
[60]	9	18	Schizophrenia	Pictures	Positive, neutral, and negative	SAM	EEG	ANOVA		Study emotion processing in the brain before and after emotional neurofeedback
[61]	229			Pictures	Sadness, anger, happiness, and neutral	Ad hoc: Valence and/or arousal	ECG and EDA	Mixed-Effect Modelling		Examine the relations of ANS activity in the parasympathetic nervous system and sympathetic nervous system with brain activity during emotional face processing in adolescents
[62]	24	16	Anorexia nervosa	Videos	Negative		ECG	Linear regression		Explore changes in HRV during and after negative emotional induction in patients suffering from restrictive type anorexia nervosa
[63]	72			Pictures	Pain		EEG	ANOVA and Bonferroni correction		Examine age-related changes in response to the perception of another’s distress or pain from early to middle childhood
[64]	17	16	Autism spectrum disorder	Pictures	Pain	Ad hoc: discomfort	EDA	Bayesian inference		Study the link between autonomic, cortical, and socio-emotional abnormalities in autism spectrum disorder ASD
[65]	40	20	Empathy deficit disorder	Pictures	Positive, neutral, and negative	Ad hoc: Valence and/or arousal	EEG	ANOVA		Analyze the emotional processing in Colombian ex-combatants with different empathy profiles
[66]	41	Healthy	Healthy	Videos	Positive, neutral, and negative		EEG	ANOVA		Identify potential behavioral and neural correlates of EI

SAM: Self-Assessment Manikin; EDA: Electrodermal Activity; EEG: Electroencephalography; ANOVA: Analysis of Variance; DEAP: Database for Emotion Analysis using Physiological Signals; SEED: SJTU Emotion EEG Dataset; KNN: K-Nearest Neighbors; RF: Random Forest; SVM: Support Vector Machines; GSR: Galvanic Skin Response; LR: Logistic Regression; RIPPER: Repeated Incremental Pruning to Produce Error Reduction; MLP: Multilayer Perceptron; HRV: Heart Rate Variability; EOG: Electrooculography; SCR: Skin Conductance Response; ERP: Event-Related Potentials; EI: Emotional Intelligence.

details the main characteristics of every study.

3.1. RQ1: Which Electrophysiological Signals May Index Emotional Processing?

To monitor human body responses related to emotion processing, several electrophysiological recording techniques have been used. EEG was the preferred method, reported in 27 studies [25,26,28,32,34,37,38,39,40,41,42,45,47,49,50,51,52,53,55,56,57,58,59,60,63,65,66]. ECG was used in 11 studies [27,29,30,33,36,44,46,48,50,61,62]. EDA was reported in 10 studies [25,27,29,31,32,43,46,54,61,64], and only one study used EOG [36]. From these studies, four of them fused ECG with EDA [27,29,46,61], two of them fused EEG with EDA [25,32], one of them fused EEG and ECG [50], and the remaining one fused ECG with EOG [36].

Table 3. Electrophysiological signals, methods and features analyzed for correlation with emotional states.

Electrophysiological Technique	Analysis Method	Analyzed Feature	Emotional Correlation
EEG	Spontaneous activity in frequency domain	Delta	High activity in schizophrenia patients
		Alpha	High band power and high recovery from discomfort
		Beta	High activity in fathers of autistic children
			Inhibitory control
		Theta	High activity with fearful pictures
			High activity in SAD participants
			Higher activity in female than male
	Evoked activity based on ERPs	P1	Larger amplitude with positive stimuli than neutral in children with ADHD
		P2 N2	Higher amplitudes for high arousal and negative valence stimuli in healthy participants
		N4	Absence in Asperger participants
			Less N400 in Neurotic participants
		EPN	The less prominent the EPN, the better the sad and afraid states were recognized
		LPP	Amplitude reduction in reappraisal conditions
			Amplitude reduction in suppression strategy (marker depression and East-Asian descendants)
			Increase with acceptance regulation
			Increments marker for SAD and schizophrenic participants, and low empathy levels
ECG	Time Domain	HR	Decrease in fear (anterior cruciate ligament reconstruction)
			Decrease while watching tender scenes
		RSA	No effect
	Time and Frequency domain	HRV	Lower high frequency component in positive, negative, and neutral stimuli (BPD+PTSD). Decrease after negative stimuli (anorexia nervosa)
EDA	Phasic activity	SCR	Increase in stimuli with high arousal, anger, fear, pain, and stress
			Decrease in pain stimuli (autism)
	Tonic activity	SCL	Increase in anger
EOG	Global velocity	Eyeball movement	Increase in happiness and sadness stimuli after disgust stimulus
			Increase in disgust stimulus after neutral stimulus

summarizes electrophysiological signals, methods and features analyzed for correlation with emotional states.

In general, the experimental setups of the studies were based on the technical configurations summarized in

Table 4. Technical information about the recording of electrophysiological signals: most frequent parameters and devices. BW: Bandwidth, Fs: Sampling frequency.

	Electrode Montage	Sampling Conditions [Hz]	Recording Systems	Computer Software	Signal Conditioning
EEG	10–132 channels according to 10/20 International System	Fs = 250–2048	Brain Vision	EEGLAB	Z < 5–60 kΩ
			Quik-Cap128 NSL	ERPLAB	Notch filtering at 50 Hz
			Emotiv EPOC	Neuroscan 4.4	Lowpass filtering at 100 and 134 Hz
			Neuron-Spectrum-1	BrainVision Analyzer	Lowpass fifth order sinc filter with a half-power cut-off at 204 Hz
			BrainVision PyCorder	Curry 7 and Neuroscan 4.5	Analogue filters were at 0.05 and 100 Hz
			V-AMP	BrainVision Analyzer 2.0 software
			g.Hlamp	Net Station, Version 4.2 software
			NuAmps	Net StationDense Array EEG
			NeuroScan
		BW = 0.05–100	BioSemi Active Two system
			Nu AmpsNeuroScan
			NeurOne
			Neuronic Medicid
			SynAmps
			actiCAP Brain Products Inc.
			Brain Products GmbH
			NeuroScan Synamp2
			HydroCel Geodesic Sensor Net
			One Hydro-Cel Geodesic Sensor Net
ECG	1–3 Ag/AgCl electrodes, electrodes in any or combinations of the following areas: arms, legs, Einthoven’s triangle, shoulders, hip, chest, wrists, clavicle, rib, wrists, sternum, abdomen	Fs = 4–2000	BIOPAC MP150 (ECG100C)	AcqKnowledge	Amplifier Gain: 2000
			Powerlab and OctalBioAmp8/30	Kubios HRV	Mode: Normal
			Biosemi Active Two system	LabView	Notch: 50 Hz
		BW= 0.05–100	Biopac fMRI compatible wireless signal logging	Mindware HRV	Band-pass: 0.5–100 Hz
				LabChart
				CMetX
EDA	Bipolar Ag/AgCl or dry-nickel plated electrodes in any or combinations of the following areas: index finger ring finger middle finger big toe second toe Dominant or non-dominant hand	Fs = 500–2000	Empatica	Acqknowledge	Amplifier gain: 5 μOhms/V, 10 μS
			BIOPAC MP150 (GSR100C)	SCRalyze
			BIOPAC MP35 (EDA100C)	Mindware EDA	High-pass filter: DC
		BW = 0.159–10	Biograph	PsPM	Low-pass filter: 10 Hz
			PowerlabNeurOne	Brain Vision Analyzer 2.1.2	Butterworth band-pass filter: 0.159–5 Hz
					Impedance level: <10 kW
EOG	Two Ag/AgC1 electrodes, placed on the outer canthi of both eyes	Fs = 1000	BIOPAC MP150 (EOG100C)	AcqKnowledge	Amplifier Gain: 2000
					Mode: Normal
		BW = 0.05–100			Notch: 50 Hz
					Band-pass: 0.05–100 Hz

.

3.1.1. EEG

Different methods were identified to analyze EEG activity. The first one focused on five typical frequency bands: delta (0–3 Hz), theta (4–7 Hz), alpha (8–12 Hz), beta (13–20 Hz) and gamma (>20 Hz). Most studies concluded that lower frequencies were effectively associated with emotional processing and regulation.

For instance, the lowest frequency range, delta, was observed to be prominent in schizophrenic patients during ER tasks [60]. Theta frequency showed an increase in activity in frontal and parietal regions with fearful pictures during the induction of emotional responses [50]. Besides, this trend was also found in ER processes, and it was used as a marker to characterize Social Anxiety Disorder patients [40]. Moreover, this increase in activity was visible in reappraisal (ER strategy) conditions to differentiate between gender, in which women tended to show higher activity than men [51]. However, this trend was not only visualized in theta frequency. Beta and theta frequencies also showed higher relative power spectra during ER tasks on fathers of autistic children [39]. Furthermore, it was proposed that beta frequency range could be an indicator of emotional, attentional, and cognitive involvement in the stimulus, and was associated with response inhibition in ER therapies [37]. On the other hand, it was identified that a weaker low beta event-related desynchronization occurred with lower psychosocial functioning in patients with schizophrenia [28]. Finally, a great alpha band power determined a quick recovery from discomfort status during ER [56].

The other type of analysis was the neural response time-locked to the event of interest, also known as Event-Related Potentials (ERP) [38]. Uusberg et al. [57] determined that there was no correlation between frequency dynamics (theta) and ERP components [57]. Many of the reviewed works only studied ERP components that are sensitive to emotions. Figure 4 summarizes the ERP associated with emotion processing and regulation observed in the reviewed studies.

Most of the studies that analyzed early potentials were associated with emotional response or recognition. For instance, the P1 component showed a large amplitude in response to positive stimuli compared to neutral stimuli in children with ADHD [38]. On the other hand, P2 and N2 presented higher amplitudes for high arousal and negative valence stimuli in healthy participants [47].

Usually, the Late Positive Potential (LPP) was employed to assess emotional processing during ER. For instance, Grecucci et al. concluded that LPP can be modulated by reappraisal strategy and observed a drastic reduction of LPP during the distancing versus attend condition [52]. Furthermore, Speed et al. determined that the LPP amplitude decreased after reappraisal training, indicating that cognitive reappraisal was an effective strategy to reduce the neural response to negative stimuli [45]. This trend was also observable in participants with depression. However, there was an important difference: the employed strategy. Those participants tended to suppress their emotions to avoid negative reactions after eliciting negative stimuli [49]. This strategy was also used as a marker to differentiate ethnicities. Suppressing emotions was used more often in interdependent self-construal East Asian descendants than European American descendants [34]. As it was previously mentioned, this potential was a marker that assesses the effectiveness of ER strategies. Boheme et al. observed a decrease in LPP amplitude for reappraisal, whereas, for acceptance, the LPP amplitude increased [25]. Moreover, the increased LPP amplitude determined electrophysiological markers of mental disorders, such as Social Anxiety Disorder (SAD) [41] and schizophrenia [55], and was also a marker for empathy (i.e., the higher the LPP, the lower the empathy level [63,65]).

3.1.2. ECG

ECG is an electrophysiological measurement used to record the electrical activity of the heart caused by the repolarization of ventricular walls and activation of the atriums. This activity is recorded by placing electrodes on specific areas of the body [14]. The features that were mostly extracted were the amplitude of Respiratory Sinus Arrythmia (RSA) [29,33] and the inter-beat R-R wave interval [36,50,61]. In a study concerning R-R intervals, data in the frequency domain was extracted using the fast Fourier and Wavelet transform [62]. The ratio between EDA and integration of the spectrum of R-R intervals was reported in [27]. High-frequency variations in beat-to-beat intervals was also reported [30]. Only one study reported computing the root mean square of successive differences of Heart Rate Variability (HRV) [48].

A contingent trend between cardiac activity and emotions was observed. An increase in HR after stimulus was associated with fear in control groups [50] and population with conditions such as bipolar disorder type I [36] was observed. Patients with anterior cruciate ligament reconstruction showed decreased HR [50]. Lower values of HR were also observed in control subjects while watching scenes inducing tenderness [29]. The ability to enhance sad emotion was associated with higher HRV [48]. One study reported no significant effect of emotional stimuli on RSA [33]. Patients with borderline personality disorder and post-traumatic stress disorder showed lower values of the high frequency component in HRV in positive, negative, and neutral stimulations [30]. A decrease in the wavelet transformation of high frequency component in HRV after negative stimuli was observed in patients with anorexia nervosa, whereas an increase was observed in healthy control subjects [62].

3.1.3. EDA

The EDA or Galvanic Skin Response (GSR) is the variation of the electrical properties of the skin in response to sweat secretion [67]. Two studies simultaneously analyzed phasic and tonic components of GSR: the Skin Conductance Response (SCR) and the Skin Conductance Level (SCL), respectively [32,46], whereas most studies only considered the SCR amplitude and peak detection [25,27,31,43,54,61,64]. On the other hand, only one study analyzed the SCL [29]. The main tendency was that with greater arousal, either pleasant or unpleasant, an increased SCR was generally shown [31,43,54,64]. Flat responses or small changes in GSR were observed in population with higher self-regulation skills [25,32,46]. Negative stimuli, such as those that induce anger, fear, pain, and stress, showed increased SCR [31,43,61,64]. Increased SCL was reported while watching anger stimuli [29]. Only Gu et al. showed an overall decreased SCR in autistic population processing stimuli inducing pain [64].

3.1.4. EOG

EOG is a technique used to record the electrical activity of eyeball movement. This method was reported in only one study. Extraction of the global velocity of eyeball movement was performed by computing the absolute values of the EOG derivatives [36]. In this study, increased eyeball movement was observed in patients with bipolar disorder type I after perceiving stimuli inducing emotions, such as happiness, disgust, and sadness, in comparison with bipolar disorder type II and control subjects. Stimuli associated with happiness and sadness increased eyeball movements when presented after a stimulus associated with disgust. Furthermore, disgust stimuli increased eyeball movements when presented after a neutral stimulus.

3.1.5. Databases of Electrophysiological Data

To analyze the relation between emotions and electrophysiological signals, most researchers collected data from an in-home sample. However, only one work [26] considered the use of SEED [68] and DEAP [69] databases for its study.

For emotion pattern recognition based on electrophysiological activity, two databases were found: SEED and DEAP. The former is a free EEG dataset developed by Shangai Jiao Tong University in 2015 [68]. It contains the electrocortical activity of 62 channels from 15 Chinese subjects (seven males, eight females) while they were watching film clips. These stimuli were 15 Chinese film clips and were categorized in positive, neutral, and negative emotions. The signals were recorded with an ESI NeuroScan System at 1000 Hz. On the other hand, DEAP includes electrophysiological data of 32 channels of EEG activity and 13 channels of peripheral physiological signals [69]. They were recorded from 32 participants (16 females) using a BioSemi ActiveTwo system at a sampling rate of 512 Hz. A total of 40 music videos were presented in 40 trials. At the end of each trial, participants rated each video in terms of arousal, valence, like/dislike, and dominance using the SAM [70].

3.2. RQ2: Which Stimuli May Be Used to Elicit Emotions?

3.2.1. Format and Nature of Stimuli

Pictures and images were the preferred stimulus type to induce emotional states. Besides, some studies used a combination of pictures with sounds [36], with words [52] or with videos [32]. Other studies used film clips [26,27,29,33,43,46,48,51,59,62] and/or music videos (DEAP dataset) [26,69]. Two studies conveyed emotions by presenting written words to the participants [45,58].

Twenty-nine studies included colored stimuli, whereas other studies preferred a greyscale [33,38,42,61] or black and white [44] presentation, and six studies did not mention the color [27,29,46,48,51,62].

As regards stimuli contents, three studies presented social scenes to the participants [63,64,65], five showed faces [33,38,39,42,47,61,66], and two studies presented a combination of social, non-social scenes and faces [28,37]. Three studies showed a combination of social and non-social scenes [31,50,53]. However, most studies did not specify any content.

Thirteen studies preferred to use in-home made stimuli [27,29,40,43,45,46,48,51,59,61,62,63,64], whereas all other studies included stimuli coming from publicly available databases.

3.2.2. Databases of Emotional Stimuli

To elicit emotions, twenty-eight studies used public and pre-validated datasets [25,28,30,31,32,33,34,35,36,37,38,39,41,42,44,45,47,49,50,52,53,54,55,56,57,58,60,66]. The IAPS was the main source of stimuli [25,28,30,31,32,34,35,36,37,41,44,49,50,52,53,54,55,56,57,60]. Some researchers complemented this dataset with some others. Particularly, Ma et al. complemented IAPS with IADS, and Grecucci et al. employed the Affective Norms for English Words (ANEW) [36,52]. Only two studies preferred to use the NimStim face stimulus set [33,38]. No other dataset was found to be repeated across reviewed studies.

3.3. RQ3: How Should Psychometrics Assess Emotional Experience?

3.3.1. Emotion Experience

From the revised literature, subjective assessments of emotion processing were performed according to seven categories (ad hoc tests): (1) valence and/or arousal [27,28,30,32,35,44,47,49,52,53,54,55,58,59,60,65], (2) discomfort [25,26,40,41,45,50,51,56,57,63,64], (3) specific emotion [46], (4) boredom and engagement [31], (5) continuum from angry to happy [42], (6) empathy [64] and (7) Theory of Mind [29,37]. Point-scale was the preferred measurement to assess valence and arousal perception: 100-point [29], 10-point [30], 9-point [25,26,35,46,50,52,53,54,56], 8-point [36], 7-point [31], 5-point [40,44,45,59,60,61] and 3-point [65]. SAM was the most popular test [25,30,37,44,50,52,53,54,55,57,60], followed by Visual Analogue Scales (VAS) [27,29,36]. Only one study used a continuous line with one mark in the middle [42].

3.3.2. Mental Health

The psychological health of participants was evaluated in half of the total reviewed studies. Anxiety disorder was tested in [25,30,35,37,40,41,43,61,62], and most studies used the State-Trait Anxiety Inventory (STAI) [25,30,43,62]. Depression was also tested in [25,30,36,43,49,58], preferentially using the Beck Depression Inventory [25,30,43,58].

3.3.3. EI- and ER-Related Monitoring

Two studies used psychometric tests related to EI per se. On the one hand, Balconi et al. employed the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT) [60], and on the other hand, Raz et al. worked with the Schutte self-report Emotional Intelligence Scale [66]. Empathy was tested by Gu et al. with the Toronto Alexithymia Scale and Empathy Quotient [64]. ER tests were used in two studies: one of them applied the Emotion Regulation Questionnaire [30], and the other one opted for the Difficulties in Emotion Regulation Scale [62].

3.4. RQ4: Under Which Conditions May Emotion Processing Be Assessed, and How Does the Experimental Paradigm Cause Biases in Emotional Perception?

3.4.1. Experimental Paradigms

Although the general aim of all the reviewed studies was to characterize emotion processing, conditions and procedures were highly variable. Therefore, it is essential to depict an outline of differences in task administration. Most studies were conducted in a laboratory setting, whereas only one study set up a real-life environment by conducting the experiment in a cinema [29]. Furthermore, most studies complemented electrophysiological objective measures with subjective assessments (as detailed in Section 3.3.1 although some did not [28,32,35,38,41,43,47,49,51,58,62,63,66].

3.4.2. Emotion Processing Mode

In most studies, emotion processing was explicit (i.e., the participant was asked to focus on the emotion conveyed). However, two studies included an implicit way of processing emotional information [56,58]. For instance, participants in one study had to focus on the discomfort induced by the emotion conveyed but not on the emotion per se [56]. Two studies included both ways of processing emotions [47,61].

3.4.3. Timing Protocol and Procedure

The time of stimulus presentation was highly variable across studies: 0.8 s, [28], 1 s [42,52,53,58], 1.5 s [47,65], 2 s [35], 2.5 s [63], 3 s [39,61,64], 4 s [25,30,33,34,37,54], 5 s [32,38,49,55,57,66], 6 s [30,31,36,41,45,50,56,60], 7 s [40], 8 s [36,50], 10 s [36,44], 15 s [32], 45 s [43], 1 min [26,48], 1.2 min [62], 1.5 to 2 min [27,29], 3 min [46], 3.5 min [51], 3 to 5 min [59] and 4 min [26]. Figure 5 details the presentation time according to stimulus types.

3.5. RQ5: How May Correlates of Emotion Processing Be Applied to Models of Statistical Analysis and Artificial Intelligence?

In biomedical areas, statistical modelling plays an important role throughout a study, from the planning stages to data interpretation. The latter is a crucial step that dictates the research outcomes. Therefore, an adequate selection of statistical tests clearly determines valid inferences and conclusions. To choose the optimal test, it is necessary to look for the data distribution, sample size, homogeneity of variances and/or covariances, and dependency between variables [71,72].

3.5.1. Statistical Analysis: Parametric Methods

Even though most studies did not specify the execution of tests for ensuring parametric conditions, most of them determined the use of parametric tests to analyze and correlate electrophysiological signals with emotions.

Table 5. Summary of parametric tests used in the revised literature. Variations of ANOVA were the most common statistical tests.

Test	Variation	Variables
ANOVA	Repeated measures	Stimuli effect on electrophysiological signals
		Hemisphere (Laterality or electrode position), regulation strategy or cognitive task
		Difference on emotional tasks (emotion recognition or regulation) considering electrophysiological signals between groups
		Associations with mental or physical issues (extraversion, neuroticism, and menstrual cycle phase)
	Two-way	Emotions and groups versus electrophysiological data
		Group and condition for ER
	Three-way	Group, electrode, and emotion stimuli vs electrophysiological data
	Four-way	Age, gender, stimuli, and electrode versus ERP
	Mixed design	Group differences and stimuli effect on electrophysiological signals
	MANOVA	Differences in tasks
	Mass univariate	Average occipital theta perturbations
	ANCOVA	Gender, regulation strategy and hemisphere
T-test	-	Emotion regulation and EEG frequency. Post hoc test: stimuli, task and time, groups, condition, and activity; electrophysiological data; groups; effectivity and time
Pearson’s test	-	Emotion regulation and electrophysiological data
F-Test	-	Emotion recognition conditions, emotions, factor time window and physiological data
Tukey’s HSD		Valence and emotion recognition vs electrophysiological data
		Comparisons between emotional conditions and electrophysiological data

summarizes the parametric tests used in the revised literature.

Repeated measures ANOVA was the most used statistical test to analyze the relation between stimuli effect and electrophysiological signals for ER or recognition [25,39,41,45,50,52]. Moreover, this test was used to assess the effect of emotion recognition or regulation on different measurement locations [47,51,53,58,65]. It was consistently used to inquire about differences between emotional tasks in the selected sample. Furthermore, it was used to correlate emotions with physical or mental conditions [46,58]. Two-way ANOVA was used to analyze the relation between groups and emotional conditions [36] during ER [55]. Three-way ANOVA was used to analyze the relation between groups, electrodes, and emotions with electrophysiological data [28,49]. Cheng et al. decided to add two more variables: age and gender [63]. The mixed-design ANOVA was employed to notice the relation between groups and stimuli with electrophysiological signals. Finally, three different versions of ANOVA were reported: (1) multivariate-ANOVA (MANOVA) to assess differences in tasks [42], (2) mass univariate ANOVA for signal perturbations [57], and (3) analysis of co-variance (ANCOVA) for regulation strategy [51]. t-test was frequently used as post hoc test after ANOVA analysis [41,42,43,45,47]. In one study, it was used to comprehend the relation between ER and EEG frequency power density [37]. Tukey’s test was used as a post hoc test to analyze valence and emotion recognition with electrophysiological data and compare between groups [30]. It was also used to analyze ERP data [65]. The Pearson test was another parametric test that was used to associate the regulation effect in valence and arousal ratings for different stimuli [53] with EEG activity [56].

3.5.2. Statistical Analysis: Non-Parametric Methods

This type of test was mainly used as a post hoc test, as shown in

Table 6. Summary of non-parametric tests used in the revised literature.

Test	Variables
Spearman’s test	Subjective evaluation and electrophysiological signals
Bonferroni adjusted pairwise comparison (post hoc)	Condition, valence, arousal, electrode, and electrophysiological data
Wilcoxon test	Time period with the tonic and phasic HRV
Bootstrapping	Emotional behavior, SCR, and dynamic causal modeling connectivity parameters
χ2 test	Pathological conditions and discrete emotions

. Bonferroni correction and method were used to analyze emotional assessments: valence and arousal, electrode position, and condition or task with the electrophysiological data [29,35]. The Spearman test was used to correlate subjective emotional assessments with electrophysiological data [54]. The studies that employed electrocardiographic and electrodermal signals used Wilcoxon test [62] and bootstrapping methods [64], respectively. Finally, χ² test was used to compare pathological conditions and discrete emotions [43].

3.5.3. Other Statistical Methods

Two studies worked with linear mixed-effects modelling, a statistical model suited to heterogenous stimuli [31,61]. Another one applied random effect models that allow modeling of the effects of stimuli simultaneously with the effect of subjects [34]. Another different tool was proposed by [37]: Network-based Statistic. It was used to identify brain regions with different degrees of connectivity for patients compared to controls within and between two measurements.

3.5.4. Classification Strategies

Classification methods have played a significant role in emotion characterization in several studies. In this section, only the results from the methods that reached the highest performances for emotion classification are presented.

The use of Support Vector Machines (SVM) reported a 73.35% accuracy classifying emotions in terms of arousal and 68.54% in terms of valence when using Independent Component Analysis (ICA) features of EEG [59]. An accuracy of 73.08% was achieved with categories of features of EDA and ECG in valence recognition [27]. Other classification methods documented in one study was auto-encoder neural network combining Decision Tree, K-Nearest Neighbors (KNN) and Random Forest (RF) [26]. Sections of EEG data from DEAP and SEED databases were used, and accuracies achieved were 62.63% (DEAP) and 74.85% (SEED). Logistic Regression (LR), Repeated Incremental Pruning to Produce Error Reduction (RIPPER) and Multilayer Perceptron (MLP) were reported in a study validating EEG activity as a good indicator of self-regulation. Groups were categorized according to EDA and reported the following best accuracies: 40.91% in non-self-regulated group using LR, 42.70% in self-regulated group using MLP, and 42.46% with RIPPER as best overall accuracy [32].

Table 7. Technical information of four studies that used machine learning algorithms for emotion classification.

Classes	Electrophysiological Features	Validation Method	Selection Method	Classifier	Performance
Positive, neutral, and negative	EDA, HRV	Leave-one-subject out	Recursive feature elimination	SVM	73.08%
Pleasant, unpleasant and neutral	EEG, EDA	10-fold cross-validation	Data divided in self-regulated and non-self-regulated groups	SVM, Logistic, RF, LR, RIPPER, MLP, KNN, Naïve Bayes, C4.5 and Radial Basis Function	Best accuracy for the non-self-regulated group: LR, 40.9091%. Best Accuracy Self-regulated group: MLP, 42.7035%. Best overall accuracy: RIPPER, 42.4606%
Valence and Arousal	EEG	Confusion matrix	ICA	SVM	73.35% arousal and 68.54% on the valence dimension
Positive, negative, and calm	EEG	K-fold cross-validation	Autoencoder neural network	Decision Tree, KNN and RF	Best accuracies: 62.63% (DEAP); 74.85% (SEED)

details information on classifiers and features introduced in machine learning algorithms.

3.5.5. Sample

The sample size calculation was not discussed in the reviewed articles, nor any correction for a small sample size. This should be relevant, as a well-defined calculation would give us further information on the robustness of the study. In the study with the least number of participants [59], the authors used an artificial intelligence algorithm and obtained similar results than the other studies when accuracy was used as a metric [26,27,32]. On the other hand, the studies with the largest sample size [36,61] included 229 and 262 recordings, respectively.

3.6. RQ6: What Are the Applications and Impacts of Assessing the Emotion Processing?

Analyzing the electrophysiological correlates of emotional processing showed a wide range of applications, emphasizing interests in both emotional perception and regulation. Figure 6 outlines the areas of application of both real-time (online) and offline electrophysiological data monitoring highlighted by the present review.

4. Discussion

The ability to perceive, use, understand and control emotions is known as EI. As this term is conformed of different dimensions in which some authors prefer to prioritize the analysis of one with respect to others. However, the core of these studies pursues to comprehend emotion processing and regulation, that human beings experience in their daily life. Furthermore, electrophysiological recording techniques are used to measure this cognitive task, as physiological responses are affordable signals that highlight the effect of emotions on human processing. To date, there are no systematic review that focuses on measuring EI based on electrophysiological analysis. For that reason, this work is the first attempt to identify studies that characterize emotions by electrophysiological activity to comprehend the ER process.

4.1. RQ1: Which Electrophysiological Signals May Index Emotional Processing?

4.1.1. Electrophysiological Measurements and Analysis

Twenty-eight studies analyzed EEG signals. Each of them executed different conditions for data acquisition. Nonetheless, the most frequent electrode montage was 32 electrodes with a sampling frequency of 500–1000 Hz and a bandwidth of 0.1–30 Hz. Some authors analyzed EEG signals by frequencies to notice increases or decreases between specific conditions, such as [51], where different ER techniques were assessed by comparing power within five frequency bands. Nevertheless, many others considered that low-frequency bands were related to motivational systems and emotional processes [60]. Specifically, theta and beta bands were associated with emotion recognition [39] and ER [56].

To study emotion recognition and regulation independently, some authors have used ERP. The former was generally examined with early ERP components, around 100–300 ms, and the latter with LPP. This latter one was the most standardized feature to study ER. A decrease in the amplitude of the LPP was a reliable indicator of the regulation of emotional arousal and was sensitive to different ER strategies [34].

The second most used technique was ECG. Features extracted from ECG for the study of emotions that reported significant results according to statistical analyses were RSA amplitude, R-R interval characteristics such as the mean and inter-beat R-wave interval, and features of high-rate variability in high-frequency range [29,33,36,44,50,61,62]. An increase in the high-frequency component of HRV, which can be derived from the R-R interval, was determined as a characteristic feature of good ER. RSA reflects the parasympathetic activity, which is part of the high-frequency component and is a reliable index of ER. Nevertheless, a trend in electrophysiological responses was not observed in the revised studies. Further research is needed to determine a correlation between extracted features of cardiac activity and emotions if existent.

Changes in EDA reflect sympathetic activity in the autonomous nervous system. Although EDA is not the primary electrophysiological recording technique for studying the relationship between cognitive emotional behavior and physiological responses, it provides a clue about changes in sympathetic arousal. The SCR was the most analyzed feature in the characterization of emotions. SCR reflects changes in arousal perception [32,54].

On the other hand, EOG is still being explored for studying emotions. In particular, the global velocity of eyeball movement was reported in only one study. Quicker eyeball movement in different targets of happiness, disgust, and sadness for participants with bipolar disorder type I was observed, relating this feature as a marker for emotional to monitor therapies. An increase of this feature was linked to the need for motor activity as a strategy to manage aversive conditions [36].

4.1.2. Open Access Databases

The quality and profile of the data analyzed are important to achieve valid results and conclusions. All but one study collected their own data, which led to a wide diversity of induction methods, psychometric testing, electrophysiological measurements and analysis, experimental paradigms, pattern recognition techniques and applications. On the other hand, one study decided to use the SEED and DEAP databases [26]. The use of previously recorded databases can limit the scope of the study, as the research goals should be adjusted to the experimental conditions and technical recording settings. However, their use reduces the efforts in collecting information, fosters the inter-agreement between scientists, and helps to fade cofounding factors due to methodological concerns.

4.2. RQ2: Which Stimuli May Be Used to Elicit Emotions?

Various methods for emotion elicitation were identified. Most studies used shared databases (e.g., IADS, IAPS) as the source of stimuli to elicit emotions. Those databases have the advantage of representing standardized and internationally known methods, and normative ratings have been extensively provided. On the other hand, a large panel of stimulus types has been outlined, with pictures and films being the most frequent. Contrary to pictures, films are dynamic, and thus more ecological as they approximate real-life environments. Therefore, films are expected to elicit higher emotional experiences than pictures. Very few of the reviewed studies considered the type of stimulus to discuss the observed behavioral and physiological responses to emotional elicitations. Besides, none of them was interested in exploring the effectiveness of different stimulus types regarding emotional and cognitive processes. The mentioned proposal might be very promising, as far as films imply more complex and multimodal emotional processing. However, as films use narrative and dramaturgic structures to tell emotional meanings, the time of exposure may need to be longer to reach conveying of the target emotion. Besides, longer exposure might cause time variations of emotional intensity from the start to the end of the video clip [73]. If such is the case, then temporal dynamics must be included in the analysis of behavioral and physiological correlates. Hence, the time of exposure must be considered rigorously according to the nature of the stimuli.

4.3. RQ3: How Should Psychometrics Assess Emotional Experience?

The complexity of emotional processes is the main difficulty in developing an accurate and precise emotion measurement system [74]. Nonetheless, many methods have been developed and validated. Throughout the review, two major tendencies were found: a small group used the discrete dimension of emotions (happiness, sadness), while the main method was the evaluation of valence and arousal. The SAM scale was the most common evaluation observed for emotion measurement. It consists of visual cues to identify perceived levels of valence, arousal, and/or dominance. This visual representation may reduce introspection and cognitive processing compared to other measuring systems (e.g., VAS scale) [75]. In the conducted research, no study compared different psychometric tests for the evaluation of emotions. Nevertheless, SAM has been validated in diverse ethnic groups [76], generally making it the best option for a diversity of subjects.

Sample characterizations include psychological health assessments. Mainly anxiety and depression levels were evaluated as an exclusion or inclusion criterion in several studies. These disorders reflect irregularities in the perception of positive and negative emotional stimuli [77]. Hence, in any study regarding emotional responses, the evaluation of anxiety and depression traits should be applied, as the neural correlates directly depend on the mental state of participants.

4.4. RQ4: Under Which Conditions May Emotion Processing Be Assessed, and How Does the Experimental Paradigm Cause Biases in Emotional Perception?

A trend of heterogeneity regarding the procedures for inducing emotions has been highlighted, thus complicating the inter-studies analysis. Previous studies emphasized the modulation of electrophysiological and behavioral mechanisms associated with emotional stimuli presentation and processing modalities [78,79].

As a case in point, [79] observed better accuracies in face emotion recognition when stimuli were presented for 50 to 100 ms rather than 16.67 ms, and [80] showed that longer duration of emotional auditory stimuli predicted higher and later peak pupil dilation. Most studies designed experimental paradigms so that a fixed time window was used to analyze physiological responses to emotional elicitation. However, [26] outlined better classification of emotions when considering specific short EEG segments (e.g., the 34 last seconds of 1-min emotional music videos). Furthermore, emotional activation processes varied with the valence (i.e., positive, or negative) and intensity of emotions. The dynamic structure of the emotional experience emphasizes two key components: an onset phase correlated with explosiveness and an offset phase associated with accumulation [81]. The explosiveness phase refers to reactivity and is an index of whether the intensity of the emotional perception has a steep versus gentle start. On the other hand, accumulation refers to whether the emotion intensity increases over time or returns to baseline [81]. Explosiveness and accumulation phases showed distinct neural correlates. The former was associated with regions that take part of the default mode network (such as the medial prefrontal cortex), whereas accumulation showed activity in regions involved in visceral sustained arousal (such as the insula) [82]. Distinct neuronal activities emphasize the essential need to consider temporal dynamics when exploring the electrophysiological basis of emotional processing. Consequently, in the future, experimental paradigms should be designed so that temporal phases of emotional processing can be analyzed appropriately.

Additionally, the conscious/unconscious nature of the emotional stimulus, which can be correlated to the explicit/implicit way of processing emotion, is another relevant factor that has previously shown to predict electrophysiological correlates [83,84]. Contrary to explicit processing, implicit processing is defined by the automatic and out-of-conscious focus on emotional information. The duality between implicit and explicit processing involves separate neuronal circuitry, although the early unconscious process covers a lower cognitive evaluation that leads to further conscious evaluation of emotion states that implies higher-order cognitive processing [84]. Nevertheless, only two studies reviewed here focused on the implicit processing of emotions [56,58].

Consequently, two important remarks must be highlighted here: (1) as variation in stimuli presentation may alter electrophysiological measures correlated to emotional processing, it would be interesting for further studies to include those parameters as covariates in within- and inter-studies signal analysis, and (2) implicit emotional processing still needs further explorations. It could be interesting for future works to correlate autonomous and central neural responses associated with implicit processing to the further explicit processing of emotional stimuli.

4.5. RQ5: How May Correlates of Emotion Processing Be Applied to Models of Statistical Analysis and Artificial Intelligence?

4.5.1. Electrophysiological Correlates in Artificial Intelligence

Classification of emotions is a field that has been extensively researched in recent years. It has gained relevance since artificial intelligence algorithms such as machine learning [85,86] and deep learning [87,88] could widen the understanding of human behavior and be implemented for ER. From the selected literature, only four studies implemented artificial intelligence algorithms for emotion classification. Two of them only implemented EEG for the acquisition of electrophysiological signals [26,59]. The others used EDA combined with other techniques, such as ECG [27] and EEG [32]. From these studies, the best classification accuracy was the algorithm that received differential entropy of EEG in the five neural bands as input [26]. The second-best result was observed in a study using ICA for spatial filtering in EEG signals [59]. From this analysis, it could be observed that filtering in time/space domains gave the best results when trying to correlate EEG signals to emotions. EEG reported better results compared to other studies combining different techniques. In fact, reliable accuracies above 80% have been previously reported in studies using EEG for emotion classification and with a sample size of at least 10 participants [89]. Contrary to the evidence aforementioned, [32] reported that EEG alone was not a good physiological indicator for emotion classification.

Accuracy, the most common classification metric, was reported in all studies. Only in [27] and [32], confusion matrices on the performance of classifiers reported recognition rates. Other classification metrics, such as recall, precision and F1 score that extend the analysis of true positives, true negatives, false positives, and false negatives, were not reported [90]. Comparisons of recognition rates based on these metrics could give a better representation of the performance of classification algorithms with the given electrophysiological data.

From the revised literature, classes in terms of valence [26] and arousal [59] showed a better identification performance. Another finding was that in these studies, classes were set in continuous emotions (e.g., from pleasant to unpleasant), including a neutral case, rather than in discrete emotions (anger, disgust, fear, happiness, neutral, sadness).

4.5.2. Signal Fusion

In a study where ECG and EDA were fused, the classification of extracted features reported accuracy lower than EEG data as unique input [27], indicating that the fusion of electrophysiological signals must be further explored to precisely classify emotions. The fusion of electrophysiological signals could increase pattern recognition in neurotechnology development by feeding classification algorithms with extracted features related to emotion conditions. This task must be performed with caution by feature selection methods. Some of the reported methods were Recursive Feature Elimination which, in essence, ranks the features by their importance, discarding the least important ones [27], and ICA, which decomposes signals into independent sources [59].

4.5.3. Experimental Sample

The sample size can be highly dependent on the study needs and considerations. It plays an important role for the performance and the validation of the study. Analyzing the research of [59], which had the least subjects (10 participants), the accuracy reported by the classifier algorithms was similar or higher than studies with a similar objective but with a sample size of at least 30 subjects (300% larger) [26,27]. No study performed a calculation for the size of the sample needed. This issue was previously addressed in [91], where the importance of a high power to reduce the probability of biased results is highlighted. Nevertheless, it is discussed that the authors alone are not to blame but that the journals should take into consideration the validity of the study groups before selection. Better metrics should be given to peer reviewers to exclude papers that do not present a valid criterion for sample size selection, as not doing so allows the publication of studies that are less probable to be replicated and more probable to be subjected to personal interests. More effort should be stressed in the research community to validate the sample size, an important aspect to increase the rigor of the study. Financially and ethically speaking, it is necessary (1) to know the number of participants needed to reach a previously fixed statistical power (generally 80%), or (2) to evaluate the estimated statistical power when, for logistic or financial reasons, the maximum sample size is known in advance. R software (R Foundation for Statistical Computing, Vienna, Austria) and G*Power [92] are valuable tools for a priori power and sample size estimation. The latter proposes a Graphical User Interface where the researcher can adjust the analysis to the statistical test needed, entering the specific test and its family (e.g., F-tests, t-tests). Sample size a priori estimation is calculated based on required power level, significance level and Cohen’s f effect size index. Besides, several packages of R software (e.g., ‘stats’ or ‘pwr’) include functions (i.e., ‘power.t.test()’ of ‘stats’ package) for a priori power and sample size estimations. Also, the “superpower” R package is a useful tool for simulation-based power analysis [93]. For instance, in the case of means comparisons with ANOVA, ‘power.anova.test()’ computes sample size (n) or statistical power according to the following input parameters: estimated standard deviation (sd) of each group, estimated mean difference (∆) and required significance level.

4.6. RQ6: What Are the Applications and Impacts of Assessing the Emotion Processing?

Analyzing the electrophysiological basis of emotional processing and regulation may be a promising tool that finds applications in a wide variety of areas. For instance, advances in emotion recognition and classification by machine learning and neural networks algorithms can nowadays find utility in video gaming by personalization of the game environment to the user [94], customer satisfaction, high-quality services [95], and aggression predictions in video surveillance tapes [96].

Although a universal way of expressing emotions has been previously emphasized [97], cross-culture [98] and gender [34] variations make artificial intelligence engineering and emotion-related electrophysiological studies a relevant source of information for sociology and anthropology investigations. As regards neurocognitive applications, the literature reviewed here highlighted direct correlations between non-pathological traits of personality (e.g., neuroticism [58], motor performances [35] and executive functions skills) [31] and emotions perception or regulation, which can be useful information in areas such as job positioning or medical assessment of pathological predispositions.

Upon medical applications, emotion recognition by artificial intelligence algorithms allows the rapid acquisition of information that could orient toward diagnosis indications. As a matter of fact, various of the reviewed studies highlighted specific profiles of electrophysiological indicators of emotional processing, allowing the significant distinction between pathological and non-pathological populations [28,38,47]. Nevertheless, none of the studies mentioned in this systematic review searched for in-between pathologies distinctions (e.g., the distinction between schizophrenia and autism spectrum disorder that are often confused [99]). Therefore, diagnosis methods based on electrophysiological correlates of emotion processing still lack investigations.

Finally, as emotion processing disabilities are part of the symptomatology of multiple pathologies (e.g., autism spectrum disorder, schizophrenia, social anxiety disorder, attention deficit/hyperactivity disorder, anorexia nervosa, depression), the assessment of their electrophysiological correlates while attending or regulating emotional stimuli may be an effective way to evaluate the pathological evolution or the effectiveness of medical interventions. For instance, [37] observed a normalization (i.e., differences were not significant between control and experimental groups anymore) of the connectivity in the beta frequency range within cingulate/prefrontal, mesio/lateral temporal, posterior parietal, and insular regions during an ER task after eight weeks of inpatient trauma treatment, which consisted of individual and group-based cognitive stabilization, and art interventions in patients experiencing complex trauma disorders. Another promising application of emotional electrophysiological correlates is the implementation of real-time biofeedback during training [100]. Only two studies included in the present review focused on the effect of neurofeedback on emotion processing or regulation abilities [43,60], highlighting promising results of such a method. Particularly, although [43] did not observe a significant change in the SCR to emotional processing and regulation, they emphasized a reduction of seizure frequency and improvement of anxiety and depression traits after a 3-month skin-conductance-based biofeedback intervention in drug-resistant patients suffering from temporal lobe epilepsy. On the other hand, [60] were interested in the subjective and electrophysiological basis of emotional perception before and after a 5-week EEG-based neurofeedback intervention. They observed the improvement of negative ER and the restoration of delta wave lateralization after neurofeedback training. In sum, neurofeedback training has the potential to be considered as a valid clinical intervention for emotional and cognitive management. However, future investigation is still needed to further understand the benefits of neurofeedback training on pathological and non-pathological populations.

5. Conclusions

In this work, an extensive literature review on methodological concerns about emotion perception and regulation assessments from 2014–2020 has been conducted. The use of electrophysiological databases was not recurrent in research studies. In fact, all but one study recorded electrophysiological signals during the experimental procedure. Furthermore, the lack of disclosure in the methodology of sample size selection is an aspect that should be discussed further, as the validity of the findings could be biased, regardless of the results in posterior statistical tests. A wide variety of emotion elicitation methods have been emphasized, each of them yielding specific electrophysiological processing. Therefore, it would be relevant to adapt data analyses to the dynamic structure of the emotional experience dependent on stimulus nature. Besides, the evaluation of mental health and personality traits is essential for a better understanding of neural correlates variations. Subjective evaluation of emotional stimuli by means of validated scales may complement the interpretation of objective electrophysiological signals. On the other hand, the most used electrophysiological technique in the literature was EEG. Research in emotions analyzed by neural activity in the cortex has been stressed, and time/frequency analyses have shown promising results for the characterization of emotions perception and regulation. Other signals, such as ECG, EDA and EOG, must be further explored to correlate with emotional conditions, and more investigation is still needed to assess the optimal signal fusion to reach a detailed vision of the emotional experience. Lastly, artificial intelligence algorithms presented the highest performance only with extracted features from EEG as input, indicating the need for innovative feature extraction and signal fusion methods for emotion classification. In conclusion, the complexity of emotional processing still needs a deeper characterization that can be achieved by the improvement of methodologies involving psychology, neurophysiology, and artificial intelligence.