Advancements in the Intelligent Detection of Driver Fatigue and Distraction: A Comprehensive Review

Abstract

Detecting the factors affecting drivers’ safe driving and taking early warning measures can effectively reduce the probability of automobile safety accidents and improve vehicle driving safety. Considering the two factors of driver fatigue and distraction state, their influences on driver behavior are elaborated from both experimental data and an accident library analysis. Starting from three modes and six types, intelligent detection methods for driver fatigue and distraction detection from the past five years are reviewed in detail. Considering its wide range of applications, the research on machine vision detection based on facial features in the past five years is analyzed, and the methods are carefully classified and compared according to their innovation points. Further, three safety warning and response schemes are proposed in light of the development of autonomous driving and intelligent cockpit technology. Finally, the paper summarizes the current state of research in the field, presents five conclusions, and discusses future trends.

1. Introduction

The rapid development and widespread application of automobiles have greatly facilitated people’s transportation and travel. According to data from the National Bureau of Statistics of China, by the end of 2022, the number of vehicles in China had reached 319 million [1]. However, with the rapid increase in the number of automobiles, traffic accidents have become increasingly prominent, with nearly 1.35 million people globally dying or becoming disabled due to traffic accidents each year [2,3,4]. It is estimated that, by 2030, road traffic injuries will become the seventh leading cause of death worldwide [5]. Therefore, the safety of automobile driving should be given full attention.

To reduce the probability of automobile accidents, improving vehicle driving safety has become a common goal in both the automotive industry and academia. Vehicle driving safety is easily influenced by multiple factors, among which, the driver’s condition is an important factor [6,7,8]. The driver’s condition is affected by various aspects, including the driver’s physiology, psychology, and emotions [9,10]. Fatigue and distraction are the two most significant adverse manifestations of the driver’s condition [11]. Studies have shown that the presence of fatigue and distraction in drivers is one of the main causes of traffic fatalities, accounting for about 36% [12].

With the rapid development of technologies such as machine vision [13], deep learning [14], and the analysis and detection of human physiological electrical signals [15], using various intelligent sensors to detect drivers’ fatigue and distraction states has become a current research hotspot [16]. Machine vision [13] emulates the human visual system to recognize, track, and classify objects, serving as a vital perceptual tool in driving the development of automotive intelligence. Deep learning [14], a subset of machine learning composed of multiple neural network layers, utilizes large datasets to autonomously learn features and patterns, providing a robust framework for understanding and interpreting drivers’ behaviors and states. Physiological electrical signals, important manifestations of the body’s internal electrical activity, primarily include brain waves, heart signals, muscle activity, and eye movements, which can indirectly reflect a person’s fatigue or distraction state [15]. By integrating intelligent sensors (such as visual and physiological signal sensors) with deep learning and other technologies, Intelligent Detection Methods have been formed.

Over the past five years, the confluence of cognitive science, vehicular technology, and artificial intelligence has catalyzed unprecedented advancements in the comprehension and mitigation of driver fatigue and distraction. Despite these remarkable technological strides, traffic incidents stemming from impaired driving states continue to pose a significant public safety challenge, highlighting the critical need for more sophisticated detection methods. This comprehensive review undertakes a meticulous exploration of the cutting-edge intelligent detection techniques developed in the past five years, categorizing and conducting a comparative analysis based on signal modalities and types of feature extraction. This endeavor aims to bridge the prevailing gaps in both academic discourse and practical implementations within this domain.

2. The Impact of Fatigue and Distraction on Driving Behavior

The phenomenon of driver fatigue and distraction has become one of the most concerning issues in the field of traffic safety [17,18]. Fatigue driving usually stems from long hours of driving or lack of sleep, leading to physical and mental exhaustion, slow reactions, decreased judgment, and problems such as inattention, increased reaction time, judgment errors, and confusion [19,20]. On the other hand, distracted driving occurs during the driving process when the driver’s attention is diverted from the task of driving due to external distractions or internal thoughts. Activities such as using a mobile phone, conversing with passengers, or adjusting the stereo can diminish a driver’s alertness to the traffic environment and reduce their ability to respond to sudden situations, thereby increasing the risk of traffic accidents [21,22]. Figure 1 illustrates the main causes and behavior patterns leading to driver fatigue and distraction.

Regarding the effects of fatigued driving and distracted driving on driver behavior, scholars have mainly investigated these through two approaches: accident database analysis and experimental research. In terms of accident database analysis, Bioulac et al. [23] analyzed driving data from 70,000 drivers, which included a significant amount of accident data. They discovered that the collision risk under fatigued driving is twice that under normal driving conditions. However, the limitation of this study is its failure to explore in depth the specific impact of different levels of fatigue on collision risk. Furthermore, Sheila et al. [24] revealed through statistical analysis of 685 traffic accidents that the probability of collisions significantly increases for novice drivers when they are distracted, particularly while using mobile phones. This study highlighted the differential impact of distraction types on accident risk, but required broader data to validate the generality of its conclusions.

In experimental research, the convenience and safety of simulators have led to a broad focus on studying the impacts of fatigue and distraction on driving. Öztürk et al. [25] used the n-back test task on a simulator platform to examine the effects of varying cognitive loads on driving behavior and detection response tasks. Bassani et al. [26] conducted experiments based on a simulator to compare the differences in speed control and lateral vehicle control capabilities between distracted and non-distracted drivers. However, it has been noted that the actual impact of driver fatigue on reaction capabilities in real driving situations is more significant than what is represented in simulator data [27]. Additionally, the reactions of drivers to visual distractions and interface interactions in real road environments show significant differences from those in simulated environments [28]. Thus, when assessing impacts and quantifying research, as well as devising safety warning and response plans, reliance should not only be on simulated data; real vehicle experiments should also be conducted. Hu et al. [29] assessed the fatigue level of drivers using the Karolinska Sleepiness Scale (KSS) before real vehicle testing. Their analysis found that reaction time is positively correlated with level of fatigue, highlighting the actual impact of fatigued driving on safety. However, this study relied on subjective assessments, which may have introduced bias. Ma et al. [30] analyzed real vehicle driving test data to study the impact of distracted driving on driving capabilities, finding that driver distraction reduces the driver’s vehicle control capabilities and that this impact varies with the type of distraction, yet they did not fully explore the quantitative assessment of distraction level.

In summary, fatigue and distraction phenomena severely affect driving behavior, increasing the risk of traffic accidents. Current research mainly explores these impacts through accident database analysis and experimental studies. However, existing research falls short of deeply investigating the specific effects of different fatigue levels and distraction types on collision risk and driving capabilities. Particularly, the actual impacts of fatigue and distraction on driving behavior in real driving situations have not been fully assessed, and methods based on subjective assessment might introduce bias. Therefore, researching real-time, high-accuracy, and robust online detection methods for driver fatigue and distraction states is of significant importance. Such research could provide objective and accurate assessments of driver states and support the design of effective traffic safety warning and response schemes.

3. Intelligent Detection Methods for Driver Fatigue and Distraction

With the rapid development of deep learning technologies and the enhancement of computing hardware power, breakthroughs have been achieved in intelligent detection techniques for driver fatigue and distraction. Based on the types of features extracted, these technologies can be divided into intelligent detection methods based on facial features, head posture, behavioral actions, physiological signals, and vehicle data, as shown in Figure 2. Among these, the first three methods require the use of cameras, infrared cameras, and other visual sensors to collect drivers’ image information for further recognition of their states [31], which can be collectively referred to as image-information-based intelligent detection methods. Furthermore, some researchers have conducted studies on multimodal fusion algorithms that integrate image information, physiological signals, and vehicle data, thereby improving the detection’s accuracy and robustness.

3.1. Intelligent Detection Methods Based on Driver’s Facial Features

Intelligent detection methods based on drivers’ facial features primarily collect facial information through cameras installed inside vehicles. After image processing, machine learning or deep learning methods are used to identify the facial region and extract feature points, followed by model training. This process enables the detection of driver fatigue or distraction states. These methods are widely recognized and applied due to their real-time performance, low cost, and non-invasive nature.

The basic workflow of intelligent detection methods based on facial features is as follows, as illustrated in Figure 3. First, the Eye Aspect Ratio (EAR) and the Mouth Aspect Ratio (MAR) are calculated. Then, based on the EAR value, eye fatigue assessment indicators are computed, such as the percentage of eyelid closure over the pupil over time (PERCLOS) [32], blink frequency, and the duration of continuous eye closure. Based on the MAR value, yawning can be determined by calculating the yawn frequency and duration. Finally, by integrating these assessment indicators into a model, a comprehensive judgment is made on whether the driver is in a state of fatigue or distraction.

Currently, Driver State Monitoring (DSM) systems widely applied in actual vehicles often use traditional machine learning libraries such as dlib to extract features of the eyes and mouth, calculating fatigue and distraction state assessment indicators using fixed thresholds. This method is widely adopted due to its simplicity and low sensor requirements. However, its recognition accuracy and detection effectiveness need improvement, and it overlooks individual differences among drivers.

To address the challenge of low accuracy in face recognition and feature point extraction, the academic community has proposed various improvement schemes. Zhu Feng et al. [33] enhanced face detection accuracy significantly by integrating an improved YOLO v3 method with the Kalman filter algorithm for face detection and using the boosting tree algorithm to extract facial feature points, effectively solving the problem of missed detections caused by glasses and hats. Although this method has made progress in accuracy, its complexity and computational cost are higher. Wang et al. [34] pointed out the correlation between drivers’ eye position and ethnicity, greatly improving positioning accuracy through a skin color recognition model and a bidirectional integral projection method, but their accuracy may be limited to specific groups. Yang et al. [35] combined 3D Convolutional Neural Networks (3D-CNN) and Bidirectional Long Short-Term Memory networks (Bi-LSTM) to propose the 3D-LTS network, which effectively recognizes subtle facial movements, improving the accuracy of mouth feature point extraction and yawn detection, as shown in Figure 4. Although this method performs excellently in dynamic feature recognition, it still faces challenges in real-time performance and resource consumption.

To address the shortcomings of traditional methods that do not consider individual differences among drivers, some studies have optimized detection approaches. Li et al. [36] improved detection effectiveness by constructing a driver identity information database and extracting facial features as a reference for assessing the driver’s state. This personalized method has obvious advantages, but its application range may be limited by the size of the offline library. You et al. [37] used a Deep Cascaded Convolutional Neural Network (DCCNN) to detect facial features and established an SVM model based on facial features, replacing traditional uniform threshold methods and adding personalized elements to fatigue and distraction detection. Although this increases the personalization of detection, the model’s general applicability and accuracy still need further verification, as shown in Figure 5. Han et al. [38] used the ShuffleNet V2K16 neural network for driver facial recognition, reducing the impact of the detection environment, and by comparing the current frame’s EAR and MAR values with the maximum EAR value and minimum MAR value of the previous 100 frames, they minimized the impact of individual differences, greatly improving experimental accuracy.

Some researchers have utilized cutting-edge deep learning networks, significantly enhancing detection precision. Liu et al. [39] introduced a dual-stream neural network into the Multi-Task cascaded Convolutional Neural Network (MTCNN), combining the static and dynamic features of drivers to improve detection performance, achieving an accuracy rate of 97.06%. Moreover, by using Gamma correction, they enhanced the accuracy of nighttime detection, effectively overcoming the challenges of detection in night conditions. Ahmed et al. [40] incorporated two InceptionV3 modules into the MTCNN network to extract the features of eye and mouth sub-samples, significantly improving the detection precision of local features. Although this method performs well in local feature recognition, its efficiency in processing large-scale data remains a consideration.

In addressing delays caused by complex algorithms, Kim et al. [41] developed a lightweight driver state monitoring system that realized end-to-end detection, significantly improving detection efficiency. This method simplified the detection process, but may have sacrificed some recognition precision. He et al. [42] used near-infrared cameras instead of traditional RGB and infrared cameras and established an integrated deep learning network model, which ensured the algorithm’s accuracy while reducing the hardware computational requirements, offering a way to achieve efficient detection in resource-limited situations. Guo et al. [43] addressed the issue that most detection methods cannot detect distracted behaviors not included in the training set well by using a semi-supervised approach with a dual-stream backbone network design, ensuring the model’s lightweight structure while enhancing its generalization capability.

Table 1. Analysis and comparison of intelligent detection methods based on facial features from the last five years.

Author	Year	Methodological Innovations	Extraction of Features	Machine Learning Models	Dataset	Accuracy	Improvement over Traditional Methods
Zhang et al. [33]	2022	Solving the problem of missed detection due to occlusion or misjudgment.	PERCLOS value, maximum eye closure time, number of yawns	Improves the yolov3+Kalman filter algorithm	Self-built real-vehicle dataset	92.50%	Solves the problem of the low accuracy of recognizing face parts in traditional methods.
Wang et al. [34]	2019	Proposing a bidirectional integral projection method to realize the precise localization of the human eye.	Blink frequency	KNN algorithm	Self-constructed simulator dataset	87.82%
Yang et al. [35]	2021	Recognizing yawning behavior based on subtle facial movements for improved accuracy.	Subtle facial changes	3D-LTS combining 3D convolution + Bi-LSTM	YawDD dataset	92.10%
Li et al. [36]	2020	Offline construction of driver identity database to analyze driving status from driver features.	Facial features	Improved yolov3-tiny + improved dlib library	DSD dataset	95.10%	Consideration of driver characteristics and their variability.
You et al. [37]	2019	Analyzing changes in binocular aspect ratio using neural network training.	Eye aspect ratio	Deep cascaded convolutional neural network	FDDB dataset + self-built simulator dataset	94.80%
Han et al. [38]	2023	Weakening environmental effects and individual differences, improving dlib method to enhance the accuracy of facial feature point extraction.	64 feature points, EAR, MAR	ShuffleNet V2K16 neural network	Self-constructed real-vehicle dataset	98.8%
Liu et al. [39]	2019	Introducing dual-stream neural network to combine static and dynamic image information for fatigue detection; utilizing gamma correction method to improve nighttime detection accuracy.	Static and dynamic image fusion information	Multi-task cascaded convolutional neural network	NTHU-DDD dataset	97.06%	Research on high-performance deep learning models to improve detection accuracy.
Ahmed et al. [40]	2022	Proposing a deep learning integration model and introducing the InceptionV3 module for feature extraction of eye and mouth subsamples.	Eye and mouth images	Multi-task cascaded convolutional neural network	NTHU-DDD dataset	97.10%
Kim et al. [41]	2019	Reducing arithmetic requirements, realizing end-to-end detection, and improving detection efficiency.	Raw images	Multi-task lightweight neural network	Self-built simulator dataset	Face orientation: 96.40% Eyes closed: 77.56% Mouth open: 93.93%	Research on lightweight models to promote technology application.
He et al. [42]	2019	Building and integrating multiple lightweight deep learning models to recognize fatigue and risky driving behaviors.	Part recognition with extended range images	SSD-MobileNet model	300 W + self-constructed validation dataset	95.10%
Guo et al. [43]	2024	Adaptive detection of distracting behaviors not included in the training set to ensure lightweight and enhance generalization capability.	Full depth images	Visual Transformer model	Self-built real-vehicle dataset MAS	98.98%

presents a comparative analysis of recent studies on new detection methods based on facial features. Overall, such methods, under experimental conditions, boast a high accuracy and convenience, becoming the primary choice for developing driver state monitoring systems [44]. However, the actual application of these methods still faces several challenges, mainly as follows:

• The actual precision of facial feature extraction is easily affected by the environment. Facial appearance may vary significantly under different lighting conditions, angle changes, and facial expressions, making it difficult to maintain stability and accuracy in facial feature extraction. Additionally, face occlusions, wearing glasses, and makeup can also affect the effectiveness of facial feature extraction.
• In detection methods based on facial features, most use facial feature parameters such as EAR and MAR, etc., to judge fatigue or distraction states. However, the setting of thresholds often relies on subjective experience, lacking objectivity and unified standards.
• Detection methods based on facial features are greatly affected by individual driver differences. While people may display facial states similar to fatigue or distraction, such as yawning, blinking, or looking down, in everyday life, it does not necessarily mean they are truly in a state of fatigue or distraction. Therefore, judging whether a driver is fatigued or distracted based solely on facial features presents potential risks of misjudgment or omission.

3.2. Intelligent Detection Methods Based on Driver’s Head Posture

Detection methods based on a driver’s head posture work by calculating the relative positions of various feature points in a three-dimensional space to obtain head posture angles, thereby detecting whether the driver is engaged in distracted behaviors, such as diverting their gaze from the driving task. Detecting the driver’s head posture can also help to identify behaviors such as looking down at a phone or nodding off due to drowsiness.

Since head rotation is a continuous process, models that consider the relationship between frames often achieve better detection results than methods that rely solely on real-time detection using single frames. Zhao et al. [45] optimized the fully connected layers of a residual error network with a composite loss function and expanded the training dataset through transfer learning. This approach accurately and continuously monitors a driver’s distracted state in real driving environments but lacks an in-depth analysis of adaptability to environmental changes and real-time requirements. Ansari et al. [46] argued that a driver’s psychological fatigue and sleepiness are reflected in changes in head posture. They introduced an improved linear unit layer into a bidirectional Long Short-Term Memory (LSTM) network structure to effectively process a 3D time series of head angular acceleration data, allowing for the effective recognition of complex head movement states. However, this method requires extensive training data to accurately identify different driver states, and data labeling is challenging.

Detection methods based on driver head posture share similarities with those based on facial features, as they can be conducted using only camera sensors, making them low-cost and easy to integrate. However, the accuracy of their detection is easily affected by head occlusions and the diversity of postures.

3.3. Intelligent Detection Methods Based on Driver’s Behavioral Actions

During the driving process, the actions and behaviors of drivers can significantly reflect their attention states. Behaviors such as making phone calls, texting, smoking, or reaching for items can lead to driver distraction. Accordingly, monitoring these behaviors can effectively determine whether a driver is distracted. Xing et al. [47] utilized a deep convolutional neural network (CNN) to design a system capable of recognizing a variety of driving behaviors, including normal driving, looking at rearview mirrors in various directions, using the car radio, texting, and making phone calls.

However, given the small differences between normal and distracted driving behaviors and the high similarity of some actions, relying solely on a CNN might lead to recognition errors. Therefore, Ma et al. [48] introduced a bilinear fusion network and attention mechanism, proposing a convolutional neural network specifically designed for driving behavior recognition, BACNN, which showed a good performance on the State Farm dataset. To further improve recognition accuracy and real-time performance, Zhang et al. [49] developed a specialized intertwined deep convolutional neural network, InterCNN. Fasanmade et al. [50] established an expert knowledge rule system and a discrete dynamic Bayesian classification model to predict the level of distraction in video frame sequences, providing a quantitative assessment of driver behavior.

In addition to using cameras to recognize driver actions and behaviors through visual images, some researchers have used wearable devices to collect signals reflecting driver behavior. However, large wearable devices are not portable and are costly, making them suitable only for experimental data collection and difficult to use for real-time detection. Thus, researching and developing portable wearable devices for the real-time monitoring of driver states is a trend. Xie et al. [51] proposed a method to identify driver distraction using a wristband equipped with an Inertial Measurement Unit (IMU), i.e., collecting acceleration data from the driver’s right wristband’s IMU and building a Convolutional Long Short-Term Memory (ConvLSTM) deep neural network model for training. The results showed that ConvLSTM achieved a better detection accuracy than CNN and LSTM models. Wagner et al. [52] specifically targeted dangerous behaviors such as using phones, eating, and smoking by drivers. They proposed a deep-learning-based classification detection method by capturing images with left and right infrared cameras.

In summary, these methods offer high real-time capabilities. Once they detect dangerous driving behaviors that could distract drivers, immediate warnings can be issued without the need for cumulative time to continuously observe changes in driver characteristics. Although these technologies have shown efficiency in the real-time monitoring of driver states and can promptly identify and warn of potential distractions, they still face challenges such as scene limitations and individual behavior differences. Future research needs to continue exploring more accurate and personalized recognition methods to adapt to the diversity and complexity of driver behaviors. Additionally, developing more convenient, lower-cost tactile motion sensing systems for broader applications is an important research direction.

3.4. Intelligent Detection Methods Based on Driver’s Physiological Characteristics

Studies indicate that significant changes in physiological characteristics occur when a driver is fatigued or distracted [53]. Monitoring a driver’s physiological signals through physiological instrument devices can analyze their attention level, emotional state, and degree of physical fatigue. Common physiological signals include the Electroencephalogram (EEG), Electrocardiogram (ECG), Heart Rate (HR), Electrooculogram (EOG), and Electromyogram (EMG).

Bundele et al. [54] monitored drivers’ skin conductivity and blood oxygen saturation using wearable devices, designing a multilayer perceptron model for training to classify and detect drivers’ psychological fatigue and drowsiness states. Chaudhuri et al. [55], based on EEG signals from drivers’ scalps, used electrophysiological source imaging and EEG source localization techniques for signal processing and feature extraction, and established a Support Vector Machine (SVM) classifier to detect whether drivers were in a state of extreme fatigue. Li et al. [56] employed convolutional neural networks and gated recurrent units to map the relationship between drivers’ distraction states and EEG signals in the time domain, verifying the effectiveness of their method through simulation experiments. Fu et al. [57] developed a non-contact vehicle-mounted driver fatigue detection system to collect drivers’ biceps femoris electromyographic signals and ECG signals. Chen et al. [58] decomposed EEG signals into wavelet sub-bands to extract nonlinear features beyond the original signal and integrated them with eyelid movement information, using an Extreme Learning Machine (ELM) for state classification.

The use of full-head EEG monitoring devices often leads to issues such as computational and storage resource wastage and low real-time performance. In light of this, Fan et al. [59] optimized EEG instruments by monitoring the frontal EEG signals of drivers to extract features like the energy, entropy, and frontal EEG asymmetry ratio of EEG signals, proposing a time-series-based ensemble learning method for fatigue and distraction detection.

Overall, intelligent detection methods based on drivers’ physiological characteristics can accurately assess drivers’ states by monitoring changes in their physiological states. However, most of these methods use invasive sensors, which may cause discomfort to drivers and have disadvantages such as poor convenience, high cost, and detection accuracy being easily affected by the environment, hindering their application.

3.5. Intelligent Detection Methods Based on Vehicle Travel Data

When drivers are fatigued or distracted, their driving skills and reaction times are compromised, leading to deviations in heading angle and difficulties in maintaining the correct direction of travel on the road. Furthermore, the emergence of fatigue and distraction can reduce a driver’s steering capability, potentially resulting in insufficient steering actions that affect their driving behavior. In severe cases, this may even lead to significant lane departures. Therefore, analyzing vehicle driving data can indirectly assess a driver’s fatigue and distraction levels.

Yang et al. [60] assessed the state and behavior of drivers based on vehicle GPS data and the Gaussian Mixture Model (GMM). Wang et al. [61] utilized a Bidirectional Long Short-Term Memory network (Bi-LSTM) with an attention mechanism to propose a method for indirectly detecting drivers using mobile phones based on vehicle performance parameters, achieving an accuracy rate of up to 91.2%. Sun et al. [62] also employed a Bi-LSTM network to process data on vehicle steering wheel angles, steering wheel angular velocity, vehicle yaw rate, lateral acceleration, and longitudinal acceleration. They used a wavelet packet analysis to extract characteristic frequency bands, thereby indirectly identifying driver fatigue and distraction.

Intelligent detection methods based on vehicle driving data generally involve data collection through a vehicle’s original onboard sensors, offering advantages in convenience and cost. However, this approach has difficulty in directly reflecting changes in drivers’ physiological and psychological states, especially under simple driving conditions, where vehicle driving data show minimal variations, making it challenging to reflect a driver’s state accurately.

3.6. Intelligent Detection Methods Based on Multimodal Fusion Feature

Driver fatigue or distraction is influenced by various physiological and psychological factors and exhibits individual differences, presenting diverse manifestations and behaviors. Therefore, it is challenging to make a comprehensive judgment based solely on single-modality signal features. Consequently, many scholars have integrated features such as driver image information, physiological signals, and vehicle data to study detection methods based on multimodal feature fusion, aiming to improve the accuracy and robustness of the detection and explore the potential connections between features and driver fatigue and distraction.

Wang et al. [63] integrated physiological signals collected by biometric instruments, such as heart rate and brain waves, with facial features and head postures extracted by cameras and trained with an RBF neural network, to achieve deployment and application at a lower cost. Du and others [64] addressed the issue of distracted driving caused by drivers conversing with others, by introducing the analysis and detection of voice signals. They proposed a dataset comprising three modalities: driver facial features, voice recognition, and vehicle signals, applied to the training of a multi-layer fusion model, achieving better detection results than single-modality models. Abbas and others [65] developed a multimodal detection system that integrates physiological and facial features for training a Deep Residual Neural Network (DRNN) model. They classified the driver states into five categories: normal, fatigued, visually distracted, cognitively distracted, and drowsy. The hardware setup is shown in Figure 6.

Detection methods based on multimodal feature fusion, by integrating various modal characteristics, can establish a more comprehensive assessment system. This approach compensates for the shortcomings of single-modality methods, collectively enhancing the accuracy and stability of detection and enabling more flexible and refined state monitoring. However, the requirement for a greater variety and number of sensors leads to increased hardware and software costs, and the processing and fusion algorithms for data from different sensors become more complex. Overall, multimodal fusion represents a research trend in the detection of driver fatigue and distraction states.

4. Safety Warning and Response Strategies Based on Fatigue and Distraction Detection

4.1. Warning Prompts Based on the Driving Cockpit

With the rapid development of intelligent cockpits, new forms of human–machine interaction (HMI) are continually emerging, providing new warning formats for detecting driver fatigue and distraction [66]. Upon the preliminary detection of driver fatigue or distraction, the design of in-cabin HMI can provide sensory stimuli to the driver in visual, auditory, and tactile forms as warning prompts. Generally, warning prompts are categorized into normal and urgent. For lower levels of fatigue or distraction, normal warnings are employed using standard visual or auditory signals [67]. When the level of fatigue or distraction is high and persists, urgent warnings should be issued by expanding the range of visual cues, changing the color of visual alerts, increasing the decibel level of auditory signals, or altering the words used in voice prompts [68,69]. Contact reminders such as vibrations in the seat and steering wheel [70] can also be utilized to enhance sensory stimulation for the driver, thereby facilitating the timely recovery of vehicle control capabilities.

4.2. Safety Response Based on Advanced Driver Assistance Systems

When drivers exhibit signs of fatigue or distraction, beyond issuing early warnings, safety responses can also be implemented through Advanced Driver Assistance Systems (ADAS). If it is detected that a driver is unable to control the vehicle promptly due to fatigue or distraction, an ADAS can automatically engage emergency braking or perform other evasive maneuvers to prevent potential collisions or accidents, thereby affording the driver more reaction time [71]. For instance, an Automatic Emergency Braking (AEB) system can autonomously apply the brakes to significantly reduce the risk of an accident if a collision is predicted and the driver has not responded in time. The Lane Keeping Assist System (LKAS) plays a vital role in preventing potential deviation accidents caused by driver fatigue or distraction by automatically adjusting the steering to help the vehicle stay in its current lane. Adaptive Cruise Control (ACC) reduces the risk of collisions caused by excessive speed or following too closely by automatically adjusting the vehicle’s speed to maintain a safe distance. The integration of these ADAS features not only enhances driving safety, but also effectively reduces the accident risk associated with driver fatigue or distraction, creating a safer driving environment for both drivers and passengers.

4.3. Multi-Level Response Mechanism Combining Autonomous Driving Technology

As the process of automobile intelligence and connectivity accelerates, integrating autonomous driving technology to establish a multi-level safety response mechanism offers a new solution for reducing the driving risks associated with driver fatigue or distraction. Within this framework, the system first attempts to alert the driver’s attention through warning prompts and provides preliminary safety interventions using Advanced Driver Assistance Systems (ADAS) in emergency situations, playing a crucial role when the driver’s response capabilities are insufficient to avoid potential risks.

If primary measures fail to effectively reduce the accident risk, or if the driver fails to respond in time, the Autonomous Driving System (ADS) is prepared to take over vehicle control. During this process, ADS uses its advanced perception and decision-making capabilities to automatically navigate the vehicle to a safe area or take the most appropriate actions to ensure the safety of both the occupants and surrounding traffic participants. Currently, despite the rapid progress in autonomous driving technology, including capabilities demonstrated in experimental and limited commercial deployments, there are still challenges to achieving full automation takeover, such as perception complexity, decision-making reliability, and legal and ethical issues [72,73].

Furthermore, it is critical to acknowledge that these intelligent vehicles are susceptible to potential cybersecurity threats from malicious actors. Such vulnerabilities necessitate the incorporation of cybersecurity measures within the autonomous driving framework to safeguard against cyberattacks that could compromise vehicle safety and functionality. Petit et al. [74] highlighted the importance of considering cybersecurity implications in cooperative automated vehicle systems, stressing the need for enhanced redundancy. Parkinson et al. [75] emphasized that, with increased connectivity and automation, vehicles face heightened risks of cybersecurity attacks, calling for a focus on addressing these risks in the autonomous vehicle sector.

In the context of cyberattacks during the vehicle handover process, the challenge of distinguishing between abnormal driving behaviors caused by driver issues, such as fatigue or distraction, and those induced by cyberattacks is significant. Addressing this challenge, several scholars have conducted in-depth studies aimed at clearly identifying the effects of cyberattacks on autonomous driving control systems. For instance, Petrillo et al. [76] explored secure adaptive control for autonomous vehicles under cyberattacks, while Guo et al. [77] focused on cyber-physical system-based path-tracking control, underscoring the necessity for sophisticated detection and mitigation strategies. These investigations reveal the complexities of maintaining autonomous vehicle safety in the face of cyber threats, emphasizing the covert nature of these threats and the critical need for advanced techniques in attack detection and response. Additionally, Sheehan et al. [78] proposed a proactive approach to cyber risk classification for CAVs, utilizing a Bayesian Network model predicated on known software vulnerabilities, representing a methodological advancement in effectively predicting and mitigating cyber risks.

Building on the understanding of cyberattacks’ impact on autonomous driving perception and control systems, numerous scholars have investigated real-time detection methods aimed at mitigating the effects of such attacks. Van Wyk et al. [79] enhanced sensor anomaly detection and identification capabilities significantly by combining Kalman filtering with multi-layered deep learning methods. Wang et al. [80] developed a comprehensive framework for detecting and isolating cyberattacks on autonomous driving systems, offering effective strategies to ensure that vehicle localization and navigation remain unaffected by cyber disturbances. Furthermore, Li et al. [81] introduced an anomaly detection model based on Generative Adversarial Networks (GANs), capable of detecting trajectory anomalies and sensor data injections in a timely manner using short-term data sequences.

Future research endeavors should not be limited to enhancing the sensory capabilities of autonomous driving systems and refining their decision-making algorithms; there is a pressing need to fortify cybersecurity defenses to safeguard against and diminish the ramifications of cyberattacks. Moreover, an intensified focus on distinguishing the origins of anomalies in autonomous vehicles is essential, a domain presently marked by scant research. Through the fusion of interdisciplinary collaboration and technological innovation, autonomous driving technology is poised to fundamentally revolutionize driving safety, promising a substantial reduction in traffic incidents attributed to driver fatigue and distraction.

5. Conclusions and Outlook

This paper comprehensively analyzes the current technological research progress on intelligent detection methods for driver fatigue and distraction. These methods are categorized into those based on image information, physiological characteristics, vehicle driving data, and multimodal feature fusion. Not only have these methods improved the accuracy and real-time capabilities of detection, but they have also provided technical support for a safer driving environment. Through the analysis and summary of various intelligent detection methods and research on safety response plans, the following conclusions are drawn:

• Detection methods based on image information, especially those relying on machine learning and deep learning technologies, have significantly improved the accuracy of facial feature recognition, particularly in the analysis of face detection, eye, and mouth movements. However, these methods are highly sensitive to environmental conditions such as lighting and obstruction of the driver’s head, while also neglecting individual differences among drivers, somewhat limiting their widespread application potential.
• Detection methods based on the physiological characteristics of drivers, by analyzing physiological signals like electroencephalograms (EEG), electrocardiograms (ECG), and heart rate (HR), provide more direct indicators for assessing the attention level and fatigue state of the driver. These methods can avoid external environmental interference to a certain extent and provide relatively stable detection results. Nonetheless, physiological signal detection often requires the use of invasive sensors, which may cause discomfort to the driver and have limited applicability in actual driving environments.
• Intelligent detection methods based on vehicle driving data assess the driver’s level of attention distraction indirectly by analyzing the correlation between driving behavior and the vehicle operation status. These methods are easy to implement and cost-effective, but their detection accuracy is influenced by the complexity of the driving environment and the difficulty in directly reflecting changes in the driver’s physiological and psychological state.
• To enhance the robustness, stability, and overall performance of detection systems, detection methods based on multimodal feature fusion have emerged, integrating features from image information, physiological signals, and vehicle data to improve the comprehensiveness and accuracy of detection. Although this approach effectively utilizes the advantages of different data sources and enhances the robustness of the detection system, it also introduces higher implementation costs, technical complexity, and computational demands.
• In terms of safety warning research, current systems primarily rely on in-cabin human–machine interaction designs, issuing warnings to drivers through visual, auditory, and tactile means. These solutions can enhance driver alertness to some extent and reduce accidents caused by fatigue or distraction. However, the effectiveness of these systems is often limited by the driver’s subjective acceptance and real-time response capability. Moreover, safety response measures, such as the intervention of Advanced Driver Assistance Systems (ADAS) and the application of autonomous driving technology, can reduce risks to some extent, but their capability to handle complex traffic environments and the collaboration between drivers and systems require further research.

Looking forward to the future development of intelligent detection technology for driver fatigue and distraction, several technical dimensions can be envisioned:

• Future research should focus on the adaptability of algorithms in complex environments, such as stability under different lighting conditions and changes in the driver’s posture, developing lightweight and efficient neural network models to ensure the rapidity of data processing and high accuracy of detection results.
• The innovation of non-invasive sensors and related algorithms could be advanced by collecting physiological signals through non-contact or minimally invasive methods, reducing driver discomfort, and expanding application scenarios.
• The application of machine learning and artificial intelligence technologies in analyzing the relationship between driving behavior and vehicle performance could be strengthened, precisely predicting driver states through detailed data collection and analysis.
• Data fusion technologies and model integration strategies could be optimized, exploring effective feature fusion algorithms to enhance the complementarity and accuracy of analysis between different data sources.
• Future safety warning schemes should pay more attention to personalization and intelligence, providing customized warning signals based on the driver’s behavior patterns and physiological state to enhance the effectiveness of warnings.
• Research should be conducted on the seamless switching mechanism between advanced driver assistance systems and autonomous driving technology, improving the safety and flexibility of the system, exploring the data fusion of in-vehicle and external environment perception systems to provide comprehensive decision support for ADAS and autonomous driving technology.

Overall, this comprehensive review critically evaluates the latest advancements in deep-learning-based intelligent detection technologies for driver fatigue and distraction over the past five years. It sheds light on their theoretical foundations, methodological innovations, and existing shortcomings. By synthesizing the research within this period, our analysis not only highlights the progress made in improving detection accuracy and real-time response capabilities, but also points out the persisting challenges and gaps, setting the stage for future research. Key contributions from our analysis include:

• An integrated framework that categorizes the intelligent detection methods based on deep learning developed over the past five years and proposes a comprehensive safety warning and response scheme. This scheme features varied warning and response mechanisms tailored to the different levels of vehicle automation.
• A critical evaluation of the limitations inherent in the current methodologies and the potential for leveraging emerging technologies such as AI and machine learning to address these challenges.
• The identification of areas that could significantly benefit from further research, including non-invasive sensing techniques, the integration of multimodal data, and the development of adaptive, personalized detection systems.

Looking forward, we advocate for a concerted effort towards interdisciplinary research that bridges cognitive science, vehicular engineering, and computer science. Such collaborative endeavors could unlock new pathways for understanding driver behavior and developing more sophisticated, context-aware technologies capable of mitigating the risks associated with driver fatigue and distraction. Furthermore, our review underscores the necessity for the rigorous validation of detection technologies in real-world settings, ensuring their efficacy and reliability across diverse driving conditions and populations.

In conclusion, this review not only consolidates current knowledge in the domain of driver fatigue and distraction detection, but also acts as a catalyst for future research. By highlighting the academic and practical value of recent advancements, we lay the groundwork for the next generation of detection technologies that promise enhanced road safety and driver well-being. As the field continues to evolve, it is imperative that future research endeavors are guided by the dual principles of innovation and inclusivity, ensuring that technological progress translates into tangible safety benefits for all road users.