Mapping the Invisible: Face-GPS for Facial Muscle Dynamics in Videos

We introduce a novel approach for analyzing facial muscle movement using commonly available video sources, such as smartphone recordings. Our method employs face detection, frame-to-frame tracking, and curvature estimation techniques to quantify the dynamics of local facial muscle movements. An attention-based deep learning network architecture is utilized to generate emotion probability distributions for each video frame. We integrate these probabilities with local kernel smoothing techniques to significantly enhance the precision of muscle movement measurements. Our adaptive kernel method visualizes the facial muscle movements and provides medical professionals with a valuable, interpretable alternative to deep learning models. It also serves as a non-invasive alternative to conventional equipment such as facial electromyography. Our proposed method has applications across various sectors, including neurosurgery, plastic surgery, and remote health monitoring for conditions like stroke, Bell’s palsy, and acoustic neuroma, as well as in emotion detection.