Facial Expression Recognition with Automatic Segmentation

Introduction

Facial expressions play a crucial role in conveying a person’s emotional state. Around 55% of nonverbal communication is captured through facial cues, making robust Facial Expression Recognition (FER) systems vitally important for many applications such as human–computer interaction, intelligent tutoring systems, and healthcare.

However, achieving high-accuracy FER under varying conditions (illumination, partial occlusions, etc.) has always been a challenge. This paper proposes an FER system that enhances performance in these scenarios by automatically segmenting the facial image into two main regions (forehead/eyes and mouth) and then combining a Gabor-filter-based feature extraction technique with a novel fuzzy-based low-complexity classifier.

Methodology Overview

1. Face Detection and Region of Interest (ROI) Segmentation

The approach starts by detecting the face using the Viola–Jones algorithm. Once the face is detected, the algorithm refines the boundaries by subtracting one color channel from another (e.g., red minus green in the RGB space) and then applying a threshold-based binarization. Using image moments and a horizontal projection technique, the face is then segmented into two critical ROIs:

The forehead/eyes region
The mouth region

This segmentation step is vital for handling partial occlusions: if one region (e.g., the mouth) is obscured, the other (the forehead/eyes) can still provide discriminative information.

2. Feature Extraction with Gabor Filters

Each ROI is subdivided into non-overlapping blocks (for example, 30×30 pixels). Each block is then correlated with a bank of 2D Gabor filters. These Gabor functions, which vary in scale and orientation, capture salient texture features that are highly effective for identifying expression-related changes in the face. The outputs of these correlations are then pooled (e.g., by taking the average of the correlation’s first coefficient) to form an initial feature vector.

3. Dimensionality Reduction (PCA)

To reduce the size of the extracted feature vector, Principal Component Analysis (PCA) is employed. This step transforms the correlation features into a more compact set of principal components, ensuring that the dimensionality is significantly reduced without losing critical information about facial expressions.

4. Fuzzy-Based Classification

The final reduced feature vectors are fed into a fuzzy-based classifier. Instead of employing a more complex (and often computationally demanding) method like Support Vector Machines for every training sample, the system builds clusters around representative feature centers using a fuzzy membership function to assign each new sample to the best matching class (emotion). This strategy yields high accuracy with lower computational cost, making it suitable for real-time or resource-constrained scenarios.

Key Results

High Accuracy: The system achieves a recognition rate around 99% when both the mouth and forehead/eyes regions are used together.

Robustness to Occlusion: Even if one region is fully or partially occluded, the other region alone can yield recognition rates over 97%.

Low Complexity Classifier: By using the fuzzy-based approach, the algorithm can be run effectively even in systems with limited computational power (such as embedded devices).

Comparisons: In benchmarks against other state-of-the-art methods (including those based on SVM and other classifiers), the proposed technique offers comparable or superior accuracy, especially under partial occlusions.

Conclusion

This paper introduces a robust FER system that achieves high accuracy by segmenting the face into two ROIs and applying Gabor-based features along with a novel fuzzy-based classifier. Experimental results confirm the system’s effectiveness, particularly its ability to handle partial occlusions. The automatic segmentation routine, aided by a thresholded color-plane subtraction and geometric constraints, further strengthens performance under changing lighting conditions.

The proposed classifier provides a simpler yet efficient approach compared to more conventional methods, making it suitable for various real-time applications where both high recognition accuracy and low latency are critical.