Figure 1. The facial expressions of the expert on facial expressions, Paul Ekman, Ph.D.

Child development, including to the ability to recognize emotional expressions, can be fast in the first year of life. A study by Caron et al. (1988) sought to determine the age at which infants learn to discriminate dynamic emotions (e.g., angry, surprised). Groups of infants four-, five-, and seven-months old were tested on some or all of five conditions involving two different emotions (displayed by two different women consecutively) with or without voice accompaniment. By measuring fixation times following habituation to one expression, the experimenters found an interesting hierarchy for the three groups. The four-month-olds were only able to recover to happy expressions following habituation to sad expressions, but not vice versa. However, in the same experiment (featuring voice), the five-month-olds were able to recover following habituation in both directions. A follow-up experiment showed that the five-month-olds readily recovered to happy expressions following habituation to sad ones in the absence of vocal accompaniment. While this suggests that voice plays less of a role in discriminating happy and sad faces, testing four-month-olds without voice would have provided a more convincing conclusion. In general, however, these two experiments suggest that an infant cannot discriminate naturalistic expressions of happiness and sadness until five months of age.

In addition, another experiment utilizing the same stimuli (with voice) showed that the seven-month-olds performed at the same level as, or better than, the other groups, and were even able to discriminate between angry and happy and vice versa, unlike the five-month-olds (p < 0.05 for age times trials interaction; four-month-olds were not tested on this dimension). However, another follow-up experiment indicated that they could only achieve this latter finding when the voices were played (p < 0.005 for a significant vocalization times trials effect). The experimenters concluded that infants could distinguish dynamic emotions (including anger) as early as seven months of age. Furthermore, they contended that infants rely more on voice than facial expression when perceiving dynamic emotions. Because of this fact, more significant findings on emotion recognition in facial expressions should be found in an older population of adolescents and adults, when more exposure to faces is inevitable.

Early Experience & Abuse

Because familiarity plays a major role in the emotion recognition ability of infants, early-life experiences may be of similar importance. The effects of atypical experiences, specifically physical abuse and improper parental care, may shed light on the level of importance. In one attempt to uncover these effects, Pollak et al. (2000) conducted two experiments to examine the emotion recognition abilities of physically abused, neglected, and normal children, all of whom were relatively young, three to five years old. In the first experiment, the children were asked to match facial expressions to emotional situations. Neglected children (those found to have caregivers who failed to meet their minimum needs) had more difficulty discriminating facial expressions (51% accuracy) than physically abused (59% accuracy) and control children (66% accuracy). In addition, the first experiment revealed two response biases: neglected children were significantly more likely to select the sad expressions (p < 0.01) compared to the other groups, and physically abused children were significantly more likely to choose the angry faces (p < 0.05).

The second experiment conducted by Pollak et al. (2000) asked the children to rate the similarity of expressions. Neglected children perceived less distinction between angry, sad, and fearful faces than did controls (e.g., angry-sad, p < 0.01; angry-fearful, p < 0.001). The neglected children’s difficulty in discriminating emotions may relate to their lack of exposure to a variety of emotions. Combining this with the abused children’s response bias for anger suggests that the extent of experience children have with the world is associated with their understanding of emotional expressions.

The response bias for angry expressions in the previous study led Pollak and Sinha (2002) to have physically abused children complete an identification task for facial emotions. Compared to normally developing children, the abused participants were equally accurate in identifying displays of all but one emotion: anger, which they over-identified (as opposed to the tendency of normal children to under-identify). This suggests that physically abused children have facilitated access to facial representations of anger, a plausible idea based on their hostile experiences early in life.

Aggression

In the Pollak and Sinha (2002) study, the maltreated children were observed as being more aggressive and hostile than the control children. Does a similar correlation between aggression and faulty emotion perception exist in adults? A study by Walz and Benson (1996) assessed the ability of men with mild to moderate mental retardation to label and discriminate facial emotions. In this case, the sample was split into two sub-groups: aggressive and non-aggressive, based on clinical observations. Although the results showed no differences in emotion recognition between the two sub-groups, the aggressive participants saw significantly more of the ambiguous expressions in a negative light; they labeled and recognized many as sad (mean number of errors for aggressive and non-aggressive at 1.89 and 1.09, respectively) or anger (mean number of errors for aggressive and non-aggressive at 2.06 and 1.09, respectively).

Another study (Larkin et al. 2002) took this finding a step further. Patients with clinical diagnoses of clinical hostility were given emotion recognition and labeling tasks. As with the previous example (Walz and Benson 1996), they mislabeled emotions to a more negative degree than did controls. For instance, significantly labeling happy as neutral (p < .03) and disgust as anger (p < .03) was common. The clinically hostile patients had a significantly higher number of errors than controls when recognizing happiness (p < .02) and disgust (p < .02). Because clinical hostility may be more severe than simple aggression, it is possible that the degree of aggression is associated with the ability to recognize emotions in faces. Further studies are needed to fully investigate this matter.

The factor of aggression may have a biological side too. One study (Best et al. 2002) involved patients diagnosed with Intermittent Explosive Disorder (IED). These IED patients were characterized by intense impulsive aggressive behavior. The researchers based their hypothesis on a case study (Blair and Cipolotti 2000) in which a patient with an orbital frontal lesion showed both impulsive aggression and a deficit in the recognition of anger and disgust. To investigate the relationship found in the case study in their larger sample, the experimenters had a sample of IED patients complete tasks that “putatively reflect functioning of the orbital/medial prefrontal cortex circuit,” including labeling and recognition tasks of facial emotion recognition. The IED sample was impaired (when compared with controls) at recognizing emotions of anger, disgust, and surprise (all at p <.025; see Figure 2). Additionally, they were biased to label disgusted faces with anger and neutral faces with disgust (p < .025) and fear (p < .05) when compared to the control group. The control group, on the other hand, was biased to label neutral faces with positive emotions, including happy (p < .05) and surprised (p < .1). These results would seem to present an initial connection between aggression, brain dysfunction, and emotion recognition. This provides a good transition to the biological perspective, which is primarily focused on brain functioning.

Figure 2. Subjects with IED made more errors and were biased to perceive negative facial expressions. On the left, the number of errors made by each group for four facial expressions are graphed. Subjects with IED made significantly more errors for anger, disgust, and surprise. On the right, the number of times subjects labeled neutral faces with each of the five expressions is graphed. Subjects with IED were more likely than controls (Con) to label neutral faces with disgust and fear. Source: Best M et al. (2002). Evidence for a dysfunctional prefrontal circuit in patients with an impulsive aggressive disorder. Proceedings of the National Academy of Sciences of the United States of America. 99:8448-8453.

Clinical Findings

Most physiological research concerning emotion recognition is based on clinical findings. Lesion studies have been crucial in eliminating and locating possible structures that control emotion recognition of faces. One study (Adolphs et al. 1996) asked 37 patients with damage to various parts of the brain to complete an emotion recognition task in faces. While all patients performed normally when the faces expressed happiness, there were significant deficits for recognition of negative emotions when compared to control subjects. The patients with these deficits were significantly more likely to have damage to the visual and somatosensory cortical sectors in the right hemisphere of the brain. Interestingly, patients with damage exclusive to the left hemisphere showed normal emotion recognition. The preliminary suggestion by the experimenters that the right hemisphere is the control center for emotion recognition is in direct conflict with the Sprengelmeyer et al. (1998) study that showed the left hemisphere to be important.

Adolphs et al. (2000) tried to examine this discrepancy further. Patients with lesions to either the right or left brain hemispheres participated in emotion recognition and identification tasks. Results revealed that right somatosensory-related cortices function together with the amygdala and the right visual cortices in retrieving emotions from faces. While lesions of the right somatosensory-related cortices were associated with impaired recognition for every individual emotion tested, impaired recognition of fear was also specifically and significantly associated with damage to the right anterior temporal lobe (p = 0.036). This study does not eliminate the role of the left hemisphere, however, since lesions to the left (and right) frontal operculum were associated with low performance scores on the emotion recognition task (p = 0.094). Nevertheless, the presence of an association between the frontal operculum (left and right), the low scores on a facial emotion naming task (p < 0.05), and the lack of an association on an emotion sorting task suggests that the frontal operculum may play a bigger role in lexical processing (language) than in the retrieval of conceptual (emotion-related but independent of language) knowledge.

Studies utilizing patients with more specific brain disorders have contributed as well. Parkinson’s Disease (PD) is essentially caused by damage to the basal ganglia. One study (Kan et al. 2002) examined the emotion recognition abilities of PD patients. They completed three discrimination tasks of emotion recognition: in faces, in voices, and in written words. Interestingly, the PD patients were found to have no problems on the voice and word tasks (when compared with controls), but significant deficits on the emotion recognition in faces discrimination task, specifically for fear (p < 0.01) and disgust (p < 0.05). The researchers concluded that the basal ganglia might play a key role in triggering recognition of emotional expressions in faces.

Epilepsy patients have also provided valuable insight. In one study (Meletti et al. 2003), patients suffering from temporal lobe epilepsy (right MTS type) participated in matching tasks for both faces and emotional expressions in faces. The epilepsy patients had significant deficits for matching the emotional expressions (especially fear: p < 0.001) compared to controls, but were normal when matching faces. These results indicate the importance of the antero-medial temporal lobe, particularly the amygdala, in emotion recognition in facial expressions. Perhaps most intriguing is the finding that the earlier the onset, the higher degree of emotion recognition impairment (p < 0.01). Future studies in this area could possibly pinpoint the ages at which seizures have their most profound effect on emotion recognition development.

A study by Boucsein et al. (2001) substantiated the above findings by testing patients with varying degrees of damage to the amygdala caused by lesions. The level of amygdala damage was found to be associated with the severity of the deficit of perception of emotional expressions (p < 0.05). This result is plausible because the amygdala has been thought to play a key role in emotion processing (Gur et al. 2002). While most theories have focused specifically on its role in fear perception (Morris et al. 1996), recent studies have indicated a broader role for the amygdala, extending to happy and sad expressions (Yang et al. 2002, 2003). Interestingly, the side of the lesion did not have an effect on emotion recognition, further muddling the debate over the dominant side of the brain in emotion recognition.

Principal Component Analysis

Like the facial component of focus, Principal Component Analysis (PCA) is a statistical technique that can be used to determine which groups of facial features elicit perception of individual emotions; in the case of PCA, this is achieved through the generation of computer models. Specifically, “PCA is used to identify a relatively small number of factors that represent the relationships among many inter-correlated variables” (Calder et al. 2001). In an initial PCA of facial expressions (which produces “eigenfaces” from the coefficients of the correlations among the faces), it was found that configural relationships between facial features play an important role in emotion recognition (Calder et al. 2000). In a follow-up study Calder et al. (2001) conducted a series of four experiments by applying pictures of faces to the PCA system and subsequently asking participants to identify the expressions on a computer. In one specific experiment, the validity of the category-based viewpoint was compared to the validity of the dimensional model. The category-based model theorizes that signals of a limited number of “basic emotions” are identified by activating discrete category representations. This contrasts with the dimensional model, a theory stating that human errors in recognizing facial expressions are not random, but actually form consistent and replicable patterns. For example, disgust is sometimes confused with anger, but never with fear, which is sometimes confused with surprise, but never with happiness. This pattern can continue for a long time. Studies have shown that both theories are applicable to the study of emotion recognition, meaning the debate will live on.

Calder et al. (2001) also addressed the components that lead to perception of facial identity (recognizing who a person is from their face) versus facial emotion. They found dissociation between identity and emotion when testing participants on the PCA models, indicating the use of different facial components in discriminating facial identity and facial emotion. For example, shape cues (eye width, jaw drop, etc.; see Figure 3) may be relatively more important than texture cues (wrinkles, shadows, etc.) in categorizing facial expressions, whereas the opposite is true in categorizing facial identity. The researchers suggested that the dissociation is due to different brain structures processing the visuo-structural properties of these two facial characteristics. They proposed that the lasting properties of a face needed to code facial identity are processed by the fusiform gyrus, whereas changeable aspects of a face (e.g., facial expressions) are processed by the superior temporal sulcus.

Figure 3. Sequences of reconstructed images for each of two eigenfaces important for categorizing facial expression. Source: Calder AJ et al. (2001). A principal component analysis of facial expressions. Vision Research. 41:1179-1208.

Situational Influences

The final factor affecting emotion recognition in faces is the basic situational influence. A study by Fernandez-Dols et al. (2002) asked adult participants whether an emotional facial expression was more plausibly associated with a social situation (two or more people communicating) or a non-social situation (one person communicating the same degree of emotion as in the social situation). They found a significant tendency for the participants to choose the social situation for every emotion (positive or negative). This suggests that social and situational influences may be triggers to perceiving emotions in facial expressions.

Finally, a combination of features to search for when trying to understand the emotions of others has been established. The specific features of the face and the social situation involved should be explored to determine what really triggers emotion recognition. If particular facial features or situations that best convey our feelings are pinpointed, behavioral techniques could then be developed to teach patients with deficits of emotion recognition.

As a diverse field with several perspectives, emotion recognition in facial expressions has been assessed through a variety of stimuli and methods. While many studies present pictures of faces to participants, others utilize recorded scenes on video of actors expressing emotions (both visibly and vocally). Although many different tasks have been used in these studies (discriminating, matching, looking, naming, sorting, etc.), they all measure the same basic construct: emotion recognition. Until one approach emerges as the established choice among researchers, the different methods will continue to be assessed very carefully.

Research on emotion recognition in faces has several exciting paths to follow, as seen by the questions left by the three perspectives. As it stands, the behavioral, biological, and cognitive views each play key parts in the motivation behind facial expression recognition. The future may not decide which perspective is most important, but it will hopefully define their roles further. As it continues, however, we will still do our best to read each other in every conversation, even if we cannot fully admit what we are looking for.