Eyes and IQ: A meta-analysis of the relationship between intelligence and “Reading the Mind in the Eyes”

Highlights

• Meta-analysis finds relationship between RMET performance and intelligence.

• Contrary to previous assumptions, RMET performance is influenced by intelligence.

• It is important to control for intelligence when using the RMET.

• Verbal and performance IQ contribute equally to RMET performance.

Abstract

Although the Reading the Mind in the Eyes Test (RMET; Baron-Cohen et al. 1997, 2001) has been used as a measure of mental state understanding in over 250 studies, the extent to which it correlates with intelligence is seldom considered. We conducted a meta-analysis to investigate whether or not a relationship exists between intelligence and performance on the RMET. The analysis of 77 effects sizes with 3583 participants revealed a small positive correlation (r = .24) with no difference between verbal and performance abilities. We conclude that intelligence does play a significant role in performance on the RMET and that verbal and performance abilities contribute to this relationship equally. We discuss these findings in the context of the theory of mind and domain-general resources literature.

Introduction

As socially intelligent beings, we make inferences regarding others' mental states during social interactions. Recently, many researchers interested in the study of mental state understanding have explored mechanisms across development that may contribute to individual differences among both healthy adults (e.g., Billington, Baron-Cohen, & Wheelwright, 2007) and relatively high-functioning individuals with disorders. The study of individual differences among adults, with and without disorders, has given rise to a demand for instruments with much greater sensitivity than traditional developmental tasks. One such instrument, the Reading the Mind in the Eyes Test (RMET; Baron-Cohen et al., 1997, Baron-Cohen, Wheelwright, Hill, Raste and Plumb, 2001), has been used in more than 250 studies (Kirkland, Peterson, Baker, Miller, & Pulos, 2013).

In the RMET, participants view photographs of eyes disembedded from the face and are asked to make a forced-choice as quickly as possible among four mental state descriptors. An important strength of the RMET is its sensitivity to individual differences in adults. For example, the RMET has been used to discriminate among parents of children on the autism spectrum (i.e., the broad autism phenotype) and IQ-matched typically developing parents (Baron-Cohen and Hammer, 1997, Hurley et al., 2007). Recently, RMET performance has been associated with aspects of academic profile and professional identity. Students in humanities outperformed those in the physical sciences (Billington et al., 2007), and those choosing certain career paths (e.g., visual arts and law) scored above average on the RMET, whereas those in computer and information technology careers did not (Strong, Russell, Germine, & Wilmer, 2011). While a number of instruments (e.g., neuropsychological test batteries that screen for disorders) have adequate sensitivity for discriminating between disordered and typically-developing samples, the RMET has been especially successful as an individual differences instrument for use with normally developing adults. To date, the RMET has been used in over 250 studies in a wide variety of disciplines (e.g., business, economics), translated into several languages, adapted for use with children (e.g., Baron-Cohen, Wheelwright, Spong, Scahill and Lawson, 2001, Hallerback et al., 2009) and adapted for brain imaging (Adams et al., 2010). Despite its widespread use, the underlying cognitive processes mediating performance on the RMET have been minimally examined. We conducted a meta-analysis to explore the contribution of intelligence to performance on the RMET.

The goal of understanding the contribution of general intelligence on RMET performance is important for two reasons. From a methodological perspective, it is important to appreciate the degree to which performance differences reflect a specific social cognitive process rather than general intelligence. To date, the RMET has been used with many different clinical groups including, for example, autism and schizophrenia (Hallerback et al., 2009). Of course, such groups may differ from comparison participants on factors associated with general intelligence. If RMET performance relates substantively to general intelligence, studies should account for possible group differences when drawing conclusions about social cognitive impairment; however, not all previous studies have adequately controlled for this possibility (e.g., Bora et al., 2007, Demurie et al., 2011). A similar criticism is relevant in studies of normally developing individuals. For example, Bailey and Henry (2008) used the RMET to investigate social understanding among older and younger individuals without disorders and did not account for potential differences in general intelligence. A recent meta-analysis (Murphy & Hall, 2011) highlights the possibility that individual differences in psychosocial functioning may reflect general intelligence. From a more theoretical perspective, an exploration of the task demands of the RMET, specifically with respect to intelligence, may inform the degree to which the task taps a relatively implicit, automatic social–cognitive process. In introducing the current revised version of the RMET, Baron-Cohen, Wheelwright, Hill et al. (2001) did not obtain a significant correlation between intelligence and the RMET, “suggesting this is independent of general (nonsocial) intelligence” (p. 247), consistent with the view that performance reflects relatively “unconscious, rapid and automatic” (p. 241) judgments. Consistent with Baron-Cohen's original description, it makes intuitive sense that the RMET would be relatively free of general intelligence demands as compared to other tasks (e.g., Strange Stories) which, by design, involve explicit verbal reasoning. Evidence for such a differential loading of intelligence on the RMET relative to other instruments such as the Strange Stories Task (Happe, 1994) would be consistent with recent speculation about a dual route model of mental state understanding (e.g., Apperly et al., 2007, Sabbagh, 2004). This model proposes that mental state understanding involves both an implicit, automatic, and inflexible process that operates independently of an explicit cognitively demanding flexible process. The proposal that RMET performance is independent of general intelligence would suggest a process that is relatively more implicit and does not require explicit, cognitive demands.

A second issue of current theoretical interest concerns the degree to which there exists an independent, somewhat-insulated mechanism that mediates social understanding (e.g., Adolphs, 2006, Leslie, 1987). Clearly, the resolution of such a complex and theoretical question will require convergent support across a range of methodologies. This is particularly true because a somewhat modular process may “co-opt” more general mechanisms in the service of social understanding (Siegal & Varley, 2002). Likewise, an implicit social cognitive process may operate in conjunction with more explicit processes. Thus, our meta-analysis cannot provide a clear refutation of models of social cognition that posit either a somewhat modular mechanism or dissociable routes. However, meta-analytic evidence that RMET performance is relatively free of demands on general intelligence might be interpreted as support for a relatively implicit and somewhat modular mechanism. In our review that follows, we first consider the study of mental state understanding that gave rise to the RMET, and then we briefly review the subtopic of intelligence. Finally, we consider the intersection of the RMET and intelligence to frame our hypothesis.

Since the inception of the study of theory of mind (ToM; Premack & Woodruff, 1978), the “false belief task” has served as the gold standard for demonstrating ToM, a cognitive milestone of the preschool years (Wellman, Cross, & Watson, 2001). Specifically, the first-order false belief task, which requires understanding the belief another person holds was extended to the second-order false belief task where the participant demonstrates understanding of a belief someone holds about the belief of another person (e.g., Suzy believes that Ray believes that the cup of coffee is hot). However, as evidenced by adult participants on the autism spectrum who manifest pervasive social difficulties while passing false belief tasks, the study of individual differences among older children and adults, with or without disorders, requires instruments with greater sensitivity. ToM tasks designed to measure individual differences in adulthood have differed with respect to task characteristics. While some tasks clearly involve explicit, language-based reasoning, others may place a greater emphasis on perceptual processing (e.g., reading facial expression). For example, the Hinting Task (Corcoran, Mercer, & Frith, 1995) consists of scenarios in which one person hints to another person (e.g., “I want to wear that blue shirt but it's very creased”). The participant must explain the meaning underlying what the character in each story says or does. Similarly, in the Faux Pas Task (Stone, Baron-Cohen, & Knight, 1998) participants identify the social faux pas that occurred in the scenario. Alternatively, other tasks have been designed to capture the relatively more implicit (i.e., less linguistically mediated) process of quickly judging mental states based on brief exposure to perceptual information. Such tasks (e.g., Reading the Mind in the Voice Task, Profile of Nonverbal Sensitivity, RMET) involve decoding nonverbal behavior conveyed in facial expression, body movement, or voice.

In comparison to instruments like the Strange Stories Task or the Faux Pas Test, it makes sense to hypothesize that the RMET would rely relatively less on general intelligence given that it would seem to involve a more implicit social–perceptual analysis. Although one must select a verbal descriptor, we assume this is less linguistically demanding than inferring a mental state based on an analysis of sentence meaning. However, to date we are not aware of any evidence to support or refute this hypothesis.

Intelligence is generally thought to be a reasoning capacity that allows one to learn from experience and adapt to changing environments (Henry, Sternberg, & Grigorenko, 2005). Frequently it is operationalized through the assessment of intelligence (g), and practically it refers to the individual variation we find in mental competence (Hunt, 2005). Tests such as the Wechsler Adult Intelligence Scale include measures of verbal ability and performance ability to capture both crystallized knowledge and fluid reasoning skills, respectively. Intelligence correlates with many real-life outcomes throughout the lifespan (Wilhelm & Engle, 2005), such as income, educational attainment, occupational status, and personality characteristics (see Brand, 1987, Kline, 1991). In many research investigations, intelligence accounts for a significant portion of the variance in the dependent variable (Brand, 1987, Jensen, 1998, Strenze, 2007, Woodward and Fergusson, 2000), but there is controversy as to the extent that intelligence significantly contributes to all mental capacities (Henry et al., 2005, Jensen, 1998).

While tests of intelligence typically focus on more cognitive aspects of intelligence, a number of researchers have speculated that there may also be individual differences in social–emotional intelligence (Matthews, Zeidner, & Roberts, 2005). The concept of “emotional intelligence” has led to a proliferation of research on one's ability to perceive, express, and reason with emotion. A meta-analysis found a correlation of .22 between emotional intelligence and general mental ability (Van Rooy & Viswesvaran, 2004), and correlations have been reported between emotional intelligence and a number of real-life outcomes (for a review, see Mayer, Roberts, & Barsade, 2008). As a result, there is a growing appreciation of the importance of social ability relating to a variety of outcomes (e.g., McGlade et al., 2008).

In a meta-analysis with adults and adolescents, Murphy and Hall (2011) reported a correlation of .19 between intelligence and interpersonal sensitivity (i.e., the ability to decode the state or trait of unfamiliar others); however, this analysis included only 36 studies published between 1931 and 2006 using a wide range of “interpersonal sensitivity” tasks requiring participants to judge emotions, intentions, or thoughts being portrayed by actors through visual and auditory modalities (e.g., Profile of Nonverbal Sensitivity, Diagnostic Analysis of Nonverbal Accuracy, Interpersonal Perception Task). Studies using the RMET were not included in this analysis. While tasks such as the Profile of Nonverbal Sensitivity and Diagnostic Analysis of Nonverbal Accuracy are used in some current research (e.g., Ingersoll, 2010, Wynn et al., 2010), the RMET has been used in hundreds of studies with both neurotypical and clinical samples and continues to be used in a wide variety of studies. Given the widespread acceptance of the RMET as a broad index of mental state understanding, an investigation of the degree to which the instrument involves intelligence is warranted.

In past research, traditional ToM tasks have yielded correlations with intelligence varying from weak negative correlations to moderate positive correlations. For example, in schizophrenia research, first-order and second-order false belief tasks correlated at .30 and .24, respectively (Bora et al., 2007) and performance on the Hinting Task correlated with verbal ability at .54 (Bora, Eryavuz, Kayahan, Sungu, & Veznedaroglu, 2006). The Strange Stories Task (SST) correlated at .28 in a group with schizophrenia and .45 in relatives of individuals with schizophrenia; similarly, the Faux Pas Task correlated at .20 in a schizophrenia group and .65 in relatives (de Achaval et al., 2010). Among individuals with autism spectrum disorders in the normal range for IQ, SST performance correlated with intelligence at .26 (Adler, Nadler, Eviatar, & Shamay-Tsoory, 2010) and .19 (Dziobek et al., 2006). In typically developing samples, SST performance correlated at .18 (de Achaval et al., 2010) and .22 (Dziobek et al., 2006), while Faux Pas Task performance correlated at .24 (de Achaval et al., 2010), and reality known and unknown ToM tasks correlated at .40 and .34 (Bailey & Henry, 2008). Further muddying the literature regarding the relationship between intelligence and ToM task performance, some correlations have been reported with combined groups consisting of clinical and nonclinical groups, which introduces increased heterogeneity. For example, Hinting Task performance correlated at .54 in a combined normal/autism spectrum disorder/delusional sample (Craig, Hatton, Craig, & Bentall, 2004) and SST at .50 in a mixed control/autism spectrum disorder sample (David et al., 2008).

Clearly, a meta-analysis is needed in order to more precisely estimate the relationship between intelligence and ToM. A review of the literature also indicates there is a lack of task-specific meta-analyses. Given the diversity of tasks included under the ToM umbrella, it is important we examine the relationship of intelligence with each type of task independently. The insights gained from this task-specific approach may offer theoretical insight into the diverse processes that contribute to overall ToM ability. Further, this approach may have practical benefit for interpreting studies involving use of the RMET with atypical samples that are likely to differ on aspects of both social and nonsocial abilities.

Given that the relationship of intelligence with ToM tasks is presumed to involve more explicit processes (e.g., SST, Faux Pas Task), it makes sense that the RMET would also correlate to some extent with intelligence (though perhaps to a lesser extent than traditional ToM tasks). In a meta-analysis conducted in our lab, the RMET correlated with Strange Stories and Faux Pas Tasks (r = .29; Kirkland, Baker, Johnson, Peterson, & Pulos, 2012), indicating the tasks may be measuring some underlying common ability or share task demands or a combination of both. Some authors have assumed there is no correlation between intelligence and RMET performance. For example, referring to the RMET, Mar, Oatley, Hirsh, dela Paz, and Peterson (2006) wrote “scores on this test do not correlate with IQ” (p. 701). However, a number of studies have, in fact, found a relationship between the two. While it is not typically the primary investigation of authors (many do not even report the correlation in their publications), we have included 77 correlations between RMET performance and intelligence in this meta-analysis to estimate the extent of this relationship.

If RMET performance does correlate with intelligence, would verbal or performance ability contribute more to this relationship? While many studies have shown a relationship between theory of mind ability and language ability in children, this is often attributed to the linguistic task demands (e.g., Lewis and Osborne, 1990, Milligan et al., 2007). However, Barrett, Lindquist, and Gendron (2007) presented evidence to support the hypothesis that language supports emotion perception in a context-rich, top-down process. Beck, Kumschick, Eid, and Klann-Delius (2012) demonstrated a moderate relationship between language ability and emotional competence in children through confirmatory factor analysis. In a study where children were asked to choose which drawing of face emotions best fits a situation evoking emotion (e.g., receiving a birthday gift), language ability explained 27% of the variance in emotion understanding in children after controlling for age (Pons, Lawson, Harris, & de Rosnay, 2003). In a recent study involving an adult sample, a significant correlation was found between RMET performance and verbal ability (r = .49) but not performance ability (r = .18, ns; Peterson & Miller, 2012). Thus, it makes sense to predict that verbal ability may play a larger role than performance ability.

We conducted a meta-analysis of the relationship between RMET performance and intelligence. Upon examining the literature, it is clear that the correlations reported between RMET performance and intelligence span a large range. Because of this variability among the correlations, a meta-analysis was necessary to precisely estimate this relationship in the general population and examine variables contributing to heterogeneity across studies. Our investigation focused on (a) the extent to which intelligence correlates with RMET performance; (b) the contribution of verbal ability relative to performance ability; (c) variables moderating the relationship between intelligence and RMET performance. We hypothesized that the RMET would have a relatively lower correlation with intelligence than more explicit ToM tasks, that verbal ability would influence RMET performance more relative to performance ability, and that moderators would not influence the relationship between intelligence and RMET performance.

First we examined whether studies using only tests of verbal ability or performance ability or both could be combined in the analysis. Next, we combined all studies and analyzed the overall relationship. Finally, we examined whether or not any study variables would influence the results or if the overall effect size would hold up across different types of studies. Studies varied on a number of factors, and these were examined as moderators when appropriate. Since the RMET is widely used, it has been translated into different languages, administered in several countries, and used with various populations. The purpose of the meta-analysis was to aggregate the correlation across these studies, but it is vital that we examine these potentially confounding factors for effects on RMET performance. When a modified version of the RMET is used, it is typically not substantiated with a psychometric study, so we can only assume these task characteristics have not been examined. Thus, we do not know if the language of administration or modifications to the RMET have any impact on performance. We recognize that these factors may influence the results and warrant examination. Thus, the following moderators were examined: (a) verbal ability versus performance ability; (b) Wechsler Intelligence Tests versus other tests of intelligence; (c) English version versus translated versions; (d) revised version versus altered versions (e.g., Bora et al. (2006) used 27 items only; Botting and Conti-Ramsden (2008) read items aloud to participants); (e) studies conducted in the UK, where the task originated (38%), versus other countries; (f) typically developing samples versus those with disorders (e.g., autism, schizophrenia); (g) adult performance versus performance in children; (h) year of publication.