The worn-out idea of "stereotype threat" suffers another defeat in a failed replication, playing no role in women's lower level of political knowledge

Does Stereotype Threat Contribute to the Political Knowledge Gender Gap? A Preregistered Replication Study of Ihme and Tausendpfund (2018). Flavio Azevedo, Leticia Micheli, Deliah Sarah Bolesta. Journal of Experimental Political Science, March 16 2023. https://doi.org/10.1017/XPS.2022.35

Abstract: The gender gap in political knowledge is a well-established finding in Political Science. One explanation for gender differences in political knowledge is the activation of negative stereotypes about women. As part of the Systematizing Confidence in Open Research and Evidence (SCORE) program, we conducted a two-stage preregistered and high-powered direct replication of Study 2 of Ihme and Tausendpfund (2018). While we successfully replicated the gender gap in political knowledge – such that male participants performed better than female participants – both the first (N = 671) and second stage (N = 831) of the replication of the stereotype activation effect were unsuccessful. Taken together (pooled N = 1,502), results indicate evidence of absence of the effect of stereotype activation on gender differences in political knowledge. We discuss potential explanations for these findings and put forward evidence that the gender gap in political knowledge might be an artifact of how knowledge is measured.

Discussion

Ihme and Tausendpfund (Reference Ihme and Tausendpfund2018) have proposed that the activation of negative gender stereotypes accounts for the variance of the political knowledge gender gap. In our independent and well-powered direct replication, we find no evidence that activation of gender stereotypes affects participants’ performance in a political knowledge test. Indeed, we find evidence of absence of this effect.

We note that some elements of our study design diverged from the original study and could have contributed to the observed non-replication. Our study was conducted with American students and working adults, whereas the original study included German students. As the United States has achieved relatively lower gender parity than Germany in political empowerment (World Economic Forum 2021), one could argue that negative stereotypes about women might be more salient for Americans than Germans, undermining women’s cognitive performance even in the absence of stereotype activation (e.g., in the control condition). Although we cannot rule out that some populations might be more vulnerable to gender stereotyping than others, we have reduced cultural biases as much as possible by devising a political knowledge test that was – at the same time – similar to the one used in the original study regarding the level of difficulty, as our data suggest, and relevant to the American political context. A comparison of the effect of stereotype threat on gender differences in political knowledge across countries with varying levels of gender equality would be beneficial for a better understanding of potential cultural differences in stereotype threat. Second, as a direct consequence of including working adults in our sample, it was necessary to adapt the measure of field of study to encompass the field of work. We argue, however, that this should not have contributed to the unsuccessful replication. If our measure of field of study/work would inadvertently make participants aware of their affiliation with a Politics or Non-Politics group, the effects of gender stereotype activation on performance would presumably become more salient. Instead, our results show that the field of study/work did not influence the results (Tables S16–S17). An argument can be made, however, that the extensive list of topics in our study reduced participants’ self-identity with Politics. Nevertheless, adding participants’ attributed importance of Politics to their study/work as a covariate in the analyses did not change results (Tables S18–S19). We have also conducted further tests restricting our sample to young and educated adults to achieve a sample more similar in composition to the respondents in the original study, but we could still not replicate the effect of stereotype activation on the gender gap in political knowledge (Table S20).

We note that our failure to replicate the effect of stereotype threat on gender differences in political knowledge is consistent with recent research efforts challenging the effect of stereotype threat on academic performance more broadly. Stoet and Geary (Reference Stoet and Geary2012) showed that only 30% of efforts aiming to replicate the gender gap in mathematical performance do succeed. In addition, a meta-analysis investigating the effect of gender stereotype threats on the performance of schoolgirls in stereotyped subjects (e.g., science, math) indicated several signs of publication bias within this literature (Flore and Wicherts Reference Flore and Wicherts2015). Given these results, it is plausible that the effect of gender stereotype activation might be small in magnitude and/or might be decreasing over time (Lewis and Michalak Reference Lewis and Michalak2019).

Furthermore, we find robust evidence of a gender gap in political knowledge even after controlling for political interest. Our results validate previous accounts that the gender gap on political knowledge may be an artifact of how knowledge is conceptualized and measured and of different gender attitudes toward standard tests. In line with previous research stating that the political knowledge gap might be artificially inflated by a disproportionate amount of men who are willing to guess rather than chose the “don’t know” option – even if that might lead to an incorrect answer (Mondak and Anderson Reference Mondak and Anderson2004) – we find that female participants attempted to answer less questions and used the “don’t know” response option in the political knowledge test more frequently than their male counterparts whereas men guessed their answers more frequently than women, resulting in a larger amount of incorrect answers (Tables S8–S14). This suggests factors other than knowledge might contribute to the gender gap in political knowledge (Mondak Reference Mondak1999). For example, gender differences in risk taking and competitiveness (Lizotte and Sidman Reference Lizotte and Sidman2009) as well as in self-confidence (Wolak Reference Wolak2020) and self-efficacy (Preece Reference Preece2016) may lead women to second-guess themselves and be less prone to attempt answering the questions of which they are unsure. Meanwhile, higher competitiveness and confidence in males might lead them to guess and “gain the advantage from a scoring system that does not penalize wrong answers and rewards right ones” (Kenski and Jamieson Reference Kenski, Jamieson and Jamieson2000, 84). Measurement non-invariance, too, appears to detrimentally affect the interpretation and validity of political knowledge scales across several sociodemographics. For example, Lizotte and Sidman (Reference Lizotte and Sidman2009) and Mondak and Anderson (Reference Mondak and Anderson2004) have shown political knowledge instruments violate the equivalence assumption for gender, while Abrajano (Reference Abrajano2015) and Pietryka and MacIntosh (Reference Pietryka and MacIntosh2013) found non-invariance across age, income, race, and education. In our own replication attempt, we also found evidence of measurement non-invariance using item response theory and showed that the magnitude of the gender systematic bias appears to be contingent on respondents’ knowledge levels such that lack of equivalence by gender is stronger at average scores and weaker at the extremes of the political knowledge continuum (see Table S21 and Figure S1).

As Politics has been essentially a male-dominated field since its creation, it should not come as a surprise that current measures of political knowledge tend to favor what men typically know. Previous studies have shown that the mere inclusion of gendered items on scales of political knowledge lessens the gender gap (Barabas, Jerit, Pollock, and Rainey Reference Barabas, Jerit, Pollock and Rainey2014; Dolan Reference Dolan2011). The investigation and validation of measures of political knowledge that capitalize on the fact that men and women might not only know different things but also may react in different ways to standard tests is paramount for a more accurate understanding of the gender gap in political knowledge and its bias.

Finally, we note that measurement issues are not unique to political knowledge and in fact are pervasive in Political Science with consequences for how we measure populism (Van Hauwaert, Schimpf, and Azevedo Reference Van Hauwaert, Schimpf and Azevedo2018, Reference Van Hauwaert, Schimpf and Azevedo2020; Wuttke, Schimpf, and Schoen Reference Wuttke, Schimpf and Schoen2020), operational ideology (Azevedo and Bolesta Reference Azevedo and Bolesta2022; Azevedo, Jost, Rothmund, and Sterling Reference Azevedo, Jost, Rothmund and Sterling2019; Kalmoe Reference Kalmoe2020), and political psychological constructs such as authoritarianism, racial resentment, personality traits, and moral traditionalism (Azevedo and Jost Reference Azevedo and Jost2021; Bromme, Rothmund, and Azevedo Reference Bromme, Rothmund and Azevedo2022; Pérez and Hetherington Reference Pérez and Hetherington2014; Pietryka and MacIntosh Reference Pietryka and MacIntosh2022). If the basic measurement properties of widely used constructs are flawed, it is likely that insights from research will be biased. Valid, invariant, and theoretically derived instruments are urgently needed for the reliable accumulation of knowledge in Political Science.