This is a follow-up to the distinctly non-sober but not wholly unuseful thread attached to my post on the Boston Review piece on Malhotra and Margalit’s survey research on anti-semitism and the financial crisis. The authors have asked for a chance to explain themselves, and their methodology, which has come in for a lot of criticism of an unavoidably speculative sort in comments to my post. Let’s hope this clears a few things up. Let’s try to be civil, shall we? The following is, obviously, not by me but by Malhotra and Margalit. And not edited by me in any way. – John Holbo
We are glad that our article generated thoughtful discussion, and we would be happy to address some of the questions people raised in the comments section. If our responses do not specifically address your particular comment, apologies in advance. Our goal here is to touch on some of the main issues.
First off, our objective in this article was scholarly, and was not to promote any particular public policy. Our main recommendation is that media outlets be careful in introducing religion and ethnicity into their reporting of the financial crisis, as this may have unintended consequences on various attitudes. Additionally, we show that the number of people who explicitly provided a response blaming “the Jews” was extremely high. We are not claiming that the point estimate of 25% is the exact percentage of prejudiced people in the population. Indeed, statistical uncertainty by itself bounds the estimate between 21-29%. Additionally, variants in question wording will of course lead to somewhat different figures. Our main point is that this figure is surprisingly high, and we were alarmed that such a high percentage of respondents would be willing to reveal this. Moreover, the survey experiment is truly the main finding in our article, and we should have ensured that this was more prominently featured. We find that associating Madoff with his ethnicity has downstream policy consequences on topics having nothing explicitly to do with anti-Semitism. Lastly, our advisors in graduate school once told us that conducting survey research is like performing surgery in a sewer. It is a very complex, messy undertaking because measurement is so difficult. Hence, we want to assure people that we have leveraged our extensive and rigorous training in public opinion research and survey methodology to do our best to study the question at hand. Reasonable people can have methodological disagreements, but keep in mind that most alternative approaches will introduce their own set of biases. For example, if we had used “Jews” instead of “the Jews,” it could easily be said that many Jews were involved in the financial crisis and that our question was not truly tapping ethnic prejudice.
We also address a few more technical details below:
1. Some people asked whether the difference between Republicans and Democrats is robust to the inclusion of various controls (education, region, gender, race, income, etc.). We find that the difference between Democrats and Republicans is highly robust to these controls when: (1) estimating a multivariate regression model; and (2) conditioning on these variables. That being said, the gap between partisans is definitely a correlation, and it not causal. But it is a highly robust correlation.
2. Some people raised some questions about how the overall blame attribution question was asked. No survey item is perfect, and you have to choose the one that is going to bias your findings in the reverse direction from your alternative hypothesis. The question we asked has been used previously, and generally underestimates the level of racial prejudice. We were very surprised that this ended up not being the case, most likely due to the anonymous Internet survey mode (as opposed to phone or face-to-face interviews). Second, any biases will likely only change baseline levels of blame, and not differences in blame between various population subgroups. And it is these subgroup differences that we are most interested in.
3. Some people wondered if people would be so disgusted with the question that they would not answer. Non-response is not an issue in this dataset. Only 0.7% of respondents refused to answer this question.
4. Some people wondered if the fact that the survey was anonymous made it less valid. Quite the opposite. Anonymity increases response validity because people do not feel a need to provide socially-desirable responses to a live interviewer. For example, in face-to-face and phone surveys, reported voter turnout rates are about 80-90%, far exceeding the 50-60% of people who actually turn out to vote. In Internet surveys, the reported turnout rates are closer to 60-70%. I think this comment had to do with some confusion over how the survey was conducted.
5. Some people asked about the representativeness of Internet surveys. The sampling design was not a random probability sample. Rather, the respondents were part of SSI’s online panel. Several researchers have argued that Web surveys can approximate the representativeness of those conducted via random digit dialing. See Clarke et al.’s paper in Political Analysis in 2007. This was not an anonymous Internet survey like the ones you see on news websites, however. Rather, panelists were invited such that the overall distribution would be representative on a host of demographic variables. As one reader pointed out, this is somewhat similar to the approach used by YouGov/Polimetrix, a highly-respected firm used by both academic researchers and major corporations. The representativeness of this survey on variables such as gender, age, and party identification approximate distributions from the 2008 American National Election Study, which used a face-to-face probability sample. Also, it is important to note that all surveys are opt-in, and many of the RDD telephone polls you read about in the media have response rates of 5-10%, meaning that we don’t observe the responses of the 90-95% of people who chose not to participate. Hence, random sampling is not a panacea for representativeness. Finally, the internal validity of the survey experiment is not compromised by a non-random sample, since the treatment itself was randomized. This is no different from medical or psychology studies that use convenience samples to assess the effects of various treatments.
6. Some people asked whether we asked people to assess blame for other groups in addition to “the Jews.” We asked them about some other groups: loose government regulations, Wall Street financial institutions, people who took out mortgages they could not afford. Unsurprisingly, people blamed these groups much more than “the Jews.” But the difference between Republicans and Democrats flipped for borrowers, which is the crucial point. That suggests that it is not simply that Democrats are blaming everyone more. Regrettably, we did not ask them to evaluate any other ethnic groups. That would be a very useful test to assess the robustness of the results, and we hope someone will do that.