David Goldenberg at “Gelf Magazine”:http://www.gelfmagazine.com has a copy of “the survey”:http://www.gelfmagazine.com/gelflog/archives/media.html#surveying_ivy_league_motherhood that Louise Story conducted as the basis for her “irritating”:https://crookedtimber.org/2005/09/20/mommy-tracking-the-ivy-leaguers/ “article”:http://www.nytimes.com/2005/09/20/national/20women.html?ei=5090&en=6a8e0c413c09c249&ex=1284868800&partner=rssuserland&emc=rss&pagewanted=all about Ivy League women and their plans for motherhood. Doing a reliable survey is hard, and by far the two biggest difficulties are sample selectivity (when the probability of participation is related to the outcome you want to measure: this a very tricky problem) and poor design of questions (where you look for what you want to find). Here are the first few questions from the survey, which was emailed to a group of freshman and senior women at Yale:
When you have children, do you plan to stay at home with them or do you plan to continue working? Why?
If you plan to continue working, do you plan to work full-time in an office, or full-time from your house, or part-time in an office, or part-time from your house? Why?

If you plan to stay at home with your kids, do you plan to return to work? If so, how old will you wait for your kids to be when you return?
Was your mom a stay-at-home mom? Explain whether she worked, and how much she worked! Were you glad with her choice (to either work or stay-at-home or whatever combination she did)?

At what age do you think you’ll have kids? How many kids do you want?
More commentary “at Gelf”:http://www.gelfmagazine.com/gelflog/archives/media.html#surveying_ivy_league_motherhood.
{ 1 trackback }
{ 26 comments }
Alex R 09.21.05 at 3:47 pm
When you have children???
Can we toss out this survey right from the first question?
P ONeill 09.21.05 at 4:06 pm
You should assign the analysis of this survey’s methodology to all the statistical experts who D2 keeps running into with the Lancet study controversy — the footsoldiers in the War on Sampling.
Timothy Burke 09.21.05 at 4:10 pm
Yeah, that’s a bad survey. It’s not a bad set of talking points for someone setting out to have qualitative, ethnographic conversations with people about their future plans, but as a survey on which one is prepared to make any systematic claims whatsoever, useless.
Uncle Kvetch 09.21.05 at 4:20 pm
Aw, you left out my favorite!
How do you think college-age men at Yale feel about whether wives should stay at home with their kids?
dp 09.21.05 at 5:38 pm
Agreed. Not a survey, but a poorly-disguised attempt to screen for particular answers, combined with a fishing expedition. Even the parenthetic comment after ‘Would you like me to keep your name anonymous?’ is far too convoluted as an opener. It would have alienated some respondents immediately. I’ve seen the consequences, having written something equally abstruse in my first questionairre.
Jay Currie 09.21.05 at 6:57 pm
Looking at the article it is hardly as if Story was running regressions on her results.
It’s a newspaper story which reflects the stories of the women Story interviewed. They said things which suggest that they think parents should raise their own children and that mums are not interchangable with nanies and daycare. What, pray, is worrying about that?
Erika 09.21.05 at 7:31 pm
Not only are the questions leading, but another thing that impacts survey response is response burden, the length of time that it takes to complete the survey. Not only are the questions leading, it might take hours to provide full answers to these questions. If I were a busy college student I’d take this survey and throw it in the garbage, unless I had an ax to grind on the subject.
JR 09.21.05 at 8:13 pm
It’s well-accepted that the only way to do surveys of this sort is with a human interviewer (in phone or in person) working from a script. Otherwise, self-selection kicks in.
In this case, the cozy, girl-talk tone of the questions (“kids” for children, “mom” for mother, the exclamation point) is an attempt at a false intimacy that would likely alienate more mature and more self-assured women while eliciting responses from less self-confident or more adolescent women.
Matt Weiner 09.21.05 at 8:40 pm
the exclamation point
Note that, though the words of the survey are quoted exactly, some of the punctuation (including that one) is original to Kieran.
Matt Weiner 09.21.05 at 8:41 pm
Should that be “original with Kieran”?
(Nice message for us too-quick posters.)
Kieran Healy 09.21.05 at 9:31 pm
Actually Matt, the exclamation point is in the version on the Gelf Magazine page, which is the only version I’ve seen. I copied it from there. Maybe it’s a typo.
Matt Weiner 09.21.05 at 9:51 pm
I’m just on drugs, then. Sorry. (That is super weird.)
Joey 09.22.05 at 1:05 am
I wonder why it is that the media in general (and NYT in particular) are so utterly obsessed with the tics and trends of undergraduate life in the Ivy League (and seemingly, Yale in particular) that they would actually run a story like this on page one. I mean, seriously — the quality of the piece was not especially inspiring; the author didn’t add much to the bare (and obviously very carefully selected) quotes.
Even that old Lisa Belkin NYT Magazine article was not as bad… although in that article, the interesting thing was to listen to Lisa Belkin trying to contort and twist her 30-something women who basically said “I really want to keep working and take care of my child too, but my boss wouldn’t let me” into “I don’t wanna work, I just wanna be a full-time mom.” These disappointingly unreflective Yale kids are _actually_ saying the latter, which makes me think maybe the quotes are more carefully screened than Lisa Belkin’s were.
Fabio rojas 09.22.05 at 8:56 am
I looked at Goldenberg’s comments. He said they got 60% response rate and worries about sample selection. This is typical, even decent, for mail surveys these days. This would be acceptable for many, if not most, social science journals.
There have even been a few articles in public opinion research journals suggesting low response rate doesn’t really change the answers much. The reason seems to be that in many cases response is a fairly idiosyncratic process.
As far as the questions go, they don’t seem so crazy:
“When you have children, do you plan to stay at home with them or do you plan to continue working? Why?”
How else would ask someone about their future plans? A simple open ended question like this seems to be a good way to at least get a sense of the answer. That’s my $.02.
Ragingpundits 09.22.05 at 9:15 am
The survey instrument is flawed but that doesn’t mean the results are incorrect. A 2001 report issues by Harvard University that tracked graduates over time showed similar results as did a fairly recent survey tracking the professional lives of several thousand gifted high school students.
Is this generation different? Perhaps. But have yet to see any evidence to suggest that it is. And my own experience is that it is not. In fact, it makes perfect sense. Insofar as the majority of ivy league students come from fairly privaleged backgrounds, this could simply be a case of wealth effects dominating substitution effects….
paul 09.22.05 at 9:45 am
If we assume that all the women who currently either intend to stay childless (or just haven’t decided whether to have children), by my experience a fairly large chunk of college women, were screened out either by themselves or the reporter, that would make a pretty big difference in the stats.
Another obvious flaw in the questions is that they appear to make work vs stay home a one-time choice. It might have been much more useful to frame things as “How long do you intend to stay out of the work force?” with possible answers ranging from, say, three weeks to indefinite. I’d say someone who had gotten to be a reporter at the NYT should know better, but then we’d all have to laugh.
theorajones 09.22.05 at 10:34 am
She didn’t get a 60% response rate. Her response rate is unknown, as is the group of women she polled (was it all freshmen/seniors, or a subset?).
I have to agree with Uncle Kvetch–my favorite question was, “what do you think the boys think about this?”
Beyond turning off all women who don’t structure their life choices around whether or not it appeals to men, the fact that she didn’t ask the boys is actually a huge failure from a social sciences perspective.
For all we know, there could have been a sea change in attitude at Yale, with everyone wanting to earn less money and have a lifestyle that included more time with the kids. That would have been a really interesting finding, with potentially large repercussions. I have no idea if that’s going on, or if something totally different is going on (maybe Yale grads all plan to adopt or live in communes or are all so wealthy they were all raised by nannies), and neither does she!
But everything in this “survey” was structured to fit into her already-written storyline: every woman (and no man) faces the same identical choice, and has only two options of how to deal with it.
That’s not research. That’s confirming your prejudices. This is such tripe, and it speaks poorly of the NYT that they’d put it on the front page.
mythago 09.22.05 at 10:50 am
As others have pointed out, the NYT runs “Rich women! Leave your jobs and stay home with your babies!” articles on a semi-regular basis.
The survey instrument is flawed but that doesn’t mean the results are incorrect.
This is about the funniest justification I’ve seen so far.
spencer 09.22.05 at 10:53 am
“When you have children, do you plan to stay at home with them or do you plan to continue working? Why?â€
How else would ask someone about their future plans? A simple open ended question like this seems to be a good way to at least get a sense of the answer. That’s my $.02.
The problem is not the open-ended nature of the question. Instead, it lies with the assumption that whoever is answering the question intends to have children. Yes, that “when” should probably be an “if,” but more important than that are the hints at the underlying bias of the interviewer.
Ragingpundits 09.22.05 at 12:31 pm
I’m not “justifying” the survey. As a survey instrument it is plainly useless. One question is what is wrong with the survey and a number of people here have pointed out the most egregious flaws. In my opinion, the more interesting question is: Regardless of the quality of the survey, is the author correct in her conclusions?
I merely pointed out that several longitudinal studies of highly educated women have produced a similar result.
Kai Jones 09.22.05 at 1:10 pm
Anybody note the heterosexual monogamous bias? Who cares what the men think, if you’re a lesbian who plans to have children?
Uncle Kvetch 09.22.05 at 2:31 pm
Anybody note the heterosexual monogamous bias?
Well, yeah; I just figured that went without saying. Lesbians aren’t “women,” silly, they’re lesbians! [/snark]
Uncle Fester 09.22.05 at 3:06 pm
Why is it a bad thing for women to choose to stay home? These are women with a choice in the matter.
God forbid they should do what they want.
Paul 09.22.05 at 3:12 pm
Regardless of the quality of the survey, is the author correct in her conclusions?
In other words, regardless of how crap her questions were, did she get the right answers?
I guess the world will never know.
Ragingpundits 09.22.05 at 4:19 pm
Or more generally, do a substantial number of female students at “elite” colleges want to stay at home as opposed to having a full-time career?
Judy Satin 09.25.05 at 12:16 pm
Jay Carrie asked what’s so bad about thinking that parents should raise their own children and that mums are not interchangable with nannies and daycare.
This anti-daycare website seems to agree…
http://www.daycaresdontcare.org
Comments on this entry are closed.