Could Busting Unions Fix America’s Schools?

I find it very revealing that those in favour of school reform routinely focus their attention on teacher unions as the source of the problem. If teacher unions were the problem than students in countries that are more highly unionized, for example Canada, Sweden, Denmark and France, should be doing worse. Since these countries are doing much better than the United States on international comparisons I think it may be safe to say that unions are not the problem.

I’m waiting in joyful hope for the day when a spike in fires or crime results in a chorus for the abolition of the police or firefighter unions. Funny how poor performance in those professions never seem to bring out these sorts of demands. I wonder what the difference might be…

There not their.
Sorry.

There are schools in this country that I would send my children to.
There are schools in this country that I would choose private (even Catholic) schools over for my children.
There are schools in this country that I would beg, borrow, steal, and kill to get my children away from.

I’m sure that all of us would agree with each of those.

I’m also pretty sure that all of us send our above average children to above average schools… and a change in the status quo would negatively affect us (and our children).

It really may be that for these kids improving their math skills somewhat and their reading skills slightly is the best thing for them (though, whereas there is a correlation between test scores and later success, we don’t have any evidance that improving children’s test scores improves later outcomes, and we have lots of evidence that these bumps in test scores from grade-specific interventions typically fade pretty quickly).

I suspect this has something to do with all those studies that say kids are mostly influenced by those around them, which is to say, parents and teachers earlier, other kids later. In any event, I won’t stop saying this: the people most responsible for their kids academic performance are the parents. That’s one group these “responsible conservatives” don’t want to take on.

When asked the best way to improve a school, Daniel Patrick Moynihan “offered the policy proposal that states wishing to improve their schools should move closer to Canada”.

On the other hand, it certainly stands to reason that a student who cannot read, or who reads at a grade level several years lower than their actual age, is going to have a significant barrier in learning, well, anything else along their academic career that the student who is reading at their own or a higher grade level will not.

There’s probably a similar, although not quite as strong, case with regards to learning math and acquiring a working grasp of rigorous logic and problem-solving skills.

(If it is the tests that are the problem, and they are not actually measuring practical reading and math skills, shouldn’t the favored approach be to go and make a better test rather than to throw up our collective hands and claim that the only way to solve the problem is to throw more money than will ever plausible actually be thrown at it at it?)

Lots of interesting comments. My wife is a teacher and we’re going to end up seeing the movie eventually because it’s gained the attention of so many parents.

A few of my wife’s fellow teachers have found WFS to be a bit adversarial, though. The existing stereotype may be fossilized teachers with tenure, but around our neck of north Seattle, teachers work for low-to-middling pay, put in lots of overtime, and are blamed for damned near everything. Many teachers do what they do because they are committed. Even if they are in error, the U.S. isn’t exactly doing a super job of building trust.

Which is not to say that everything’s going swimmingly, because it’s not. But there are aspects of a child’s education which we don’t care to seriously discuss because magic worlds like “socialism!” are uttered. Habits, diet, and home-life fall under that fear. The last of those three is among the most important, too.

We clearly have different ideas of what education should be, how success is measured, and what to focus on. My worry is that we’ll treat “charter schools” as yet another magic word that solves everything. You know, the way we hitch our notions to fad diets to lose weight.

Then it could fail, we’d lose faith, and feel more helpless. Charter schools may be a super duper awesome idea, but pushing it via a movie starring an administrator that boasts about how many teachers have been fired is a thoroughly lousy sell.

“The union busters’ case seems to rest on the claim that American teachers and/or American teachers’ unions are somehow intrinsically worse than those of other developed countries. Maybe that is true but I have never seen an explanation why that would be the case.”

To start the conversation do we need ‘why’? First we need to agree that it is true, if it is. It seems to me that it is true, but if can’t agree on that, we probably need to start there.

I would say that American teacher’s unions, and American teachers, are probably not much different than those in other countries.

What is very different is the American school system.

In France (and it’s my understanding that this is pretty typical in Europe), publicly schools are much less monolithic than in the US. There are publicly funded religious schools, schools for various levels of academic achievement, and so forth.

If you think that the US schools are probably as good as you can get with a one-size-fits-all solution, that the teachers union is going to provide pressure for stability, and that a more varied set of options would be better than a one-size-fits-all solution (all of which I would think), then the teachers union will be problematic in the US in a way that they won’t in Europe.

Americans, especially conservative Americans, are deeply conflicted about education. They want the benefits of having an educated population because even the most moss-clad reactionary is in favor of the money and military power that go along with technology, but they resent educated people as elitists. We want education to be cheap and not threatening to our cultural prejudices. Above all, the last thing we want to do is respect teachers. Of course there are and always have been strands in our tradition with the opposite tendency, but they tend to be associated in the popular mind with the despised Europeans. Over and beyond hatred for unions, the tendency to blame the supposedly wretched state of our schools on teachers arises from American anti-intellectualism.

One obvious way to improve our educational performance would to emulate the attitudes of peoples such as the Jews, Chinese, and Japanese, who love and treasure learning. If that prescription seems hopelessly impractical, I have to ask why it is any sillier than the suggestion that mediocre teachers are automatically going to be replaced by better ones if only we disbanded the unions.

It’s perfectly clear what the union busters want to do. Their goal is to hire lots of young teachers, preferably without family, who they can pay badly and get to work very long hours – and then when those people burn out, or even insist on working only the hours they are paid for, they want the right to fire them. This will save loads of money they can then give away to their rich friends or spend on their pet wars or both.

Anyone who thinks they actually give a d***n about learning is deluding themselves.
If they did, they would at the very least take notice of the little research that is out there.

Whoops. Screwed up the html tag there. Sorry.

To think the “education reform” movement is sincere, you have to think that rightwingers agonize over the plight of black children. And you have to think that middle class parents are agitating to improve education so that millions of poor children will now be able to compete with their own children for good jobs after school.

Anyone really think this?

By the way, the whole education-reform wave reminds one of nothing so much as the late-Victorian push by women’s rescue and so on organizations to reduce prostitution by housing them in shoddy hostels and making them work in badly paid textile jobs. It’s completely hopeless.

as if the entire point of public education is to produce good little workers to stamp out bits of GDP

But that is the point of public education. Ask any ex-CEO of an oil company!

I wonder how much of the motivation is political rather the being based on a desire for better outcomes, especially for lower class students. I can recall circa 1980 thinking teachers unions were simply fronts for communist revolution. I don’t ever recall, anyone arguing against this point of view, so probably the characterization of teachers unions, and to a lessor extent, any unions, as organizations intent on forcing leftwing viewpoints onto society are probably still pretty rampant, -especially on the right. So an indirect attack on them isn’t exactly unexpected.

To a significant demographic, school and curricula, is a means of waging culture ware, first, and a means of helping young people prepare for life second.

We do have issues, with bureaucracy and funding mandates however. My son, who did have some problems, was kept in special ed a lot longer than necessary -we suspect that the desire to collect state funding strongly biased their judgment.

There are very valid reasons to focus on teachers rather than the factors outlined in Rothstein’s article, namely that teachers are a controllable factor (in districts that have had their unions busted) and that mass unemployment and social ills are so multifactorial and poorly addressable that they are definitely higher hanging fruit.

Here’s the problem: K-12 teachers are something like 2 or 3% of the workforce. That’s a lot of people. I think it’s highly unlikely that, absent major structural changes in the incentives to become a teacher, you could replace a significant chunk of those teachers with smarter/more competent/more awesome people, especially in poorly run schools that serve high-needs populations. There’s not exactly a line out the door to work 80 hour weeks at a low status job for $50,000/year with a lot of people yelling at you. You can get different people, but they will either be new, uncertified teachers who leave the poorly run school as fast as they can (i.e. once they get their teaching certificates and have enough experience to get hired at a charter or better public school) or experienced teachers who were unable to get a better position. Even if we entirely ignore the problem of having very little idea in advance who will or will not be a good teacher, that’s not a particularly fabulous pool. Teacher quality is way, way more difficult to fix than most people think. Giving teachers ongoing feedback or time for collaboration or some other structural assistance might actually be much easier and more effective, because it doesn’t depend on magically uncovering a group of amazing teachers just waiting to work very hard for not that much money under difficult circumstances.

Still, the comparison to fad diets is a good one. Certainly there are many things about changing teacher work rules that might plausibly affect student success later in life, as there are many dietary changes that might affect health. We don’t know that test/scale numbers are a good marker for long-term success/health, and we don’t even know if work rules changes/particular fad diets reliably move the marker.

As a Crooked Timber reader who plays a leadership role in American teacher unions, I would be interested in some specificity to the general talk about “stupid and backward union structures.” That sort of broad brush characterization seems to me be partaking in the discursive “common sense” of the corporate attack on public sector unions in an unquestioning way, without ever interrogating it.

Please address the rubber room, New York City. Again, I don’t actually think getting rid of rubber rooms makes that much difference, but the rubber rooms are there.

Sebastian 21 says it is obvious that American teachers are worse than other countries’. It’s nice that somebody owns up to that statement. Do you also think that American doctors are bad? Scientists? Engineers? Lawyers? Plumbers? Mechanics? Farmers? Firefighters?

Are Americans just dumb in general or is there something particular that draws incompetents into the teaching profession? If that is so, then shouldn’t we identify what it is?

“teaching to even a moderately good math test is teaching math”

Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests. Maths proficiency has to be tested by doing maths. Which is probably true for any subject.

But there are aspects of a child’s education which we don’t care to seriously discuss because magic worlds like “socialism!” are uttered. Habits, diet, and home-life fall under that fear. The last of those three is among the most important, too.

Just about every teacher I know has this same impression. We can talk about any kind of “reform” we want to and “and all the options are open”, just so long as the teacher’s union is weakened and the neglect of the parents and incompetent craven administrators are not mentioned.

But in the half-decade since I graduated, it’s gone downhill quite rapidly. Why? Most signs point to a severely incompetent and at times actively malicious administration.

From what I’ve seen and heard, I’d say about 80% of this is due to caving in to hostile parents and school boards made up of hostile parents.

Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests. Maths proficiency has to be tested by doing maths. Which is probably true for any subject.

Yeppers. Got it in one, especially that last sentence.

Principals can perfectly well get rid of teachers in their probationary years, but because most know nothing about instruction, and the rest don’t see improving instruction or managing human capital as any part of their job, they often don’t even bother observing. Get rid of the teachers unions, and teachers are subject to the arbitrary power of people who know nothing and have less than no interest in what really matters in the school.

Thank you, Harry.

Sebastian, again, if by “it is my impression …” you mean to say “I have no clue whatsoever but I am guessing …”, please don’t bother. If you have specific knowledge about how teachers unions in the US differ from those in other countries, let us know.

Sebastian, you seriously think that teaching math tests using Scantron is a good idea? You would be alone in that belief. Universities do it because it’s cost-effective, not because it’s good pedagogy. You’ll say anything to stick it to the teachers, won’t you?

Myles is a fine example of the reason why it’s all about teacher’s unions: because teachers unions are a defined evil. The charge has been repeated the requisite number of times to become fact among people who deal with such facts. It’s a big, fat, ‘all of the above’ irony that the people striving in this particular battle of the culture war are supposed to be educators is not relevant. Hence the famous ‘rubber room’ in New York, that single apocryphal place full of apocryphal terrible teachers paid full salary to sit around safely not harming kids–can stand for the entire field of education, can carry the entire argument over whether teachers unions are evil.

My union is occasionally exasperating, and occasionally a bit slimy, but a necessary fact of life in the litigious and ugly world of American public education where culture warriors routinely urge parents to blame teacher for their childrens’ shortcomings (and, occasionally, blame them for their successes in, say, grasping evolution). The best teachers are the most vulnerable, since it is the best teachers who challenge kids to change and grow. My union will provide me an unlimited supply of pit-bull lawyers to defend my job and my livelihood in those circumstances. I can tolerate the minor negatives of teaching with a few hacks and ninnies, simply because at the moment it is far too risky to give it my all when any administrator or parent can be poltroon enough to sandbag me then walk away humming. In fact, the threat of a real firing or lawsuit is slight, about the same as the tornado or fire I’m also insured against. It’s the peace of mind that I’m tenured and backed by a union that allows me to opt for the high-challenge, low bullshit approach that I know works because I’ve been at this for a long time and I’ve seen it work (and yes, part of my evaluation of success is test scores, because there are a few good ones.)

By the way, I’m sure the research shows that it’s poverty or race or good teachers that account for success. But what accounts for failure? I have a suggestion. How can society reasonably expect schools to generate smart, thoughtful kids when the society is proudly ignorant, nasty, greedy, violent, and hypocritical? We worry that kids watch too much TV or play too many video games; the influence of those must be tame compared to listening to Sharron Angle or Christine O’Donnell or Michele Bachmann or Sarah Palin, to name just four especially stellar role models. Our society’s prevailing level of factual accuracy, persuasive argument, clear language, intellectual consistency, historical accuracy, and essential honor are abysmal; it’s comical to imagine that schools operating in that environment could make progress.

ice9

“Sebastian, you should have learned in school that if you want to say the teachers union here in the U.S. is worse than elsewhere, you actually have to provide proof, evidence, lines of reasoning, that sort of thing. Trolling about with the “I’m not sure” line trying to sucker people into trying to convince you when quite obviously from many previous conversations that nothing will convince is dishonest.”

SoV, hmmm ok. You always play the burden of proof game. And interestingly it is never your burden. Funny, that.

I think Sebastian has just caused my head to momentarily explode. You know, Sebastian, maybe this is because I was taught good reasoning skills in school. If you make some sort of assertion, the burden of proof is upon you to prove it. That’s just good scientific practice. So I try to be very careful about what I say and how I say it, and I’m usually pretty forthright about when my opinion is just my opinion, gut-checked with anecdotal data.

You didn’t learn about any of this in school? Seriously? Or is this some sort of lawyer thing, some sort of rhetorical game you’re playing that the rest of us aren’t aware of? I think most of us here are interested in the science and the facts; rather less of us in attempting to get others to accept a certain narrative.

Now, do you have any actual evidence for your claim? If you do, spit it out. Stop being coy and stop wasting our time.

There is another huge problem with standardized tests (especially high stakes ones)–the people who take them can be subject to things like stereotype threat or even simple test anxiety that can seriously (and, in the case of stereotype threat, systematically) distort the meaning of the test results.

I have seen ‘Waiting for Superman’. However, I am not a student of education policy in the US (or anywhere else), I have not read the Klein/Rhee manifesto, etc. I have also not read this whole thread, so I apologize for any repetition.

Several of the movie’s messages strike me as simplistic at best, and critics here and elsewhere have been noting them. One glaring omission in ‘WforS’ is that there is no mention (or virtually no mention) of the history of racial and class segregation and how this has contributed to “failing” schools; instead there is simply the bald and simplistic statement/suggestion that ‘bad’ schools cause neighborhood deterioration, not the other way around.

The US Supreme Court’s 1974 decision in Milliken v. Bradley prohibited cross-district busing and ensured that schools in northern cities such as Detroit (the city involved in the Milliken case) would remain or become largely segregated by race. By contrast, in at least some other cities where the local authorities drew the lines in such a way as to include suburbs and central city schools in the same district, the educational and social outcome tended to be different, and better. At least this was apparently the case in Raleigh, NC: see Gerald Grant, Hope and Despair in the American City. How a movie purporting to be about education in the US can be silent about these issues is beyond me.

Re testing etc: I’m thankful that I went through school long before the testing mania reached its current pitch. There were standardized tests, of course, and the kids in my suburban (and virtually all white) public high school, many of them bound for ‘elite’ colleges and universities, took them seriously; but I would be very hard pressed to recall a single occasion in which a teacher in a classroom referred to a standardized test. They did not loom over the curriculum the way they apparently do today in many schools (and not only ‘disadvantaged’ ones).

So now that we have established that testing, at least for math, can be a good thing, I guess we’re getting somewhere.

And here we have more – and more blatant – trolling. I’m wondering if Sebastian has the forthrightness to name the names of those who did not think that math testing “can be a good thing”.

Sebastian, if you want to be disruptive . . . don’t you have your own blog you can play on?

Fuck you, Ned. Is there any evidence that Scantron is a good evaluation method, rather than a cheap evaluation method? I taught math for several years, and know many people who have taught it for many more, and none of them have ever used Scantron except out of expediency (like they’re teaching a class of 300 with not enough TAs). He offers no argument that multiple-choice testing is a good idea, just the assertion, and then various patronizing comments towards everyone who disagrees, such as dismissing all real-world experience on the subject as “silly”.

Every thread on this subject with Sebastian is the same: no one cares about students unless they agree with Sebastian that the teachers union needs to be smashed. He’s monomaniacal on the subject. Honestly, we’re being politer than he deserves.

However, the importance of the written sections – where partial credit was possible – made grading them a labor-intensive process.

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

BINGO!

SoV “Please try to remember what you’ve already said. ”

Please try to remember what we’re talking about. Try to keep the context in mind, thanks. It isn’t all calculus. In fact if we really were worrying about whether or not all (or even significantly most of) the kids were deeply understanding calculus and whether or not that deep understanding of calculus ought to be measured via scantron, this would be a much better world.

Bite me, Sebastian. You’re not being even remotely honest at this point. I quoted your own words back at you, you know, the part where you did say that yes, scantrons for stuff like geometry, trigonometry and calculus were just fine and dandy.

Why don’t you just say that you were wrong, instead of going through all this song and dance with the weaseling and bafflegab? It certainly wouldn’t hurt your reputation after the hits it’s already taken.

To amplify my point: It is irresponsible for a movie such as Waiting for Superman, which after all will be seen by a great many people who are not professional students and observers of education policy, to suggest via omission that the trouble with US education is entirely a result of what goes on in classrooms and who controls that. That is why a single-minded focus on whether teachers unions are good or bad, and on what specific, immediate interventions will or won’t work, risks obscuring another point: a movie like this could itself educate its audience, and to produce a portrait of American education that is completely devoid of a huge chunk of relevant historical background is a gross disservice to the ordinary person who walks in to see this movie knowing nothing about education debates beyond whatever he or she may have picked up from the media. A context-less, ahistorical, often tendentious “documentary,” which wastes a good half-hour at the end on those heartrending but basically pointless lottery scenes, is probably worse than no movie at all. The more I think about it, the more I am convinced that Davis Guggenheim should be ashamed of this film. I would urge people to think twice and thrice before paying to see it.

Jim Harrison: “One obvious way to improve our educational performance would to emulate the attitudes of peoples such as the Jews, Chinese, and Japanese, who love and treasure learning.”

They love and cherish learning and teachers. :)

I’m not against teachers’ or any other workers’ unions in general.

But the ACLU is against the actions of this one in particular — http://laist.com/2010/02/24/lausd_layoffs_lead_to_lawsuit_from.php.

And I think they’re right.

Well, SoV, I’m glad to know what passes for civil from you. You can’t stay on point, you can’t follow an argument, and you have to be repeatedly insulting.

The point remains. Math can be tested.

Especially at the competency testing levels that are the topic of the conversation.

I would feel friendlier towards unions if I didn’t think that their mission was to protect their members’ jobs, no matter what. At least that’s the lesson I took away from the remarks of the head of the Hawai’i teachers’ union, a number of years ago; he boasted that no teachers had been fired while he was in office.

I can’t believe that none of the teachers deserved firing. People are people; they mess up. They disappoint us. Teachers are people.

Sebastian

You appear to be working under the impression that if someone says something uncivil to you, you have won – and will be adjudged by bystanders to have won – the argument.

I don’t know whether that’s a rule that works well in your cultural and professional milieu. Maybe it does.

In mine, though, there’s an important exception to that rule. You don’t win the argument if you respond to reasoned argument by deliberately attempting to provoke incivility.

At 63 you say

SoV: “Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.”

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

It’s very difficult to read that as anything other than a playground level taunt, designed to provoke the incivility it received in reply.

Remind me what the evidence that scantron tests were effective in teaching maths. Because like Scent of Violets, the only thing I saw in support of that claim was that they are used in universities. And you’ve had two people with actual expereince tell you that, and why, this point doesn’t support the claim you’re making.

(Incidentally, I’d say that when there’s someone in a conversation who is trying to win the argument by provoking people into being uncivil, there’s a case that the conversation will be more productive with more interventionist moderating. But I realise that that’s a very time-consuming option.)

I’m reposting this, because a paragraph break messed up the formatting in such a way as to obscure the main point I wanted to make. If there’s someone moderating this thread, they might want to suppress one version or other.

(Just so this gets caught in the spam filter,and thus doesn’t mess up thread numbering, I’ll insert a gratuitous reference to So Cialis M here)

Sebastian

You appear to be working under the impression that if someone says something uncivil to you, you have won – and will be adjudged by bystanders to have won – the argument.

I don’t know whether that’s a rule that works well in your cultural and professional milieu. Maybe it does.

In mine, though, there’s an important exception to that rule. You don’t win the argument if you respond to reasoned argument by deliberately attempting to provoke incivility.

At 63 you say

SoV: “Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.”

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

It’s very difficult to read your contribution there as anything other than a playground level taunt, designed to provoke the incivility it received in reply.

Remind me what the evidence that scantron tests were effective in teaching maths. Because like Scent of Violets, the only thing I saw in support of that claim was that they are used in universities. And you’ve had two people with actual expereince tell you that, and why, this point doesn’t support the claim you’re making.

(Incidentally, I’d say that when there’s someone in a conversation who is trying to win the argument by provoking people into being uncivil, there’s a case that the conversation will be more productive with more interventionist moderating. But I realise that that’s a very time-consuming option.)

There is no mention here of actual test scores despite the theme of WfS. The NAEP is a standardized test used throughout the US and perhaps looking at the actual scores might shed some light on what teachers, as a class, have accomplished. Bob Somerby has done a lot of writing on this movie and I think his background of teaching in the Baltimore city school district for 11 years might lend some credibility to his analysis.
Per Bob, who wrote several weeks’ worth of posts about the movie, one of his 9 critical elements is:
Stop lying about our test scores: If we claim to care about public schools, could we possibly start to tell the truth about our most basic data? In the last 20 years, minority students have recorded increased scores on both sections of the NAEP, around 16 points, or almost 2 grade levels. Does that mean teachers, especially those in poor districts, have done a good job with those kids?
You might want to spend some time at The Daily Howler and see what he has to say.

Again, we’re expending enormous amounts of heat and light on a proposal that is basically equivalent to trying to form an orchestra by recruiting people on the street, paying bonuses to some of them, sacking the ones who make the most bum notes, and replacing them with more random recruits. It’s going to be a long time before you stumble on a competent instrumentalist, and a hell of a long time before you get them all playing in time, let alone in tune.

Why isn’t there enormously more interest in improving observation, and finding out what it is that the “good teachers” do that the others don’t? That would be a gigantic advance however education was organised, funded, or delivered. If we don’t know that, then you’re at the level of the old story about Billy Wilder making one of his actors do take after take after take and yelling “CUT!” every time. Eventually the player turns to Wilder and says “Well – what is it you want?” Wilder says “Just…be better.”

Just knowing that some of the teachers are doing better than others isn’t very useful unless you also know why. If you know why, you can teach the other teachers. You can base the teacher training curriculum on why so they all start off doing it.

Even if it turns out that you can’t define teaching that well, that it’s some sort of inherent property of personality, well, that’s useful information. You might be able to find out who has it in advance rather than waiting for them to fail.

Also, I suggest the following operational definition of excessive testing; if you’re doing tests to prepare for the tests, that’s too much. When I did A levels we had mock exams of the mock exams.

Anyone else notice how disrespectful Sebastian is towards teachers? He says something silly like multiple choice and scantrons can be used to test geometry, trigonometry and calculus, several people who are actually math teachers tell him that no, that’s simply not true, and here’s why . . . and we get a heaping helping of disrespect thrown in our faces.

Note that Sebastian doesn’t teach math at any level (he claims to be some sort of lawyer, I believe), but he’s so sure that we’re all wrong, that he’s right, and that they to get this across is to be snide and nasty. I’d say we’re seeing – live – a bit of the problem teachers face in their day to day work.

On perhaps a lighter note, this talk about testing math reminded me of an experience that may be amusing to a third party. ;)

On a test in my introductory physics class in college there was a 10 point problem to derive a certain mathematical result. I thought the problem was quite easy. But the instructor only gave me 5 points credit. I checked my paper again, and saw no mistake. So I asked the instructor about it, and he said that my derivation was wrong. I asked him to show me my mistake. He pointed to a certain step and said that I should have made a binomial approximation. He continued with his derivation from that point, and at a later point made another binomial approximation that, as it happened, canceled out the first. I protested that my derivation was exact and required no approximation, but he would not listen to reason. {grin, I think}

He continued with his derivation from that point, and at a later point made another binomial approximation that, as it happened, canceled out the first. I protested that my derivation was exact and required no approximation, but he would not listen to reason. {grin, I think}

That sounds odd. Not him, you. Binomial approximations throw away information; doing it twice can’t “cancel out” because then extra information would be coming out of thin air. Spoken as someone who did very well on the Putnam – which is not multiple choice :-)

And re: the French educational system: if you are saying that a CFG curriculum, a CAP curriculum and a Bac curriculum are the same, you’re using a definition of curriculum with which I’m unfamiliar.

No, I wasn’t saying that, but I can see where we were sort of talking past each other. I’m just saying that a Bac curriculum (for instance) is going to be virtually the same in every lycée in every part of France, whether public or (publicly-funded) private. And this is in sharp contrast to the high school diploma in the US.

The idea of streaming secondary students into more pre-professional programs like the CAP is an interesting one that might hold some lessons for the US system, but I think it might be getting a little far afield of the topic at hand.

RE: France–that makes sense–yes, we were talking about different things.

RE: binomial approximations–in some cases, you can do the approximation at any of several different points in a process and lose the same info (same results, but the earlier you do the approximation the more concise the model is; figuring out when to do your approximations to balance accuracy and ease of modeling is one of the big questions in modeling–the rare cases where it doesn’t matter are rare.)

92: “The actuarial prelims and the math GRE are both pretty good measures of subject competency, and are multiple-choice tests.”

I thought that the math GRE was a joke. I don’t think it measures “subject competency” in any meaningful way. One problem with this particular type of test is that the underlying material is relatively simple. Nothing more advanced than high school algebra. In order to make the test seem “selective” enough (GRE is used for graduate school admission), the designers often come up with confusing questions and deliberate traps. The skill required to score high on that kind of test is not so much working out a math problem as being able to spot the traps while working fast under pressure and keeping one’s nerves. Maybe this is precisely what the test aims to evaluate – but it is not math proficiency. If you don’t believe me, have a look at some practice tests: http://www.greguide.com/gre-practice-tests.html

Many of the analogies in the verbal part of the test are patently absurd and are only there to add selectivity to the test. The designers use obscure vocabulary that many test-takers have never heard of. I understand why the designers do this – otherwise the test would be easy for too many people. I also understand that the non-standardized part of the test – the short essays graded by hired graders for low pay – is at least as problematic: grading is inconsistent and it is known that longer essays get higher scores. These tests are attempts at solving a ranking problem efficiently. They are not totally arbitrary and are somewhat related to what they are supposed to test. But meaningful evaluation tools they are not, and the importance that kind of test assumes in the US educational system is worrying.

piglet–I’m thinking of the Math subject test, not the math portion of the standard GRE.
http://www.ets.org/Media/Tests/GRE/pdf/gre_0809_math_practice_book.pdf

And for an actuarial exam, look here:
http://www.soa.org/files/pdf/course1_0503.pdf

Sam: I don’t know. I’m not going to spend much time on this but I had a short look. Question 2: Which of the following is an equation of the line tangent to the graph of y =x+e**x at x = 0 ?

The experienced test taker will know that they don’t actually have to differentiate, just to eliminate the obviously wrong answers. In this case 3 out of 5 choices don’t go through (0,1), so they can be eliminated. Yes, to come up with that reasoning requires some basic understanding. You have to know what a tangent is etc. But it does bother me that test-takers spend a lot of preparation on learning techniques for finding short-cuts rather than doing actual maths. In fact they have to because they lose too much time otherwise. The other thing that bothers me is that these tests consist of isolated, context-free bits. This kind of test is an end in itself. What is the effect on students of teaching to that kind of test?

“Yes, but high school should teach calculus to as many people as possible. It should be a core part of the curriculum. And that’s exactly where I have a problem with standardized tests: because they misdirect class time and teaching time towards a nice basic level of pabulum. That’s a serious problem.”

This is a very weird objection to testing. The reason we aren’t teaching calculus to as many people as possible is because we are having trouble teaching more basic math skills to a large portion of the students. That is the fault of testing. That is a fault of teaching, and of curriculum design. The point of testing (at the end) is not to DO the teaching, it is to verify that the teaching was done. A huge number of the objections here seem to skate right over that point. See for example:

“Remind me what the evidence that scantron tests were effective in teaching maths.”

and

“Apart from entry exams, an exam is of value only insofar as it provides teachers with usable information about what the student doesn’t know. Making the exam multiple choice needlessly deprives the teacher of basic information on why they made the mistakes they did – especially in mathematics, where you can get all your reasoning right and be laid low by a simple arithmetic mistake.”

I’m not advocating scantron tests as the only method of TEACHING math. And for all those who are getting so high and mighty about my reading comprehension, it would be nice to see where you picked up the idea that I was making that argument. I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred. That is what teacher’s unions in the United States are against. They don’t like the end-of-year/end-of-high-school testing to verify that the teaching/learning actually occurred.

And you all aren’t even getting the objection to standardized testing right. The legitimate objection is that, for individual students, at the very highest levels of performance measured by the test, standardized testing isn’t great at separating out “very good” from “brilliant”. They are however quite capable of determining average levels of competence from the vast portion of the middle. Which coincidentally is exactly what we want them to be able to do. We don’t need them to evaluate fine-grain differences between individual students. What we want from these tests is that they give an overall view of about how many students have learned basic competency of certain skills. If the test gets it slightly wrong on one of the students, it will tend to average out over the very many students. The tests don’t have to be hyper-precise at the student level, they evaluate at the class/school/curriculum level. And in most cases (including the much maligned NCLB) they do it across multiple years. Such tests are perfectly adequate for that purpose. Even scantrons. Especially in math.

Whatever reason such tests are so strenuously resisted, it isn’t because they are ineffective tools at detecting basic competency among the vast majority of the tested population.

Scent of Violets: “billikin said was that doing two approximations in succession canceled out. I’m not saying that it’s impossible, but the professional in me was intrigued enough to wonder just what sort of problem this was where such a thing was possible. Of course, this could just be an misunderstanding over what “canceled out” means.”

By “canceled out” I mean he got the correct result. :)

@Scent of Violets:

It has been a long time, but, as I recall, the guy did not like to divide. Maybe it was something like this:

Instead of dividing by (a + b), he made the following approximation:

(a + b)^(-1) -> a^(-2) * (a – b)

Then later, instead of dividing by the result, he made the following approximation:

(a^(-2) * (a – b))^(-1) -> a^2 * a^(-2) * (a + b) = (a + b)

Go figure. ;)

Sebastian: “And in most cases (including the much maligned NCLB) they do it across multiple years. Such tests are perfectly adequate for that purpose.”

Is there an apples and oranges problem? To assess student learning over a school year, shouldn’t you have a test at the beginning of the year and one at the end? (OC, you expect the kids to flunk the first test, but you want to know what they know and don’t know coming in.) But the tests don’t do that, do they?

Landru: “one supremely pale edu-wonk pointed to a bunch of charts and graphs and pronounced that if the US were to fire just the bottom 6% of teachers, then we’d ascend to educational nirvana (here defined as being on par with Finland, if I recall). That seems to encompass the whole philosophy: teachers are generally good, but a tiny minority of bad performers—just 1 in 17—are enough to drag the whole system down; once you agree to that, then it’s only natural to ask why it isn’t easy, or even possible, to just prune that tiny bit of rot away and let Finland bloom.”

Do you buy that?

Let me try those approximations again, just for fun:

(a + b)^-1^ -> a^-2^ * (a – b)

(a^-2^ * (a – b)) -> a^2^ * a^-2^ * (a + b) = (a + b)

Math is hard.

Sebastian, it is quite amazing that you seem to think you can judge the effectiveness of multiple choice math testing better than numerous professional mathematicians.

Christian, you are obviously unfamiliar with the great number of discussions on this blog involving Sebastian, which, as a rule, pit the genuine expertise in various fields of human learning and experience of his interlocutors against Sebastian’s, ahem, first-rate command of English prose and American tax law and unflagging confidence in his own opinions. I am envious!

Isocrates: “‘rhetoric’, referring to an academic discipline in its own right, with its own journals, conferences, and graduate programs that indeed does teach critical thinking”

I remember when I first saw a critical thinking textbook. It made me think of my grandmother’s high school rhetoric textbook, and, IMO, critical thinking suffered by comparison. But a critical thinking course is better than nothing, which was the case when I was in school.

If what was being debated in the fight over testing was AP vs. nothing, my guess is that you wouldn’t see any objection from the teacher’s unions. The APs are high quality tests that measure learning as well as aptitude, are subject-specific and comprehensive, contain a mix of multiple choice and written questions, etc.

The interesting question is why the Rhees and Kleins of the world haven’t jumped at the chance to square the circle. I would argue that the APs don’t lend themselves to their preferred management strategies – among other things, AP exams do not lend themselves to the give-a-letter-grade-to-the-school, rank-the-teachers-by-value-added, success-or-failure, annual-improvement model. Indeed, an AP system would likely show some of the more complex realities of education that go beyond the strange fixation we have that everyone in the country should be educationally ranked on the basis of their reading and math skills, nothing else, and that everyone must be good at both.

StevenAttewell@127: Yes! Precisely!

I think it would be possible – even desirable – to develop a universal testing model based on the AP tests. Currently, they cater solely to high-achieving students, but I think it would be possible to write tests and curricula for a range of aptitude levels. You could imagine an algebra or trigonometry test that followed a similar model to the AP Calculus exam.

(I would argue that the success of the AP exams is due to the fact that they were developed with teaching in mind. Questions on the test are integrally tied into a class’s curriculum and pedagogical purpose. So for example, the AP US History exam doesn’t just require students to know when the Civil War was, or who was the president at the time. It requires them to be able to navigate and decide between arguments about why the Civil War occurred – which is after all the real goal of a history course. State- or NCLB-mandated standardized testing, by contrast, has little to no connection with the curriculum in any given year, and hence it’s largely futile.)

But for this to work, the goal would have to be to give students qualifications rather than to assess whether they were achieving. It’s a subtle but important difference. Think of the British school system, where you can leave school with O- or A-level qualifications rather than just a high school diploma that mainly indicates that you haven’t drooled too much in class. Testing based around qualifying students in specific subjects would be a good thing, as it requires sophisticated tests that are integrally linked with curricula.

Thank you Stuart. Please, SoV and others, explain why these tests are so horribly inappropriate.

I’m not advocating scantron tests as the only method of TEACHING math. And for all those who are getting so high and mighty about my reading comprehension, it would be nice to see where you picked up the idea that I was making that argument. I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred.

Sigh. Sebastian, will you please keep track of what you have already written:

Universities do it all the time using scan-tron tests for pretty much all the classes from pre-algebra straight through differential equations, which is probably further into math than at least half the readers here have gotten.

And more of the same. No, universities don’t “do it all the time” and your presumption that they do is laughable. And no, per your other stuff, it looks pretty clear to me that you were saying that teaching to the multiple choice test is teaching math. Do you not understand how your posting style makes you something of a figure to be mocked, the very stereotype of the dull and ignorant school choice crowd?

Let me try those approximations again, just for fun:

(a + b)-1 -> a-2 * (a – b)

(a-2 * (a – b)) -> a2 * a-2 * (a + b) = (a + b)

Rats! I left out the exponent in the second one, but the formatting came out OK.

Okay, I see, basically divide by a quantity then divide by it’s reciprocal. Yeah, that’ll get you back to where you started if you don’t do any expansion tricks, but it’s just luck that it worked out on this one. If you do a three-term expansion twice, you get back a+b, plus some other stuff: -2b^3/a^2+b^4/a^3. So your guy just got lucky. Assuming I’ve done the expansion correctly, of course :-)

Looking at the 10th grade math test for this state’s high stakes tests (required for graduation), and given what I remember about the level of pre-calculus understanding and instruction even among kids bound for selective colleges (say, 1/3 probably reliably mastered the material, at most), I find it difficult to believe 50% of high school students could answer some of these questions on the basis of their regular math classes.

Right. There’s a big difference between using a test as a filter, and using it to accurately assess the student’s knowledge and ability. Multiple choice can, in some instances, work for the first bit. But not, generally speaking, for the second one.

AP tests are also normed. That makes them very interesting diagnostic tools.

Plus they can count for college credit, so there’s a strong incentive.

Elsewhere, incentive is, shall we say, thin. For many, perhaps most of the ‘high stakes’ tests the main emphasis in ‘teaching’ is actually persuading kids to take the tests seriously, to do their best, or maybe just to show up unstoned. Example: A few years ago my high school failed to make AYP for one category: male African Americans, reading. That one category failure cost the school a number of, you know, gold stars or whatever. The whole faculty was shaking their heads and looking dejected like they’d just let the team down, killed the Gipper. Actually, I was the one who let the team down. On Test Dayz!!! everybody is pressed into service to administer the test (and you can’t look at it, oh noez–not a peek, and if you should post one on an Internet Blog Site you’d be cashiered on the spot–our hands signed a paper. Really.) Anyway, in my group of test takers were three 11th grade black males, all together by virtue of the alphabet and their standardized anglo-American slave names. The test hit their desks, and their heads soon after; the three together bubbled nary a bubble. Those were three of the 11 african-american 11th graders in our entire school of 3,000 plus; their abstention on the exam cost us our gold star because of those dang statistics and the math that SOV keeps plugging away at above. I tried to get them to bubble, I really did, but they smiled politely and shook their heads. The problem: they’d been told that they couldn’t text during the exam, and that made them unhappy, so they gave it a miss.

ice9

“Right. There’s a big difference between using a test as a filter, and using it to accurately assess the student’s knowledge and ability. Multiple choice can, in some instances, work for the first bit. But not, generally speaking, for the second one.”

I’m confused by this statement. Aren’t we asking for a filter, by which we set a minimum proficiency level and note those who pass? We aren’t asking for a fine grained analysis tool which sorts the students into 100 levels of achievement. We are asking for a test that can on average identify the general number of students who have achieved minimum competence. We aren’t even asking it to be as fined grained as an SAT, GRE, or AP test. If you think standardized tests can do that, what in the world are we arguing about?

Those were three of the 11 african-american 11th graders in our entire school of 3,000 plus; their abstention on the exam cost us our gold star because of those dang statistics and the math that SOV keeps plugging away at above. I tried to get them to bubble, I really did, but they smiled politely and shook their heads. The problem: they’d been told that they couldn’t text during the exam, and that made them unhappy, so they gave it a miss.

Plugging away is right; there’s no substitute for practice, practice, practice.

Let me tell you the rest of the story. Those three guys graduated despite profound academic deficiencies and a really terrible work ethic because people like me aren’t allowed to flunk them out. Later on, people like me (can you tell this is . . . irksome) will be forced to take at least one section of what is now 40 (and used to be 5, back in the 1980’s) sections of remedial math for entering freshmen to bring these jokers up to speed. Then we’ll be barraged constantly by parents telling us that their kid “needs” to be passed despite their abysmally low performance and their terrible work ethic, alternately begging for mercy and threatening terrible retributions (and you know, if their boy isn’t learning, it’s my fault. Despite their complete unwillingness to back me up on doing the homework.)

No, ma’am, your kid is a sorry little excuse for a human being, and a big part of that is on you. Had you been doing your job at home, showing some backbone and some discipline, your child might have been of better character and might actually have the self-discipline we used to take for granted as a necessary perquisite for academic success.

And until that issue is addressed, you’re simply not going to see a lot of reform; only a round robin of finger pointing and cheapo “solutions” that doesn’t cost anyone anything (except the teachers.) Rinse and repeat.

Hi Sebastien:

On principle I see nothing wrong with testing and have had difficulty explaining this to others, so I can sympatize (a little) with your plight here. I don’t want to get into ad hominem attacks or debates about tone or literary style, but I do have a question.

You mention specifically that in your preface to one of your posts that, “The point of testing is not to DO the testing, it is to verify that the teaching was done.” Do you believe that standardised testsare a reliable and valid method of determining that “the teaching was done”? If yes, can you think of any possible objections to this stance?

Now, I know that you said “verify the teaching/learning has occurred” but perhaps what has got so many people’s backs up is your (seemingly, and I fully admit that this is merely my interpretation) emphasis on these tests ability to measure whether teaching is done ( I’m not referencing the other debate as to whether or not standarised tests can validly measure whether learning has occured– that’s a whole other black box problem).

I know what teachers with multiple years of experience will say, but I’m interested in your opinion. Do you think standardised tests provide a useful metric for teacher performance?

“Uh-huh. To repeat myself, Juilliard gives a test to prospective students to see if they can read sheet music. So if they pass, they must be Juilliard material on account of the fact that they demonstrated the ability to sight read. Right.” [and then a paragraph of extraneous character attack]

See the thing is, we are talking about BASIC math skills. We are most specifically NOT talking about deciding whether kids are Julliard material, or what level of math wizard they are. So yes, we are testing to “can you read sheet music” levels. Which btw, a frightening number of kids are failing. So let’s quit with the condescension and talk about it.

See the thing is, we are talking about BASIC math skills.

Here are a couple of attempts at assessment of the learning of kids from various countries:

http://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment
http://en.wikipedia.org/wiki/Trends_in_International_Mathematics_and_Science_Study

Where should the US be on such charts for you to consider American schooling an overall success?

Substance McGravitas @149
Here are a couple of attempts at assessment of the learning of kids from various countries…

Here’s another fun thing that can be done with these charts: We could take a look at the countries that consistently rank in the top 10 and try to figure out what they’re doing that the US isn’t. But on the other hand that might be hard and perhaps it would be better to look for a silver bullet that’s amenable to political sloganeering (e.g. “Test the kids! Fire the bad teachers!”).

Since we’re agreed on that, I’ll start: To the best of my knowledge, education funding and core curriculum in each of the top-performing countries is centralized rather than Balkanized across 50 states and thousands of school districts.

Oh wait! That’s an observation that lends itself to a stunning and practical policy measure: Let’s Balkanize the education system further by encouraging school vouchers and the establishment of privately-run and sometimes privately-funded charter schools. Brilliant!

Sebastian @152 said:

“This actually isn’t a good idea, at least as a long term plan. 3% of the worst over the next 5 years might be good, (after 5 years of evaluation, testing, training etc). 3% every year sets up a problem if you are unlucky one year, and it sets up a long term dynamic where you are subject to firing even if your kids are learning to the competency level.”

3% isn’t much, especially since you’re going to have turnover, retirements, new hires, etc, anyway. In fact, if the mean duration of employment is 33 years, you’re going to have 3% per year turnover anyway. And a teacher with kids learning at the competency level could be exempted. The 3% wouldn’t be compulsory. If all teachers are showing competency, it could be 0%. Further, the union could have some say in who was fired. They would just have to allow 3%.

Also, even when we have a fine-grained tool it tends to measure one-dimensionally. What we need is greater refinement of our tools in both depth and breadth. It’s something I was hoping to see from NCLB, and the focus on standards — the Program of Studies in my state generated to be NCLB-compatible, for example, is quite good, in that it specifies several independent dimensions in which a person can be mathematically competent (geometrical reasoning, algebraic thinking and abstraction, measurement and comparison of quantity, etc).

The ‘core content for assessment,’ by comparison, is [a] considerably narrower and more streamlined to be computational, and [b] measured and aggregated along a one-dimensional scale, rather than retaining fine measurements across those dimensional categories so carefully constructed for the POS. It’s a shame.

‘Teach to the test’ here literally means ‘ignore 50-80% of the POS, especially the stuff that’s hard to test in standardized formats because the relevant questions are kinda long.’ And several folks upthread are right: given a better, finer, more sensitive battery of qualifications tests, teaching to the POS-thorough test would be just fine. (It would be also impossible, because you can’t teach kids to fake their way through that kind of stuff, which is what ‘teach to the test’ is all about in practice: exploiting the deficiencies of the test format to produce unrepresentatively high results.)

148: and how does this “incentive” actually have any impact whatsoever, if we can’t actually define what constitutes good teaching? what is the mechanism of action? How do you improve in the absence of information?

After you control for all the typical demographic variables (ethnic origin, parents’ income, % second language, % special needs, yadda yadda), the result you get will still have a distribution of some kind because, y’know, some kids are dafter than others. If it’s a normal distribution and we’re using the 3rd (or 97th, depending on how you look at it) percentile as the trigger-level, you’ll have a bit less than a 1 in 33 chance of being fired every term purely by chance. With three terms a year, you’re looking at firing the entire workforce about once every 11 years on the basis of pure statistical noise.

So why not come out straight and stand up for a return to the great truths of the classical canon? Decimation. Let’s sack 10% of the teachers at random. That’ll keep them on their toes.

Hence my Western Canon Plan. Random elimination FTW!

p.s. To be exact, I graduated from high school in 1975.

I know you think that you’re being clever by engaging us as a lawyer, but that doesn’t appear to be cutting any ice; the rest of us are by and large scientists.
They eat lawyers for breakfast ;-)

Sebastian is not a litigator, Sebastian is a pension and benefits specialist, to the best of my knowledge. A first rate litigator will not get eaten for lunch by a scientist (usually encountered in the form of doctors), and instead will devour any pureyor of junk science. Usually, when an honest, first rate scientist meets a first rate litigator, the result is truth, or a reasonable approximation thereof.

Sebastian is a remarkably decent and honorable guy for a conservative, but on some issues . . . well, he knows what he knows, and nothing will convince him otherwise.

LFC –

My experience is: the proliferation of AP courses [history here has not markedly reduced the quality of courses. And the newer courses I’ve looked at, e.g. Environmental Science, seem solid and excellently well designed — AP ES stacked up pretty solidly compared to the intro to environmental science course I took at UW-Madison. Students who scored 3+ on calculus BC and take Multivariate Calculus here at the uni where I teach score on par with students who took our 1-year Calc sequence; scoring a 5 on the AP test for Calculus BC seems to correlate strongly with earning both an A in M-v Calc and respect for one’s mathematical facility from the M-v Calc instructor. Folks around here speak pretty well of AP tests’ predictive value. TIMSS offers corroboration of this [PDF] for math and physics.

Mileage may vary depending on the school and teacher and etc, of course, but the advantage of having the national College Board program is clear — when students have to pass an exam not written by the teacher, and the exam is allowed to range over the entire breadth of course content as well as material that is newly presented on the test for students to digest and respond to in real-time, parents have less room for complaint or protest against demanding and thorough teachers, and more room to complain about and protest vapid or undemanding teachers. The incentives are well-aligned. “Teach my kid to do well on this test” equates to “teach my kid to know the foundational material well, and to think critically about new material.”

Note that I mean incentives for pouty pushy parents, not for teachers. That’s where we really need to adapt and mould the incentive structure: the most consistent and loud and urgent social force for micro-changes in education, at the local level, is the group of parents who attend PTA meetings because they want their kid to get ahead. It’s important to mobilize that particular force in socially productive directions (because it’s going to be there and it’s going to impact policy). To me, that means nationalizing the curriculum and credentialing process as much as possible.

Distancing the requirements to get a diploma from teachers — that is, separating teaching from summative assessment — seems to me to be an unqualified good. Giving teachers more power to teach flexibly, but taking the power to control all the outcomes of final summative assessment out of their hands, means that teachers can point to the ‘bad cop’ of e.g. the College Board and play the ‘good cop’ who is demanding, edgy, unconventional, etc because they want their students to succeed. I think it should be mandated that 20% of a student’s grade in non-elective courses must be determined by performance on some national or international exam — perhaps in place of a teacher-written final exam.

So specifically I think an AP-style program, extended to proficiency tests in core content and the standard program of studies, with requirements to get 3+ in order to receive a high school diploma, would be an amazingly good thing.

Problem is, there are strong incentives for states to design crappy tests and make their deficiencies easy to exploit. My own state separates the basic “core content for assessment” from the actual full program of studies, and the net effect is to… not teach anything that’s not core content. And the tests [even the non-multiple choice portions] are mostly designed so that students who are taught exam-specific tricks can back their way into many of the answers without even fully understanding the questions. Just as importantly, passing or failing the test has no impact on graduation or course grade. Them’s some perverse incentives.

I guess this is getting away from union-busting, so I should mention that teachers’ unions help insulate teachers somewhat from the force of pouty pushy parents. When the PPP force is pushing in more productive directions than it currently is, I’ll be a little more sympathetic to calls for union busting — at least I’ll listen with a more open ear.

For now, no. As it is, when I hear about a PPP-force movement to get a teacher fired around here, it’s frankly usually because the teacher is atheist or allegedly homosexual, not because the teacher is underperforming. (The actual reason given, the teacher is “disrespectful to students’ beliefs” or some such thing. What I hear later is that the disrespect was merely a reasonable demand that was provoked, e.g., by a student insisting on reading their bible instead of their history book in a world history class. Hmm.)

Since it got lost in me being Long on the Internets, I’ll shorter-me this in: tests which present brand-new content and require students to digest it in real time, test critical thinking and analysis. Such tests are hard but not impossible to write, and are essential if we want to teach ‘critical thinking’ and verify results.

Cf. the DLAB test for language-learning proficiency. I suggest, only somewhat tongue-in-cheek, that we outsource design of all these tests to the military.

Salient @ 167- when I taught molecular genetics I always wrote exams on which most of the questions worked that way- usually by presenting fictional experimental data and asking the students to analyze it using the principles which they hopefully had learned. I agree that it’s a much more effective way of assaying learning than asking them to spit out rote-memorized factoids. Sadly, many students even then (early 90s) were conditioned by their K-12 education to prefer the latter and to find the former uncomfortable and “unfair”. I imagine it’s even worse after years of the damage wrought by NCLB.

“Surely the need for fine-grained distinction is intuitive. I would hope someone who loves math and invests more time in it could clear a higher bar, and have their pairing of effort and facility acknowledged as worthwhile (and I would hope that teachers who facilitate this achievement would be likewise acknowledged).

If we teach ten more students to add fractions, at the expense of teaching two fewer students how to think about differentiation, I am not convinced [or perhaps, I am no longer convinced] that we have achieved a worthwhile trade-off for society.”

I’d be thrilled with a fine-grained assessment. But I’m being told that even the binary assessment can’t be done, so I’m not going to try to argue for an even better one with people who don’t believe that minimum proficiency can be detected in a test. And the last paragraph is a different question of values altogether.

I’d still be thrilled to find out how “teaching to the test” like the Florida test example given above is likely to cause problems in learning basic math, and how it would be unable to detect basic math competency.

I agree that it’s a much more effective way of assaying learning than asking them to spit out rote-memorized factoids.

Yep. And your example of “here’s a sample situation, analyze it” is basically the way to go about it, in all fields. English is kicking our arses for that reason; giving students a new story passage and asking them reading-comp questions about it is standard operating procedure. In maths-n-sciences it seems surprisingly controversial, because [exactly as you say] we’re not doing that in the K-12 environment, so “digesting new information in real-time” is not a skill that is being taught.

That, I would think, is a shame and a huge problem, which seems under-mentioned in the popular discussions of these things. Even a kindergartener should be expected to make real-time extensions of learned material to new material. I see lots of great HS teachers doing that in formative assessment, but far fewer HS teachers requiring that in summative assessment — and the teachers most likely to include that in summative assessment, tend to be teachers teaching the high-level courses. Students in the mainstream are sorely underserved in that regard.

And the last paragraph is a different question of values altogether.

Uhh, yeah, and values are important to this discussion.

I’d still be thrilled to find out how “teaching to the test” like the Florida test example given above is likely to cause problems in learning basic math, and how it would be unable to detect basic math competency.

Well, for one thing, when someone says “teach to the test” this usually does not mean “teach the material that the test is testing.” It means, in part, “teach tricks that allow students to answer questions correctly, or improve their guesses at more questions’ answers, without understanding the material.”

By definition of what the phrase means and how it is used, teaching to the test is not teaching course content.

If what you’re saying is: develop tests that are comprehensive and thorough, and which also introduce new and unannounced extensions of core course material for students to digest in real-time, and then organize the course so that successful completion of the course will prepare students to succeed in that environment… well, that’s a rephrasing of what education with summative assessment basically already is. You’re just describing “Teach stuff and then assess learning” is not new. You’re just recapitulating a core tenet of the uncontroversial Understanding by Design course-design template. I would “let me google that for you” Understanding by Design, but I am trying to conscientiously reduce how sarcastic and bitter I come across on the Internets, and maybe you’re not aware that you are broadly suggesting teachers should do what they’re already doing and mostly have been doing since time immemorial.

Either you’re advocating that we teach test-exploitation strategies [I take this as prima facie bad] or you’re advocating that we teach material and then assess learning [not a new statement, therefore not worth hammering on about]. To get better responses from us, how about you articulate a little more specifically what you’re advocating?

Either you’re advocating that we teach test-exploitation strategies [I take this as prima facie bad] or you’re advocating that we teach material and then assess learning [not a new statement, therefore not worth hammering on about].

I can’t speak for Sebastian, but I’m advocating the second. And I think it is worth “hammering on about”, because currently, it isn’t being done in a way that makes subject mastery comparable even between teachers in the same school, much less between schools, in the American public school system.

It seems like there are two very different questions being discussed, with extensive attendant confusion.

Sebastian Teaching to even a moderately good math test is teaching math.

Piglet Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests.

I’m arguing that piglet is wrong, and that it is entirely possible to make moderately-good math tests (such that “teaching to the test” requires teaching math, not test-taking tricks) that are multiple-choice tests.

I am not arguing that currently-used tests (here in VA, the SOL’s) are moderately good. They aren’t. They’re extremely gameable, and don’t test core knowledge well.

I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.

I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.

Define “high-stakes”. Be sure to specify as part of your definition whether any attempt will be made to measure value added, rather than punishing any teacher whose kids start out behind at the beginning of the year. Also be sure to indicate whether you think tests score are the ONLY measure that should be used to judge teacher performance. Then we can begin assessing the truth value of this claim and the intentions (and level of intellectual competence) behind it.

I’m arguing that piglet is wrong, and that it is entirely possible to make moderately-good math tests (such that “teaching to the test” requires teaching math, not test-taking tricks) that are multiple-choice tests.

And what is the basis for this “argument”? Because you are not arguing, you’re just stating your own opinion with nothing to back it up; no facts, no research, no cites, not even a bare-bones reason. Just a flat statement.

Have you taught math classes before? Do you have experience teaching math, and if so, what was it, and why do the math teachers here seem to be rather uniformly against the multiple choice format?

Let’s not miss the point. The goal of testing, for the union busters, is NOT to assess student learning. It is to evaluate teacher performance. Even the best-designed tests will buckle under the strain of being used for the latter while pretending to do the former.

why do the math teachers here seem to be rather uniformly against the multiple choice format?

At least in part: Because you’re here.

I haven’t actually seen all that many people coming out vociferously against the College Board’s AP exams or the GRE subject test, here, except for you, SoV. Some of us have been staying mum on that particular issue because we’ve learned not to provoke your ire.

I just finished grading about 500 multiple-choice tests.^1^ In each case, students were required to show their work on the exam for each problem, then report the answer requested.

Although each problem required about half a page of handwritten computation, I received almost zero extra information from looking at the students’ work, just frequent confirmation of what I thought each available wrong answer on the test would indicate. So-and-so deeply goofed and thought the product formula was (fg)'(x) = f'(x)g'(x), this other so-and-so made a minus sign error, etc.

As you know, after a few years of teaching the same stuff, a great majority of errors are predictable, and tests can be written so that [a] the predictable errors always result in a wrong answer, [b] each predictable wrong answer is made available to the student, and [c] “none of the above” can be made always available, in case students err in unexpected ways.

If students double-check their work because their answer is unlisted, and thereby locate an error in their work, I’m okay with that happening. I want them double-checking every response anyway.

On the other hand, I don’t particularly care if they were careless with minus signs or didn’t know the formula or what, though I can track the difference because a wrong answer from an minus-sign error at a key step was always available. The idea that I should award partial credit for a wrong answer because ‘at least the student knew the formula’ bothers me. I give them enough more than enough time to finish and check their work carefully, and then expect perfect responses: I expect accurate knowledge, correct application of that knowledge, and careful procedural work.

Incorrect reported answers are marked -1, unanswered questions -0, correct answers +2. Students have been told repeatedly that guesses with little to no work shown will be marked -5 instead of -1, even if correct, so nobody took a random stab at any question. This eliminated possible statistical noise from happening to guess right. There are probably ways I could automate this verification, if I bothered to think of them.

Now, I don’t really feel like being harassed to the ends of the earth by you, SoV, so kindly either don’t reply or at least reply civilly. Regardless, acting aggressively to commenters who disagree with you over multiple threads, and then drawing conclusions from the lack of disagreement with you later on, is problematic at best, and I see no reason to let it stand unprotested here.

^1^Sure, I could ask more theoretical essay-response questions in exams which can’t be assessed via multiple choice, but am currently following the department’s advised guidelines very strictly, which I don’t have a say in. I’m a graduate student, not a tenured professor. (Regardless, my exams are known for being damn hard relative to the norm, and my RateMyProfessor ranking & comments both reflect that.) Besides, I ask those questions in quizzes, which can be quickly graded 0/1/2 in a light read. Having read over old Calc 1 exams from a wide variety of universities as made available on their professors’ web pages, it seems to me that the content assessed and well-assessed by my exams pretty much matches the content they assess, assuming they never award partial credit for an incomplete or erroneous response.

Now, I don’t really feel like being harassed to the ends of the earth by you, SoV, so kindly either don’t reply or at least reply civilly. Regardless, acting aggressively to commenters who disagree with you over multiple threads, and then drawing conclusions from the lack of disagreement with you later on, is problematic at best, and I see no reason to let it stand unprotested here.

Translation: After I’ve been rude and uncivil, I don’t want you to follow me in kind.

What a tool. And as for “following to the ends of the Earth” . . . just what the hell are you smoking? What’s with the ‘tude anyway? Looking up above, I don’t see anywhere I’ve unloaded on you either; but you seem to think that’s the case, so why don’t you point it out? Further:

I haven’t actually seen all that many people coming out vociferously against the College Board’s AP exams or the GRE subject test, here, except for you, SoV.

Really? Let’s rewind the tape:

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

BINGO!

That bingo was me. Doesn’t look like I’m exactly dead set against those exams now, does it?

So, do you have something meek and sweet to say to me? Like maybe an apology?

Couple points:

(a) The GRE, and its subject tests, are useless. It’s a form of rent extraction. when I am on the committee doing graduate admissions I never once look at the scores. The only function it properly has is to eliminate the very worst candidates – and it is superfluous for that purpose. So what remains is that it forces students to pony up hundreds of dollars for nonsense.

(b) Sebastian is being dishonest. He still pretends that “assessing learning” is the same as “evaluating teacher performance”. This allows him to make the unfounded claim that teachers’ union object to assessing student learning.

(c) I do manifestly not agree that multiple choice tests are a good way to assess mathematics learning. They are a cheap way. Not at all the same thing.

(d) The whole idea that you learn for a year, then write one 4 hour test to determine what was learned is laughable beyond belief. The fact that anyone will actually defend this idea, when it is patently absurd, mystifies me.

Salient@185:
Don’t be a wimp. We’ve all been assaulted by SoV for our patent disregard of logic and bad faith and rudeness and provably incorrect assumptions. Just say what you think is right, in terms that will appeal to our lesser minds, if not the great mind of SoV. And then ignore the troll.

“As we have repeatedly said, the actual tests in use bear very little relation to curricula (unlike the AP exams) and consequently have a tendency to divert class time away from course aims. ”

Yes that is the *claim*. But the tests we actually see, (see linked above), don’t actually seem to be such a problem. It looks to me like it tests actual math understanding relatively well for a basic competency level. I understand that the teacher’s union claim that the tests are so horrifically awful. I just don’t see any evidence of that.

And actually I’m not even sure I’ve heard the the majority of the NCLB *tests* are bad, just that the 100% pass crap was problematic. Which is a perfectly legitimate gripe about what you do with the test, but not the test itself.

And on a related note, does anyone here actually know of a case where busting a union improved quality in an industry?

What do you mean, fix American schools? If it ain’t broke, don’t fix it.

http://scienceblogs.com/mikethemadbiologist/2010/10/i_agree_with_bob_somerbys_educ.php#more

It seems like U. S. schools have been doing a pretty good job, over the years. :)

The reason that multiple-choice tests don’t actually do a good job of testing whether people know something is because they don’t allow you access to the reasoning process. Certainly at the college level, no self-respecting teacher would administer one.

Salient:

On the other hand, I don’t particularly care if they were careless with minus signs or didn’t know the formula or what, though I can track the difference because a wrong answer from an minus-sign error at a key step was always available. The idea that I should award partial credit for a wrong answer because ‘at least the student knew the formula’ bothers me.

I don’t get this at all. Why would this bother you, exactly? It certainly didn’t bother any of my professors if I made a sign error somewhere. Yes, attention to detail is certainly important and one should always try to get the right answer; however, someone may simply make a mechanical mistake, whether it’s missing a factor of 2 or writing down a sign wrong or whatever. I think of that the same way I think of a spelling mistake on a writing test: unfortunate, but not so far as to undermine the rest of the work. This of course is the fundamental problem of multiple-choice tests: they don’t distinguish between someone who made a mechanical error and someone who just guessed (or, as happened in my case, someone who literally made a mistake filling out the Scantron).

As for the GRE, as christian h. mentioned above, it is nothing more than rent-seeking, a method of extorting money from a captive population. As far as I’m concerned, ETS ought to be one of the first organizations up against the wall when the revolution comes.

I note with interest that SamChevre has time to blow nonsense about climate science out his ass on another thread, but no time to respond to my #179 here. The conclusion is left as an exercise for the reader.

Yes, but you’re not arguing that only math teachers should be assessed on the basis of their students’ performance on standardized tests. You’re arguing that all teachers should be evaluated thusly.

Heh. Sebastian hasn’t shown that about multiple choice testing for math though. He’s done his usual:

a) Made a claim with no evidence or reasoning to back it up

b) Dared others to prove him wrong

c) Asserted that the people who challenge his claims “haven’t convinced him”.

d) Declared victory and accused others of being blindly partisan for not agreeing with him.

In any of the classes I teach, that sort of logic would earn him a straight F. To go along with what sg says above I’ll point out that not only do students seem to be disappointed to learn that tests won’t be multiple choice, but that since multiple choice is much easier to grade, math teachers pointing out their lack of utility in most circumstances aren’t doing themselves any favors.

Let me give yet another concrete example: we’ll be having an exam next week where I’ll give the students a rational function (a polynomial of m degree over a polynomial of n degree) and then ask them to graph it, showing x- and y-intercepts, vertical, horizontal and oblique asymptotes, etc. Now, I know from past experience that if I give the same rational function and then ask one multiple choice question at a time what are the x-intercepts, what are the y-intercepts, etc, the students tend to do much better than in open format. Why is that? Is the multiple choice format more accurate in assessing the student’s level of knowledge on curve sketching in this instance? Or the open format? Speaking as a math guy who’s administering the test, I’m going to go with the latter option. If Sebastian wants to he can try to “convince me otherwise” ;-)

I don’t get this at all. Why would this bother you, exactly? It certainly didn’t bother any of my professors if I made a sign error somewhere. Yes, attention to detail is certainly important and one should always try to get the right answer; however, someone may simply make a mechanical mistake, whether it’s missing a factor of 2 or writing down a sign wrong or whatever.

Salient seems to be a much harder grader than other math teachers who have opined on this one. In fact, I do give credit for “having the right idea”, and I tend not to grade overly hard on sign mistakes, improperly done elementary options like subtraction, etc. And this is for the excellent reason that I not only teach math, I do math, and I make those kinds of stupid mistakes myself all the time. I’d like to think I don’t make them because of insufficient mastery of the concept :-) The difference is, I have all the time in the world to catch those mistakes and correct them. The kids taking my exam? They’ve got just 60 minutes.

This of course is the fundamental problem of multiple-choice tests: they don’t distinguish between someone who made a mechanical error and someone who just guessed (or, as happened in my case, someone who literally made a mistake filling out the Scantron).

As I said above, I continually remind my students that an 80% correct answer in a multiple choice format is wrong.

As for the GRE, as christian h. mentioned above, it is nothing more than rent-seeking, a method of extorting money from a captive population. As far as I’m concerned, ETS ought to be one of the first organizations up against the wall when the revolution comes.

To be fair, these sorts of tests seem to be used to disqualify prospective students as opposed to qualifying them. To go with yet another analogy, on “House” it’s very easy to run a test that shows the patient isn’t suffering from, say, heavy metal poisoning. But it takes most of the show to figure out exactly what the patient does have.

OK, let’s me try answering (again) the question of what I mean by high-stakes testing.

I mean what the VEA uses it to mean–testing that is required for promotion (to another grade) or for graduation. This is a student-focused definition.

I’m not talking about evaluating teachers, I have nowhere in this thread been talking about evaluating teachers, and I’m not interested in talking about how to evaluate teachers.

Certainly at the college level, no self-respecting teacher would administer one.

This is bullshit, spoken by a person who either hasn’t spent much time coursing through universities’ old exams for introductory classes, or is provocatively calling a rather large portion of the field self-respect-free.

The kids taking my exam? They’ve got just 60 minutes.

Well, hell, strict time constraints will do more to inhibit reasoning than the way that final answers are reported. I have evening exams with a soft limit of 2 hours and a shared understanding that students can have some more time if they feel they need it. About two-thirds turn the thing in ~45 minutes early, and say they had more than enough time to review and check their work for errors — it’s an “hour exam” with time allotted. Increasing demand for precision needs to correspond to relaxing the time constraint, definitely.

But for example, some students mix up the quotient rule, effectively getting it backwards. The final result from this forgetfulness is a minus sign error, sure. However, a student who is thinking carefully about the problem, and checked their work by viewing the function on a graphing calculator, ought to realize the sign of their answer must be incorrect. If they don’t, that’s either a conceptual deficiency (not understanding that an increasing function has positive slope) or maybe a failure to carefully check work. I want my students to be mindful enough to engage in that sort of work-checking, and I model how to do that in class. Seems fair to expect it on tests, and my average scores etc do seem very comparable to grade distributions elsewhere.

On the other hand, weird random minus sign errors that are unrelated to formula mismatch or some such thing usually result in an answer that’s not available on the answer form. The student then (hopefully) knows to review and revise their work, (hopefully) catching the error in a second pass.

I do give credit for “having the right idea”

I protest vociferously the notion that ‘memorizing the formula accurately’ or even ‘knowing which formula applies’ is the same thing as, or is even indicative of, “having the right idea.”

But this is getting a bit far afield from the topic of union-busting, and I do happen to have very limited control over my testing format. I just think there’s a bit too much hyperbole being dumped on multiple-choice answer sheets, as if the moment you let someone compare their final answer to a list of possible final answers all hell breaks loose.

SG: “Can you tell me why my students prefer multiple choice? Do you think it’s because they think it provides a more accurate assessment of how well they have learnt the material, or do you think it’s because they realize multiple choice exams are easier to pass?

And on the topic of multiple choice, do you accept that maybe it’s a lot easier for an academic, usually untrained in teaching theory per se, to set an exam with written content that correctly judges what the student has learnt, than to set a multiple choice exam with the same goal?”

My problem with this argument is that you’re arguing in the opposite direction from what a competency test measures and why teachers unions appear to be so scared of them.

You are arguing that multiple choice tests are too easy. But when teacher’s unions reject linkage between student performance on exit tests and teacher’s performance (even when tracked over many years, and even when paired with entrance exams to show the value added to performance over the year so that a teacher doesn’t get hit with the fact that they just got a bunch of students who were behind) they aren’t doing so because they worry that the test is too easy. They aren’t worried that too many students will pass the competency test when they should be failing. They aren’t worried that too many bad teachers are actually going to get good grades because their students will overperform.

Your objections are the exact opposite of what they are worried about.

As for test design, I’m not sure what you’re asking from me. I don’t think I’m suggesting that every single teacher should individually design a state or national competency test. The fact that most academics would have trouble designing a good one doesn’t imply that the actual people designing the test can’t do it.

So far as I can tell, the problem isn’t that we are having too many kids passing competency EXAMS when they don’t have real competence. The problem we have is that too many kids pass years and years of *teacher assessments*, not controlled by outside assessments, and then at the end graduate from high school without the ability to do even basic math or basic reading.

But even on a theoretical basis, your objection falls on the high end of achievers. Yes, multiple choice tests have trouble appropriately discerning between the students who have very good understanding of the topic, and those who have an even more deep mastery of the field. Yes, for *individuals* you might get one person who is slightly better than another look worse because the slightly better one got tripped up in an odd way. But over populations of students, especially over multiple years, and especially since you are attempting to measure general floor-level competency and not trying to finely discriminate between different varieties of brilliant, your objections don’t apply.

Peter Nunns: “Math teachers are teachers.
Math teachers can be evaluated effectively by their students’ performance on a standardized exam.
Therefore, all teachers can be evaluated effectively by their students’ performance on a standardized exam.

But logic doesn’t work like that. Engage with the hard cases, or GTFO.”

Learning, teaching and understanding does work like that. When you want to debate about anything, you have to get at the building blocks. If you want to talk about calculus, you don’t do it until you can talk about arithmetic first. If you want to do trig you have to cover algebra first.

The argument has essentially boiled down to: standardized tests can’t measure basic competency (with some arguing that maybe they *could* but that they never do in practice). There are various explanations for this contention, some of them contradictory, but a lot of them came down to

it causes teaching to the test (i.e. teaching kids to pass without them being likely to get actual competence).

I said (in comment #20, so it isn’t a recent development in my thinking) “My major criticism of the “teaching to the test” criticism is that while it may be an excellent point for certain disciplines (most obviously history for example, where for a sufficiently narrow test, teaching directly to the test could lower overall perspective in the discipline), teaching to even a moderately good math test is teaching math. For the levels of reading that we are talking about, teaching to the reading test is teaching reading. Teaching the test for those kind of things, at that basic a level, isn’t likely to be bad.”

This has provoked quite a bit of disagreement.

When you disagree over the easy case (math) it makes no sense to jump right to the harder cases.

I’m not abandoning the hard cases. I’m just not willing to tackle them with people who can’t even admit that math competency (not math brilliance just basic competence) can be detected on standardized test–including ones, like linked above–that are already in use. I’m not committing a logical fallacy by focusing on the easy case first. I’m committing basic logic–talking about premise level ideas first rather than skipping them and then invariably getting muddled because we ignored premise level disagreement.

Which brings us back to the real world. The actual test that Stuart Buck linked in #128, actually used by Florida, which is actually one of the larger states, appears to be just fine. Like all tests, is it probably gameable on the extreme margins? Sure. But does it seem to require a pretty non-awful level of math understanding to get by it? Yes. Is it multiple choice? Yes. Oh no!

SoV, is as usual trying to distract by attacking the people instead of the argument. So I’ll redirect attention to the fact that you haven’t answered Salient’s points about multiple choice tests. You just attacked his character. And so far as I can tell you aren’t engaging any of my points whatsoever. I know you always play the burden of proof game. You’ve been provided an actual test, actually in use, by one of the actual largest states in the US. You haven’t even attempted to show us what is so awful about it. Are you going to avoid any actual analysis whatsoever by asserting burden of proof? I would say the premise that low level competence can be detected in testing is relatively intuitive. Especially when we can say–see here is a test that does it. Maybe, just this once, for the very first time ever since you began arguing on the internet, you have the burden of proof. Or even the burden of suggestive hinting.

My problem with this argument is that you’re arguing in the opposite direction from what a competency test measures and why teachers unions appear to be so scared of them.

I judge you by your utterly pathetic standard of argumentation here, and say that you clearly cannot be a competent lawyer and your employer should fire you.

See any problem with that?

I can only repeat what Harry said way back at #7:

I’ve said this before, but in my own interactions with (apparently recalcitrant) teachers union leaders, they understand perfectly well that the evaluation system is stupid, and that incentives need to be put in place. They deeply, deeply, distrust the competence of administrators at the school and district levels. This reflects, in my view, their ability to see what is patently obvious, rather than any paranoia.

That really should have been the end of this particular branch of the discussion.

Well, to leave this discussion agreeing with someone–good MC questions and answers are very labor-intensive to write. Good and not easily gameable is even harder.

That’s why I’d argue that they are appropriate for end-of-year assessments on a statewide level, but very poor tools for normal everyday classroom use.

It looks to me like the educational counterpart of security theater: something is being done for the sake of showing others that we’re doing something and meanwhile no effective measures are being undertaken.

Excellent analogy- I intend to shamelessly steal it.

Rhetorical questions, of course. It’s exactly what we should be doing, but we’re not. As such, I’m pretty skeptical of all this pro-testing hoopla. It looks to me like the educational counterpart of security theater: something is being done for the sake of showing others that we’re doing something and meanwhile no effective measures are being undertaken.

Of course. Contrary to what Sebastian has mischaracterized, most (i.e., approaching 100%, at least those I know of) teachers aren’t against accurate year-end assessments. The problem is that the people who are really pushing for this want to do it on the cheap (and therefore poorly), in a way that doesn’t put any expectations on certain constituencies (like a certain type of parent), and done by people who give every indication of wanting to use the results as a club against individual teachers and their union, as opposed by the people best able to administer and interpret the results: the teacher’s themselves.

@172: My girlfriend tells about how, when she was in primary school learning arithmetic, to multiply a number by 48 she would first multiply it by 50, then multiply it by 2, and then find the difference.

Your girlfriend didn’t have these teachers.

The version in the printed paper also included a grade 5 question asking for a proof involving arithmetic sequences.

Salient 185, a couple points. You say that incorrect answers get a -1, no answer gets a 0, and correct answer +2. Why are you punishing the wrong tries? I don’t get this at all. Isn’t this needlessly intimidating? Many students are unsure of themselves and need positive reinforcement rather than “don’t even bother trying, you will screw up anyway”. Granted testing time may be too late to deal with this problem but the way you test is probably related to the way you teach. There has been a lot of emphasis recently on the need to encourage students to try something out, experiment, play, encourage learning from mistakes instead of turning mistakes into a demoralizing, discouraging experience. The emphasis on correct results, rather than the intellectual process, in traditional math instruction has been heavily criticized for promoting “math anxiety”. This is not to advocate credit for wrong answers but punishing wrong tries over not trying at all seems hard to justify (although I can see your point as a TA having to grade 500 tests – but what I am saying is precisely that testing needs adequate resources).

From your comment I gather that you are testing for mechanical applications of mathematical rules. Perhaps MC tests are adequate for testing this but I simply don’t agree with reducing math to a mechanical discipline. Do your students ever have to derive a result? Do they ever execute a proof? Do they learn to use mathematical tools for solving real-world problems? Maybe I am idealistic but that is what I think the essence of math instruction should be.

Because I have better things to do with my time than to provide you with evidence that you have done things that you have done. But sure, ok, google provides—wasn’t that hard to google Yglesias LaTeX ScentofViolets. IIRC, we first crossed paths here, comments 9, 10, etc. to comment 47 or so. Of course, now that I have provided exactly what reminders you’re asking for, you’ll probably want to argue about how reasonable you were and how unreasonable I was for hours. We both have better things to do.

Ah, that’s who you are. You are aware of course, that you’ve just disproven your rather wild allegations, right:

I owe an apology to identifying as a bully someone who once followed me from here over to Matthew Yglesias’ blog (which is, assuredly, the end of the earth) in order to continue arguing with me about whether my use of LaTeX was appropriate?

So. In both threads, opinion was running rather heavily against you, and they were completely unrelated. Btw, you do know you can go to both discussions can do a find on LaTeX, right? Let me know when you find that occurrence in both of those discussions.

Geeze, you really carry a grudge around don’t you? Next time, don’t be so quick to accuse someone of stalking or “harassing” you. Not that I’m expecting an apology of course.

Couldn’t you guys just drop the part of your conversation that is unrelated to this thread? Don’t respond, just drop it. Please.

Granted testing time may be too late to deal with this problem but the way you test is probably related to the way you teach.

No, the way I test is mostly mandated by the department and therefore almost entirely out of my control, partly because I’ve gotten in trouble [for some mild definition of that word] for being too creative with my examinations, specifically for being too theoretically or structurally inclined and therefore too ‘hard’ on the students, by requiring proofs and derivations and the like. Ironically, my tests for those semesters were substantially more like what other folks seem to be advocating for here. But then, students complained of the course being too demanding (by comparing my tests to others’ tests which weren’t doing the things that are lauded here), and here I am, effectively demoted-without-demotion to r&g duties for the year.

Hm. Perhaps I’ve grown too reactionary in response to having been rebuffed, though. I was meditating earlier today on the fact that I probably wouldn’t be using a test format anything remotely like this, if I had control over it, and frankly would be rather likely to be joining the camp of folks critical of multiple-choice tests. (And I’d still be dazedly grading at 3am tonight.) That line about people being very good at understanding what their bosses pay them to understand, probably comes into play here against me. Strange to realize this. :-/

Anyhow, piglet – I probably agree with [or could easily come around to] the entirety of your comment, even where it disagrees with stuff I said above. Still, looking at exams for comparable courses, including those written in non-MC-format — proofs and derivations generally aren’t required here or at comparable institutions, and the real-world examples are reduced to mostly mechanical problems. I actually have access to all of the information that I would have had access to on a comparable non-MC exam, as showing work’s required, etc.

Of course, it seems several institutions call this class remedial and outsource it to a local community college, and several others don’t seem to even offer it, so there may be a bit of “we wash our hands of this” going on.

Still, the class [where it exists] is applications-oriented, set apart from the Calculus of a Single Variable sequence for students who won’t ever take another math class beyond this. The goal of the class is explicitly to put elementary calculus tools in the student’s computational toolbox. I’m okay with folks calling that ‘not doing math,’ though I wouldn’t be so harsh, but I can see some argument for offering students a computational-facility-oriented class that doesn’t delve into the theory more than superficially, and teaches them to be fluent computers for technical work.

Why are you punishing the wrong tries?

Hm. This (and what follows it) is a heartwrenchingly good point, and seems like the sort of thing I would have said myself even just a year ago. Sheesh, what kind of corner have I backed myself into? Something for me to think on further. Short cop-out answer is, I didn’t choose the test format, and while I contribute some questions to the exam in proposal form, I don’t have too much more control than that. Of course, that’s a cop-out, but I’ll leave it at that.

although I can see your point as a TA having to grade 500 tests

Technically, we only have to put effort into grading quizzes, due to automation.

Old business: First, how about Pax, okay? What’s been said and done is water under the bridge for both of us, whatever our individual perceptions. Are we cool?

New business:

No, the way I test is mostly mandated by the department and therefore almost entirely out of my control, partly because I’ve gotten in trouble [for some mild definition of that word] for being too creative with my examinations, specifically for being too theoretically or structurally inclined and therefore too ‘hard’ on the students, by requiring proofs and derivations and the like. Ironically, my tests for those semesters were substantially more like what other folks seem to be advocating for here. But then, students complained of the course being too demanding (by comparing my tests to others’ tests which weren’t doing the things that are lauded here), and here I am, effectively demoted-without-demotion to r&g duties for the year.

Congratualations. Yes, that is exactly what a lot of us would like to see more of. And yes, a lot of us have gotten complaints from our students about the tests being “too hard”. And here is my essential beef: the department heads and administrators will let the kids (more realistically, their parents) get away with dictating policy. So on the one hand, I can’t have too many F’s or D’s, but on the other, all of us (other teachers that is) get flack from the administration when these same students don’t do so well on standardized tracking tests. So I’m put in the position of not assigning enough homework, grading easy, and turning out high-scoring alumni.[1]

I think that’s a pretty indefensible position to be put in – especially as I and a lot of other teachers I know put in a lot of unpaid extra time just to help the students who say they can’t meet during regular office hours. Heck, I’ve come in on Saturdays, and have seen at least three other doors open. I think these kids need (and deserve) a good education. And I’m willing more than willing to devote extra time to see that this happens. But this has got to be a two-way street, you can lead a horse to water and all that.

[1]I should mention that there are several tracks. Students on the hard sciences/engineering track still tend to be pretty good students and we have a calculus series just for them, for example. On the other tracks? Well, that’s where the problems show up. I also teach business calc every so often, and you have finance students complaining that the math is “too hard”. To name but one but by no means the most egregious example.

Sebastian, your whole answer to me at 213 seems to consist of an innuendo, suggesting as far as I can tell that teachers objections to multiple-choice tests are out of fear of being assessed themselves… is that right?

You are doing what SoV accuses you of, because it’s very clear from comments here that teachers only objct to MCE on the basis of its value for assessing learning. So why are you claiming otherwise?

You haven’t taken account of my point about Asian languages I see. And you obviously don’t understand either a) what maths teachers are aimin to assess (or teach) or b) how MCEs are written.

For a) you seem to think maths tests check whether someone got the right number, i.e. if they added up correctly; and for b) you seem to think a good MCE is easy to set.

The quality of arguments being raised against you is so much higher than anything you have offered, it’s as if you’re being actively rude by persisting…

Sebastian, the AP test has a free response section for calculus, not a multiple-choice section. The AP test is the only example you have “presented” (free of links) and it doesn’t agree with your position. From the AP Test website:

The AP Calculus AB Exam covers differential and integral calculus topics that are typically included in introductory calculus courses at the college level. Because graphing calculator use is an integral part of the course, the exam contains questions that require students to use a graphing calculator. Students cannot take both the Calculus AB and Calculus BC exams during the same year.

You can find additional free-response questions and scoring guidelines on AP Central, along with grade distributions and examples of actual students’ responses and commentary that explains why the responses received the scores they did.

And your original claim was that MCEs are sufficient for calculus.

So, can we now agree? You were wrong. You’re also wrong about Japanese and Chinese (the writing component at any level can’t be assessed by multiple choice), and most of the teachers here think that more basic maths can’t be assessed that way either. Given you’ve been wrong about everything else, perhaps you should give up while you’re ahead?

Somebody please let Sebastian know about the First Rule of Holes. This is getting embarrassing.

Sg, “Sebastian, the AP test has a free response section for calculus, not a multiple-choice section. The AP test is the only example you have “presented” (free of links) and it doesn’t agree with your position.”

The AP calculus test is 50% multiple choice. The weight of those questions is normally 45% but sometimes changes depending on other factors. The fact that you didn’t discover that, especially when I had already said that, suggests to me you didn’t look very closely. Thanks.

(In case you don’t believe I said that, see 232: “Where did we get the misapprehension that the well designed AP tests aren’t multiple choice? I did a quick survey of them, and all the ones I looked at are at least half multiple choice by time, and no lower than 45% multiple choice by scoring weight. And that includes even the ‘hard to grade’ things like English, History, and Music Theory. The idea that multiple choice tests can’t form the core of good assessment isn’t supported by the so called ‘good’ tests that you’re citing.”)

Jerry, I’m not sure what you don’t like about the stem and leaf question. It is the very most basic introduction to ideas like frequency and can be taught without algebra. It is a fairly intuitive concept for primary school kids who understand the relationship between numbers. I suppose if you think it shouldn’t be on the curriculum at all, then I guess it shouldn’t be there.

Your problem with question #2 is especially weird. The kids are being led to the correct answer by reinforcing the relationship between addition and multiplication. The prefacing paragraph isn’t an example of the kind of word traps that people worry about upthread, the test takers have given you two forms of the question, both leading to exactly the same answer, while simultaneously reinforcing the relationship between the symbolic question and the real world question. You see that again and again in this test–the problems reinforce, every single time, the relationship between the symbolic representation of numbers and the actual instance of numbers in a problem. And for someone who has seen too many high school graduates who have trouble with that, I think that is a great thing to put in a test for sixth graders.

The only questions that don’t really follow that format are #3 (Which of the following numbers has a value between 0 and 1?)–a straight test of fractions, #7–a straight test of knowing the definition of the word ‘mean’ in math, and #11 which is pretty much just a ‘do you understand what the sign means’ question. Rather than being a negative property, it looks like you’ve identified a positive property of the test.

Sg, I’m not sure what concession you’re trying to wring from me. That the writing portion of writing tests can’t be multiple choice? Ummm, ok. I don’t know where I’ve suggested otherwise.

But unless you are restricting yourself to JUST the writing portion of writing tests, you’re wrong about the use of multiple choice on AP tests even for the Asian languages. As you can see on this test for the AP Chinese Language and Culture course, 50% of the weight of the exam is in the form of multiple choice. I can’t personally judge how good the questions are, as I neither speak nor read any dialect of Chinese. But those pesky multiple choice assessments seem to form an astonishingly high percentage of even the ‘good’ tests. Which may come as a surprise to people like SoV, who assert that no self respecting person would use them for anything more complicated than remedial algebra. (Which of course is often a higher level of sophistication than we are even testing for, a fact which seems to go mostly unacknowledged in this discussion).

A colleague has pointed out to me that without the long-form quizzes we generate and grade, the information gleaned from exams would be far less useful. I was taking for granted the opportunity to formatively assess students each week in long form, and thereby get fairly intricate samples of their cognitive process in action. Without that data, I would be at a a substantial disadvantage as an assessor of learning. So I must take back a good deal of what I said upthread, as regards the utility of multiple choice testing (at least as a sole or predominant measure of performance). At best, it allows for quick verification of learning between the formative assessment and summative assessment stages: but I wouldn’t know how to interpret those scores if I didn’t have long-form assessment data handy.

It’s like the difference between sampling a output of a function f(t) where input t = 10, and having the graph for f(t) for 0 < t < 9 and then sampling the output where t = 10. If I can already see a pattern of growth or stagnation in student learning, of course a quick check to see if that pattern has continued or changed course is about all that's necessary; my confidence in my analysis of earlier long-form assessments is carrying over to confidence that the MC-exam assessment result is meaningful.

But for one student, I'm interpreting their score of 34/40 one way, and for a different student, I'm interpreting their score of 34/40 a different way, and that [a] requires long-form data appropriately analyzed, [b] leads me to often significantly different conclusions about how much learning has actually occurred, for identical MC scores, and [c) therefore clarifies that the MC scores alone do not all by themselves evaluate student learning in a fair or accurate or precise way, and thus can’t form the core of any evaluation of teacher value added.

It is a fairly intuitive concept for primary school kids who understand the relationship between numbers.

A stem and leaf plot can help students, in a particular formative stage of their learning, the role of digit in the concept of number. It can also be produced mathematically without any underlying understanding of digit. So it’s not useful for assessment of learning.

It’s a linguistic sandtrap that hurts the students because it forces them to expend effort in places where it’s not required.

Agreed! And then high school teachers are told to “teach to the test” by teaching students to ignore the explanatory chunk of text and just read the question, which is actually destructive to their ability to apply what they’ve learned in non-testing environments.

My objection to the stem-and-leaf plot question would be that there is no application. Students are taught what a stem-and-leaf plot is but are they also taught what it is good for and why one might want to use it (i. e. as a simple tool of data exploration)? We can’t tell from that test.