Could Busting Unions Fix America’s Schools?

by Harry on October 18, 2010

(Preface: if you, sensibly enough, want to avoid my rambling and get straight to the point, just go and read Richard Rothstein on the Rhee/Klein manifesto now. Update: a nice related post, which will now be followed by interesting discussion, at Laura’s).

I’ve managed to resist seeing Waiting For Superman so far. The trailers promise me that I won’t like it much. My wife gets a free showing on Wednesday, so she can report to me. Ironically, the book on which it is apparently based, Paul Tough’s Whatever It Takes, is really not at all bad. It is true that, as a friend of mine said, “He has drunk the Kool-Aid”. But unlike many Kool-Aid drinkers (and there are a lot of them, I gather), he displays pretty clearly all the evidence you need in order to judge its toxicity. His account of the social science around low-end academic achievement is pretty careful, and entirely readable, and the narrative is well paced, and the story informative. For Tough, urban school districts need more Promise Academies and KIPP schools. I tend to be in the, “lets try it and see if it works” camp myself, though with an emphasis on actually trying to figure out whether it really does work. More than any academic study I have seen, Tough’s book makes me sceptical. He makes clear that Geoffrey Canada, the CEO of the Harlem Children’s Zone, became obsessed with driving up test scores. Remember “test scores” means “math and reading test scores”. From a low base it is not that difficult to drive them up — simply restrict your attention to driving them up and allow math and reading to be the more or less exclusive focus of the curriculum. This is what he did, with modest effects on reading scores (which are harder to manipulate by teaching to the test) and greater effects on math scores. It really may be that for these kids improving their math skills somewhat and their reading skills slightly is the best thing for them (though, whereas there is a correlation between test scores and later success, we don’t have any evidance that improving children’s test scores improves later outcomes, and we have lots of evidence that these bumps in test scores from grade-specific interventions typically fade pretty quickly). Maybe, maybe not. Whether any child in that school actually learned more, or anything more useful, as a result of this, we have no way of knowing.

Back to the movie.

It has certainly become the catalyst for debate, which I suppose could be good. Last week saw the publication of the Klein/Rhee manifesto, much of which is a lightly coded version of a call to get rid of the teacher’s unions, and the promulgation of a standard narrative that all we need are good teachers to save these children from educational disaster. Beyond the obvious question — once you’ve gotten rid of the teachers’ unions what are you going to do about the school boards and administrators who have been signing contracts with them for decades and who, unlike the unions, actually have managerial responsibility and authority? — there are many others. Last Thursday Richard Rothstein responded extremely quickly, and devastatingly, to the manifesto. Because his writing is so tight there’s nothing dispensable — you have to read it all. Here’s a representative paragraph:

Even the president’s more careful statement — that teacher quality is the most important in-school factor — is actually without solid foundation in research. It is true that some studies have found that variation in teacher quality has more of an influence on test scores than do the size of classes or average district-wide per pupil spending. In other words, you are better off having a good teacher in a larger class than a poor teacher in a smaller class. But that’s it. It is on this thin reed that Joel Klein and Michelle Rhee are mounting a campaign to make improving teacher quality, and removing teachers whose students’ test scores are lower, the centerpiece of national efforts to improve the life chances of disadvantaged students.

There are plausibly many other in-school factors, not quantified in research, that could have as much if not more of an influence on student test scores than teacher quality. Take the quality of school leadership. Would an inspired school principal get better student achievement from a corps of average-quality teachers than a mediocre principal could get from high-quality teachers? Studies of organizations would suggest the answer is yes, but there have been no such studies of school leadership. Take the quality of the curriculum. Would average teachers given a well-designed curriculum get better achievement from their students than would high-quality teachers with a poor curriculum? A very few research studies in this field suggest the answer might be yes as well.

Or take another in-school factor, teacher collaboration. Even when elementary school students sit in a single classroom for most of the day, several teachers influence their achievement. Teachers can meet to compare lesson plans that worked well and those that didn’t. Teachers in lower grades can successfully align their instruction with what will be most helpful for learning in the next grade. Teachers of the arts can reinforce the writing curriculum, and vice-versa. Will average-quality teachers who work well together as a team with the common purpose of raising student achievement get better results than higher-quality teachers working in isolation? Plausibly, the answer is yes. Will promising to pay individual teachers more if their students get higher test scores than the students of another teacher reduce the incentives for teachers to collaborate? Again, a plausible answer is yes.

{ 249 comments }

1

MPAVictoria 10.18.10 at 3:11 pm

I find it very revealing that those in favour of school reform routinely focus their attention on teacher unions as the source of the problem. If teacher unions were the problem than students in countries that are more highly unionized, for example Canada, Sweden, Denmark and France, should be doing worse. Since these countries are doing much better than the United States on international comparisons I think it may be safe to say that unions are not the problem.

2

laura 10.18.10 at 3:13 pm

3

epist 10.18.10 at 3:16 pm

I’m waiting in joyful hope for the day when a spike in fires or crime results in a chorus for the abolition of the police or firefighter unions. Funny how poor performance in those professions never seem to bring out these sorts of demands. I wonder what the difference might be…

4

MPAVictoria 10.18.10 at 3:18 pm

epist:
Just you wait friend. These people will not rest until their is not a unionize job in the whole world.

5

MPAVictoria 10.18.10 at 3:18 pm

There not their.
Sorry.

6

tarun 10.18.10 at 3:18 pm

There are very valid reasons to focus on teachers rather than the factors outlined in Rothstein’s article, namely that teachers are a controllable factor (in districts that have had their unions busted) and that mass unemployment and social ills are so multifactorial and poorly addressable that they are definitely higher hanging fruit.

Of course there are other problems, US schools are ridiculously costly, high on overhead, and subject to political manipulation.

It also doesn’t help teachers that they operate on strict seniority and that they resist any attempt to measure effectiveness (which just leaves test scores). Rothstein throws out a bunch of theories on other approaches, but there is a sense these are only offered because the current status quo is under siege – why did it take the threat of busting unions for him to suggest “working as a team” or “incentive pay” or implicitly assert that the research just doesn’t seem very conclusive (i.e. it isn’t very good)?

7

Harry 10.18.10 at 3:42 pm

Nonsense, lots of people have been saying the sorts of thing Rothstein is saying about in-school factors, for years. Of course, all the research is inconclusive, but, for example, the view that leadership matters a good deal has a long history. And one might think that, given that there are many fewer leaders than teachers, and they are already not unionized, and the apparent average quality is astonishingly bad, they are the lowest hanging fruit.

I’ve said this before, but in my own interactions with (apparently recalcitrant) teachers union leaders, they understand perfectly well that the evaluation system is stupid, and that incentives need to be put in place. They deeply, deeply, distrust the competence of administrators at the school and district levels. This reflects, in my view, their ability to see what is patently obvious, rather than any paranoia. Principals can perfectly well get rid of teachers in their probationary years, but because most know nothing about instruction, and the rest don’t see improving instruction or managing human capital as any part of their job, they often don’t even bother observing. Get rid of the teachers unions, and teachers are subject to the arbitrary power of people who know nothing and have less than no interest in what really matters in the school. Lousy leadership is bad for teachers and for student achievement.

I agree about such things as reducing child poverty — it is clear that American voters are not interested in that, and just want quick fixes. But early childhood education is do-able and popular, and early screening for poor hearing and eyesight, and making needed prescriptions is cheap, has educational benefits for the kids afflicted and other kids in their classes, and is valuable independently of that. Any policymaker has limited space for action, and rather than thinking there are in-school and out-of-school menus, the responsible policymaker looks at a large single menu, and figures out which, among the feasible items, are most cost-effective.

By the way, busting unions takes a hell of a lot of work, and is politically very risky. Winning their trust is what competent leaders do.

8

Miles Townes 10.18.10 at 4:12 pm

My reaction to the Rhee-Klein argument is that they are working from the wrong assumptions. They point to the need to prepare students for the “21st century global economy” – as if the entire point of public education is to produce good little workers to stamp out bits of GDP. I can’t think of a worse way to motivate somebody to learn science and math than to point to the economic needs of this country moving forward: “Our generation borrowed a lot of money that we expect your generation to pay for, so here’s some books about differential calculus and organic chemistry. Now get to work – the Japanese are four years ahead of you.”

Moreover, they’ve nicely ignored the fact that the problems of the very present 21st century global economy are political failings, not a lack of nerds to do sci-tech jobs. And if our schools are bad on science and math, they are terrible – appalling! – at teaching kids how to be civic and political actors in society. Insofar as our problems are political, the last thing we need is more drones. I’m not sure that teachers are totally of the hook for that – I can point to some history and gov’t teachers in my education – but I think the bigger problem is the slashing of curricula designed to teach civics, politics, history, literature, and – yes – economics. That seems to me a failing at the superintendent level, if anything.

9

Jaybird 10.18.10 at 4:19 pm

There are schools in this country that I would send my children to.
There are schools in this country that I would choose private (even Catholic) schools over for my children.
There are schools in this country that I would beg, borrow, steal, and kill to get my children away from.

I’m sure that all of us would agree with each of those.

I’m also pretty sure that all of us send our above average children to above average schools… and a change in the status quo would negatively affect us (and our children).

10

Davis X. Machina 10.18.10 at 4:19 pm

“These people will not rest until their is not a unionize job in the whole world.”

Because so long as one of us, somewhere, is covered by a collective bargaining agreement, none of us, anywhere, is truly free.

11

ScentOfViolets 10.18.10 at 4:28 pm

It really may be that for these kids improving their math skills somewhat and their reading skills slightly is the best thing for them (though, whereas there is a correlation between test scores and later success, we don’t have any evidance that improving children’s test scores improves later outcomes, and we have lots of evidence that these bumps in test scores from grade-specific interventions typically fade pretty quickly).

I suspect this has something to do with all those studies that say kids are mostly influenced by those around them, which is to say, parents and teachers earlier, other kids later. In any event, I won’t stop saying this: the people most responsible for their kids academic performance are the parents. That’s one group these “responsible conservatives” don’t want to take on.

12

Dirty Davey 10.18.10 at 4:32 pm

The best evidence for the union-busting argument is the universally high quality of public education throughout the South. If those silly northern and urbanized states would just get rid of their teachers’ unions, they’d end up with school systems as good as the ones in Mississippi, Alabama, and North and South Carolina.

13

Jaybird 10.18.10 at 4:38 pm

When asked the best way to improve a school, Daniel Patrick Moynihan “offered the policy proposal that states wishing to improve their schools should move closer to Canada”.

14

Sebastian 10.18.10 at 4:48 pm

“If teacher unions were the problem than students in countries that are more highly unionized, for example Canada, Sweden, Denmark and France, should be doing worse.”

This would be true if teacher’s unions in those countries were fundamentally similar to those in the US. I’m not so sure that is obviously the case. Do unions in Canada, Sweden, Denmark and France resist replacing evaluation systems that they know are stupid, and work to avoid incentives (as Harry put it in #7?). My impression is that the teacher’s unions don’t function that way, or at least not to the extreme that the US teacher’s unions do.

“It really may be that for these kids improving their math skills somewhat and their reading skills slightly is the best thing for them (though, whereas there is a correlation between test scores and later success, we don’t have any evidance that improving children’s test scores improves later outcomes, and we have lots of evidence that these bumps in test scores from grade-specific interventions typically fade pretty quickly). ”

Isn’t later success what we want? We don’t want good test scores for the hope of better test scores later. We want good test scores for the hope of better outcomes later, right? Why in the world would we want to be dismissive of that?

15

Harry 10.18.10 at 4:56 pm

Sebastian — sorry, I should have been clearer. The point is this. We have evidence that correlates test scores with later success. But no evidence that increasing test scores correlates with, let alone causes, later success. And lots of evidence that interventions that do little or nothing for test scores enhance prospects of later success a lot. In other words driving schools to raise test scores may be a terrible waste of resources if we are trying to produce later success.

16

piglet 10.18.10 at 5:26 pm

Sebastian: “My impression is that the teacher’s unions don’t function that way, or at least not to the extreme that the US teacher’s unions do.” Your “impression” is informed by what direct knowledge of teacher’s unions in Canada, Sweden, Denmark, and France? If your answer is “none” then please spare us.

The union busters’ case seems to rest on the claim that American teachers and/or American teachers’ unions are somehow intrinsically worse than those of other developed countries. Maybe that is true but I have never seen an explanation why that would be the case.

17

Jeff R. 10.18.10 at 5:27 pm

On the other hand, it certainly stands to reason that a student who cannot read, or who reads at a grade level several years lower than their actual age, is going to have a significant barrier in learning, well, anything else along their academic career that the student who is reading at their own or a higher grade level will not.

There’s probably a similar, although not quite as strong, case with regards to learning math and acquiring a working grasp of rigorous logic and problem-solving skills.

(If it is the tests that are the problem, and they are not actually measuring practical reading and math skills, shouldn’t the favored approach be to go and make a better test rather than to throw up our collective hands and claim that the only way to solve the problem is to throw more money than will ever plausible actually be thrown at it at it?)

18

SamChevre 10.18.10 at 5:39 pm

I would strongly recommend reading Ta-Nehisi Coates posts, and the comments, on this subject:

Flight to Canada
Flight to Canda, continued
The ignorance of what is Possible

http://www.theatlantic.com/national/archive/2010/10/flight-to-canada/64428/

19

Matt Warren 10.18.10 at 5:40 pm

Lots of interesting comments. My wife is a teacher and we’re going to end up seeing the movie eventually because it’s gained the attention of so many parents.

A few of my wife’s fellow teachers have found WFS to be a bit adversarial, though. The existing stereotype may be fossilized teachers with tenure, but around our neck of north Seattle, teachers work for low-to-middling pay, put in lots of overtime, and are blamed for damned near everything. Many teachers do what they do because they are committed. Even if they are in error, the U.S. isn’t exactly doing a super job of building trust.

Which is not to say that everything’s going swimmingly, because it’s not. But there are aspects of a child’s education which we don’t care to seriously discuss because magic worlds like “socialism!” are uttered. Habits, diet, and home-life fall under that fear. The last of those three is among the most important, too.

We clearly have different ideas of what education should be, how success is measured, and what to focus on. My worry is that we’ll treat “charter schools” as yet another magic word that solves everything. You know, the way we hitch our notions to fad diets to lose weight.

Then it could fail, we’d lose faith, and feel more helpless. Charter schools may be a super duper awesome idea, but pushing it via a movie starring an administrator that boasts about how many teachers have been fired is a thoroughly lousy sell.

20

Sebastian 10.18.10 at 6:03 pm

Ok, I think what you’re getting at, and while it isn’t personally convincing, it certainly is more convincing than the first stab at it.

My major criticism of the “teaching to the test” criticism is that while it may be an excellent point for certain disciplines (most obviously history for example, where for a sufficiently narrow test, teaching directly to the test could lower overall perspective in the discipline), teaching to even a moderately good math test is teaching math. For the levels of reading that we are talking about, teaching to the reading test is teaching reading. Teaching the test for those kind of things, at that basic a level, isn’t likely to be bad.

21

Sebastian 10.18.10 at 6:07 pm

“The union busters’ case seems to rest on the claim that American teachers and/or American teachers’ unions are somehow intrinsically worse than those of other developed countries. Maybe that is true but I have never seen an explanation why that would be the case.”

To start the conversation do we need ‘why’? First we need to agree that it is true, if it is. It seems to me that it is true, but if can’t agree on that, we probably need to start there.

22

MPAVictoria 10.18.10 at 6:20 pm

Sebastian why does it seem to you that teacher’s unions in the United States are worse than those in say Canada? I have a great deal of personal experience with this issue and I would not consider this to be the case. Please tell me why you do.

23

SamChevre 10.18.10 at 6:38 pm

I would say that American teacher’s unions, and American teachers, are probably not much different than those in other countries.

What is very different is the American school system.

In France (and it’s my understanding that this is pretty typical in Europe), publicly schools are much less monolithic than in the US. There are publicly funded religious schools, schools for various levels of academic achievement, and so forth.

If you think that the US schools are probably as good as you can get with a one-size-fits-all solution, that the teachers union is going to provide pressure for stability, and that a more varied set of options would be better than a one-size-fits-all solution (all of which I would think), then the teachers union will be problematic in the US in a way that they won’t in Europe.

24

Gene O'Grady 10.18.10 at 6:39 pm

A word of thanks to those who have focused on leadership rather than teachers as an area where positive change is possible. Looking back over my kids’ time in school (in Palo Alto, a district with putatively excellent schools) nothing made nearly as much difference as the quality of the school principals. This includes good ones instantly improving a whole school and bad ones wrecking the place.

25

Jim Harrison 10.18.10 at 6:42 pm

Americans, especially conservative Americans, are deeply conflicted about education. They want the benefits of having an educated population because even the most moss-clad reactionary is in favor of the money and military power that go along with technology, but they resent educated people as elitists. We want education to be cheap and not threatening to our cultural prejudices. Above all, the last thing we want to do is respect teachers. Of course there are and always have been strands in our tradition with the opposite tendency, but they tend to be associated in the popular mind with the despised Europeans. Over and beyond hatred for unions, the tendency to blame the supposedly wretched state of our schools on teachers arises from American anti-intellectualism.

One obvious way to improve our educational performance would to emulate the attitudes of peoples such as the Jews, Chinese, and Japanese, who love and treasure learning. If that prescription seems hopelessly impractical, I have to ask why it is any sillier than the suggestion that mediocre teachers are automatically going to be replaced by better ones if only we disbanded the unions.

26

nick s 10.18.10 at 6:43 pm

How much of this is tied up to the “Nice White Lady Breaks All The Rules And Saves The Day” cultural trope so frequently used to depict the American education system?

27

christian h. 10.18.10 at 6:47 pm

It’s perfectly clear what the union busters want to do. Their goal is to hire lots of young teachers, preferably without family, who they can pay badly and get to work very long hours – and then when those people burn out, or even insist on working only the hours they are paid for, they want the right to fire them. This will save loads of money they can then give away to their rich friends or spend on their pet wars or both.

Anyone who thinks they actually give a d***n about learning is deluding themselves.
If they did, they would at the very least take notice of the little research that is out there.

28

StevenAttewell 10.18.10 at 6:49 pm

Harry at 7 – distrust of administrators and corporate-backed reformers has been my experience in talking with AFT folks as well. That’s why I’ve always argued that the way to tell a real education reformer from an Econ-101-overapplier is to argue for giving the teachers’ unions control over hiring and firing and build performance standards into the union contract. If reformers care more about improving outcomes than they care about breaking unions, they’ll agree.

29

StevenAttewell 10.18.10 at 6:50 pm

Whoops. Screwed up the html tag there. Sorry.

30

Uncle Kvetch 10.18.10 at 7:06 pm

In France (and it’s my understanding that this is pretty typical in Europe), publicly schools are much less monolithic than in the US. There are publicly funded religious schools, schools for various levels of academic achievement, and so forth.

Not sure where you got that impression about France, but it’s quite the opposite, actually. The French education system is notoriously centralized and top-down. The curriculum — right down to the choice of textbooks — is essentially uniform throughout the country, in sharp contrast to the state- and district-level variation you find in the US. It’s true that there are publicly funded religious schools, but in exchange for that funding they teach the exact same curriculum as their public counterparts.

As for “schools for various levels of academic achievement,” that’s hardly unusual in the US. Here in NYC you have elite public high schools like Stuyvesant and Bronx Science with very tough admission requirements.

31

mcd 10.18.10 at 7:28 pm

To think the “education reform” movement is sincere, you have to think that rightwingers agonize over the plight of black children. And you have to think that middle class parents are agitating to improve education so that millions of poor children will now be able to compete with their own children for good jobs after school.

Anyone really think this?

32

Myles SG 10.18.10 at 7:47 pm

I actually with the Left (whatever it is) on this issue. I just don’t there is very much you can do, absent altering broad cultural and socioeconomic factors, to improve school achievement. The children of poor people are less likely to be academic because they have been brought up in a less academic environment, and the more chronic poverty there is the more pervasive the culture of academic failure.

Unions don’t really enter much into this picture. Stupid and backward union structures makes things worse, but not by very much, and changing stupid and backward union structures, even when they achieve their maximum results, is still tempering at the margins.

Basically, there isn’t much we can do. Blaming unions is a nice psychological escape, a sort of punching bag, but it’s not reality.

33

Myles SG 10.18.10 at 7:50 pm

By the way, the whole education-reform wave reminds one of nothing so much as the late-Victorian push by women’s rescue and so on organizations to reduce prostitution by housing them in shoddy hostels and making them work in badly paid textile jobs. It’s completely hopeless.

34

Peter Nunns 10.18.10 at 8:39 pm

Gene O’Grady@24:
A word of thanks to those who have focused on leadership rather than teachers as an area where positive change is possible. Looking back over my kids’ time in school (in Palo Alto, a district with putatively excellent schools) nothing made nearly as much difference as the quality of the school principals. This includes good ones instantly improving a whole school and bad ones wrecking the place.

As someone who attended a public high school much more recently than most people in this debate, I wholly agree with this comment. Here’s what I observed:

* Teachers don’t live off the fat of the land – they tend to lead pretty modest middle-class lives. If they stick around long enough to get tenure, they are generally doing it out of a love of teaching.

* The teachers who benefited from tenure weren’t generally the incompetents but those with some verve and a desire to teach potentially controversial topics. Two examples: My (notoriously vapid) junior-year honors English teacher was reassigned to teaching at a lower level after it was discovered that nobody was learning anything in her classes. The next year, my (excellent and passionate) senior year AP English teacher fought a battle with the administration over teaching The Laramie Project. I saw the same thing in other places: the administration was happy to manage incompetence, but it tried to crush creativity. I have no doubt that without tenure they would have fired him, and probably kept her, which would be a totally insane result.

* My school was well-funded and generally regarded as a good school. But in the half-decade since I graduated, it’s gone downhill quite rapidly. Why? Most signs point to a severely incompetent and at times actively malicious administration. The teachers were generally good – but as the older ones retired, they weren’t being replaced by people of a similar caliber. And career development for younger teachers wasn’t being managed effectively. This is a problem with management, not unions. But nobody ever thinks to criticize the bosses in America.

* By the time I graduated, I was spending roughly a week of class time sitting standardized tests. This was wasted time for me and most of my fellow students. I don’t think that middle-aged people, who haven’t been subjected to this full-court-press of Scantron sheets, can quite understand how utterly useless such tests are. They test reading and mathematics skills on an extremely minimal level – essentially, you could pass the test without knowing 90% of what a high school should teach you. Standardized test, by their nature, test what is easy to test. They couldn’t assess my writing ability, nor my chemistry, physics and calculus, so they didn’t. (More sophisticated instruments like the AP tests do so, but they’re quite labor-intensive to grade.) Had my teachers taught to that test, it would have undermined the quality of my education.

* Furthermore, there was no sign that anybody in the local school district was doing anything with the information being gathered. Testing mandates came from above – the state of California, NCLB – and were used to make funding decisions. But it wasn’t information that was useful to teachers or administrators, because it was only marginally relevant to what was being taught.

35

engels 10.18.10 at 8:57 pm

as if the entire point of public education is to produce good little workers to stamp out bits of GDP

But that is the point of public education. Ask any ex-CEO of an oil company!

36

billikin 10.18.10 at 9:01 pm

Matt Warren: “Charter schools may be a super duper awesome idea”

From what I hear, the stats don’t bear that out. Charter schools do not do a demonstrably better job than public schools, do they?

37

Omega Centauri 10.18.10 at 9:03 pm

I wonder how much of the motivation is political rather the being based on a desire for better outcomes, especially for lower class students. I can recall circa 1980 thinking teachers unions were simply fronts for communist revolution. I don’t ever recall, anyone arguing against this point of view, so probably the characterization of teachers unions, and to a lessor extent, any unions, as organizations intent on forcing leftwing viewpoints onto society are probably still pretty rampant, -especially on the right. So an indirect attack on them isn’t exactly unexpected.

To a significant demographic, school and curricula, is a means of waging culture ware, first, and a means of helping young people prepare for life second.

We do have issues, with bureaucracy and funding mandates however. My son, who did have some problems, was kept in special ed a lot longer than necessary -we suspect that the desire to collect state funding strongly biased their judgment.

38

Harry 10.18.10 at 9:05 pm

Sebastian

I agree with you about math. The problem there is that unless the teaching to the test is relentless, and reasonably good, over years , the gains fade quickly. You have to affect motivation which is hard. Reading is different, either because it really is different, or because its not clear how to teach to the test. The problem is that to learn reading you really have to read, a lot, and schools have a hard time motivating kids from low-income backgrounds to do that. But in principle I’m sort of with you. The crowding out worry is very serious though, especially with respect to reading in which the gains tend to be pretty limited.

Thanks Steven, I’ll try to fix the link from within…

39

Laurel 10.18.10 at 9:14 pm

There are very valid reasons to focus on teachers rather than the factors outlined in Rothstein’s article, namely that teachers are a controllable factor (in districts that have had their unions busted) and that mass unemployment and social ills are so multifactorial and poorly addressable that they are definitely higher hanging fruit.

Here’s the problem: K-12 teachers are something like 2 or 3% of the workforce. That’s a lot of people. I think it’s highly unlikely that, absent major structural changes in the incentives to become a teacher, you could replace a significant chunk of those teachers with smarter/more competent/more awesome people, especially in poorly run schools that serve high-needs populations. There’s not exactly a line out the door to work 80 hour weeks at a low status job for $50,000/year with a lot of people yelling at you. You can get different people, but they will either be new, uncertified teachers who leave the poorly run school as fast as they can (i.e. once they get their teaching certificates and have enough experience to get hired at a charter or better public school) or experienced teachers who were unable to get a better position. Even if we entirely ignore the problem of having very little idea in advance who will or will not be a good teacher, that’s not a particularly fabulous pool. Teacher quality is way, way more difficult to fix than most people think. Giving teachers ongoing feedback or time for collaboration or some other structural assistance might actually be much easier and more effective, because it doesn’t depend on magically uncovering a group of amazing teachers just waiting to work very hard for not that much money under difficult circumstances.

Still, the comparison to fad diets is a good one. Certainly there are many things about changing teacher work rules that might plausibly affect student success later in life, as there are many dietary changes that might affect health. We don’t know that test/scale numbers are a good marker for long-term success/health, and we don’t even know if work rules changes/particular fad diets reliably move the marker.

40

Leo Casey 10.18.10 at 9:20 pm

As a Crooked Timber reader who plays a leadership role in American teacher unions, I would be interested in some specificity to the general talk about “stupid and backward union structures.” That sort of broad brush characterization seems to me be partaking in the discursive “common sense” of the corporate attack on public sector unions in an unquestioning way, without ever interrogating it.

For example, it is simply not true that teacher unions are supporting broken teacher evaluation systems, or that we do not have a substantial body of efforts to create professional systems of teacher evaluation. What is true is that teacher unions have resisted efforts to reduce teacher evaluations to value-added metrics off standardized test scores of their students, and that we will advocate for the use of multiple measures for teacher evaluations, much as we insist upon their use for student assessments. A professional evaluation system will include measures of student achievement from performance assessments, in which students actually demonstrate the skill and knowledge they need to possess, such as the writing of a persuasive essay, as well as value-added metrics. It will include both supervisory and peer observations of actual classroom instruction, as well as a portfolio of artifacts of teaching, such as lesson plans, unit designs and curriculum. That doesn’t make for the 10 second soundbite of anti-teacher union staples, but it is the actual reality.

41

Myles SG 10.18.10 at 9:37 pm

As a Crooked Timber reader who plays a leadership role in American teacher unions, I would be interested in some specificity to the general talk about “stupid and backward union structures.” That sort of broad brush characterization seems to me be partaking in the discursive “common sense” of the corporate attack on public sector unions in an unquestioning way, without ever interrogating it.

Please address the rubber room, New York City. Again, I don’t actually think getting rid of rubber rooms makes that much difference, but the rubber rooms are there.

42

Myles SG 10.18.10 at 9:40 pm

I mean, the drain on resources aren’t actually as horrifying as the NY Times makes it out to be, but they drains on resources. It’s just the sheer ludicrousness of paying incompetent teachers full salary to do nothing that is so striking.

43

piglet 10.18.10 at 9:47 pm

Sebastian 21 says it is obvious that American teachers are worse than other countries’. It’s nice that somebody owns up to that statement. Do you also think that American doctors are bad? Scientists? Engineers? Lawyers? Plumbers? Mechanics? Farmers? Firefighters?

Are Americans just dumb in general or is there something particular that draws incompetents into the teaching profession? If that is so, then shouldn’t we identify what it is?

44

StevenAttewell 10.18.10 at 9:53 pm

Regarding teaching reading, I actually don’t think the tests are teaching people to read. They’re teaching them to answer very specific kinds of multiple choice questions about what’s going on in short passages or sentences.

The problem is that this tends to push reading to the shallowest level. I’ve seen this in history classes I’ve TAed for – freshmen come in being very practiced at a particular kind of reading comprehension that focuses on the basic “what happened?”, but completely unprepared to do deeper thinking about the author’s intentions, audience, the strengths and weaknesses of the argument, the use or misuse of evidence, and the context of the author and audience.

45

piglet 10.18.10 at 9:55 pm

“teaching to even a moderately good math test is teaching math”

Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests. Maths proficiency has to be tested by doing maths. Which is probably true for any subject.

46

ScentOfViolets 10.18.10 at 10:34 pm

To start the conversation do we need ‘why’? First we need to agree that it is true, if it is. It seems to me that it is true, but if can’t agree on that, we probably need to start there.

The same old same old I see. Sebastian, you should have learned in school that if you want to say the teachers union here in the U.S. is worse than elsewhere, you actually have to provide proof, evidence, lines of reasoning, that sort of thing. Trolling about with the “I’m not sure” line trying to sucker people into trying to convince you when quite obviously from many previous conversations that nothing will convince is dishonest. Yeah, I know that you want the default to be going in the other direction, but that’s just not going to happen.

So why are you “not sure” to put it into your own words?

47

ScentOfViolets 10.18.10 at 10:37 pm

But there are aspects of a child’s education which we don’t care to seriously discuss because magic worlds like “socialism!” are uttered. Habits, diet, and home-life fall under that fear. The last of those three is among the most important, too.

Just about every teacher I know has this same impression. We can talk about any kind of “reform” we want to and “and all the options are open”, just so long as the teacher’s union is weakened and the neglect of the parents and incompetent craven administrators are not mentioned.

48

ScentOfViolets 10.18.10 at 10:41 pm

One obvious way to improve our educational performance would to emulate the attitudes of peoples such as the Jews, Chinese, and Japanese, who love and treasure learning.

Without necessarily buying into those particular prejudices, you’re exactly right. And that’s why “reform” is so hard.

And the people who tell me how much they “respect” education, learning, and the professionals who do the teaching? I’ll believe them when they put their money where their mouth is. Not before.

49

ScentOfViolets 10.18.10 at 10:43 pm

But in the half-decade since I graduated, it’s gone downhill quite rapidly. Why? Most signs point to a severely incompetent and at times actively malicious administration.

From what I’ve seen and heard, I’d say about 80% of this is due to caving in to hostile parents and school boards made up of hostile parents.

50

ScentOfViolets 10.18.10 at 10:45 pm

I agree with you about math. The problem there is that unless the teaching to the test is relentless, and reasonably good, over years , the gains fade quickly.

Speaking as a math teacher, “teaching to the test” is either very hard, or else is vacuously the default condition. The problem here is that people don’t have the right model for math – they need to be thinking about music as opposed to history.

51

ScentOfViolets 10.18.10 at 10:48 pm

Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests. Maths proficiency has to be tested by doing maths. Which is probably true for any subject.

Yeppers. Got it in one, especially that last sentence.

52

Peter Nunns 10.18.10 at 11:30 pm

As someone who was taking large whumps of standardized tests relatively recently, there is no doubt in my mind that they are almost completely worthless as evaluation instruments. Anything worth learning in high school is too complex to be reduced to five choices on a Scantron sheet.

I guess this is my major issue with this. The proponents of “union busting plus standardized testing” spend a lot of time talking about things like “incentives” – i.e. giving teachers the incentives to teach properly. What they never stop to consider is that they are proposing to set up a system with very strong incentives to not teach anything.

53

jeer9 10.19.10 at 12:05 am

Principals can perfectly well get rid of teachers in their probationary years, but because most know nothing about instruction, and the rest don’t see improving instruction or managing human capital as any part of their job, they often don’t even bother observing. Get rid of the teachers unions, and teachers are subject to the arbitrary power of people who know nothing and have less than no interest in what really matters in the school.

Thank you, Harry.

54

Sebastian 10.19.10 at 12:19 am

Math is tested by doing math. And it is quite testable. Universities do it all the time using scan-tron tests for pretty much all the classes from pre-algebra straight through differential equations, which is probably further into math than at least half the readers here have gotten.

The idea that math isn’t testable by multiple choice tests is frankly silly. How in the world do we get so many people here agreeing that it isn’t? What exactly is it about math, especially at the lower levels that we are talking about, that makes it non-amenable to testing?

At the levels of addition, subtraction, multiplication, division, simple algebra, geometry, trigonometry, and calculus at the high school level–which is to say nearly all of the math would might be talking about–you can make tests with one correct and 4 incorrect answers–at least 3 of them stemming from common misunderstandings or confusions.

Teaching to that test IS teaching math. Teaching to the test without teaching math is really difficult. Quite a few of the people here seem happy to attribute all manner of assumptions about the bad faith of anti-teacher’s union people, but how can you be arguing in good faith by suggesting that math isn’t readily testable?

If you are going to claim that it isn’t testable you’ll need to get a lot more specific about why it isn’t testable.

“Sebastian, you should have learned in school that if you want to say the teachers union here in the U.S. is worse than elsewhere, you actually have to provide proof, evidence, lines of reasoning, that sort of thing. Trolling about with the “I’m not sure” line trying to sucker people into trying to convince you when quite obviously from many previous conversations that nothing will convince is dishonest.”

SoV, hmmm ok. You always play the burden of proof game. And interestingly it is never your burden. Funny, that.

I responded to an assertion in comment 1. The assertion implied that the teacher’s unions were the same in all the modern countries. It is my impression that other teacher’s unions (outside the US) are not as resistant to actually measuring children’s learning and to evaluating teacher’s with it in mind. I’m open to being corrected on that point, if it turns out that they are equally resistant. But I’ll freely admit that I doubt you could possibly come up with that.

As for the proposition that US teachers are resistant to that, take it up with Harry if you don’t believe. Even he, and by that I mean something like even a person pretty darn open to union control of things, understands that it is true (see comment #7). That fact isn’t seriously in dispute. Which of course doesn’t mean that you won’t dispute it. But that would put the burden of proof firmly in your court, which I know you hate.

55

piglet 10.19.10 at 12:31 am

Sebastian, again, if by “it is my impression …” you mean to say “I have no clue whatsoever but I am guessing …”, please don’t bother. If you have specific knowledge about how teachers unions in the US differ from those in other countries, let us know.

56

Stuart 10.19.10 at 12:58 am

Sebastian, you seem to be relying on #7 to prove your point about US Teachers Unions being recalcitrant. Actually reading that comment seems to do nothing of the sort, i.e. “I’ve said this before, but in my own interactions with (apparently recalcitrant) teachers union leaders, they understand perfectly well that the evaluation system is stupid, and that incentives need to be put in place. “

Tom me, this seems to indicate that Harry believes that the idea that the union leaders are recalcitrant does not match reality, at least in his experience. Did you maybe score highly in a multiple choice English exam and are over reliant on it as evidence of your level of reading comprehension?

57

Walt 10.19.10 at 1:22 am

Sebastian, you seriously think that teaching math tests using Scantron is a good idea? You would be alone in that belief. Universities do it because it’s cost-effective, not because it’s good pedagogy. You’ll say anything to stick it to the teachers, won’t you?

58

Cryptic ned 10.19.10 at 1:42 am

Walt’s comment 57 looks kind of dumb when you go back and see that Sebastian actually tried to back up his statement with a lot of explanation. Try harder. Not just anti-anti-union-backlash backlash.

59

ice9 10.19.10 at 2:03 am

Myles is a fine example of the reason why it’s all about teacher’s unions: because teachers unions are a defined evil. The charge has been repeated the requisite number of times to become fact among people who deal with such facts. It’s a big, fat, ‘all of the above’ irony that the people striving in this particular battle of the culture war are supposed to be educators is not relevant. Hence the famous ‘rubber room’ in New York, that single apocryphal place full of apocryphal terrible teachers paid full salary to sit around safely not harming kids–can stand for the entire field of education, can carry the entire argument over whether teachers unions are evil.

My union is occasionally exasperating, and occasionally a bit slimy, but a necessary fact of life in the litigious and ugly world of American public education where culture warriors routinely urge parents to blame teacher for their childrens’ shortcomings (and, occasionally, blame them for their successes in, say, grasping evolution). The best teachers are the most vulnerable, since it is the best teachers who challenge kids to change and grow. My union will provide me an unlimited supply of pit-bull lawyers to defend my job and my livelihood in those circumstances. I can tolerate the minor negatives of teaching with a few hacks and ninnies, simply because at the moment it is far too risky to give it my all when any administrator or parent can be poltroon enough to sandbag me then walk away humming. In fact, the threat of a real firing or lawsuit is slight, about the same as the tornado or fire I’m also insured against. It’s the peace of mind that I’m tenured and backed by a union that allows me to opt for the high-challenge, low bullshit approach that I know works because I’ve been at this for a long time and I’ve seen it work (and yes, part of my evaluation of success is test scores, because there are a few good ones.)

By the way, I’m sure the research shows that it’s poverty or race or good teachers that account for success. But what accounts for failure? I have a suggestion. How can society reasonably expect schools to generate smart, thoughtful kids when the society is proudly ignorant, nasty, greedy, violent, and hypocritical? We worry that kids watch too much TV or play too many video games; the influence of those must be tame compared to listening to Sharron Angle or Christine O’Donnell or Michele Bachmann or Sarah Palin, to name just four especially stellar role models. Our society’s prevailing level of factual accuracy, persuasive argument, clear language, intellectual consistency, historical accuracy, and essential honor are abysmal; it’s comical to imagine that schools operating in that environment could make progress.

ice9

60

Leo Casey 10.19.10 at 2:03 am

Myles SG at 41:

1. The ‘rubber rooms’ were not a creation of collective bargaining or the teachers union, but of the Joel Klein administration at the NYC DoE which established them after 40+ years of collective bargaining and union representation.

2. The ‘rubber rooms’ were finally ended last year, after the union threatened to bring on a recently retired and highly respected prosecutor to head a team of lawyers to work through all of the cases of teachers held at these sites. Joel Klein had wanted to keep the ‘rubber rooms’ open because he believed that they gave him a way to attack tenure, due process and the union. To that end, Klein had brought Steve Brill, a personal friend of Klein’s then Deputy Chancellor Chris Cerf, into the “rubber rooms” to write his “expose,” while having a teacher who was doing a film documentary arrested for going in. When the Mayor realized the potential embarrassment of having a credible third party go through the cases, he overrode Klein and agreed to close the “rubber rooms” and adjudicate the cases of the teachers.

3. The “rubber rooms” are presented in “Waiting for Superman” as a reality a year after the agreement to close them and adjudicate the cases for the same reason that Joel Klein wanted to keep them open — to use as a bludgeon in a campaign to eliminate due process for tenured teachers.

61

ScentOfViolets 10.19.10 at 2:21 am

“Sebastian, you should have learned in school that if you want to say the teachers union here in the U.S. is worse than elsewhere, you actually have to provide proof, evidence, lines of reasoning, that sort of thing. Trolling about with the “I’m not sure” line trying to sucker people into trying to convince you when quite obviously from many previous conversations that nothing will convince is dishonest.”

SoV, hmmm ok. You always play the burden of proof game. And interestingly it is never your burden. Funny, that.

I think Sebastian has just caused my head to momentarily explode. You know, Sebastian, maybe this is because I was taught good reasoning skills in school. If you make some sort of assertion, the burden of proof is upon you to prove it. That’s just good scientific practice. So I try to be very careful about what I say and how I say it, and I’m usually pretty forthright about when my opinion is just my opinion, gut-checked with anecdotal data.

You didn’t learn about any of this in school? Seriously? Or is this some sort of lawyer thing, some sort of rhetorical game you’re playing that the rest of us aren’t aware of? I think most of us here are interested in the science and the facts; rather less of us in attempting to get others to accept a certain narrative.

Now, do you have any actual evidence for your claim? If you do, spit it out. Stop being coy and stop wasting our time.

62

ScentOfViolets 10.19.10 at 2:32 am

The idea that math isn’t testable by multiple choice tests is frankly silly. How in the world do we get so many people here agreeing that it isn’t? What exactly is it about math, especially at the lower levels that we are talking about, that makes it non-amenable to testing?

Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.

But the other, bigger part, is that the tests are for me a diagnostic tool. If I can’t see their work as well as their answers, it’s much harder for me to correct any systemic problems. And if their answer is wrong because of one stupid sign change that didn’t carry over to the next line, I don’t tend to take off very much for that if we’re covering, say integral substitution.

I’ve got to ask – you seem to be completely clueless as to how teaching is done. In fact, your stuff reads like so much right-wing boiler-plate. What personal experience do you have with teaching, and where do you get your information about what’s supposedly wrong on this particular subject?

63

rosmar 10.19.10 at 3:00 am

There is another huge problem with standardized tests (especially high stakes ones)–the people who take them can be subject to things like stereotype threat or even simple test anxiety that can seriously (and, in the case of stereotype threat, systematically) distort the meaning of the test results.

64

Sebastian 10.19.10 at 3:24 am

SoV: “Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.”

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

“Tom me, this seems to indicate that Harry believes that the idea that the union leaders are recalcitrant does not match reality, at least in his experience. Did you maybe score highly in a multiple choice English exam and are over reliant on it as evidence of your level of reading comprehension?”

Stuart, Harry in 7 suggests that the union leaders are recalcitrant, but he softens it by suggesting that they may have a good reason for being so. He also writes, ” they understand perfectly well that the evaluation system is stupid, and that incentives need to be put in place.” This strongly suggests that he thinks the current evaluation systems are stupid and that incentive systems should be put into to place.

Now you can read that however you like I suppose, but if I were you I’d be rather more careful about throwing around suggestions of poor reading comprehension.

So now that we have established that testing, at least for math, can be a good thing, I guess we’re getting somewhere.

65

LFC 10.19.10 at 3:40 am

I have seen ‘Waiting for Superman’. However, I am not a student of education policy in the US (or anywhere else), I have not read the Klein/Rhee manifesto, etc. I have also not read this whole thread, so I apologize for any repetition.

Several of the movie’s messages strike me as simplistic at best, and critics here and elsewhere have been noting them. One glaring omission in ‘WforS’ is that there is no mention (or virtually no mention) of the history of racial and class segregation and how this has contributed to “failing” schools; instead there is simply the bald and simplistic statement/suggestion that ‘bad’ schools cause neighborhood deterioration, not the other way around.

The US Supreme Court’s 1974 decision in Milliken v. Bradley prohibited cross-district busing and ensured that schools in northern cities such as Detroit (the city involved in the Milliken case) would remain or become largely segregated by race. By contrast, in at least some other cities where the local authorities drew the lines in such a way as to include suburbs and central city schools in the same district, the educational and social outcome tended to be different, and better. At least this was apparently the case in Raleigh, NC: see Gerald Grant, Hope and Despair in the American City. How a movie purporting to be about education in the US can be silent about these issues is beyond me.

Re testing etc: I’m thankful that I went through school long before the testing mania reached its current pitch. There were standardized tests, of course, and the kids in my suburban (and virtually all white) public high school, many of them bound for ‘elite’ colleges and universities, took them seriously; but I would be very hard pressed to recall a single occasion in which a teacher in a classroom referred to a standardized test. They did not loom over the curriculum the way they apparently do today in many schools (and not only ‘disadvantaged’ ones).

66

ScentOfViolets 10.19.10 at 3:44 am

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

Sigh. Actually, you said something about trigonometry and calculus:

At the levels of addition, subtraction, multiplication, division, simple algebra, geometry, trigonometry, and calculus at the high school level—which is to say nearly all of the math would might be talking about—you can make tests with one correct and 4 incorrect answers—at least 3 of them stemming from common misunderstandings or confusions.

Please try to remember what you’ve already said. And in that spirit, I guess your silence on your evidence for the significant differences between teachers unions here and abroad mean that you have no evidence. So please cease all future concern trollery with your “wondering” and “not convince”. I will from here on out will assume you concede that there aren’t any significant differences until you have given some evidence otherwise.

67

ScentOfViolets 10.19.10 at 3:46 am

So now that we have established that testing, at least for math, can be a good thing, I guess we’re getting somewhere.

And here we have more – and more blatant – trolling. I’m wondering if Sebastian has the forthrightness to name the names of those who did not think that math testing “can be a good thing”.

Sebastian, if you want to be disruptive . . . don’t you have your own blog you can play on?

68

Peter Nunns 10.19.10 at 3:50 am

So now that we have established that testing, at least for math, can be a good thing, I guess we’re getting somewhere.

No, you’ve established that math testing is in theory possible. No dispute there.

But I would contend that math testing as it is currently being practiced under high school standardized testing regimes is no good. Again, I was in high school when standardized testing came into vogue – NCLB started while I was there, and California implemented its high school exit exam at around the same time. I was one of the test subjects for these programs. And they were simply no good!

As I said above, the actually-existing tests were inflexible and irrelevant to the curriculum, meaning that they didn’t provide my school district or teachers with any actionable information. The CA high school exit exams produced a nice statistic that meant almost nothing.

However, I had another experience with large-scale testing that was quite positive: The national Advanced Placement tests. Your average AP test had both a quantitative section (scantron bubbles) that tested basic knowledge of concepts and ability to perform relatively mechanical calculations/analyses, and a qualitative section with written responses. And because they were generally constructed with curriculum development in mind, teaching directly to the test (as some of my AP teachers did) resulted in a great deal of practical learning. However, the importance of the written sections – where partial credit was possible – made grading them a labor-intensive process.

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

69

Walt 10.19.10 at 3:51 am

Fuck you, Ned. Is there any evidence that Scantron is a good evaluation method, rather than a cheap evaluation method? I taught math for several years, and know many people who have taught it for many more, and none of them have ever used Scantron except out of expediency (like they’re teaching a class of 300 with not enough TAs). He offers no argument that multiple-choice testing is a good idea, just the assertion, and then various patronizing comments towards everyone who disagrees, such as dismissing all real-world experience on the subject as “silly”.

Every thread on this subject with Sebastian is the same: no one cares about students unless they agree with Sebastian that the teachers union needs to be smashed. He’s monomaniacal on the subject. Honestly, we’re being politer than he deserves.

70

Sebastian 10.19.10 at 4:02 am

“Every thread on this subject with Sebastian is the same: no one cares about students unless they agree with Sebastian that the teachers union needs to be smashed. He’s monomaniacal on the subject. Honestly, we’re being politer than he deserves.”

Care to point to anywhere on this thread where I suggest anything like this? Or on a recent thread here?

I didn’t think so.

SoV “Please try to remember what you’ve already said. ”

Please try to remember what we’re talking about. Try to keep the context in mind, thanks. It isn’t all calculus. In fact if we really were worrying about whether or not all (or even significantly most of) the kids were deeply understanding calculus and whether or not that deep understanding of calculus ought to be measured via scantron, this would be a much better world.

Of course we aren’t really talking about it. And so far as I know, none of the state level basic requirements testing requires calculus.

We are talking about basic math skills. People who can’t do simple algebra because our schools are failing them. THAT is what we are testing for. And THAT can quite well be tested for via pretty basic testing methods. It isn’t a great mystery how to test for THAT. I can see why you might want to pretend that the (somewhat dubious) idea that calculus is tougher to test for in a standardized way (though the AP tests belie that) applies to the vast majority of kids that we are talking about. But it doesn’t. At all.

Basic math can be tested for basic competence.

Pretending that it can’t suggests studied and willful blindness. Trying to turn me into a troll for suggesting it, is an act of bad faith in argumentation.

71

ScentOfViolets 10.19.10 at 4:06 am

However, the importance of the written sections – where partial credit was possible – made grading them a labor-intensive process.

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

BINGO!

72

ScentOfViolets 10.19.10 at 4:09 am

However, the importance of the written sections – where partial credit was possible – made grading them a labor-intensive process.

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

BINGO! Cue the standard plaint that wanting to test and evaluate is not the problem; it’s wanting to do it on the cheap and mean that’s the problem. But that goes back to people who “respect” education and educators putting their money where their mouth is.

73

ScentOfViolets 10.19.10 at 4:14 am

SoV “Please try to remember what you’ve already said. ”

Please try to remember what we’re talking about. Try to keep the context in mind, thanks. It isn’t all calculus. In fact if we really were worrying about whether or not all (or even significantly most of) the kids were deeply understanding calculus and whether or not that deep understanding of calculus ought to be measured via scantron, this would be a much better world.

Bite me, Sebastian. You’re not being even remotely honest at this point. I quoted your own words back at you, you know, the part where you did say that yes, scantrons for stuff like geometry, trigonometry and calculus were just fine and dandy.

Why don’t you just say that you were wrong, instead of going through all this song and dance with the weaseling and bafflegab? It certainly wouldn’t hurt your reputation after the hits it’s already taken.

74

David 10.19.10 at 4:15 am

My friend Carl Chew quit a career as a successful artist and went to teach in the Seattle school system for a dozen years. He recently retired/resigned. Here is his take on the movie:

http://www.ctchew.com/pages/2010/superman.html

Somewhat more nuanced than the usual blame the unions and teachers crap.

75

LFC 10.19.10 at 4:15 am

To amplify my point: It is irresponsible for a movie such as Waiting for Superman, which after all will be seen by a great many people who are not professional students and observers of education policy, to suggest via omission that the trouble with US education is entirely a result of what goes on in classrooms and who controls that. That is why a single-minded focus on whether teachers unions are good or bad, and on what specific, immediate interventions will or won’t work, risks obscuring another point: a movie like this could itself educate its audience, and to produce a portrait of American education that is completely devoid of a huge chunk of relevant historical background is a gross disservice to the ordinary person who walks in to see this movie knowing nothing about education debates beyond whatever he or she may have picked up from the media. A context-less, ahistorical, often tendentious “documentary,” which wastes a good half-hour at the end on those heartrending but basically pointless lottery scenes, is probably worse than no movie at all. The more I think about it, the more I am convinced that Davis Guggenheim should be ashamed of this film. I would urge people to think twice and thrice before paying to see it.

76

billikin 10.19.10 at 4:24 am

piglet: “Are Americans just dumb in general or is there something particular that draws incompetents into the teaching profession?”

American teachers are underpaid.

77

billikin 10.19.10 at 4:53 am

Jim Harrison: “One obvious way to improve our educational performance would to emulate the attitudes of peoples such as the Jews, Chinese, and Japanese, who love and treasure learning.”

They love and cherish learning and teachers. :)

78

Myles SG 10.19.10 at 5:36 am

Leo Casey at #59:

I thought the whole reason the rubber rooms exist was because it was easier in this manner to get rid of bad teachers instead of keeping them in the classroom and waiting a very long time for the adjudication process, which seems very complicated, to play out? I mean, yes, due process, but there is no due process right for someone to be kept teaching in a classroom where he’s doing harm.

Again, I just don’t think it matters as much as people think it does. It doesn’t seem to me that the urban school system in the US are any actively worse than schools in French banlieues, or for British council estates.

Speaking as someone who before going to prep school went through the Canadian system, which is structurally similar, my impression is that most teachers at the high-school level are competent only to the extent that the very degraded system defines competency. You can get straight A’s in English without being able to analyze John Donne with much in the way of perceptiveness and erudition, and hell, some English teachers can’t either. The vast majority of people graduate high school functionally illiterate.

79

trotsky 10.19.10 at 5:50 am

I’m not against teachers’ or any other workers’ unions in general.

But the ACLU is against the actions of this one in particular — http://laist.com/2010/02/24/lausd_layoffs_lead_to_lawsuit_from.php.

And I think they’re right.

80

christian h. 10.19.10 at 5:58 am

Sebastian in 54. claims that universities are using multiple choice testing in math classes, up to and including “differential equations”. This is simply false. I have taught math at four US universities – one private, three top tier public – and know people teaching at dozens more. As somebody wrote above the only cases of multiple choice testing I have ever even heard of are out of desperation and lack of grading resources. No mathematics professor in their right mind would use them by choice, period.

Multiple choice testing may be adequate for the job of eliminating students from the competition for medical school places or internships at top investment banks. It is completely useless to evaluate actual learning success. This is of course true even for basic algebra if that is fresh material the students are currently learning.

81

Sebastian 10.19.10 at 6:17 am

Well, SoV, I’m glad to know what passes for civil from you. You can’t stay on point, you can’t follow an argument, and you have to be repeatedly insulting.

The point remains. Math can be tested.

Especially at the competency testing levels that are the topic of the conversation.

82

Geoff Swenson 10.19.10 at 6:40 am

My sister was a teacher for several years in the Chicago area, but eventually grew weary of the bureaucracy interfering with good teaching and low pay. She now works as a salesperson of industrial products at more than twice the pay rate.

The low levels of teacher’s pay has to be a major factor in the quality of our schools. The unions are sometimes a problem, but I’m sure that if a decent pay raise was offered along with some carefully negotiated improvements to teacher evaluations, it would be accepted by the unions.

While we are at it, can we also have some improvements in evaluating the administrators, and in finding ways to capture the techniques that work and cloning them in other schools?

83

Zora 10.19.10 at 7:21 am

I would feel friendlier towards unions if I didn’t think that their mission was to protect their members’ jobs, no matter what. At least that’s the lesson I took away from the remarks of the head of the Hawai’i teachers’ union, a number of years ago; he boasted that no teachers had been fired while he was in office.

I can’t believe that none of the teachers deserved firing. People are people; they mess up. They disappoint us. Teachers are people.

84

Peter Nunns 10.19.10 at 7:37 am

Sebastian @69:
And so far as I know, none of the state level basic requirements testing requires calculus.

Yes, but high school should teach calculus to as many people as possible. It should be a core part of the curriculum. And that’s exactly where I have a problem with standardized tests: because they misdirect class time and teaching time towards a nice basic level of pabulum. That’s a serious problem.

Peter Nunns @52:
The proponents of “union busting plus standardized testing” spend a lot of time talking about things like “incentives” – i.e. giving teachers the incentives to teach properly. What they never stop to consider is that they are proposing to set up a system with very strong incentives to not teach anything.

I guess my problem with actually-existing standardized testing is that it’s basically an educational straitjacket. In extremely poorly-performing schools, it might be a useful mechanism for raising standards or fossicking out areas of weakness. But because it’s also likely to lower the performance of successful schools by encouraging them to teach to a dumbed-down test. Again, I was in high school while this was being tested and it was not good.

Why is it that we’re decades into the post-Fordist, flexible-production world and the best idea we can come up with for improving schools is to implement a testing regime that amounts to mass-production of intellectually sterile compliance?

85

praisegod barebones 10.19.10 at 7:53 am

Sebastian

You appear to be working under the impression that if someone says something uncivil to you, you have won – and will be adjudged by bystanders to have won – the argument.

I don’t know whether that’s a rule that works well in your cultural and professional milieu. Maybe it does.

In mine, though, there’s an important exception to that rule. You don’t win the argument if you respond to reasoned argument by deliberately attempting to provoke incivility.

At 63 you say

SoV: “Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.”

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

It’s very difficult to read that as anything other than a playground level taunt, designed to provoke the incivility it received in reply.

Remind me what the evidence that scantron tests were effective in teaching maths. Because like Scent of Violets, the only thing I saw in support of that claim was that they are used in universities. And you’ve had two people with actual expereince tell you that, and why, this point doesn’t support the claim you’re making.

(Incidentally, I’d say that when there’s someone in a conversation who is trying to win the argument by provoking people into being uncivil, there’s a case that the conversation will be more productive with more interventionist moderating. But I realise that that’s a very time-consuming option.)

86

Z 10.19.10 at 7:59 am

It is my impression that other teacher’s unions (outside the US) are not as resistant to actually measuring children’s learning and to evaluating teacher’s with it in mind.

Sebastian is actually right about (at least) one of the country he mentions: France. French teachers unions are not as resistant as American unions to actually measuring children’s learning and to evaluating teacher’s with it in mind; they are much much more resistant. In fact, the idea that tests scores could be in any way related to teachers evaluation would probably immediately put 99% of French schools on strike and no government would dare dreaming about proposing something like NCLB. In fact, I am not even sure correlating teacher’s evaluations to students test scores would be legal (even in theory) under French law.

So Sebastian was right: unions in France are very different than unions in the US. But the direction of the difference is not what he thinks it is. This is truly a question where the American left (someone like Obama) would be at the right-most end of the French political spectrum and the American right is so far to the right that their propositions are simply unheard of.

I say this as French citizen and the son, grandson and husband of members of teacher’s union.

87

praisegod barebones 10.19.10 at 8:01 am

I’m reposting this, because a paragraph break messed up the formatting in such a way as to obscure the main point I wanted to make. If there’s someone moderating this thread, they might want to suppress one version or other.

(Just so this gets caught in the spam filter,and thus doesn’t mess up thread numbering, I’ll insert a gratuitous reference to So Cialis M here)

Sebastian

You appear to be working under the impression that if someone says something uncivil to you, you have won – and will be adjudged by bystanders to have won – the argument.

I don’t know whether that’s a rule that works well in your cultural and professional milieu. Maybe it does.

In mine, though, there’s an important exception to that rule. You don’t win the argument if you respond to reasoned argument by deliberately attempting to provoke incivility.

At 63 you say

SoV: “Speaking as a real math guy teaching math, it is silly for all but the most trivial exercises. We only use it for the most very basic algebra (myself, I don’t think the university should even have accepted these kids, given their demonstrated level of mathematical ability.) Partly this is because there are too many operations to perform to make a bubbled scantron a good format – as I tell my class, getting 80% of a five-point problem right means you got it wrong.”

Well, since the problem areas in elementary school are all of the pre-algebra and very basic algebra type, I’m thrilled that you agree with me that for the type of schooling we’re talking about scantron tests are appropriate. Perhaps you would like to convince piglet and walt. I leave them to you.

It’s very difficult to read your contribution there as anything other than a playground level taunt, designed to provoke the incivility it received in reply.

Remind me what the evidence that scantron tests were effective in teaching maths. Because like Scent of Violets, the only thing I saw in support of that claim was that they are used in universities. And you’ve had two people with actual expereince tell you that, and why, this point doesn’t support the claim you’re making.

(Incidentally, I’d say that when there’s someone in a conversation who is trying to win the argument by provoking people into being uncivil, there’s a case that the conversation will be more productive with more interventionist moderating. But I realise that that’s a very time-consuming option.)

88

X 10.19.10 at 9:27 am

Regarding the value of multiple choice exams:

* Apart from entry exams, an exam is of value only insofar as it provides teachers with usable information about what the student doesn’t know. Making the exam multiple choice needlessly deprives the teacher of basic information on why they made the mistakes they did – especially in mathematics, where you can get all your reasoning right and be laid low by a simple arithmetic mistake.

* I did both British A-levels and US SATs (and scored well on both, if that’s relevant.) The SATs struck me as laughably easy in comparison to the A-levels – or, let’s be honest, even the GCSEs – and one big reason for that was simply that, unlike A-levels or GCSEs, they were multiple choice. It’s a lot easier to get the right answer when it’s sitting in front of you jogging your memory.

89

Tom M 10.19.10 at 11:23 am

There is no mention here of actual test scores despite the theme of WfS. The NAEP is a standardized test used throughout the US and perhaps looking at the actual scores might shed some light on what teachers, as a class, have accomplished. Bob Somerby has done a lot of writing on this movie and I think his background of teaching in the Baltimore city school district for 11 years might lend some credibility to his analysis.
Per Bob, who wrote several weeks’ worth of posts about the movie, one of his 9 critical elements is:
Stop lying about our test scores: If we claim to care about public schools, could we possibly start to tell the truth about our most basic data? In the last 20 years, minority students have recorded increased scores on both sections of the NAEP, around 16 points, or almost 2 grade levels. Does that mean teachers, especially those in poor districts, have done a good job with those kids?
You might want to spend some time at The Daily Howler and see what he has to say.

90

ice9 10.19.10 at 11:38 am

Regarding use of scantron in math in public schools: I teach in a large public high school in the Midwest. Our math department has tremendous success: high admissions, terrific test scores, state and national championships in various competitions, etc.

I don’t teach math, but I am a keen observer of the department because, like our social studies department, they are in constant conflict with the administration of the school. Virtually all of these conflicts are parent- and student-initiated and arise from student dissatisfaction with grades relative to expectation, ambition, or ‘work’. Spoiled and lazy students demanding more than they have earned, in other words. Invariably the parents, usually backed by administrators, suggest or demand more ‘objective’ measurements (such as scantron read multiple choice tests) and more ‘common assessments’ as part of these conflicts.

God bless ’em, our math department resists, only using the scantrons for the most basic types of assessments and sticking to good old paper-and-pencil show-your-work exams for when it counts.

We have a lot of kids who feel entitled and who have uncritical parent advocates (and the brass to bring them in). Enough–a critical mass of complainers– see the standardized, multiple choice assessment as their friend. THese people take this view without a hint of irony. That is, they demand better teaching, better learning, and higher grades from the teachers, and demand that scantron-style MC tests be used to achieve it.

They’re right, given their givens; such tests are the friend of the weak and entitled student. That’s because the tests make it easier to teach and easier to succeed by reducing the standard of demonstration. There’s a conscious movement, primarily among the same people who hate unions reflexively, to mechanize and Nickleby the teaching process wherever possible because it yields better grades to this particular stratum of society and of student. They also rejoice at it because it drains the power of teachers to teach in a way that holds students accountable for real learning, and, most important, it makes each class more interchangeable, reducing barriers to the privileged when they want to hand-pick their kid’s teacher. Administrators like the uniformity, too.

I wish the standardized test mania were only system- or state-wide.

ice9

91

Alex 10.19.10 at 1:08 pm

Again, we’re expending enormous amounts of heat and light on a proposal that is basically equivalent to trying to form an orchestra by recruiting people on the street, paying bonuses to some of them, sacking the ones who make the most bum notes, and replacing them with more random recruits. It’s going to be a long time before you stumble on a competent instrumentalist, and a hell of a long time before you get them all playing in time, let alone in tune.

Why isn’t there enormously more interest in improving observation, and finding out what it is that the “good teachers” do that the others don’t? That would be a gigantic advance however education was organised, funded, or delivered. If we don’t know that, then you’re at the level of the old story about Billy Wilder making one of his actors do take after take after take and yelling “CUT!” every time. Eventually the player turns to Wilder and says “Well – what is it you want?” Wilder says “Just…be better.”

Just knowing that some of the teachers are doing better than others isn’t very useful unless you also know why. If you know why, you can teach the other teachers. You can base the teacher training curriculum on why so they all start off doing it.

Even if it turns out that you can’t define teaching that well, that it’s some sort of inherent property of personality, well, that’s useful information. You might be able to find out who has it in advance rather than waiting for them to fail.

Also, I suggest the following operational definition of excessive testing; if you’re doing tests to prepare for the tests, that’s too much. When I did A levels we had mock exams of the mock exams.

92

SamChevre 10.19.10 at 1:54 pm

It seems that three questions are getting mixed up:

Can a multiple choice test accurately measure math proficiency?
Do the tests currently used for that purpose accurately measure math proficiency?
Can a multiple-choice test provide all the information a teacher needs?

I’d say that the answer to the second aand third questions is agreed, by everyone, to be NO. The first is the one that seems to be in dispute. I’d say yes. The actuarial prelims and the math GRE are both pretty good measures of subject competency, and are multiple-choice tests. The fact that the SOL’s are really lousy measures of math proficiency is a problem with the SOL’s, not standardized multiple-choice tests.

And re: the French educational system: if you are saying that a CFG curriculum, a CAP curriculum and a Bac curriculum are the same, you’re using a definition of curriculum with which I’m unfamiliar.

93

ScentOfViolets 10.19.10 at 2:00 pm

Anyone else notice how disrespectful Sebastian is towards teachers? He says something silly like multiple choice and scantrons can be used to test geometry, trigonometry and calculus, several people who are actually math teachers tell him that no, that’s simply not true, and here’s why . . . and we get a heaping helping of disrespect thrown in our faces.

Note that Sebastian doesn’t teach math at any level (he claims to be some sort of lawyer, I believe), but he’s so sure that we’re all wrong, that he’s right, and that they to get this across is to be snide and nasty. I’d say we’re seeing – live – a bit of the problem teachers face in their day to day work.

94

billikin 10.19.10 at 2:00 pm

Zora: “I would feel friendlier towards unions if I didn’t think that their mission was to protect their members’ jobs, no matter what. At least that’s the lesson I took away from the remarks of the head of the Hawai’i teachers’ union, a number of years ago; he boasted that no teachers had been fired while he was in office.

“I can’t believe that none of the teachers deserved firing. People are people; they mess up. They disappoint us. Teachers are people.”

I agree with you that it is highly unlikely that some teachers did a bad job, and I might have fired them if it were up to me. However, I also think that teachers are underpaid for what we want from them. One way to increase their effective pay is through job security. Perhaps that is what is going on here. The bargain is job security instead of higher pay. If we do not like that bargain, maybe we should make a new one: higher pay but less job security. :)

95

billikin 10.19.10 at 2:11 pm

On perhaps a lighter note, this talk about testing math reminded me of an experience that may be amusing to a third party. ;)

On a test in my introductory physics class in college there was a 10 point problem to derive a certain mathematical result. I thought the problem was quite easy. But the instructor only gave me 5 points credit. I checked my paper again, and saw no mistake. So I asked the instructor about it, and he said that my derivation was wrong. I asked him to show me my mistake. He pointed to a certain step and said that I should have made a binomial approximation. He continued with his derivation from that point, and at a later point made another binomial approximation that, as it happened, canceled out the first. I protested that my derivation was exact and required no approximation, but he would not listen to reason. {grin, I think}

96

billikin 10.19.10 at 2:13 pm

Correction:

I said, “I agree with you that it is highly unlikely that some teachers did a bad job”

OC, I meant, “I agree with you that it is highly likely that some teachers did a bad job”

97

ScentOfViolets 10.19.10 at 2:20 pm

He continued with his derivation from that point, and at a later point made another binomial approximation that, as it happened, canceled out the first. I protested that my derivation was exact and required no approximation, but he would not listen to reason. {grin, I think}

That sounds odd. Not him, you. Binomial approximations throw away information; doing it twice can’t “cancel out” because then extra information would be coming out of thin air. Spoken as someone who did very well on the Putnam – which is not multiple choice :-)

98

MPAVictoria 10.19.10 at 2:31 pm

I think the comment above focusing on the pay issue is particularly important. In Northern Alberta (my Father was a teacher, than a principal and then finally a Superintendent) teachers max out at around 80ish thousand a year. Now that is not yacht money but it is a decent wage for an educated professional. From what I gather in this thread teachers in the US make, in many cases, substantially less than 80,000. That kind of wage difference could be responsible for part of the difference in performance between Canadian and American schools.

99

Uncle Kvetch 10.19.10 at 2:32 pm

And re: the French educational system: if you are saying that a CFG curriculum, a CAP curriculum and a Bac curriculum are the same, you’re using a definition of curriculum with which I’m unfamiliar.

No, I wasn’t saying that, but I can see where we were sort of talking past each other. I’m just saying that a Bac curriculum (for instance) is going to be virtually the same in every lycée in every part of France, whether public or (publicly-funded) private. And this is in sharp contrast to the high school diploma in the US.

The idea of streaming secondary students into more pre-professional programs like the CAP is an interesting one that might hold some lessons for the US system, but I think it might be getting a little far afield of the topic at hand.

100

MPAVictoria 10.19.10 at 2:33 pm

Myles:
I don’t think it is really the job of a high school English teacher to teach John Donne in a deep way to high school students. That is what university English departments are for.

101

SamChevre 10.19.10 at 2:46 pm

RE: France–that makes sense–yes, we were talking about different things.

RE: binomial approximations–in some cases, you can do the approximation at any of several different points in a process and lose the same info (same results, but the earlier you do the approximation the more concise the model is; figuring out when to do your approximations to balance accuracy and ease of modeling is one of the big questions in modeling–the rare cases where it doesn’t matter are rare.)

102

ScentOfViolets 10.19.10 at 3:10 pm

I don’t think it is really the job of a high school English teacher to teach John Donne in a deep way to high school students. That is what university English departments are for.

Ah, the critical thinking bit. I note this because for all the talk about math, it’s really not critical thinking I’m teaching in my classes; in fact, in just about every undergraduate math course we’re striving to make the process as non-critical, automatic, and unthinking as possible. Sorta like being able to proficiently read and play sheet music. Critical thinking would be in English, History, um “Rhetoric”, that sort of thing I would think. And in high school we got “Invisible Man”, “Crime and Punishment” and “The Great Gatsby”. Good stuff but – as you say – not anything we performed any deep analysis on. How could we, given the time constraints? Certainly we never got much in the way of historical context for any of these novels, and TGG was mostly about character analysis and symbolism, the spectacles of T. J. Eckleburg looking out over the wasteland and all that. In short, the prep skills you’d need to tackle Donne in college.

Sam on binomial approximations: What you say is true, but what billikin said was that doing two approximations in succession canceled out. I’m not saying that it’s impossible, but the professional in me was intrigued enough to wonder just what sort of problem this was where such a thing was possible. Of course, this could just be an misunderstanding over what “canceled out” means.

103

piglet 10.19.10 at 3:19 pm

92: “The actuarial prelims and the math GRE are both pretty good measures of subject competency, and are multiple-choice tests.”

I thought that the math GRE was a joke. I don’t think it measures “subject competency” in any meaningful way. One problem with this particular type of test is that the underlying material is relatively simple. Nothing more advanced than high school algebra. In order to make the test seem “selective” enough (GRE is used for graduate school admission), the designers often come up with confusing questions and deliberate traps. The skill required to score high on that kind of test is not so much working out a math problem as being able to spot the traps while working fast under pressure and keeping one’s nerves. Maybe this is precisely what the test aims to evaluate – but it is not math proficiency. If you don’t believe me, have a look at some practice tests: http://www.greguide.com/gre-practice-tests.html

Many of the analogies in the verbal part of the test are patently absurd and are only there to add selectivity to the test. The designers use obscure vocabulary that many test-takers have never heard of. I understand why the designers do this – otherwise the test would be easy for too many people. I also understand that the non-standardized part of the test – the short essays graded by hired graders for low pay – is at least as problematic: grading is inconsistent and it is known that longer essays get higher scores. These tests are attempts at solving a ranking problem efficiently. They are not totally arbitrary and are somewhat related to what they are supposed to test. But meaningful evaluation tools they are not, and the importance that kind of test assumes in the US educational system is worrying.

104

JM 10.19.10 at 3:25 pm

If teacher’s unions are supposed to be the problem, then why do the least unionized states, the South, have such lousy school systems?

The focus on teacher’s unions is so obviously unfounded that I have to wonder what they’re really after.

105

SamChevre 10.19.10 at 3:27 pm

piglet–I’m thinking of the Math subject test, not the math portion of the standard GRE.
http://www.ets.org/Media/Tests/GRE/pdf/gre_0809_math_practice_book.pdf

And for an actuarial exam, look here:
http://www.soa.org/files/pdf/course1_0503.pdf

106

MPAVictoria 10.19.10 at 3:35 pm

JM
I think it is obvious what they are really after.

107

piglet 10.19.10 at 4:01 pm

Sam: I don’t know. I’m not going to spend much time on this but I had a short look. Question 2: Which of the following is an equation of the line tangent to the graph of y =x+e**x at x = 0 ?

The experienced test taker will know that they don’t actually have to differentiate, just to eliminate the obviously wrong answers. In this case 3 out of 5 choices don’t go through (0,1), so they can be eliminated. Yes, to come up with that reasoning requires some basic understanding. You have to know what a tangent is etc. But it does bother me that test-takers spend a lot of preparation on learning techniques for finding short-cuts rather than doing actual maths. In fact they have to because they lose too much time otherwise. The other thing that bothers me is that these tests consist of isolated, context-free bits. This kind of test is an end in itself. What is the effect on students of teaching to that kind of test?

108

Landru 10.19.10 at 4:28 pm

ice9: ” I can tolerate the minor negatives of teaching with a few hacks and ninnies,”

I gather that a lot of the argument comes down to what constitutes “minor” and “few” in various peoples’ judgements.

I saw “Waiting for Superman” this past weekend, and the nugget that stood out for me was when one supremely pale edu-wonk pointed to a bunch of charts and graphs and pronounced that if the US were to fire just the bottom 6% of teachers, then we’d ascend to educational nirvana (here defined as being on par with Finland, if I recall). That seems to encompass the whole philosophy: teachers are generally good, but a tiny minority of bad performers — just 1 in 17 — are enough to drag the whole system down; once you agree to that, then it’s only natural to ask why it isn’t easy, or even possible, to just prune that tiny bit of rot away and let Finland bloom.

109

Sebastian 10.19.10 at 4:30 pm

“Yes, but high school should teach calculus to as many people as possible. It should be a core part of the curriculum. And that’s exactly where I have a problem with standardized tests: because they misdirect class time and teaching time towards a nice basic level of pabulum. That’s a serious problem.”

This is a very weird objection to testing. The reason we aren’t teaching calculus to as many people as possible is because we are having trouble teaching more basic math skills to a large portion of the students. That is the fault of testing. That is a fault of teaching, and of curriculum design. The point of testing (at the end) is not to DO the teaching, it is to verify that the teaching was done. A huge number of the objections here seem to skate right over that point. See for example:

“Remind me what the evidence that scantron tests were effective in teaching maths.”

and

“Apart from entry exams, an exam is of value only insofar as it provides teachers with usable information about what the student doesn’t know. Making the exam multiple choice needlessly deprives the teacher of basic information on why they made the mistakes they did – especially in mathematics, where you can get all your reasoning right and be laid low by a simple arithmetic mistake.”

I’m not advocating scantron tests as the only method of TEACHING math. And for all those who are getting so high and mighty about my reading comprehension, it would be nice to see where you picked up the idea that I was making that argument. I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred. That is what teacher’s unions in the United States are against. They don’t like the end-of-year/end-of-high-school testing to verify that the teaching/learning actually occurred.

And you all aren’t even getting the objection to standardized testing right. The legitimate objection is that, for individual students, at the very highest levels of performance measured by the test, standardized testing isn’t great at separating out “very good” from “brilliant”. They are however quite capable of determining average levels of competence from the vast portion of the middle. Which coincidentally is exactly what we want them to be able to do. We don’t need them to evaluate fine-grain differences between individual students. What we want from these tests is that they give an overall view of about how many students have learned basic competency of certain skills. If the test gets it slightly wrong on one of the students, it will tend to average out over the very many students. The tests don’t have to be hyper-precise at the student level, they evaluate at the class/school/curriculum level. And in most cases (including the much maligned NCLB) they do it across multiple years. Such tests are perfectly adequate for that purpose. Even scantrons. Especially in math.

Whatever reason such tests are so strenuously resisted, it isn’t because they are ineffective tools at detecting basic competency among the vast majority of the tested population.

110

billikin 10.19.10 at 4:41 pm

Scent of Violets: “That sounds odd. Not him, you. Binomial approximations throw away information; doing it twice can’t “cancel out” because then extra information would be coming out of thin air.”

Yes, it is unusual. But I do not know why you point the finger at me. I didn’t make the approximations. In this case they did cancel out. For one thing, there was a change of sign. True story.

111

billikin 10.19.10 at 4:52 pm

Scent of Violets: “billikin said was that doing two approximations in succession canceled out. I’m not saying that it’s impossible, but the professional in me was intrigued enough to wonder just what sort of problem this was where such a thing was possible. Of course, this could just be an misunderstanding over what “canceled out” means.”

By “canceled out” I mean he got the correct result. :)

112

billikin 10.19.10 at 4:58 pm

JM: “If teacher’s unions are supposed to be the problem, then why do the least unionized states, the South, have such lousy school systems?”

Poverty. And two forms of racism/classism, one against Blacks, one against Rednecks. (My observations from having grown up in the South.)

“The focus on teacher’s unions is so obviously unfounded that I have to wonder what they’re really after.”

Gee, maybe they’re really after teachers’ unions. Ya think? ;)

113

billikin 10.19.10 at 5:23 pm

@Scent of Violets:

It has been a long time, but, as I recall, the guy did not like to divide. Maybe it was something like this:

Instead of dividing by (a + b), he made the following approximation:

(a + b)^(-1) -> a^(-2) * (a – b)

Then later, instead of dividing by the result, he made the following approximation:

(a^(-2) * (a – b))^(-1) -> a^2 * a^(-2) * (a + b) = (a + b)

Go figure. ;)

114

billikin 10.19.10 at 5:28 pm

The formatting came out wrong, but I think you know what I mean. :)

115

billikin 10.19.10 at 5:35 pm

Sebastian: “And in most cases (including the much maligned NCLB) they do it across multiple years. Such tests are perfectly adequate for that purpose.”

Is there an apples and oranges problem? To assess student learning over a school year, shouldn’t you have a test at the beginning of the year and one at the end? (OC, you expect the kids to flunk the first test, but you want to know what they know and don’t know coming in.) But the tests don’t do that, do they?

116

Isocrates 10.19.10 at 5:52 pm

SoV @ 102: I suppose I may regret the question, but why is the term ‘rhetoric’, referring to an academic discipline in its own right, with its own journals, conferences, and graduate programs that indeed does teach critical thinking as much as English (Literature) or History, treated to scare quotes in your comment?

117

billikin 10.19.10 at 5:53 pm

Landru: “one supremely pale edu-wonk pointed to a bunch of charts and graphs and pronounced that if the US were to fire just the bottom 6% of teachers, then we’d ascend to educational nirvana (here defined as being on par with Finland, if I recall). That seems to encompass the whole philosophy: teachers are generally good, but a tiny minority of bad performers—just 1 in 17—are enough to drag the whole system down; once you agree to that, then it’s only natural to ask why it isn’t easy, or even possible, to just prune that tiny bit of rot away and let Finland bloom.”

Do you buy that?

118

christian h. 10.19.10 at 6:00 pm

Sebastian, it is quite amazing that you seem to think you can judge the effectiveness of multiple choice math testing better than numerous professional mathematicians. To recapitulate:

1. Multiple choice tests do not adequately evaluate learning success in mathematics. What they do reflect is learning success in test taking skills.

2. Using multiple choice testing to evaluate teachers (reformer speak for lowering costs) will deform mathematics learning to test preparation. The net effect is the creation of misapprehensions in students minds’ about what math is, and how it is learned. We deal with the result in our colleges – or try to anyway. By now a majority of math grad students at leading institutions have not been educated in US schools, and most of them are from countries that do not use multiple choice standardized testing.

3. Using standardized testing to identify teachers to be fired will not in fact improve teaching ability in general since you can’t pull perfect teachers out of your ass. Rather it will likely have the opposite effect by lowering morale and making an economically undesirable line of work even less desirable.

4. School reformers are by and large smart enough to know these things. It therefore stands to reason that improved learning is not, in fact, their goal. Given the actual outcomes it is much more likely that their goals are (a) cost cutting, (b) destroying the self-confidence of yet another group of workers, and (c) opening up of yet another public service to private profit making.

119

billikin 10.19.10 at 6:14 pm

Let me try those approximations again, just for fun:

(a + b)^-1^ -> a^-2^ * (a – b)

(a^-2^ * (a – b)) -> a^2^ * a^-2^ * (a + b) = (a + b)

120

billikin 10.19.10 at 6:15 pm

Rats! I left out the exponent in the second one, but the formatting came out OK.

121

Substance McGravitas 10.19.10 at 6:20 pm

Math is hard.

122

Cathy Fiorello 10.19.10 at 6:54 pm

People, the AP exams are standardized tests. If what you are bitching about is low-level, poorly designed tests, bitch about that. There are a lot of standardized tests that are damn good, and teaching to them is real teaching.

123

engels 10.19.10 at 8:12 pm

Sebastian, it is quite amazing that you seem to think you can judge the effectiveness of multiple choice math testing better than numerous professional mathematicians.

Christian, you are obviously unfamiliar with the great number of discussions on this blog involving Sebastian, which, as a rule, pit the genuine expertise in various fields of human learning and experience of his interlocutors against Sebastian’s, ahem, first-rate command of English prose and American tax law and unflagging confidence in his own opinions. I am envious!

124

rosmar 10.19.10 at 8:30 pm

I would like to elaborate on my earlier comment about the systematic problems caused by stereotype threat. You can lower the average score of girls on math grades just by reminding them (by having them fill out a “gender” bubble at the top of the test, for example) that they are girls. Similarly for Black and Latino students on many kinds of tests. Stereotype threat (the fear of fulfilling a stereotype, which makes people slow down and second-guess themselves, exactly the wrong thing to do if you want to score as high as possible on a timed standardized test) all by itself makes standardized tests unjust.

125

billikin 10.19.10 at 8:35 pm

Isocrates: “‘rhetoric’, referring to an academic discipline in its own right, with its own journals, conferences, and graduate programs that indeed does teach critical thinking”

I remember when I first saw a critical thinking textbook. It made me think of my grandmother’s high school rhetoric textbook, and, IMO, critical thinking suffered by comparison. But a critical thinking course is better than nothing, which was the case when I was in school.

126

Peter Nunns 10.19.10 at 10:02 pm

Cathy Fiorello @122
People, the AP exams are standardized tests. If what you are bitching about is low-level, poorly designed tests, bitch about that. There are a lot of standardized tests that are damn good, and teaching to them is real teaching.

This is exactly what I have been saying. It hasn’t been picked up in the discussion.

That being said, test strategy – such as knowing how many answers you have to eliminate to make guessing worthwhile – and familiarity with test formats plays an incredibly important role even on the AP tests. When teachers have to spend their time explaining these things, classroom time is being diverted from more valuable purposes (such as teaching calculus).

Sebastian @109:
“Yes, but high school should teach calculus to as many people as possible. It should be a core part of the curriculum. And that’s exactly where I have a problem with standardized tests: because they misdirect class time and teaching time towards a nice basic level of pabulum. That’s a serious problem.”

This is a very weird objection to testing. The reason we aren’t teaching calculus to as many people as possible is because we are having trouble teaching more basic math skills to a large portion of the students. That is the fault of testing. That is a fault of teaching, and of curriculum design. The point of testing (at the end) is not to DO the teaching, it is to verify that the teaching was done. A huge number of the objections here seem to skate right over that point.

And you seem to skate over the substance of my comment (@84) – which is that standardized testing regimes create a major incentive to teach to the test, which is to say, focus on a lot of rather irrelevant things. So you get the absurdity of math teachers teaching students how to eliminate the two or three obviously wrong answers and then guess between the rest rather than how to complete a proof or calculate a value. That’s not an education.

Furthermore, you don’t even bother to comment on the tendency for standardized testing at the appropriate-pabulum level to degrade academic standards at high-achieving schools. For the nth time, I am probably the only commentator here who has been subjected to standardized testing in school – I’m telling you what I observed happening. In all good conscience, I can’t support a system that undermines successful schools, and successful teachers, in the name of “higher standards”.

127

StevenAttewell 10.19.10 at 10:14 pm

If what was being debated in the fight over testing was AP vs. nothing, my guess is that you wouldn’t see any objection from the teacher’s unions. The APs are high quality tests that measure learning as well as aptitude, are subject-specific and comprehensive, contain a mix of multiple choice and written questions, etc.

The interesting question is why the Rhees and Kleins of the world haven’t jumped at the chance to square the circle. I would argue that the APs don’t lend themselves to their preferred management strategies – among other things, AP exams do not lend themselves to the give-a-letter-grade-to-the-school, rank-the-teachers-by-value-added, success-or-failure, annual-improvement model. Indeed, an AP system would likely show some of the more complex realities of education that go beyond the strange fixation we have that everyone in the country should be educationally ranked on the basis of their reading and math skills, nothing else, and that everyone must be good at both.

128

Stuart Buck 10.19.10 at 10:20 pm

Given all of the discussion of math tests above, it might help if some of the professional mathematicians could examine an actual state test and explain why it doesn’t do a useful job measuring whether elementary students have a good foundation in math skills.

Here’s a 6th grade math test from Florida: http://fcat.fldoe.org/pdf/releasepdf/07/FL07_G6M_TB_Rel_WT_C002.pdf
Please explain that test’s inadequacy in light of the technical document about its construction and validity: http://fcat.fldoe.org/pdf/fc06tech.pdf

129

Peter Nunns 10.19.10 at 10:31 pm

StevenAttewell@127: Yes! Precisely!

I think it would be possible – even desirable – to develop a universal testing model based on the AP tests. Currently, they cater solely to high-achieving students, but I think it would be possible to write tests and curricula for a range of aptitude levels. You could imagine an algebra or trigonometry test that followed a similar model to the AP Calculus exam.

(I would argue that the success of the AP exams is due to the fact that they were developed with teaching in mind. Questions on the test are integrally tied into a class’s curriculum and pedagogical purpose. So for example, the AP US History exam doesn’t just require students to know when the Civil War was, or who was the president at the time. It requires them to be able to navigate and decide between arguments about why the Civil War occurred – which is after all the real goal of a history course. State- or NCLB-mandated standardized testing, by contrast, has little to no connection with the curriculum in any given year, and hence it’s largely futile.)

But for this to work, the goal would have to be to give students qualifications rather than to assess whether they were achieving. It’s a subtle but important difference. Think of the British school system, where you can leave school with O- or A-level qualifications rather than just a high school diploma that mainly indicates that you haven’t drooled too much in class. Testing based around qualifying students in specific subjects would be a good thing, as it requires sophisticated tests that are integrally linked with curricula.

130

ScentOfViolets 10.19.10 at 10:35 pm

SamChevre @105:

I took that Math Subject exam and looked at it again just now. Believe me, that’s merely an extremely low floor to screen out the bottom. It does not measure ability, knowledge, what have you except in a very rudimentary sense. Sort of like taking a screening test to see if you can read sheet music as part of the process to get into Juilliard. If you pass, it really doesn’t mean much, if you fail, you’ve been very cheaply weeded out.

131

Sebastian 10.19.10 at 10:41 pm

Thank you Stuart. Please, SoV and others, explain why these tests are so horribly inappropriate.

132

bianca steele 10.19.10 at 10:44 pm

Looking at the 10th grade math test for this state’s high stakes tests (required for graduation), and given what I remember about the level of pre-calculus understanding and instruction even among kids bound for selective colleges (say, 1/3 probably reliably mastered the material, at most), I find it difficult to believe 50% of high school students could answer some of these questions on the basis of their regular math classes. If questions are drawn from a small part of the subject, it would be easier to teach to them. (I looked for a way to map raw scores to “advanced,” “proficient,” etc., and didn’t find it.) We certainly took out two weeks a year to drill arithmetic, fractions, etc., for the standardized tests we were subjected to, back in the stone age (which were used to determine the top 12%, etc., for tracking, e.g., eligibility for AP classes).

133

ScentOfViolets 10.19.10 at 10:48 pm

I’m not advocating scantron tests as the only method of TEACHING math. And for all those who are getting so high and mighty about my reading comprehension, it would be nice to see where you picked up the idea that I was making that argument. I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred.

Sigh. Sebastian, will you please keep track of what you have already written:

Universities do it all the time using scan-tron tests for pretty much all the classes from pre-algebra straight through differential equations, which is probably further into math than at least half the readers here have gotten.

And more of the same. No, universities don’t “do it all the time” and your presumption that they do is laughable. And no, per your other stuff, it looks pretty clear to me that you were saying that teaching to the multiple choice test is teaching math. Do you not understand how your posting style makes you something of a figure to be mocked, the very stereotype of the dull and ignorant school choice crowd?

134

ScentOfViolets 10.19.10 at 10:52 pm

SoV @ 102: I suppose I may regret the question, but why is the term ‘rhetoric’, referring to an academic discipline in its own right, with its own journals, conferences, and graduate programs that indeed does teach critical thinking as much as English (Literature) or History, treated to scare quotes in your comment?

Oh, it’s just that we never had classes in high school that were specifically called “rhetoric”, but that’s really what they were about. A very valuable set of skills. Has this changed since my time? That’s all I meant with the quotes, nothing derogatory. To the contrary.

135

ScentOfViolets 10.19.10 at 11:00 pm

Let me try those approximations again, just for fun:

(a + b)-1 -> a-2 * (a – b)

(a-2 * (a – b)) -> a2 * a-2 * (a + b) = (a + b)

Rats! I left out the exponent in the second one, but the formatting came out OK.

Okay, I see, basically divide by a quantity then divide by it’s reciprocal. Yeah, that’ll get you back to where you started if you don’t do any expansion tricks, but it’s just luck that it worked out on this one. If you do a three-term expansion twice, you get back a+b, plus some other stuff: -2b^3/a^2+b^4/a^3. So your guy just got lucky. Assuming I’ve done the expansion correctly, of course :-)

136

ScentOfViolets 10.19.10 at 11:05 pm

Ah, -2b^3 / a^2 + b^4 /a^3 . That’s the extra bit. Sorry.

137

ScentOfViolets 10.19.10 at 11:08 pm

Looking at the 10th grade math test for this state’s high stakes tests (required for graduation), and given what I remember about the level of pre-calculus understanding and instruction even among kids bound for selective colleges (say, 1/3 probably reliably mastered the material, at most), I find it difficult to believe 50% of high school students could answer some of these questions on the basis of their regular math classes.

Right. There’s a big difference between using a test as a filter, and using it to accurately assess the student’s knowledge and ability. Multiple choice can, in some instances, work for the first bit. But not, generally speaking, for the second one.

138

Uncle Kvetch 10.19.10 at 11:48 pm

Please, SoV and others, explain why these tests are so horribly inappropriate.

Please, Sebastian, explain what makes teachers’ unions in the US qualitatively “worse” than those in countries that get better educational results. You made that claim several times upthread — never offering any evidence, mind you, it was always solely based on your “impression” and what you “think” — and once people started calling you on it, you went completely mute on the subject.

Just this once, just to switch things up a bit and keep it interesting, how about you actually present an argument and defend it? Or is “This is what I want to be true, so I hereby declare it true — now prove me wrong” the only song you know?

139

ice9 10.20.10 at 12:40 am

AP tests are also normed. That makes them very interesting diagnostic tools.

Plus they can count for college credit, so there’s a strong incentive.

Elsewhere, incentive is, shall we say, thin. For many, perhaps most of the ‘high stakes’ tests the main emphasis in ‘teaching’ is actually persuading kids to take the tests seriously, to do their best, or maybe just to show up unstoned. Example: A few years ago my high school failed to make AYP for one category: male African Americans, reading. That one category failure cost the school a number of, you know, gold stars or whatever. The whole faculty was shaking their heads and looking dejected like they’d just let the team down, killed the Gipper. Actually, I was the one who let the team down. On Test Dayz!!! everybody is pressed into service to administer the test (and you can’t look at it, oh noez–not a peek, and if you should post one on an Internet Blog Site you’d be cashiered on the spot–our hands signed a paper. Really.) Anyway, in my group of test takers were three 11th grade black males, all together by virtue of the alphabet and their standardized anglo-American slave names. The test hit their desks, and their heads soon after; the three together bubbled nary a bubble. Those were three of the 11 african-american 11th graders in our entire school of 3,000 plus; their abstention on the exam cost us our gold star because of those dang statistics and the math that SOV keeps plugging away at above. I tried to get them to bubble, I really did, but they smiled politely and shook their heads. The problem: they’d been told that they couldn’t text during the exam, and that made them unhappy, so they gave it a miss.

ice9

140

Sebastian 10.20.10 at 1:07 am

Scentofviolets: “And no, per your other stuff, it looks pretty clear to me that you were saying that teaching to the multiple choice test is teaching math.”

Let’s try some context on for size shall we? When having a discussion, it sometimes works to actually read and comprehend the context.

I was responding to this statement: “Apart from entry exams, an exam is of value only insofar as it provides teachers with usable information about what the student doesn’t know. Making the exam multiple choice needlessly deprives the teacher of basic information on why they made the mistakes they did – especially in mathematics, where you can get all your reasoning right and be laid low by a simple arithmetic mistake.”

The way you can tell that I was responding to this statement is that I quoted it, and then responded to it.

I wrote: “I’m not advocating scantron tests as the only method of TEACHING math. And for all those who are getting so high and mighty about my reading comprehension, it would be nice to see where you picked up the idea that I was making that argument. I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred.”

I even preface the quote with: “The point of testing (at the end) is not to DO the teaching, it is to verify that the teaching was done. A huge number of the objections here seem to skate right over that point.”

Now if you want to, I suppose you can try to pretend that is me advocating scantrons as a method of teaching, instead of a possible method of end-of-class-minimimum-proficiency testing.

But that isn’t the claim I make anywhere here. The claim I make is that “teaching to the test” is teaching math (for the level of math that we are talking about here). If you teach math, the students will on average be capable of demonstrating proficiency. I’m not advocating “needlessly depriv[ing] the teacher of basic information on why they made the mistakes they did” because I’m not advocating that the scantrons be the only method of teaching during the year. And depriving them of that at the end doesn’t matter, because the kids already have learned or not learned the subject.

And now that Stuart has provided a good test, actually used by one of the biggest states in the union, I’d love to see you explain how “teaching to the test” is likely to leave us with lots and lots of kids who CAN pass the test but DON’T actually know math.

141

Sebastian 10.20.10 at 1:20 am

“Right. There’s a big difference between using a test as a filter, and using it to accurately assess the student’s knowledge and ability. Multiple choice can, in some instances, work for the first bit. But not, generally speaking, for the second one.”

I’m confused by this statement. Aren’t we asking for a filter, by which we set a minimum proficiency level and note those who pass? We aren’t asking for a fine grained analysis tool which sorts the students into 100 levels of achievement. We are asking for a test that can on average identify the general number of students who have achieved minimum competence. We aren’t even asking it to be as fined grained as an SAT, GRE, or AP test. If you think standardized tests can do that, what in the world are we arguing about?

142

StevenAttewell 10.20.10 at 1:32 am

Peter – Precisely. One of the things I’m interested in – especially since this was very much the case with my brother and myself when we were going through high school – is the way in which our education system looks for a very precise kind of polymath that has a lot to do with regurgitation and test-taking, but not much to do with other skills.

One of the reasons why I favor the AP test is precisely its similarity to the O and A levels in the British system – which allow for students to develop particular strengths to offset their weaknesses; this allows students to shape their college applications in that direction (i.e, allowing someone who’s great at math and sciences but terrible at Shakespeare the ability to present themselves to MIT as a prospective math or sciences major, and allowing someone who’s great at literature, history, etc. but terrible at math to present themselves as a prospective liberal arts major, without being held back by their specialization). It also allows universities to better balance their intake.

But this rather conflicts with our model of education, which irrationally insists that people should be prized for showing facility with quantitative and qualitative subjects more than people who show a higher facility in one area but a weaker facility in the other. For example, in many high schools, certain subjects – usually math – are used as “gatekeepers” for advanced courses in other subjects.

143

ScentOfViolets 10.20.10 at 1:56 am

Those were three of the 11 african-american 11th graders in our entire school of 3,000 plus; their abstention on the exam cost us our gold star because of those dang statistics and the math that SOV keeps plugging away at above. I tried to get them to bubble, I really did, but they smiled politely and shook their heads. The problem: they’d been told that they couldn’t text during the exam, and that made them unhappy, so they gave it a miss.

Plugging away is right; there’s no substitute for practice, practice, practice.

Let me tell you the rest of the story. Those three guys graduated despite profound academic deficiencies and a really terrible work ethic because people like me aren’t allowed to flunk them out. Later on, people like me (can you tell this is . . . irksome) will be forced to take at least one section of what is now 40 (and used to be 5, back in the 1980’s) sections of remedial math for entering freshmen to bring these jokers up to speed. Then we’ll be barraged constantly by parents telling us that their kid “needs” to be passed despite their abysmally low performance and their terrible work ethic, alternately begging for mercy and threatening terrible retributions (and you know, if their boy isn’t learning, it’s my fault. Despite their complete unwillingness to back me up on doing the homework.)

No, ma’am, your kid is a sorry little excuse for a human being, and a big part of that is on you. Had you been doing your job at home, showing some backbone and some discipline, your child might have been of better character and might actually have the self-discipline we used to take for granted as a necessary perquisite for academic success.

And until that issue is addressed, you’re simply not going to see a lot of reform; only a round robin of finger pointing and cheapo “solutions” that doesn’t cost anyone anything (except the teachers.) Rinse and repeat.

144

ScentOfViolets 10.20.10 at 2:02 am

I’m confused by this statement. Aren’t we asking for a filter, by which we set a minimum proficiency level and note those who pass? We aren’t asking for a fine grained analysis tool which sorts the students into 100 levels of achievement. We are asking for a test that can on average identify the general number of students who have achieved minimum competence.

Uh-huh. To repeat myself, Juilliard gives a test to prospective students to see if they can read sheet music. So if they pass, they must be Juilliard material on account of the fact that they demonstrated the ability to sight read. Right.

Sebastian, would it hurt you so much to actually listen to what people say occasionally? And would it be so tiresome to actually present some evidence for your assertions and back up what you’re saying, rather than challenging people to prove you wrong? Because right now, it seems that the rough consensus is that you suck at general logic and reasoning. I know you think that you’re being clever by engaging us as a lawyer, but that doesn’t appear to be cutting any ice; the rest of us are by and large scientists.

They eat lawyers for breakfast ;-)

145

subdoxastic 10.20.10 at 2:12 am

Hi Sebastien:

On principle I see nothing wrong with testing and have had difficulty explaining this to others, so I can sympatize (a little) with your plight here. I don’t want to get into ad hominem attacks or debates about tone or literary style, but I do have a question.

You mention specifically that in your preface to one of your posts that, “The point of testing is not to DO the testing, it is to verify that the teaching was done.” Do you believe that standardised testsare a reliable and valid method of determining that “the teaching was done”? If yes, can you think of any possible objections to this stance?

Now, I know that you said “verify the teaching/learning has occurred” but perhaps what has got so many people’s backs up is your (seemingly, and I fully admit that this is merely my interpretation) emphasis on these tests ability to measure whether teaching is done ( I’m not referencing the other debate as to whether or not standarised tests can validly measure whether learning has occured– that’s a whole other black box problem).

I know what teachers with multiple years of experience will say, but I’m interested in your opinion. Do you think standardised tests provide a useful metric for teacher performance?

146

Walt 10.20.10 at 2:14 am

I can’t endorse SoV’s last comment. I eat Cheerios for breakfast.

147

Sebastian 10.20.10 at 3:08 am

“Uh-huh. To repeat myself, Juilliard gives a test to prospective students to see if they can read sheet music. So if they pass, they must be Juilliard material on account of the fact that they demonstrated the ability to sight read. Right.” [and then a paragraph of extraneous character attack]

See the thing is, we are talking about BASIC math skills. We are most specifically NOT talking about deciding whether kids are Julliard material, or what level of math wizard they are. So yes, we are testing to “can you read sheet music” levels. Which btw, a frightening number of kids are failing. So let’s quit with the condescension and talk about it.

148

Charles St. Pierre 10.20.10 at 3:22 am

If I may make a practical suggestion: Devil take the hindmost. Let us say 3% per year be subject to firing, criteria to be negotiated between the union and the employer. Surely, a union can’t object to its worst 3% being liable to firing, or to put it the other way, can’t justify forcing the employer to retain the worst 3% of their employees.

Since nobody will want to be in the lowest 3%, this will create incentive to a certain minimum performance, an incentive that will increase with time, as the very worst are progressively culled.

(I already posted this at over at Marginal Revolution.)

149

Substance McGravitas 10.20.10 at 3:26 am

See the thing is, we are talking about BASIC math skills.

Here are a couple of attempts at assessment of the learning of kids from various countries:

http://en.wikipedia.org/wiki/Programme_for_International_Student_Assessment
http://en.wikipedia.org/wiki/Trends_in_International_Mathematics_and_Science_Study

Where should the US be on such charts for you to consider American schooling an overall success?

150

Sebastian 10.20.10 at 3:57 am

Substance, your cite says:

“Data for US students is further tracked for ethnic and racial groups, which can be tracked as the nation. As a whole, grade four students in the United States lagged the best Asian and European nations in the 2007 TIMSS international math and science test. However, broken down by race, Asian Americans scored comparably to Asian nations, European Americans scored comparably to the best European nations (although European nations aggregate their own result independent of race or origin). Hispanic Americans averaged 505, comparable to students in Austria and Sweden, while African Americans at 482 were comparable to Norway.

Grade eight students in the United States also lagged the best Asian and European nations in the 2007 TIMSS international math and science test. Broken down by race, US Asians scored comparably to Asian nations, white Americans scored comparably to the best European nations, Hispanic Americans averaged 475, comparable to students in Malaysia , while African Americans at 457 were comparable to Bosnia and Herzegovina and Lebanon.”

I’d like to see African Americans and Hispanic Americans doing better by grade 4 and not losing such a tremendous amount of ground by grade 8. I would however note the concern (discussed above) that such tests tend to be better and minimum level competency identification rather than rank ordering, and as such I’m not sure that asking where we should be on the rank ordering is a question that makes sense.

151

Peter Nunns 10.20.10 at 3:59 am

Substance McGravitas @149
Here are a couple of attempts at assessment of the learning of kids from various countries…

Here’s another fun thing that can be done with these charts: We could take a look at the countries that consistently rank in the top 10 and try to figure out what they’re doing that the US isn’t. But on the other hand that might be hard and perhaps it would be better to look for a silver bullet that’s amenable to political sloganeering (e.g. “Test the kids! Fire the bad teachers!”).

Since we’re agreed on that, I’ll start: To the best of my knowledge, education funding and core curriculum in each of the top-performing countries is centralized rather than Balkanized across 50 states and thousands of school districts.

Oh wait! That’s an observation that lends itself to a stunning and practical policy measure: Let’s Balkanize the education system further by encouraging school vouchers and the establishment of privately-run and sometimes privately-funded charter schools. Brilliant!

152

Sebastian 10.20.10 at 4:03 am

“If I may make a practical suggestion: Devil take the hindmost. Let us say 3% per year be subject to firing, criteria to be negotiated between the union and the employer. Surely, a union can’t object to its worst 3% being liable to firing, or to put it the other way, can’t justify forcing the employer to retain the worst 3% of their employees.”

This actually isn’t a good idea, at least as a long term plan. 3% of the worst over the next 5 years might be good, (after 5 years of evaluation, testing, training etc). 3% every year sets up a problem if you are unlucky one year, and it sets up a long term dynamic where you are subject to firing even if your kids are learning to the competency level. I wouldn’t support a 3% per year thing. I’m not even particularly worried about setting up a big firing thing. But in order to make things better we have to be willing to identify what works and what doesn’t. And that is impossible if you hide from the data by refusing to allow testing.

153

Charles St. Pierre 10.20.10 at 4:47 am

Sebastian @152 said:

“This actually isn’t a good idea, at least as a long term plan. 3% of the worst over the next 5 years might be good, (after 5 years of evaluation, testing, training etc). 3% every year sets up a problem if you are unlucky one year, and it sets up a long term dynamic where you are subject to firing even if your kids are learning to the competency level.”

3% isn’t much, especially since you’re going to have turnover, retirements, new hires, etc, anyway. In fact, if the mean duration of employment is 33 years, you’re going to have 3% per year turnover anyway. And a teacher with kids learning at the competency level could be exempted. The 3% wouldn’t be compulsory. If all teachers are showing competency, it could be 0%. Further, the union could have some say in who was fired. They would just have to allow 3%.

154

Salient 10.20.10 at 4:51 am

Aren’t we asking for a filter, by which we set a minimum proficiency level and note those who pass? We aren’t asking for a fine grained analysis tool which sorts the students into 10 levels of achievement. [I changed 100 to 10 to reduce hyperbole.]

Why not ask for a reasonably fine-grained analysis tool? Math ‘proficiency’ for a future engineer or physicist isn’t the same as math ‘proficiency’ for a future restaurant owner or nurse, which isn’t the same as ‘proficiency’ for a future foreman. Why set the same bar for everyone, regardless of what they find the most joy in or pursue most fervently? Cripes, small wonder kids become disinterested in school en masse. There’s no greater social force for homogenization in their lives.

I would argue that the lack of interest in fine-grained metrics, and a disproportionate interest invested in binary metrics, is compromising the credibility of many accountability-from-testing arguments. Even the binary-oriented NCLB allows for distinguishing ‘Distinguished’ students from ‘Proficient’ students, and awarding weight to the school for producing more Distinguished students.

Surely the need for fine-grained distinction is intuitive. I would hope someone who loves math and invests more time in it could clear a higher bar, and have their pairing of effort and facility acknowledged as worthwhile (and I would hope that teachers who facilitate this achievement would be likewise acknowledged).

If we teach ten more students to add fractions, at the expense of teaching two fewer students how to think about differentiation, I am not convinced [or perhaps, I am no longer convinced] that we have achieved a worthwhile trade-off for society.

155

Salient 10.20.10 at 5:03 am

Also, even when we have a fine-grained tool it tends to measure one-dimensionally. What we need is greater refinement of our tools in both depth and breadth. It’s something I was hoping to see from NCLB, and the focus on standards — the Program of Studies in my state generated to be NCLB-compatible, for example, is quite good, in that it specifies several independent dimensions in which a person can be mathematically competent (geometrical reasoning, algebraic thinking and abstraction, measurement and comparison of quantity, etc).

The ‘core content for assessment,’ by comparison, is [a] considerably narrower and more streamlined to be computational, and [b] measured and aggregated along a one-dimensional scale, rather than retaining fine measurements across those dimensional categories so carefully constructed for the POS. It’s a shame.

‘Teach to the test’ here literally means ‘ignore 50-80% of the POS, especially the stuff that’s hard to test in standardized formats because the relevant questions are kinda long.’ And several folks upthread are right: given a better, finer, more sensitive battery of qualifications tests, teaching to the POS-thorough test would be just fine. (It would be also impossible, because you can’t teach kids to fake their way through that kind of stuff, which is what ‘teach to the test’ is all about in practice: exploiting the deficiencies of the test format to produce unrepresentatively high results.)

156

Geoff Swenson 10.20.10 at 6:35 am

I had a engineering calculus instructor in college that graded tests based on the presentation. Getting the answer correct was not good enough. He wanted to see the intermediate steps needed to get tot he final answer. This is not something you can grade on a standardized test.

If you messed up on the calculation, but presented the steps in a clear fashion, he was fine with that and assigned 9 out of 10 points for the answer.

Even if you missed canceling out a term in an early step and got a wildly different answer, as long as the remaining steps were correctly presented he also gave nearly full credit for the answer. He occasionally gave an extra point for an correct answer with a good presentation.

Getting the answer correct was only good for about half of the points, if he didn’t see a presentation that made sense.

He justified this saying that if you make a good presentation, it is easy for others to look at your work and find the step that isn’t correct.

157

Alex 10.20.10 at 11:07 am

148: and how does this “incentive” actually have any impact whatsoever, if we can’t actually define what constitutes good teaching? what is the mechanism of action? How do you improve in the absence of information?

After you control for all the typical demographic variables (ethnic origin, parents’ income, % second language, % special needs, yadda yadda), the result you get will still have a distribution of some kind because, y’know, some kids are dafter than others. If it’s a normal distribution and we’re using the 3rd (or 97th, depending on how you look at it) percentile as the trigger-level, you’ll have a bit less than a 1 in 33 chance of being fired every term purely by chance. With three terms a year, you’re looking at firing the entire workforce about once every 11 years on the basis of pure statistical noise.

So why not come out straight and stand up for a return to the great truths of the classical canon? Decimation. Let’s sack 10% of the teachers at random. That’ll keep them on their toes.

158

dsquared 10.20.10 at 11:52 am

My understanding is that when pressed on the subject of where it is that they plan to hire all these excellent teachers from, given that the ones we had are the ones who showed up at the wage and conditions offered, people retreat to arguments about “incentives” – ie, that the existing stock of teachers will be motivated by fear.

159

Alex 10.20.10 at 1:01 pm

Hence my Western Canon Plan. Random elimination FTW!

160

LFC 10.20.10 at 1:04 pm

SAttewell@142:
this rather conflicts with our [U.S.] model of education, which irrationally insists that people should be prized for showing facility with quantitative and qualitative subjects more than people who show a higher facility in one area but a weaker facility in the other. For example, in many high schools, certain subjects – usually math – are used as “gatekeepers” for advanced courses in other subjects.

This does not conform to my experience in high school, which was admittedly several decades ago. I was not that interested in math and not especially good at it, compared to some of my peers. I took high school math through trigonometry but did not take calculus, though the school offered it and quite a few people took it. But I did take the AP English and history courses (European and American), and math was in no way a “gatekeeper” for those courses. Perhaps things have changed since the 70s, when I was in high school. One change I am aware of, at least in the one school district with which I have some familiarity, is that the number and variety of AP courses has greatly proliferated; this may have both an upside, in terms of increasing access, and a downside, in terms perhaps of weakening some of the courses themselves.

161

LFC 10.20.10 at 1:06 pm

p.s. To be exact, I graduated from high school in 1975.

162

nick s 10.20.10 at 1:19 pm

Sebastian is:

a) butthurt
b) tendentious
c) both a) and b)
d) neither a) nor b)

163

rea 10.20.10 at 1:53 pm

I know you think that you’re being clever by engaging us as a lawyer, but that doesn’t appear to be cutting any ice; the rest of us are by and large scientists.
They eat lawyers for breakfast ;-)

Sebastian is not a litigator, Sebastian is a pension and benefits specialist, to the best of my knowledge. A first rate litigator will not get eaten for lunch by a scientist (usually encountered in the form of doctors), and instead will devour any pureyor of junk science. Usually, when an honest, first rate scientist meets a first rate litigator, the result is truth, or a reasonable approximation thereof.

Sebastian is a remarkably decent and honorable guy for a conservative, but on some issues . . . well, he knows what he knows, and nothing will convince him otherwise.

164

Landru 10.20.10 at 2:50 pm

Usually, when an honest, first rate scientist meets a first rate litigator, the result is truth, or a reasonable approximation thereof.

Anecdotally, this is not my experience. I’ve had a few conversations/arguments with first-rate litigators — I’m a scientist, though I won’t reveal my rating — and I would say that whenever they were in a losing position regarding facts or logic they would turn to make every effort to nitpick on to tiny, typo-level errors in language while avoiding the main subject completely. This may save the litigators from serving as breakfast fare but it certainly did not help in getting closer to the truth, in any approximation.

165

Salient 10.20.10 at 2:52 pm

LFC –

My experience is: the proliferation of AP courses [history here has not markedly reduced the quality of courses. And the newer courses I’ve looked at, e.g. Environmental Science, seem solid and excellently well designed — AP ES stacked up pretty solidly compared to the intro to environmental science course I took at UW-Madison. Students who scored 3+ on calculus BC and take Multivariate Calculus here at the uni where I teach score on par with students who took our 1-year Calc sequence; scoring a 5 on the AP test for Calculus BC seems to correlate strongly with earning both an A in M-v Calc and respect for one’s mathematical facility from the M-v Calc instructor. Folks around here speak pretty well of AP tests’ predictive value. TIMSS offers corroboration of this [PDF] for math and physics.

Mileage may vary depending on the school and teacher and etc, of course, but the advantage of having the national College Board program is clear — when students have to pass an exam not written by the teacher, and the exam is allowed to range over the entire breadth of course content as well as material that is newly presented on the test for students to digest and respond to in real-time, parents have less room for complaint or protest against demanding and thorough teachers, and more room to complain about and protest vapid or undemanding teachers. The incentives are well-aligned. “Teach my kid to do well on this test” equates to “teach my kid to know the foundational material well, and to think critically about new material.”

Note that I mean incentives for pouty pushy parents, not for teachers. That’s where we really need to adapt and mould the incentive structure: the most consistent and loud and urgent social force for micro-changes in education, at the local level, is the group of parents who attend PTA meetings because they want their kid to get ahead. It’s important to mobilize that particular force in socially productive directions (because it’s going to be there and it’s going to impact policy). To me, that means nationalizing the curriculum and credentialing process as much as possible.

Distancing the requirements to get a diploma from teachers — that is, separating teaching from summative assessment — seems to me to be an unqualified good. Giving teachers more power to teach flexibly, but taking the power to control all the outcomes of final summative assessment out of their hands, means that teachers can point to the ‘bad cop’ of e.g. the College Board and play the ‘good cop’ who is demanding, edgy, unconventional, etc because they want their students to succeed. I think it should be mandated that 20% of a student’s grade in non-elective courses must be determined by performance on some national or international exam — perhaps in place of a teacher-written final exam.

So specifically I think an AP-style program, extended to proficiency tests in core content and the standard program of studies, with requirements to get 3+ in order to receive a high school diploma, would be an amazingly good thing.

Problem is, there are strong incentives for states to design crappy tests and make their deficiencies easy to exploit. My own state separates the basic “core content for assessment” from the actual full program of studies, and the net effect is to… not teach anything that’s not core content. And the tests [even the non-multiple choice portions] are mostly designed so that students who are taught exam-specific tricks can back their way into many of the answers without even fully understanding the questions. Just as importantly, passing or failing the test has no impact on graduation or course grade. Them’s some perverse incentives.

I guess this is getting away from union-busting, so I should mention that teachers’ unions help insulate teachers somewhat from the force of pouty pushy parents. When the PPP force is pushing in more productive directions than it currently is, I’ll be a little more sympathetic to calls for union busting — at least I’ll listen with a more open ear.

For now, no. As it is, when I hear about a PPP-force movement to get a teacher fired around here, it’s frankly usually because the teacher is atheist or allegedly homosexual, not because the teacher is underperforming. (The actual reason given, the teacher is “disrespectful to students’ beliefs” or some such thing. What I hear later is that the disrespect was merely a reasonable demand that was provoked, e.g., by a student insisting on reading their bible instead of their history book in a world history class. Hmm.)

166

Steve LaBonne 10.20.10 at 2:53 pm

A first rate litigator will not get eaten for lunch by a scientist (usually encountered in the form of doctors

Bad example. Doctors understand less than nothing about science (cramming their heads with masses of disconnected facts does not constitute training in scientific reasoning, quite the contrary), unless they have serious research training and experience subsequent to their medical training. Of course, they THINK they know everything- which makes them far more vulnerable in court than a real scientist.

167

Salient 10.20.10 at 2:57 pm

Since it got lost in me being Long on the Internets, I’ll shorter-me this in: tests which present brand-new content and require students to digest it in real time, test critical thinking and analysis. Such tests are hard but not impossible to write, and are essential if we want to teach ‘critical thinking’ and verify results.

Cf. the DLAB test for language-learning proficiency. I suggest, only somewhat tongue-in-cheek, that we outsource design of all these tests to the military.

168

engels 10.20.10 at 3:07 pm

Hence my Western Canon Plan. Random elimination FTW!

And what was wrong with my proposal?

169

Steve LaBonne 10.20.10 at 3:07 pm

Salient @ 167- when I taught molecular genetics I always wrote exams on which most of the questions worked that way- usually by presenting fictional experimental data and asking the students to analyze it using the principles which they hopefully had learned. I agree that it’s a much more effective way of assaying learning than asking them to spit out rote-memorized factoids. Sadly, many students even then (early 90s) were conditioned by their K-12 education to prefer the latter and to find the former uncomfortable and “unfair”. I imagine it’s even worse after years of the damage wrought by NCLB.

170

Alex 10.20.10 at 3:31 pm

Engels, that wouldn’t be crunchy. Remember “Crunchycons”? What happened to them?

171

Sebastian 10.20.10 at 4:02 pm

“Surely the need for fine-grained distinction is intuitive. I would hope someone who loves math and invests more time in it could clear a higher bar, and have their pairing of effort and facility acknowledged as worthwhile (and I would hope that teachers who facilitate this achievement would be likewise acknowledged).

If we teach ten more students to add fractions, at the expense of teaching two fewer students how to think about differentiation, I am not convinced [or perhaps, I am no longer convinced] that we have achieved a worthwhile trade-off for society.”

I’d be thrilled with a fine-grained assessment. But I’m being told that even the binary assessment can’t be done, so I’m not going to try to argue for an even better one with people who don’t believe that minimum proficiency can be detected in a test. And the last paragraph is a different question of values altogether.

I’d still be thrilled to find out how “teaching to the test” like the Florida test example given above is likely to cause problems in learning basic math, and how it would be unable to detect basic math competency.

172

billikin 10.20.10 at 4:26 pm

On teaching to the test in math:

My girlfriend tells about how, when she was in primary school learning arithmetic, to multiply a number by 48 she would first multiply it by 50, then multiply it by 2, and then find the difference. Had I (my current self) been her teacher I would have praised her to the skies. She was thinking mathematically, with both imagination and rigor. Her teacher, however, told her that she was wrong. She should follow the mechanical procedure preached by the textbook. (Multiply by 8, then by 4, writing the result of the latter operation one space to the left of the result of the first, and then add the two results.)

Now consider the movie, “Stand and Deliver”. Not to disparage Escalante’s accomplishments in real life, but what the movie showed of his high school math teaching was drilling his students in mechanical operations to solve certain types of problems. Such drill is teaching to the test. It enables the test taker to rapidly solve certain problems automatically. Getting a good test score may help a minority kid get into college and get a scholarship, and teaching to the test can serve that end. However, what the teacher in the movie was doing was little different from what my girlfriend’s primary school teacher, who obviously did not understand math, was doing: teaching a set of mechanical operations. That has little to do with teaching math, with teaching students to think with both imagination and rigor.

Philosophers will note the similarity to Searle’s Chinese Room. Just because someone can follow a prescribed sequence of operations does not show that they understand what they are doing. Teaching to the test can be the enemy of education.

173

Salient 10.20.10 at 4:28 pm

I agree that it’s a much more effective way of assaying learning than asking them to spit out rote-memorized factoids.

Yep. And your example of “here’s a sample situation, analyze it” is basically the way to go about it, in all fields. English is kicking our arses for that reason; giving students a new story passage and asking them reading-comp questions about it is standard operating procedure. In maths-n-sciences it seems surprisingly controversial, because [exactly as you say] we’re not doing that in the K-12 environment, so “digesting new information in real-time” is not a skill that is being taught.

That, I would think, is a shame and a huge problem, which seems under-mentioned in the popular discussions of these things. Even a kindergartener should be expected to make real-time extensions of learned material to new material. I see lots of great HS teachers doing that in formative assessment, but far fewer HS teachers requiring that in summative assessment — and the teachers most likely to include that in summative assessment, tend to be teachers teaching the high-level courses. Students in the mainstream are sorely underserved in that regard.

And the last paragraph is a different question of values altogether.

Uhh, yeah, and values are important to this discussion.

I’d still be thrilled to find out how “teaching to the test” like the Florida test example given above is likely to cause problems in learning basic math, and how it would be unable to detect basic math competency.

Well, for one thing, when someone says “teach to the test” this usually does not mean “teach the material that the test is testing.” It means, in part, “teach tricks that allow students to answer questions correctly, or improve their guesses at more questions’ answers, without understanding the material.”

By definition of what the phrase means and how it is used, teaching to the test is not teaching course content.

If what you’re saying is: develop tests that are comprehensive and thorough, and which also introduce new and unannounced extensions of core course material for students to digest in real-time, and then organize the course so that successful completion of the course will prepare students to succeed in that environment… well, that’s a rephrasing of what education with summative assessment basically already is. You’re just describing “Teach stuff and then assess learning” is not new. You’re just recapitulating a core tenet of the uncontroversial Understanding by Design course-design template. I would “let me google that for you” Understanding by Design, but I am trying to conscientiously reduce how sarcastic and bitter I come across on the Internets, and maybe you’re not aware that you are broadly suggesting teachers should do what they’re already doing and mostly have been doing since time immemorial.

Either you’re advocating that we teach test-exploitation strategies [I take this as prima facie bad] or you’re advocating that we teach material and then assess learning [not a new statement, therefore not worth hammering on about]. To get better responses from us, how about you articulate a little more specifically what you’re advocating?

174

billikin 10.20.10 at 4:29 pm

Correction:

I wrote, “writing the result of the latter operation one space to the left of the result of the first”

Better: “writing the result of the latter operation under the result of the first and shifted one place to the left”

175

SamChevre 10.20.10 at 5:44 pm

Either you’re advocating that we teach test-exploitation strategies [I take this as prima facie bad] or you’re advocating that we teach material and then assess learning [not a new statement, therefore not worth hammering on about].

I can’t speak for Sebastian, but I’m advocating the second. And I think it is worth “hammering on about”, because currently, it isn’t being done in a way that makes subject mastery comparable even between teachers in the same school, much less between schools, in the American public school system.

176

Salient 10.20.10 at 8:13 pm

…currently, it isn’t being done in a way that makes subject mastery comparable even between teachers in the same school, much less between schools, in the American public school system.

I’m amenable to that criticism — though the stuff I’ve read on grading mostly comments on how remarkably and counter-intuitively homogeneous grades are across schools, states, etc.

Note, though, you indeed aren’t speaking for Sebastian, and are much more in agreement with stuff I said upthread, which diverges from Sebastian: his argument runs a bit more along the lines of, the existing testing system is doing a sufficiently good job of differentiating weak teachers from strong ones, so we should destroy teachers’ unions and then use existing testing data [or slight modifications thereof] to weed out bad teachers. These bad teachers are already [in Sebastian’s formulation] identified and known to be bad, because of existing testing data, it’s somehow already obvious that they don’t belong [or would become immediately obvious if we only gave their students the right test], and all that stands between us and school-improvement-by-firing-the-bad-guys is a union that prevents the firing.

Cf. stuff like this:

My major criticism of the “teaching to the test” criticism is that while it may be an excellent point for certain disciplines … teaching to even a moderately good math test is teaching math. For the levels of reading that we are talking about, teaching to the reading test is teaching reading. Teaching the test for those kind of things, at that basic a level, isn’t likely to be bad.

Most of the reading teaching-to-the-test that occurred at the high school where I taught at, I would classify as teaching how to avoid reading and answer questions correctly, by exploiting deficiencies in the test format. The teachers of those workshops readily agreed. One said, “teach the players to play the game.” And smiled wanly at me, as if to say: ain’t it a crazy world?^1^

Or this:

I’m advocating testing (which may or may not include scantrons per se) at the end, presumably after the teaching/learning has occurred, to verify that it has occurred. That is what teacher’s unions in the United States are against.

Well, no. They’re not. Not at all. They’re likely to be against the use of poorly designed and easily exploited tests for this purpose. They’re also likely to protest the use of that test data as a sole or predominant basis for firing teachers, for reasons Sebastian continues to elide.

I’ll note that Sebastian got into a huge argument upthread with individuals who pointed out that the currently-in-practice Scantron tests are not effective at checking for mathematical facility. Sebastian responded that at a sufficiently basic level, a bar set sufficiently low, these tests should be able

But they don’t. For example, students get fed up with this crap and don’t bother to perform on the low-bar tests. Kids are, well, rational like that. Who wants to learn how to quasi-read, e.g. skimming for frequency of name appearance to identify the name of a story’s protagonist?^2^

Anyway, this afternoon, I took the time to call two teacher’s union representatives (who happen to be acquaintances of mine) and asked them about this, word for word, off the record. One laughed and mentioned that no the union was not anti-final-exam, the other sighed and said “that’s silly.” This is not an accurate representation of a teacher’s union position, and I defy Sebastian to provide any quotation from any union representative speaking in their official capacity which states, to wit, that the union is against testing students at the end of a semester to verify that learning occurred.

For all that Sebastian may be a nice guy, he puts a hell of a lot of crazy words in teachers’ unions’ mouths.

^1^This was actually the moment where I understood concretely what a ‘wan smile’ looks like.

^2^An actual technique which “teach to the test” teaching teaches… It amazed me how many high school students could answer their reading questions with reasonable accuracy without being able to even summarize for me what the story was about.

Oh god I’m being long on the internet again. I’d best clam up.

177

SamChevre 10.20.10 at 8:49 pm

It seems like there are two very different questions being discussed, with extensive attendant confusion.

Sebastian Teaching to even a moderately good math test is teaching math.

Piglet Maybe but are the math tests “moderately good”? I contend that if they are designed to be easily gradable (i. e. multiple choice), they are not good tests.

I’m arguing that piglet is wrong, and that it is entirely possible to make moderately-good math tests (such that “teaching to the test” requires teaching math, not test-taking tricks) that are multiple-choice tests.

I am not arguing that currently-used tests (here in VA, the SOL’s) are moderately good. They aren’t. They’re extremely gameable, and don’t test core knowledge well.

I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.

178

Barry 10.20.10 at 9:28 pm

dsquared 10.20.10 at 11:52 am

“My understanding is that when pressed on the subject of where it is that they plan to hire all these excellent teachers from, given that the ones we had are the ones who showed up at the wage and conditions offered, people retreat to arguments about “incentives” – ie, that the existing stock of teachers will be motivated by fear.”

Positive incentives are the only thing which will spur the elite 1% on to GGP (Greater Galtian Production).

For the rest of us, it’s the lash until morale improves.

179

Steve LaBonne 10.20.10 at 9:46 pm

I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.

Define “high-stakes”. Be sure to specify as part of your definition whether any attempt will be made to measure value added, rather than punishing any teacher whose kids start out behind at the beginning of the year. Also be sure to indicate whether you think tests score are the ONLY measure that should be used to judge teacher performance. Then we can begin assessing the truth value of this claim and the intentions (and level of intellectual competence) behind it.

180

ScentOfViolets 10.20.10 at 10:16 pm

Why not ask for a reasonably fine-grained analysis tool? Math ‘proficiency’ for a future engineer or physicist isn’t the same as math ‘proficiency’ for a future restaurant owner or nurse, which isn’t the same as ‘proficiency’ for a future foreman.

Just to make sure that we’re all on the same page here: I don’t think anyone is against a good test at the end of the semester to find out how much the kids have learned and how proficient they are in various subtopics.

The problem is that word “good” and multiple- choice format doesn’t fit that definition; what multiple choice can do (and is used for sometimes) is weed out the obviously unsuitable. And while colleges processing thousands or tens of thousands of applicants might find such a test useful, they aren’t using it to determine proficiency.

No, the “good” tests are going to cost you some money and manpower to administer. If you’re serious about determining proficiency that is.

181

ScentOfViolets 10.20.10 at 10:21 pm

I’m arguing that piglet is wrong, and that it is entirely possible to make moderately-good math tests (such that “teaching to the test” requires teaching math, not test-taking tricks) that are multiple-choice tests.

And what is the basis for this “argument”? Because you are not arguing, you’re just stating your own opinion with nothing to back it up; no facts, no research, no cites, not even a bare-bones reason. Just a flat statement.

Have you taught math classes before? Do you have experience teaching math, and if so, what was it, and why do the math teachers here seem to be rather uniformly against the multiple choice format?

182

christian h. 10.20.10 at 10:26 pm

The union busters speak about incentives. But then they magically forget about it and ignore the patently obvious: punishing teachers for bad test scores (or bad VAM scores which are as useless from a statistical point of view) provides incentives to teach test taking skills and tricks, or even cheat. If you look at that Florida test (btw nice try to post a full test and manual and then demand test critics point out all the flaws…. in a blog comment) for example you’ll notice the hardest part of it isn’t any of the math – it’s the instructions how to mark the answer sheets. Since test scores are non-scalable (one of the reasons VAM is nonsense) a teacher threatened with lay-off or salary cuts due to bad test scores will be very well advised to prioritize class time as follows:

1. Drill kids on the mechanics of test taking – no math taught.

2. Drill kids on test taking tricks that will improve the very worst performers to not quite as bad.

3. Drill kids on very specific types of questions likely to come up (this is how you get perfect GRE scores, trust me on that).

4. If any time’s left, teach some math.

This priority order is almost independent of test design. It is a function of (a) the attempt to evaluate and reward/ punish teachers (and administrators for that matter) based on the test design and (b) the economic necessity to design such high stakes test in a way that is inexpensive to grade.

183

christian h. 10.20.10 at 10:28 pm

Let’s not miss the point. The goal of testing, for the union busters, is NOT to assess student learning. It is to evaluate teacher performance. Even the best-designed tests will buckle under the strain of being used for the latter while pretending to do the former.

184

Salient 10.20.10 at 11:21 pm

I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.

What Steve said.

Also: I have seen no evidence that firefighters would be happy with high-stakes testing of their response times if only the tests were randomly distributed throughout the city. (Some stations take longer because the average house is further away? Tough. Innovate something.)

Of course, my not seeing this does not imply ‘firefighter unions are objectively anti-response-time-testing; ergo, we should abolish them,’ or even anything approaching that.

185

Salient 10.20.10 at 11:26 pm

why do the math teachers here seem to be rather uniformly against the multiple choice format?

At least in part: Because you’re here.

I haven’t actually seen all that many people coming out vociferously against the College Board’s AP exams or the GRE subject test, here, except for you, SoV. Some of us have been staying mum on that particular issue because we’ve learned not to provoke your ire.

I just finished grading about 500 multiple-choice tests.^1^ In each case, students were required to show their work on the exam for each problem, then report the answer requested.

Although each problem required about half a page of handwritten computation, I received almost zero extra information from looking at the students’ work, just frequent confirmation of what I thought each available wrong answer on the test would indicate. So-and-so deeply goofed and thought the product formula was (fg)'(x) = f'(x)g'(x), this other so-and-so made a minus sign error, etc.

As you know, after a few years of teaching the same stuff, a great majority of errors are predictable, and tests can be written so that [a] the predictable errors always result in a wrong answer, [b] each predictable wrong answer is made available to the student, and [c] “none of the above” can be made always available, in case students err in unexpected ways.

If students double-check their work because their answer is unlisted, and thereby locate an error in their work, I’m okay with that happening. I want them double-checking every response anyway.

On the other hand, I don’t particularly care if they were careless with minus signs or didn’t know the formula or what, though I can track the difference because a wrong answer from an minus-sign error at a key step was always available. The idea that I should award partial credit for a wrong answer because ‘at least the student knew the formula’ bothers me. I give them enough more than enough time to finish and check their work carefully, and then expect perfect responses: I expect accurate knowledge, correct application of that knowledge, and careful procedural work.

Incorrect reported answers are marked -1, unanswered questions -0, correct answers +2. Students have been told repeatedly that guesses with little to no work shown will be marked -5 instead of -1, even if correct, so nobody took a random stab at any question. This eliminated possible statistical noise from happening to guess right. There are probably ways I could automate this verification, if I bothered to think of them.

Now, I don’t really feel like being harassed to the ends of the earth by you, SoV, so kindly either don’t reply or at least reply civilly. Regardless, acting aggressively to commenters who disagree with you over multiple threads, and then drawing conclusions from the lack of disagreement with you later on, is problematic at best, and I see no reason to let it stand unprotested here.

^1^Sure, I could ask more theoretical essay-response questions in exams which can’t be assessed via multiple choice, but am currently following the department’s advised guidelines very strictly, which I don’t have a say in. I’m a graduate student, not a tenured professor. (Regardless, my exams are known for being damn hard relative to the norm, and my RateMyProfessor ranking & comments both reflect that.) Besides, I ask those questions in quizzes, which can be quickly graded 0/1/2 in a light read. Having read over old Calc 1 exams from a wide variety of universities as made available on their professors’ web pages, it seems to me that the content assessed and well-assessed by my exams pretty much matches the content they assess, assuming they never award partial credit for an incomplete or erroneous response.

186

Salient 10.20.10 at 11:32 pm

I mangled that last sentence when editing it. Meant: Having read over old Calc 1 exams from a wide variety of universities as made available on their professors’ web pages, it seems to me that the content assessed and well-assessed by my exams pretty much matches the content they assess, assuming they do award partial credit for an incomplete or erroneous-due-to-trivial-error response.

187

ScentOfViolets 10.20.10 at 11:50 pm

Now, I don’t really feel like being harassed to the ends of the earth by you, SoV, so kindly either don’t reply or at least reply civilly. Regardless, acting aggressively to commenters who disagree with you over multiple threads, and then drawing conclusions from the lack of disagreement with you later on, is problematic at best, and I see no reason to let it stand unprotested here.

Translation: After I’ve been rude and uncivil, I don’t want you to follow me in kind.

What a tool. And as for “following to the ends of the Earth” . . . just what the hell are you smoking? What’s with the ‘tude anyway? Looking up above, I don’t see anywhere I’ve unloaded on you either; but you seem to think that’s the case, so why don’t you point it out? Further:

I haven’t actually seen all that many people coming out vociferously against the College Board’s AP exams or the GRE subject test, here, except for you, SoV.

Really? Let’s rewind the tape:

If standardized tests resembled the AP exams, I’d be convinced of their worth. But they don’t, and so I’m not.

BINGO!

That bingo was me. Doesn’t look like I’m exactly dead set against those exams now, does it?

So, do you have something meek and sweet to say to me? Like maybe an apology?

188

Sebastian 10.21.10 at 12:21 am

Wow. A teacher who thinks that multiple choice tests can work. With reasons. And SoV can’t respond to the argument, but rather just sputters incoherently. I’m shocked..

Salient, I said (about your suggestion that teaching 10 more students fractions might not be worth it if it convinces 2 students that they don’t want to go on to calculus) “And the last paragraph is a different question of values altogether.”

And you replied:
“Uhh, yeah, and values are important to this discussion.”

To be clear I was trying to be dismissive of that particular value one way or the other. I just figured it was so much a different topic, and so likely to spin out the conversation in a totally different direction that I didn’t want to get bogged down in it. We’re having enough trouble with simpler topics.

You also write: “Either you’re advocating that we teach test-exploitation strategies [I take this as prima facie bad] or you’re advocating that we teach material and then assess learning [not a new statement, therefore not worth hammering on about].”

I’m advocating that we teach material and assess learning. And of course it isn’t a new statement, schools do it all the time. The problem is that there is this huge argument *right here* that has as its basic premise that of course we can’t do that. And I’ve been told repeatedly that multiple choice tests of course can’t possibly do that. And that they can’t *even identify basic competency* much less identify a couple of levels of competency. I’ve been harassed up and down for the temerity of suggesting that basic competency testing is possible. (And for the double horror of suggesting that some of the actual tests actually do it). The reason I am focusing on math is because it seems the most obvious case where very clear competency testing can be done, and is done. And we can’t even get past that part of the conversation. But to be super clear, yes teach, then assess, then analyze what is working and what isn’t, then assess again.

“Anyway, this afternoon, I took the time to call two teacher’s union representatives (who happen to be acquaintances of mine) and asked them about this, word for word, off the record. One laughed and mentioned that no the union was not anti-final-exam, the other sighed and said “that’s silly.” This is not an accurate representation of a teacher’s union position, and I defy Sebastian to provide any quotation from any union representative speaking in their official capacity which states, to wit, that the union is against testing students at the end of a semester to verify that learning occurred.”

I’ll be happy to look for teacher’s union quotes in just a moment. But in that vein, can you show me the testing assessments that teacher’s unions support? Ones that actually test for completed learning and demonstrated competence? Because I’ve never seen that support. (And according to SoV I haven’t seen it because it can’t exist).

189

christian h. 10.21.10 at 1:44 am

Couple points:

(a) The GRE, and its subject tests, are useless. It’s a form of rent extraction. when I am on the committee doing graduate admissions I never once look at the scores. The only function it properly has is to eliminate the very worst candidates – and it is superfluous for that purpose. So what remains is that it forces students to pony up hundreds of dollars for nonsense.

(b) Sebastian is being dishonest. He still pretends that “assessing learning” is the same as “evaluating teacher performance”. This allows him to make the unfounded claim that teachers’ union object to assessing student learning.

(c) I do manifestly not agree that multiple choice tests are a good way to assess mathematics learning. They are a cheap way. Not at all the same thing.

(d) The whole idea that you learn for a year, then write one 4 hour test to determine what was learned is laughable beyond belief. The fact that anyone will actually defend this idea, when it is patently absurd, mystifies me.

190

Peter Nunns 10.21.10 at 1:45 am

Sebastian@189:

Wow. A teacher who thinks that multiple choice tests can work.

If you had actually bothered to read what we have been saying, we agree that standardized testing is possible, and we cited an example of a test that is good: The AP tests administered by the College Board.

I’ve been harassed up and down for the temerity of suggesting that basic competency testing is possible.

You’re incorrect about this, and attempting to play the victim to boot. Again, we agree that testing is possible. We do not agree that the type of testing being practiced under NCLB etc is adequate to the task. As we have repeatedly said, the actual tests in use bear very little relation to curricula (unlike the AP exams) and consequently have a tendency to divert class time away from course aims. This is the problem. Good tests – standardized or not – will support curricula. Bad tests undermine them, and in the real world – which you can’t seem to be bothered with as evidenced by the fact that you don’t bother to talk to teachers or seek out relevant research – they are undermining them.

The reason I am focusing on math is because it seems the most obvious case where very clear competency testing can be done, and is done.

A more intellectually honest argument in favor of standardized competency testing would engage not with the easiest case but with the hardest. If you really cared about the subject, you’d be trying to make the case for testing in, say, English or biology. But that would be difficult, so you’ve chosen not to.

191

Ebenezer Scrooge 10.21.10 at 2:24 am

Salient@185:
Don’t be a wimp. We’ve all been assaulted by SoV for our patent disregard of logic and bad faith and rudeness and provably incorrect assumptions. Just say what you think is right, in terms that will appeal to our lesser minds, if not the great mind of SoV. And then ignore the troll.

192

Sebastian 10.21.10 at 3:03 am

“A more intellectually honest argument in favor of standardized competency testing would engage not with the easiest case but with the hardest. If you really cared about the subject, you’d be trying to make the case for testing in, say, English or biology. But that would be difficult, so you’ve chosen not to.”

Ummm no. If you can’t be convinced in math, I really can’t be bothered to argue it because that shows that you aren’t really listening.

193

Sebastian 10.21.10 at 3:21 am

“As we have repeatedly said, the actual tests in use bear very little relation to curricula (unlike the AP exams) and consequently have a tendency to divert class time away from course aims. ”

Yes that is the *claim*. But the tests we actually see, (see linked above), don’t actually seem to be such a problem. It looks to me like it tests actual math understanding relatively well for a basic competency level. I understand that the teacher’s union claim that the tests are so horrifically awful. I just don’t see any evidence of that.

And actually I’m not even sure I’ve heard the the majority of the NCLB *tests* are bad, just that the 100% pass crap was problematic. Which is a perfectly legitimate gripe about what you do with the test, but not the test itself.

194

sg 10.21.10 at 7:14 am

Sebastian, I have a question for you. I teach statistics and public health to undergraduates, and at the beginning of every lecture series they always ask me whether the exam will be multiple choice or “an actual exam.” when I tell them “an actual exam,” they always look profoundly disappointed, even more so when I tell them this means they’ll have to memorize stuff.

Can you tell me why my students prefer multiple choice? Do you think it’s because they think it provides a more accurate assessment of how well they have learnt the material, or do you think it’s because they realize multiple choice exams are easier to pass?

And on the topic of multiple choice, do you accept that maybe it’s a lot easier for an academic, usually untrained in teaching theory per se, to set an exam with written content that correctly judges what the student has learnt, than to set a multiple choice exam with the same goal?

It is, after all, very easy to set 4 answers to a question, two of which are obviously false, and mistakenly make it very easy for students with limited grasp of the course to rule out the false ones.

Finally I should point out something very obvious: when testing pictographic languages, no multiple choice test will ever work, because it tests your ability to read but not to write.

Do you really think this kind of trickery only exists in those language tests, or do you accept it might possibly be a little more universal?

195

sg 10.21.10 at 7:15 am

And on a related note, does anyone here actually know of a case where busting a union improved quality in an industry?

196

Peter Nunns 10.21.10 at 7:22 am

Sebastian @192:

“A more intellectually honest argument in favor of standardized competency testing would engage not with the easiest case but with the hardest. If you really cared about the subject, you’d be trying to make the case for testing in, say, English or biology. But that would be difficult, so you’ve chosen not to.”

Ummm no. If you can’t be convinced in math, I really can’t be bothered to argue it because that shows that you aren’t really listening.

Yes, but you’re not arguing that only math teachers should be assessed on the basis of their students’ performance on standardized tests. You’re arguing that all teachers should be evaluated thusly.

Let me make it clear: this is dishonest. You’re actually practicing a form of logical fallacy. It goes like this:
X is a member of Y.
For all X, Z.
Therefore, for all Y, Z.

See how that works? You’re basically saying:
Math teachers are teachers.
Math teachers can be evaluated effectively by their students’ performance on a standardized exam.
Therefore, all teachers can be evaluated effectively by their students’ performance on a standardized exam.

But logic doesn’t work like that. Engage with the hard cases, or GTFO.

197

billikin 10.21.10 at 1:36 pm

What do you mean, fix American schools? If it ain’t broke, don’t fix it.

http://scienceblogs.com/mikethemadbiologist/2010/10/i_agree_with_bob_somerbys_educ.php#more

It seems like U. S. schools have been doing a pretty good job, over the years. :)

198

Salient 10.21.10 at 1:50 pm

Engage with the hard cases, or GTFO.

Nah, I’m not ok with that, for one. If math teachers could be evaluated and selected based entirely on their students’ performance on a test, then it would seem reasonable to propose we apply such a policy to math teachers, regardless of whether one extends it to other teachers. So Sebastian, if he argued persuasively for that and only that, would still be contributing to the discussion.

But I’m not willing to concede that point about evaluation of math teachers. I may disagree [mildly] with christian h. about his point {d} — after all, folks regularly have two-hour exams count for 50% of a student’s grade or more in 100-level courses, even if one can’t test everything, it seems it must be possible to test quite a lot, if we are willing to allow one test to stand in for 40% or even 55% of a student’s grade in a course — but I completely agree that {b} is a dangerously misleading shift of focus.

[As an aside, (a) is good to know. Thanks for mentioning it.]

199

Jerry Vinokurov 10.21.10 at 2:37 pm

The reason that multiple-choice tests don’t actually do a good job of testing whether people know something is because they don’t allow you access to the reasoning process. Certainly at the college level, no self-respecting teacher would administer one.

Salient:

On the other hand, I don’t particularly care if they were careless with minus signs or didn’t know the formula or what, though I can track the difference because a wrong answer from an minus-sign error at a key step was always available. The idea that I should award partial credit for a wrong answer because ‘at least the student knew the formula’ bothers me.

I don’t get this at all. Why would this bother you, exactly? It certainly didn’t bother any of my professors if I made a sign error somewhere. Yes, attention to detail is certainly important and one should always try to get the right answer; however, someone may simply make a mechanical mistake, whether it’s missing a factor of 2 or writing down a sign wrong or whatever. I think of that the same way I think of a spelling mistake on a writing test: unfortunate, but not so far as to undermine the rest of the work. This of course is the fundamental problem of multiple-choice tests: they don’t distinguish between someone who made a mechanical error and someone who just guessed (or, as happened in my case, someone who literally made a mistake filling out the Scantron).

As for the GRE, as christian h. mentioned above, it is nothing more than rent-seeking, a method of extorting money from a captive population. As far as I’m concerned, ETS ought to be one of the first organizations up against the wall when the revolution comes.

200

Steve LaBonne 10.21.10 at 2:40 pm

As far as I’m concerned, ETS ought to be one of the first organizations up against the wall when the revolution comes.

As one who recently had to go through the college-admissions zoo with my daughter, I’d like to reserve a place on the firing squad.

201

Steve LaBonne 10.21.10 at 2:49 pm

I note with interest that SamChevre has time to blow nonsense about climate science out his ass on another thread, but no time to respond to my #179 here. The conclusion is left as an exercise for the reader.

202

SamChevre 10.21.10 at 3:00 pm

Well, I’ve said nothing about evaluating teachers in the entire thread. I’m interested in evaluating student learning, and don’t have any strong interest in how to evaluate teachers given student learning.

203

ScentOfViolets 10.21.10 at 3:02 pm

Yes, but you’re not arguing that only math teachers should be assessed on the basis of their students’ performance on standardized tests. You’re arguing that all teachers should be evaluated thusly.

Heh. Sebastian hasn’t shown that about multiple choice testing for math though. He’s done his usual:

a) Made a claim with no evidence or reasoning to back it up

b) Dared others to prove him wrong

c) Asserted that the people who challenge his claims “haven’t convinced him”.

d) Declared victory and accused others of being blindly partisan for not agreeing with him.

In any of the classes I teach, that sort of logic would earn him a straight F. To go along with what sg says above I’ll point out that not only do students seem to be disappointed to learn that tests won’t be multiple choice, but that since multiple choice is much easier to grade, math teachers pointing out their lack of utility in most circumstances aren’t doing themselves any favors.

Let me give yet another concrete example: we’ll be having an exam next week where I’ll give the students a rational function (a polynomial of m degree over a polynomial of n degree) and then ask them to graph it, showing x- and y-intercepts, vertical, horizontal and oblique asymptotes, etc. Now, I know from past experience that if I give the same rational function and then ask one multiple choice question at a time what are the x-intercepts, what are the y-intercepts, etc, the students tend to do much better than in open format. Why is that? Is the multiple choice format more accurate in assessing the student’s level of knowledge on curve sketching in this instance? Or the open format? Speaking as a math guy who’s administering the test, I’m going to go with the latter option. If Sebastian wants to he can try to “convince me otherwise” ;-)

204

Steve LaBonne 10.21.10 at 3:05 pm

Well, I’ve said nothing about evaluating teachers in the entire thread.

You made a specific claim @177 about the attitude of teacher’s unions to evaluating teachers. I will take this evasion as admission that you were lying. Thank you for clearing that up.

205

ScentOfViolets 10.21.10 at 3:07 pm

I don’t get this at all. Why would this bother you, exactly? It certainly didn’t bother any of my professors if I made a sign error somewhere. Yes, attention to detail is certainly important and one should always try to get the right answer; however, someone may simply make a mechanical mistake, whether it’s missing a factor of 2 or writing down a sign wrong or whatever.

Salient seems to be a much harder grader than other math teachers who have opined on this one. In fact, I do give credit for “having the right idea”, and I tend not to grade overly hard on sign mistakes, improperly done elementary options like subtraction, etc. And this is for the excellent reason that I not only teach math, I do math, and I make those kinds of stupid mistakes myself all the time. I’d like to think I don’t make them because of insufficient mastery of the concept :-) The difference is, I have all the time in the world to catch those mistakes and correct them. The kids taking my exam? They’ve got just 60 minutes.

206

SamChevre 10.21.10 at 3:13 pm

Huh?

I said “I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.”

From the SOL website. “For high school students, the scenario in Virginia is much different, as Standards of Learning tests are required for graduation and thus become high-stakes testing. ”

I’m accustomed to seeing “high-stakes testing” used in this context–as tests required for graduation or advancement; I’m not sure where evaluating teachers comes into this.

207

ScentOfViolets 10.21.10 at 3:16 pm

This of course is the fundamental problem of multiple-choice tests: they don’t distinguish between someone who made a mechanical error and someone who just guessed (or, as happened in my case, someone who literally made a mistake filling out the Scantron).

As I said above, I continually remind my students that an 80% correct answer in a multiple choice format is wrong.

As for the GRE, as christian h. mentioned above, it is nothing more than rent-seeking, a method of extorting money from a captive population. As far as I’m concerned, ETS ought to be one of the first organizations up against the wall when the revolution comes.

To be fair, these sorts of tests seem to be used to disqualify prospective students as opposed to qualifying them. To go with yet another analogy, on “House” it’s very easy to run a test that shows the patient isn’t suffering from, say, heavy metal poisoning. But it takes most of the show to figure out exactly what the patient does have.

208

Steve LaBonne 10.21.10 at 3:22 pm

I said “I have seen no evidence that teacher’s unions would be happy with high-stakes testing if only the tests were better designed.”

And you were asked specific questions about specific BAD testing practices that any sensible educator or union SHOULD oppose, and you were asked for evidence that if testing were done properly it would still be reflexively opposed (anybody can go to, say, the AFT’s site and see that the answer is “no”,) AND about what YOU meant by “high-stakes”, specifically in terms of whether you believe it’s appropriate to use ONLY test scores in evaluating teachers. You still refuse to address any of those questions but instead continue with childish evasive tactics.

You’re a dishonest coward who loves to make hit-and run unsupported claims. Typical wingnut. Quite like the behavior of Clive Crook documented in the OP to the other thread that you’re currently polluting.

209

SamChevre 10.21.10 at 3:38 pm

OK, let’s me try answering (again) the question of what I mean by high-stakes testing.

I mean what the VEA uses it to mean–testing that is required for promotion (to another grade) or for graduation. This is a student-focused definition.

I’m not talking about evaluating teachers, I have nowhere in this thread been talking about evaluating teachers, and I’m not interested in talking about how to evaluate teachers.

210

Steve LaBonne 10.21.10 at 3:43 pm

Well, then, now support your claim that teacher’s unions oppose the use of properly designed tests for that promotion and graduation. That’s going to be rather difficult, since it took me all of a minute to find the following: In 1999 Texas enacted legislation, which the Texas Federation of Teachers initiated, that made passing the third-grade reading test a requirement for promotion to fourth grade.

So, are you an ignoramus or a liar? (Not mutually exclusive catergories, of course.)

211

Salient 10.21.10 at 4:14 pm

Certainly at the college level, no self-respecting teacher would administer one.

This is bullshit, spoken by a person who either hasn’t spent much time coursing through universities’ old exams for introductory classes, or is provocatively calling a rather large portion of the field self-respect-free.

The kids taking my exam? They’ve got just 60 minutes.

Well, hell, strict time constraints will do more to inhibit reasoning than the way that final answers are reported. I have evening exams with a soft limit of 2 hours and a shared understanding that students can have some more time if they feel they need it. About two-thirds turn the thing in ~45 minutes early, and say they had more than enough time to review and check their work for errors — it’s an “hour exam” with time allotted. Increasing demand for precision needs to correspond to relaxing the time constraint, definitely.

But for example, some students mix up the quotient rule, effectively getting it backwards. The final result from this forgetfulness is a minus sign error, sure. However, a student who is thinking carefully about the problem, and checked their work by viewing the function on a graphing calculator, ought to realize the sign of their answer must be incorrect. If they don’t, that’s either a conceptual deficiency (not understanding that an increasing function has positive slope) or maybe a failure to carefully check work. I want my students to be mindful enough to engage in that sort of work-checking, and I model how to do that in class. Seems fair to expect it on tests, and my average scores etc do seem very comparable to grade distributions elsewhere.

On the other hand, weird random minus sign errors that are unrelated to formula mismatch or some such thing usually result in an answer that’s not available on the answer form. The student then (hopefully) knows to review and revise their work, (hopefully) catching the error in a second pass.

I do give credit for “having the right idea”

I protest vociferously the notion that ‘memorizing the formula accurately’ or even ‘knowing which formula applies’ is the same thing as, or is even indicative of, “having the right idea.”

But this is getting a bit far afield from the topic of union-busting, and I do happen to have very limited control over my testing format. I just think there’s a bit too much hyperbole being dumped on multiple-choice answer sheets, as if the moment you let someone compare their final answer to a list of possible final answers all hell breaks loose.

212

ScentOfViolets 10.21.10 at 4:24 pm

Salient, you owe me an apology. Either give it or stop addressing me.

213

Sebastian 10.21.10 at 4:33 pm

SG: “Can you tell me why my students prefer multiple choice? Do you think it’s because they think it provides a more accurate assessment of how well they have learnt the material, or do you think it’s because they realize multiple choice exams are easier to pass?

And on the topic of multiple choice, do you accept that maybe it’s a lot easier for an academic, usually untrained in teaching theory per se, to set an exam with written content that correctly judges what the student has learnt, than to set a multiple choice exam with the same goal?”

My problem with this argument is that you’re arguing in the opposite direction from what a competency test measures and why teachers unions appear to be so scared of them.

You are arguing that multiple choice tests are too easy. But when teacher’s unions reject linkage between student performance on exit tests and teacher’s performance (even when tracked over many years, and even when paired with entrance exams to show the value added to performance over the year so that a teacher doesn’t get hit with the fact that they just got a bunch of students who were behind) they aren’t doing so because they worry that the test is too easy. They aren’t worried that too many students will pass the competency test when they should be failing. They aren’t worried that too many bad teachers are actually going to get good grades because their students will overperform.

Your objections are the exact opposite of what they are worried about.

As for test design, I’m not sure what you’re asking from me. I don’t think I’m suggesting that every single teacher should individually design a state or national competency test. The fact that most academics would have trouble designing a good one doesn’t imply that the actual people designing the test can’t do it.

So far as I can tell, the problem isn’t that we are having too many kids passing competency EXAMS when they don’t have real competence. The problem we have is that too many kids pass years and years of *teacher assessments*, not controlled by outside assessments, and then at the end graduate from high school without the ability to do even basic math or basic reading.

But even on a theoretical basis, your objection falls on the high end of achievers. Yes, multiple choice tests have trouble appropriately discerning between the students who have very good understanding of the topic, and those who have an even more deep mastery of the field. Yes, for *individuals* you might get one person who is slightly better than another look worse because the slightly better one got tripped up in an odd way. But over populations of students, especially over multiple years, and especially since you are attempting to measure general floor-level competency and not trying to finely discriminate between different varieties of brilliant, your objections don’t apply.

Peter Nunns: “Math teachers are teachers.
Math teachers can be evaluated effectively by their students’ performance on a standardized exam.
Therefore, all teachers can be evaluated effectively by their students’ performance on a standardized exam.

But logic doesn’t work like that. Engage with the hard cases, or GTFO.”

Learning, teaching and understanding does work like that. When you want to debate about anything, you have to get at the building blocks. If you want to talk about calculus, you don’t do it until you can talk about arithmetic first. If you want to do trig you have to cover algebra first.

The argument has essentially boiled down to: standardized tests can’t measure basic competency (with some arguing that maybe they *could* but that they never do in practice). There are various explanations for this contention, some of them contradictory, but a lot of them came down to

it causes teaching to the test (i.e. teaching kids to pass without them being likely to get actual competence).

I said (in comment #20, so it isn’t a recent development in my thinking) “My major criticism of the “teaching to the test” criticism is that while it may be an excellent point for certain disciplines (most obviously history for example, where for a sufficiently narrow test, teaching directly to the test could lower overall perspective in the discipline), teaching to even a moderately good math test is teaching math. For the levels of reading that we are talking about, teaching to the reading test is teaching reading. Teaching the test for those kind of things, at that basic a level, isn’t likely to be bad.”

This has provoked quite a bit of disagreement.

When you disagree over the easy case (math) it makes no sense to jump right to the harder cases.

I’m not abandoning the hard cases. I’m just not willing to tackle them with people who can’t even admit that math competency (not math brilliance just basic competence) can be detected on standardized test–including ones, like linked above–that are already in use. I’m not committing a logical fallacy by focusing on the easy case first. I’m committing basic logic–talking about premise level ideas first rather than skipping them and then invariably getting muddled because we ignored premise level disagreement.

Which brings us back to the real world. The actual test that Stuart Buck linked in #128, actually used by Florida, which is actually one of the larger states, appears to be just fine. Like all tests, is it probably gameable on the extreme margins? Sure. But does it seem to require a pretty non-awful level of math understanding to get by it? Yes. Is it multiple choice? Yes. Oh no!

SoV, is as usual trying to distract by attacking the people instead of the argument. So I’ll redirect attention to the fact that you haven’t answered Salient’s points about multiple choice tests. You just attacked his character. And so far as I can tell you aren’t engaging any of my points whatsoever. I know you always play the burden of proof game. You’ve been provided an actual test, actually in use, by one of the actual largest states in the US. You haven’t even attempted to show us what is so awful about it. Are you going to avoid any actual analysis whatsoever by asserting burden of proof? I would say the premise that low level competence can be detected in testing is relatively intuitive. Especially when we can say–see here is a test that does it. Maybe, just this once, for the very first time ever since you began arguing on the internet, you have the burden of proof. Or even the burden of suggestive hinting.

214

Jerry Vinokurov 10.21.10 at 4:59 pm

This is bullshit, spoken by a person who either hasn’t spent much time coursing through universities’ old exams for introductory classes, or is provocatively calling a rather large portion of the field self-respect-free.

That was fairly provocative, but I have perspective on this issue. I was a double major in physics and math at a large public university not terribly long ago and along the way I took all the intro classes, including some in other technical disciplines like engineering. I never encountered a multiple-choice test. Not once. Then I went to grad school at a smaller private university where I was a TA for multiple intro classes (physics for med students, basically) in which I graded midterms and finals. We never gave out multiple choice exams.

I can only think of one good reason to give an MCE, and that’s if you just don’t have the resources to grade everything individually. Otherwise, I think an MCE is just an incorrect tool for assessing learning at the college level.

215

Steve LaBonne 10.21.10 at 5:07 pm

My problem with this argument is that you’re arguing in the opposite direction from what a competency test measures and why teachers unions appear to be so scared of them.

I judge you by your utterly pathetic standard of argumentation here, and say that you clearly cannot be a competent lawyer and your employer should fire you.

See any problem with that?

I can only repeat what Harry said way back at #7:

I’ve said this before, but in my own interactions with (apparently recalcitrant) teachers union leaders, they understand perfectly well that the evaluation system is stupid, and that incentives need to be put in place. They deeply, deeply, distrust the competence of administrators at the school and district levels. This reflects, in my view, their ability to see what is patently obvious, rather than any paranoia.

That really should have been the end of this particular branch of the discussion.

216

Steve LaBonne 10.21.10 at 5:13 pm

And to add to what Jerry just said, MCEs may be easier to grade, but good MC questions- or more exactly, the wrong-but-plausible answers needed to make them work- are a damn sight harder to write than essay- or problem-type exam questions. My first teaching gig was in a medical biochemistry department which gave the students FLEX-style tests to prepare them for the (now defunct) real thing, and I sweated like crazy over the questions I was assigned to write.

217

SamChevre 10.21.10 at 5:36 pm

Well, to leave this discussion agreeing with someone–good MC questions and answers are very labor-intensive to write. Good and not easily gameable is even harder.

That’s why I’d argue that they are appropriate for end-of-year assessments on a statewide level, but very poor tools for normal everyday classroom use.

218

Jerry Vinokurov 10.21.10 at 5:46 pm

But why are they even appropriate for end-of-year assessments? I mean, let’s take something like an ideal case: you want to design a test that faithfully measures what students have learned. The best case scenario that has been presented here for multiple choice tests is that they’re not entirely worthless. Let’s grant that; but isn’t it also pretty clear by comparison with actually well-designed tests (e.g. the AP tests) that such tests are not really that worthwhile? In other words, shouldn’t we be trying to design a system that does a good job of evaluating what students have learned, instead of taking the first thing that comes along and is administratively easiest, and then pegging everything to that?

Rhetorical questions, of course. It’s exactly what we should be doing, but we’re not. As such, I’m pretty skeptical of all this pro-testing hoopla. It looks to me like the educational counterpart of security theater: something is being done for the sake of showing others that we’re doing something and meanwhile no effective measures are being undertaken. If we were really serious as a country about all of this, we’d design much more thorough tests that complimented curricula (not just in math but in everything) and really try to get at the root of the problem. Instead, administrators decree from on high tests that may or may not reflect anything interesting or relevant, and whose noise seems of an order of magnitude with their data, and on the basis of that we’re going to bust teachers’ unions or whatever? Sorry, don’t buy it. From where I sit (again, not that far removed from my secondary schooling experience) none of this looks like a serious attempt to solve the problem of educational assessment.

219

Steve LaBonne 10.21.10 at 5:48 pm

It looks to me like the educational counterpart of security theater: something is being done for the sake of showing others that we’re doing something and meanwhile no effective measures are being undertaken.

Excellent analogy- I intend to shamelessly steal it.

220

ScentOfViolets 10.21.10 at 5:51 pm

And to add to what Jerry just said, MCEs may be easier to grade, but good MC questions- or more exactly, the wrong-but-plausible answers needed to make them work- are a damn sight harder to write than essay- or problem-type exam questions.

Well, here’s the problem: if I ask a student to solve 6x+17=9x-10, yeah, at that level, I can write a multiple choice question that can accurately assess what the student doesn’t know, or more accurately, what they know that ain’t so (and yes, questions very much like that one appear on the only multiple choice math exam we administer at our university, “intermediate” – read remedial – algebra.) But for something like, say, graphing x^3 -8/x^2 -4, showing all intercepts, asymptotes, end behaviour, etc, well, that is a question about basic competency, imho, but I don’t think breaking this problem down into several questions in a multiple choice format is helpful. Quite the contrary; on questions like this it is quite common for students to do better on each individual question as opposed to the open-ended question that is asked instead. As a math teacher, that tells me something about their curve sketching ability.

And of course, the converse can also be true – students can do better on open-ended questions than they do on multiple choice by virtue of receiving partial credit.

221

ScentOfViolets 10.21.10 at 5:58 pm

Rhetorical questions, of course. It’s exactly what we should be doing, but we’re not. As such, I’m pretty skeptical of all this pro-testing hoopla. It looks to me like the educational counterpart of security theater: something is being done for the sake of showing others that we’re doing something and meanwhile no effective measures are being undertaken.

Of course. Contrary to what Sebastian has mischaracterized, most (i.e., approaching 100%, at least those I know of) teachers aren’t against accurate year-end assessments. The problem is that the people who are really pushing for this want to do it on the cheap (and therefore poorly), in a way that doesn’t put any expectations on certain constituencies (like a certain type of parent), and done by people who give every indication of wanting to use the results as a club against individual teachers and their union, as opposed by the people best able to administer and interpret the results: the teacher’s themselves.

222

Salient 10.21.10 at 6:09 pm

Salient, you owe me an apology.

I owe an apology to identifying as a bully someone who once followed me from here over to Matthew Yglesias’ blog (which is, assuredly, the end of the earth) in order to continue arguing with me about whether my use of LaTeX was appropriate? …Eh, okay. Bygones. Sorry. I apologize. Now let’s move on with life.

223

bianca steele 10.21.10 at 6:10 pm

@172: My girlfriend tells about how, when she was in primary school learning arithmetic, to multiply a number by 48 she would first multiply it by 50, then multiply it by 2, and then find the difference.

Your girlfriend didn’t have these teachers.

224

ScentOfViolets 10.21.10 at 6:27 pm

Salient, you owe me an apology.

I owe an apology to identifying as a bully someone who once followed me from here over to Matthew Yglesias’ blog (which is, assuredly, the end of the earth) in order to continue arguing with me about whether my use of LaTeX was appropriate? …Eh, okay. Bygones. Sorry. I apologize. Now let’s move on with life.

!?!?!?!?! Why don’t you post those offending missives, rather than just baldly saying that this is so? I have no idea who you are or why I would “follow” you. And you’re accusing me of harassment or stalking or holding grudges or somesuch? Can you say projection? I note that you still haven’t apologized for your bad behaviour (oh, but my supposed bad behaviour excuses yours, uh-huh), nor for your nonsense about my position on the AP exams. Well, I’m not holding my breath.

225

bianca steele 10.21.10 at 6:28 pm

The version in the printed paper also included a grade 5 question asking for a proof involving arithmetic sequences.

226

Salient 10.21.10 at 7:41 pm

Why don’t you post those offending missives, rather than just baldly saying that this is so?

Because I have better things to do with my time than to provide you with evidence that you have done things that you have done. But sure, ok, google provides — wasn’t that hard to google Yglesias LaTeX ScentofViolets. IIRC, we first crossed paths here, comments 9, 10, etc. to comment 47 or so. Of course, now that I have provided exactly what reminders you’re asking for, you’ll probably want to argue about how reasonable you were and how unreasonable I was for hours. We both have better things to do.

I note that you still haven’t apologized for your bad behaviour

If you don’t accept “Sorry. I apologize. Now let’s get on with our lives” as sufficient apology on the Internets then well, there’s nothing more I’m willing to do for you. Surely the rest of CT readers don’t want to see even this much back and forth, much less a long comment going on and on in thorough apology to another.

227

piglet 10.21.10 at 7:42 pm

Salient 185, a couple points. You say that incorrect answers get a -1, no answer gets a 0, and correct answer +2. Why are you punishing the wrong tries? I don’t get this at all. Isn’t this needlessly intimidating? Many students are unsure of themselves and need positive reinforcement rather than “don’t even bother trying, you will screw up anyway”. Granted testing time may be too late to deal with this problem but the way you test is probably related to the way you teach. There has been a lot of emphasis recently on the need to encourage students to try something out, experiment, play, encourage learning from mistakes instead of turning mistakes into a demoralizing, discouraging experience. The emphasis on correct results, rather than the intellectual process, in traditional math instruction has been heavily criticized for promoting “math anxiety”. This is not to advocate credit for wrong answers but punishing wrong tries over not trying at all seems hard to justify (although I can see your point as a TA having to grade 500 tests – but what I am saying is precisely that testing needs adequate resources).

From your comment I gather that you are testing for mechanical applications of mathematical rules. Perhaps MC tests are adequate for testing this but I simply don’t agree with reducing math to a mechanical discipline. Do your students ever have to derive a result? Do they ever execute a proof? Do they learn to use mathematical tools for solving real-world problems? Maybe I am idealistic but that is what I think the essence of math instruction should be.

228

piglet 10.21.10 at 8:12 pm

SoV: “But for something like, say, graphing (a function), showing all intercepts, asymptotes, end behaviour, etc, well, that is a question about basic competency, imho, but I don’t think breaking this problem down into several questions in a multiple choice format is helpful.”

I am thinking along similar lines. You can test each of these concepts in isolation but the goal should be for students to understand the interconnected whole. I am concerned about a test format that necessitates breaking down the material into isolated bits and pieces, and the consequences for how the material is taught when “teaching to the test” becomes the overriding principle. That’s the longer explanation for my earlier statement “Maths proficiency has to be tested by doing maths”.

229

ScentOfViolets 10.21.10 at 8:50 pm

Because I have better things to do with my time than to provide you with evidence that you have done things that you have done. But sure, ok, google provides—wasn’t that hard to google Yglesias LaTeX ScentofViolets. IIRC, we first crossed paths here, comments 9, 10, etc. to comment 47 or so. Of course, now that I have provided exactly what reminders you’re asking for, you’ll probably want to argue about how reasonable you were and how unreasonable I was for hours. We both have better things to do.

Ah, that’s who you are. You are aware of course, that you’ve just disproven your rather wild allegations, right:

I owe an apology to identifying as a bully someone who once followed me from here over to Matthew Yglesias’ blog (which is, assuredly, the end of the earth) in order to continue arguing with me about whether my use of LaTeX was appropriate?

So. In both threads, opinion was running rather heavily against you, and they were completely unrelated. Btw, you do know you can go to both discussions can do a find on LaTeX, right? Let me know when you find that occurrence in both of those discussions.

Geeze, you really carry a grudge around don’t you? Next time, don’t be so quick to accuse someone of stalking or “harassing” you. Not that I’m expecting an apology of course.

230

ScentOfViolets 10.21.10 at 9:00 pm

@172: My girlfriend tells about how, when she was in primary school learning arithmetic, to multiply a number by 48 she would first multiply it by 50, then multiply it by 2, and then find the difference.

Your girlfriend didn’t have these teachers.

Yet another thing that should be taught but that we don’t have time for. Maybe it’s using calculators, maybe it’s something else, but if my students multiply, say 87 and 92, they aren’t at all surprised or skeptical of an answer like 14,332. No sense at all that the number must end in four, or that it must be less than 10,000.

Contrariwise, I had to multiply something like 25 times 37 the other day, and in about three seconds or less, came up with 925 – faster than anyone in the class could even enter the numbers. Of course, they looked at me like I was some sort of Martian, even after I explained the trick: 25 is like one quarter, and 4 quarters make a dollar. 9 times 4 is 36, so you’ve got 900 plus one 25 left over, which makes 925. Maybe I’m showing my age, but this sort of computation used to be quite common. My mother – in her 70’s – does simple multiplications in just this way (that Catholic school education she both hated and is proud of.)

231

piglet 10.21.10 at 9:02 pm

Couldn’t you guys just drop the part of your conversation that is unrelated to this thread? Don’t respond, just drop it. Please.

232

Sebastian 10.21.10 at 9:21 pm

“The best case scenario that has been presented here for multiple choice tests is that they’re not entirely worthless. Let’s grant that; but isn’t it also pretty clear by comparison with actually well-designed tests (e.g. the AP tests) that such tests are not really that worthwhile?”

Where did we get the misapprehension that the well designed AP tests aren’t multiple choice? I did a quick survey of them, and all the ones I looked at are at least half multiple choice by time, and no lower than 45% multiple choice by scoring weight. And that includes even the ‘hard to grade’ things like English, History, and Music Theory. The idea that multiple choice tests can’t form the core of good assessment isn’t supported by the so called ‘good’ tests that you’re citing.

SoV: “Well, here’s the problem: if I ask a student to solve 6x+17=9x-10, yeah, at that level, I can write a multiple choice question that can accurately assess what the student doesn’t know, or more accurately, what they know that ain’t so (and yes, questions very much like that one appear on the only multiple choice math exam we administer at our university, “intermediate” – read remedial – algebra.) ”

Jesus H Christ. What the hell do you think sixth graders are being tested on? How can you write this? You’ve been arguing against the possibility of multiple choice tests for BASIC COMPETENCY in primary school and high school and now you come up with this? Are you aware of the subject of this post?

Surely all this vitriol wasn’t just about the idea that sometimes even universities use multiple choice tests. That wasn’t even the main focus of my argument. Surely you couldn’t be ranting against me all this time across all these comments when you agreed with me all along for the tests that are the subject of the general conversation?

Really?

What the hell do you think we are trying to test sixth graders on? Do you think it is differential calculus? Do you think it is imaginary number theory? Do you think it LaPlace Transforms? We aren’t even testing them at the remedial-for-university algebra level yet. We are testing them for VERY BASIC math understanding.

I guess I shouldn’t be so furious. At least you finally admitted that for the kind of testing that is the subject of the post, multiple choice exams are in fact appropriate.

If we someday get to the point where it makes sense to test the general high school near-graduate for competence in calculus, I’ll be right there with you in wanting multiple choice to be about half the assessment.

233

Salient 10.21.10 at 9:23 pm

Granted testing time may be too late to deal with this problem but the way you test is probably related to the way you teach.

No, the way I test is mostly mandated by the department and therefore almost entirely out of my control, partly because I’ve gotten in trouble [for some mild definition of that word] for being too creative with my examinations, specifically for being too theoretically or structurally inclined and therefore too ‘hard’ on the students, by requiring proofs and derivations and the like. Ironically, my tests for those semesters were substantially more like what other folks seem to be advocating for here. But then, students complained of the course being too demanding (by comparing my tests to others’ tests which weren’t doing the things that are lauded here), and here I am, effectively demoted-without-demotion to r&g duties for the year.

Hm. Perhaps I’ve grown too reactionary in response to having been rebuffed, though. I was meditating earlier today on the fact that I probably wouldn’t be using a test format anything remotely like this, if I had control over it, and frankly would be rather likely to be joining the camp of folks critical of multiple-choice tests. (And I’d still be dazedly grading at 3am tonight.) That line about people being very good at understanding what their bosses pay them to understand, probably comes into play here against me. Strange to realize this. :-/

Anyhow, piglet – I probably agree with [or could easily come around to] the entirety of your comment, even where it disagrees with stuff I said above. Still, looking at exams for comparable courses, including those written in non-MC-format — proofs and derivations generally aren’t required here or at comparable institutions, and the real-world examples are reduced to mostly mechanical problems. I actually have access to all of the information that I would have had access to on a comparable non-MC exam, as showing work’s required, etc.

Of course, it seems several institutions call this class remedial and outsource it to a local community college, and several others don’t seem to even offer it, so there may be a bit of “we wash our hands of this” going on.

Still, the class [where it exists] is applications-oriented, set apart from the Calculus of a Single Variable sequence for students who won’t ever take another math class beyond this. The goal of the class is explicitly to put elementary calculus tools in the student’s computational toolbox. I’m okay with folks calling that ‘not doing math,’ though I wouldn’t be so harsh, but I can see some argument for offering students a computational-facility-oriented class that doesn’t delve into the theory more than superficially, and teaches them to be fluent computers for technical work.

Why are you punishing the wrong tries?

Hm. This (and what follows it) is a heartwrenchingly good point, and seems like the sort of thing I would have said myself even just a year ago. Sheesh, what kind of corner have I backed myself into? Something for me to think on further. Short cop-out answer is, I didn’t choose the test format, and while I contribute some questions to the exam in proposal form, I don’t have too much more control than that. Of course, that’s a cop-out, but I’ll leave it at that.

although I can see your point as a TA having to grade 500 tests

Technically, we only have to put effort into grading quizzes, due to automation.

234

ScentOfViolets 10.21.10 at 9:28 pm

Couldn’t you guys just drop the part of your conversation that is unrelated to this thread? Don’t respond, just drop it. Please.

Not a problem. I think we’ve all decided who’s zooming who here and nothing else will change that.

Moving on:

I am thinking along similar lines. You can test each of these concepts in isolation but the goal should be for students to understand the interconnected whole.

Exactly so. I can teach my students to do simple differentiation, and if the functions are simple enough, say polynomials, I can even get away with multiple choice questions. But if I ask my students to find the zeros of a function and the range they must lie in, well, to say I get a variety of different answers is an understatement :-) Oddly enough, those answers also tell me where they went off the rails, or where they’re just completely clueless.

235

ScentOfViolets 10.21.10 at 9:46 pm

Old business: First, how about Pax, okay? What’s been said and done is water under the bridge for both of us, whatever our individual perceptions. Are we cool?

New business:

No, the way I test is mostly mandated by the department and therefore almost entirely out of my control, partly because I’ve gotten in trouble [for some mild definition of that word] for being too creative with my examinations, specifically for being too theoretically or structurally inclined and therefore too ‘hard’ on the students, by requiring proofs and derivations and the like. Ironically, my tests for those semesters were substantially more like what other folks seem to be advocating for here. But then, students complained of the course being too demanding (by comparing my tests to others’ tests which weren’t doing the things that are lauded here), and here I am, effectively demoted-without-demotion to r&g duties for the year.

Congratualations. Yes, that is exactly what a lot of us would like to see more of. And yes, a lot of us have gotten complaints from our students about the tests being “too hard”. And here is my essential beef: the department heads and administrators will let the kids (more realistically, their parents) get away with dictating policy. So on the one hand, I can’t have too many F’s or D’s, but on the other, all of us (other teachers that is) get flack from the administration when these same students don’t do so well on standardized tracking tests. So I’m put in the position of not assigning enough homework, grading easy, and turning out high-scoring alumni.[1]

I think that’s a pretty indefensible position to be put in – especially as I and a lot of other teachers I know put in a lot of unpaid extra time just to help the students who say they can’t meet during regular office hours. Heck, I’ve come in on Saturdays, and have seen at least three other doors open. I think these kids need (and deserve) a good education. And I’m willing more than willing to devote extra time to see that this happens. But this has got to be a two-way street, you can lead a horse to water and all that.

[1]I should mention that there are several tracks. Students on the hard sciences/engineering track still tend to be pretty good students and we have a calculus series just for them, for example. On the other tracks? Well, that’s where the problems show up. I also teach business calc every so often, and you have finance students complaining that the math is “too hard”. To name but one but by no means the most egregious example.

236

Jerry Vinokurov 10.21.10 at 11:09 pm

No, the way I test is mostly mandated by the department and therefore almost entirely out of my control, partly because I’ve gotten in trouble [for some mild definition of that word] for being too creative with my examinations, specifically for being too theoretically or structurally inclined and therefore too ‘hard’ on the students, by requiring proofs and derivations and the like.

My impression is that you have been exceptionally ill-served by your department, and I’m really sorry to hear that.

237

sg 10.22.10 at 2:25 am

Sebastian, your whole answer to me at 213 seems to consist of an innuendo, suggesting as far as I can tell that teachers objections to multiple-choice tests are out of fear of being assessed themselves… is that right?

You are doing what SoV accuses you of, because it’s very clear from comments here that teachers only objct to MCE on the basis of its value for assessing learning. So why are you claiming otherwise?

You haven’t taken account of my point about Asian languages I see. And you obviously don’t understand either a) what maths teachers are aimin to assess (or teach) or b) how MCEs are written.

For a) you seem to think maths tests check whether someone got the right number, i.e. if they added up correctly; and for b) you seem to think a good MCE is easy to set.

The quality of arguments being raised against you is so much higher than anything you have offered, it’s as if you’re being actively rude by persisting…

238

Sebastian 10.22.10 at 4:18 am

“So why are you claiming otherwise?”

Because we know that they can assess learning, we’ve seen it. And even SoV admits that for the kind of tests we are talking about they can. She is claiming that they aren’t great for assessing calculus and higher level math, and others are asserting that they can’t distinguish between the highest levels of achievement.

But fortunately we aren’t asking it to do any of those things at all. We want it to measure very basic achievement at a much lower level than calculus.

And you have been provided a link to a test, in use in Florida, which does that. And so far as I’ve seen, no one has done anything but make extremely bald assertions that it can’t measure basic math competency.

Every time someone objects they bring up some ridiculous crap about how impossible multiple choice is at assessing calculus (which is rather noticeably belied by the AP tests), but which is entirely beside the point anyway. We are talking about BASIC math competency. See the test linked at 128. It is certainly more sophisticated than your silly “you seem to think maths tests check whether someone got the right number”. But it still measures essentially very basic math.

Right?

Please don’t answer unless you click the link.

239

sg 10.22.10 at 8:47 am

Sebastian, the AP test has a free response section for calculus, not a multiple-choice section. The AP test is the only example you have “presented” (free of links) and it doesn’t agree with your position. From the AP Test website:

The AP Calculus AB Exam covers differential and integral calculus topics that are typically included in introductory calculus courses at the college level. Because graphing calculator use is an integral part of the course, the exam contains questions that require students to use a graphing calculator. Students cannot take both the Calculus AB and Calculus BC exams during the same year.

You can find additional free-response questions and scoring guidelines on AP Central, along with grade distributions and examples of actual students’ responses and commentary that explains why the responses received the scores they did.

And your original claim was that MCEs are sufficient for calculus.

So, can we now agree? You were wrong. You’re also wrong about Japanese and Chinese (the writing component at any level can’t be assessed by multiple choice), and most of the teachers here think that more basic maths can’t be assessed that way either. Given you’ve been wrong about everything else, perhaps you should give up while you’re ahead?

240

Jerry Vinokurov 10.22.10 at 3:12 pm

Ok, quit carping about the stupid Florida test. I took a look at it and fortunately the idiocy begins with question 1:

Each weekday for two weeks, Ms. Rubio counted the number of cars parked at noon in a city parking lot. She recorded her results in the table below… [table follows]. Which stem-and-leaf plot below includes all the correct numbers of cars Ms. Rubio recorded in her table?

Here’s a great example of a GIGO situation. All the comprehensive psychometric assessments in the world aren’t going to help you if you’re asking the wrong questions. Who cares which “stem and leaf plot” represents this data? I’ve been making plots for some time now, and while I vaguely recall being asked to make such a thing at approximately grade 6 or so, I don’t think I’ve seen it since. It’s an utterly useless manipulation that doesn’t teach anything worthwhile about math. I didn’t even remember at first what the hell such a thing even was, though it’s easy to figure out from the form.

What’s being tested here exactly? Why is this relevant or interesting and how does it relate to learning math? I don’t see any connection between this question and understanding, say, elementary statistics, which one might reasonably think is where such a question is going. My grade: 0/1 quality questions so far.

Consider question 2:

In the parking lot of Flora’s apartment complex there are 6 rows of parking spaces. In each row there are 10 parking spaces on the left and 12 parking spaces on the right. The total number of parking spaces in 6 rows of the lot is represented by the expression below.
6 x (10 + 12)
Which of the following is equivalent to this expression?

Again, what’s the purpose of formulating the question in this way? It’s full of linguistic cruft that has no bearing on the actual question being asked. If you want to test whether people understand the equivalence of expressions, just ask them that. Why is there a whole paragraph preceding this when the resulting expression is just given? If the question asked “which expression represents the number of cars in the parking lot,” that would make sense, but that’s not what it does. Again, a stupid formulation even if the concept being tested is somewhat legitimate. My grade: 0.5/2 quality questions so far (I’m generous and giving partial credit for at least having a legitimate goal in mind).

Not that all of the questions on this test are so bad, but I think it just goes to show you that if you don’t understand whether the skill you’re testing is worthwhile or whether the text you’re presenting students with has any bearing on the problem, you’re not necessarily doing a good job of testing even if your validation metrics are “right.” Content matters.

My secondary objection to these questions is that many of them rely on confusion, misdirection, or even optical illusion (see question 12) to trick students into giving the wrong answer. What’s more, the strict time limits enforced on such tests make it hard for many students (especially those not proficient in English) to give answers to relatively involved word problems. I mean, I’m a fast reader and a competent English speaker, but when I see a question that says “Which inference is supported by this data?” I have to read the options. Maybe there’s a shortcut there that I’m forgetting, but in any case, you have to think about it, and one thing you don’t have on these tests is the luxury to think.

241

Steve LaBonne 10.22.10 at 3:16 pm

Somebody please let Sebastian know about the First Rule of Holes. This is getting embarrassing.

242

Jerry Vinokurov 10.22.10 at 3:17 pm

By the way, let me just also point out that many (most) of these problems involve multi-step reasoning. Knowing that someone made a mistake anywhere along the line (see, for example, the Venn diagram problem, #20) doesn’t tell you what it is that they don’t understand. And that’s something you need to know because if you don’t know that, you don’t know what you’re doing wrong. In other words, students aren’t black boxes that give right or wrong answers, they’re real people with real cognitive processes that matter.

243

Sebastian 10.22.10 at 4:00 pm

Sg, “Sebastian, the AP test has a free response section for calculus, not a multiple-choice section. The AP test is the only example you have “presented” (free of links) and it doesn’t agree with your position.”

The AP calculus test is 50% multiple choice. The weight of those questions is normally 45% but sometimes changes depending on other factors. The fact that you didn’t discover that, especially when I had already said that, suggests to me you didn’t look very closely. Thanks.

(In case you don’t believe I said that, see 232: “Where did we get the misapprehension that the well designed AP tests aren’t multiple choice? I did a quick survey of them, and all the ones I looked at are at least half multiple choice by time, and no lower than 45% multiple choice by scoring weight. And that includes even the ‘hard to grade’ things like English, History, and Music Theory. The idea that multiple choice tests can’t form the core of good assessment isn’t supported by the so called ‘good’ tests that you’re citing.”)

Jerry, I’m not sure what you don’t like about the stem and leaf question. It is the very most basic introduction to ideas like frequency and can be taught without algebra. It is a fairly intuitive concept for primary school kids who understand the relationship between numbers. I suppose if you think it shouldn’t be on the curriculum at all, then I guess it shouldn’t be there.

Your problem with question #2 is especially weird. The kids are being led to the correct answer by reinforcing the relationship between addition and multiplication. The prefacing paragraph isn’t an example of the kind of word traps that people worry about upthread, the test takers have given you two forms of the question, both leading to exactly the same answer, while simultaneously reinforcing the relationship between the symbolic question and the real world question. You see that again and again in this test–the problems reinforce, every single time, the relationship between the symbolic representation of numbers and the actual instance of numbers in a problem. And for someone who has seen too many high school graduates who have trouble with that, I think that is a great thing to put in a test for sixth graders.

The only questions that don’t really follow that format are #3 (Which of the following numbers has a value between 0 and 1?)–a straight test of fractions, #7–a straight test of knowing the definition of the word ‘mean’ in math, and #11 which is pretty much just a ‘do you understand what the sign means’ question. Rather than being a negative property, it looks like you’ve identified a positive property of the test.

244

Jerry Vinokurov 10.22.10 at 4:23 pm

Jerry, I’m not sure what you don’t like about the stem and leaf question. It is the very most basic introduction to ideas like frequency and can be taught without algebra. It is a fairly intuitive concept for primary school kids who understand the relationship between numbers. I suppose if you think it shouldn’t be on the curriculum at all, then I guess it shouldn’t be there.

Because it’s a worthless concept to test. Who cares whether you can make a stem-and-leaf plot? Unless my experience is dramatically atypical, I have literally never had to make one after middle school. It’s junk; it doesn’t reflect anything even remotely interesting about math. It’s like when kids are asked to put an answer in “standard form,” (there’s a question on this test that asks what 21^3 is in standard form. Who cares?!), it doesn’t contribute anything to the learning of actual mathematical concepts.

Your problem with question #2 is especially weird. The kids are being led to the correct answer by reinforcing the relationship between addition and multiplication.

My argument is that it’s not useful information. Note that I said that if the goal were to find the right expression, then the text would make sense. Otherwise it’s just a question that asks, “which of these expressions is identical to that one?” Except that unless you read the question first, you don’t know that, so now you’re spending time reading some nonsense about parking lots and that’s time you don’t have to read other problems. It’s a linguistic sandtrap that hurts the students because it forces them to expend effort in places where it’s not required. Unless you’re the kind of savvy test-taker who just reads the question and then goes back to extract the relevant info from it, in which case all you’re doing is teaching people test-taking skills and not math skills.

Anyway, I don’t want to sink into a debate about these questions. What I’m trying to say is that MCEs are a poor methodology for evaluating teacher quality, and a likewise poor methodology for understanding what students are doing wrong. Certainly a test like this is going to bring out a bias in favor of students who are fast readers against those who are not (for whatever reason). In my view, that neither fair nor useful. You seem to think that this evaluation is unproblematic and I (and a lot of other people) are telling you that actually, it’s not as unproblematic as you think, even at lower levels.

The very most we’ll get out of a test like this is a diagnosis that says, “the kids who are doing very poorly on this test are likely to have great difficulty with learning math.” So what? That’s not informative except to the extent that it points to that population. It doesn’t tell you how anything about how to teach them effectively, what factors are responsible, and whether the test is testing the right things.

As for the APs, they are dramatically different because they are explicitly calibrated towards the material learned in classes with the help of competent people who identify the relevant portions of the curriculum to be tested. But for what it’s worth, what I think is good about the APs is exactly the fact that they don’t limit themselves to simply a multiple choice section. If I had my way, I would eliminate that section altogether and turn it into a short-response section. That would of course require more work from the readers, and frankly the current compromise is quite acceptable to me in terms of quality, so I’m not inclined to push on that point overmuch. But that’s my ideal system, in case you care to know what it is.

245

Sebastian 10.22.10 at 4:25 pm

Sg, I’m not sure what concession you’re trying to wring from me. That the writing portion of writing tests can’t be multiple choice? Ummm, ok. I don’t know where I’ve suggested otherwise.

But unless you are restricting yourself to JUST the writing portion of writing tests, you’re wrong about the use of multiple choice on AP tests even for the Asian languages. As you can see on this test for the AP Chinese Language and Culture course, 50% of the weight of the exam is in the form of multiple choice. I can’t personally judge how good the questions are, as I neither speak nor read any dialect of Chinese. But those pesky multiple choice assessments seem to form an astonishingly high percentage of even the ‘good’ tests. Which may come as a surprise to people like SoV, who assert that no self respecting person would use them for anything more complicated than remedial algebra. (Which of course is often a higher level of sophistication than we are even testing for, a fact which seems to go mostly unacknowledged in this discussion).

246

Jerry Vinokurov 10.22.10 at 5:11 pm

Which may come as a surprise to people like SoV, who assert that no self respecting person would use them for anything more complicated than remedial algebra. (Which of course is often a higher level of sophistication than we are even testing for, a fact which seems to go mostly unacknowledged in this discussion).

You seem intent on ignoring actual criticisms of MCEs in favor of mischaracterizing other people’s opinions about them. This is a silly technique that everyone can see past.

247

Salient 10.22.10 at 5:52 pm

A colleague has pointed out to me that without the long-form quizzes we generate and grade, the information gleaned from exams would be far less useful. I was taking for granted the opportunity to formatively assess students each week in long form, and thereby get fairly intricate samples of their cognitive process in action. Without that data, I would be at a a substantial disadvantage as an assessor of learning. So I must take back a good deal of what I said upthread, as regards the utility of multiple choice testing (at least as a sole or predominant measure of performance). At best, it allows for quick verification of learning between the formative assessment and summative assessment stages: but I wouldn’t know how to interpret those scores if I didn’t have long-form assessment data handy.

It’s like the difference between sampling a output of a function f(t) where input t = 10, and having the graph for f(t) for 0 < t < 9 and then sampling the output where t = 10. If I can already see a pattern of growth or stagnation in student learning, of course a quick check to see if that pattern has continued or changed course is about all that's necessary; my confidence in my analysis of earlier long-form assessments is carrying over to confidence that the MC-exam assessment result is meaningful.

But for one student, I'm interpreting their score of 34/40 one way, and for a different student, I'm interpreting their score of 34/40 a different way, and that [a] requires long-form data appropriately analyzed, [b] leads me to often significantly different conclusions about how much learning has actually occurred, for identical MC scores, and [c) therefore clarifies that the MC scores alone do not all by themselves evaluate student learning in a fair or accurate or precise way, and thus can’t form the core of any evaluation of teacher value added.

It is a fairly intuitive concept for primary school kids who understand the relationship between numbers.

A stem and leaf plot can help students, in a particular formative stage of their learning, the role of digit in the concept of number. It can also be produced mathematically without any underlying understanding of digit. So it’s not useful for assessment of learning.

It’s a linguistic sandtrap that hurts the students because it forces them to expend effort in places where it’s not required.

Agreed! And then high school teachers are told to “teach to the test” by teaching students to ignore the explanatory chunk of text and just read the question, which is actually destructive to their ability to apply what they’ve learned in non-testing environments.

248

sg 10.23.10 at 2:30 am

Sebastian, that test you linked to states explicitly that the writing component is a freeform test. This is because any multiple choice test of writing must, by design, be a reading test. There is difference between reading and writing in Chinese; you can’t test writing by setting a reading test.

Your argument about the AP Tests was that they’re an example of assessing maths achievement using MCEs; they’re not. They’re an example of assessing maths achievement using MCEs and free-form questions. This is a subtle but important difference.

249

piglet 10.24.10 at 6:58 pm

My objection to the stem-and-leaf plot question would be that there is no application. Students are taught what a stem-and-leaf plot is but are they also taught what it is good for and why one might want to use it (i. e. as a simple tool of data exploration)? We can’t tell from that test.

Comments on this entry are closed.