Brookings Fails to Make its Case for Annual Testing

This from Curmudgucation:

I kind of love the guys at Brookings. They are such a reliable source of earnest amateur writing about education. They're slick, polished, and professional, and they rarely know what they're talking about when it comes to education.

Like most everybody paying attention, they see the writing on the wall for an ESEA rewrite by the GOP Congress, and the four (!) authors of this piece would like to put their oar in for maintaining the regimen of annual testing.

"The Case for Annual Testing," by Grover J. "Russ" Whitehurst, Martin R. West, Matthew M. Chingos and Mark Dynarski of the Brown Center on Educational Policy, presents an argument that they contend is composed of four part. And not one of them is correct. The central foundation of the structure is that testing, standards and accountability are discrete and totally separable. So we're in trouble already with this argument. But let's go ahead and look at the four legs of this stool.

Federal control of standards and accountability is unnecessary, but the provision of valid and actionable information on school performance is a uniquely federal responsibility.

Information on school performance in education is a public good, meaning that individuals cannot be effectively excluded from using the information once it exists. Because it is impossible to prevent consumers who have not paid for the information from consuming it, far too little evidence will be produced if it is not required by the federal government.

IOW, local districts won't produce information because they are afraid that someone will see it, so only the federal government can force the production. And, the authors continue, only the feds can produce the high-grade top-quality stuff. The argument is some combination of "nobody else as good as the feds" and "others can do it, but they won't unless the feds make them.

The states, they argue, are perfectly capable of setting standards and holding schools accountable. But somehow, only the feds can get good information. How does that even make sense? States are perfectly capable of making a good pancake and telling if it's any good, but only the feds can go to the store for the ingredients? How would states set standards or hold schools accountable if they couldn't also come up with the information implicit in each of those activities?

Nevertheless, Brookings says sternly, "If the federal government doesn't support it,  it will not happen."

Note: they have made another bad assumption here, but I'll wait a bit to bring it up.

Student learning impacts long-term outcomes that everyone should value, and test scores are valid indicators of such learning.

Neither half of this sentence is correct.

The first half of the sentence is supported entirely and only in the article by the work of Chetty, Friedman and Rockoff. This is the infamous study asserting that a good teacher in elementary school will make a difference of $250,000ish dollars in future earnings. Disproving the study is a popular activity, made extra popular because much of the proof is right there in the original study's own data set. If you'd like to read a scholarly takedown, try this. If you'd like one with plain English and a Phineas and Ferb reference, try this. Either way, the study is bunk.

But while the first half is substantially wrong, but still kind of right (yes, student learning results in stuff that people should care about), the last half is just silly.

The authors try to shoe-horn some Chetty et al in to prove the second as well, but it doesn't. This whole argument boils down to, "There's one paper that shows some teeny tiny correlation between test scores and doing well later in life."

But in terms of offering support for the assertion that test scores are a valid measure of important learning, they offer nothing at all. Nothing. At. All.

And here's the other thing-- even if they were a valid measure, so what? What is the purpose of knowing before the fact which students are headed for greater success as adults?

Many school management and improvement functions depend on annual measures of student growth.

The functions they're talking about include marketing charter schools and "differentiating" teachers. They assert, with a straight face, that you can't run VAM systems without test data, which they suggest is important by alluding again to Chetty, thereby managing to cram two discredited and debunked pieces of work into a single paragraph.

They also assert that test results are needed to evaluate policies that are foisted on schools (because, I guess, the schools themselves don't know or won't say). And they are looking out for the schools, which won't be credited for their success (credited? by whom? who is out there giving schools credits for doing a good job?).

Finally, you can't disaggregate data for subgroups if you don't have data.

Most of the opponents of federally imposed standards, testing, and accountability should be in favor of federally imposed annual testing shorn of standards and accountability.

Brookings' fourth and final point is that everybody really ought to love annual testing once you remove accountability and standards from the mix (if I could insert a Jon Stewart "Do tell" gif here, I surely would).

Conservatives should love it because testing data can be used to feed school choice. And to assuage their fears of federal oversight, the writers offer this astonishing assertion:

And it doesn’t have to be the same test across the nation to provide this information, or even a single end-of-the-year test as opposed to a series of tests given across the year that can be rolled-up into an estimate of annual growth.  All that is required is something that tests what a school intends to teach and is normed to a state or national population.

I have no words. Apparently this entire article is a waste of time because when they say they're in favor of annual testing, they just mean that at least once a year teachers should give some sort of test. Well, hey! Done!! I will leave it to you guys to figure out how those tens of thousands of tests will be normed up so that all of those schools doing testing a completely different way can somehow be legitimately compared. Get back to me when you sort that one out, in a decade or two.

For progressives, we offer the argument that disagregated test data is a useful tool for lobbying on behalf of whatever subgroup you're concerned about. I've contemplated this argument before, and while I understand the appeal of keeping groups from disappearing, I have serious ethical issues with using students as tools to generate talking points. If your argument for testing is, "Well, no, it doesn't really serve the kids. It might even be damaging for the kids. But it generates some real good lobbying material for advocates," I think you're on shaky ground, indeed.

And parents? Well, there's this:

Surely, such parents no more want to be in the dark about a K-12 school’s academic performance than they would want to ignore the quality of the college to which their child will eventually seek enrollment.

Because, of course, all students will eventually seek enrollment in a college. Beyond that, I'm wondering as always-- where is this great mass of parents clamoring for and demanding federal testing? Where are all these parents who have no idea how well their child's school is doing and so are desperately demanding federal test results so they will know?

Brookings finally notes that teachers unions might be a lost cause on this issue because 9and they use very nice fancy language to say this) teachers are all afraid of being evaluated and punished for the results. But teachers should be practical enough to see the value in trading an end to test-linked evaluations in exchange for keeping the annual tests themselves.

To wrap up

As always, Brookings really captures the point of view of economists who haven't an actual clue about what goes in actual schools.

The biggest gaping hole in their proposal is an unfounded belief in the validity of The Big Test. They believe that The Big Test is a valid measure of learning, and that is an assumption that nobody, anywhere has backed up. The closest these guys come is throwing around the infamous Chetty results, and all that Chetty shows is that there is a slight correlation between test scores and later financial success (thereby creating supremely narrow definitions of learning and success). For their purposes, that means nothing. I'll bet you that there's a correlation between how nice a student's shoes are and how successful that student is later in life, but that doesn't mean buying nice shoes for every student would make the student successful later in life.

But every piece of the Brookings argument rests on that foundation-- that a narrow bubble test with some questions about math and some reading questions somehow measures the full depth and breadth of a student's education. Brookings assumes that people are just upset about the High Stakes part of High Stakes Testing; they fail to grasp that a major reason for being upset about the High Stakes portion is that the Testing is crap. You can play with the data from the crap test all day, but at teh end of the day, you'll just have crap data in a shiny report.

Final verdict? Brookings has completely failed to make a case for annual testing.