Arne-Ology & the Bad Incentives of Evaluating Teacher Prep with Student Outcome Data

The absolutely worst, most anti-scientific expression of the corporate school reform movement I have seen is related to evaluating Teacher Education Programs by the test scores of their "grandstudents." That is to say, the students of teachers who graduated from a given higher education institution. There are well-documented problems with value-added measures of student progress in the K-12 world already. But this idea takes those flaws and elevates them to new heights.

The Obama administration should be congratulated for wanting every teacher preparation institution to produce outstanding candidates. Of course they should. And they can be forgiven for wanting the ability to measure that quality and communicate progress to the public. But what is unforgivable is to proceed with plans to use metrics to evaluate important variables when it is well-established that those metrics are flawed to the point of meaninglessness.

The knowledgeable misuse of pseudo-scientific data for political purposes is every bit as damaging to actual scientific principles as the defunding of research projects.

But if the feds insist on such a fundamentally unwise policy, EKU could respond by changing our admission policies. To be accepted, our candidates would have to enter into an agreement with Eastern that, after graduation, they will only accept teaching jobs working with girls in schools where the free/reduced lunch rate is less than 35%. That will ensure a bright future for EKU beyond anything else we might undertake. Because such measures reward schools whose graduates go on to teach in affluent schools, millions will become available through federal grants to our students. Our student population will grow, and right along with it, our national prominence (so long as no one debunked what we were really doing) and our future budgetary problems will melt away. 

This from Politico:

The Obama administration plans to use tens of millions in federal financial aid as leverage to reward teacher training programs that produce teachers who routinely raise student test scores — and to drive the rest out of business.

Education Secretary Arne Duncan will announce the revival of a push to regulate hundreds of teacher preparation programs Friday at a town hall meeting with White House policy director Cecilia Muñoz. He plans to release a draft regulation by summer and aims to enact it within a year.

The goal: To ensure that every state evaluates its teacher education programs by several key metrics, such as how many graduates land teaching jobs, how long they stay in the profession and whether they boost their students’ scores on standardized tests. The administration will then steer financial aid, including nearly $100 million a year in federal grants to aspiring teachers, to those programs that score the highest. The rest, Duncan said, will need to improve or “go out of business.” ...

This from School Finance 101:

As I understand it, USDOE is going to go ahead with the push to have teacher preparation programs rated in part based on the student growth outcomes of children taught by individuals receiving credentials from those programs. Now, the layers of problems associated with this method are many and I’ve addressed them previously here and in professional presentations.

1. This post summarizes my earlier concerns about how the concept fails both statistically and practically.
2. This post explains what happens at the ridiculous extremes of this approach (a warped, endogenous cycle of reformy awesomeness)
3. These slides present a more research based, and somewhat less snarky critique

Now, back to the snark.

This post builds on my most recent post in which I challenged the naive assertion that current teacher ratings really tell us where the good teachers are. Specifically, I pointed out that in Massachusetts, if we accept the teacher ratings at face value, then we must accept that good teachers are a) less likely to teach in middle schools, b) less likely to teach in high poverty schools and c) more likely to teach in schools that have more girls than boys.


Extending these findings to the policy of rating teacher preparation programs by the ratings their teachers receive… working on the assumption that these ratings are quite strongly biased by school context, it would make sense for Massachusetts teacher preparation institutions to try to get their teachers placed in low poverty elementary schools that have fewer boys. 

Given that New Jersey growth percentile data reveal even more egregious patterns of bias, I now offer insights for New Jersey colleges of education as to where they should try to place their graduates – that is, if they want to win at the median growth percentile game.


It’s pretty simple – New Jersey colleges of education would be wise to get their graduates placements in schools that are:

• 20% of fewer free lunch (to achieve good math gains)
• 5% or lower black (to achieve good math gains)
• 11% or lower free lunch (to achieve good LAL gains)
• 2% or lower black ( to achieve good LAL gains)

Now, the schools NJ colleges of ed should avoid (for placing their grads) are those that are:

• over 50% free lunch
• over 30% black

That is, if colleges of education want to play this absurd game of chasing invalid metrics.
Let’s take a look at some of the specific districts that might be of interest.

Here are the districts with the highest and lowest growth producing teachers (uh… assuming this measure has any attribution to teacher quality). 



Now, my New Jersey readers can readily identify the differences between these groups, with a few exceptions. Ed schools in NJ would be wisest to maximize their placements in locations like Bernards Twp, Essex Fells, Princeton, Mendham and Ridgewood. After all, what young grads wouldn’t want to work in these districts? And of course, Ed schools would be advised to avoid placing any grads in districts like East Orange, Irvington or Newark.

Let me be absolutely clear here. I AM NOT ACTUALLY ADVOCATING SUCH DETRIMENTAL UNETHICAL BEHAVIOR.

Rather, I am pointing out that newly adopted USDOE regulations in fact endorse this model by requiring that this type of data actually be used to consequentially evaluate teacher preparation programs.

It’s simply wrong. It’s bad policy. And it must stop!

And yes… quite simply… this is WORSE THAN THE STATUS QUO!

For further discussion on this point, I refer you to this post!