Wednesday, March 23, 2016

How do the schools really stack up?

Over at School Finance 101 Bruce Baker is doing his year end review of education data on the New Jersey schools. But his review contains some news that applies to Kentucky and all other states as well. I have added a few bits to highlight Kentucky's performance.

This from School Finance 101:
On many occasions I’ve pointed out better and worse uses of national and international assessment data – of Mis-NAEPery and PISA-Palooza… wherein the media and punditocracy go wild with gross misrepresentations and misinterpretations of relatively limited albeit not entirely useless assessment data.

In New Jersey, we’ve been told in recent years that while our average scores remain high, we must not rest on our laurels, because our gains pale by comparison to reformy standout states like Tennessee. We’ve been told that while our average performance is high, our gaps in achievement are among the largest in the nation and certainly not improving at any reasonable rate. We’ve also been told that these findings provide strong proof that all the money New Jersey has thrown at schools in response to years of litigation over school funding has not only been unhelpful, but that the additional funding to high poverty settings has actually caused harm. As such, the way to repair that harm is to reduce funding to high need settings and redistribute those harmful resources across other, less needy districts likely to use it more wisely (yeah… really… they did say that… and they’ve followed through on that redistribution plan!) And that will help fix the achievement gap!

But what really is the state of student outcomes in New Jersey, if we apply a few basic principles to the analysis of NAEP data – guidelines I have addressed in numerous previous posts:
  • First, average state contexts differ and those differences strongly influence average NAEP scores at all grade levels. In short, poverty matters! As such, performance levels should be adjusted for poverty.
  • Second, NAEP gains over time (which are cohort gains), are strongly influenced by initial NAEP performance. That is, those who started lower and had more to gain, gained more. As such, changes over time should be adjusted for initial scale scores.
  • Third, achievement gaps between low income and non-low -income, or among races are substantially influenced by the income gaps between these groups. As such, achievement gaps should be adjusted for differences in income between groups.

Average Performance Level Adjusted for Poverty

This first figure shows us the scatterplot of state average poverty rate and average 8th grade NAEP scale scores. Those states falling above the line have greater than expected scale scores and below the line have lower than expected scale scores. Notably, the correlations are quite strong. New Jersey beats the odds on both 8th grade reading and math. That is, NJ scores are higher than would be expected even given New Jersey’s low poverty rate?

Figure 1

If we rank states by their average difference from expectations, New Jersey comes in fifth (averaging the math and reading differentials). [Kentucky comes in 7th!]

Figure 2

But that’s only because the whole U.S. stinks, right?

Of course, some might argue that its really nothing to cheer about – doing better than other U.S. states, because of course, we all know that the U.S. performs miserably compared to other nations. But as I’ve pointed out previously, when conducting similar poverty adjusted comparisons across countries, the U.S. doesn’t look so bad. (see here for more explanation/discussion)

Figure 3

Ah… but you say… outcomes of even high performing – non-low income kids in the U.S. still stink compared to those in other countries. Again, I respond by pointing out that most such comparisons are deeply conceptually flawed. Perhaps most importantly, as I’ve explained on numerous previous posts, the U.S. average is only as low as it is in international comparisons because of the large number of low performing (relatively high poverty) states that have largely thrown their education systems under the bus for the past several decades...

Here are a few more pictures...


Richard Innes said...

There is some smoke and mirrors in this analysis, too.

The biggest problem is that there is no reliable way to estimate poverty in the 2015 NAEP data. That is because all states now participate in the federal free and reduced cost lunch "Community Eligibility Program." Under this program, if 40% of a school's students can qualify for lunches using one of several options, then the WHOLE school goes on the school lunch program, even the children of the wealthiest families.

At present, the National Center for Education Statistics is working on this problem -- they even have a special team established -- but no answers were approved as of 2015 NAEP testing.

So, forget all school lunch based analysis in the NAEP. The way states have reported their 2015 lunch data to NCES for NAEP reporting is total chaos and the figures are not comparable across states.

BTW, Kentucky still needs real, needs-based lunch eligibility figures for use in the SEEK formula, so the data reported in the Kentucky School Report Cards is supposedly still solid for needs analysis. But, this all is out the window when NAEP is considered.

Richard Innes said...

One more thing: when doing NAEP analysis looking only at poverty isn't enough. You absolutely have to break things out by race to begin to get a real understanding of relative state performance.

In some of the stuff shown in the main blog item, a lot of poor whites in Kentucky are being compared to a lot of poor blacks and Hispanics elsewhere. A good number of those kids in other states are also still learning English, too. So, the comparisons shown would be problematic even if the school lunch data wasn't in total chaos.

Richard Day said...


There is neither smoke nor mirrors. That implies deception.

What there is, is a very complex dynamic that we try to make quantitative sense of while the environment is constantly changing. The fact that our measures are imperfect is something we live with constantly. But Baker's point, that poverty impacts results, is indisputable. We should not forget it. But showing poverty's impact (or real student achievement for that matter) with flawless statistical precision remains a distant dream.

At present only 104 Kentucky districts participate in CEP. But of course, that makes your point.

I'm very concerned by the implications of viewing race as determinitive of student performance. Aren't you? Race is not a biological construct. Talk about bad data! You're worried about free lunch data. How would one ever control for racial mixing?

Focusing on performance differences by race may be useful politically, but it's a statistical quagmire.

Have you ever wondered why Whites aren't being compared to Asians?