Same exam, two scores

More on the CATS test, this from the Cincinnati Enquirer:

Year-to-year comparison difficult

When state accountability test scores are released Tuesday, they'll look a little different from past years.

The Commonwealth Accountability Testing System, also known as CATS, measures students' knowledge in core content areas.

CATS underwent several changes before students were tested this past spring, including redesigned tests, the addition of tests and the addition of grade levels that take those tests.

As a result, a school can't make a straight comparison of this year's score with last year's, according to the Kentucky Department of Education. Therefore, the department will release two scores to each school and district next week.

The first score is the non-adjusted score, the same they've received every year since CATS went into effect in 2000. But that one is more for informational purposes.

The second score - the result of a concordance table, or statistical link - is the one that matters toward each school's goal of reaching proficiency (a score of 100) by 2014.

The concordance table is created by lining up each school's CATS scores this year from highest to lowest. The CATS score growth trend at each grade level from 2000-2006 is then factored in to determine the final score for each school this year...

... "We have different CATS now - the rules have changed," [said KDE Communications Director Lisa Gross.]

Gross likened the table to a bell curve. She gave a baseball analogy as to how it works."Say the Yankees batting average in 2006 was .300, then in 2007 the rules changed," Gross said.

For example, maybe the pitching mound was moved or maybe aluminum bats were legalized.

"Then in 2007, the team average was .330. You can't say they were a better batting team in 2007, but if you line up their average and the other teams' averages in 2007 and compare them to 2006, you can get a sense of where they fall."

And it looks like this year's test data gets an asterisk. Whether or not the dead ball era is over will be known soon enough. If NCLB data run true to form, when CATS scores are released next week a sharper than normal increase is expected - at least for lower performing schools. Higher performing schoools have been warned that their scores may be depressed.

Barb Martin, my old Ludlow High School classmate and present director of assessment and accountability for the Kenton County School District, used an ACT/SAT example.

"If you score a 24 on the ACT, you can get a concordance table to figure out what you would need to score on the SAT to be comparable to that," Martin said. "It's a statistical way of linking two systems that are not exactly the same."

So, the concordance is a way to compare apples to oranges?

I'm not laying this on Barb, but...If the concordance is a technically sound assessment practice, and there's no problem deriving a score for a test one does not take...then I've got an idea.

Let's just establish a concordance between CATS and whatever assessment Massachusettes uses - so we can determine how our students would have scored on the Massachusetts exam - because they just kicked our butts on NAEP.

How well this sits with superintendents and principals remains to be seen.

Gross said her department has gotten "a pretty good reaction" from superintendents. However, some district assessment coordinators haven't gotten
the same reaction from principals.

Schools received their scores this week for review before they're made public Tuesday.

"Some of them are not happy with the information they've received," said Charlene Ball, district assessment coordinator for Boone County Schools.

Then Gross said something very interesting. "Because of the bell curve, schools at the top and bottom will fluctuate the most, with the higher scores likely coming down and the lower scores going up. Those in the middle will likely adjust the least."

Her specific mention of a bell curve, and her description of the statistical effect called "regression toward the mean" strikes me as curious. Is she suggesting that CATS has been put through a normative process?

If so, doesn't that suggest that CATS is no longer a performance-based assessment? More Multiple choice; fewer open response; long gone are performance assessments; math portfolios...

Doesn't the law require the Board of Education to maintain a performance-based system?

And if that's true, shouldn't we all be more concerned that teaching to the test may, in fact, galvanize our students into a restricted curriculum - just as many public school critics currently claim?

Charlene Ball, district assessment coordinator for Boone County Schools said,

"The thing that's so frustrating is that schools see their non-adjusted score, then (the higher-performing schools) see their adjusted score lower," Ball said. "But these are two unrelated tests. There was no way to report scores this year with last year because you'd be comparing apples to oranges."