CATS: Marxist Psychobabble?!

The mainstream media has picked up on the frustration of school folks that results when higher authorities monkey with the assessment system - subordinating the need of school leaders for a stable system in the process.

The issue is aired out in ths morning's Cincinnati Post:

'A bunch of psychometric babble'

Changes made this year to Kentucky's standardized testing make it harder than ever to accomplish the primary goal of the program -- measuring improvement, many Northern Kentucky educators say.

The Kentucky Department of Education, which approved the changes in test questions and how they're scored and subjects weighted for the CATS program, doesn't disagree.

Comparing scores to previous years, the linchpin of monitoring progress since the first annual statewide CATS results were released in 2000, means little now, at least for this year.

That is the lament today in local schools and district offices, as superintendents try to figure out if their students have progressed and if so, how much, for the testing cycle that will be released to the public Tuesday.

"It's a bunch of psychometric babble," Walton-Verona School Superintendent Bill Boyle said Thursday. "We're going to sit down tomorrow and ascertain where we are. We have no idea."

The assessment program began under House Bill 53, when the KIRIS testing program (which seemed to change annually) was replaced by the Commonwealth Accountability Testing System (CATS). The shift to CATS may not ahve been perfect, but it was a stable system that allowed consistent long-term comparisons which are vital to longitudinal analysis. Consistency across place and time is what makes NAEP data so valuable, for example.

What possible advantage is it to ask schools to hit a moving target?

The heart of the CATS test, The Kentucky Core Content for Assessment (the academic parts of the test) has been revised. Questions have been changed, as has the "weight" given to each subject. The ACT college-entrance exam was added for all Kentucky high school juniors, and for the first time this year it will be factored into high schools' accountability scores. But cut scores that determine what constitutes "Proficiency" were adjusted, and their meaning was lost in the process.

KDE told superintendents:

"Your overall accountability index will likely decrease a few points if you have been a successful school."

It goes on: "Your overall accountability index will likely increase a few points if you have been a struggling school."

"I don't know who wrote this," Boyle responded. "Karl Marx?"

(KSN&C has speculated {on scant data} that it is now easier for schools to achieve proficiency. We'll see if this is true next week.)

Some of the changes were apparently forced by NCLB. For example, the state's dalliance with Depth of Knowledge (DOK) - Professor Norman Webb's reinvention of Professor Benjamin Bloom's famous taxonomy - supposedly absolved the state when it increased use of multiple choice questions (and reduced use of open response questions) making the test less performance-based; something that is required by Kentucky statute.

KDE says the changes are "technically sound."

The Prichard Committee' s Cindy Heine says, "On balance, it's been good. I fully believe our schools are more focused on curriculum and instruction, and helping all children be successful..."

CATS uses results from the Kentucky Core Content Tests (KCCT), which are tied directly to the state's standards. The key goal is progress at each school.

But at least for this year, that's harder to figure. To ballpark how the new scores translate to previous performances, school districts must apply a concordance table....

...Since CATS results will now be released as a bell curve, schools will be better able to compare their scores to others. But it makes it more difficult to compare a school's scores to its results from last year because the rules, and the ways to interpret them, have changed.

That's why a school or district could perform better but get a lower score, or vice versa. That won't, however, affect state-set goals for progress by which schools and districts are judged by the Kentucky Department of Education....

KDE says this is the first year of a new cycle, so this year's scores aren't make-or-break. But I doubt many folks in the field are going to see it that way. No one is suggesting that school leaders won't be replaced based on poor performance as measured by the new yardstick.

Boyle's been in education for 30 years, and strongly applauds the past decade's push for accountability.

There was little accountability on educators when he started his career, he said, and he reflects the mood of many Northern Kentucky school superintendents when he says CATS was well-intended and has been largely helpful in shaping curriculum improvement plans.

But he and others see this year's changes as creating a shifting target.

"We're just tired of the continual change," he said. "You adjust your curriculum
accordingly, who you hire, but it deals with people's livelihoods, and it gets frustrating."

Timing, of course, is everything. And that's another major complaint educators have about CATS results. Students take the tests in March and April, but results aren't typically released until September.

This year, it was pushed back even further, to Tuesday, in the wake of districts'
complaints that No Child Left Behind scores were inaccurate.

The problem is, by then, the school year is well under way, so it requires adjusting curriculum, or subject emphasis, in mid-stream.

"My biggest criticism isn't that we have an aggressive accountability, it's the timing," said Newport School Superintendent Michael Brandt.

"It's almost in October ... and the data is almost irrelevant," Brandt said. "Some of it's still very subjective, and the open-response part takes too long to grade. I have kind of a silly way of looking at it. If you can get a child into Harvard, Yale, UC or Northern, based on him or her taking a four -hour ACT test, why can't this be done quicker ...

This needs to be back (to schools) in 30, 45 days to implement corrections sooner."

School folks are broadly supportive of the intent of the CATS assessment and see it as a great improvement over past times when the pressure for student achievement was slight. But with the changes, CATS supporters are now waiting to see.

Erlanger-Elsmere School Superintendent Mike Sander...remains cautiously optimistic about the current changes. "Overall, is it better?" he said. "That's difficult to say. Let's see in two, three years."

As Kenton County Schools Superintendent Tim Hanner points out,

"This year it really is a one-year snapshot of how our students did."

A one-year snapshot is right. Nothing less.

But as some frustrated superintendents point out, it's also nothing more.