CATS Confusion Abounds

The most important issue - by far - facing returning legislators this week is the state budget. But for some legislators, it is the elimination of CATS. Let's hope the big issue does not get lost within the smaller issue.

After reading various comments in the debate over the fate of CATS it seems that the conversation would benefit from a little clarification.

The first thing I think of when I hear someone say, ‘Let’s get rid of CATS,’ is that the speaker is either not fully informed about what CATS really is, or the speaker is assuming everyone understands the real meaning of CATS, and is casually misusing the name on purpose.

I suppose it's no big deal when a lay person makes the mistake. In fact, the media and KDE contribute to that misunderstanding in some ways. And clearly, some teachers, who know better, use CATS as a short-hand reference to the Kentucky Core Content Test (KCCT). But it is bothersome when a legislator fails to understand the implications of such a comment.

So let's clarify a couple of things.

First, CATS is a multi-component system designed to produce some measure of school accountability; it is not a test. The Commonwealth Accountability Testing System has many parts that are still going to be with us long after the General Assembly adjourns - particularly the $9 million component required of Kentucky by No Child Left Behind (NCLB).

But CATS is the KCCT, the ACT, the Explore, the Plan, the alternative assessment, writing portfolio…and everything.

Furthermore, some are confused about the relationship between CATS and NCLB. It seems the public is being led to believe these are two separate tests, when in fact, the NCLB scores are produced by the KCCT. It that sense, NCLB is a portion of the Core Content Test.

But this is a case where the public ought to be confused because the data are published in two separate reports. The NCLB report comes out early, in order to meet the federal deadline. The KCCT report comes later. Because they are reported separately, the general public seems to think they are separate tests.

But lately, a new idea has been introduced into the conversation - the idea that CATS should be able to compare the performance of Kentucky students to the performance of students in other states.

KSN&C discussed this idea recently with Kentucky Board of Education member, Dorie Combs, who chairs the Department of Curriculum and Instruction at Eastern Kentucky University where I work.

Combs told KSN&C that,

This idea of, ‘Let’s give a test where we can compare ourselves to other states.’ That’s probably the biggest misunderstanding. And it takes a little explanation.

First of all, every state has to come up with a test to measure that state’s standards. So every state has a different test that they use for NCLB.

And they aren’t using off-the-shelf tests to do this. In most cases they’re doing what we used to do. They’re augmenting an off-the-shelf test. So they might have the CTBS, which they have taken and added to; with additional questions. And then adding in, in many cases, an open response questions, which is what we do as well.

This makes state-to-state comparisons inappropriate. Combs says,

There’s really only one test out there that compares the states and that’s the NAEP. [The National Assessment of Educational Progress]

Of course, the concept behind CATS was never to compare Kentucky to other states. It’s a criterion test designed to look at how the schools perform. And any inference that CATS was intended to, designed to, or has done so, is simply wrong.

Combs added,

Even if we used the ACT: Again, the ACT would not yield a fair comparison because not every state takes the ACT. And we’re [testing] juniors, here. Other states test their seniors.

Combs was recently at a meeting with one of the New York Regents (like Kentucky's school board members) and they discussed the long-standing New York Regent's Exam. She learned that,

They know that their high school graduation requirement tests “at about the ninth or tenth grade level. What does that do?

Coincidentally, I have two students in one of my EDF 203 sections both of whom passed their high school proficiency exams, in Ohio, as eighth graders; and to hear them tell it, were never tested again.

Kentucky chose to give the ACT in the fall and the KCCT in the spring. The idea was get some performance data early in the junior year with the hope of lighting a fire under some kids going into the spring KCCT testing, where the schools are held accountable.

KSN&C: You know Dorie, the fundamental thing behind this test that was so revolutionary was the notion of measuring how well schools performed as opposed to how well kids performed. Do you think that, if there was adequate funding, that the notion of measuring school performance might give way to the notion of measuring individual performance?

Combs: There are those who take that view; that the best way to get kids to perform is to hold their feet to the fire. But what we’ve seen happening in other states – it doesn’t work that way. Inevitably, you get into this cycle - where some students don’t perform well enough to move to the next level, or pass, or graduate – yet they have good grades. The families start suing the school districts. Saying, ‘How can you say they can’t graduate when you’ve passed them every year and given them good grades?’

And, then [those states] come back, and they change the test. Or, they start letting them take it over and over and over. So every state that’s used a student accountability model eventually falls into this quicksand. And then going back and saying ‘Well we’ve got too many people not passing. There must be something wrong with the test.

I understand that teachers and some school administrators feel like they should not be put under the stress of the student’s performance.

And some are complaining about how schools use test data to fire young teachers; an idea that probably ought to be challenged in court. Teachers do not control enough of the variables. It’s totally unfair. But if that’s the way the game is going to be played, then let’s put the superintendents under the same stress. Just as superintendents have objected to such plans – on justifiable grounds – so too should the teachers be heard on this issue. What’s good for the lower and middle levels of the organizational chart ought to be true for the top as well.

Combs says,

It concerns me that schools use test score data to fire teachers.

I don’t think you can take that test score and say, ‘It’s your fault.’ Because all the way down the line, you have to say, ‘Well, it’s their fault too.’ We can’t isolate it to teachers. I’m opposed to that approach where all you say is, ‘What does this teacher contribute.’ It can make it difficult for that teacher to work with a child.

So this idea of getting rid of teachers is not coming from [the Kentucky Board of Education].

The idea is to say, ‘What can we do to get better?’ Not to beat everybody up and say, ‘This is your fault.’

And when you look at it, the punishment for the school is usually more money.

...arguably, a disincentive to obtaining a target. Yet the most exalted examples among our superintendents and principals seem to be those who move quickly to get the "right people" on the bus and the "wrong people" off.

Combs contends that,

This idea that beginning teachers are disposable is a shame because it doesn’t give teachers the time to develop into mature, strong teachers because there’s so much you learn in that first year…

At present, the proposal to dump CATS is supported by arguments that it would save the state money.

The Herald-Leader recently opined that Senate President David Williams was being disingenuous when he claimed the state would realize savings by eliminating CATS.

He's just using the budget emergency to get in a little pandering. He's playing to that small segment of Republicans and soreheads who like nothing about the public schools and to a larger segment disenchanted with what they see as an overemphasis on testing.

The Commonwealth Accountability Testing System accounts for about 1 percent of the education budget or about $14 million. Kentucky is facing nearly a $500 million shortfall.

Trying to replace CATS with an off-the-shelf achievement test could end up costing Kentucky millions in federal dollars by putting the state out of compliance with the No Child Left Behind law. There also would be costs and penalties for breaking agreements with current testing contractors.

Killing CATS could dig the state even deeper into the hole.

To put a finer point on the actual dollars, Combs points out that the proposed elimination of CATS,

is not going to help us money-wise, because it doesn’t cost that much... Assessment [in Kentucky] is $15 million all together. We can’t get rid of the whole thing. We’ve still got to address NCLB.

While the Obama administration has been fairly clear that growth models are about to become de rigueur for NCLB, that hasn’t happened yet and it could be a while before it does.

Let’s say they vote to [eliminate CATS]. We still have to keep the components that are required for NCLB. So that means keeping, [grades] 4 through 8, reading, writing; science at elementary [through] high school, and reading and math in high school. So we’ve got to continue that. So now we’re already spending $9 million. Now, we’re only going to save five or six million.

Here’s the likely scenario. There’s not going to be any forethought to - when we take this away - What’s going to be in its place?

So we could well end up having to create one test while paying for another.

But in any case, arguments over how much money could potentially be saved should not exceed $6 million, tops.

In my view, we had an imperfect, but decent enough, assessment until NCLB requirements came in on top of CATS. In fact, NCLB did not benefit Kentucky one bit. We already had Senate Bill 168 which requires the disaggregation of student achievement data. If we wanted to stick sharper “teeth” into SB 168, it would have done everything Kentucky needed.

Combs says she hopes,

that changes might be made [to NCLB] to give us some flexibility. And pull some of this, ‘If you’re not meeting 100% of your goals, you’re failing.

[I just want to] make sure everyone understands that KCCT fills the requirements of NCLB. There’s really no test that is a national test - other than NAEP - and that NAEP does actually make these state comparisons. That’s the best way to compare state-to-state and those data are available.

Has CATS simply become a bad brand? It certainly seems to this observer that the popular sentiment has turned against CATS.

Combs countered,

If I was a teacher – the best thing to make my life easier - it’d be to get rid of that dang test.

Well….duh. That would make everybody glad.

Does that make it the right thing to do?

Not necessarily.

The Council of Chief State School Officers maintains a careful review of state accountability systems. Turns out,

There’s only one state that uses an off-the-shelf achievement test. And, you know who that is?

I’d like to think that if I had a minute, I would have come up with that one. But I forgot - the Iowa Test of Basic Skills. Of course.