Measuring the Gap under SB 1

The new SB1 test got a first reading before the Board of Education in February and is scheduled for a second reading in April. We are in a 60-day window set aside for public comment, so let's talk about it.

One of the issues that has worried me is, How should the achievement gap be measured?

This is important because we know that however the state calculates it, teachers will plan their strategies based on whatever focus is likely to provide the best test score results. One approach might induce teachers to focus on a small subset of the kids who are said to be “in the gap.” A different approach would broaden that focus.

KSN&C spoke to KDE testing guy Ken Draut, at the AdvanceEd Conference in December, and learned of KDE’s plans to use ACT benchmarks to measure the gap. Uh oh.

Generally, KDE is looking at what they call a balanced accountability approach.

Achievement score data would come from five days of testing with the new SB1 Test for grades 3-8 built around the new Kentucky Core Achievement Standards, as they are implemented. Scores will be calculated around “proficiency,” meaning that cut scores will be set to determined performance levels. Novice = 0; Apprentice = 0.5; Proficient = 1.0; and Distinguished = 1.5. The + .5 amount given to students scoring in the Distinguished range is thought of as a Bonus, and it will be offset by a negative .5 for each Novice student in the group.

But what about measuring the achievement gap?

Draut told conference attendees that “we really feel like, in the gap world, we got a really innovative model to measure gap.” Draut explained that KDE had three problems with measuring the gap which they have tried to address.

1. The number of different sub groups, which can number to as many as 45 different goals.
2. A lot of the kids fall into more than one group. For example, 80 percent of Kentucky’s African American kids are receiving free and reduced lunch. 80 percent of our ELL kids are in the free and reduced lunch group. 70 percent of our special education students receive free and reduced lunch. As a result, we end up counting one student multiple times. So under the present system, a single student might be counted four times. Miss one target and you are likely to miss four.
3. Comparing “closed gap to group.” Draut said, “You want to close African American to White; American Indian to white… Because of the way testing works, you can end up with a “wavy” pattern.” For example, in JCPS one year, the white kids at Southern Middle School dropped backwards and the African American kids stayed the same, the Courier-Journal reported that Southern Middle was closing the gap. And the opposite can happen (which was our experience at Cassidy) where white kids can go up 8 points and African American kids go up 6 points, but the gap increases.

To address this, KDE plans to present all of the gap students’ data, but will create a new single group of underperforming “gap kids” who would only be counted one time. Instead of comparing the gap kids to the group, it would be measured against the goal of 100 percent proficiency, or what is called “gap to goal.” The gap is to be divided by the number of years schools are given to reach their goal and schools would be awarded points based on the percentage of that goal they were able to close. A school that had a six goals to meet and closed three of them would earn 50 percent of their points.

Growth is to be measured by using a regression of the reading and mathematics scores, the only tests given every year from grades 3-8. It will compare a student’s progress to other students who have been performing similarly. Given a proficient 5th grade student with a scale score of 230, who then earns a score of 240 in 6th grade: the model asks if this level of growth is typical of other Kentucky students, above average, or below average, and awards points accordingly.

KSN&C caught up after his presentation.

KSN&C: Ken, as you may be aware, a number of statisticians…like Skip Kifer, say that the modeling that underlie the statistics of the ACT Benchmarks are a bunch of crap, basically. [chuckles]

Draut: Right.

KSN&C: Are you concerned about that?

Draut: Well, this is how we answered the board the other day: It’s you guys. You guys drive this. If you say the ACT is a bunch of crap, lets’ throw it out…

KSN&C: Well, not the ACT. Just the benckmarks.

Draut: Well, I’m just saying, if you all say it, and then you put something else in, we’ll line right up, because we’re trying to get them ready for you.

KSN&C: OK, but you lost me. Tell me who “you” is. Because you’re saying the board…

Draut: Universities.

KSN&C: Oh.

Draut: You see, we’re driven by the universities. We can’t get our kids into the universities unless we meet your criteria.

KSN&C: So if the universities say, this standard isn’t appropriate, or the metric’s wrong, or something, then that’s going to be a problem for you guys.

Draut: Well, we’ll put in whatever you say, but I tell you, what the issue is, and we’ve said this to several people, tell us what you’d replace it with.

KSN&C: Uh huh.

Draut: Just tell us.

KSN&C: So, the benchmarks are useful, because they are there…But you have to know what they mean or they’re meaningless. And you can’t replace it with the ACT really, because that cuts out middle school and…causes you some other problems.

Draut: Right.

KSN&C: So, then what do I replace it with. I’ve got a bad yardstick, but it’s the best one I’ve got?

Draut: And what are the universities going to accept to get the kids in the door? Because whatever the universities accept, that’s what I’ve got to get my kids ready for.

KSN&C: Are you getting that kind of pushback from the universities?

Draut: No

KSN&C: So the question’s been raised but nobody’s pushing the issue?

Draut: No. It’s kinda like just what you said, tell me what’s in its place?

KSN&C: And nothing comes to mind.

Draut: So now you open up fifty years of research saying, hey, we can tell you it works. It does predict…

KSN&C: Do we know the degree to which those benchmarks are bad, or in what direction they are bad? Or is it that we just don’t know?

Draut: I think that you’d have to do some reading, both the pro and con, when I read, and I’ve heard Skip, but when I read the ___of it, it makes a lot of sense. And when I hear Skip it makes sense, too. I can’t get a sense of which one’s right…But that whole issue is driven by CPE and the universities because if you’re sitting there in the university saying we’re only going to take the kids that make the CPE benchmark, and we’re only going to take the COMPASS, then we say, OK, and we line up with you. But if universities change…and say, you know, we’re not going to use ACT, we’re going to use some new testing, then we’ll realign everything [to that]. ..But I think it would be useful to look at both the pro and the con.

For a few months now, I've been pondering Draut's position that decisions made at CPE should drive the model ultimately adopted by the Kentucky Board of Education. Generally I agree that we can not lower standards and KDE must hit college-ready targets. But I'm much less convinced that CPE ought to dictate how the achievement gap in measured in our elementary and middle schools.

NOTE: It is my understanding that the EXPLORE can predict results on the PLAN test, but not the ACT. The PLAN test can predict performance on the ACT but not performance in college. The ACT can predict performance in college up to a point, and its arguably not the best way, but is made better by the inclusion of other measures.

Holliday's chances. Wanna Bet?

While attending the P-20 Innovation Conference at the Lexington Center this week, I had the opportunity to catch up with a few folks who are tuned in to national education politics and get their assessments of Education Commissioner Terry Holliday's chances of getting the Obama administration to waive the onerous parts of the NCLB accountability model.

At present, NCLB's fixed targets unfairly label many schools as failing, even when they show significant progress toward desirable goals. As Kentucky educators will recall, it was this measure, when layered on top of the pre-existing CATS assessment, that took the CATS' 9th life.

Kentucky, Vermont and other states sought waivers from the US Office of Education under Margaret Spellings, but were turned away empty-handed. Initially, Obama Education Secretary Arne Duncan appeared to some pundits as Maggie's mirror image, but he has since taken a clear stand against NCLB's blindness to the benefits of valuing growth as a measure of improved student achievement.

Therein lies Holliday's hope.

This week the Kentucky Board of Education took its first step toward building a better mouse trap. In approving a white paper outlining a possible future "Senate Bill 1 test" they have solidified just what a better set of school and district goals might look like. KDE Assessment guy Ken Draut told the assembly in Lexington today that the "plans are getting very firm." The state Board will take a first reading on the new system in February, a second in April and receive public input for a 60-day period after that. By July, we should know the plan.

Draut describes the new plan, as consistent with the prescription in SB1. It is a "balanced" assessment that gauges student achievement, assesses teachers and principals, considers support systems, program reviews, and provides accountability for schools and districts. More on this later.

Over at Prichard, Susan Weston suggests that the Commish can effectively argue that our new goals (based on the Obama-supported national core standards) will be tougher than the NCLB's more pedestrian expectations, as well as fairer. This is true. But persuasive arguments have been known to pale in the face of political motivations, and right now, it would take a savant to figure out the national politics. Events of this week have certainly convinced me that I don't have a clue.

Council of Chief State School Officers honcho Gene Wilhoit told KSN&C today that reauthorization of NCLB may be a couple of years away and rest with the well-tanned and mercurial soon-to-be-Speaker of the House John Boehner. But in the lurch, President Obama can handle some changes administratively, if he will.

Will he?

Kentucky Education Commissioner Terry Holliday told KSBA's Brad Hughes recently that he is very optimistic that the AYP waiver can be obtained.

“We think (the Obama administration) is very open to replacing AYP,” he said. “We think Kentucky will be the first state to take that waiver request forward, but we think there will quite a few others. It’s all based on the new common core standards and growth models that could replace AYP.”

KSN&C asked AdvancED President & CEO Mark Elgart if he was a betting man, what odds would he give that Holliday's gambit wil pay off. "Better than 50%," Elgart said.

But UK Ed Leadership Chair Lars Bjork worries about the competing tensions the Obama administration faces between their desire to continue driving change and their reservations about AYP. He opined that if Kentucky goes it alone under the banner of a reform state, it might not go too well. But all-in-all, he too gave Holliday a better than 50-50 chance.

This sentiment was echoed by Wilhoit, who when asked about Holliday's chances said, "If he goes it alone, not very good. But if he goes with a group, I think he will be successful." And Wilhoit wants to help. He thinks the CCSSO can pull together about 15 states to go to Education Secretary Arne Duncan en masse.

Duncan is on the record as disapproving of NCLB's rigid (anti-research based) accountability model which seems to have been designed to identify an increasing number of schools as "failing." But nationally, Republicans have, and continue to, dangle NCLB Reauthorization in the balance. Elgart, Wilhoit, Draut...virtually everybody I talk to (and read) seems to think ESEA reauthorization is still a couple of years away. And at least one well-placed national observer thinks Boehner may be looking to position NCLB reauthorization as a Republican political victory just in advance of the next presidential election.

But what if Holliday fails in his quest? During the Q&A following his presentation on assessment and accountability, I asked Draut about the impact.

KSN&C: Specifically, what waiver is Commissioner Holliday seeking [from the Obama administration] and what impact will their decision have on [Kentucky’s] plan?

Draut: Oh, that's a great question. The picture right now, we've got NCLB and this model of AYP that we're living with, and reauthorization is not on the fast track. So we might be living with that in 2011 and 12, and I don't know - 13. We don't know where this is going. We have this new state accountability model coming and what we are trying to avoid is having two models - just like we have now - where one says you're terrible and the other says you're great... And in NCLB, for years and years, there has been this waiver procedure where you could appeal to the US Office of Ed for a waiver of your accountability model outside of the NCLB model. It's just that until recently...

KSN&C: They didn't grant any.

Draut: ...they were kind of closed ears...but now, in support of innovation and these kinds of things, they are talking, and Gene, you can jump in here, they are talking achievement gap, growth, college readiness and graduation rate, and BINGO, that's us. So, what the thought is, is that we take our model, and present it as a waiver...We will still identify persistently low [schools] as required, identify low achieving and high achieving...still do what they'd like us to do.
KSN&C: But if they say "No," is it back to the drawing board?

Draut: Well, I think we'd probably push forward with ours.

KSN&C: But we'd have the same problem we've got now?

Draut: We'd have the same problem we've got now.

Writing Score Change Worth about 11 points

Now that I'm hanging out in the Ivory Tower with college kids I don't run into my old colleagues much any more. So it was great to see Phillip Shepard and Michael Miller, a couple of Fayette County refugees now working in the big house at KDE. It was also good to say "Hi" to Susan Weston Perkins, Roger Marcum, Dale Brown and Dick Innes. Yes, we smile and shake hands when we meet.

I missed greeting Bob Sexton and Cindy Heine, but got to meet Sharon Oxendine.

Thanks to Elaine Farris for the hospitality and to Harry Moberley for the shout out to my class during the meeting. And thanks to Moberley and Dan Kelly who have agreed to chat politics and social policy with EKU's doc students this fall.

Susan Weston Perkins commented about KSN&C's prior post on the inflation of high school writing scores when the scoring system was adjusted this year. With the disclaimer that she hasn't talked with Ken Draut about it yet, here's her take:

PERFORMANCE LEVEL CHANGE

Depending on the subject, student performance may be assessed to be

• At one of five performance levels: non-performance, novice, apprentice, proficient, or distinguished.
• At one of eight performance levels: non-performance, medium novice, high novice, low apprentice, medium apprentice, high apprentice, proficient, or
  distinguished.

From 1999-2006, writing was assessed at the five levels.

In 2007, elementary and middle school writing switched to eight levels, but high school stayed at five levels.

In 2008, high school switched to eight.

THE INDEX CHANGE

The Writing Portfolio Index on a 0-140 scale is calculated by multiplying the percent of students at each level by a weight.

In the five-level version, all apprentices are worth 0.60, but in the eight-level version, low is worth 0. 40, medium 0.60, and high 0.80.

So, if a school had a lot of high apprentices, they’d be better off with the eight-point scale. If a lot of low apprentices, they might prefer the five-point version.

There’s a similar issue with the novice scoring.

As it happens, the 2008 high school writing scores included [a] high number of high apprentices and high novices—so the switch to eight levels gave us a an impressive statewide jump...

(Apologies for the poor quality scan, KSN&C)

ROUGH IMPACT

From 2007 (five-level) to 2008 (eight level), the high school writing index went from a 56 to a 72.

It’s possible to go back and use the five-level formula. You just weight all apprentices at .60 and treat both medium and high novices as .13. If the state had done that, the Index would be just a 61.

In other words, 11 point come from that giant weighting switch.

THE REMAINING PUZZLES

Why did they switch elementary and middle in 2007 and high in 2008? KDE clearly decided to move to the eight-level scoring some time ago. Why the delay?

Also, why did they present the 2007 and 2008 data as though it’s a continuous trend? They made all that effort to say you couldn’t show trendlines for similar changes from 2006 to 2007. Why didn’t they repeat that on this one 2007 to 2008 change?

I don’t know how Ken Draut would answer these questions and need to make
time to ask him.

Thanks Susan.