Guskey Weighs in on Charters

Pretty much says what we've said....

This from the Herald-Leader:

Charter schools should not have unfair advantage

...Allowing charter schools would certainly enhance the state's application for Race to the Top funds. The federal government believes that giving parents the option of enrolling their children in a charter school may serve to increase educational quality and accountability.

But certain influential education organizations in Kentucky don't want charter schools because they fear charter schools will detract from the viability of our public school systems.

As an educational researcher, I have read much of the research on charter schools. From what I can determine, the findings are inconclusive at best. Some studies say charter schools work, while others suggest they offer little or no advantage.

Unfortunately, the quality of most of these studies is not very good. The majority have significant methodological flaws that challenge the validity of their results.

The most we can say at this point is that we really don't know whether charter schools are necessarily good or bad. No strong evidence shows students learn better in charter schools than they would if enrolled in regular public schools. But no confirming data shows that charter schools harm students or public school systems either.

The real issue comes down to the policies set for launching charter schools.

Guskey suggests two rules for charters in Kentucky:

1. Charters must accept any child who wants to enroll.
2. Charters have to keep that child and will be held accountable for his or her learning progress, no matter what.

Guskey says these rules would level the playing field and that any school can look good if allowed selective admissions or if allowed to "counsel out" students who don't fit.

That is why saying charter schools work for students who stay for three years has little significance if all those students not doing well were "encouraged" to leave.

Guskey on Why the ACT is Inappropriate for State-wide Accountability

In today's Herald-Leader, CASA assessment guru Tom Guskey points out the problems associated with using tests for purposes that run counter to their design. Without mentioning Senate Bill 1, he shows why it won't fix what's broken.

Selecting school tests a matter of competence

...What's missing in these discussions about testing is a clear understanding about what...certain tests can and cannot do.

...College entrance exams such as the ACT and SAT help colleges and universities decide whom to admit [but they] do not reflect any particular level of knowledge... rather where each student ranks in relation to others. Ranking makes the selection process easier.

Problems arise when a test designed for one purpose is used for another. ... tests like the ACT and SAT are labeled "instructionally insensitive." If instruction helps most students answer a question correctly, then that question is removed from the test, for it no longer serves its purpose. Even if the question asks about a vitally important concept, it no longer differentiates students and is eliminated.

This is why scores on selection tests are more strongly related to social and economic factors than are scores on competence tests. Aspects other than those influenced by instruction often account for the differences among students. It is also why it makes little sense to use a selection test like the ACT or SAT as a measure of the quality of instructional programs. Doing so would be analogous to using a ruler to measure a person's weight.

Having all students take a selection test such as the ACT or SAT may help some realize that they rank high enough to get into a college or university. That would be a good thing, especially for non-traditional students and those who come from disadvantaged backgrounds.

But to use the results of an "instructionally insensitive" selection test to assess the quality of instructional programs is educational sacrilege. No testing expert would agree to it -- and neither should any legislator or policy-maker.

Hot off the presses: CASA cites Problems with Senate Bill 1

As KSN&C readers may know that when it comes to assessment, I play favorites.

Over the years, the best, most reliable, and independent sources of information regarding assessment in Kentucky have consistently come from three individuals, Skip Kifer, Ben Oldham and Tom Guskey. I am fortunate to have studied under Ben and Skip at UK. Furthermore, as a principal, I would share my data with Skip annually and he would selflessly sit with me and analyze gap closing and the overall performance of Cassidy students. Tom has built a fine national reputation for scholarly excellence.

If we ever disagree - they're probably right.

If I had a magic wand - they'd redesign and oversee the state's assessment program rather than KDE, but that's another story.

Recently - owing largely to Skip and Tom's disillusionment with some things at UK - they became ripe for recruitment by Vice President Ben Oldham at Georgetown College. Their formation of the Center for the Advanced Study of Assessment is a major coup for the small liberal arts college.

Today they weigh in on Senate Bill 1 in a 19-page analysis posted at the Prichard Committee.

Read the whole thing. But here is their conclusion:

Enacting Senate Bill 1 will change dramatically Kentucky’s assessment and accountability systems. There are four major departures from the existing assessment that we find problematic and should be carefully scrutinized. Careful scrutiny includes identifying changes that have broader implications than might at first be evident.

1. The first major change is moving the focus away from school outcomes to individual academic achievement.

This could substantially reduce the amount of information that is available to schools and districts, information they have used in the past to judge in a particular content area what part of the core content they have done well and what part not so well. The reason for the reduction is that previous assessments produced more than one form for the assessment thereby increasing the breadth and depth of what was sampled. This change also may not have its intended results of better measurements at the individual level. Because the test is expected to be an adequate sample of Kentucky’s core content, produce national norm-referenced results, and provide diagnostic information, the attempt to do all three things may limit how well it does any one of them.

2. The second major change is moving parts of the assessment from the state level accountability portion to the district level where there is no formal accountability.

Moving things to the district level and making the district responsible for parts of the assessment is not necessarily a bad thing to do. Other states have done similar things. But it may not be wise to move just some of the assessment to a new level. An assessment that is part state accountability and part district responsibility may prove to weaken both. We are particularly concerned about the effects of such changes on Kentucky’s long standing commitment to teaching students to write well.

3. The third major change is assessing with only multiple choice items.

Open-ended responses, the writing portfolio, and on-demand writing have been eliminated. Given the nature of the goals, standards, and expectations for Kentucky schools, we do not believe that an assessment that relies exclusively on multiple-choice items can adequately describe those outcomes. Measurements that tap more easily more complex skills and knowledge are both necessary and desirable.

4. The fourth, and perhaps most important, major change is reducing and perhaps eliminating the participation of teachers in the formal assessment.

In the past teachers have worked on content standards, created test questions, evaluated portfolios, and graded open-response items. Although a cadre of teachers became experts in these areas, creating technical expertise was not the main purpose of the involvement. The purpose was to provide opportunities for teachers to see how instruction, standards, and assessmentsshould be intimately tied together. These activities are powerful ways to make teachers both better teachers and better assessors.

The new NAEP data are released. That means it's time to spin the news!

The release this week of national test scores in reading and math has generated a fresh round of conversation about how Kentucky's students are performing.

We have this conversation at least three times a year when the various "yardsticks" (NAEP; CATS; NCLB) trot out their measurement data. Then there's SAT, ACT... EIEIO....

Wouldn't it be great if such data were integrated into a comprehensive value-added system? But I digress.

NAEP scores nationally, and in many individual states, showed modest gains from 2005 to 2007.

As Diane Ravitch explains in today's New York Post,

The federally sponsored National Assessment of Educational Progress (NAEP) is known in the education world as the gold standard of testing. In 2002, Congress authorized NAEP testing in every state to serve as a check of the states' own claims about their progress. (Congress rightly worried that individual states would dumb-down tests that they themselves develop and administer.)

And, there is at least reason to be suspicious of Kentucky's "new and improved" test. It appears Kentucky may have joined a number of other states in a race to the bottom by the redefining of proficiency.

Whenever test score data are released the spinning begins. The Kentucky Department of Education has an interest (some might say a duty) in pointing out the progress made by the schools. So they publicly shine a light on the best numbers, and privately express concern for the worst.

It's a little thing called spin. Everybody seems to delight in the practice these days.

In a Tuesday press release, the Kentucky Department of Education said,

"The results of the 2007 National Assessment of Educational Progress (NAEP) in reading and mathematics show that Kentucky's 4th and 8th-graders made gains when compared to the state's performance in previous NAEP assessments..."

True. Gains were made. Kentucky's student achievement, as measured by the NAEP, has trended steadily upward overall. (See charts below.)

So that's KDE's headline; Progress over time.

On the other side of the argument, assessment watchdogs are sniffing out specific areas of concern. Writing for the Bluegrass Institute this week, Richard Innes took issue with KDE's discounting of declines in 8th grade reading.

KDE claims of eighth grade reading since 1998, “Kentucky’s 8th-graders’ scores have remained steady, with minor gains and losses.”

Is that a fair description? Let’s examine the facts.

In the new ... NAEP assessments ... Kentucky had a reading proficiency rate of 30 percent in 1998. That rose to 32 percent in 2002 and went up again to 34 percent in 2003.

Then, things came unglued.

Eighth grade reading proficiency decayed to 31 percent in 2005, and in 2007 it slid again to just 28 percent. The 2007 proficiency rate is statistically significantly lower than both the 2002 and 2003 scores and is clearly six points lower than the 2003 performance. That six point difference isn’t just statistically significant – it’s just plain SIGNIFICANT.

No other state lost more ground in this time frame.

What’s more, during the same time period, the CATS Kentucky Core Content Test reading proficiency rates for eighth graders continuously rose. Do you believe that?
CATS up 10 points while NAEP declined six?

I'll take Dick's word for it that the declines are statistically significant and that Kentucky's decline is the nation's worst. The decline certainly looks significant to me.

We don't generally consider six point drops minor, and more to the point, downward trends are antithetical to progress. If it were me...I think I'd have left the word "minor" out of KDE's statement.

A BETTER ATTACK?

Innes has discovered some bad news, but arguably, not the worst news.

While Kentucky has progressed steadily, so have other states. Growth is a vital factor to consider, but so is excellence. Kentucky's relative standing among the states frequently leaves the state in all too familiar territory.

For example, who do Kentucky students outperform in 4th grade math? (See map below)

New Mexico, Louisiania, Mississippi and Alabama. All other states are roughly equal to (9 states), or exceed (36 states) Kentucky's progress. You're not going to hear that in a KDE headline.

It's a little better at 8th grade. Add California, Nevada, Oklahoma, Tenessee, Hawaii and West Virginia to the list.

Kentucky only outscores eight states in 8th grade reading.

But clearly the best news for Kentucky is in 4th grade reading where Kentucky joins the national leaders and is only outscored by seven states. What happens between 4th grade and 8th grade in reading ought to be of concern.

MARK YOUR CALENDARS

We're less than a week away from KDE's next big announcement of progress. I predict the new CATS assessment will show average performance gains of 7% or so across-the-board and in some places, jumps will be huge based at least partly upon changes...
a) to the test itself
b) to the "cut scores" used to define proficiency

The new test data can not be compared to the previous tests - but it will be. It's the data school folks have.

We discussed the NCLB data situation last night at UK. Without advance comment, I asked a group of graduate students (and future principals) to analyze the NCLB proficiency rates in Kentucky. The general reaction to the sharp increase was "Wow!" One of the students shared her experience working with the assessment company to establish the new cut scores. We discussed changes to the system that might account for the dramatic increases, and how school leaders could "present" the data. That's when one of the students came up with the best spin ever. (Pay attention Lisa. Here's your angle.) The new assessment is a truer reflection of the content actually taught in Kentucky's schools, and therefore the 7-point spike in proficiency levels is a fairer measure of the actual progress Kentucky students have made than under the old test.

Terrific.

Now, if we can only get the NAEP data to bear that out....

We have a fundamental problem in our current accountability system. It's initial purpose was political (to garner the support of the business community for KERA's big price tag). It not focused so much on student achievement and curriculum. The focus was school accountability.

Better, would have been a assessment system that began with content and then folded the data into a value-added system, such as the one used in Tennessee. If CATS had been designed to improve instruction for individual students, it would have looked very different.

To their credit, and after the fact, many educators began to look at interim assessment systems that would help teachers identify learning problems early and intervene quickly. There has been a lot of good work done in the trenches, but the state system has become a hodgepoge under NCLB.

Interpreting test data to the public is a national problem, and "interested" parties will always spin the data to suit their own purposes. What we really need is a "disinterested" assessment/accountability reporting source.

As Ravitch understands, we need...

an independent, nonpartisan, professional audit agency to administer tests and report results to the public.

Such an agency should be staffed by testing professionals without a vested interest in whether the scores go up or down. Right now, when scores go down, the public is told that the test was harder this year - but when scores rise, state officials never speculate that the test might have been easier. Instead, they high-five one another and congratulate the state Board...for their wise policies and programs.

What the public needs are the facts. No spin, no creative explanations, no cherry-picking of data for nuggets of good news.

Just the facts.

I may even know the right folks for the job. I understand Ben Oldham at Georgetown College has recruited Skip Kifer and Tom Guskey to form an assessment center at Gtown. I studied under Ben and Skip and am familiar (along with most of the national academic community) with Tom's work. I can't think of a more valuable state resource to guide a more effective and fair assessment program that the guys at Georgetown.

KENTUCKY SCHOOL DATA from the 2007 NAEP Exam

Student Characteristics
Number enrolled: 679,878
Percent in Title I schools: 60.6%
With Individualized Education Programs (IEP): 16.0%
Percent in limited-English proficiency programs: 1.5%
Percent eligible for free/reduced lunch: 52.4%

School/District Characteristics
Number of school districts: 176
Number of schools: 1,426
Number of charter schools: N/A
Per-pupil expenditures: $7,2541
Pupil/teacher ratio: 16.0
Number of FTE teachers: 42,413

Racial/Ethnic Background
White: 86.3%1
Black: 10.6%1
Hispanic: 2.1%1
Asian/Pacific Islander: 0.9%
American Indian/Alaskan Native: 0.2%

Scale Scores for Mathematics

Kentucky vs. National Public

For more specific data check out NAEP's Data Explorer.

Cross State Comparison

Percent At or Above Proficient

4th Grade Mathematics

Where:
Blue = Kentucky
Green = States above Ky
Yellow = States about the same as Ky
Red = States below Ky

For more specific data check out NAEP's Data Explorer.