The Commish on SEEK and Accountability Issues

On Friday, Jan. 7, Education Commissioner Terry Holliday presented a webcast on SEEK and Accountability Issues, directed at local school district superintendents. That webcast has been archived.

The archived version of the webcast is available at the following links:

The webcast PowerPoint presentation --
video and audio --
downloadable audio podcast --

Holliday Calls for Refocus on Student Success

On September 23, the Kentucky Department of Education will release reports that preview Kentucky's proposed accountability model- one that focuses on college and career readiness, proficiency, growth and closing gaps.

Commissioner Terry Holliday says the new model will be an improvement over the Adequate Yearly Progress (AYP) measures mandated by No Child Left Behind.

How could it be anything other than better? AYP took everything we know about research-based student assessment, and threw it out the window in favor of an approach designed to identify failure wherever it could be found - even in places where it wasn't.

Now, Holliday has a more pleasant recollection of NCLB's history than I do. On Dr H's blog the Commish says,

As an educator, I can’t deny that the purpose of NCLB is very laudable. We all want every child to be successful. The measure of AYP was never intended to label schools as failures, nor have teachers felt like they are failures. However, each year around this time, news reports all over the state and nation come out about NCLB and how schools have failed to make the grade or how schools have come up short on the NCLB scale.

While I agree that NCLB's aspiration to improve the achievement of all students is the right goal, the underlying motivation of NCLB's infamous AYP has been questioned by many.

I quibble because, as I recall, initially NCLB actually used the term "Failing" to describe schools that did not make AYP - language that was later softened after many complaints that questioned the true motivation behind NCLB. Was NCLB really a cynical effort to label as many pubic schools as possible as "failures" in support of a move toward privatizing public education?

And, I suspect many teachers were made to feel like failures along the way. As principals and superintendents bore down on the entire faculty, on many occassions, the "true motivations" of whole groups of teachers were questioned, as though their attitudes were perpetuating achievement gaps. Of course, in some percentage of cases that was surely true - but certainly not in most cases. NCLB cast its blame widely and unfairly, much as some school administrators did.

But that is not to downplay the importance of a new accountability system - one that holds standards high while valuing growth. While still not perfect, such a system would be more realistic and more fair to Kentucky teachers.

Early drafts of the new testing system, shared with the state board of education late last year, began in third grade with annual reading and math tests, periodic science, social studies and on-demand writing tests, ACT exams, and end of course exams for certain high school classes.

This is all a part of a long transition from a pre-KERA teacher-centered approach to teaching and testing to a standards-centered approach. It is also part of a transition from the days when test scores were withheld from the public to today's approach of publicizing every piece of school performance data we can get our hands on. Don't get me wrong, I like a lot of data and I like it public. But the data must be used appropriately. That's something many folks fail to do, just like NCLB.

The new system will collect individual student achievement data over time and look at student performance individually.

Now the Commish would like our help chatting up the politicians.

No Child Left Behind is coming up for re-authorization in the next year or so. I encourage all citizens to help teachers and staff communicate to our politicians that we support helping children be successful; however, we need other ways to announce the results of our efforts and the progress, not failure, of our schools. In Kentucky, the requirements of Senate Bill 1 will serve as a model for the nation as we consider reauthorization of NCLB.

Holliday is referring to an accountability system that does not forget to value student growth. And that's a good thing to value for every child in any school.

Build a Value-Added Assessment System for P-12 Before Moving on to Teacher Prep Institutions

"A key assumption of using test scores to judge teachers

is that students are randomly assigned,

first, to schools, and, second, to classes.

Neither is true."

Recently, there has been increased talk of holding teacher preparation institutions accountable for the performance of Kentucky teachers - in effect holding colleges accountable for student achievement in the state's public schools.

It's a good motivation, but in practice, there's a lot wrong with the idea.

It simply takes all of the problems associated with teacher-accountability-by-standardized-test-score, a central tenant of the Obama Administration, and multiplies that unfairness many times over. While it may feel good to teachers to know that they are not the only ones tied to an unfair system, it does not fix what's broken.

But it may become a better idea, if those who are enthusiastic for change slow down long enough to create a system that addresses fairness by trying to address the many technical problems, and that might actually work to an acceptable degree. That means building Kentucky's new accountability system from the bottom up:

• Curriculum standards, first
• Assessments built on those standards
• Lessons taught on those standards (in that order)
• Assessment results fed into a value-added accountability system that is sensitive to the great variability in children and one that establishes an individual baseline for each child and controls (to the degree possible) for that demographic variability.
• At this point the public should consider a cost benefit ratio: the investment, in relation to the reliability and validity of the system.
• Professionals should gauge the practical limitations of social science research, both quantitative and qualitative. Unlike the natural sciences, our variables refuse to hold still; which bears heavily on the precision of the system.
• Individual student progress is measured from each individual student's established baseline
• Individual student achievement data is collected over the student's entire academic career
• The teacher accountability system should be quantitative and qualitative.
• The principal accountability system should be quantitative and qualitative
• There should be a planned review of the accountability system after collecting about three years of data (barring some unforeseen data catastrophe), with any significant adjustments to the accountability formula made at that time.
• Then, and only then, a teacher preparation institute accountability system should be ready to track the performance of teachers who graduated from each institute, based on that value-added system; and that system should be quantitative and qualitative

Tying teacher accountability to test scores is going to happen. Fairly or unfairly, states are now scrambling to meet the strictures of RTTT grant funding. But desire for much needed funding doesn't make the myriad technical problems go away.

The Century Foundation recently outlined their Eight Reasons Not to Tie Teacher Pay to Standardized Test Results. In a nutshell,

Reason #1: Tying test scores to teacher compensation suggests that teachers are holding back on using their experience, expertise, and time because they are not being paid for the extra effort.

Reason # 2: The standardized tests in most states are lousy and so are the standards they are designed to measure.

Reason #3: The idea of compensating teachers individually in order to differentiate their performance from their school colleagues defeats a principal tenet of good instruction—that teachers need to learn from one another to solve difficult pedagogical challenges.

Reason #4: Most teachers do not teach a grade or subject that is subject to standardized testing.

Reason # 5: Even reliable standardized tests are valid only when they are used for their intended purposes.

Reason #6: A key assumption of using test scores to judge teachers is that students are randomly assigned, first, to schools, and, second, to classes. Neither is true.

Reason #7: State data systems are in their infancy. It turns out that it is harder, is more expensive, and takes longer for states to produce reliable, accurate, and secure longitudinal data on students and teachers than widely assumed.

Reason #8: The rationale for tying tests to compensation is not clear.

The non-profit, non-partisan Century Foundation argues that No Child Left Behind has narrowed instruction too much already, that one does not need a standardized test to identify the worst and best teachers, and no system could be constructed with sufficient precision to withstand the inevitable court challenges.

At the heart of the argument in favor of tying pay to test scores is the idea that it will improve practice. But that can only work if the economy provides the anticipated financial incentives. In this recession,

"if teacher compensation does not keep up with inflation because of poor student performance, then teachers will . . . what? Work harder? Dig deeper? Stay longer? There is no evidence that such measures improve instructional practices or student outcomes."

Secretary Duncan is correct when he catalogues the weaknesses in the present system of preparing, recruiting, mentoring, retaining, inspiring, retraining, promoting, and dismissing teachers. but this is an idea that is way ahead of just about everything it would need to have even a chance of working fairly and reliably, if at all.

Viewpoint: SB 1 and raising standards

Over at Prichard, Susan offers some observations on Senate Bill 1 and its opposition to Kentucky's lowered standards.

Since 1990, Kentucky testing has had three scoring systems, each easier than the last. By easier, I mean two things.

First, we set new "cut points" when we changed from 1998 KIRIS to 1999 CATS and when we changed from the 1999-2006 version of CATS to the revised 2007 and 2008 CATS. At the moment that we put the new cut points to work, many more
students were counted as proficient.

Second, we asked for less complex kinds of student performance. From KIRIS to CATS, we dropped performance events and added multiple choice. From "CATS I" to "CATS II" we gave multiple choice more weight than it had before.

Senate Bill 1 demands something different. It calls for P-12 standards to be aligned with college expectations...

Sexton Decries Early Death of Accountability

The Prichard Committee has been generally supportive (with reservations) of efforts in the House to produce an alternative to the Senate Bill 1. Today they are wondering what hit them.

Somewhere, Sharron Oxendine is smiling.

This from Bob Sexton:

CATS: What today's news stories missed?

In a surprise last minute move on legislation affecting testing and accountability (incorporated now in SB 1), the House amended the bill to "suspend" accountability in most subjects for at least 3 years starting this spring. The consequences will be highly disappointing for parents, students, and the public because these measures of school progress underpin Kentucky's push for improved student learning --- and support the case for adequate school funding.

But now it will be more difficult for parents, taxpayers and teachers to figure out how their schools are progressing in core subjects like writing, science, history, geography economics and civics, and there will be no measure of how they are doing overall (calculated by the index of all the subjects).

All they will know under "suspended" accountability is what No Child Left Behind testing in math and reading tells them, which isn’t enough. There’s a strong chance too that all the other subjects---the ones that "don't count"--- will get less attention in the classroom.

This is a hugely ironic outcome for the teachers who lobbied for this change. Teachers who have bitterly complained about NCLB now get nothing but NCLB! They also saw their best case for adequate funding --- evidence of making progress with children --- thrown in the garbage. Be careful what you wish for, as the old saying goes.

So with data on school results "suspended" what's next? Who knows. What information will school boards and superintendents use to push local schools to improve? Will we suspend aid for struggling schools and efforts to reduce achievement gaps? Will we suspend calls for adequate school funding?

Stay tuned --- schools and kids are in for a bumpy ride.

SOURCE: Prichard release

Upper Elementary Grades Bear the Brunt of Accountability

This from the Phi Delta Kappan:

Educators claim that accountability forces them to narrow the curriculum. But a comparison of teachers' schedules before and after NCLB shows that little has changed.

Upper elementary teachers won't be surprised to learn that in every state, students enrolled in grades 3 through 8 bear the brunt of educational accountability.

All states test all students at these grade levels in English/language arts and mathematics (Toye et al. 2006). Furthermore, an increasing number of states are testing students at selected elementary and middle school grade levels in science and, to a lesser extent, in social studies.

Although all states test high school students, testing is done less uniformly; states vary on the grades and subjects tested. Finally, only six states test students at grade levels lower than grade 3 (California, Delaware, Georgia, Idaho, Iowa, Mississippi) (Toye et al. 2006). Although the burden of educational accountability in the United States rests with children ages 8 through 12, not much research has evaluated the impact of educational accountability on schools, teachers, and students.

Teachers provide the bulk of the evidence of the impact of accountability on teaching. Not surprisingly, teachers report that accountability has adversely affected how they teach, impacting curriculum, quality of instruction, and instructional time...

Teachers report that the major effect of educational accountability on the curriculum has been "narrowing" the curriculum. Hutton, Curtis, and Burstein, for example, conclude that the emphasis on high-stakes testing "is impacting how the core subjects are being taught, with the social studies curriculum being relegated to the background and only taught when there is time" (2006, p. 18). Similarly, M. Gail Jones and her colleagues (1999) state that, as a result of the ABC program in North Carolina, the teachers they interviewed spent most of the school day preparing students in the basics, that is, reading, writing, and arithmetic.

It's true that different subjects in the elementary school receive different emphasis in terms of allocated time. But it's not true that educational accountability legislation has caused this...

Salvaging Accountability

This from Thomas Toch & Douglas N. Harris in Ed Week:

George W. Bush rode to the White House pledging high standards for all students. He’ll leave Washington with the nation’s public education system focused on teaching basic skills to disadvantaged student populations, with the United States lagging in international comparisons of educational attainment, and with his signature education law plagued by so many problems and mired in so much controversy that it has put at serious risk two decades of work to improve public schooling by making educators accountable for their students’ success.

The most important thing Barack Obama or John McCain could do quickly to salvage the accountability movement is change the way that the federal No Child Left Behind Act judges schools. Not by abandoning NCLB’s focus on students’ meeting standards, a move that would be unwise on both policy and political grounds, but by making the law a more legitimate report card of school performance, one that provides a fair and accurate gauge of educators’ contribution to their students’ achievement. Since its inception, NCLB has instead held schools responsible for factors they can’t control and perversely encouraged states to set standards low....

A New Approach to Accountability in Kentucky

From time to time I have referred vaguely to a "value-added"system for assessing the effectiveness of Kentucky schools. Today the Dallas Morning News gives us a peek at the kind of thing I have in mind.

If I had a magic wand, testing would exist to inform better instruction and provide parents with annual information about the progress of their child. But accountability would look something like this:

'School Effectiveness Index' changes wisdom

on what makes a good Dallas ISD campus

'Effectiveness index' considers demographics in measuring campuses

How good is your kid's school?

It seems like a simple question, but factors like student backgrounds can make clear-cut answers elusive.

What if you could find a way to evaluate schools that doesn't penalize them for their students' language barriers, lack of parental involvement or other social factors that impact learning?

With virtually no fanfare, the Dallas Independent School District does exactly that. Every year, it produces a "School Effectiveness Index," a rating that levels the playing field between schools, no matter where their students come from or what they lack in life.

DISD has been calculating the scores since 1992 but has never publicized the ratings, even though parents are hungry for information showing how their neighborhood schools stack up. The Dallas Morning News recently collected nine years of effectiveness scores from the district and has made them available at dallasnews.com/disdblog.

One of the district's top researchers, who helped develop the ratings, cautioned parents not to view the scores in isolation. Many other factors also are important, he said, such as personal interactions with teachers.

"From a parent standpoint, I would be far less concerned about the school [ratings] than I would be about what my kids are telling me about their teacher," said Robert Mendro, the district's director of evaluation. "A kid will tell them about which teacher requires homework, which teacher is tough."

The News recently shared the effectiveness scores with about 40 parents, educators and community members. Some found the scores helpful in a general sense.

"Rather than choosing the highest-ranking school in a measure of best, I would look for consistency in the top 10 percent of performing schools," said parent Randy Hazlett, who has a child at Townview Magnet Center. "Still, the rating system will reward those schools which accept students from low-performing feeder schools and significantly boost their test scores."

The scores turn conventional wisdom about which schools are "good" on its head. For example, the most effective school in the district last year, according to the ratings, was Rusk Middle School. By way of comparison, the Talented and Gifted magnet school – widely lauded as one of the best public schools in America – ranked No. 29.

That happens because the ratings compare schools by isolating the impact teachers have on student achievement. The ratings measure how far a student advances with his teacher in one year.

Through a complex statistical analysis, the district isolates student characteristics that affect learning, such as family income, ethnicity and English mastery. A school's score is designed to eliminate advantages campuses gain from the social and demographical characteristics of their students.

That's a completely different approach from the state's accountability system, which looks at how many students pass annual standardized exams. Low-income students tend to score lower on such tests.The district's "value-added" ratings focus on how much students learn in a year, not whether they pass certain tests, such as the TAKS, Dr. Mendro said.

Such calculations are a growing trend in education as researchers try to determine how far schools push their kids along the learning curve...