More on multiple measures and NCLB

An interesting post, and comments, over at This Week in Education on multiple measures.

It seems that US Rep and education committee Chairman George Miller's speech at the National Press Club last month struck a nerve with conservative Stanford researcher Erik Hanushek.

Miller had signalled his openness to looking at some additional measures of success, like high school graduation rate in assessing the success of our schools. This make perfectly good sense to me.

We ought to clearly define it so that "graduation" in Georgia means the same thing it does in Kentucky, then keep the data and track it.

I'm not a fan of Hanushek's. Some of his sworn testimony in school funding cases has appeared to me to be deliberately stupid. I say deliberately, because clearly, he is not, in fact, stupid. Sometimes, what he says is. To hear him tell it, money just doesn't matter much when it comes to quality schooling - it can't be shown scientifically to have an impact. But I'll bet he understands that it costs you more if you want fries with that burger. If he wants the BMW with the navigational system, he knows it will cost him more. Just like hiring an extra high quality teacher who will make a difference for a bunch of kids. But I digress...

This time, I think Hanushek is a little smarter.

Haunshek is worried by some of the discussion over multiple measures and argues for measures that are clear and, in fact, measurable. Me too. Some argue that school assessment ought to include such measures as portfolio assessment, writing assessments and public speaking. Talk about an undefined measures. These may well be the right idea - but as assessment goes - it will be difficult to agree that one teacher's assessment of student writing would match anther's.

It is an inter-rater reliability nightmare.

Hanushek also correctly points out the powerful effect that disaggregating student achievement data has had on forcing schools to focus on all groups of students, rather than hiding behind mean scores while persistently allowing the same groups of children to fail year after year...without accountability.

Without arguing whether federal imposition of NCLB's federal accountability system is actually constitutional - it would be the right way to go, even under a state accountability system.

And Hanushek concedes that adding a growth model improves NCLB.

But Hanushek only wants to look at data that focuses on "basic cognitive skills."

Why?

We collect a ton of data about the performance of our schools. School leaders across the country pour over that data (some of it demographic like graduation rates) gauging the relative success of school programs; looking for trends. The data tell a story.

Part of that story is what it will take for all school and all students to be successful; what it takes to close achievement gaps.

Perhaps what Hanushek really fears from an expanded data set, is not so much that it might damage kids, but what it might reveal; that it's not enough for us to simply declare our best aspirations - that no child will be left behind - reaching that goal is going to require a greater, more comprehensive effort, than simply putting the thumb screws to America's principals and teachers.

Of course, some of the broader discussion of multiple measures is at cross purposes.

Hanushek is thinking of an accountability system.

Many teachers look at assessment - quite properly - as a useful tool for guiding and adjusting instruction. The teachers are correct. But when policy makers wanted accountability, their heads were in a different place.

Of course, it would have been best if we first built a strong instructional assessment system which started with the content and then provided teachers timely feedback on their student's progress on specific objectives... and then, built an assessment system around that.

But it didn't work that way for political reasons.

Policy makers wanted accountability more than improved instruction...so that's where they put their effort. The two ought to be the same, but they are not.

Sound bite style reports: More confusion with multiple measures.

This from Edspresso.

Exposing an Ugly Paradox

Question: How can 6 Nevada high schools in one district get NCLB "F"s and make the "Top 5% of Public Schools" in Newsweek?

The Nevada District's Answer: NCLB including special education programs equally

Edspresso's Answer: The achievement gap

First, consider what Newsweek really means by Top 5%. Newsweek awards this title by taking IB/AP/Cambridge test scores, and dividing by the number of graduating seniors, so the "Top 5% of Public Schools" is really just the "Top 5% of Public Schools' Smartest Kids Who Are Graduating" in the nation.

What's really ugly is that this Nevada district administration blames it on the special education students' scores (13.2% of the district's student body). "More than half of the Washoe County schools getting a failing grade under No Child Left Behind standards did so because of special-education programs, school trustees were told by staff Tuesday."The article uses the Top 5% title to suggest that NCLB isn't grading them properly.

But the whole point is that this paradox is a perfect example of the achievement gap.

Aren't special education students the ones that commonly get left behind? This district's schools are top performing on AP and other college prep courses across the nation; at the same time they're failing the students at the bottom.

The point of NCLB is to expose any problems with even the smallest of groups - problems that used to be completely masked among standardized tests that simply "melted" everyone's scores together.

Superintendant Paul Dugan said, "If we don't make it in one category the whole school is categorized as not making adequate yearly progress."And conversely, if you do make it in one category (IB/AP exams), the whole school is categorized as a Top 5% school nationwide.

Good job, smart students' teachers and smart students' curriculum and smart students' administrators... now what are you going to do about those you left behind?

What Testing Guru Bill Sanders Really Meant About Multiple Measures

This from Alexander Russo at This Week in Education.

Once in a while, I actually do some reporting, and today I happened to talk Prof. William Sanders, the testing guru whose recent letter to Congressman Miller was leaked to the press and seemed (according to an Ed Daily story) to put Sanders squarely against Miller's proposed use of multiple measures in AYP.

Well, it turns out that Sanders is against the use of portfolios and classroom observations that are often called multiple measures, but not against end of course tests, college entrance tests, and the like that he thinks Miller is talking about. "Those things have a place," says Sanders, who points out that they are already part of the growth model projections that he has developed and are being used in some pilot states.

To those who are concerned about the complexity and transparency of both the current AYP and proposed changes, Sanders says such intricacies are the price of a nuanced and reliable rating system. "A simple system could be developed," he says, noting that some states are going that direction, "but it would be less reliable and more biased [than a more complex one]."

His main accountability concern, however, is not so much that the current AYP relies on "a single test" (a description he says irks him and ignores the fact that there are three years of tests and hundreds if not thousands of test item responses that go into each year's AYP calculations), but rather that it encourages too much focus on lower-performing kids rather than "early high-achieving kids" who get ignored. He proposes a rating system that evaluates schools not only on reducing the achievement gap but also on helping already-proficient kids do even better -- [See] ... Tennessee ...and Nevada.

This from the Center for Greater Philadelphia at the University of Pennsylvania:

Value-Added Assessment in Tennessee

Tennessee is the state most strongly identified with value-added assessment. Its system dates back to 1992, when value-added was implemented as an integral part of a comprehensive education reform measure. Using a complex statistical method developed by Dr. William Sanders, then a statistician at the University of Tennessee , the Tennessee Value-Added Assessment System (TVAAS) provides:

Data to the public on the performance of districts and schools, and data for appropriate administrators on the performance of teachers;

Information to teachers, parents and the public on how schools are doing in helping each child make academic gains each year;

Information to school administrators to help identify weaknesses in even the strongest schools.

TVAAS is a statistical methodology that begins with testing each student in each grade in a number of subjects. Through 1997, Tennessee tested second through eighth grades in Reading , Math, Language, Science, and Social Studies. Tennessee began testing grades three through eight in 1998.

The TVAAS statistical model aggregates student growth increases using a design that accommodates missing data. Because of a philosophical belief that schools should insure that all students progress at equivalent rates, no matter their disadvantages, the model does not include other data on students.

In a report by the Council of Chief State School Officers, Tennessee's 8% increase in math and science scores was linked to TVAAS. In addition, Tennessee is one of the few states that have shown improvement on the National Assessment of Education Progress since TVAAS was implemented in 1992.

The state has both rewards, aid, and sanctions linked to its school rating system.There is no specific value-added teacher evaluation as part of Tennessee 's accountability system, but school administrators have access to teacher level data that can be used to improve instruction. Value-added scores can be used for up to 8 percent of a teacher's evaluation.

The incentive funds are only available to schools, not to teachers.

Although teachers and administrators were suspicious at first, they are now finding they can actually use TVAAS to improve teaching, something that no other accountability system has afforded.

Russo sees shift toward multiple measures in Miller's speech on NCLB

Alexander Russo at This Week in Education, opines that George Miller's speech this week could mean a broken heart for Pro-NCLB groups.

Is Miller Breaking Up With Pro-NCLB Groups?

Perhaps the most compelling interpretation I've heard of the Miller speech ...is that Miller was emphasizing that (a) the reauthorization process is still moving along despite recent delays, and (b) multiple measures are going to be part of his bill no matter what.

If multiple measures are definitely in, then this represents the first big break by Miller from the groups that helped craft and defend NCLB 1.0 and the EdSec -- and a big win symbolically at least for NCLB 1.0 critics like the NEA who have been clamoring for years now that annual standardized tests were a bad way to go and, more recently, working hard on freshman lawmakers that previous compromises (like the growth model or the idea of treating schools that just miss AYP one year differently from those that miss it all the time, every year) aren't enough.