This from Susan Weston at the Prichard Blog:
There's been some talk recently about the number of Kentucky students excluded from NAEP participation based on disabilities. The chart [below], drawn from the NAEP "Nation's Report Card" documents for each subject, shows those rates. To me, it offers two clear messages.First, Kentucky has been doing something different from the rest of the country with NAEP reading...Second, NAEP science and mathematics have worked differently. Kentucky's exclusion rate has been a perfect match to the country in science, and it has differed by a single point in mathematics.So, what if we look at reading, and leave out students with disabilities for a moment? ... All the excluded students in the chart above above were in the group with disabilities. Even if the disability exclusion rate made a difference, it wouldn't change the scores for the students without disabilities. Here's that comparison.For both grades, Kentucky students scored above the national average by a statistically significant margin in reading.That is, Kentucky's recent record of relative NAEP success does not, in fact, evaporate when exclusion rates are considered...the notion of Kentucky as distinctively weak among the states is one we need to lay to rest...
4 comments:
The issue is testing accommodations. A student may have a plan that changes how he or she takes a test. The most common accommodation is allowing more time to take the test. Others include, for example, a human reader for blind students.
In reading, Kentucky provides accommodations that NAEP does not allow. That is why Kentucky's exclusion rate is higher than average.
Varied exclusion rates for different states produce problems in interpreting scores. If Kentucky's 5% difference from the national rate contains only low scores, the Kentucky mean goes down. If all of those scores fall in the bottom tail of the distribution, the downward adjustment would be substantial.
It is not clear, however, how one would determine where the scores might be. For any of a number of reasons, it is unlikely that each of those hypothetical 5% scores is extremely low.
Dr. Kifer’s somewhat confusing post is correct, though incomplete, about one point – Kentucky’s nation-leading exclusion rates for students with learning disabilities are definitely related to our state’s use of an unreasonable testing accommodation not allowed on NAEP reading assessments.
Such inflation has been present in Kentucky’s NAEP reading since 1998 and makes comparison of Kentucky’s scores to other states problematic.
As Kifer points out, no-one knows exactly how the excluded students would score if they took NAEP reading. However, there are some pretty interesting clues regarding this, information that Kifer and Prichard both overlook. Given the ready availability of the information, such oversight is surprising.
During the past six months the Kentucky Board of Education has engaged in extensive and very public discussions about NAEP exclusion and why it is so excessive in Kentucky’s reading samples (which were the highest for any state in 2011 NAEP for both 4th and 8th grade). Those board discussions make it clear: our high rate of NAEP exclusion involves a long-running testing accommodation used in Kentucky that allows providing special education students with readers for the state’s reading assessments.
Keep this in mind: learning disabled students who get readers on Kentucky’s state tests must routinely receive the same accommodation on all classroom tests they take. The provision must be written into their Individual Education Plans (IEP). Clearly, the vast majority of the learning disabled students who receive this accommodation are very poor readers, perhaps mostly even non-readers. These students never get a real, printed text decoding and comprehension test, which is exactly what the NAEP administers. Their IEPs don’t allow real reading assessment.
If students who routinely get readers on all of their tests suddenly were to take the challenging NAEP reading assessment without readers, it would be very surprising to see anything other than a very, very low overall group score. Many of these students would probably react very emotionally if they were suddenly required to demonstrate a skill they previously never had to demonstrate. Probably, many of these students would produce scores at or very close to a non-performance level.
By the way, let’s clarify Kifer’s confusing comment about how excessive exclusion impacts scores. NAEP reading scores actually being reported for Kentucky students who do take this federal test are HIGHER, not lower, than they would be if the excluded students also were tested and included in the average. Exclusion creates inflated, not depressed, scores.
So, there is very good reason for concern that Kentucky’s reported NAEP reading scores have notable inflation. You just have to know all the facts.
Speaking of knowing all the facts, I’ll have more to say about Prichard’s very weak post later this evening.
Mr. Innes should learn how to read. He should attack what I said.
Also, if he wishes to look at research on this issue I could locate some for him. Of course, since it doesn't conform to his weird notions, he would find it confusing.
Now that a couple of supporting posts are on line at www.bipps.org/bipps-blog, let’s talk about some really serious problems with Prichard’s blog.
For one thing, Prichard’s blog doesn’t contain any mention of the very well-known major cause of Kentucky’s excessive exclusion rates on NAEP reading assessments. Lots of learning disabled students in Kentucky have IEPs that stipulate all tests, including reading tests, are to be read to those students. That accommodation is totally incompatible with the construct tested by the NAEP, which is true printed text decoding and comprehension. This leads to Kentucky’s nation-leading levels of exclusion on NAEP.
This accommodation is also hostile to the impacted students getting much, if any, real instruction in reading, as well. The accommodation creates negative inducements to teach reading to these special students.
The situation is no secret; the Kentucky Board of Education has been talking about the problem for half a year.
Prichard’s NAEP analysis of scores for students who are not learning disabled also ignores years of advice on how NAEP score comparisons need to be conducted. As the 2011 NAEP Reading Report Card puts it:
“Differences in states’ demographic makeup should be taken into consideration when interpreting state results.”
Prichard undertakes no such demographic analysis.
As soon as the NAEP 2011 reading results are disaggregated by race, Kentucky’s supposed advantage in NAEP reading evaporates for the state’s dominant racial population. Instead of scoring statistically significantly higher than the national average, Kentucky’s white, non-learning disabled students score statistically significantly lower than the national average for whites in both fourth and eighth grade.
There is a lot more information available on this; your interested readers can find that in two blog items here:
http://www.bipps.org/naep-exclusion-rates-what-prichard-didn%e2%80%99t-say-%e2%80%93-part-1/
and here:
http://www.bipps.org/naep-exclusion-rates-what-prichard-didn%e2%80%99t-say-%e2%80%93-part-2/
Your readers could benefit by first learning the rest of the story before blindly citing Prichard’s blog to anyone.
Post a Comment