The Bluegrass Institute
recently picked a fight with the
folks over at the Kentucky Long-Term Policy Research Center.
At issue is whether Kentucky's system of public education (preschool through college) has improved in recent years and how Kentucky's schools compared to those in other states.
It all started when the
KLTPRC issued its recent study indicating that Kentucky had made educational progress since 1992. This must have rubbed the Bluegrass Institute folks the wrong way. They spent some time
looking for holes in the report.
BGI's response
criticized the KLTPRC report alleging: misstatements of NAEP data, inappropriate use of dropout data, using the ACT to rank states, and more.
But a blogger on the NAEP beat,
Susan Ohanian suggested to BGI that their biases were showing.
The Bluegrass Institute offers "free-market solutions to Kentucky's most pressing problems." That, of course, puts them in the Choice camp, which is a euphemism for vouchers. Not that they are at all hesitant about proclaiming their love of vouchers: "Vouchers allow parents to choose better schools."
So are they offering a scholarly critique of Kentucky's testing program--or a manifesto to drain public confidence in their schools?
As the Courier-Journal demonstrated recently, this kind of claim is a fundamental problem for any "Think Tank." It is incredibly hard for anyone to ignore their own biases, especially if they believe strongly in them.
Lord knows, I have biases.
I don't think I could fully trust anyone who told me they had no biases. I just want to know what they are.
My biases come from a career as an elementary school administrator in two Kentucky counties before KERA, during its rocky implementation and after Sen. Gerald Neal's SB 168 (and its data disaggregation, which is significantly more powerful than I first realized; and which predates NCLB.) Since graduation I have taught at UK, EKU and the private Georgetown College.
I am biased by the sum of my experiences; and compelling data.
I don't need reports from anyone to convince me that significant improvements have been made in Kentucky's system of public schools. Like other social institutions, they are far from perfect. But given the relatively modest financial resources Kentucky invests, on the whole, Kentucky should be right proud of its schools.
As
education historian and scholar Diane Ravitch understands, what we really need is...
"...an independent, nonpartisan, professional audit agency to administer tests and report results to the public.
Such an agency should be staffed by testing professionals without a vested interest in whether the scores go up or down. Right now, when scores go down, the public is told that the test was harder this year - but when scores rise, state officials never speculate that the test might have been easier. Instead, they high-five one another and congratulate the state Board...for their wise policies and programs.
What the public needs are the facts. No spin, no creative explanations, no cherry-picking of data for nuggets of good news.
Just the facts.
In her presentation of the KLTPRC report at the group's
recent conference (video), Dr Amy Watts presented data in support of two propositions: 1) that public education, writ large, in Kentucky is progressing, and that 2) Kentucky's relative standing among the states is improving.
The following is paraphrased from Watts' presentation.
On an incomplete data set, KLTRPC derived an Index considering an adjusted set of 11 indicators; which revealed across-the-board improvements roughly from the 43rd rank to the 34th. This report mirrors two previous reports done outside of Kentucky...Kentucky was 34th in Education Week's Quality Counts 2007 Achievement Index and was 31st in the Morgan Quinto 2006-2007 Smartest State Index."
The KLTPRC data set looks at two kinds of indicators: educational attainment and student Achievement.
Educational attainment indicators
These data refer to Kentucky adults (HS diploma, 2-year degree, Bachelor's degree) and show an upward trend...although a relatively flat one, that leaves Kentucky ranked near the bottom of US states.
Dropout rate shows progress; declining (which is good) about 2% over the last 7 years.
(KLTPRC used the definition insisted upon by the National Center of Educational Statistics: "The percent of high school students who left high school between the beginning of one school year, and the beginning of the next, without earning a high school diploma or its equivalent.")
Since not all states reported the data for all years, a percentile indicator was used
to show progress over time. That measure showed that Kentucky made progress
while outperforming other states; growing from the 32nd percentile to the 61st.
Student Achievement data
Rather than looking at the scores, these data focus on the percentage of students who performed at the proficient level or above - which is Kentucky's goal. The CATS Accountability Index grew over time for all levels (Elem, Middle & HS).
The 4th and 8th grade Reading, Math and Science data are reported using the National Assessment of Educational Progress. To put Kentucky's growth into a national context the scores are reported using a percentile scale.
4th grade Reading improved (in percentile terms, from about 25th to about 50th).
For 8th graders the trend is flat - not much growth - and in percentile terms Kentucky has fallen behind - dropping from average to below average).
In Math, 4th grade achievement is up from 13 percent proficient to 31%, but the gap between Kentucky's progress and that of the nation has widened over time (we're falling behind.)
8th grade Math is up from 10% up to 27% proficient, which is roughly equivalent to the rest of the nation.
In percentile terms, while Kentucky has progressed in 4th grade Math, it has not kept pace with the national average and Kentucky has fallen behind from a 20th percentile ranking, down now to the 16th percentile.
In 8th grade math the story is more mixed. Kentucky was making progress up to the year 2000, rising from 17th percentile to 37th, but has fallen off since that time. Now it's down to 27th. This is a concern...as math is seen by economists as a crucial area for the economic future of the state.
Science is a bright spot for Kentucky with 4th graders rising from average performance rankings up to the 81st percentile, while 8th grade rankings rose from below average to average.
To gauge high school performance, ACT composite scores were used. On the ACT, Kentucky has shown improvement while narrowing the gap between Kentucky and
the nation. This has resulted in a higher standing for Kentucky, up from 10th percentile rank to 24th.
Chris Derry: "You and I have discussed this, but I want to take these questions to a higher level; and frankly because we're on the Internet. I would like for the basis of discussing the policy consequences of some of the data you've used - might lead people to conclude otherwise, if other data were used (sic).
For instance, you've used the data point of dropout rates, and in fact...the Kentucky Auditor Crit Luallen came out with
a report that challenged the accuracy of the Kentucky...Department of Education's dropout rate, which are the dropout rates used in your calculation.
(Luallen criticized the student information system with preventing accurate calculations - off by as much as 30%. That system is currently being replaced as per Luallen's 2006 suggestion.)
So I would say, if the Auditor has questioned that accuracy, why would you introduce it into this report?
Amy Watts: Regarding the dropout rate ...the dropout rate that I'm using here is a consistent definition that is used over time and the NCES, the National Center for Education Statistics, and officials at KDE work diligently to ensure the accuracy of these data.
In fact, the [NCES] can't even compute a national dropout rate because several of the states don't adhere to their very strict definition.
So in this context, with these data, we are able to place Kentucky in the national setting and how we've progressed in this particular indicator over time based on the fact that these are nationally recognized standards of data collection and methodology and that is consistent. That's the biggest benefit of using that particular indicator.
As far as the Auditor's report, it really did help to point out where they can ...continue to make sure that ...the systems that they already have, the ...internal audit systems that they already have in place, to ensure ...the accuracy and ...how clean these data are, that that is maintained. So it helped highlight some places where they can really strengthen this and make it even better than it already is.
Derry: ...the ACT is an essential test that is really is an international standard because of so many students both abroad and domestically, who are required to take that test, to qualify to enter college. ...And Kentucky, I believe, Illinois and Colorado are the only states, beginning this year, in which all students in those states are required to take the ACT.
But because other states do not have that 100% requirement, and because their percentages of test takers is so low, ACT has cautioned against comparing one state versus another; yet you've done that in this report.
Watts: The ACT, does not warn against using these data in the context in which we have used them.
They've cautioned that you have to take into account demographic differences across that states and we were very transparent in the data that we used here. You can go to the ACT website and the methodology and some of these cautions that you've talked about are made readily apparent there.
So, definitely, we were very transparent in these data so that you could understand exactly what went into calculating the indexes used here and the rankings that we found.
Derry: ...a big part of the emphasis in this report is on census data; and looking at the age category of age 25 to 64.
As indicated in your report you say...'sufficient time has passed since the Kentucky Education Reform Act of 1990 and the Postsecondary Education Improvement Act of 1997 to ask, 'Are we making educational progress in Kentucky and if so are we gaining relative to the nation?'
When you use census data you're including ...hundred of thousands of people who were not in KERA. And yet, you're making the claim in this, that they should be included as an indication of the progress under KERA.
Watts: Yes the census data, those are, again, readily accepted indicators of educational attainment and progress that are used time and time again by tons of different researchers.
So ...it seemed like it would be ...incomplete without this data in an index of educational progress.
...They do not reflect directly upon how KERA or the Postsecondary Education
Act of 1997 have contributed to where we've come.
This is kind of a spot check of; 'let's look. Let's see where we are now. Let's see where we were. Now, let's begin to see where we're going based on these data that really give us an overall accurate picture of what's happening in Kentucky.'
Before any of this happened, I had already read the report, saw its construction as well as its transparency.
Kentucky's funding levels may still be in the basement, and some areas of progress are certainly stronger than others, but overall student achievement gains are undeniable.
Student achievement is a lagging indicator and Kentucky is realizing the benefits of earlier (and continued) effort. This would seem to be the very definition of "an efficient system of common schools throughout the state," which is the legislature's constitutional mandate.
Viewed as a cost/benefit ratio, Kentucky schools are providing its citizens a better educational program than the state has a right to expect.
This is essentially what Judge Thomas Wingate alluded to when he struck down the Council for Better Education's most recent effort to force the General Assembly to keep its commitment to school reform.
For the amount of oil we put in the engine, this seventeen-year-old Chevy is runing pretty well.
