Showing posts with label Richard Innes. Show all posts
Showing posts with label Richard Innes. Show all posts

Monday, March 28, 2011

BREAKING: BIPPS Offers Good Advice to School Boards on Superintendent Vetting

It's been a while since the Bluegrass Institute has issued any kind of advice for school board members - or anyone else for that matter - that wasn't dripping with bias, obfucation or thinly veiled political motivations.

But today BIPPS released tips for school board members who are looking for new superintendents, and they are worth considering. I only found two places where BIPPS allowed their biases to slip in. That's pretty good for them.

Superintendent hiring: Advice to school boards

  • Tip #1: Understand that some will hide problems – but information is out there

  • Tip #2: Research before hiring a search firm

  • Tip #3: Make some phone calls yourself

  • Tip #4: Check the newspapers

  • Tip #5: Resumes are critical – check out everything

  • Tip #6: Beware of Secrecy

  • Tip #7: Ask informed questions – be respectful, but understand that softball questions won’t help you

  • Tip #8: Never forget, this is YOUR responsibility, not a search firm’s, not the public’s

  • Tip #9: Consider other resources

  • Tip #10: Other considerations

  • Tip #11: NEVER FORGET: The focus is on student preparation for college and careers
BIPPS says,
Included are pointers on how to use the Internet to find out about an applicant’s past performance, along with ideas on how to use media sources and suggestions on informed questions to ask candidates to help determine what education reforms they have successfully undertaken.

Richard Innes, the institute’s education analyst, says a lack of due diligence on the front end of a hiring process can lead to embarrassment, and even resignations, later on.
Innes is exactly correct about this. In fact, BIPPS and KSN&C worked pretty well together, in 2007, when the Kentucky Board of Education locked in on hiring Barbara Erwin despite emerging evidence of an impending trainwreck. The Board believed Ray & Associates, their search firm instead of certain prophets of doom and it turned out that the prophets were right.

Here's the postmortem on the Erwin affair from C-J.

A subsequent state board had a similar problem in 2009, but followed one of Innes's tips. When a board member asked a certain blogger I know to look into a rumor about Commissioner finalist Dennis Cheek, that blogger was able to help the board avoid a substantial political problem.

Wednesday, February 03, 2010

Shelly Berman on Charter Schools

Shortly after the 16 January KET show on charter schools I got a call from Dick Innes of the Bluegrass Institute. Innes apparently felt that I had "been a little rough" on BIPPS and that there were apparently some unnamed people who were thinking about picketing my home!?

Well shucks. First of all I'm never home, so I'd suggest my office at EKU. But I couldn't remember what it was I had said about them. At least, not recently - although whatever it was, I'm sure I meant it.

As usual, Innes and I agreed on some things and disagreed on others; but came at things from different directions.

At any rate, the conversation turned to the KET debate on charter schools (KET VIDEO). Bill Goodman's participants Rev. Jerry Stephenson, chair of the Kentucky Education Restoration Alliance, Jim Waters from BIPPS (who, now that I think of it, I have criticized Waters for his passing acquaintance with accuracy and his introduction of creationism into conversations about Kentucky schools - was that it?) KEA's Sharron Oxendine, and JCPS Superintendent Sheldon Berman.

Innes and I agreed that Berman's assertion that Charter Schools were a failed reform, was at least premature, if not inaccurate. Then Innes informed me that he had uncovered an earlier (and inconsistent) quote from a report Berman wrote:
"What we're saying is it's possible to have public schools and charter schools stand side by side and learn from each other."
Innes wrote about it over at his place. I said I'd look into it when I got the chance.

The public record on Sheldon Bernman and charter schools seems to begin in 2004 when he served as superintendent of the Hudson School District in Massachusetts. At that time the Massachusetts Board of Education was considering their first seven charter schools. (Boston Globe 16 Jan 2004)

Berman attended a rancorous meeting where accusations were exchanged in a packed auditorium. Berman and other officials were "concerned about the charter school's proposal because it lists a 15 percent attrition rate" which suggested that the charter schools did not intend to educate everyone, as it promised, but instead planned to create an elite school, skimming the best pupils from the communities.

The Massachusetts board approved three of the seven charter applications including the Advanced Math and Science Academy Charter School, Community Charter School of Cambridge, and KIPP Academy Lynn Charter School. (Globe 25 Feb 2004)

Berman complained to the Boston Globe that residents were never able to comment on changes that were made after the period for public input. "This final proposal did not receive public review," Berman said. "To create an elitist academy is antithetical to the role of public education."

Berman responded with a lawsuit along with school officials from Marlborough and Maynard. "I think there are some substantive issues that have not been fully addressed and deserve a second look," Berman said. (Globe 26 Feb 2004)

The three communities contended in the suit that the Board of Education's review of the charter school was "egregious enough" to justify legal action, said Hudson School Superintendent Sheldon Berman . "We were appalled by the lack of thoroughness and fairness in the process," Berman said. "The [charter school's] application did not meet the standards set by the board itself." Berman said the suit had "a viable chance" to succeed. "We're on strong ground here." (Globe 21 Mar 2004)

The Boston Herald called our Berman by name in its (27 Mar 2004) editorial titled, Oh no, not competition!
In an astonishing display of narrow-mindedness, school committees in Marlborough, Maynard and Hudson are suing the state because they DON'T want to compete with a charter school approved for the region.Talk about fear and loathing! The three school committees filed suit against the state Board of Education in an effort to halt the scheduled fall 2005 opening of the Advanced Math and Science Academy Charter School. Sheldon Berman , superintendent of the Hudson schools, said, "I think we have a ate] [sic] board with an ideological agenda. It's supportive of privatization and not that clearly in support of public education."

A year later, the Advanced Math and Science Charter School has said it exceeded enrollment projections and held a partial lottery. Berman said the enrollment figures only represented parents of students who were interested in attending the school. The Massachusetts Association of School Superintendents, which opposes charter schools on the grounds that they drain funding from public schools, will issue a report next week asking the state to require official enrollment registration before funding the schools, Berman said. (Globe, 20 Mar 2005)

Then, in what reads like an olive branch toward charter school advocates, a report co-authored by Berman and released by the Massachusetts Association of School Superintendents Task Force on Charter Schools said that tensions between charter school advocates and traditional public school advocates had become so intense that it was difficult to dispassionately assess the effectiveness of charter schools. But the report also called for a host of changes to shore up the charter school application process.

Marc Kenen, executive director of the Massachusetts Charter School Association, said he was encouraged by the tone of the superintendents' report. ''For them to say that charter schools can coexist with traditional public schools, it's certainly a big step for them to take," Kenen said. (31 Mar 2005)

But it didn't end there.

By April 14 2005, (Globe) Berman and his colleagues had amended their lawsuit saying that the charter school's proposal for teaching special-education students was deficient. Berman said newly discovered documents "lend validity to the issues we raised" particularly the special education issue and strengthen the school districts' position. Four of seven independent evaluators who assessed the school's application in 2003 as part of the Department of Education's charter school review process gave the school low marks in its plan's "commitment to serving the needs of special populations of students." Berman said state officials only recently disclosed these documents, which the suit called "clear evidence of the arbitrary and illegal nature of the board's decision."

The Boston Globe called Berman an unofficial spokesman against the charter schools, being one of three districts that sued the school and the state Board of Education in an attempt to overturn the board's approval of the academy. "And he has rallied at least a dozen other school systems from Worcester to Boston to join a complaint against the state Department of Education for approving the charter school, one of the largest complaints against a single charter school so far." (Globe 22 May 2005)

Berman said he had visited Russia and Japan to study their educational systems. He said that not all students with special learning needs would be able to keep up with such rigorous programs and would ultimately drop out. It's an issue that Berman understands well. Several years ago, he authored a report on special education that received national attention and led to increased funding in Massachusetts and other states. Berman also contends that the academy would drain funding and talent from the public schools and undermine their efforts to beef up curriculum. "We have some of the most advanced science and math courses in the state," he said of Hudson High School. "We offer four calculus courses. I don't think there's another school around that does that."

Berman said he does not oppose charter schools in general but considers the Marlborough academy an attempt to create an elite private-styled school at taxpayer expense, he said. He said the school only recently provided more than 4,000 pages of documents that lawyers had requested, and that many of the signatures on petitions used to show support for the charter school were names that could not be verified. "They're very intent on getting their own way," Berman said. "They have essentially subverted what we believe are important regulations . . . and misrepresented themselves."

Berman, 56, said a background in competitive fencing he was a former assistant fencing coach at Harvard and Massachusetts Institute of Technology has taught him a lot about public policy debate. "It isn't all about attack," he said. "It's about being calm and handling things in the most appropriate way. In fencing, it's most effective to let [the opponent] fall on your point. It's also that way in public policy." But Berman is not sitting idly by. Brookline School Superintendent William Lupini said he joined the complaint after realizing his school would be affected by the new charter school. "Shelley has been the workhorse, the driving force in putting together very research-based, very reasonable position papers," Lupini said. "I can't think of anybody I'd rather be working with on this issue."

Boston Globe readers responded to Berman's position saying,

"The basis of the suit, filed by Superintendent Sheldon Berman of Hudson, seems
to be that "not all students with special learning needs would be able to keep up with such rigorous programs." Yet it is many of those "special learners" not being served in their current public schools who AMSA is attracting. Massachusetts testing procedures ensure that children are held to a minimum standard of learning. But where are the services for those who go beyond the minimum?"
And this:

Berman is a master at whipping local school boards into a frenzy about things like the academy because he is absolutely terrified of competition. So are the other school superintendents. Several are already changing curriculum to better compete and that's good. Hopefully Berman's specious arguments against the academy will be rejected by any judge that has to review the case. It is the future. So is increased competition in the K-12 education business. It's been a monopoly for way too long. As a result, as Alan Greenspan often says, our children are behind the rest of the world in math and science after about the fourth grade. Folks who have lots of money already have, and take advantage of, a plethora of alternatives to public schools. It's time that folks who can't afford to pay twice for schools have access to some alternatives. (Globe 29 May 2005)
Berman's colleagues responded with recognition. Opposition to charter schools was among the reasons that Superintendent Sheldon Berman recently received the prestigious President's Award from the Massachusetts Association of School Superintendents. Berman believes he was recognized because most of the state's superintendents "have become aware of how significant this issue is," he said last week. This is the second time he has gotten the award. "I am very, very honored," he said. (16 Jun 2005)

Amid cries that Berman had somehow manipulated the data, a report by Worcester Polytechnic Institute graduate students showed that "several local charter schools [were] enrolling disproportionately small numbers of special education and limited English proficient students."
Robert Harrington, superintendent of the Abby Kelley Foster Charter Public School, accused Mr. Berman of being "full of "hatred." Berman, who co-chair the Massachusetts Association of School Superintendents' Charter School Task Force, said they proposed the study to WPI because the state didn't seem to be researching the issue. (Worcester Telegram & Gazette, 7 June 2007)

If the implication from Innes is that Berman didn't tell the whole truth by declaring charter schools to be a failed reform - having already declared that public schools and charter schools could stand side by side - I'm not sure that holds up.

(I would have asked Berman myself if past information I had received from had held up, but it didn't. When I was defending him I exchanged several amiable messages with Berman. But when his information turned out to be untrustworthy, and I asked him about it, he stopped writing. I'm still waiting for his explanation as to why that is the case.)

The record would seem to indicate that Berman has been and remains a charter school foe, the 2005 report notwithstanding.

The record also suggests that Berman didn't have to wait two seconds before he considered charters to be a failed idea. He told the Globe in 2005 that "he does not oppose charter schools in general," but the evidence suggests he does.

But Innes had another concern.
If I heard this right, at one point, Jefferson County Superintendent of Schools Sheldon Berman claimed that only two of the 50 charter schools in Massachusetts – where he served before coming to Kentucky – were really performing well.

I wondered about the same thing. Innes wanted me to know he had the goods on this one from a 2006 Massachusetts Board of Education report. It looks to me like he does.
The study of 56 Massachusetts charter schools found:
  • When there is a statistically significant difference in MCAS performance, it is much more likely to favor the charter school than the CSD.
  • In both English Language Arts and Mathematics, at least 30 percent of the charter schools performed statistically significantly higher than their CSD in each year with the exception of 2001. In 2001, 19% of the charter schools performed statistically significantly higher than their CSD in English Language Arts and 26% in Mathematics.
  • The percentage of charter schools performing higher than their CSD each year has remained fairly constant in English Language Arts and Mathematics while the number of charter schools and the number of students tested in charter schools has increased.
  • The percentage of charter schools performing lower than their CSD has declined to approximately 10 percent in Mathematics and dropped below 10 percent in English Language Arts.

Similar patterns existed for all demographic subgroups, with the likelihood of the significant difference favoring the charter school being most prevalent for the African American, Hispanic, and Low Income subgroups.

Because eighteen charter schools were located within the city of Boston, the aggregate performance of those schools was also compared to the Boston Public Schools. In this comparison, the analysis showed that:

  • Charter school students in Boston as a combined cohort have performed statistically significantly higher than students enrolled in the Boston Public Schools each year from 2001 to 2005 in both English Language Arts and Mathematics, except there was no statistically significant difference in performance in English Language Arts in 2001.
  • Among the African American, Hispanic, Low Income, and Special Education subgroups, charter school performance was statistically significantly higher than the CSD in each year since 2002 in both content areas.
  • Charter school students in the White subgroup performed statistically significantly higher than their counterparts in the Boston CSD in 3 of the 5 years in Mathematics and 4 of the 5 years in English Language Arts.
Now, the only problem with all this is that the study acknowledges that the demographic characteristics of students attending a particular charter school may differ from the demographic characteristics of the students in the CSD...which was one of Berman's original complaints.

Thursday, March 12, 2009

Attention Bluegrass Institute: This is How it's Done

The Kentucky Department of Eucation today issued a retraction for some misleading information contained in charts showing overall average scores for PLAN and EXPLORE tests given to Kentucky students.

The errors were called to their attention by Richard Innes, an analyst with the Bluegrass Institute, here, here and here.

KSN&C agrees with Innes's analysis.

This from KDE:

In News Release 09-017 (EXPLORE and PLAN Results Released), please note that the tables showing overall average scores for those assessments may be misleading.

The tables show Kentucky's average English, mathematics, reading, science and composite scores for 2006, 2007 and 2008, along with average scores nationwide for those years.

The tables should have indicated that the national scores are based on the averages of students who took those tests in 2005, when EXPLORE and PLAN were normed as part of a national study. The 2005 nationally normed score also...used the national average for 2006, 2007 and 2008 and will continue to be the "score of record" for national purposes until the tests are re-normed.

The corrected tables were attached to the press release and show the 2005 normed score only.

Now, if we can just the Bluegrass Institute to retract their exaggeration of of a recent UK study, we'll have two groups who have demonstrated some integrity this week.

Saturday, January 31, 2009

Poor Little Think Tank

When he's not muscling the government bureaucracy for being bad for us, or extolling the virtues of a free-market idealism that made the American economy what it is today, it turns out that the Bluegrass Institute’s David Adams has a softer side.

Writing at Kentucky Progress, Adams seemed to be sad about a lone anonymous commenter on the new PrichBlog. "Anonymous" hoped that the new Prichard Committee blog would be “a welcome alternative to the Bluegrass Institute.”

Obviously hurt by this, Adams took the single comment and exprapolated a conspiracy theory claiming,
protectors of the status quo have, instead of making a case for the efficacy of their programs, made a habit of taking potshots at The Bluegrass Institute and anyone else who dares to threaten their base of power.
Poor Bluegrass institute. I had no idea it was so tough being in a think tank.

Adams apparently thought he would open a meaningful conversation with the Prichard folks. As is his wont and right, he started with a little name-calling,
The Bluegrass Institute has tried for years to engage KDE sychophants like Prichard Committee in a meaningful debate.
Sychophants?! Adams followed by describing the Prichard Committee as
“the chief protector of failed bureaucratic education policies that have held Kentucky back for decades.”
One wonders how the conversation with the Prichard Committee is working out.

Adams defends his name-calling with this:

I consider calling someone who uses my own money against me a name or two well within my rights as a taxpayer.
The problem is the Prichard Committee is not a government agency. They don't spend anybody's tax dollars. ...for or against anyone. And if the taxpayer bill of rights includes a name-calling provision, I missed it.

So I fear Adams may have hampered his own effort to engage the Prichard Committee in meaningful debate by choosing an opening gambit that was both disrespectful and off-the-mark; regrettably, an increasingly disappointing Bluegrass Institute trait.

I remain confounded by the Bluegrass Institute's apparent lack of concern over its own credibility. But as David puts it,

I'm not too concerned about my credibility with those who refuse to hear well-documented complaints about the KDE.

In fact, some of BIPPS's complaints are better documented than others. And some of BIPPS's arguments are known to be exaggerations, but remain unacknowledged by BIPPS - further undermining its cerdibility.

Adams's persistent loyalty to BIPPS's principles is unflinching no matter the documentation, or lack thereof. He serves the BIPPS leadership, apparently takes his cues from other BIPPS analysts and, promotes them and their ideas; and has, by that means, advanced personally. I don’t believe I’d be calling anyone a sycophant if I were him.

But the blogosphere would be less fun without Adams's occasionally prescient insights. Take this one for example:

No one in Kentucky is going to freeze to death living indoors this winter. We are going to have to get past the point of these feel-good appropriations should times really get
hard in America.
Oh wait. That wasn't prescient at all, was it?

But it was surely a public service when Adams offered advice to the non-rich on how to be poor while facing problems of college affordability.

And he does produce funny headlines from time to time.

By contrast, BIPPS education analyst Richard Innes reached out the left hand of welcome to Prichard by pimping one of his own blog posts, and inferring that Prichard had arrived late to the party.

Then on January 21, 2009 4:14 PM he wrote the funniest thing of all.

“Note that I linked back to the Prichard Blog, as we really do want to engage in productive conversation.”

What some think of as following copyright law, Innes offers as an olive branch.

I predict that the Prichard Committee will tire of the BIPPS good-cop/bad-cop routine fairly quickly.

Friday, October 24, 2008

Starve the Schools. Blame the Schools. Bleed the Schools?

The not-too-subtle warrant behind a new study by Richard Innes of the Bluegrass Institute, "How Whites and Blacks Perform In Jefferson County Public Schools," is that after 18-years of KERA, educational problems in Kentucky should now be largely solved - and if they are not, the problem lies with KERA.

Some thoughts on the achievement gap:

The achievement gap was deliberately created and sustained by whites over centuries in America. Slavery, Jim Crow and recalcitrant racism remain America's great historical shame.

So the Bluegrass Institute is to be commended for taking a serious look at the performance of African American students in Jefferson County and the dismal graduation rates that describe their performance. The data remind us of how far we have left to go.

But there is an unchallenged premise in the very first paragraph of the study that bears some comment.

One of the most important tenets of the Kentucky Education Reform Act of 1990 is that all children can learn, regardless of race or economic status. However, during the 18 years since KERA’s enactment, serious questions have arisen regarding the performance of Kentucky’s public schools in meeting that goal for all children, especially black students.

This is all true. But a couple of things leap to mind.

First, BGI is quite correct to suggest that viewing students as capable (able to learn) is an historic first that came to Kentucky schools with KERA in 1990 - and it would be wrong on many levels to retreat from that goal. Prior to KERA (actually more like 1995), it was fully acceptable for a third of Kentucky's students to fail - and in many places, more than a third did just that.

Second, the 18 year period since KERA has not seen that problem solved. But seriously, was that expectation realistic? I have no doubt that KERA's goals can be largely achieved in time, but Kentucky sure did spend a whole lot longer digging the hole it's students are in than has spent climbing out of it. Patience and nutruring are indicated.

Let's review the history briefly.

The best thing that can be said about efforts to educate African Americans in antebellum Kentucky, is that it was not illegal to teach a black person to read.

Following the civil war the legislature's first effort provided that only taxes collected from blacks could be used to educate black children; later amended to add...after taking care of paupers. That "progress" accounted for 74 years of Kentucky's history.

In 1874, eight years later, a school system for blacks began - and where schools existed, they were funded at about 1/3rd the rate given to the pitiful schools whites attended.

During the progressive era at the turn of the century, more schools were built and conditions improved, but under Jim Crow laws blacks were legally separated, and socially "kept down."

Testing was employed to measure human potential. The results were interpreted to suggest that blacks were genetically less capable and therefore, not worthy of equal investment. Still, local schools for blacks developed, and for a time, African American teachers were better educated than white teachers - since the brightest and best were precluded from pursuing many other employment options.

Conditions changed - legally, but only in small measure - with the Brown v Board of Education decision in 1954. It was another 14 to 18 years before a substantial percentage of blacks attended school with whites - then only after the federal government withheld funds from (or troops were called in to) schools that refused to desegregate.

Photo: Louisville, 1955, Black students segregated within a desegregated school

There was nothing in KERA that should not have already been achieved under Brown. But that didn't happen, and by that time Kentucky was in it's 162nd year of the achievement gap.

Conditions improved over time but relative to other states, not so much. By 1984, investment inequities reached a disproportionate 8:1 ratio between the rich schools and poor schools in the state. If parents wanted a good education for their child, where they lived mattered.

It was only since Senate Bill 168, in 1998, that Kentucky schools began any real effort to ensure substantially equitable student achievement outcomes. Student data was disaggregated and schools were held responsible, not only for maintaining high average performance, but for the improvement of each subgroup of students. Prior to that time if a third of a school's students failed to perform up to expectations, it was considered the parents' or students' fault. This marked the beginning of a philosophical shift, away from "equality of educational opportunity" toward "equity of student achievement outcomes" - a philosophy also inherent in NCLB.

In the face of 200 years of a deliberately created achievement gap - ten years of substantial progress in an under-funded system seems somewhat understandable. It might even be crazy to expect that the intractable problems of poverty might be solved, through legislation, by the schools alone.

Now, I know it is heresy to not swallow the "KERA Kool-aid" that all it takes to produce better schools is strong leadership, a whip and a chair. But I believe we need a broader more comprehensive approach.

Without having read the indivdual school reports referred to in the BGI study, I have no reason to dispute the findings of Richard Innes' report. His analysis shows:

  • Blacks remain well behind academically in the key subjects of reading and mathematics.
  • In a significant number of Jefferson County schools – 47 out of the 120 schools with usable data on reading and 44 out of 120 for math – the gap between white and black students is widening.
  • Graduation rates remain extremely low for significant numbers of blacks – especially black males – in the majority of Louisville’s public high schools. Using a graduation-rate estimation formula created by Johns Hopkins University, black males in only three of the 19 high schools in the study had graduation rates equal to or greater than the statewide graduation rate for all students. The graduation rate also is low for black females and even for white students in these 19 schools. Two of these schools reported abysmal graduation rates of less than 60 percent, qualifying them as “dropout factories” using the Johns Hopkins formula. In both of these Jefferson County schools, the graduation-rate estimates using the Johns Hopkins formula fell well below 50 percent for whites and blacks of both sexes.
I'm sure he's correct. I particularly commend Innes for this:

Black students do not benefit if achievement gaps in their schools close only because whites also perform poorly.
Those who would close gaps by lowering standards perform a disservice to all students.

But I do take issue with his conclusion that:

...the CATS assessment process holds no consequences for poor performance with student subgroups and really does nothing to deal with gaps."

This ignores the part of the process where the superintendent chews some principals' butts off, demotes them to a third grade classroom, and replaces them with the next victims. I promise, that part of the process counts for something.

Innes reviews the graduation rates for Jefferson County high schools noting that,

...Kentucky’s existing NCLB high school graduation-rate reports fail to provide accurate data overall and do not contain information on how races perform.

This well-documented problem exists nationally, and is being addressed nationally. Using a measure from Johns Hopkins, Innes finds that the graduation rates for far too many schools, well, ... suck.

In his closing, Innes takes two more shots at his point:

Jefferson County Schools face a considerable amount of work to raise academic performance and high school graduation rates to acceptable levels. After nearly two decades of KERA, real improvement – especially for the district’s black students – remains a long way off.

Kentucky cannot afford schools with low and declining graduation rates, and with low academic achievement. Clearly, as KERA approaches 19 years in force, the trends outlined in this report raise questions about whether the public school system in Kentucky is capable of meaningful reform for all students.

There it was. That last phrase. And, Innes could have written that conclusion as early as 1994 by which time he was publicly (Kentucky Post) acknowledged as a KERA "foe."

Would Innes have us believe that Kentucky schools (parents, teachers, kids...) are not capable? Would he cure these bad schools with a healthy dose of leeches, called school vouchers?

None of that speculation should mask what Innes gets right.

In too many cases student achievement gains are lacking and reported graduation rates remain a fantasy. If Kentucky is serious about reaching its goals, the state needs to get serious about its investment. It is going to take a comprehensive approach to meet KERA's lofty goals.

Sunday, October 19, 2008

Innes smacks Kifer. Kifer smacks back.

We recently had a little dust up over testing between education analyst Richard Innes and Georgetown/UK Professor Skip Kifer.

The Bluegrass Institute's Richard Innes, argues that credibility and stability problems with the CATS are reason enough to dump it. In this argument, the GREAT is the enemy of the GOOD. Innes pounds on the CATS' several flaws; hoping that by seeking perfection the utility of the instrument will be doomed. He thinks there's a better idea. Just throw the CATS out. Maybe, replace it with the ACT. After all, the ACT is now a super test with super powers. Just ask ...the folks at ACT.

Ben Oldham recently wrote, "Since the ACT is administered to all Kentucky juniors, there is a tendency to over-interpret the results as a measure of the success of Kentucky schools."

Innes, assisted that over-interpretation mightily, and decided he would "school" Professor Oldham saying,
Oldham pushes out-of-date thinking that the ACT is only a norm-referenced test. The ACT did start out more or less that way, years ago, but the addition of the benchmark scores, which are empirically developed from actual college student performance to indicate a good probability of college success, provides a criterion-referenced element today, as well.

Well, I'm no testing expert but even I knew that was wrong. Wrong enough, that I began to worry that BGI's testing expert might have some holes in his own preparation.

KSN&C responded,

The problem of over-interpretation has been somewhat exacerbated by the inclusion of benchmark scores in the ACT. But benchmarking does not change the construction of the test nor the norming procedures. It does not turn the ACT into a criterion-referenced exam as Innes tries to suggest...

The National Association for College Admission Counseling Commission — led by William Fitzsimmons, dean of admission and financial aid at Harvard University — issued a report last month the sparked a lot of conversation about how tests like the ACT and SAT were being misused.

This is a recurring theme in the college admissions business but this year's report warned that the present discussion of standardized testing has come to be “dominated by the media, commercial interests, and organizations outside of the college admission office.” Some of those groups have other items on their agenda.

Skip Kifer mentioned the report's warning about over-emphasizing test scores and argued for prudent use of the ACT in selecting students. He also warned folks not to get snookered by ACT officials new claims that without having changed the nature of the test, their scores now tell whether a student "meets expectations" or is "ready" to attend college.

As Kifer pointed out,

The benchmark stuff is statistically indefensible. Hierarchical Linear Modeling(HLM) was invented because people kept confusing at what level to model things and how questions were different at different levels. The fundamental statistical flaw in the benchmark ... is that it ignores institutions. Students are part of institutions and should be modeled that way.

Innes persisted and tried to play it off saying,

I guess Kifer and his compatriots at Georgetown College ... will never get the idea behind the ACT’s Benchmark Scores. It really isn’t hard to understand the Benchmarks.

This is when I should have suspected an academic spittin' contest was on the way and somebody was going to have to put up or shut up. (Actually, in the blogosphere, nobody really shuts up, but you know what I mean.)

Stung by the suggestion, and always the professor, Kifer challenged Innes to "Describe the statistical models used to determine the ACT benchmarks." This question would show whether Innes understood the nature of Kifer's argument and at the same time would disprove his claims.

Innes said he was waiting for some information from ACT (which in my experience is a lot like waiting for Godot) and changed the subject to how lousy the CATS is.

We're still waiting to hear him defend his position that the ACT Benchmarks constitute a valid criterion-reference test and that "the Benchmarks can fairly be considered a real measure of proficiency." That may be like waiting for Godot as well.

KSN&C has taken the position that all social science tests are imperfect. This includes the CATS, the KIRIS, the ACT, the SAT, the CTBS, the NAEP, and the EIEIO.

Monday, September 29, 2008

What the Bluegrass Institute Doesn't Seem to Know about the ACT

There is no way to sugar coat this.
Somebody doesn't know what he's talking about
- and it's not Oldham.

Well, Ben Oldham and I got called out by Richard Innes of the Bluegrass Institute the other day for "ignoring" the ACT benchmarks scores - apparently the holy grail of assessment in his mind.

Of course, this is all part of a larger conversation about Senate Bill 1 and the on-going Task Force on Assessment at KDE.

I wasn't planning on getting into all of this this fall. It's tedious, inside baseball kind of stuff. But the fundamentals are still the same. First, it's just a test. Second, every test has been designed to do a specific job. If test designers wanted a test to do something else, they would begin with that fact in sight. Third, have I mentioned it's just a test?

OK, let's talk about the ACT.

Here's the problem with the American College Test: Nothing, really.

It is a well-designed test intended to help admissions officers at competitive colleges determine which students are most likely to be successful at the university level. Ben Oldham recently went further saying, the "American College Test (ACT) is a highly regarded test developed by a cadre of some of the best measurement professionals in the world and is used by a number of colleges..."

But the ACT is only ONE factor that colleges use to make such determinations.

Why?

I mean, if the ACT can predict success in life, as Innes un-credibly argues (below), why don't colleges simply rely on it and quit wasting time compiling grade point averages and other data they say they need to made the best choices for their school?

The answer lies in the fact that test data are only reliable up to a point. It's just a test score and it shouldn't be turned into anything more.

As Richard C. Atkinson and Saul Geiser recently pointed out,

the problem with general-reasoning tests like the SAT [and ACT] is their premise: that something as complex as intellectual promise can be captured in a single test and reflected in a single score. It is tempting for admissions officers--and parents, legislators, policymakers and the media--to read more into SAT [and ACT] scores than the numbers can bear. Although measurement experts know that tests are only intended as approximations, the fact that scores frequently come with fancy charts and tables can create an exaggerated sense of precision.

And such exaggerations persist.

Newspapers and bloggers rank scores that ought not be ranked - because people like rankings. Some "think tanks" act as though test scores equal "truth" and look for any opportunity to twist data into a pre-existing narrative that Kentucky schools are going to hell in a handcart - this, despite trend data to the contrary, about which they are in full denial.

Georgetown College Distinguished Service Professor Ben Oldham correctly warned that, "since the ACT is administered to all Kentucky juniors, there is a tendency to over-interpret the results as a measure of the success of Kentucky schools. His excellent article clarifies what the test is, and what it isn't.

The problem of over-interpretation has been somewhat exacerbated by the inclusion of benchmark scores in the ACT. But benchmarking does not change the construction of the test nor the norming procedures. It does not turn the ACT into a criterion-referenced exam as Innes tries to suggest - unless all one means by "criterion" is that the ACT derived a cut score. Under that definition a People Magazine Celebrity Quiz could be considered criterion. Socre 18 and you're a Hollywood Insider!

The ACT's "criteria" simply does not measure how well Kentucky students are accessing the curriculum. It is much more sensitive to socio-economic factors attributable to most of the college-going population.

Using a "convenience sample" of schools (those willing to participate) the ACT looks at student success in particular college courses; and then looks at the ACT scores obtained by "successful" students. But regardless of what data such a design produces, "there is no guarantee that it is representative of all colleges in the U.S." Further the ACT "weighted the sample so that it would be representative of a wider variety of schools in terms of their selectivity."

That is to say, they tried to statistically adjust the data produced by the sample to account for more highly selective schools as well as the less selective. This process of weighting data to produce a score that the original sample did not produce should be viewed suspiciously. It would be like...oh, let's say like....using a concordance table to give students a score on a test they didn't take.

If KDE had done anything like this, Innes' buddies at BGI would be crying "fraud."

If we are going to test, and if our tests are going to be used to determine placement in programs within schools, and eventually in college, then we need to understand what the ACT means when it says "college-ready." And we don't. The most important flaw of the ACT benchmarks is conceptual: What is "readiness" for higher education?

As one delves deeped into the statistics other problems arise. Skip Kifer who serves on the Design and Analysis Committee for NAEP told KSN&C,

The benchmark stuff is statistically indefensible. Hierarchical Linear Modeling
(HLM) was invented because people kept confusing at what level to model things
and how questions were different at different levels. The fundamental
statistical flaw in the benchmark ... is that it ignores institutions. Students
are part of institutions and should be modeled that way.

But the ACT models at the "student" level when it should be modeling at the "students nested within institutions" level.

It is possible that the ACT took a kind of average of those "correct" models but that can not be determined that from their Technical Report.

Perhaps Innes could help us understand: How is it that the ACT's benchmarks could have been empirically defined and yet managed to get the same relationship for the University of Kentucky and Lindsey Wilson College?

Unfortunatley, the ACT folks did not respond an inquiry from KSN&C.

But none of this will likely stop the exaggeration of the ACT's abilities.

In response to a KSN&C posting of Ben Oldham's article, Innes made the following claim:

Oldham pushes out-of-date thinking that the ACT is only a norm-referenced test. The ACT did start out more or less that way, years ago, but the addition of the benchmark scores, which are empirically developed from actual college student performance to indicate a good probability of college success, provides a criterion-referenced element today, as well.

"Criterion-referenced element?!" A cut score? The ACT is a timed test too - but that doesn't make it a stopwatch.

So, Oldham is old fashioned and out-of-date? Au contraire. It is Innes who is over-reaching.

Innes argues,

the ACT says that many employers for ... better paying jobs now want exactly the same skills that are needed to succeed in college. So, the Benchmark scores are more like measures of what is needed for a decent adult life. Thus, it isn’t out of line to say that the Benchmarks can fairly be considered a real measure of proficiency. And, that opens the door to compare the percentages of students reaching EXPLORE and PLAN benchmarks to the percentages that CATS says are Proficient or more.

Bull.

One could derive as much "proficiency" evaluating Daddy's IRS form 1040 and then comparing percentages of students reaching EXPLORE and PLAN benchmarks to the likelihood of owning a BMW or affording cosmetic surgery.

I'm afraid what we have here is something other than a nationally recognized assessment expert who is out-of-date.

We have a pundit who thinks the ACT benchmarks constitute a criterion-referenced assessment of the performance of Kentucky students and their prospects for a decent adult life!? This, absent any connection between the ACT and Kentucky's curriculum beyond pure happenstance. There is no relationship between a student's ACT score and any specified subject matter - which is typically the definition of a criterion-referenced test.

There is no way to sugar coat this. Somebody doesn't know what he's talking about - and it's not Oldham.

The best spin I can put on this is that Innes got snookered by ACT's marketing department, which seems to do a fine job, but has been known to overstate the abilities of ACT's EPAS system.

But none of this makes the ACT a bad test. It just means that assessment experts have to take care to understand the nature of the exams and not to rely on them to do too much.

And it is commendable that Kentucky is working toward building an actual relationship between Kentucky's curriculum and that of the ACT through the development of content tests. That work will get Innes closer to to where he wants to be. He should wait for the actual work to be done before making claims.

Just as Atkison, Geiser, Oldham, Kifer, Sexton and virtually everybody else says, the results should not be over-interpreted to suggest relationships that just aren't there. And trying to argue causal chains that are completely unproven is certainly not best practice.

But more to the point, Kentucky recently committed to use the ACT's EPAS system including EXPLORE and PLAN as yet another measure - a norm-reference measure - of student performance. As long as Kentucky is cognizant of the test's limitations we ought to strengthen the connections between Kentucky high schools and universities and gauge student readiness for college. It was because of the large numbers of college freshmen in need of developmental courses that CPE pushed for the ACT/EPAS system to begin with.

Kifer wonders why Kentucky's Advanced Placement (AP) Tests receive so little attention. After all, unlike the ACT, the AP tests are a direct measure of a high school student's ability to do college work; AP courses are particularly well-defined; the tests exist across the curriculum; good AP teachers abound; course goals and exams are open to scrutiny.

When a high schooler passes an AP test he or she not only knows what it means, but the school of their choice gives them college credit for their effort.

Aware of CPE's commitment to the ACT as one measure of student readiness, KSN&C contacted newly named Senior Vice President of the Lumina Foundation Jim Applegate, who until recently served as CPE's VP for Academic Affairs.

Here's what Jim had to say:

Richard,

The article recently referenced in your publication from the admissions officer group addresses the use of ACT for college admissions. The
organizations sponsoring assessments such as ACT, SAT, and others have made clear that no single standardized test should be used to make such decisions. Postsecondary institutions, to implement best practice, should use a
multi-dimension assessment to make admissions decisions. A test score may play a
role in these decisions, but not the only role.

Kentucky uses the ACT/EPAS system (the Explore, Plan, and ACT tied to ACT ‘s College Readiness Standards) to help determine college readiness, place students in the right high school courses to prepare them for college, and place them in the right courses once they go to college. Kentucky’s revised college readiness standards are
about placement, not admission. For the foreseeable future, the postsecondary
system will, as it has always done, accept large numbers of students with ACT
scores below readiness standards, but will provide developmental educational
services to these students to get them ready for college-level work. The large
number of underprepared students coming into Kentucky’s postsecondary system led the Council a couple of years ago to initiate an effort to improve developmental
education order to make sure these students receive the help they need to
succeed in college.

A growing number of states are adopting the ACT or the entire EPAS system to more effectively address the challenge of getting more high school graduates ready for college or the skilled workplace (e.g., Colorado, Illinois, and Michigan). These states also want to better understand the performance of their students in a national and international context. Globalization no longer allows any state’s educational
system to remain isolated from these contexts.

The use of ACT/EPAS is, of course, only one necessary strategy to improve the college/workplace readiness of Kentucky’s traditional and adult learners. Kentucky is working to implement statewide placement tests in mathematics, reading, and English that will be administered to high school students who fall below statewide college readiness benchmarks tied to ACT scores (few states have gotten this far in
clarifying standards to this level). These placement tests will provide more
finely grained information about what students need to know to be ready for
college-level work. We are also working to more strongly integrate college
readiness goals into our teacher preparation and professional development
programs to ensure teachers know how to use the assessments beginning in middle
school to bring students to readiness standards.

The postsecondary system is hopeful the implementation of ACT/EPAS will promote partnerships between postsecondary and high/middle schools to improve student achievement. Some of that has already begun since the first administration of the EPAS college readiness system. For the first time in my time in Kentucky (I grew up
here and returned to work here in 1977) we now know where every 8th grader is on
the road to college readiness thanks to the administration of the Explore. If in
five years the number of students needing developmental education is not
significantly less than it is today then shame on all of us.

Jim Applegate

All of this reminds me of the old Crest Toothpaste disclaimer I read daily while brushing my teeth over the decades.

Crest has been shown to be an effective decay preventive dentifrice that can be of significant value when used as directed in a conscientiously applied program of oral hygiene and regular professional care.

Let's see if I can paraphrase:

The ACT/EPAS system has been shown to be an effective norm-reference assessment that can be of significant value when used as directed in a conscientiously applied assessment program based on clear curriculum goals, direct assessments of specific curriculum attainment and effective instruction from a caring professional.

Friday, June 27, 2008

Jack Jennings on the new CEP Study

KSN&C recently posted a story on the new report from the Center on Education Policy, "Has Student Achievement Increased Since 2002?"

CEP presents what is at least the fourth study in the past two years to confirm the significant progress being made in Kentucky schools - one of only four states to show moderate-to-large gains in both reading and math.

Richard Innes, a Bluegrass Institute researcher, doubted the credibility of the study.

Last year, a Kentucky Long-Term Policy Research Center study revealed that Kentucky made across-the-board improvements in its rank among the states; growing from roughly from 43rd to the 34th.

The Bluegrass Institute doubted the credibility of that study, too.

The KLTPRC report mirrored two previous studies done outside of Kentucky. Kentucky was 34th in Education Week's Quality Counts 2007 Achievement Index and was 31st in the Morgan Quinto 2006-2007 Smartest State Index."

I assume BGI didn't think much of those studies either.

When Richard questioned the quality of this study, I contacted Jack Jennings, President of CEP for a comment.

Here's his response:

Mr. Day, you sent a message on Wednesday saying that a commentator had a question about the report we released on Tuesday about student achievement. In
particular, he claimed our report would have credibility problems because Kentucky had a decline in 8th grade reading on NAEP that had not been reflected in the results on the state’s test. I asked the authors of the report to look at this issue and to respond. This is what they say:

First, this is consistent with the CEP report. Table 4-F (in the national summary) shows a moderate-to-large gain on the state assessment but a moderate-to-large
decrease on NAEP for grade 8 reading.

In the state profile, the first bullet says: "From 2002 to 2006, student achievement increased in both reading and math at all grade levels analyzed, according to percentages proficient and effect sizes." This is true for the state assessment, and presumably this is what the e-mail author is referring to.

Kentucky changed its assessment in 2007, so it's possible (although admittedly quite a stretch) that performance dropped in 2007... which showed up on NAEP but we couldn't see on state testing.

The report provides several reasons why NAEP and state tests might differ.

After receiving your e-mail, I got a call from a person from an organization in Kentucky who was asking the same question about our report. I asked him if he had read the report and he said that he had not, and that he was probably the source of your question. I urged him to read the report before he started raising concerns about it.

By the way, the report—Has Student Achievement Increased Since 2002?—and the fifty state profiles, as well as all the test score data from the states is on our web site, CEP-DC.org.

Thanks.

Jack Jennings, President, Center on Education Policy


Now, Innes's suspicions were not irrational. Knowing that there was a decline in 8th grade reading on the NAEP seems inconsistent with the study's conclusions and knowing that the researchers accounted for the NAEP data is important.

But taken as a whole, BGI tends only to report the bad news about public schools...and I suspect he's was looking to discount the good news about Kentucky schools any time he can. It's probably in his job description.

It would seem any study that produces what BGI believes to be "the wrong answer" must necessarily be a bad study. This is a problem whenever one is hired to substantiate their boss's predetermine perception of reality.

Friday, September 28, 2007

The new NAEP data are released. That means it's time to spin the news!

The release this week of national test scores in reading and math has generated a fresh round of conversation about how Kentucky's students are performing.

We have this conversation at least three times a year when the various "yardsticks" (NAEP; CATS; NCLB) trot out their measurement data. Then there's SAT, ACT... EIEIO....

Wouldn't it be great if such data were integrated into a comprehensive value-added system? But I digress.

NAEP scores nationally, and in many individual states, showed modest gains from 2005 to 2007.

As Diane Ravitch explains in today's New York Post,
The federally sponsored National Assessment of Educational Progress (NAEP) is known in the education world as the gold standard of testing. In 2002, Congress authorized NAEP testing in every state to serve as a check of the states' own claims about their progress. (Congress rightly worried that individual states would dumb-down tests that they themselves develop and administer.)
And, there is at least reason to be suspicious of Kentucky's "new and improved" test. It appears Kentucky may have joined a number of other states in a race to the bottom by the redefining of proficiency.

Whenever test score data are released the spinning begins. The Kentucky Department of Education has an interest (some might say a duty) in pointing out the progress made by the schools. So they publicly shine a light on the best numbers, and privately express concern for the worst.

It's a little thing called spin. Everybody seems to delight in the practice these days.

In a Tuesday press release, the Kentucky Department of Education said,

"The results of the 2007 National Assessment of Educational Progress (NAEP) in reading and mathematics show that Kentucky's 4th and 8th-graders made gains when compared to the state's performance in previous NAEP assessments..."

True. Gains were made. Kentucky's student achievement, as measured by the NAEP, has trended steadily upward overall. (See charts below.)

So that's KDE's headline; Progress over time.

On the other side of the argument, assessment watchdogs are sniffing out specific areas of concern. Writing for the Bluegrass Institute this week, Richard Innes took issue with KDE's discounting of declines in 8th grade reading.

KDE claims of eighth grade reading since 1998, “Kentucky’s 8th-graders’ scores have remained steady, with minor gains and losses.”

Is that a fair description? Let’s examine the facts.

In the new ... NAEP assessments ... Kentucky had a reading proficiency rate of 30 percent in 1998. That rose to 32 percent in 2002 and went up again to 34 percent in 2003.

Then, things came unglued.

Eighth grade reading proficiency decayed to 31 percent in 2005, and in 2007 it slid again to just 28 percent. The 2007 proficiency rate is statistically significantly lower than both the 2002 and 2003 scores and is clearly six points lower than the 2003 performance. That six point difference isn’t just statistically significant – it’s just plain SIGNIFICANT.

No other state lost more ground in this time frame.

What’s more, during the same time period, the CATS Kentucky Core Content Test reading proficiency rates for eighth graders continuously rose. Do you believe that?
CATS up 10 points while NAEP declined six?

I'll take Dick's word for it that the declines are statistically significant and that Kentucky's decline is the nation's worst. The decline certainly looks significant to me.

We don't generally consider six point drops minor, and more to the point, downward trends are antithetical to progress. If it were me...I think I'd have left the word "minor" out of KDE's statement.


A BETTER ATTACK?

Innes has discovered some bad news, but arguably, not the worst news.

While Kentucky has progressed steadily, so have other states. Growth is a vital factor to consider, but so is excellence. Kentucky's relative standing among the states frequently leaves the state in all too familiar territory.

For example, who do Kentucky students outperform in 4th grade math? (See map below)

New Mexico, Louisiania, Mississippi and Alabama. All other states are roughly equal to (9 states), or exceed (36 states) Kentucky's progress. You're not going to hear that in a KDE headline.

It's a little better at 8th grade. Add California, Nevada, Oklahoma, Tenessee, Hawaii and West Virginia to the list.

Kentucky only outscores eight states in 8th grade reading.

But clearly the best news for Kentucky is in 4th grade reading where Kentucky joins the national leaders and is only outscored by seven states. What happens between 4th grade and 8th grade in reading ought to be of concern.

MARK YOUR CALENDARS

We're less than a week away from KDE's next big announcement of progress. I predict the new CATS assessment will show average performance gains of 7% or so across-the-board and in some places, jumps will be huge based at least partly upon changes...
a) to the test itself
b) to the "cut scores" used to define proficiency

The new test data can not be compared to the previous tests - but it will be. It's the data school folks have.

We discussed the NCLB data situation last night at UK. Without advance comment, I asked a group of graduate students (and future principals) to analyze the NCLB proficiency rates in Kentucky. The general reaction to the sharp increase was "Wow!" One of the students shared her experience working with the assessment company to establish the new cut scores. We discussed changes to the system that might account for the dramatic increases, and how school leaders could "present" the data. That's when one of the students came up with the best spin ever. (Pay attention Lisa. Here's your angle.) The new assessment is a truer reflection of the content actually taught in Kentucky's schools, and therefore the 7-point spike in proficiency levels is a fairer measure of the actual progress Kentucky students have made than under the old test.

Terrific.

Now, if we can only get the NAEP data to bear that out....

We have a fundamental problem in our current accountability system. It's initial purpose was political (to garner the support of the business community for KERA's big price tag). It not focused so much on student achievement and curriculum. The focus was school accountability.

Better, would have been a assessment system that began with content and then folded the data into a value-added system, such as the one used in Tennessee. If CATS had been designed to improve instruction for individual students, it would have looked very different.

To their credit, and after the fact, many educators began to look at interim assessment systems that would help teachers identify learning problems early and intervene quickly. There has been a lot of good work done in the trenches, but the state system has become a hodgepoge under NCLB.

Interpreting test data to the public is a national problem, and "interested" parties will always spin the data to suit their own purposes. What we really need is a "disinterested" assessment/accountability reporting source.

As Ravitch understands, we need...
an independent, nonpartisan, professional audit agency to administer tests and report results to the public.

Such an agency should be staffed by testing professionals without a vested interest in whether the scores go up or down. Right now, when scores go down, the public is told that the test was harder this year - but when scores rise, state officials never speculate that the test might have been easier. Instead, they high-five one another and congratulate the state Board...for their wise policies and programs.

What the public needs are the facts. No spin, no creative explanations, no cherry-picking of data for nuggets of good news.
Just the facts.
I may even know the right folks for the job. I understand Ben Oldham at Georgetown College has recruited Skip Kifer and Tom Guskey to form an assessment center at Gtown. I studied under Ben and Skip and am familiar (along with most of the national academic community) with Tom's work. I can't think of a more valuable state resource to guide a more effective and fair assessment program that the guys at Georgetown.


KENTUCKY SCHOOL DATA from the 2007 NAEP Exam

Student Characteristics
Number enrolled: 679,878
Percent in Title I schools: 60.6%
With Individualized Education Programs (IEP): 16.0%
Percent in limited-English proficiency programs: 1.5%
Percent eligible for free/reduced lunch: 52.4%

School/District Characteristics
Number of school districts: 176
Number of schools: 1,426
Number of charter schools: N/A
Per-pupil expenditures: $7,2541
Pupil/teacher ratio: 16.0
Number of FTE teachers: 42,413

Racial/Ethnic Background
White: 86.3%1
Black: 10.6%1
Hispanic: 2.1%1
Asian/Pacific Islander: 0.9%
American Indian/Alaskan Native: 0.2%


Scale Scores for Mathematics
Kentucky vs. National Public




For more specific data check out NAEP's Data Explorer.


Cross State Comparison
Percent At or Above Proficient
4th Grade Mathematics
Where:
Blue = Kentucky
Green = States above Ky
Yellow = States about the same as Ky
Red = States below Ky



For more specific data check out NAEP's Data Explorer.

Saturday, September 22, 2007

New assessment company: errors commonplace in Kentucky

Faulty data, appeals, recalculations leading to new accountability decisions - this has become business as usual for too many Kentucky schools. It forces one to wonder about the trustworthiness of the assessment instrument and the company hired to administer it.

The Cincinnati Post reported:
While there's been much talk in Kentucky education circles that there were an unusual number of complaints from schools, department of education spokeswoman Lisa Gross said the number wasn't extraordinary given that the state changed the company doing the test and increased the number of grades tested.

"The breadth of the issues with the NCLB data was typical of that in previous years and had no major statewide effects," she said.

I'd like to be comforted by that thought, but I'm not sure I am. From the minute NCLB scores were released some western Kentucky superintendents claimed the data were contaminated.


Is Gross saying it's always this bad?

If so, why did KDE decide to delay release of the CATS data this year, as opposed to last year...or the year before?

Is Gross blaming the new company?

Covington District Assessment Coordinator Bill Grien told the Cincinnati Post, " When (the state department of education) listened to all the complaints from around the state, they decided to take time to make sure data was correct."


The clear message for every school in the state is, "Don't trust the data!" Double check it yourself.

Changes in the testing program this past year make all of the data suspect. Errors undermine confidence.

In addition, Kentucky School News and Commentary began hearing reports from teachers last spring (when the new tests were first being administered) that the new test was easier on its face - an unverifiable opinion of numerous teachers; suspicious nonetheless.

Even if the released data had been pristine, the fact that the assessment itself has changed has caused the Kentucky Department of Education to warn schools that the data can not be compared to past years.


Of course that did not stop district superintendents from proclaiming progress based on the new data. It's the only data they have.

Around the state local superintendents took bows and praised teachers for district progress. Much of that progress may well be accounted for by the new test alone. Superintendent Stu Silberman claimed that Fayette County had "broken some records" with this year's data. If the data cannot be compared...how can that be?

The question is...Has Kentucky joined the growing list of states who have lowered standards to avoid looking bad under NCLBs accountability system?

It looks like the answer might be yes.

Dick Innes, over at the Bluegrass Institute, has been dissecting the assessment program for years now. See his MUST READ post on the current KDE testing woes. His preliminary data on 2007 NCLB proficiency rates seem to confirm my own suspicions - that those teachers who said the new test is less demanding are correct.


Consider: Reading proficiency among all Ky students stood at 48.24% in 2002. In 2003, it rose a couple of points to 50.16%. The average growth per year was 2.15% as the proficiency percentage advanced to 53.45% in 2004; 55.95% in 2005 and 56.84% by 2006.

Then the test was changed.

In 2007 the percentage of students scoring proficient in reading jumped to 66.12%! A 9.28% increase in one year - which would really be worth celebrating if we knew it was the result of increased student achievement.

By changing the test, Kentucky made more "progress" toward meeting its NCLB goals in one year than it had over the previous four years.

The results are similar for Reading and Math and for African American and Free Lunch subgroups.

Average yearly gains for "all students" in Math were 2.49% until the new test. This year it jumped to 9.86%.

Average yearly gains for "African Americans" in Math were 2.24% until the new test. This year it jumped to 10.15%.

Average yearly gains for "Free lunch" students in Math were 2.73% until the new test. This year it jumped to 10.28%.

Average yearly gains for "African Americans" in Reading were 2.40% until the new test. This year it jumped to 10.03%.

Average yearly gains for "Free Lunch" students in Reading were 2.65% until the new test. This year it jumped to 10.14%.

(The sources for his data: KDE, NCLB Progress Reports for 2003 - 2007.)

Is it safe to assume (after multiple administrations of the old test) that the established pattern of 2.5% (approximate) growth per year was about right? If so, the new data should be viewed with suspicion. Reported student achievement growth may be inflated by something around 7%. If that's true...the celebrating ought not get to big unless a school's growth exceeds 7% this year.

If my suspicions are correct, when the CATS data are released on October 2nd, we ought to see some monster "progress." Annual increases may well break some records. In baseball parlance, the dead ball era is over. And in places where progress has historically been slow, but where the instructional focus has been the most intense, we may see numbers that are huge.

Just for fun

Any Fayette County folks want to guess what kind of gains Booker T Washington Academy will post when the data are released next month?

Principal Peggy Petrilli recently resigned (or was released, depending upon who you ask) amid a lot of fussing from parents about her methods. Petrilli was changing a school climate whose legacy was one of long-standing failure. She rubbed a lot a folks the wrong way. She canned some politically connected people. Lots of claims. A few investigations. So far...apparently nothing...no wrongdoing substantiated.

But what about the results?

POP QUIZ

The CATS results for 2007 will show that the percentage of proficient readers at Booker T Washington Academy will...

a. decline
b. stay about the same
c. increase by less than 7 points
d. increase by a little more than 7 points
e. increase by more than 10 points
f. increase by more than 15 points

The correct answer is.... f.

Thursday, July 19, 2007

Bluegrass Institute blames KDE for problem that could have been avoided

Take a peek at yesterday's post by Richard Innes on this year's late release of NCLB results in Kentucky.

He blames KDE for being unprepared for a situation they have known about since 2001; congratulates Fayette County for their aleting the parents they could; and liked the Post Op-Ed, as I did.

We don't fully agree on the proper response to the situation, but the Bluegrass Institute definately scores some points.