JCPS rise in state rankings not all it seems

“If percentile ranks were used to determine proficiency and recalculated each year, then there would always be 70 percent of schools classified as ‘(needs) improvement’ and this method would not consider how much a school or district improved.” 

---Dena Dossett, JCPS’ planning and program evaluation director.

This from the Courier-Journal:

When Donna Hargens, superintendent of Jefferson County Public Schools, wants to illustrate how the state’s largest school district is “moving and improving,” she tends to reach for one statistic in particular:

Donna Hargens

“We’ve risen from the ninth percentile to the 51st,” Hargens said during a GLI event in May. She used nearly the same words during a Louisville Forum event last year, as well as at several community conversations and school board meetings.

And that fact is true. In 2011, the state ranked JCPS as being in the ninth percentile of all school districts in Kentucky, making it one of the lowest-performing in terms of state accountability measurements. In 2014, the state announced that JCPS had risen to the 51st percentile.

But those numbers may be misleading to those who don’t understand how the Kentucky Department of Education calculates scores, in particular because what the state has been calculating in recent years is not truly a percentile rank.

In fact, if you ranked districts in 2013-2014 based on their accountability scores, JCPS was only in the 27th percentile last school year, not the 51st.

The difference in numbers is due to an intricacy in the state's calculations.

“It’s disappointing that something that seemed so (easy to reference) doesn’t mean what it means in plain English,” said David Jones Jr., JCPS’ board chairman. He said he has often used the percentile rankings as “shorthand” to try to explain JCPS’ progress, saying he didn’t realize until after talking with a Courier-Journal reporter exactly how the percentile rankings were calculated.

To be sure, JCPS has improved over the years. Last year, 96 of its schools met or exceeded yearly progress goals set by the state — 20 more than did so the year before. The district also has increased the percentage of students who are college and career ready, and its graduation rate rose last school year.

But other districts have made progress at the same time, meaning JCPS has not moved up through the district ranks as their percentile scores would suggest.

So as the state prepares to release the latest round of test scores and accountability data Oct. 1, here’s a rundown of what percentile ranks of schools and districts really means.

The basic gist is this: to allow school districts to show improvement, the state periodically “locks” schools’ and districts’ scores to a prior-year percentile, meaning that more recent “percentile rankings” are not true comparisons of how well one school district did compared to others.

Here’s how it works: schools and districts get an overall score based on a number of factors, such as graduation rates, test scores, the growth of different groups of students and college and career readiness.

In 2011-2012, schools and districts were ranked using those overall scores.

But the next year, the state placed the new overall scores that schools and districts received on the prior year’s percentile ranking scale.

For instance, in 2011-2012, a score of 58.4 put districts in the 71st percentile. In 2012-2013, districts who scored 58.4 were labeled in the 71st percentile, even though a score of 58.4 in 2012-2013 only put a school district about halfway through the pack instead of better than 70 percent of other districts.

It gets a bit trickier after that.

After the 2012-2013 results were released, the state added more components that it knew would be part of the 2013-2014 accountability system, creating a new overall score for 2012-2013 and resetting the percentile ranking.

When the 2013-2014 scores came out, they were tied back to that new percentile ranking.

Now, the state will go through the same process again with the 2014-2015 scores.

Confused? You’re not alone.

Some educators and others have lamented the percentile rankings as convoluted and confusing for parents and others.

Confusion abounds

Even some district administrators struggle when trying to accurately explain the percentile rankings.

"The percentile rank and growth have become so complex even people like me struggle to explain it to parents and community groups," said Dewey Hensley, JCPS' chief academic officer. "Hopefully the state will address the complexity and ensure our accountability system communicates clearly and fairly the progress districts make in this multi-layered system."

Indeed, Hensley last week mentioned that JCPS had moved to the 51st percentile during a meeting with the Louisville chapter of the NAACP. Hargens, who was sitting with him at the meeting, attempted to clarify the meaning of the percentile ranking, saying that  “the percentile rank is locked. … Our kids are progressing, but our state has locked the percentiles.”

She didn’t explain what it meant to lock a percentile, and no one in the room asked.

Hargens later said the statistic about JCPS’ percentile rankings is “never intended to be the whole picture,” saying that understanding where schools and students are requires looking at a number of data points. “The more you try to make one number describe the whole picture, the more you lose."

She said she tries to be honest with the community, showing other data points and noting that while JCPS is improving, “we’re still climbing the mountain.”

“We’re working to make progress every day,” she said. “Our schools are working to make progress every day. When they do, we recognize that and when they don’t, we recognize that, as well. Behind every data point is the heartbeat of a kid. … That’s that part people need to understand, the big picture.”

The reason the state does these convoluted calculations has to do with the way Kentucky’s accountability system is set up to rely on percentile rankings to categorize performance.

Under the state’s accountability system, schools in the 90th percentile or higher are considered “distinguished,” while those in the 70th percentile or higher are “proficient.” Any school or district below the 70th percentile is classified as “needs improvement.”

“If percentile ranks were used to determine proficiency and recalculated each year, then there would always be 70 percent of schools classified as ‘(needs) improvement’ and this method would not consider how much a school or district improved,” said Dena Dossett, JCPS’ planning and program evaluation director.

“This model gives schools a set standard to use as a goal, rather than setting a goal based on a relative ranking that would change yearly.”

But the bottom line is that schools’ and districts’ percentile rankings are not exactly what they seem.

In 2012-2013, JCPS was listed as being in the 32nd percentile, when actually it was only in the 14th percentile. In 2013-2014, its rank of 51st belied the fact that it was actually in the 27th percentile of school districts in the state for its score.

And JCPS is of course not alone; many districts and schools’ rankings are affected by the way the state calculates the percentiles.

For instance, Bullitt County Public Schools was ranked in the 75th percentile in the 2013-2014 school year. But a comparison of district scores showed that Bullitt was only actually in the 50th percentile.

Jan Stone, Bullitt’s director of assessment, data and research, said her superintendent asks her every year when test scores and accountability results come out to calculate where Bullitt stands on a true ranking of the state’s school districts.

“I’m not sure people generally understand what that overall score means because it’s so complex,” Stone said. “It’s my job to understand it, but when I talk to teachers and others, I ask, do you really understand what your overall score means?”

Changes sought

Some district testing administrators and others have been suggesting changes, saying that the “percentile ranking” calculation is difficult for the average parent and consumer to understand.

“If it takes me more than five minutes to explain the metrics, maybe it’s too hard,” said Jennifer Davis, director of elementary and secondary programs with Bowling Green City Schools. She noted that part of the point of an accountability system is for people to understand how well their school or district is performing.

“The system … is so complex that a lot of people outside of education don’t understand it,” Davis said. She said many people know what a percentile should mean, so they often turn to that number as an easy way to understand where a school or district stands.

She suggested the state use overall scores to determine schools’ and districts’ classifications and “eliminate the percentile ranking altogether.”

“We are all calling it a percentile, and the state (is), too. In actuality, it isn’t a percentile,” she said.

When the 2014-2015 score results are released next week, Kentucky parents will again see the percentile rankings.

They will be locked back to recalculated 2012-2013 rankings that will take into account a number of changes in the 2014-2015 accountability system, including the fact that the state is not counting elementary and middle school science scores this year and is adding new program reviews.

But Rebecca Blessing, spokeswoman for the Kentucky Department of Education, recently announced that this year, for the first time, the state will release a ranking of schools and districts based on current-year data, as well as the locked calculation because it can be so tricky.

“We hope that will provide some clarity and some transparency,” said Rhonda Sims, an associate commissioner with the Kentucky Department of Education. She noted that the state will also add a footnote about the percentile rankings explaining the way the scores are locked.

Sims added that the state is open to hearing ideas on how to improve upon the accountability system, noting the state's new education commissioner, who is expected to be Stephen Pruitt, may have a different take.

But in the meantime, some remain frustrated.

“I have yet to talk to anyone who says, ‘We’re so glad to talk about percentiles,’” Davis said.

What is a percentile ranking?

Percentiles are used to rank a group of things in comparison to each other. Percentiles are used on a scale of 100, regardless of how many things are in the ranking. For instance, if a group of 12 students took a test, and one student did better than 92 percent of her classmates on the test, she would be in the 92nd percentile. If a 6-year-old boy weighs more than 80 percent of other boys his age nationwide, he would be in the 80th percentile.

How the state calculated its percentile rankings, by year:

2011-2012: The state used schools' overall scores to rank them in order, giving them a true percentile ranking based on where they fell compared to each other.

2012-2013: The state took schools' overall scores, but instead of comparing schools to each other like it did the previous year, it compared schools' scores to where the scores would have fallen on the 2011-2012 ranking. For instance, in 2011-2012, a score of 58.4 put districts in the 71st percentile. In 2012-2013, districts who scored 58.4 were labeled in the 71st percentile, even though a score of 58.4 in 2012-2013 only put a school district about halfway through the pack instead of better than 70 percent of other districts. 

2013-2014: Recognizing that it had made a number of changes to its accountability system, including adding a new program reviews element, the state retroactively recalculated schools' and districts' 2012-2013 rankings to include the new components. Then, it took schools' overall scores for 2013-2014 and compared them to where they would have fallen on that recalculated 2012-2013 ranking.

2014-2015: The state said it will again compare schools' overall scores to 2012-2013 rankings, but that the 2012-2013 rankings will have again been recalculated to account for more changes in this year's accountability system.