“If
percentile ranks were used to determine proficiency and recalculated
each year, then there would always be 70 percent of schools classified
as ‘(needs) improvement’ and this method would not consider how much a
school or district improved.”
---Dena Dossett, JCPS’ planning and
program evaluation director.
This from the
Courier-Journal:
When Donna Hargens, superintendent of Jefferson County Public
Schools, wants to illustrate how the state’s largest school district is
“moving and improving,” she tends to reach for one statistic in
particular:
 |
| Donna Hargens |
“We’ve risen from the ninth percentile to the
51st,” Hargens said during a GLI event in May. She used nearly the same
words during a Louisville Forum event last year, as well as at several
community conversations and school board meetings.
And
that fact is true. In 2011, the state ranked JCPS as being in the ninth
percentile of all school districts in Kentucky, making it one of the
lowest-performing in terms of state accountability measurements. In
2014, the state announced that JCPS had risen to the 51st percentile.
But
those numbers may be misleading to those who don’t understand how the
Kentucky Department of Education calculates scores, in particular
because what the state has been calculating in recent years is not truly
a percentile rank.
In fact, if you ranked districts in
2013-2014 based on their accountability scores, JCPS was only in the
27th percentile last school year, not the 51st.
The difference in numbers is due to an intricacy in the state's calculations.
“It’s
disappointing that something that seemed so (easy to reference) doesn’t
mean what it means in plain English,” said David Jones Jr., JCPS’ board
chairman. He said he has often used the percentile rankings as
“shorthand” to try to explain JCPS’ progress, saying he didn’t realize
until after talking with a Courier-Journal reporter exactly how the
percentile rankings were calculated.
To be sure, JCPS has
improved over the years. Last year, 96 of its schools met or exceeded
yearly progress goals set by the state — 20 more than did so the year
before. The district also has increased the percentage of students who
are college and career ready, and its graduation rate rose last school
year.
But other districts have made progress at the same
time, meaning JCPS has not moved up through the district ranks as their
percentile scores would suggest.
So as the state prepares
to release the latest round of test scores and accountability data Oct.
1, here’s a rundown of what percentile ranks of schools and districts
really means.
The basic gist is this: to allow school
districts to show improvement, the state periodically “locks” schools’
and districts’ scores to a prior-year percentile, meaning that more
recent “percentile rankings” are not true comparisons of how well one
school district did compared to others.
Here’s how it
works: schools and districts get an overall score based on a number of
factors, such as graduation rates, test scores, the growth of different
groups of students and college and career readiness.
In 2011-2012, schools and districts were ranked using those overall scores.
But
the next year, the state placed the new overall scores that schools and
districts received on the prior year’s percentile ranking scale.
For
instance, in 2011-2012, a score of 58.4 put districts in the 71st
percentile. In 2012-2013, districts who scored 58.4 were labeled in the
71st percentile, even though a score of 58.4 in 2012-2013 only put a
school district about halfway through the pack instead of better than 70
percent of other districts.
It gets a bit trickier after that.
After
the 2012-2013 results were released, the state added more components
that it knew would be part of the 2013-2014 accountability system,
creating a new overall score for 2012-2013 and resetting the percentile
ranking.
When the 2013-2014 scores came out, they were tied back to that new percentile ranking.
Now, the state will go through the same process again with the 2014-2015 scores.
Confused? You’re not alone.
Some educators and others have lamented the percentile rankings as convoluted and confusing for parents and others.
Confusion abounds
Even some district administrators struggle when trying to accurately explain the percentile rankings.
"The
percentile rank and growth have become so complex even people like me
struggle to explain it to parents and community groups," said Dewey
Hensley, JCPS' chief academic officer. "Hopefully the state will address
the complexity and ensure our accountability system communicates
clearly and fairly the progress districts make in this multi-layered
system."
Indeed, Hensley last week mentioned that JCPS
had moved to the 51st percentile during a meeting with the Louisville
chapter of the NAACP. Hargens, who was sitting with him at the meeting,
attempted to clarify the meaning of the percentile ranking, saying
that “the percentile rank is locked. … Our kids are progressing, but
our state has locked the percentiles.”
She didn’t explain what it meant to lock a percentile, and no one in the room asked.
Hargens
later said the statistic about JCPS’ percentile rankings is “never
intended to be the whole picture,” saying that understanding where
schools and students are requires looking at a number of data
points. “The more you try to make one number describe the whole picture,
the more you lose."
She said she tries to be honest with
the community, showing other data points and noting that while JCPS is
improving, “we’re still climbing the mountain.”
“We’re
working to make progress every day,” she said. “Our schools are working
to make progress every day. When they do, we recognize that and when
they don’t, we recognize that, as well. Behind every data point is the
heartbeat of a kid. … That’s that part people need to understand, the
big picture.”
The reason the state does these convoluted
calculations has to do with the way Kentucky’s accountability system is
set up to rely on percentile rankings to categorize performance.
Under
the state’s accountability system, schools in the 90th percentile or
higher are considered “distinguished,” while those in the 70th
percentile or higher are “proficient.” Any school or district below the
70th percentile is classified as “needs improvement.”
“If
percentile ranks were used to determine proficiency and recalculated
each year, then there would always be 70 percent of schools classified
as ‘(needs) improvement’ and this method would not consider how much a
school or district improved,” said Dena Dossett, JCPS’ planning and
program evaluation director.
“This model gives schools a
set standard to use as a goal, rather than setting a goal based on a
relative ranking that would change yearly.”
But the bottom line is that schools’ and districts’ percentile rankings are not exactly what they seem.
In
2012-2013, JCPS was listed as being in the 32nd percentile, when
actually it was only in the 14th percentile. In 2013-2014, its rank of
51st belied the fact that it was actually in the 27th percentile of
school districts in the state for its score.
And JCPS is
of course not alone; many districts and schools’ rankings are affected
by the way the state calculates the percentiles.
For
instance, Bullitt County Public Schools was ranked in the 75th
percentile in the 2013-2014 school year. But a comparison of district
scores showed that Bullitt was only actually in the 50th percentile.
Jan
Stone, Bullitt’s director of assessment, data and research, said her
superintendent asks her every year when test scores and accountability
results come out to calculate where Bullitt stands on a true ranking of
the state’s school districts.
“I’m not sure people
generally understand what that overall score means because it’s so
complex,” Stone said. “It’s my job to understand it, but when I talk to
teachers and others, I ask, do you really understand what your overall
score means?”
Changes sought
Some
district testing administrators and others have been suggesting
changes, saying that the “percentile ranking” calculation is difficult
for the average parent and consumer to understand.
“If it
takes me more than five minutes to explain the metrics, maybe it’s too
hard,” said Jennifer Davis, director of elementary and secondary
programs with Bowling Green City Schools. She noted that part of the
point of an accountability system is for people to understand how well
their school or district is performing.
“The system … is
so complex that a lot of people outside of education don’t understand
it,” Davis said. She said many people know what a percentile should
mean, so they often turn to that number as an easy way to understand
where a school or district stands.
She suggested the
state use overall scores to determine schools’ and districts’
classifications and “eliminate the percentile ranking altogether.”
“We are all calling it a percentile, and the state (is), too. In actuality, it isn’t a percentile,” she said.
When the 2014-2015 score results are released next week, Kentucky parents will again see the percentile rankings.
They
will be locked back to recalculated 2012-2013 rankings that will take
into account a number of changes in the 2014-2015 accountability system,
including the fact that the state is not counting elementary and middle
school science scores this year and is adding new program reviews.
But Rebecca
Blessing, spokeswoman for the Kentucky Department of Education,
recently announced that this year, for the first time, the state will
release a ranking of schools and districts based on current-year data,
as well as the locked calculation because it can be so tricky.
“We
hope that will provide some clarity and some transparency,” said Rhonda
Sims, an associate commissioner with the Kentucky Department of
Education. She noted that the state will also add a footnote about the
percentile rankings explaining the way the scores are locked.
Sims
added that the state is open to hearing ideas on how to improve upon
the accountability system, noting the state's new education
commissioner,
who is expected to be Stephen Pruitt, may have a different take.
But in the meantime, some remain frustrated.
“I have yet to talk to anyone who says, ‘We’re so glad to talk about percentiles,’” Davis said.
What is a percentile ranking?
Percentiles
are used to rank a group of things in comparison to each other.
Percentiles are used on a scale of 100, regardless of how many things
are in the ranking. For instance, if a group of 12 students took a test,
and one student did better than 92 percent of her classmates on the
test, she would be in the 92nd percentile. If a 6-year-old boy weighs
more than 80 percent of other boys his age nationwide, he would be in
the 80th percentile.
How the state calculated its percentile rankings, by year:
2011-2012:
The state used schools' overall scores to rank them in order, giving
them a true percentile ranking based on where they fell compared to each
other.
2012-2013:
The state took schools' overall scores, but instead of comparing
schools to each other like it did the previous year, it compared
schools' scores to where the scores would have fallen on the 2011-2012
ranking. For instance, in 2011-2012, a score of 58.4 put districts in
the 71st percentile. In 2012-2013, districts who scored 58.4 were
labeled in the 71st percentile, even though a score of 58.4 in 2012-2013
only put a school district about halfway through the pack instead of
better than 70 percent of other districts.
2013-2014:
Recognizing that it had made a number of changes to its accountability
system, including adding a new program reviews element, the state
retroactively recalculated schools' and districts' 2012-2013 rankings to
include the new components. Then, it took schools' overall scores for
2013-2014 and compared them to where they would have fallen on that
recalculated 2012-2013 ranking.
2014-2015:
The state said it will again compare schools' overall scores to
2012-2013 rankings, but that the 2012-2013 rankings will have again been
recalculated to account for more changes in this year's accountability
system.