NCTQ Method Stirs Backlash from Ed Schools

Sorry for the formatting on this long piece. Something has changed at Blogger or with IE and it has gone C R A Z Y. Richard

Instead of making an open records request, last week, I found myself responding to one. The request for my course syllabi came from the National Council on Teacher Quality a Bush-era education non-profit, established for the purpose of opposing, or "countering," traditional teacher organizations and current structure of the profession.

So, who are we talking about here?

NCTQ was one of a handful of groups, partly funded by government grants, whose advocacy for certain Bush policies through Op-Eds and other publicity, came under scrutiny for failing to follow the anti-propaganda rules. According to the Department of Education's Office of Inspector General report on Department PR expenditures from 2002 - 2004, NCTQ and the Oquirrh Institute (which former Ed Sec Margaret Spellings apparently paired) received $677,318 to "increase the American public's exposure and understanding of the research and full spectrum of ideas on teacher quality." In 2005, the Office of the Inspector General for the U.S. Department of Education called the NCTQ grant into question on a couple of counts. First, the unsolicited grant to NCTQ had been approved although two out of three reviewers had recommended against it. Secondly, NCTQ president Kate Walsh had run Op-E ds without including the legally required EDGAR disclosure to protect against covert propaganda.

The Inspector General raised two principal concerns:

• "The failure of these grantees to include the required disclaimer appears to have resulted in an improper expenditure of grant funds that should now be recovered.


• We did find…that the level of involvement by the Office of the Secretary in the initial approval of these unsolicited proposals and the oversight of these grants raised some concerns.

Oh great. Is this just another political group, posing as something it isn't, while looking to do a hatchet-job on the profession?

Walsh has a little history herself. NCTQ was a spinoff of the Education Leaders Council, a conservative-leaning group of education officials that also got the American Board for Certification of Teacher Excellence (ABCTE) started in 2001, and was labeled a "high-risk grantee" for its questionable handling of millions of dollars in federal grants. See SourceWatch for lots more on NCTQ.

Questions about the proposed NCTQ "research" methodology for assessing teacher prep programs have been percolating for more than a year. In an April 29, 2010 in the Chronicle of Higher Education article concerns were raised.

"The report in question is an assessment of teacher education programs. Yet, there is no explicit attention to the core concepts of assessment: validity and reliability. Further, there is no literature review establishing a basis for what comprises an appropriate and effective teacher education program and many key assertions in the report are unsupported by citations from the teacher education literature. The report was apparently not peer reviewed, being published by its authors and their sponsoring agency, not by a reputable journal. There is also evidence that those conducting this report subscribe to a contentious, narrow view of reading and how it should be taught…"

According to the NCTQ method, Teacher Beat reported, each school will be reviewed against 19 standards and graded on 17 of them. These standards have been revised and reduced in number since NCTQ did its review of Illinois and Texas education schools, last year, essentially a "field test" for the larger project. Alternative routes to teacher certification will be included in the review, but it won't include Teach For America or district-created programs that operate independently of ed. schools.

Penelope Peterson, the dean of the School of Education and Social Policy at Northwestern University and a participant in the NCTQ field test in Illinois said,

"There is nothing wrong with rating schools as long as the rankings are transparent and accurate. I would call it a Consumer Reports approach to rating teacher education. I don't think in any way you could call this a sophisticated research study."

Peterson has first-hand knowledge of the NCTQ process and her experience is revealing. Northwestern ranked highest out of 111 programs during the Illinois evaluation and was the only school to receive an A. But the rating was only given after granting NCTQ more access.

"Basically, we would have failed for the standards if we hadn't presented additional data and argued with them. If they just used the public data — for example, whatever they could find on our website — it would have been very inaccurate and misleading."

So cooperating, arguing, and providing information beyond that used in the paper-review process NCTQ will use for the vast majority of Ed schools (including those in Kentucky) was necessary for the prestigious Northwestern to go from an F to an A.

Kentucky is not cooperating. It would seem that Kentucky is already screwed.

For example, NCTQ rates schools based on their "value-added," or effectiveness of education students in the classroom after they graduate, Peterson told the Daily Northwestern. While states like Florida track these statistics, Illinois does not. Neither does Kentucky, and there are very serious concerns about the validity of such data to begin with.

"I don't know where all my students are that have graduated from our school, and, furthermore, I don't have access to their student achievement scores. So we got a zero on it."

Flunking states that are unable to provide the particular data NCTQ believes is important (without regard for local control) is unfair to schools, and when it produces a knowingly false result, which is then reported, it is dishonest.

It would seem that Walsh has already made up her mind (a red flag for bias and pseudoscience) and all that remains is to discredit the schools.

Kenneth Teitelbaum, the Dean of the College of Education and Human Services at Southern Illinois University Carbondale drew a clear analogy to demonstrate this problem in Group's Report Poorly Done, Lacking in Data:

Imagine an organization that decides to assess doctor preparation by establishing its own standards rather than those embraced by the American Medical Association. Or something similar for lawyers, engineers, nurses, police officers, etc. This is what NCTQ does. Whatever our own major professional associations subscribe to, or whatever the research shows, NCTQ assumes its own standards and then assesses our programs based on them. In addition, they do no direct observations of practice, no interviewing of students and school and community partners, and very little follow up of the factual errors that we call to their attention. They simply look at course syllabi, our website and the University catalog, all very limited indicators of what actually takes place in our courses and field experiences and intended as such. How can one come to grand conclusions about the quality of an elementary education or special education program from such limited information? Apparently NCTQ thinks you can. In my view, and those of my colleagues, their efforts would not be sufficient to pass an undergraduate research course.

As pointed out at the Quick and The Ed, NCTQ evaluation completely fails to assess what teachers know and can do. It does not look at how, or whether, what teachers do impacts student learning. Rather, judgments appear to be based on what content is included in course syllabi. How the syllabi are reviewed, coded, or rated is not apparent. There is little evidence that the content NCTQ is looking for on syllabi affects teacher effectiveness.

Among the remaining concerns: The methods that NCTQ will use; the perception that NCTQ is biased against colleges of education and against teacher education; The NCTQ ratings will not be based on what teacher candidates learn in their programs; The survey does not meet standards of high-quality research; concerns over prior NCTQ surveys of teacher education (in Illinois and Texas); The NCTQ does not rely on other professional consensus documents for the standards it is using.

In short, the survey is not research, and its "methods" do not seem appropriate for any professional consideration.

In January 2011, NCTQ announced its partnership with U.S. News and World Report, and launched a review of the nation's 1,400 education schools that is due to be published in 2012. But when they announced their methodology, problems remained.

The American Association of Colleges for Teacher Education responded with a member alert.

"[NCTQ's] efforts have utilized methodologies that do not meet the standards of basic scientific research. For example, they have assessed course syllabi and handbooks against "standards" – that are neither research based nor representative of any established consensus – as a means of evaluating teacher preparation programs. AACTE has consulted with its member institutions as they have encountered difficulties with NCTQ...Institutions being contacted by NCTQ can thus be aware of the experience of others as they consider participation in any way with NCTQ "studies".

In a February letter, officials from 35 leading education colleges and graduate schools — including Columbia, Harvard, Michigan State and Vanderbilt — denounced an "implied coercion" if they do not cooperate with the ratings. The deans said,

"Equating missing data with instrumental failure is simply dishonest. And doing so would surely result in a devaluing of the overall rating."

NCTQ President Kate Walsh responded to concerns saying,

"As for the consequences for institutions that choose not to cooperate, let me be clear that all institutions will be rated regardless of their decisions. There are standards which do not require cooperation, and we will rate those as planned. After hearing your concerns, we have decided, however, not to automatically fail institutions that do not participate. We instead expect to estimate for the remaining standards based on the material that we are able to assemble. Those ranking based on estimates will be clearly labeled. However, the public will be informed that the school refused to supply information needed and that alternative methods were used to develop the rankings. This is a format used by U.S. News. If you have an alternative method that will encourage cooperation, we are certainly open to considering it."

Brian Kelly, the editor of US News and World Report, acknowledged this concern during an interview with New York Times and said NCTQ will stop failing schools that don't have statistics for certain standards. "We regret that language. "It's really not the way we want to be doing business."

"new rankings will not be published until sometime in the second half of 2012. NCTQ is all ready working on data collection and has sent surveys to the schools. NCTQ will collect all the data and produce the ranking. U.S. News will carefully vet the results before they are published to make sure they are up to our standards."

In mid March, Kentucky Education Commissioner Terry Holliday, along with EPSB's Phil Rogers, CPE's Bob King joined the leadership of Kentucky's private and public colleges and universities and a growing list of state education leaders who have objected to the methodology used by the NCTQ.

In a letter to NCTQ composed by UK Ed Dean Mary John O'Hair, the Kentucky leadership said,

"we have met with additional individuals and agencies in Kentucky to reassess our decision regarding endorsing the NCTQ report to be sure we are doing what is best to address that important, commonly held goal. After we carefully examined NCTQ's response, participated in webinars and attended the NCTQ presentation in San Diego at the 2011 AACTE annual conference, we stand with the majority of colleges and universities across the U.S. that have elected not to endorse the NCTQ investigative report…

"…we remain convinced that your proposed investigative report fails to measure vital activities that we believe would more accurately inform the public of the quality of our teacher preparation programs. Hence, we cannot in good conscience endorse the methodology or results of your effort."

James Cibulka president of NCATE (and former UK Ed Dean) acknowledged the flaws in the NCTQ paper-review methodology without specific comment, and vowed that NCATE accreditation assessments would

"remain totally independent. How an institution fares in the ratings or even whether it decides to participate with NCTQ/USNWR at all can have no bearing on our accreditation decisions, and there should be no appearance of entanglement. We will fairly consider for accreditation both those that participate in the NCTQ/USNWR ratings and that do not."

Cibulka, and virtually everyone would agree that teacher education programs can be improved. But he rejected as false information NCTQ President Kate Walsh's claim that he agreed with her view that teacher education in schools of education is broken.

The federal education secretary, Arne Duncan, has said that many, if not most, teacher-training programs are mediocre. "It is time to start holding teacher-preparation programs more accountable for the impact of their graduates on student learning," Duncan said in a speech in November.

So, if Duncan's plan comes to pass, teacher educators (but not general education teachers in other colleges within the universities, and not Teach for America) will be held accountable for the actions of their "grandstudents" – whom they will have never met. I'm trying to think of another field where such an arcane notion might be in practice.

It also must be noted that while Holliday objects to the bad methodology in this case, he has flirted with significant parts of the same shaky ideology for use within Kentucky. He told the 2010 gathering of the Kentucky Association of Teacher Educators that Ed schools "ought to have some skin in the game" and he signaled an interest in evaluating teacher ed programs by looking at the results of these grandstudents.

Considering that Kentucky does not yet have an effective and (relatively) fair way of assessing the value-added to student achievement by P-12 educators (one that factors in all variables, including socio-economic status) one shudders to think of the level of reliability, validity and fairness one might expect from this new plan. While I fully understand the need to motivate the troops, we don't need ideas that are big-hearted but empty-headed.

The Kentucky leaders clearly indicated to NCTQ, that those who are supposed safeguard the integrity of research methodology ought not support what they know to be bad design.

Assessments are underway [in Kentucky] as multiple partners have developed a comprehensive accountability system for Kentucky's teacher preparation programs. The system spans across the teacher's career with monitoring points built into the system that our programs will use for program improvement and the EPSB will use for accountability. The following points will be monitored and provided to the public through the EPSB Data Dashboard:

• 
  
  Admission standards/criteria continuously monitored including minimum GPA of 2.75, a rigorous basic skills test for all teacher candidates, and candidate selection procedures that consider whether a candidate has skills in communication, collaboration, critical thinking, and flexibility.


• 
  
  Teacher Performance tasks built into student teaching with a focus on K-12 student learning within a co-teaching setting with the cooperating teacher.


• 
  
  Minimum cut scores on Praxis II with recognition for programs with candidates scoring in the 85th percentile and above.


• 
  
  The use of teacher performance assessments within the teacher's first year of employment.


• 
  
  The inclusion of teacher working conditions survey results.


• 
  
  Aggregate results for reviews of teacher preparation programs.


• 
  
  Teacher retention data included in program reviews.


• 
  
  School performance data of K-12 schools where at least 50% of the teachers are from a specific teacher preparation program.

The framework and the accountability systems in Kentucky, while not perfect, provide a much more accurate picture of quality and continuous improvement than NCTQ's present attempt.

"…we agree that grounded clinical practice, preparation of elementary school teachers, preparation of secondary school teachers, preparation of special education teacher, and entry into the program and profession are all vital elements in determining the effectiveness and quality of a teacher preparation program…we believe the methodology that you have developed to measure these standards inadequately demonstrates the effectiveness of a program, we cannot endorse your study."

EKU Education Dean Bill Phillips sees a narrow set of concerns underlying NCTQ's method. It all has to do with teaching reading and the touting of certain brain research, but that is not readily apparent.

Phillips' first involvement with the issues came when Kentucky's education deans were recruited to participate in the study. That started a conversation among the deans "as to whether or not we should [participate] because of the past history of their methodology." Then followed a national discussion via email with the US education deans and various organizations like AACTE; and the consensus of almost all of the deans was that "[NCTQ] really had just been there to try and stir up the public against colleges of education."

That led to Dean O'Hair writing a letter that was shared and edited by the Kentucky group.

And what is the thinking behind not participating?

"Our thought was to send the letter and tell them what our concerns are and then, hopefully, they will change the methodology." Much of Kentucky's concern was that the NCTQ method only looked at inputs to the system and then saying, without evidence, that those inputs affect the outcomes for students.

"It's partially true. Certainly, if you take the very top minds in the country and put them in teacher education; the students are coming out, from Harvard, with a good education, and from Columbia and schools like that. They come out very bright. But they started very bright. Because of the work on…dispositions…a lot of, particularly regional universities around the country that say, we really can change a person's knowledge and behaviors coming out of the university," Phillips said.

Phillips thinks the argument NCTQ and others are trying to make is that Ed schools should only accept the very best and brightest into teacher education – the top third. "They claim," Phillips said, "that we take the bottom third…and as a result, we have poor quality."

One wonders what would happen to regional universities' service to their states and the impact such a policy might have on multicultural diversity among the American teaching force. One wonders if such a policy suggests that education is wasted on the undereducated.

Phillips discovered in the process of reviewing the requested information that NCTQ is particularly interested in the latest brain research, which was not readily apparent from the initial request. Phillips didn't know what they were looking for until the open records request provided some revelations.

"They want to make the case that we're not using the latest brain research in teaching reading. So they have asked for the syllabi, and then gone on to ask for all kinds of other information about who we admit to programs, how we admit them, our policies on student teaching, and tenure, agreements we have with school districts, and, it's pretty exhaustive. Basically it's undergraduate, initial certification that they're looking for. And what they said was that if you don't participate in this then we're going to give you an F and we'll embarrass you. Well, what they found out was that only 10 percent of the colleges in the country agreed to the survey."

That group includes Western Kentucky University.

Now, in states like Wisconsin and Michigan, other conservative groups are asking for open records on faculty emails. This new tactic is being compared to McCarthyism and perhaps underscores the importance of maintaining tenure for faculty.

My quick take is: We'd better keep faculty tenure, but turn over the emails.

As for the true motivations of Walsh and NCTQ, evidence supports a conclusion of outrageous bias. Seattle Education 2010 did some digging but found Walsh's vitae tough to locate. The closest SE2010 got to her background in education was an excerpt from a book titled The Teachers We Need vs. The Teachers We Have by Lawrence Baines. In chapter 7 he writes: The

National Council on Teacher Quality (NCTQ) is another organization that promotes alternative certification while attempting to masquerade as an objective, research-focused agency. The similarity of the name of the National Council on Teacher Quality to the National Council for the Accreditation of Teacher education (NCATE) is no coincidence.

Whereas NCATE advocates rigorous standards for teachers, including a full-semester or longer of student teaching and challenging and relevant course work, NCTQ advocates a student teaching experience of a few weeks and limited course work. Unsurprisingly, the president of NCTQ is an alternatively certified teacher who started the first alternative certification program in Maryland. Chester Finn, who sits on the board of directors of the NCTQ also happens to be the President of ABCTE [from which Walsh resigned amid controversy in 2005].

Thus, two organizations (NCEI and NCTQ) who provide the federal government and state agencies with data on alternative certification are also dependent upon the continuing proliferation of alternative certification for their survival. Given this reality, it seems unlikely that either NCEI or NCTQ will ever have anything negative to say about alternative certification.

Let's review the facts. The Chief Executive Officer of a business that provides alternative certification for a fee (Chester Finn, of ABCTE) is on the board of directors of the organization that provides the reports (NCTQ) that promote the benefits of alternative certification. Not only has the federal government failed to launch an independent evaluation of teacher quality, it has relied upon NCEI and NCTQ to provide data about the quantity and quality of alternatively-certified teachers.

When a wolf is appointed to guard the sheep, one must expect that casualties will be heavy. As teacher certification across the United States has gotten easier, quicker, and more profitable for the wolves, the sheep have started disappearing.

Eventually, one would hope that the rationale upon which the alternative certification business empire has been built—that unprepared, inexperienced students with poor academic records are somehow superior to well-prepared, experienced teachers with stellar academic records—will not stand.