NCTQ Ranking of Teacher Prep Programs Gets an F

This from A "Fuller" Look at Education Issues

Background

In January, 2011, the National Council on Teacher Quality (NCTQ) and U.S. News & World Report announced their intention to evaluate and rank teacher education programs in the United States. The goal of this study is to review the quality of teacher preparation programs across the U.S., based on NCTQ’s standards for teacher education. This followed a series of reports assessing state-level Teacher Preparation policies across the nation.

U.S. News/NCTQ Ratings Project Discussion at AACTE 63rd Annual Meeting & Exhibits from AACTE on Vimeo.

On February 25, AACTE hosted a session at its Annual Meeting featuring Kate Walsh of the National Council on Teacher Quality, Bob Morse of U.S. News and World Report, Michael Feuer of George Washington University, Rick Ginsberg of the University of Kansas and Sharon Robinson of AACTE. Ms. Walsh and Mr. Morse spoke about the motivations for and methodologies employed in the U.S. News/NCTQ project, and Dr. Feuer and Dr. Ginsberg served as respondents offering concerns and recommendations for the project. Dr. Robinson moderated a Q & A session with the audience.

Because there are at least 1,400 university-based teacher preparation programs in the US, NCTQ relied on two methods to assess the practices of teacher preparation programs and the content in the courses offered by the teacher preparation programs. The first method was to ask preparation program representatives to respond to queries from NCTQ about programs. The second method was to collect and analyze the syllabi from the courses taken by students in teacher preparation programs.
The purpose of this blog is to examine the effort by NCTQ to evaluate, judge, and rank university-based teacher preparation programs. My comments are separated into five sections: (1) Inputs versus outputs; (2) Lack of research foundation; (2) Methodology employed; (3) Alternative programs ignored; (4) Superintendent critiques; and, (5) Ultimate impact.

Approach Focuses on Inputs and Largely Ignores Outputs

One of the two most important criticisms of the NCTQ effort is the almost unilateral focus on inputs and the lack of any consideration of outputs. While inputs are certainly important to a preparation program, the outputs are what matter. By outputs, I mean such outcomes as teacher placement, teacher longevity in the profession, actual behaviors of teachers in the classroom, and the effect of teachers on various student outcomes.

NCTQ claims the barriers to assessing outcomes are simply too large to overcome without the investment of hundreds of millions of dollars. In fact, I would agree with NCTQ on this point—assessing the outcomes of teacher preparation programs would be quite costly and difficult. For example, analyzing outputs such as placement, retention, and impact on student test scores would require states to collect and make available detailed data in a number of areas such as: teacher characteristics and prior experiences; teacher production, placement, and retention; the link between test scores and students;  the link between students and their teachers and teachers and their preparation programs; a wide variety of school characteristics; and, the characteristics of the principal. Most states do not collect such data and, even if they wanted to collect such data, lack the financial and human resources to accomplish such a huge undertaking. Further, even if states were able to collect such data and NCTQ was granted access to the data, appropriately analyzing such data is extremely difficult and may simply be impossible. Indeed, as Sass and his colleagues found in Florida, fairly comparing teacher preparation programs based on graduates’ impact on student test scores is not possible because of the need to control for unobserved characteristics of schools.

Given the extreme difficulty in assessing outcomes, I can understand why NCTQ did not try to assess outcomes. What is terribly troubling, however, is that NCTQ makes the quantum leap from inputs to quality preparation. Essentially, NCTQ claims it can assess the quality of a preparation program’s teachers based almost solely on a review of syllabi in some, but not all, courses taken by students in a program. There is simply not enough evidence or research to make such a leap (see below for further discussion of this) and NCTQ should have simply stated that they were evaluating and ranking preparation programs on inputs only and left it at that. This is simply another example of a think-tank not understanding issues surrounding research.

Lack of Research Foundation

The second major critique of the NCTQ effort is that there is little or no empirical research that substantiates the use of such standards in an effort to evaluate and judge the quality of a teacher preparation program. NCTQ admits that their standards are not grounded on a body of research. Note that the “study” lists the number of research reports underlying each standard. Yet, they do not list the papers. Are they peer-reviewed? What was the quality of the research? Was the paper a case study of one program or a large analysis of multiple programs? If NCTQ was really confident and transparent about the research foundation of their standards, they would have listed the papers and even provided links to them. Give me any education topic and I can list 20 papers on that tropic. Many won’t be pertinent or of high-quality, but it will sure look impressive to the non-researcher that I could say “there are 20 research reports that substantiate this standard.” You need both research QUANTITY and QUALITY in order to legitimately adopt a standard.

For example, NCTQ states: “[Our] standards were developed over five years of study and are the result of contributions made by leading thinkers and practitioners from not just all over the nation, but also all over the world. To the extent that we can, we rely on research to guide our standards.

However, the field of teacher education is not well-studied.” Note that the word researcher is omitted from this description. NCTQ relied on thinkers and practitioners, but not researchers. While thinkers and practitioners can provide useful insight, researchers are critical to such standard setting. In fact, many beliefs based on common sense turn out to be incorrect after research examines and issue.

While NCTQ is correct in that there is not a large body of research examining the inputs of teacher preparation programs and any outcome measures, there is some research and NCTQ apparently did not read it. Indeed, Eduventures (http://www.units.muohio.edu/eap/deansmessge/documents/EduventuresNCTQMethodologyCritique.pdf) correctly points out that the NCTQ standards include many inputs for which there is no research base that links the inputs to any type of output while some inputs that are important indicators of teacher preparation program quality are completely ignored by NCTQ and do not appear in the standards.

Eduventures (2010) notes that important inputs that should be assessed in an effort to link programs to outputs would include: the quality of instruction provided in teacher preparation and content courses, student support services, mentoring and induction provided by the program, and the length of the clinical experience required of students. Note that many of these inputs are simply absent from the NCTQ standards. NCTQ will respond that they don’t have enough money to properly conduct the study. So why do the study at all then? Is it okay to do a bad study because the money is not available to conduct a proper study? What would NCTQ say if a teacher preparation program claimed that they could do better if they had more money? I seriously doubt NCTQ would be sympathetic. Why should the public and the media be sympathetic with NCTQ?

Examples of Incorrect Reading of the Research

There are numerous examples of NCTQ standards that are simply not supported by existing research that was clearly available to NCTQ.

For example, NCTQ states that middle and high school preparation programs should require students to obtain a content-area major or at least 30 hours of content courses. Yet, Monk (1994) found diminishing returns to teacher effectiveness past five courses at the high school level. Moreover, at the middle school level, Alexander and Fuller (2003) and Darling-Hammond (personal; communication) found that middle school teachers trained in as elementary teachers were more effective than middle school teachers trained as secondary subject area specialists.

With respect to student teaching, NCTQ requires that preparation programs ensure that cooperating teachers for student teachers be proven effective instructors as measured by student achievement. While this makes common sense, there is no research base to support this contention. Further, NCTQ does not say how districts should use student achievement to assess teacher effectiveness. There is certainly no consensus in this area and there will be great variation in the quality of efforts of districts to do this. Ultimately, the measure is meaningless because there is no method to ensure districts assess teachers in an appropriate and accurate manner.

Also with respect to student teaching, NCTQ states:

When evaluated in the context of teacher preparation programs that are in relative geographic proximity, the proportion of a program’s student teaching placements that are made in schools that can be classified as “high functioning and high needs” can signal a commitment to ensuring that all teacher candidates experience teaching in such learning environments.  For purposes of classification, schools are designated as “high functioning and high needs” if:  

• Average student performance in reading and mathematics both exceed the district average or the school has been designated by its state as having recently made significant improvements in average student performance in reading and mathematics.
  AND
• Forty percent or more of students are eligible to receive free or reduced-price meals.

Again, this standard has no research foundation. Moreover, it is highly problematic. If districts use percent proficient to determine student performance, then the determination could very well be incorrect. Even more problematic would be the use of the change in percent proficient to assess growth. This almost always provides an inaccurate judgment about progress (Koretz, 2008).  This makes me strongly suspect NCTQ does not even understand basic assessment issues.

Another aspect of student teaching that is excluded from the standards is the existence of a capstone project. NCTQ uses the standard of five observations that was found to be statistically significantly associated with teacher practice in a study by Boyd and his colleagues (2008). Yet, the very same study found the existence of a capstone project has the same impact. Why did NCTQ pick one finding and ignore the other?

Perhaps most importantly, the standards for reading/English and mathematics instruction are not grounded in the research that is clearly evident in the standards developed by the National Council of Teachers of English or the National Council of Teachers of Mathematics. Why NCTQ believes they know more about instruction in these fields than actual experts is beyond me.

Incorrect Use of Research

As is typical with think-tank reports, the authors at NCTQ do not understand how research should be used. There is actually some high-quality research on the link between teacher preparation practices and outcomes, but we certainly need much more to make definitive conclusions about best practice. But using research to identify potential best practices and using research to rank institutions are two totally different uses of research. When researchers like Donald Boyd and his colleagues conduct research, they are looking for patterns in the data to be able to say a certain characteristic of teacher preparation programs is associated with improved teacher practice or greater student achievement. Such research is extremely useful. However, such research does NOT conclude that EVERY teacher preparation that employed a particular strategy was high-performing or that EVERY teacher preparation program that did not use a particular strategy was low-performing. The researchers conclude that teacher preparation programs tend to have better outcomes if they use a particular strategy, but that some programs that use the strategy are low-performing and some that don’t use the strategy are high-performing.

NCTQ, however, completely mis-uses the research by contending EVERY program MUST use a certain strategy. That is simply NOT what research says or what researchers would advocate in terms of how the data should be used. There is widespread consensus that research should not be used this way which is why researchers are loath to rank programs. They know that rankings will be inaccurate and cause harm to good people who run effective programs and give undue recognition to ineffective programs...

 Emphasis in original.