Think Tank Reports “Severely Flawed”
Most reports from education policy think tanks are
not worth the paper on which they are printed, according
to a new education policy research center. The Think
Tank Review Project, a collaborative research project
at the Education
and the Public Interest Center (EPIC) at the University
of Colorado at Boulder and the Education
Policy Research Unit (EPRU) at Arizona State University,
seeks to review reports from think tanks and to comment
on their research methods, reliability, and validity.
Kevin Welner, Director of EPIC, and Alex Molnar, Director
EPRU, are the Co-founders and Co-directors of the Think
Tank Review Project, and they described
the findings of the project’s first year of
operation in a February issue of Education Week.
The Project’s review
panel, which brings together 24 top education researchers
from across the country, evaluated 13 major think tank
reports from 2006 and found many “disturbing”
elements in the reports.
The Worst Offenders
The worst offenders, according to the Project, are
the Arlington, Virginia-based Lexington
Institute, the Washington-based Thomas
B. Fordham Institute, the Harvard Program
for Education Policy and Governance (PEPG), and
the New
York City-based Manhattan Institute. These think
tanks, according to reviewers, published reports that
were “selectively data-mined” and “seriously
lacking in methodological rigor.” To list a few
charges, Welner and Molnar criticized these organizations
for their presentation of “conclusions that their
own data and analyses flatly contradicted” and
also for their “resolute use of statistics to
achieve a desired outcome.”
Welner and Molnar gave special recognition to certain
reports for being particularly biased or problematic.
The Fordham Institute’s “The State of State
Standards 2006,” for example, was panned by reviewers
for its bias and its methodological flaws. Another special
mention went to the PEPG’s “On the Public-Private
School Achievement Debate.” After the National
Center for Education Statistics released a report last
summer showing that the NAEP
scores of public and private school students did not
significantly differ after correcting for the socio-economic
background of the students, Paul Peterson and Elena
Llaudet re-analyzed the data – in a manner criticized
by the review panel – in order to show private
schools performed better.
Caveat Emptor
The problem with think tank reports, according to Welner
and Molnar, is that they are not actual research, because
that they do not study original research questions,
nor do they use rigorous methodological standards. Think
tank “policy briefs,” Welner and Molnar
say, are heavily biased by “highly selective reviews
of the literature” and “a necessarily skewed
reading of the insights offered by that research.”
Even the reports that claim to be analyses of data have
“problems ranging from severely flawed data to
inappropriate methods, to broad conclusions not supported
by the evidence provided.”
The reason for the Think Tank Review Project, Welner
and Molnar say, is that academics generally ignore think
tanks as non-scientific and non-academic, while media
outlets give think tanks enormous coverage as valid
and unbiased. The Project exists to provide a warning:
beware of what you buy; these reports are not a substitute
for real research and rigorous debate.
Prepared by Matthew Samberg, March 15, 2007
|
