What Happens When a Publisher Doesn’t Have Scientific Evidence?

A letter from Citizens for Responsibility and Ethics in Washington (CREW) to the Inspector General of the U.S. Department of Education raises important issues. Although the letter is written in a very careful, thorough, and lawyerly manner, no doubt most readers will notice right away that the subject of the letter are the business practices of Ignite!, the company run by the president’s brother Neil.

CREW documents that Ignite! has sold quite a few units of Curriculum on Wheels (COW) to schools in Texas and elsewhere and that these were purchased with NCLB funds. They also document that there is no accessible scientific evidence that COWs are effective. Given the NCLB requirement that funds be used for programs that have scientifically-based evidence of effectiveness, there appears to be a problem. The question we want to raise is: whose problem is this?

The media report that Mr. Bush has responded to the issues. For example, this explanation appears in eSchool News (Nov. 17, 2007):

* In his interview with eSchool News, Bush said the watchdog group has misinterpreted the federal statute.
* “We’re proud we have a product that has the science of learning built into its design, with tons of anecdotal evidence,” the Ignite! founder said. “But we don’t yet have efficacy studies that meet the What Works Clearinghouse standards–in fact, I challenge you to find any educational curriculum that has met that standard.”

Mr. Bush appears to suggest that NCLB requires only that products incorporate scientific principles. This suggestion is doubtful, outside Reading First, which had its own rules. With respect to actually showing scientifically valid evidence of effectiveness, he concedes that none exists for COWs, but points to the fact that his company’s competitors also lack that kind of evidence.

We came to two conclusions about CREW’s contentions: First, their letter suggests that Ignite! did something wrong in selling its product without scientific evidence. A perspective we want to suggest is that nothing in NCLB calls for vendors to base their products on the “science of learning,” let alone conduct WWC-qualified experimental evidence of effectiveness. Nowhere is it stated that vendors are not allowed to sell ineffective products. Education is not like the market for medical products, in which the producers have to prove effectiveness to get FDA approval to begin marketing. NCLB rules apply to school systems that are using federal funds to purchase programs like COW. The IG investigation has to be directed to the state and local agencies that allow this to happen. We think that the investigators will quickly discover that these agencies have not been given much guidance as to how to interpret the requirements. (Of course with Reading First, the Department took a hands-on approach to approving only particular products whose effectiveness was judged to be scientifically based, but this approach was exceptional.)

Our second conclusion is that the current law is unenforceable because there is insufficient scientific evidence about the effectiveness of the products and services for which agencies want to use their NCLB funds. The law needs to be modified. But the solution is not to water down the provisions (e.g., by allowing anecdotal evidence if that’s all that is available) or remove them altogether as some suggest. The idea behind having evidence that an instructional program works is a good one. The law has to address how the evidence can be produced while supporting local innovation and choice. State and local agencies will need the funds to conduct proper evaluations. Most importantly, the law has to allow agencies to adopt “unproven” programs under the condition that they assist in producing the evidence to support their continued usage.

CREW’s letter misses the mark. But an investigation by the IG may help to ignite a reconsideration of how schools can get the evidence they need. —DN