School and district leaders have extraordinarily difficult jobs. They make decisions every day about how to improve student learning, keep kids safe, and manage large staffs, often with limited time, resources, and information. They should not need a research PhD to judge whether the evidence presented by an education technology company can be trusted.
Yet they frequently do.
Companies are going to present their products in the most favorable light. That happens in every industry. But there is a meaningful difference between positive marketing and outright misrepresentation about whether a product has been shown to improve student learning. Too often, that line is being crossed.
The consequences extend beyond individual purchasing decisions. When education leaders cannot trust the evidence they’re shown, it becomes harder to distinguish effective products from ineffective ones. And companies that communicate their findings responsibly face pressure to make more aggressive claims to compete.
The result is an edtech evidence market with a serious trust problem.
What misleading claims look like
Misleading claims take several forms. Sometimes companies leave out inconvenient findings. Sometimes they overstate what their studies show. And sometimes they make claims directly contradicted by the evidence.
Consider four examples:
- Omission. A vendor reported that students who used its reading app scored higher on state tests than those who didn’t. That claim was technically true, but left out that students chose whether to use the app, meaning they may have differed from their peers in ways that the study couldn’t measure, like motivation. The study showed that app use and higher scores were correlated, not that the app caused the better outcomes.
- Overclaiming. A vendor advertised that its product delivered seven months of additional learning, but those gains were confined to a small group of “super users,” just 4% of students, who completed far more problems than their peers. Presenting results that way implies an effect that may not be representative or achievable at scale. This is commonly referred to as the 5% problem. When a small group of heavy users see outsize benefits, vendors should work to get more students using the tool that way, not present the super users’ results as typical.
- Cherry-picking. A vendor’s website highlighted large gains for English learners. The study’s overall effect was modest, and the finding for English learners was based on an exploratory look at a small group of students, too few to determine whether the difference was real or due to chance. The researchers flagged it as a pattern worth further study, but the vendor presented it as a proven benefit.
- Misrepresentation. In reviewing grant applications, we have encountered claims about research findings that were directly contradicted by the research reports attached to those same applications. In one case, an applicant claimed that a small pilot had demonstrated positive impact, but the attached report showed notable implementation challenges and mixed results — some positive, some negative, and none statistically significant.
These aren’t obscure methodological disagreements. They are examples of organizations presenting evidence in ways that leave decision-makers with a fundamentally inaccurate understanding of what the research found.
Of course, district leaders understand that vendors are trying to sell them something. They know to be skeptical. But recognizing that you are being marketed to doesn’t make it easier to evaluate which claims are credible and which are not.
And when competitors are routinely inflating claims, the incentives for everyone else to follow become difficult to resist.
AI makes the stakes especially high
The speed at which AI products can scale brings this problem to the forefront.
One of the striking observations from reviewing Instruction Partners’ recent AI Learning Tour is the wide gap between product reach and evidence of effectiveness. Some products report reaching millions of students while having limited or no independent evidence that they improve student learning.
A lack of evidence does not mean a product is ineffective. Emerging technologies may hold promise, and rigorous research takes time. We should be creating opportunities to pilot, evaluate, and improve these tools, rather than adopting them wholesale without knowing the consequences.
Using an edtech product is not a do-no-harm scenario, even when the underlying idea seems sound. Poorly designed or implemented tools could lead to student disengagement, deliver inappropriate instructional content, exacerbate inequities, or reduce the cognitive struggle that learning requires.
Even a product that does no direct harm takes up instructional time that could have been put to better use.
The growing skepticism toward AI and educational technology is driven by many concerns beyond whether the products improve student learning. And the gap between inflated claims and limited evidence exacerbates it, since every overstated claim makes the next honest one harder to believe.
More evidence is necessary, but not sufficient
At Accelerate, we have invested in independent research on AI-powered educational technology through our Call for Effective Technology (CET) program. We believe the field needs much stronger evidence about whether these tools work, for whom, under what conditions, and at what cost.
But generating more research is only part of the solution.
As the evidence base grows, a second question arises: who shapes what school leaders hear about the findings?
An independent evaluation may carefully explain uncertainty, study limitations, and the distinction between correlation and causation. But if those caveats disappear when the results are translated into a press release, marketing presentation, or sales pitch, the research may do little to improve decision-making.
We risk moving from a market with too little evidence to one with more evidence but systematically unreliable claims about what it means.
That’s not much progress.
Funders face these pressures too. We want to find wins, celebrate promising results, and maintain positive relationships with the organizations we support. But our commitment is to finding what works for students, and that only happens if we report the evidence as it is, not as we hope it will be. That’s why providers and districts in our CET program commit up front to sharing findings publicly, whether they are encouraging, disappointing, or inconclusive. Independent researchers conduct the studies, and when results come in, we review them with external experts and publish our own “explainers” outlining what each study does and does not show. We have a responsibility to hold ourselves to standards at least as high as those we demand of others.
The goal is not to discourage innovation or set a burden of proof so high that edtech tools can’t reach the classroom. It is to establish shared expectations for transparency, rigor, and accuracy, including that companies communicate honestly about their evidence.
Building a more trustworthy market
We could hope that edtech leaders will act with integrity and demonstrate their commitment to students through their actions, not just their words.
But hope is not a strategy. The incentives in the market are unlikely to change without greater accountability from the organizations that help shape it.
Funders have a responsibility to hold their grantees accountable. Funding independent research is inadequate if the organizations we support subsequently misrepresent findings. We should expect public reporting of outcomes, including null or disappointing results, and accurate communication of what the evidence does and does not show.
Education journalists and trade media have an important role to play. Research claims deserve scrutiny that looks beyond a company press release. Journalists should examine the underlying studies directly, seek independent research perspectives, and treat claims that outrun the evidence as a story in its own right.
States and districts must regulate their markets. Preferred vendor lists and procurement processes should include evidentiary standards and demand transparency about research quality, implementation requirements, and student outcomes. Promising tools should have opportunities to demonstrate their value through pilots and phased implementation, with decisions about broader adoption informed by credible evidence of impact.
School and district leaders should have access to independent, trustworthy interpretations of evidence. They should ask hard questions of providers, look beyond marketing materials and testimonials, and seek independent expertise when assessing research claims. But it is unreasonable to expect overextended education leaders to independently adjudicate every company’s evidence on their own. The rest of the ecosystem needs to make that job easier.
And providers themselves have an opportunity to lead and show themselves to be honest brokers. A company that openly acknowledges what it has not yet proven, shares disappointing results, and makes improvements based on research findings should be rewarded for doing so, not disadvantaged relative to its peers making claims unmoored from the evidence.
Ultimately, the purpose of education research is not to supply marketing ammunition for companies competing for public dollars. It is to help us all make better decisions about how to improve student learning.
If we want an edtech market that rewards products that actually work, we need a market that rewards telling the truth about what works.
Nakia Towns is President and Luke Kohlmoos is Managing Director, Strategy at Accelerate.
[email protected] | [email protected]
