Why a worldwide standard of quality needs to be applied to the evidence reform of EdTech

The epidemic’s impact on the use of educational technology (EdTech) shed light on systemic flaws in EdTech, from its conceptualization to its financing to its selection and implementation in classrooms. One proposed solution is the evidence reform for EdTech. Only by unifying disparate national efforts with a global strategy on what constitutes “evidence” in educational technology can the reform be realised.

Several proponents of educational technology have proposed major changes to the field in the wake of the pandemic. EdTech reform was proposed in reports from various national governments (including England’s), foundations (including the Jacobs Foundation), and research groups (including EdTech Exchange). The universal agreement that schools should only choose technologies that have been shown to have a positive impact on children’s learning is at the heart of the reform. Evidence in support of EdTech, however, is defined, measured, and mandated in significantly different ways in different nations.

According to the ESSA Standards of Evidence, which are used in the United States, randomised control trials are the gold standard. The United States government has established four tiers of evidence standards, each with progressively higher standards for efficacy. The What Works Clearinghouse catalogue features supporting non-regulatory guidance on how to measure the individual levels, as well as a list of recommended resources.

There is a wide range of EdTech evidence requirements and enforcement across European countries. There are countries that have invested in the creation of educational technology for widespread use (e.g. the Octavo Digital Library in Malta). Educators and government officials in other countries are given more autonomy in making policy decisions (e.g. Norway). Numerous academic groups, think tanks, and businesses in the UK have contributed to an extensive body of evidence framework (e.g. Educate Ventures or What Worked ). Countries outside of the Global North adhere to a wide variety of recommendations, the majority of which are both less rigorous and broader than the ESSA standards.

Based on national experiences, the 2023 GEM Report on Technology in Education aims to provide an overview of technology policy in education. The question of how to coordinate efforts on a national and international scale to collect and analyse more evidence of the effectiveness of EdTech is crucial. Most ed tech is made with a global audience in mind. While the content of individual platforms can be adapted to meet the needs of national curricula, the evidentiary foundation must adhere to international norms.

Academics agree that evidence must come from original research published in a reputable journal with strict peer review policies. For the past decade, however, a different definition of evidence has been used in the context of educational technology (EdTech): evidence in the form of teacher reports and reviews. EdTech products that have received high ratings from educators on sites like EdTech Impact and the Educational App Store tend to be prioritised by school purchasing departments.

Teachers’ experiences and intuitions about what works in the classroom are not at odds with objective evidence. In fact, scientific assessments of EdTech’s efficiency and effectiveness in fostering children’s learning should be combined with teachers’ firsthand accounts. Neither educators nor researchers have been able to coordinate their assessments of the available evidence. Currently, this void is being filled by a wide range of EdTech evidence providers, some of which rely on composite ratings when certifying or approving individual EdTech products. As an illustration, consider the recent mergers of two major EdTech certification bodies, the International Society for Technology in Education (ISTE) and the Association for Supervision and Curriculum Development (ASCD), as well as LearnPlatform and Instructure.

It takes a lot of trial and error and testing with lots of kids from lots of different schools to build up a good body of evidence. For this reason, it is prudent to employ a unified efficacy framework, like that proposed by ESSA, in order to streamline the process of testing the available evidence. Federal funding and VC investments in EdTech that aims to be more evidence-led would also be a wise move (e.g. as modelled by the Vital Prize). Because of the difficulty in defining evidence solely in terms of efficacy, randomised controlled trials (RCTs) have become the norm for educational technology. This contradicts the more expansive evidence definitions put forth by different countries. Efficacy standards have also been criticised for their potential to stifle innovation by inhibiting the growth of smaller, more agile companies.

As a capital-intensive sector, EdTech is particularly vulnerable to the impact of changes in international policy on its competitive landscape. For its part, the European Union (EU) has promised and become a counterweight to US “dominance” in EdTech with respect to privacy, but it is falling behind in the EdTech evidence race. The evidence framework and market mechanisms are the driving forces behind the United States’ educational technology market dominance. Threats to international efforts to create inclusive and welcoming EdTech spaces. This reality calls for a multifaceted strategy from the GEM Report that brings the demand for EdTech evidence in line with a defined body of international norms.