AI Bias in Schools Demands New Testing Before Contracts

Shauna D. A. Knox, a former subject matter expert on human trafficking and child labor exploitation at the U.S. Department of Education, warns that schools are repeating past mistakes by failing to protect children from biased artificial intelligence. She argues no AI tool for teaching, assessing, or tracking students should be eligible for a school district contract until it passes a bias test built around the students digital tools have historically harmed.

Studies have documented racial bias in existing AI systems used in educational contexts. Last year, one study found AI assistants used by teachers recommended harsher approaches for struggling students whose names were perceived as Black. A separate study found an AI grader gave essays by Black students lower scores than essays by Asian students, replicating a gap present in human scoring.

The Case for Pre-Contract Bias Testing

Most school districts already conduct privacy checks before purchasing technology, evaluating whether a tool will keep a child's data private and secure. Knox proposes that bias testing should be integrated into this existing procurement checkpoint. The testing process would involve an evaluator giving the AI tool nearly identical student work, changing only details like the name on each sample or the background and speech patterns signaled in the writing, and then comparing the tool's responses. The evaluator would also examine what the tool teaches students about people, including which stereotypes it repeats and whose history it tells in full. A tool failing either examination should lose its eligibility for a school contract.

State Laws and District Bans Are Insufficient

While nearly every state has moved to regulate AI, and a handful have enacted chatbot laws for children, Knox contends these measures are inadequate. For instance, California requires a chatbot to tell a child they are using AI and to remind them to take breaks. New York State requires chatbots to detect expressions of suicidal thinking. However, the child protected while using AI at home has no comparable protections at school, as only a few states have passed laws directing education agencies and districts to manage AI tools.

Oklahoma's law is an exception, requiring an educator to review any AI-generated content before classroom use and prohibiting AI from being the primary basis for grades or promotion decisions. More typically, state laws are preliminary, merely directing agencies and districts to develop their own policies. Some large districts have implemented bans. This month, New York City barred students through eighth grade from using generative AI, and Los Angeles Unified blocked all students from using it on district devices. These bans, however, are a reaction to an inability to test tools, not a guide to which tools are safe.

Why Privacy Checks Cannot Catch AI Bias

The standard privacy checks run by most districts do not test for bias, as no state requires it and no law mandates companies disclose AI bias. These checks, focused solely on data protection, cannot determine if a tool built on a large language model is safe. The underlying AI can infer a child's race from their name, neighborhood, school, home language, or writing style, leading to differential treatment even when race is never explicitly named. Companies have trained models to stop saying explicitly biased things but have not removed the underlying bias itself.

Research illustrates the subtlety of this problem. When researchers gave AI models writing samples in the dialect many Black Americans speak at home, the models said nothing negative about Black people yet judged the writers themselves as less intelligent and matched them to less prestigious work. This demonstrates why tools must be retested even after they are deployed in classrooms, as the systems can change after they are sold.

A Weakened Federal Enforcement Mechanism

Previously, a biased tool could be removed from schools if statistical proof showed it was harming one group of children more than another. However, the U.S. Department of Education recently withdrew the rule that made this action possible. Now, its Office for Civil Rights will act only when someone can prove a school or district intended to discriminate. Statistical proof of disproportionate harm is no longer sufficient, meaning a biased tool can remain in use. Knox, writing for The Hechinger Report, concludes that without proactive bias testing mandated at the district procurement level, everything about the arrangement is protected except for the children.