Following a strong backlash against the “Gemini” pilot program, Google is facing calls to pause the use of its artificial intelligence in schools.

CHICAGO — On Wednesday, families in seven cities across the country held protests against what they view as Google’s attempt to embed its artificial intelligence models into schools—effectively turning children of all ages into guinea pigs for its “Gemini” AI and fostering an addiction to the technology.
The protests follow a statement released by Google last week, which confirmed that students would be unable to use Google Search without the integration of AI. The move has sparked significant backlash. While Google claims the change is intended to provide schools with “more choices, not fewer”—by allowing them to toggle off Google Search, Gemini, and Gemini Notebook—it fails to offer a search option that is free from AI. Parents are concerned that the only way for students to avoid interacting with the AI is to remember to append “-ai” to every search query they perform.
Despite cold, rainy weather in Chicago, families gathered to voice their opposition to Google. Dozens of protesters, carrying signs, marched toward Google’s downtown office just as employees appeared to be wrapping up their workday. They chanted slogans such as, “Hey, Google, don’t even try—we don’t need your Gemini!” and “Hey, Google, leave our kids alone!”
Led by the non-profit grassroots advocacy group “Illinois Families for Public Schools” (ILFPS), a coalition of parents and students has secured a victory in a significant battle over the use of artificial intelligence. In August, the group pressured Chicago Public Schools (CPS) to withdraw from a pilot program called “Gemini”—an initiative that had been slated to include all high school students across the district without prior consultation with their families. ILFPS is now calling for a moratorium on the use of AI until there is a better understanding of the risks posed to students, and until CPS can establish a policy to protect them from potential learning setbacks or other, more serious consequences stemming from the use of Google’s AI-powered chatbots. Should the group ultimately prevail, it would follow the lead of cities like Los Angeles and New York, which have taken a cautious approach by banning the use of AI in schools.
Lawsuit: Gemini Acts as a “Suicide Coach” for Students
Perhaps most concerning for parents is a lawsuit currently facing Google, filed on behalf of “Emma Doe,” a 13-year-old student from Durham, North Carolina. Emma claims that before she began using the Gemini chatbot on her school-issued Chromebook, she was a happy, well-adjusted, straight-A student. Her parents were completely unaware of her interactions with the AI; the lawsuit alleges that Google “deliberately” concealed this, effectively luring Emma into what it describes as an “AI-induced psychosis.” Over time, what began as routine school-related interactions with the chatbot escalated into increasingly intense conversations—eventually, the chatbot allegedly encouraged Emma to plan her own suicide and funeral. A school monitoring system flagged a chat log containing the word “funeral,” alerting her parents and averting a tragedy. Emma’s experience has sparked widespread alarm among parents, fueling protests not only in Durham but also in Boston, Boulder (Colorado), Chicago, Los Angeles, New York City, and Portland (Oregon).
The lawsuit alleges that Google leveraged its dominant position in the educational technology sector to surreptitiously integrate Gemini into Chromebooks, while falsely marketing the product as safe.
The filing states: “Because of Google’s choices, Emma was lured by its flawed product into repeated interactions with a sycophantic AI chatbot on her school-issued Chromebook—all without her parents’ knowledge. Had it not been for a fortunate turn of events, Google’s AI might well have manipulated Emma into taking her own life.” It further asserts: “Google must be held accountable for treating children’s lives as expendable in the pursuit of its own interests.”
Google has only responded publicly to concerns regarding potential harm to the learning process.
The company has pledged to “continue working closely with educators to ensure we provide schools with more choices, greater control, and enhanced search features for learning.” However, to date, Google’s response to parental concerns has focused solely on documented risks to learning. In this regard, Google claims it is addressing the issue of artificial intelligence undermining students’ critical thinking skills by launching another pilot program for students. According to Google, this pilot will provide students using AI models and “AI Overviews” with “step-by-step, guided learning—ensuring that students do not simply receive quick answers when completing their assignments.”
Cassie Creswell, executive director of ILFPS, told Ars that she has been focused on issues in the educational technology sector for over a decade. Even so, Google’s moves regarding artificial intelligence caught her off guard.
“I knew that AI overviews appeared in search results—and that they were problematic—but I had no idea that every student in Chicago Public Schools had an interface on their Chromebooks that allowed them to interact with Gemini just like a chatbot,” Creswell said. “It’s deeply unsettling.”
She acknowledges that children encounter AI on their personal devices, noting that her own daughter mentioned her friends prefer using ChatGPT over Gemini. However, she argues that the fact that some children use AI on their own devices does not mean schools should encourage its adoption—especially when children’s brains are still developing and the risks remain poorly understood.
“There are all sorts of dangerous things in the world,” Creswell said, “but our school district doesn’t go around handing them to children to keep on their desks.”
No one knows how Google uses student data.
Meanwhile, the schools themselves appear to be part of the transparency problem.
Creswell stated that the official position of Chicago Public Schools (CPS) is that “students do not have access to generative AI tools, and that because existing guidelines are already in place, there is no need for a district-wide policy on generative AI.” Yet, upon reviewing the contract between Google and CPS, she discovered that students are, in fact, being exposed to generative AI and cutting-edge models. This ongoing confusion has led parents to demand a “real policy”—one that would be “voted on by the Chicago Board of Education and developed with input from teachers, parents, students, and experts who do not receive funding from the tech industry.”
Creswell told Ars that parents are particularly concerned about Google’s lack of transparency regarding the handling and use of student data. Through a Freedom of Information Act (FOIA) request, she conducted an exhaustive review of Google’s contracts. She told Ars that none of the dozens of agreements she examined—involving partners of Chicago Public Schools such as Google, Anthropic, and OpenAI—specify how student data is handled. While she acknowledges that Google’s integration of artificial intelligence into its search function may simply be an “objective fact,” she argues that parents have a right to know how their children’s personal data is being incorporated into Google’s models.
“There is a complete lack of transparency regarding exactly what data is being collected and how it is being used,” Creswell said, adding that “it remains unclear what data they are actually retaining.”
Creswell and other advocates argue that children’s data is likely being collected regardless, noting that “even if you use the ‘-AI’ search term and don’t receive a text summary, your search data and the returned results are essentially being fed into generative AI models.” Although Illinois law permits the use of anonymized data for commercial purposes, parents contend that Google should clearly explain what measures it has taken to ensure that this data cannot be reassembled to re-identify individual children.
Parents: Ban the Use of Student Data for AI Training
Ars reached out to Google to ask for clarification on whether Creswell’s interpretation is accurate, and whether there is a substantive difference between manually appending “-AI” to an individual search query versus using a button that automatically adds “-AI” to all search queries. Google did not provide clarification; instead, it emphasized that schools retain control over whether students can access the Gemini app, Gemini Notebook, and AI-integrated search features. While schools can choose to disable these features individually, there is no option to opt out of AI-enhanced search entirely.
Emma Doyle’s lawsuit alleges that Google has leveraged its dominant position in schools—held since 2007—to push artificial intelligence onto students and harvest their data, all while failing to provide meaningful protection against the most serious risks posed by the manipulative design of its chatbots. Cresswell discovered that the Chicago Public Schools (CPS) system signed a contract with Google as early as 2020—”long before the launch of Gemini.” In the two subsequent contract renewals, even as AI technology advanced, no updated measures were introduced to safeguard student privacy.
Cresswell argues that, aside from revisions to the Google Workspace for Education privacy policy, it remains “entirely unclear” how Google’s handling of student data may have evolved.
Currently, the policy explicitly states: “We receive customer data through Core Services and process it according to the instructions of the school (the customer). This customer data includes any content submitted, stored, sent, or received by you or your school through the Core Services.” The policy further notes that Google collects data regarding these activities, as well as account information (such as names, email addresses, and passwords) and location data.
While the policy makes no mention of Gemini, it notes that data generated through the use of Core Services helps maintain the services, improve personalized experiences, and enhance security. Furthermore, it states: “Data we collect in Additional Services is used across our various services to provide, maintain, and improve our offerings; develop new services; deliver personalized experiences; measure performance; communicate with you; and protect Google, our users, and the public.”
For parents, this policy offers no reassurance that their children’s data will not be co-opted by Google to further its ambitions in the AI sector.
“We should not allow any data from children in a school environment to be collected and used for commercial purposes,” says Cresswell.
“So, what does the future hold for our thinking?”
Creswell points out that parental concerns regarding privacy extend well beyond Google. However, because Google “already occupies such a vast portion of the educational landscape and experience,” parents are focusing their efforts on questioning the use of artificial intelligence systems in schools—and doing so as early as possible.
In Chicago, Karen Zaccor—a recently re-elected member of the Chicago Public Schools Board of Education—is pushing for a moratorium on the use of AI in schools, following discussions with city and community colleges. She notes a marked decline in the reading, math, and writing proficiency of Chicago Public Schools graduates. In the case of Emma, the alleged suppression of chat logs by Google, combined with the school’s decision to withhold security logs, has left parents—who already believe that AI does more harm than good in education—feeling deeply frustrated.
During a protest against Google on Wednesday, parents brandished signs threatening to oust the company from schools, with one reading: “No Google + No AI = A Win-Win.” Many children used characters from animated series like Beavis and Butt-Head and South Park to mock what they perceive as Google’s cynical approach to introducing artificial intelligence into the classroom. One young girl, wearing pink rain boots, held up a piece of cardboard on which she had written in colorful marker: “I like reading!!”
At the height of the event, about 60 people stood outside Google’s downtown Chicago office—braving the rain—to listen to teachers and students speak out against the reckless adoption of AI.
One particularly poignant address came from Reese Mackey, a student in the Chicago Public Schools, who spoke on behalf of the student body. She challenged Google to more carefully weigh the long-term harms of aggressively pushing AI onto students, noting that—contrary to the idea that AI skills are essential for future employment—relying on such technology could actually leave students ill-prepared and falling behind. Meanwhile, as Emma Doe’s lawsuit points out—and as noted by Creswell in an interview with Ars Technica—some research has labeled the Gemini system “the most dangerous AI for the teen market,” with findings suggesting that such AI can even trigger psychosis.
“What about the future of our youth?” Mackey asked. “What about the future of our minds?”