AI Text Detectors: Can We Really Trust Them?

According to a study by Diplomeo, “78% of 16-25 year olds use AI for their studies and guidance, with 25% using it weekly.” This trend is particularly concerning for educators who fear that assignments submitted may not have been written by the students themselves, but by AI. Of the 78% of young people utilizing AI, “35% admit to writing all or part of their homework using sites like ChatGPT.” Therefore, artificial intelligence detectors play a crucial role in this hunt for cheating. But are they reliable, and can they really be depended upon?

How Does an AI Detector Work?

AI checkers aren’t magical; they don’t provide an exact percentage of text generated by artificial intelligence. These tools operate on probabilities. Thus, the percentage given by the detector is merely an estimate of the likelihood that the text was generated by AI. But what does the tool base its claims on that AI was used or not?

Most often, the detectors rely on language models like ChatGPT, Gemini, or Claude themselves. This helps them more easily check if the entered text could have been created by AI. Models like ChatGPT attempt to predict the words that are most likely to follow in the text. If the detector notices similarities between the text written by the AI and the one entered into the tool, there’s a higher chance that it was authored by artificial intelligence.

Finally, grammar or spelling errors are an important aspect checked by the detector. Errors are possible, but they are mostly human. The more errors your text contains, the more likely AI will think it’s written by a human.

Can We Really Trust AI Detectors?

The answer is clear: no. Although these detectors are improving daily, it is still very difficult to trust them completely. Texts copied from ChatGPT are relatively easy to detect, but many contents written by humans are still mistakenly identified as AI-generated.

These AI errors are called “false positives,” and they mainly occur when a text follows very structured formats, such as introductions or conclusions. These sections often reuse recurrent phrases like “as we have seen” or “in conclusion, we can state that,” which wrongly makes them suspected of being produced by artificial intelligence. News articles or list-form articles are also regularly recognized as false positives.

Contents with little complexity and logical sequence such as: “Our parents wait outside the gym for us to finish our sports session” could be analyzed as AI-generated. This sentence is logical and the words are easily predictable, which resembles the workings of language models. Conversely, a text like: “Our parents are waiting outside the gym to pick up bread and catch a flight” is less predictable. Therefore, it is quite unlikely that an AI could generate such a sentence.

By understanding how AI and detection tools work, it becomes relatively easy to circumvent these systems. This raises a significant issue, especially in the field of education, where there is uncertainty about the real origin of the assignments submitted: were they written by the students themselves or generated by AI? Punishing a work suspected to have been produced by a language model then becomes tricky, especially in the absence of concrete proof. And it’s hard to imagine the academic world encouraging spelling errors…