How Do AI Detectors Work?

If you’ve ever come across an AI written story online, you’ve probably experienced that feeling of reading something that looks coherent and well structured, but still has something off about it. Today it’s all too common to stumble across AI writing, and much of it isn’t even edited thoroughly, which is why it’s important to understand the characteristics of AI writing and how to detect it.

What Are AI Detectors?

AI detectors are tools designed to identify if a piece of content was written by an actual person or by an AI bot. With the rise of AI text generators in recent years, many people have grown concerned about transparency and the ethics of using AI for professional and academic purposes. 

To use an AI detector, you simply paste in the text and hit submit. Oftentimes there will be a character limit or a usage limit for the number of detections that are possible within a 24 hour period.

How Do They Work?

Most AI detectors compare patterns of human language and patterns seen in AI written text, so when you feed a piece of text into a detector, the first thing that happens is an examination of how predictable or random the text is. If the next word is almost always a predictable choice, the text has what’s referred to as low perplexity and is therefore more likely to be AI generated. This involves using common words, phrases that fit, or a bland text flow. By contrast, text written by a person is more unpredictable and might have grammatical errors or unusual word choices..

Another thing detectors look at is the variation in sentence structure and length, often referred to as burstiness. People tend to write with a mix of short and long sentences, creating a natural rhythm but if not specifically instructed to vary its output, AI can produce more uniform sentences that all have similar length or structure so by measuring burstiness, a detector sees if the text has a healthy mix of sentence lengths and styles.

Developers have also trained AI classifier models on large datasets of writing, labeling a ton of samples as either human written or AI written. These models learn the subtle differences in word choice, phrasing, pacing, and even punctuation that tend to differentiate AI text so when you input new text, the classifier compares the text to everything it has seen before and gives a probability or score for how likely it is to belong in either category.

There are also specific tell tale markers that serve like a watermark for AI generated content.  One idea is to have the AI subtly prefer certain synonyms or sentence structures in a pattern that wouldn't be noticeable to a reader but statistically would stand out if you know to look for it. As of now, most AI generated text doesn't have these watermarks, but if they become common, detectors could simply scan for those hidden patterns. 

Keep in mind that AI detectors often combine multiple signals instead of relying on a single one, since each of these methods has strengths and weaknesses. 

A few of the most popular AI detectors include GPTZero, OpenAI’s Text Classifier, and Turnitin. GPTZero was one of the first widely known free AI detectors, created by a Princeton student. It looks at perplexity and burstiness, so if the text is too orderly or predictable, GPTZero leans towards calling it AI generated. 

OpenAI released its own detector which was trained on pairs of human and AI text but it was known for an abundance of false flags and was eventually taken down. 

Turnitin, known for plagiarism checking in schools, introduced an AI writing detector built into its system but users have reported mixed results. 

Each of these tools uses a slightly different mix of techniques, but none of them can guarantee accuracy so it’s always best to tread with caution.

Limitations and Challenges

In practice AI detection comes with a lot of caveats, the biggest being false positives. If a student writes in a very formal or impersonal style, or if an author naturally writes in a way that coincidentally resembles the patterns of AI text, the detector might get it wrong, errors that errors can have serious consequences if people take the detector's word at face value.

On the flip side, AI detectors can also miss AI written text that has been cleverly edited or produced by advanced models, so if you take an AI generated paragraph and shuffle a few words around, or ask the AI to write in a more chaotic  style, you can often lower its predictability enough to dodge some detectors. 

Some detection algorithms might also inadvertently be biased, for example detectors sometimes flag writing by non-native English speakers more often because they're still learning the language. There’s also a lack of context, since detectors look at the text in isolation and can't tell why a certain style is the way it is. 

Additionally, language and AI models change so fast so a detection method that works today might not work next year and because detectors have to be updated with new training an old detector might start failing if it isn't retrained to recognize the fingerprints of the latest AI techniques. Using a multi-model AI option, like Vear.com, can help you compare AI answers side by side and pick the best parts to avoid being detected.

AI detectors should be used as helpful tools, not final judges which is why an actual person needs to review and decide what to do with that information.

We've seen that these detectors are far from all knowing, and they can be misled by unusual human writing or by advanced AI that learns to sound more human. My exploration into this topic left me with an appreciation for how nuanced writing is. It drives home a key point: when it comes to judging a piece of writing, there's still no substitute for human judgment. The detectors can assist and highlight, but we humans remain the ultimate detectors of meaning, context, and intent in what we read and write.