利用经典机器学习检测LLM生成文本

本文当前为实验性机器翻译，可能包含错误。如有不清楚的地方，请参阅原始中文版本。我会持续改进翻译。

截至2026年初，主流的大语言模型(LLM)生成的文本表现出强烈的统计模式，使用传统机器学习模型可以有效地区分其与人类写作的内容。我怀疑许多所谓的“AI抄袭检测工具”实际上就是以这种方式在后台工作。

在线演示：https://lyc8503.github.io/AITextDetector/：https://lyc8503.github.io/AITextDetector/

本演示中使用的模型并未以通用数据进行训练，也未经过严格优化或迭代。它在测试集上的单句检测准确率约为85%。请在使用前阅读本文，以了解潜在的限制。

核心代码（草稿）和训练模型文件可在GitHub获取：lyc8503/AITextDetector：https://github.com/lyc8503/AITextDetector

半年前我还在学校写论文的时候，已经有关于论文AIGC（AI生成内容）检测的传闻流传开。我测试了几个平台——CNKI、万方，以及一些第三方AIGC检测服务——发现它们确实能以相当不错的准确率区分我手写的文本与LLM生成的文本。

这引发了我对AIGC检测实际工作原理（以及如何绕过它）的好奇。

但那时我同时忙于太多事情——沉迷于无线电：https://blog.lyc8503.net/categories/硬件の研究/无线射频/、Minecraft、东方Project——几次尝试失败后，我暂时搁置了这个想法。

最终，我还是糊弄完成了论文，生活继续前行。但最近，在浏览Lofter时，我偶然发现整个标签页充斥着低质量、极度脱离角色的AI生成同人小说。

我怎么一眼就能看出它们是AI生成的？嗯，有些人（或女生）甚至懒得在发布前清理Markdown格式或AI生成的章节标题——然后将文章的一半内容放在付费墙后

然而，大多数 AI 生成的文本更难被识别——它们隐藏在各种写作风格、多样的提示语中，并且不易被立即察觉。当你意识到某些地方感觉不对时，往往已经太晚了。有些文本几乎无法证明是 AI 生成的，这让我感到偏执。在吞下太多 AI 生成的糟糕内容之后，我终于受够了。Lofter 浏览到此为止——是时候打开 VS Code 了！

是的，这就是我最终重新开始周末项目的理由：构建一个 AI 生成文本检测器……

现在，在互联网上搜索 AIGC 检测几乎完全被广告污染。每一个结果只是另一个论文 AI 改写服务。那时候，我在噪音中挖掘，发现了一个叫文本困惑度的东西：https://zhuanlan.zhihu.com/p/114432097。

这个想法很简单：使用现有的大型语言模型（LLM）来估计每个单词在给定句子中出现的概率。如果几乎每个单词在 LLM 的预测中排名都很高（Top-N），那么这句话很可能是 AI 生成的。相反，如果许多单词出乎意料，更可能是人类写的。

听起来很有希望，对吧？我花了一些时间尝试这种方法，但结果令人失望——假阳性和假阴性很多，而且无法设定合理的阈值。此外，还有实际问题：高推理成本、跨模型泛化能力差、大模型难以本地部署以及封闭权重模型难以集成。总体来说，这种方法既不优雅也不可靠。

尝试失败——被一堆软广告“教程”欺骗了

既然在线资源无用，那就回归老派炼金术吧。

Scikit-learn，启动！按照它的路线图：https://scikit-learn.org/stable/machine_learning_map.html，我们可以直接选择线性 SVC 和朴素贝叶斯作为分类任务的良好起点。

（悄悄说：这也符合我的直觉——LLM 有可检测的用词模式；即使是朴素贝叶斯分类器也应该能识别它们。我只是没想到信号会这么强。）

老派炼金术的传统分类器需要标注数据——所以我们需要人类写的文本和确认过的 LLM 生成文本来训练。

我的方法：我提取了2023年我从某个类似Ford和River的平台抓取的数据，筛选了2010–2022年（ChatGPT出现前）发布的文章。我只剔除了极低互动或篇幅非常短的内容，然后随机抽取了近10,000篇多千字的人类撰写文本作为样本。

然后，我使用了一个大型语言模型（LLM）生成这些文本的章节摘要，把摘要再输入LLM，让它重新生成完整文章。这给了我大致相等数量的LLM生成样本，在体裁上多样化，并且与原有人类内容高度匹配。

理论上至少是这样。但是LLM的API很贵，我可不想在一个周末项目上花费数千美元。所以我变得有创意——并绕过规则，利用了多个低成本或免费的API渠道：

免责声明：这不是推荐。这些行为违反平台服务条款，可能会导致你被封号。但这些平台忙于营销宣传，无暇顾及，我也不打算全价支付。

许多面向编程的LLM API奇怪地按调用收费，但我们可以通过将任务批量化处理来优化利用，每次调用迫使LLM生成更多内容。于是……

你说我用了价值2000美元的3亿多Gemini代币是什么意思？！

最终，我使用gemini-3-flash生成摘要，并用七个不同的模型（gemini-3-pro、qwen-coder-plus、glm-5、glm-4.7、kimi-k2.5、doubao-seed-code、deepseek-v3.2）生成了七组LLM生成样本。

在数据生成进行到一半时，我等不及就开始训练。

我让Claude写分类器代码，但它愚蠢地把整个原始文本直接输入模型——达到了可疑的99.45%准确率……等等，真的吗？

Claude没用。我自己来。训练时，我用中文标点将所有文本拆分为句子，清理非中英文字符，然后应用scikit-learn的TF-IDF → LinearSVC。清理了一些噪声后，句子级分类仍能达到约85%的准确率！

即便这个有bug的第一个版本也达到了88%的准确率（后优化为85%）

单个句子信息有限，但每句85%的准确率意味着对于较长的文章，我们可以非常有信心判断是否由AI生成。这次表现远超我的预期。老式机器学习依然很有吸引力——远比那些只问LLM“嘿，这文本是AI生成的？”的愚蠢在线工具好多了。

完成所有数据后，我尝试训练一个8类模型（人类7个AI），但这些大型语言模型看起来太相似——很可能是彼此提炼出来的——所以分类很混乱，准确率只有~50%。

多职业结果——显然不可分割。也许我的模型不好，但无所谓，不重要

最终，我训练了七个独立的二元分类器，并采用了多数投票：如果≥2个模型检测到一句话，就会被标记为AI。

所有模型的准确率都超过85%，F1比例超过80%——相当扎实！我还注意到，AI生成的文本经常被多个模型标记，所以投票非常合理。

我试过MultinomialNB和SGDClassifier，但准确度略有下降。BERT提供了小幅提升，但对GPU使用时间过多——最终被淘汰。甚至测试了AutoGluon，结果竟然只有53%的二进制准确率。我不会深入讲那些。

到这个地步，我本可以直接发布仓库，然后就此打住。但每次都启动Python实在太不方便了。我本可以托管一个Python API，但那意味着服务器维护——这违背了我严格的无服务器理念。

我最初的计划是：导出模型到 ONNX，然后通过 WASM 的 ONNX Web Runtime 进行推理。但当我请我的硅片仆人Claude帮忙时，我没有明确说明——结果它偏离了剧本，裁剪并导出模型成了JSON......然后完全用JavaScript实现了TF-IDF SVM，用于浏览器推理。

嗯......其实这是个不错的主意。我在一封一百万字符的文本上测试过——在我的机器上大约花了10秒，可以接受。对于典型的几千字符输入，是即时的。

好吧，既然这只是演示，JavaScript的方法更透明，我就保留这个有点傻的实现。（怪克洛德，不是我。）

关于准确性：我测试了不同的特征限制。最终优先考虑性能，保留了50万特征。存储为JSON，占用107MB（虽然服务器端gzip压缩后约为38MB）。较小版本（5万–8万特征）仅损失3–4%的准确率，但最终AI检测率差异显著——尤其是在人类文本上，相对差异 ±50%，导致误报。因此我坚持使用50万特征。

最终准确率下降：约1%，如下所示：

以下所有测试均使用精简的网页版本，其性能应与完整的joblib模型相似。

当前逻辑：将输入文本拆分为句子，清理后使用7个二元模型进行分类。如果≥2个模型标记某句子，则标记为疑似AI并高亮显示。最终AI得分为被标记字符的比例。分类：

首先，测试常见模型的检测率，如Doubao和Deepseek——两者都在训练数据中。提示：写一篇3000字的故事。很容易被捕捉：

现在测试未见过的模型——泛化能力如何？

我测试了几种未在训练中出现的其他模型（MiMo-V2、Doubao-Seed-2.0、GPT-4o）——检测率均约为70%，有些甚至超过90%。表现稳健。

还测试了更复杂的提示——例如输入20章人类写作的文本，让LLM模仿风格并续写。检测率略降至67.8%（但记住，我们也在复杂提示上进行了训练）。由于篇幅限制，结果未展示。

然后我从我的订阅列表中挑选了10部已完结的网页小说（2022年前）——类型、作者、年代多样，很可能未在训练数据中。

它们的AI检测率分别为：22.7%、24.2%、25.0%、24.5%、19.0%、13.7%、29.1%、4.9%、27.3%、19.2%——均低于30%。我还随机抽样了Lofter的同人文；由于性质更随意，检测率通常低于10%。但当我输入怀疑为AI生成的文本时，检测率飙升至83.4%，强烈表明未披露使用了LLM。

[2026年3月5日更新] 为了更严格的测试，我随机抽取了10,000篇高互动（浏览量>5000）、长篇（字数>2000）的Lofter同人文，均发表于2022年前。它们的AI检测率分布（使用7模型投票，≥2票）：

以60%作为阈值 → 假阳性率：0.04% 以70% → 假阳性率：（几乎为零）上面四篇超过60%的文章都是合集索引，而非实际故事——因链接过多而被标记。

即使在50%的阈值下，假阳性率也仅为0.33%。

然后，我抓取了Lofter安卓端排名前20的热门标签（周榜）下的所有文章，按长度筛选后进行检测：

32.22%的文章得分 >50% AI——可能部分或完全由AI生成……难道已经没有人类了吗？？更重要的是，没有一篇主动声明是AI生成的内容。

“法之衰落时代，，，”——群里的一个朋友

好了，我们已经建立了一个AIGC检测器。现在是时候建立一个反检测器了。

但让我们测试一些常见的反AIGC检测技巧：

谷歌翻译往返（CN→EN→CN）：89.9% → 85.0% 有道翻译往返（CN→EN→CN）：89.9% → 79.2% 搜狗翻译往返（CN→EN→CN）：89.9% → 86.0%

This article is currently an experimental machine translation and may contain errors. If anything is unclear, please refer to the original Chinese version. I am continuously working to improve the translation.

As of early 2026, mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human-written content using traditional machine learning models. I suspect this is how many so-called “AI plagiarism checkers” actually work under the hood.

Online Demo: https://lyc8503.github.io/AITextDetector/：https://lyc8503.github.io/AITextDetector/

The model used in this demo is not trained on general-purpose data , nor has it undergone rigorous optimization or iteration. Its single-sentence detection accuracy is approximately 85% on the test set. Please read through this article before use to understand potential limitations.

The core code (drafts) and trained model files are available on GitHub: lyc8503/AITextDetector：https://github.com/lyc8503/AITextDetector

Back when I was still writing my thesis at school half a year ago, rumors were already spreading about checking papers for AIGC (AI-generated content). I tested several platforms—CNKI, Wanfang, and some third-party AIGC detection services—and found they could indeed distinguish between my hand-written text and LLM-generated text with decent accuracy.

That sparked my curiosity about how AIGC detection actually works (and how to bypass it) .

But I was juggling too many things at the time—obsessed with radio：https://blog.lyc8503.net/categories/%E7%A1%AC%E4%BB%B6%E3%81%AE%E7%A0%94%E7%A9%B6/%E6%97%A0%E7%BA%BF%E5%B0%84%E9%A2%91/, Minecraft, Touhou—and after a few failed attempts, I shelved the idea.

Eventually, I faked my way through the thesis, and life moved on. But recently, while browsing Lofter, I stumbled upon entire tags flooded with low-quality, wildly out-of-character AI-generated fanfics.

How can I tell they’re AI at a glance? Well, some folks (or gals) don’t even bother cleaning up Markdown formatting or AI-generated section headers before posting—and then they slap half the article behind a paywall

Most AI-generated texts, however, are harder to spot—they’re buried among diverse writing styles, varied prompts, and not immediately obvious. By the time you realize something feels off, it’s too late. Some texts are borderline impossible to prove as AI, leaving me paranoid. After swallowing a few too many AI-generated turds, I’d finally had enough. Lofter browsing stops here—time to open VS Code!

Yes, that’s how I ended up reviving my weekend project idea: building an AI-generated text detector…

The internet is now almost entirely polluted with ads when searching for AIGC detection. Every result is just another essay-AI-rewriting service. Back then, I dug through the noise and found something called text perplexity：https://zhuanlan.zhihu.com/p/114432097.

The idea is simple: use an existing LLM to estimate the probability of each word appearing in a given sentence. If nearly every word ranks high in the LLM’s predictions (Top-N), the sentence is likely AI-generated. Conversely, if many words are unexpected, it’s more likely human-written.

Sounds promising, right? I spent some time trying this method, but results were disappointing—plenty of false positives and false negatives, and no reasonable threshold could be set. Plus, there are practical issues: high inference cost, poor cross-model generalization, difficulty deploying large models locally, and closed-weight models being hard to integrate. Overall, this approach isn’t elegant or reliable.

Failed attempt—got tricked by a bunch of soft-ad "tutorials"

Since online resources were useless, back to old-school alchemy.

Scikit-learn, activate! Following its Roadmap：https://scikit-learn.org/stable/machine_learning_map.html, we can directly pick Linear SVC and Naive Bayes as good starting points for our classification task.

(Whisper: this also matched my gut feeling—LLMs have detectable word-choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong.)

Old-school alchemy traditional classifiers need labeled data—so we need human-written texts and confirmed LLM-generated texts for training.

My approach: I pulled data I’d scraped in 2023 from a certain Ford-like and River-like platform, filtering for articles published between 2010–2022 (pre-ChatGPT). I only filtered out extremely low-engagement or very short pieces, then randomly sampled nearly 10,000 multi-thousand-character texts as human-written samples.

Then, I used an LLM to generate chapter summaries of these texts, fed the summaries back into the LLM, and had it regenerate full articles. This gave me a roughly equal number of LLM-generated samples, diverse in genre and closely matching the original human content.

In theory, at least. But LLM APIs are expensive, and I wasn’t about to spend thousands on a weekend project. So I got creative—and skirted the rules leveraged multiple low-cost or free API channels:

Disclaimer: This is not a recommendation. These actions violate platform ToS and may get you banned. But the platforms are too busy with marketing hype to care, and I wasn’t about to pay full price.

Many programming-focused LLM APIs strangely charge per call, but we can abuse optimize this by batching tasks into massive inputs, forcing the LLM to generate more content per call. And so…

What do you mean I used over 300M Gemini tokens worth $2000 at full price?!

Ultimately, I used gemini-3-flash to generate summaries, and seven different models ( gemini-3-pro , qwen-coder-plus , glm-5 , glm-4.7 , kimi-k2.5 , doubao-seed-code , deepseek-v3.2 ) to generate seven sets of LLM-generated samples.

While I was halfway through data generation, I couldn’t wait and started training.

I asked Claude to write the classifier code, and it naively dumped the entire raw text into the model—achieving a suspicious 99.45% accuracy… Wait, really?

Claude’s useless. I’ll do it myself. For training, I split all texts into sentences using Chinese punctuation, cleaned non-Chinese/English characters, then applied scikit-learn’s TF-IDF → LinearSVC . After cleaning up some noise, sentence-level classification still hit ~85% accuracy!

Even this buggy first version hit 88% accuracy (later optimized to 85%)

Individual sentences carry limited info, but 85% accuracy per sentence means that for a longer article, we can be highly confident in judging whether it’s AI-generated. This performance far exceeded my expectations. Old-school ML still slaps—way better than those dumb online tools that just ask an LLM, “Hey, is this text AI-generated?”

After finishing all data, I tried training an 8-class model (human + 7 AIs), but the LLMs seem too similar—probably distilled from each other—so classification was messy, with only ~50% accuracy.

Multi-class results—apparently not separable. Maybe my model sucks, but whatever, not important

Eventually, I trained seven separate binary classifiers and used majority voting: a sentence is flagged as AI if ≥2 models detect it.

All models achieved over 85% accuracy and over 80% F1—pretty solid! I also noticed that AI-generated texts were often flagged by multiple models, so voting made perfect sense.

I tried MultinomialNB and SGDClassifier , but accuracy dropped slightly. BERT gave a minor boost but required too much GPU time—discarded. Even tested AutoGluon , which somehow managed only 53% binary accuracy. Won’t dive into those.

At this point, I could’ve just published the repo and called it a day. But launching Python every time is way too inconvenient. I could’ve hosted a Python API, but that means server maintenance—violates my strict Serverless philosophy.

My original plan: export model to ONNX, run inference via ONNX Web Runtime in Wasm. But when I asked my silicon servant Claude to help, I didn’t specify clearly—and it went off-script, trimming and exporting the model as a JSON… then implemented TF-IDF + SVM entirely in JavaScript for browser inference.

Hmm… actually not a bad idea. I tested it on a 1-million-character text—it took about 10 seconds on my machine, acceptable. For typical few-thousand-character inputs, it’s instant.

Fine, since this is just a demo, and the JS approach is more transparent, I’ll keep this slightly silly implementation. (Blame Claude, not me.)

As for accuracy: I tested different feature limits. Ultimately prioritized performance and kept 500k features. Stored as JSON, it’s a bloated 107MB (though gzipped server-side, it’s ~38MB). Smaller versions (50k–80k) only lost 3–4% accuracy, but final AI detection rates varied significantly—especially on human texts, with ±50% relative differences, leading to false positives. So I stuck with 500k.

Final accuracy drop: ~1%, as shown below:

All tests below use the pruned web version, which should perform similarly to the full joblib models.

Current logic: split input text into sentences, clean and classify using all 7 binary models. If ≥2 models flag a sentence, it’s marked as suspected AI and highlighted. Final AI score is the proportion of flagged characters. Classification:

First, test detection rate on common models like Doubao and Deepseek—both were in training data. Prompt: write me a 3000-word story . Easily caught:

Now test on unseen models—how’s generalization?

I tested several other models not in training (MiMo-V2, Doubao-Seed-2.0, GPT-4o)—all detected at ~70%, some even >90%. Solid.

Also tested more complex prompts—e.g., feeding 20 chapters of human-written text and asking the LLM to mimic style and continue. Detection rate dipped slightly to 67.8% (but remember, we trained on complex prompts too). Results not shown due to space.

Then I picked 10 completed web novels (pre-2022) from my subscription list—diverse genres, authors, eras, and likely not in training data.

Their AI detection rates: 22.7%, 24.2%, 25.0%, 24.5%, 19.0%, 13.7%, 29.1%, 4.9%, 27.3%, 19.2%—all under 30%. I also sampled random Lofter fanfics; since they’re more casual, detection rates were often below 10%. But when I fed in texts I suspected were AI-generated, detection spiked to 83.4%, strongly suggesting LLM use without disclosure.

[Mar 5, 2026 Update] For more rigorous testing, I randomly sampled 10,000 high-engagement (views >5000), long-form (word count >2000) fanfics from Lofter, all posted before 2022. Their AI detection rate distribution (using 7-model voting, ≥2 votes):

Using 60% as threshold → false positive rate: 0.04% Using 70% → false positive rate: (effectively zero) The four texts above 60% were all collection indexes, not actual stories—flagged due to excessive links.

Even at 50% threshold, false positive rate is only 0.33% .

Then, I scraped all articles from Lofter Android’s top 20 trending tags (weekly榜单), filtered by length, and ran detection:

32.22% of articles scored >50% AI —likely partially or fully AI-generated… Is there any human left?? Moreover, not a single one has proactively declared AI-generated content.

“Age of Dharma’s decline, , ,” —a friend in the group

Alright, we’ve built an AIGC detector. Time to build an anti-detector now.

But let’s test some common anti-AIGC detection tricks:

Google Translate roundtrip (CN→EN→CN): 89.9% → 85.0% Youdao Translate (CN→EN→CN): 89.9% → 79.2% Sogou Translate (CN→EN→CN): 89.9% → 86.0%