将大语言模型（LLM）作为分类器，即向上下文应用提示并返回标签，使用起来很糟糕。尤其令人痛苦的是，它们的性能往往相当不错。
但让我们考虑一下我们希望分类器具备的一些特性，以及将LLM作为分类器在这些方面表现如何：
LLM 内置了一些先验信息，这些信息可能与我们的分布不符。例如，LLM 不会知道我们是在测试一个正类稀少的人群，还是在测试一个正类相对普遍的选择人群。我想你可以给它提供这些上下文，但现在你必须为每一个新的人群修改它，而且，正如我们的第一个观点所述，还不清楚这是否会被适当地纳入LLM的判断。
这些失败并不是LLM的错：它并不是作为分类器设计的，实际上也没有合理完成这类任务的机制。但这只是因为我们在思考问题时方法不对……
通过适当的框架来利用LLM的能力，我们可以获得LLM的强大功能，同时享有标准机器学习算法的便利。想了解可能性的话，可以考虑用一个简单的逻辑回归来包装LLM的判决：
\[ p(y = 1 \mid x) = \sigma(\alpha + \beta \cdot LLM(x)) \]
注意，在\(\beta \rightarrow \infty\)的特殊情况下，这基本上就是恢复了我们的LLM分类器！！但那是一个愚蠢的参数选择策略。我们应该采用通常的方法，使用一些训练数据来估计参数。这样就会归结为两种情况，我们只需获得经验估计。
\[ p（y = k \mid LLM（x） = 1） = \frac{\sum_{i} I（y_{i} = k \text{ 和 } LLM（x_{i}） = 1）}{\sum_{i} I（LLM（x_{i}） = 1）} \]
现在让我们重新审视我们的期望：
我们基本上已经恢复了我们模型所希望的所有良好特性！我们还能进一步吗？
假设我们对分类器的性能不满意：我们应该怎么办？在LLM作为分类器的情况下，我们唯一的选择是尝试改动提示。这是一项神秘的工作，互联网上有很多建议，但真正的智慧却很少。祝你好运。
从机器学习的角度来看，让你的模型变得更好是：
我们用一个例子让情况更具体。我们将使用SemEval 2018任务3数据集：https：//github.com/Cyvhee/SemEval2018-Task3 2：#fn：2，这是一组由专家标注者标记为讽刺的4618条推文（3834条train / 784条测试）。讽刺是这篇文章的自然契合，这是一个NLP任务，LLM显然有真实信号，我们从LLM中隐含的世俗知识中受益。
我们的提示要求模型做出二元讽刺判断，我们会一次性对所有推文进行批量处理：
仅凭这个提示，我们就得到了以下的表演
这款游戏作为一次性游戏的表现其实相当惊人。你不会指望没有学习就能做到这一点，这正是大型语言模型的好处。当然，整体还是很一般：Brier分数很差，因为我们没有校准（实际上，随便猜测的Brier分数只有0.25）。
我们可以用逻辑回归做得更好，因为它能实现校准（不过注意，这不会影响排序，所以F1是一样的）。
让我们来看看一些额外的大型语言模型特性。首先，让我们快速看看我们的错误分类（无论哪个模型都是一样的）
鉴于此，我们修改一下提示如下
我们还会开始添加一些确定性特征，这些特征我们可以计算：
那么我们看到改进了吗？我们比较了三个嵌套模型：仅判决模型、判决+所有LLM特征、判决+所有特征（LLM+基于规则）。
我们看到每个附加功能层面的明显益处，包括那些位于LLM之外的确定性特征。
系数图显示模型实际依赖的特征，并控制其他所有特征：
图1：逻辑回归系数（± 1 SE），按|系数|排序。
我们的方法与该数据集的已发表文献相比如何？
我们看到初始LLM分类器远远领先竞赛冠军（0.747对0.705）。从特征工程角度看，我们有重叠的CI与竞赛后最先进的技术，仅基于LLM提取特征进行逻辑回归。
让大型语言模型（LLM）能够可靠地作为分类工具是一项艰巨的工作，但潜力巨大。越来越多的研究依赖LLM进行分类：比如《How People Use ChatGPT》：https：//minimallysufficient.com/ephemera/how-people-use-chatgpt/，利用LLM对与LLM的对话进行分类，5：#fn：5，或者Huggingface事件中的“slop-vestigation：https：//x.com/RyanGreenblatt/status/2092692685224325542？s=20”。我们需要从这些工具中获得高质量的结果。
幸运的是，越来越多的论文正在提出这一观点。
我个人对研究代理分类器很感兴趣。你不是固定的特征集或类陈述，而是赋予LLM自我调查的能力。LLM可以在分类时利用调查过程的特征作为特征：本质上是根据调查的严谨性和全面性来给自己打分。有了可靠的测试集，我们就能对结果做出统计上有效的推断！
这在多模态模型中甚至不使用图片时也能看到：https：//arxiv.org/abs/2603.21687。︎ ↩：#fnref：1
Van Hee， C.， Lefever， E.， 和 Hoste， V.（2018）。SsemiEval-2018 任务3：英语推文中的讽刺检测。载于第12届国际语义评估研讨会论文集（第39–50页）。计算语言学协会。https://aclanthology.org/S18-1005/:https://aclanthology.org/S18-1005/ ↩︎：#fnref：2
Wu 等（2018）。THU/_NGN 于2018年半期评估任务3：通过密集连接LSTM和多任务学习进行推文讽刺检测。2018年学期评估论文集。︎ ↩：#fnref：3
Baziotis 等人（2018）。NTUA-SLP 在 SemEval-2018 任务 3：利用词语和字符级别注意力 RNN 集合追踪讽刺推文。SemEval 2018 会议论文集。︎ ↩：#fnref：4
有反馈吗？我很想听听你的声音——邮箱：#或保持匿名：https：//www.admonymous.co/minimallysufficient。
LLMs-as-classifiers, prompts applied to a context and returning a label, suck to work with. This is especially painful because they often perform pretty decently.
But let’s consider some of the things we’d want in a classifier and see how an LLM-as-classifier stacks up:
The LLM has some prior information baked in which might be a poor fit for our distribution. For instance the LLM won’t know whether we’re testing on a population where our positive class is rare or an enriched population where our positive class is relatively prevalent. And I guess you can give it that context but now you’ve got to modify that for each new population and also, as in our first point, it’s not clear that this will be appropriately incorporated into the LLM’s judgement.
These failures are not the fault of the LLM: it’s not designed as a classifier and indeed has no mechanism for plausibly doing some of these things. But only because we’re thinking of things incorrectly…
With the proper framework that harnesses the LLM’s power we can get the power of the LLM with the convenience of stock ML algorithms. For a taste of what’s possible consider wrapping the LLM verdict with a simple logistic regression:
\[ p(y = 1 \mid x) = \sigma(\alpha + \beta \cdot LLM(x)) \]
Note that in the special case of \(\beta \rightarrow \infty\) this basically recovers our LLM classifier!! But that’s a dumb parameter selection policy. We should instead do our usual approach of estimating our parameters using some training data. This will then collapse into two cases and we just get the empirical estimates.
\[ p(y = k \mid LLM(x) = 1) = \frac{\sum_{i} I(y_{i} = k \text{ and } LLM(x_{i}) = 1)}{\sum_{i} I(LLM(x_{i}) = 1)} \]
Now let’s revisit our desiderata:
We’ve basically recovered all of the nice properties we wanted from our model! Can we go even further?
Suppose we are not pleased with the performance of our classifier: what should we do? In the LLM-as-classifier case our only option is to try messing with the prompt. This is an arcane undertaking about which advice abounds on the internet but wisdom is scarce. Best of luck to you.
From a ML point of view the way you make your model better is:
Let’s make this more concrete using an example. We’ll use the SemEval 2018 Task 3 dataset：https://github.com/Cyvhee/SemEval2018-Task3 2：#fn:2 , a collection of 4618 tweets (3834 train / 784 test) labeled for irony by expert annotators. Irony is a natural fit for this post it’s an NLP task where an LLM clearly has real signal and we benefit from the worldly knowledge implicitly embedded in the LLM.
Our prompt asks the model to make a binary irony judgment, and we run it over all the tweets at once as a batch job:
We get the following performance just from this prompt
It’s actually quite remarkable how well this does as one-shot. You wouldn’t expect this to be possible without learning which is the cool thing about LLMs. Of course it’s still pretty meh: the Brier score is quite bad as we don’t have calibration (indeed just random guessing gets us a Brier score of 0.25).
We can do better with our logistic regression which achieves calibration (though note it doesn’t affect the ordering so F1 is the same).
Let’s consider some additional LLM features. Firstly let’s take a quick look at our misclassifications (they’re the same from either model)
In light of this let’s modify our prompt as follows
We’ll also start adding in some deterministic features that we can compute:
So do we see improvements? We compare three nested models: verdict only, verdict + all LLM features, verdict + all features (LLM + rule-based).
We see a clear benefit from each level of additional features including the deterministic features which lie outside of the LLM.
The coefficient figure shows which features the model actually relies on, controlling for all others:
Figure 1: Logistic regression coefficients (± 1 SE), sorted by |coefficient|.
How does our approach compare to the published literature on this dataset?
We see that our initial LLM classifier beats the competition winner handily (0.747 vs 0.705). With the feature engineering perspective we have overlapping CIs with the post-competition state of the art, using nothing but a logistic regression on top of LLM-extracted features.
Getting LLMs into shape to reliably serve as classifiers is hard work but potentially highly impactful. There’s more and more research that relies on LLMs for classification: like the How People Use ChatGPT：https://minimallysufficient.com/ephemera/how-people-use-chatgpt/ which uses LLMs to classify conversations with LLMs 5：#fn:5 or the “slop-vestigation：https://x.com/RyanGreenblatt/status/2092692685224325542?s=20” of the Huggingface incident. We’re going to need to get high quality results out of these tools.
Fortunately, there’s a growing body of papers which are making this point.
Personally I am interested in investigating agentic classifiers. Instead of a fixed feature set or class statement you empower the LLM to investigate itself. The LLM can use features of the investigative process as features when classifying: essentially grading itself on the rigor and comprehensiveness of the investigation. And with a reliable test set we can make statistically valid inferences on the results!
This shows up with multimodal models not even using the images：https://arxiv.org/abs/2603.21687. ↩︎：#fnref:1
Van Hee, C., Lefever, E., & Hoste, V. (2018). SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation (pp. 39–50). Association for Computational Linguistics. https://aclanthology.org/S18-1005/：https://aclanthology.org/S18-1005/ ↩︎：#fnref:2
Wu et al. (2018). THU\_NGN at SemEval-2018 Task 3: Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning. Proceedings of SemEval 2018 . ↩︎：#fnref:3
Baziotis et al. (2018). NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs. Proceedings of SemEval 2018 . ↩︎：#fnref:4
Have feedback? I'd love to hear from you — email：# or keep it anonymous：https://www.admonymous.co/minimallysufficient.
扫码直达文章详情