LLMs-as-classifiers, prompts applied to a context and returning a label, suck to work with. This is especially painful because they often perform pretty decently.

But let’s consider some of the things we’d want in a classifier and see how an LLM-as-classifier stacks up:

The LLM has some prior information baked in which might be a poor fit for our distribution. For instance the LLM won’t know whether we’re testing on a population where our positive class is rare or an enriched population where our positive class is relatively prevalent. And I guess you can give it that context but now you’ve got to modify that for each new population and also, as in our first point, it’s not clear that this will be appropriately incorporated into the LLM’s judgement.

These failures are not the fault of the LLM: it’s not designed as a classifier and indeed has no mechanism for plausibly doing some of these things. But only because we’re thinking of things incorrectly…

With the proper framework that harnesses the LLM’s power we can get the power of the LLM with the convenience of stock ML algorithms. For a taste of what’s possible consider wrapping the LLM verdict with a simple logistic regression:

\[ p(y = 1 \mid x) = \sigma(\alpha + \beta \cdot LLM(x)) \]

Note that in the special case of \(\beta \rightarrow \infty\) this basically recovers our LLM classifier!! But that’s a dumb parameter selection policy. We should instead do our usual approach of estimating our parameters using some training data. This will then collapse into two cases and we just get the empirical estimates.

\[ p(y = k \mid LLM(x) = 1) = \frac{\sum_{i} I(y_{i} = k \text{ and } LLM(x_{i}) = 1)}{\sum_{i} I(LLM(x_{i}) = 1)} \]

Now let’s revisit our desiderata:

We’ve basically recovered all of the nice properties we wanted from our model! Can we go even further?

Suppose we are not pleased with the performance of our classifier: what should we do? In the LLM-as-classifier case our only option is to try messing with the prompt. This is an arcane undertaking about which advice abounds on the internet but wisdom is scarce. Best of luck to you.

From a ML point of view the way you make your model better is:

Let’s make this more concrete using an example. We’ll use the SemEval 2018 Task 3 dataset：https://github.com/Cyvhee/SemEval2018-Task3 2：#fn:2 , a collection of 4618 tweets (3834 train / 784 test) labeled for irony by expert annotators. Irony is a natural fit for this post it’s an NLP task where an LLM clearly has real signal and we benefit from the worldly knowledge implicitly embedded in the LLM.

Our prompt asks the model to make a binary irony judgment, and we run it over all the tweets at once as a batch job:

We get the following performance just from this prompt

It’s actually quite remarkable how well this does as one-shot. You wouldn’t expect this to be possible without learning which is the cool thing about LLMs. Of course it’s still pretty meh: the Brier score is quite bad as we don’t have calibration (indeed just random guessing gets us a Brier score of 0.25).

We can do better with our logistic regression which achieves calibration (though note it doesn’t affect the ordering so F1 is the same).

Let’s consider some additional LLM features. Firstly let’s take a quick look at our misclassifications (they’re the same from either model)

In light of this let’s modify our prompt as follows

We’ll also start adding in some deterministic features that we can compute:

So do we see improvements? We compare three nested models: verdict only, verdict + all LLM features, verdict + all features (LLM + rule-based).

We see a clear benefit from each level of additional features including the deterministic features which lie outside of the LLM.

The coefficient figure shows which features the model actually relies on, controlling for all others:

Figure 1: Logistic regression coefficients (± 1 SE), sorted by |coefficient|.

How does our approach compare to the published literature on this dataset?

We see that our initial LLM classifier beats the competition winner handily (0.747 vs 0.705). With the feature engineering perspective we have overlapping CIs with the post-competition state of the art, using nothing but a logistic regression on top of LLM-extracted features.

Getting LLMs into shape to reliably serve as classifiers is hard work but potentially highly impactful. There’s more and more research that relies on LLMs for classification: like the How People Use ChatGPT：https://minimallysufficient.com/ephemera/how-people-use-chatgpt/ which uses LLMs to classify conversations with LLMs 5：#fn:5 or the “slop-vestigation：https://x.com/RyanGreenblatt/status/2092692685224325542?s=20” of the Huggingface incident. We’re going to need to get high quality results out of these tools.

Fortunately, there’s a growing body of papers which are making this point.

Personally I am interested in investigating agentic classifiers. Instead of a fixed feature set or class statement you empower the LLM to investigate itself. The LLM can use features of the investigative process as features when classifying: essentially grading itself on the rigor and comprehensiveness of the investigation. And with a reliable test set we can make statistically valid inferences on the results!

This shows up with multimodal models not even using the images：https://arxiv.org/abs/2603.21687. ↩︎：#fnref:1

Van Hee, C., Lefever, E., & Hoste, V. (2018). SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation (pp. 39–50). Association for Computational Linguistics. https://aclanthology.org/S18-1005/：https://aclanthology.org/S18-1005/ ↩︎：#fnref:2

Wu et al. (2018). THU\_NGN at SemEval-2018 Task 3: Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning. Proceedings of SemEval 2018 . ↩︎：#fnref:3

Baziotis et al. (2018). NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs. Proceedings of SemEval 2018 . ↩︎：#fnref:4

Have feedback? I'd love to hear from you — email：# or keep it anonymous：https://www.admonymous.co/minimallysufficient.