TutorMoments： Do AI tutors know when to help and when to hold back？

一个好导师需要具备什么？：#what-makes-a-good-tutor TutorMoments 的工作原理：#how-tutormoments-works 初步结果：#preliminary-results 限制与后续步骤：#limitations-and-next-steps 📄 技术报告: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf：https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | 📊 数据: https://huggingface.co/datasets/allenai/tutormoments-preview：https://huggingface.co/datasets/allenai/tutormoments-preview | 💻 代码: https://github.com/allenai/tutormoments：https://github.com/allenai/tutormoments

：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LlOloe_6sFprFVqSBBk5t.png

今天我们要介绍的是 TutorMoments 的预览版：https://tutormoments.allen.ai/，这是一个衡量前沿法学硕士是否能够平衡教育中最困难的权衡之一的框架：何时介入并帮助学生，何时阻止并让学生做更多的工作。

TutorMoments 是基于重放的评估工具，建立在真实一对一的数学辅导课程之上。经验丰富的数学教师会查看从美国辅导项目收集的课程记录，并标记导师必须在“让问题更容易以便学生入手”与“推动学生自己进行更多推理”之间做出选择的时刻。然后，TutorMoments 会截取该决策点之前的课程记录，把它交给语言模型，让模型在模拟课程中接管导师角色——由另一语言模型扮演学生——以观察大型语言模型导师会如何操作。

仅被告知“好好辅导”，我们发现模型倾向于过度帮助，提供过多支持，很少促使学生进行更深入的思考。在导师提示中明确说明这一权衡（何时帮助何时克制）能够提升表现，但仍无法达到持续匹配情境的人类辅导水平，而且不同 LLM 在如何可靠地做出这一判断上仍存在很大差异。

作为我们对开放研究承诺的一部分，我们正在发布一个已去标识的辅导对话数据集：https://huggingface.co/datasets/allenai/tutormoments-preview，以及用于运行我们回放管线的代码：https://github.com/allenai/tutormoments，同时还提供我们评估这些对话中关键时刻的模型导师回放，以便可复现性。我们希望 TutorMoments 能为教育工作者、研究人员以及构建 AI 导师的团队提供更清晰的方法，了解模型如何处理最重要的教学决策——并帮助该领域构建可以适应每个学生的导师，而不是替他们完成工作。

向一个优秀的数学导师寻求帮助时，你很可能会得到这样一个问题作为回应：“你对这个问题的要求了解多少？”这并不是不够帮助——优秀教学的一部分就是诊断学生已经掌握了什么，并在当下提供合适的支持。立刻提供支持会剥夺学生进行智力工作的机会，而这种工作正是帮助他们学习的关键。有时需要支持；而有时最有效的做法是推动学生通过解释正确答案来巩固理解。

然而，语言模型是被训练成提供帮助的，有帮助的助手往往会为你完成困难的部分——解释概念、列出步骤，并引导你找到答案。在辅导课堂上，这可能会缩短建设性挣扎的时间——即那种努力、有时令人沮丧的解决问题的过程，而学习研究长期以来将其与更强的理解力联系在一起。

大多数作为导师的语言模型的基准测试并未捕捉到这种张力。它们往往会特别奖励某种行为——例如绝不直接给出问题答案，或者总是提供提示——而不考虑这是否是针对学生实际理解情况的正确做法。但良好的辅导并不是可以在所有情况下固定识别的单一行为。这是一种判断：这个学生在当前这个问题上现在最需要的是什么？

TutorMoments 建立在真实辅导数据之上。我们发布的数据集 TutorMoments-Preview：https://huggingface.co/datasets/allenai/tutormoments-preview，包含 462 份去标识化的、仅文本的一对一数学辅导记录，针对美国二至七年级学生，由教师标注了超过 1,500 个关键时刻，以及 27 位美国教师注释者提供的数千条自由文本注释。这些记录来自一个高剂量辅导项目，项目学生大多就读于 Title I 学校，根据家长和监护人同意的研究条款共享；所有数据在提供方处理后，再经过一个附加的数学智能管道去除了身份信息。

所有注释均来自经验丰富的数学教师，我们要求他们阅读记录并标记关键学习时刻——记录发生了什么，辅导老师做了什么，以及学生的反应如何。每个关键时刻都是辅导老师必须权衡支架式教学（使问题更易理解）和推动严格性（鼓励学生进行更高层次思考）的决策点。

TutorMoments 的运作方式是在某个关键时刻暂停记录，并将会话交给语言模型，由其作为辅导老师与模拟学生进行五轮辅导。我们将每个由模型生成的延续称为重放。然后基于 LLM 的评分管道对每次重放进行三方面评分：模型是否（1）在学生需要支持时提供了支架，(2) 在学生准备好接受更大挑战时推动了严格性，以及 (3) 避免了过度支架（降低挑战超过该时刻所需）。

评分管道以教师定义的真实结果开始：对于每个关键时刻，是需要支架还是需要推动严格性。每个时刻由多位教师注释，当意见不一致时，我们取多数标签——例如，如果三位教师为某个时刻注释，其中两位选择严格性，而一位选择支架，真实结果就为严格性。一个独立的语言模型分类器会根据教师注释进行验证，然后决定辅导老师的实际行为是否与该时刻所需一致——“适当”的回合意味着辅导老师的分类行为（支架、推动严格性或过度支架）与教师判断的时刻需求相符。

我们用两个提示将七个大型语言模型（LLMs）通过 TutorMoments 进行测试：一个普通提示，没有提供实际指导——它只告诉模型利用已知的良好辅导方法来回应学生——以及一个评估意识提示，明确说明了支架、过度支架和推动严谨性之间的权衡。每个模型都根据辅导记录中的关键时刻进行评分，这些时刻平均分布在适合使用支架的方法和需要严格要求的情况下。

表格中的每个数字都是介于 0 和 1 之间的评分——表示模型在相关时刻做出适当行为的比例——因此分数越高，模型做出正确判断的次数越多。例如，适当严谨性的 0.50 意味着模型在需要严谨性的一半时刻成功地提出了严谨要求。

阅读评分时，需要注意以下几点：

人类辅导员是自然参考，而非上限。我们不把人类辅导员视为理想实践的模型——即便是有经验的辅导员在瞬间也会做出不够理想的选择。按照相同的决策点同样评分，我们记录的辅导员得分为 0.458（适当支架）、0.182（适当严谨）和 0.496（避免过度支架）——都低于模型的评估意识得分，并大致处于普通提示分数的范围内。但这并不意味着 AI 辅导优于人类教师。标注者专门寻找辅导可以做得更好的时刻，因此数据集集中于错失的机会，而非理想实践。

这些分数衡量的是辅导员行为，而不是学习效果。重播中使用的是模拟的“oracle”学生，因此数字反映的是模型在决策点的行为——而非真实学生是否学会了知识。

严谨性比支架更不稳定。评分流程对严谨性的识别不如支架可靠，并且在基础标注中，严谨时刻（260 个）比支架时刻（738 个）要少。

：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/iJjIPaPGPMCpnMrFiYxIZ.png

表格中最清晰的模式是提示的重要性：每个模型在有评估意识的提示下的得分都高于普通提示下的得分。这表明，模型默认的“有帮助的助手”行为本身不足以成为优秀的辅导者。但在提示中明确说明权衡虽然有所帮助——它提升了每个分数——模型在解释增强提示的方式上仍存在很大差异，即使是得分最高的模型也有很大提升空间。

我们还分析了辅导者在每种情境下所采取的动作。虽然提示鼓励模型追求严谨，但它们使用的策略比人类少，通常依赖于要求学生解释答案。相比之下，人类辅导者采用的策略更为多样，更有可能后退一步，让学生独立完成任务。

TutorMoments 仍处于早期开发阶段，目前存在一些局限性。最大的问题是，自动化评估可以提供模型在决策点的行为信号，但不能替代真实学生和真实学习成果的研究。数据集也比较狭窄：基于美国，主要是小学和初中数学，由单一教育者群体标注。我们的发现可能无法推广到其他学科、年级或环境。

我们分享这一预览是为了在构建更大、多模态数据集、更强评分流程和更深入分析的过程中收集反馈。

该项目部分得益于盖茨基金会和学习共享平台（Learning Commons）的支持。

What makes a good tutor? ：#what-makes-a-good-tutor How TutorMoments works ：#how-tutormoments-works Preliminary results ：#preliminary-results Limitations and next steps ：#limitations-and-next-steps 📄 Tech Report: https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf：https://tutormoments.allen.ai/static/paper/tutormoments-preview.pdf | 📊 Data: https://huggingface.co/datasets/allenai/tutormoments-preview：https://huggingface.co/datasets/allenai/tutormoments-preview | 💻 Code: https://github.com/allenai/tutormoments：https://github.com/allenai/tutormoments

：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/LlOloe_6sFprFVqSBBk5t.png

Today we're introducing a preview of TutorMoments ：https://tutormoments.allen.ai/, a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work.

TutorMoments is a replay-based evaluation built off real one-on-one math tutoring sessions. Experienced math teachers go through transcripts collected from a U.S. tutoring program and flag the moments where a tutor had to choose between making a problem easier to get started on and pushing the student to do more of the reasoning themselves. TutorMoments then takes the transcript up to that decision point, hands it to a language model, and has the model take over as the tutor in a simulated session – with the student played by another language model – to see what the LLM tutor does.

Told only to "tutor well," we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.

As part of our commitment to open research, we're releasing a dataset of de-identified tutoring transcripts：https://huggingface.co/datasets/allenai/tutormoments-preview, the code for running our replay pipeline：https://github.com/allenai/tutormoments, and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them.

Ask a good math tutor for help and you'll likely get a question back like, "What do you know about what the problem is asking?" That isn't unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what's most effective is a push to solidify understanding by explaining a correct answer.

Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding.

Most benchmarks for language models acting as tutors don't capture this tension. They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn't a single fixed behavior you can identify across the board. It's a judgment call: what does this student need, right now, on this problem?

TutorMoments is built on real tutoring data. The dataset we're releasing, TutorMoments-Preview：https://huggingface.co/datasets/allenai/tutormoments-preview , is 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7, with more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline.

All annotations came from experienced math teachers, whom we asked to read the transcripts and mark key learning moments—noting what was going on, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding (making a problem more accessible) against pushing for rigor (encouraging the student to do harder thinking).

TutorMoments runs by pausing a transcript at one of those key moments and handing the session to a language model, which takes over as the tutor for five turns with a simulated student. We call each of these model-generated continuations a replay . An LLM-based scoring pipeline then rates each replay on three things: whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready for more challenge, and (3) avoided over-scaffolding (reducing the challenge more than the moment called for).

The scoring pipeline starts from a teacher-defined ground truth: for each key moment, whether it called for scaffolding or for a push for rigor. Several teachers annotated each moment, and when they disagreed we took the majority label—if three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor. A separate LM classifier validated against teacher annotations then decides whether the tutor's actual move matches what the moment called for—an "appropriate" turn means the tutor's classified action (scaffold, push for rigor, or over-scaffold) lines up with what teachers judged the moment to call for.

We ran seven LLMs through TutorMoments using two prompts: a plain prompt that gives no real guidance – it only tells the model to use what it knows about good tutoring to respond to the student – and an evaluation-aware prompt that spells out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Each model was scored over key moments drawn from the tutoring transcripts, split evenly between moments where scaffolding was the right approach and moments that called for rigor.

Every number in the table is a rating between 0 and 1 – the share of the relevant moments where the model did the appropriate thing – so a higher score means the model made the right call more often. A 0.50 on appropriate rigor, for instance, means the model pushed for rigor in half of the moments that called for it.

A few things to keep in mind when reading the scores:

Human tutors are a naturalistic reference, not a ceiling. We don't treat human tutors as a model of ideal practice—even experienced tutors make less-than-optimal choices in the moment. Scored the same way at the same decision points, the human tutors in our transcripts get 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding)—all below the models' evaluation-aware scores and around the range of their plain-prompt scores. But this isn't a claim that AI tutors outperform human teachers. Annotators specifically looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice.

The scores measure tutor behavior, not learning. Replays use a simulated "oracle" student, so the numbers reflect how a model acts at a decision point—not whether a real student learned.

Rigor is noisier than scaffolding. The scoring pipeline detects rigor pushes less reliably, and there are fewer rigor moments (260) than scaffolding moments (738) in the underlying annotations.

：https://cdn-uploads.huggingface.co/production/uploads/638e39b249de7ae552d977b5/iJjIPaPGPMCpnMrFiYxIZ.png

The clearest pattern in the table is how much the prompt matters: every model scores higher under the evaluation-aware prompt than under the plain one. That suggests a model's default "helpful assistant" behavior isn't enough on its own to tutor well. But spelling out the trade-off in the prompt only goes so far—while it lifts every score, models still differ widely in how they interpret the enhanced prompt and even the best scorers have plenty of room to improve.

We also break down the moves that tutors made under each scenario. While prompting encourages models to push for rigor, they use fewer strategies than humans do, often relying on asking students to explain their answers. In contrast, human tutors employ more varied strategies and are much more likely to step back and let students work independently.

TutorMoments is still early in its development, and it has several limitations at this stage. The biggest is that automated evaluation gives us signal about how a model behaves at a decision point, but it can't stand in for studies with real students and real learning outcomes. The dataset is also narrow: U.S.-based, mostly elementary and middle-school math, annotated by a single pool of educators. Our findings may not generalize to other subjects, grade levels, or settings.

We're sharing this preview to gather feedback as we build toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis.

This project has been made possible in part through support from the Gates Foundation and Learning Commons.