Apple 研究：Embarrassingly Simple Self-Distillation 提升代码生成能力

Apple 机器学习研究提出一种极简的自我知识蒸馏方法，让大语言模型仅使用自身原始输出即可提升代码生成能力，无需验证器、教师模型或额外数据。

该方法在多个代码生成基准上取得显著改进，且计算成本极低。

研究展示了自我蒸馏在代码任务中的有效性，为模型自我改进提供了新思路。

Embarrassingly Simple Self-Distillation Improves Code Generation

Authors Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, Yizhe Zhang

View publication：https://arxiv.org/abs/2604.01193

Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation.

BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks

August 3, 2024 research area Human-Computer Interaction：/research/?domain=Human-Computer%20Interaction, research area Tools, Platforms, Frameworks：/research/?domain=Tools%2C%20Platforms%2C%20Frameworks conference IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)：/research/?event=IEEE%20Symposium%20on%20Visual%20Languages%20and%20Human-Centric%20Computing%20(VL%2FHCC)

This paper was accepted at IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2024.

Programmers frequently engage with machine learning tutorials in computational notebooks and have been adopting code generation technologies based on large language models (LLMs). However, they encounter difficulties in understanding and working with code produced by LLMs. To mitigate these challenges, we introduce a novel workflow into…

Applying RLAIF for Code Generation with API-usage in Lightweight LLMs

July 1, 2024 research area Methods and Algorithms：/research/?domain=Methods%20and%20Algorithms, research area Speech and Natural Language Processing：/research/?domain=Speech%20and%20Natural%20Language%20Processing Workshop at ACL：/research/?event=ACL%20Workshop

This paper was accepted at the Natural Language Reasoning and Structured Explanations workshop at ACL 2024.

Reinforcement Learning from AI Feedback (RLAIF) has demonstrated significant potential across various domains, including mitigating harm in LLM outputs, enhancing text summarization, and mathematical reasoning. This paper introduces an RLAIF framework for improving the code generation abilities of lightweight (<1B parameters) LLMs. We…

Our research in machine learning breaks new ground every day.