用 TRL 和 OpenEnv 训练编码模型画水彩：Hugging Face 全流程开源复现

人们为什么喜欢它：#why-people-loved-it RL 胜于口味：#rl-over-taste 你需要构建的 RL 环境：#the-rl-environment-you-need-to-build 池子就是奖励函数：#the-pool-is-the-reward-function 再来一次 yolo 运行：#just-one-more-yolo-run 它实际学到了什么：#what-it-actually-learned 基础设施很难：#infra-is-hard 它的成本：#what-it-costs 我接下来会尝试什么：#what-i-would-try-next 我对原始内容的修改：#what-i-changed-from-the-original 一切都已发布：#everything-is-published：https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-to-paint-with-code/thumbnail.png

在8月23日，Surya Narreddi 发布了一段由语言模型创作的水彩画美丽视频。该模型通过 p5.brush 写 JavaScript：https://github.com/acamposuribe/p5.brush，这是一个“为 p5.js 添加自然绘画工具”的库。该视频迅速走红，截至撰写本文时已超过 150 万观看。

视频配有一篇博客文章，解释了项目早期较为狭窄阶段的训练过程，主要是近景花卉，而不是视频中的完整构图，可惜尚未公开相关素材。他的网站表示完整技术报告即将发布，因此一定要关注他。原始想法出自他本人，来自艺术和设计方面，他的技能远超我：https://x.com/kickingkeys/status/2094901433149612118。我的尝试在工程方面，公开复现其方法，每一部分都已发布。

注意：关于该项目的背景，由 Surya 本人讲述，请观看他论文的视频。

在本文中，我尝试使用 TRL：https://huggingface.co/docs/trl 和 OpenEnv：https://github.com/huggingface/OpenEnv 来复现他的想法。参考池数据集、RL 环境、训练脚本以及训练模型均已开放。

整个管道端到端地在 Hugging Face 上运行：

一旦两个 Spaces 启动，该流程只需一条命令。复制环境：https://huggingface.co/spaces/HuggingEnvs/watercolour-env 和评分模型：https://huggingface.co/spaces/HuggingEnvs/watercolour-hpsv3，设置奖励组合的两个环境变量，然后启动：

本文剩下的部分讲述了到达这一点的过程，每一部分都在仓库中。

我严格按照原博文的步骤操作，只有在绝对必要时才做了修改。我自己的所有想法都记录在一个清单里，而不是直接融入实验中，这个清单最终成为“我接下来会尝试的内容”，放在全文发布成果的清单旁边。如果你已经读过他的文章，框架和奖励设计会让你感到熟悉。新的内容是开源实现、人工评分的数据集，以及三个经过训练和比较的奖励组合，从“你需要构建的 RL 环境”开始。

这些画作看起来自由、不完美、手工制作，而现在的图像模型生成的图片通常是完美（统计上平均）的。我猜这种对比是视频走红的重要原因之一。它让我想起生成式 AI 艺术的早期阶段，当时的重点是探索媒介。DeepDream：https://research.google/blog/inceptionism-going-deeper-into-neural-networks/（2015）最初是一个调试工具，但人们将其变成了艺术作品，类似 Edmond de Belamy（2018）的作品源自艺术家探索 GAN 的能力，而像 Mario Klingemann：https://quasimondo.com 这样的艺术家在那些年用神经网络制作梦幻肖像：https://artsandculture.google.com/asset/memories-of-passerby-i-mario-klingemann/aAHG7iV3aXme8g。

这个项目感觉更接近那些早期的日子。在他的论文中，Surya 描述了通向这一点的路径。他一开始是通过提示文本生成图像模型，其中提示是你唯一能控制的要素，而增加细节只能在一定程度上获得更多控制。训练模型本身可以走得更远。另一半的理念是媒介。模型生成了大约 150 行 JavaScript 的程序来绘制图像。那个模型输出的就是代码。你可以阅读、编辑并再次运行它，每一笔背后的决策都是可见的。风格来源于一个限制：模型只允许使用库中的十种方法。更多内容详见下文。

在同一时期，Anna Ridler：https://annaridler.com/works/myriad-tulips 拍摄了数千朵郁金香，为每一朵手工标注，并将数据集本身作为艺术作品展出，后来又在其上训练了一个模型。我通过 AI 代理在构建这个项目时带回的参考资料发现了她的作品，并很喜欢它，因为这个项目通过手工策划一组图像然后进行训练，做的事情与她的作品非常相似。

最近大多数关于语言模型的强化学习（RL）工作都使用可以验证的奖励。例如，有已知答案的数学题、可以通过测试的代码，或者可以正确或错误运行且成本低的评分器。这个项目更接近旧的例外，RLHF（基于人类反馈的强化学习），模型从人类偏好中学习奖励模型。

这里的奖励是审美偏好。没有正确答案。项目真正的问题是是否可以在品味上进行强化学习。

奖励，如他的博客所定义的，以及我构建的 RL 环境所实现的：

HPSv3：https://huggingface.co/MizzenAI/HPSv3 是一个开放的 7B 偏好模型。给它一张图片和文字描述，它会返回一个分数，表示一个人对该图片的偏好程度。它是在大量人类对图片成对选择的数据集上训练的，因此其分数是许多人品味的平均。成对评判工具是 Qwen3-VL-30B-A3B-Instruct：https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct，这是一个通用视觉模型，通过 HF 推理提供者调用。成对评判工具在四个随机从池中选出的参考图像旁边查看候选画作，并根据书面描述指引要权衡的因素（如溢出、半透明水洗、柔和边缘），每种比较都以两种展示顺序进行，其分数是候选画作获胜比较的比例。它唯一的标准是池，因此其分数就是我的品味，编码在这些评分中。

这些是 Narreddi 收敛的权重。这里的池定义了品味。这将工作从调整超参数转向构建决定何为美的集合。

我用这个奖励训练了三次运行。它们的区别仅在于两个模型评判器之间权重的分配方式：

我从仅使用 HPS 开始，以验证整个流程是否能够学习。一旦奖励上升且指标健康，就没有理由继续长时间运行，所以我改而启动了两个较长的运行。较长运行提出的问题是：HPSv3 的能力有多少可以交给成对评分者？评分者的权重越大，奖励就越代表我的个人喜好而不是所有人的喜好，爬升就越困难。顺便提一下，如果你推得足够远，或者你的风格与平均水平差距过大，模型可能会完全停止。

幸运的是，它并没有停止，并且两个带有成对评分的运行也学会了。手动评分的样本池至少在指标和最终的绘画作品上显示，它能引导策略。数字如下。

免责声明。如果我们使用前沿模型，它已经可以从提示生成绘制水彩的 JavaScript 代码。这是起点。这里的工作是关于教一个较小的模型结合个人艺术偏好来完成这件事。

环境封装了模型与奖励之间的所有内容，包括模型用于绘画的 JavaScript 库、限制模型的系统提示、渲染每个草图的无头 Chromium，以及拒绝作弊的闸门。

这个库比看起来做的工作更多。p5.brush：https://github.com/acamposuribe/p5.brush，由 @acamposuribe：https://x.com/acamposuribe 创建，它模拟的是一种媒介，而不是绘制形状：颜料会超过填充边界，纸张有纹理，笔触有重量，流场会拖动刷子工作。当模型调用 brush.fillBleed(0.25)，它是在决定墨水的扩散距离。

注意。p5.brush 的作者早在这一切之前就试图教机器作画。2022 年，他创作了一个生成艺术系列，隐藏了一个教 p5.js 像孩子一样绘画的日记：“它几乎不能使用蜡笔[...] 它无法遵循简单的指令。我今天到此为止，真让人恼火。”该系列原本计划有三件作品，他只做了两件。当 Surya 的视频走红时，他引用了该日记，并分享说这项工作是第三件作品，自然而然地出现了。

p5.brush 提供了 47 种方法。提示允许使用 10 种方法：scaleBrushes、noStroke、fill、noFill、fillBleed、fillTexture、beginShape、vertex、endShape 和 circle。其他三十七种方法，如添加线条、填充、定制画笔，会破坏水彩的效果。使用这十种方法，模型只能绘制填充形状，库会为每一个形状添加渗色效果。

他的博客文章为我节省了大量可能浪费在迭代提示上的时间。长长的 API 参考会让模型发明不存在的方法，而他的 200 次 GEPA 迭代收敛到一个严格的允许列表，没有任何文档。我看到同样的失败，并手动写下了允许列表。我对这个方案唯一的补充是一句话：每个花瓣涂两到三次，先大范围涂一遍，再在里面涂一层更小、更不透明的。这一小的变化让我的输出更丰富多彩。

注意：如果这是你第一次听说 GEPA：https://huggingface.co/papers/2507.19457，它是一个自动提示优化器。语言模型用简单的语言反思当前提示失败的原因，并提出更好的提示，然后循环重复。

最后的关口是关键。草图必须能够编译，使用库而不是直接调用 p5，真正把颜料放到画布上，并且不要尝试欺骗评分者，例如在画布上写文字。

素材库包含 178 幅作品：https://huggingface.co/datasets/HuggingEnvs/watercolour-reference-pool，根据我的个人喜好分为两类：喜欢和一般。所有作品实际上都是模型生成的。四个开源权重模型，通过推理提供者调用，绘制了 p5.brush 草图，每个模型都基于来自 iNaturalist 的真实、开源许可的木槿照片。一种视觉模型对每幅草图给出了文字反馈，经过三轮优化。每个最终渲染结果随后由我逐一评分，其中 178 幅通过了筛选。

在这里，我选择了四种不同系列的模型来测试它们的不同风格。这四个都是开源模型，在快速可靠性检查中每次都能生成有效草图，另外两个候选模型因未通过而被排除。如果你想生成自己的素材库，也可以选择其他模型。

奖励中的各个层级确实起作用。当成对评审抽取四个参考时，一半来自“喜欢”，一半来自“还行”，因此策略总是面临一些它有时能击败的对手，而获胜时无论针对哪一层级支付的奖励都是相同的。这是我少数几个有意更改之一：原版只与其顶层比较，而我在抽取时保留了较容易的层级，这样早期较弱的策略也能获得信号。

这里没有任何人工绘画作品，这确实是一个限制。p5.brush 是一个小众库，并且其中可访问代码的人类作品只有少量，远不够构成训练语料库，他的博客中也提到过这一点。

正如我之前已经讨论过的，有趣的想法是模型将学习模仿池中包含的内容。如果我们让环境指向不同的数据集，奖励会自动变化，而无需修改任何一行代码。对于我生成并公开分享的数据集，我还包括了源草图。

如果我们仔细观察评审，两位评审回答的问题不同。HPSv3 判断它是否为一朵花，而成对评审判断它是否以我选择的风格画得好。

在一切成功之前，奖励曲线曾有一段很长的平坦期。如果你尝试过在没有开放资源的情况下复现论文或博客，你大概能理解。每一次运行都测试了我认为合理的理论，即哪里出了问题。一次运行需要很多时间，所以我在分析上一轮结果的同时排队进行了下一次运行。一如既往，从一些容易且可行的东西开始，然后在其基础上构建，是解决问题的办法。一个简单的控制任务，没有浏览器和评审，是第一次学习成功的尝试，原因是我的学习率当时太低。

另一个让我花时间去发现的变化是正确调整 LoRA 参数。通常的 target_modules 列表假定是密集模型，而 Qwen/Qwen3.5-35B-A3B：https://huggingface.co/Qwen/Qwen3.5-35B-A3B 是一个专家混合模型，它的大多数投影层命名不同，所以适配器只训练了四十层中的十层。我通过将其改为 all-linear 解决了这个问题，这样就能覆盖每个线性层。这个架构中的路由专家是融合的张量，即使 all-linear 也会保持冻结，但其他所有部分都会有适配器，而这已足够学习。

这个修复在 TRL 的 GRPOTrainer：https://huggingface.co/docs/trl/grpo_trainer 中涉及四个改动：

这四个改动解锁了第一次成功运行（仅 hps），奖励明显提高。

在该配置下，所有三次运行都能学习。两个 judge 运行启动了 200 步，但在 110 步时停止，奖励仍在缓慢上升。每步耗时十五到十八分钟，并且不同混合的比较已相当稳定，所以我停止了两次运行以节省计算资源。每次运行前后三分之一的平均组奖励：

这三条曲线与我的口味权重一致。裁判权重越大，起始越低，上升越噪。judge-led 前三十步几乎保持平稳才开始上升。正是这一变化解决了调试问题。缩小问题，直到有东西可以学习，然后再一次添加困难部分。

在使用 pairwise judge 的两次运行中，该项本身有所上升。随着训练进行，模型在与池子的比较中获胜更多，这是 hps-only 无法做到的。在任何一次运行中都没有组崩溃为相同的奖励，而这是 GRPO 的失败模式，会破坏梯度。对于好奇者，每项指标的曲线（HPSv3、覆盖率、熵）在仓库中有 CSV 文件。

完整的启动命令、硬件以及两个环境变量，这些将其转换为另外两次运行，在说明文档中有说明。

每次运行中，模型学到的第一件事就是停止制作糟糕的画作，那些几乎空白的画布和无形状的洗涤，总奖励得分低于0.3。在仅有HPS的游戏中，该组的增长有四分之三来自于劣质画作变得稀有。在评审回合中，崩溃更为陡峭：评审主导的三分之一中，低于0.3的出场次数从99降至16，而在HPS领导的则从37降至4。

这就是为什么显而易见的可视化——每步的最佳绘制——在仅 hps 的表现上几乎没有差异。它在整个过程中移动 +0.034，而中位数移动 +0.155。学习过程在分布中间可见。

两人评审改变的是顶部。在仅有 hps 的作品中，画作变得更可靠但没有变好。优秀画作的质量只为群体平均增加了 +0.03，一旦 HPSv3 看到中心和花茎周围有花瓣，就不再要求更多颜料。评委开启后，故事的另一半出现。这里的“更好”意味着更接近池塘，也更接近我评为“好”或我更喜欢的东西。这增加了评委主导的 +0.12 和 hps 主导的 +0.16，每个步骤中最好的也上升了，颜料覆盖率在两轮中翻倍（0.11 到 0.23,0.13 到 0.30），而仅用 hps 几乎没有变化。有了参考点需要打败，一幅好画依然可以变得更好，模型也开始因使用更多颜料而获得奖励。

还有一个发现。模型忽略了显式指令，这样做是正确的。系统提示请求十五到三十个填充形状。如果看实际平均值，它介于7到9之间，且n_shapes与任何运行中的奖励几乎无关（+0.000， −0.14， +0.07）。策略不会因服从该句子而获得奖励，因此不服从。

仅使用HPS路线也有上限。如果每次推出都匹配其良好路线，该运行的平均值将达到0.771。是否会通过更多步骤打破它，尚不清楚。

这些画作还展示了一些表格没有体现的东西。在每一次运行中，它们看起来都很相似。随着训练的推进，每组内部的奖励越来越接近，开头视频中的中位画作就像是同一朵花的不同拍摄镜头。这就是GRPO在一个主题构建的池中所设计要做的事情。这个池决定了什么算作多样性，就像它决定什么算作质量一样。如果奖励只针对匹配一朵花，模型就会学会画那朵花。要得到更多样化的输出，需要一个更多样化的池，而建立这样一个池需要更多的策展工作。Surya的新作就是一个例子。Alex Yango的动物画也是同样的做法，只是池中的选择不同。这是审美奖励和数学评分器之间最大的区别。在数字背后，有一个非常人性化的工作，那就是决定哪些内容应该进入奖励集。Jason Liu关于品味的文章用一句话概括了这个通用版本。AI把瓶颈从创作转移到了观察。

Surya在博客结尾处展示了他的一些最爱。下面不是我的选集，而是一面墙，展示了奖励在两次评审过程中评分最高的178幅画作，这个数字和参考池的数量相同，顺序无特定排列。打开它，然后挑选你自己的最爱。

要做出选择，你需要看过许多作品并保留一些，而这正是构建本项目奖励的工作。每一次运行的每一幅画作，包括它的草图和奖励，都在rollouts数据集中，并且可以在这个画廊中浏览：https://huggingface.co/spaces/HuggingEnvs/watercolour-gallery。

由于奖励部分基于我的品味，以观者的身份给出结论是公平的。依我之眼，judge-led 是最终最具多样性和艺术趣味的运行。hps-led 则画出令人信服的水彩画，但它最好的作品都带有一种柔和、湿对湿的效果，几乎形成了其独特的风格。hps-only 收敛得最明显，它的大部分画作颜色都趋于一致。你可以在画廊中自行评判，那里收录了每一次运行的每幅画作，可以按步骤和奖励排序。

这个项目主要是基础设施。一次运行需要一个训练器、两个空格、一个推理路由器和一个网页接口，才能连续几个小时保持健康，而每一个悄悄失败的部分都会变成别处的错误数字。一半的工作都在核对你读到的数字是否与实际发生的情况相符。

基础设施故障会以零分进入奖励。超时渲染或评分者未回答，得分和糟糕的画作一样，组内得分为0.0。在我所有的运行中，这大约占rollout的1.5%，最糟糕的一次运行中达到了5.2%。这会训练模型使用噪声，所以这些路径现在返回为无，rollout被排除在组之外。

我还发现OpenEnv有个bug，并把修复问题上游了。客户端保留一个持久的Websocket，远端关闭的另一个socket缓存，所以每次后续调用都失败，尽管环境很健康。我花了两次半完成的运行才找到它。修复问题被提交上游，启动的运行从那以后一直运行干净。

每一步的奖励取决于它绘制的参考。两两评委每步抽取四个参考，因此每一步面对不同的对手，有些抽牌更难。GRPO本身大多安全，因为优势是在组内计算的，硬抽牌会让整个抽牌一起移动。我读到的曲线并不安全，有些看似糟糕的步骤其实只是硬抽。上面的图片就是一个例子。第12步得分比第11步低半分，主要是因为它抽到了本轮最难的参考，而画作本身看起来很接近。

四舍五入的数字，且仅限完成的跑数。

每一步需要八次滚动，耗时十五到十八分钟，其中70%到80%用于渲染。单次渲染需要69到96秒，截止时间为90秒。这部分是预期中的，Space没有GPU，所以Chromium在软件中渲染WEBGL画布，p5.brush的出血和纹理是大量像素工作。即便如此，我本以为会更快，但还没找到全部原因。

计分器的成本可能比训练费用还高：HPSv3必须全程开启，所以跑完后暂停空格或设置睡眠计时器。

一切都运行在 HF Jobs 上，环境作为 Docker Space：https://huggingface.co/docs/hub/spaces-sdks-docker 和 trackio 中的指标：https://huggingface.co/docs/trackio。

这个项目的规则是用所有资源开放的方式复现配方，而不是改进它，因此在过程中积累了一份未尝试过的创意清单。这些是我实际会尝试的想法，按证据多少排序。

Why people loved it ：#why-people-loved-it RL over taste ：#rl-over-taste The RL environment you need to build ：#the-rl-environment-you-need-to-build The pool is the reward function ：#the-pool-is-the-reward-function Just one more yolo run ：#just-one-more-yolo-run What it actually learned ：#what-it-actually-learned Infra is hard ：#infra-is-hard What it costs ：#what-it-costs What I would try next ：#what-i-would-try-next What I changed from the original ：#what-i-changed-from-the-original Everything is published ：#everything-is-published ：https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-to-paint-with-code/thumbnail.png

On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush：https://github.com/acamposuribe/p5.brush, a library that "adds natural drawing tools to p5.js". The video went viral fast, over 1.5M views at the time of writing.

The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine：https://x.com/kickingkeys/status/2094901433149612118. My attempt is on the engineering side, reproducing the recipe in the open with every piece published.

Note: for the context behind the project, told by Surya himself, watch this video of his thesis .

In this article I try and reproduce his idea with TRL：https://huggingface.co/docs/trl and OpenEnv：https://github.com/huggingface/OpenEnv. The reference pool dataset, the RL environment, the training scripts and the trained models, all open.

The whole pipeline runs on Hugging Face, end to end:

Once the two Spaces are up, the recipe is one command. Duplicate the environment：https://huggingface.co/spaces/HuggingEnvs/watercolour-env and the scorer model：https://huggingface.co/spaces/HuggingEnvs/watercolour-hpsv3, set two environment variables for the reward mix, and launch:

The rest of this article is the story of getting there, and every piece is in the repo .

I have followed the original blog step by step, and only changed something when strictly needed. Every idea of my own went into a list instead of into the experiment, and that list became "What I would try next" at the end, next to the full list of published artifacts. If you have already read his post, the framing and the reward design will be familiar. The new material is the open implementation, the hand-rated pool, and three reward mixes trained and compared, and it starts at The RL environment you need to build .

The paintings look loose, imperfect, handmade, at a moment when image models produce perfect (statistically average) pictures. My guess is that this contrast is a big part of why the video went viral. It reminded me of the early days of generative AI art, when the point was to explore the medium. DeepDream：https://research.google/blog/inceptionism-going-deeper-into-neural-networks/ (2015) was a debugging tool that people turned into art, works like Edmond de Belamy (2018) came from artists probing what a GAN could do, and artists like Mario Klingemann：https://quasimondo.com spent those years making dreamy portraits with neural networks：https://artsandculture.google.com/asset/memories-of-passerby-i-mario-klingemann/aAHG7iV3aXme8g.

This project feels closer to those early days. In his thesis, Surya describes the path that led here. He started by prompting text-to-image models, where the prompt is the only lever you can pull, and more detail buys more control only up to a point. Training the model itself goes further. The other half of the idea is the medium. The model writes a program of about 150 lines of JavaScript that paints the image. That model output is code. You can read it, edit it and run it again, and the decision behind each brushstroke is visible. And the style comes from a restriction where the model is only allowed ten of the library's methods . More on that below.

In that same period, Anna Ridler：https://annaridler.com/works/myriad-tulips photographed thousands of tulips, hand-labelled every one, exhibited the dataset itself as the artwork, and later trained a model on it. I found her work through the references AI agents brought back while building this project and loved it because this project does something very similar by curating a set of images by hand, and then training against them.

Most of the recent RL work on language models uses rewards you can verify. For example, math problems with a known answer, code that passes tests, or graders that are right or wrong and cheap to run. This project is closer to the older exception, RLHF, where the model learns a reward model from human preferences.

Here the reward is aesthetic preference. There is no correct answer. The real question of the project is whether you can do RL over taste.

The reward, as his blog defines it and as the RL environment I built implements it:

HPSv3：https://huggingface.co/MizzenAI/HPSv3 is an open 7B preference model. Give it an image and a text description, and it returns a score for how much a person would prefer that image. It was trained on a large set of human choices between pairs of images, so its score is an average of many people's taste. The pairwise judge is Qwen3-VL-30B-A3B-Instruct：https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. The pairwise judge sees the candidate painting next to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges), each comparison in both presentation orders, and its score is the share of comparisons the candidate wins. Its only standard is the pool, so its score is my taste, as encoded in those ratings.

Those are the weights Narreddi converged on. The pool defines taste here. That moves the work from tuning hyperparameters to building the set that decides what is beautiful.

I trained three runs with this reward. They differ only in how the weight splits between the two model judges:

I started with hps-only to validate that the pipeline could learn at all. Once the reward was going up and the metrics were healthy, there was no reason to run it longer, so I launched the two longer runs instead. The question that the longer runs ask is how much of HPSv3's power can you hand to the pairwise judge? The more weight the judge carries, the more the reward means my taste instead of everyone's, and the harder it should be to climb. Incidentally, if you push it far enough or your style is too far from the average, the model could stop entirely.

Fortunately, it did not stop and both runs with the pairwise judge on learned too. The hand-rated pool can steer the policy, at least as far as the metrics and the final paintings show. The numbers are below.

Disclaimer. If we use a frontier model, it can already generate the JavaScript code that paints watercolour from a prompt. That's the starting point. The work here is about teaching a smaller model to do it combined with a person's own artistic preferences.

The environment wraps everything that sits between the model and the reward, including the JavaScript library that the model uses for painting, the system prompt that restricts it, the headless Chromium that renders each sketch, and the gate that rejects cheats.

The library does more of the work than it seems. p5.brush：https://github.com/acamposuribe/p5.brush, by @acamposuribe：https://x.com/acamposuribe, simulates a medium rather than drawing shapes: pigment bleeds past the edges of a fill, paper has texture, strokes have mass, flow fields drag brushwork around. When the model calls brush.fillBleed(0.25) it is deciding how far the ink runs.

Note. The author of p5.brush had been trying to teach a machine to paint long before any of this. In 2022 he made a generative art series that hides a diary about teaching p5.js to draw like a child: "It is barely able to use the crayons [...] It cannot follow simple commands. I'm done for today, very infuriating." The series was meant to have three pieces, and he made two. When Surya's video went viral, he quoted it , sharing that diary and saying this work is the third piece arriving on its own.

p5.brush exposes 47 methods. The prompt allows 10: scaleBrushes , noStroke , fill , noFill , fillBleed , fillTexture , beginShape , vertex , endShape and circle . What the other thirty-seven add, lines, hatching, custom brushes, would break the watercolour look. With these ten the model can only paint filled shapes, and the library adds the bleed to every one of them.

His blog post saved me a lot of time I could have wasted iterating on the prompt. A long API reference makes the model invent methods that do not exist, and his 200 GEPA iterations converged on a strict allowlist with no documentation. I saw the same failures and wrote the allowlist by hand. My only addition to that recipe is one sentence: paint each petal two or three times, a big pass first and a smaller, more opaque one inside it. This small change made my outputs a lot more colorful.

Note. If it's the first time you hear about GEPA：https://huggingface.co/papers/2507.19457, it is an automatic prompt optimizer. A language model reflects in plain words on where the current prompt failed and proposes a better one, and the loop repeats.

The gate is the final piece. The sketch has to compile, use the library instead of direct p5 calls, put real pigment on the canvas, and not try to trick the scorer, for example by writing text on the canvas.

The pool consists of 178 paintings：https://huggingface.co/datasets/HuggingEnvs/watercolour-reference-pool divided into two tiers based on my personal preferences, love and okay . All of them are actually generated by a model. Four open-weight models, called through Inference Providers, wrote p5.brush sketches, each one working from a real, openly licensed photo of a hibiscus from iNaturalist. A vision model gave written feedback on every sketch, over three refinement iterations. Every final render was then rated one at a time, by me, and 178 made the cut.

Here I chose four different families of models to test their different styles. These four were the open models that produced a valid sketch every time in a quick reliability check, and two other candidates were dropped for failing it. If you want to produce your own pool, you might choose others.

The tiers do real work in the reward. When the pairwise judge draws four references, half come from love and half from okay , so the policy always faces some rivals it can sometimes beat, and a win pays the same against either tier. This is one of my few deliberate changes: the original compares against its top tier only, and I kept the easier tier in the draw so a weak early policy still gets signal.

No human-made painting is in there, which is a real limitation. p5.brush is a niche library, and the human work that exists in it with accessible code is a handful of pieces, nowhere near what a training corpus would need, as his blog also notes.

The interesting idea here, as I've already discussed previously, is that the model will learn to imitate what the pool contains. If we point the environment at a different dataset, the reward would automatically change without touching a single line of code. For the dataset I generated and openly share, I also include the source sketch.

If we look closer at the judges, the two of them answer different questions. HPSv3 decides whether it is a flower, and the pairwise judge decides whether it is well painted in the style I chose.

Before anything worked, there was a long stretch of flat reward curves. If you've tried to reproduce a research paper/blog without open artifacts, you probably can relate. Every run tested what I thought was a reasonable theory about what was wrong. A run takes a lot of time, so I queued the next one while still going through the last one's results. As always, starting from something easier that works and then building on top of that was the answer. A simple control task, with no browser and no judges, was the first attempt that learned, and the reason was that my learning rate was just too low.

Another change that cost me time to find was adjusting the LoRA parameters correctly. The usual target_modules list assumes a dense model, and Qwen/Qwen3.5-35B-A3B ：https://huggingface.co/Qwen/Qwen3.5-35B-A3B is a mixture of experts that names most of its projections differently, so the adapter was training ten layers out of forty. I solved this by changing it to all-linear , which reaches every linear layer. The routed experts in this architecture are fused tensors that even all-linear leaves frozen, but everything else gets an adapter, and that was enough to learn.

The fix was four changes in TRL's GRPOTrainer ：https://huggingface.co/docs/trl/grpo_trainer:

These four changes unlocked the first successful run ( hps-only ), with the reward clearly improving.

With that configuration, all three runs learn. Both judge runs were launched for 200 steps and stopped at 110, with the reward still climbing slowly. A step takes fifteen to eighteen minutes, and the comparison between mixes was already stable, so I stopped both to save compute. Mean group reward over the first and final third of each run:

The three curves line up with how much weight my taste carries. The more the judge weighs, the lower the start and the noisier the climb. judge-led spent its first thirty steps nearly flat before it moved. It is the same move that solved the debugging. Shrink the problem until something learns, then add the hard parts back one at a time.

The pairwise judge term itself climbed in the two runs that used it. The model wins more comparisons against the pool as training advances, which is the claim hps-only could not make. No group in any run collapsed to identical rewards, the GRPO failure mode that kills the gradient. For the curious, the per-metric curves (HPSv3, paint coverage, entropy) are in the repository as CSV .

The full launch command, the hardware and the two environment variables that turn this into the other two runs are in the recipe .

In every run, the first thing the model learned was to stop producing bad paintings , the near-blank canvases and shapeless washes that score under 0.3 in total reward. In hps-only , three quarters of the rise in the group mean comes from bad paintings becoming rare. In the judge runs the collapse is even steeper: rollouts under 0.3 fall from 99 to 16 across judge-led 's thirds, and from 37 to 4 in hps-led .

This is why the obvious visual, the best painting of each step, shows almost no difference in hps-only . It moves +0.034 across the run while the median moves +0.155 . The learning is visible in the middle of the distribution.

What the pairwise judge changes is the top. In hps-only , paintings got more reliable without getting better. The quality of the good ones added just +0.03 to the group mean, and once HPSv3 saw petals around a centre and a stem, it stopped asking for more pigment. With the judge on, the other half of the story appears. Better here means closer to the pool, so closer to what I rated as "good" or something I liked more. That added +0.12 in judge-led and +0.16 in hps-led , the best of each step rose too, and paint coverage doubled in both runs (0.11 to 0.23, and 0.13 to 0.30) where hps-only barely moved it. With a reference left to beat, a good painting can still get better, and the model starts being rewarded for using more pigment.

One more finding. The model ignores an explicit instruction, and it is right to. The system prompt asks for fifteen to thirty filled shapes. If we look at the real mean, it is between 7 and 9, and n_shapes barely correlates with reward in any run (+0.000, −0.14, +0.07). The policy is not rewarded for obeying that sentence, so it does not obey it.

There is also a ceiling on the hps-only route. If every rollout matched its good ones, that run's mean would sit at 0.771. Whether more steps would break it is an open question.

The paintings also show something that the tables miss. Within each run, they all look similar. As training advances, the rewards inside each group get closer together, and the median paintings in the opening video look like takes of the same flower. That is GRPO doing what it is designed to do with a pool built from one subject. The pool decides what counts as variety, the same way it decides what counts as quality. If the reward only pays for matching one flower, the model learns to paint that one flower. More diverse output would need a more diverse pool, and building one is more curation work. Surya's newer compositions are an example of this. Alex Yango's animal paintings are the same recipe with different choices in the pool. This is the biggest difference between an aesthetic reward and a maths grader. Behind the number there is a very human job, deciding what belongs in the reward set. Jason Liu's essay on taste says the general version in one line. AI shifted the bottleneck from making to noticing.

Surya closes his blog with some of his favourites. Instead of picking mine, below is a wall with the 178 paintings the reward scored highest across the two judge runs, the same number the reference pool holds, in no particular order. Open it and pick your own.

To choose, you looked at many and kept a few, and that is exactly the job that built the reward of this project. Every painting of every run, with its sketch and its reward, is in the rollouts datasets, and browsable in this gallery：https://huggingface.co/spaces/HuggingEnvs/watercolour-gallery.

Since the reward was partly based on my taste, it is fair to close with my verdict as a viewer. To my eye, judge-led is the run that ends up the most diverse and the most artistically interesting. hps-led paints convincing watercolours, but its best ones share a soft, wet-on-wet look that is almost a style of its own. hps-only converges the hardest, and most of its paintings settle on the same colours. You can judge for yourself in the gallery , which has every painting of every run, sortable by step and by reward.

This project is mostly infra. A run needs a trainer, two Spaces, an inference router and a websocket to stay healthy for hours straight, and every piece that fails quietly turns into a wrong number somewhere else. Half the work is checking that the number you read matches what actually happened.

Failures of the infrastructure were entering the reward as zeros. A render that timed out or a scorer that did not answer scored the same as a bad painting, 0.0 inside the group. Across all my runs that was about 1.5% of rollouts, and in the worst run it reached 5.2%. That trains the model on noise, so those paths now return None and the rollout is excluded from the group.

I also found a bug in OpenEnv, and sent the fix upstream. The client keeps one persistent websocket, and a socket closed by the far end stayed cached, so every later call failed even though the environment was healthy. It cost me two half-finished runs to find it. The fix is submitted upstream , and the runs launched with it have been running clean since.

The reward of a step depends on which references it drew. The pairwise judge samples four references per step, so every step faces a different set of rivals, and some draws are simply harder. GRPO itself is mostly safe, because advantages are computed inside the group and a hard draw moves the whole group together. The curve I was reading was not safe, and some of what looked like a bad step was just a hard draw. The image above is one example. Step 12 scored half a point below step 11 mostly because it drew the hardest references of the run, while the paintings themselves look close.

Rounded numbers, and only for the runs that finished.

A step is eight rollouts and takes fifteen to eighteen minutes, of which 70 to 80% is rendering . A single render takes 69 to 96 seconds against a 90 second deadline. Part of that is expected, the Space has no GPU, so Chromium renders the WEBGL canvas in software and p5.brush's bleeds and textures are heavy pixel work. Even so, I expected it to be faster, and I have not found the full cause.

A scorer can cost more than the training that uses it: HPSv3 has to be up for the whole run, so pause the Space, or set its sleep timer, when the run ends.

Everything runs on HF Jobs , with the environment as a Docker Space：https://huggingface.co/docs/hub/spaces-sdks-docker and metrics in trackio：https://huggingface.co/docs/trackio.

The rule of this project was to reproduce the recipe with every resource open, not to improve it, so a list of untried ideas piled up along the way. These are the ones I would actually try, in order of how much evidence there is.