我们以两个生产相关问题结束。首先，我们在四个仅批量大小不同的图之间共享内核缓存，并计时每次构建，这样我们可以看到后续形状重用已经编译好的内核，而不必再次支付 JIT 成本。然后我们在 CUDA 图中捕获卷积计划，将 cuDNN 句柄的流设置为捕获流。这样工作就落在图中，我们可以测量重放消除后的每次迭代启动开销。
总之，我们在这里构建的内容在代码上很小，但范围很广：一个卷积、一个矩阵乘法和一个注意力内核，每个都表示为图而不是库调用。在这个层次上工作改变了我们能够做出的决策。我们选择了哪些操作合并到单个内核中，因此我们折叠到尾部的偏置加法、激活函数和 AMAX 归约从未将中间结果写入内存。我们自己选择了引擎，而不是接受启发式，每次对候选配置进行计时都告诉我们这个选择的价值。我们还选择何时支付编译成本，通过序列化计划、跨形状共享的内核缓存以及 CUDA 图捕获将其推到热路径之外。与 PyTorch 的对比检查和计时同样重要，因为我们仅刚好匹配 PyTorch 的地方通常是 PyTorch 已经在底层调用 cuDNN 的地方。这标志着这个 API 的价值所在：没有框架层等价物的融合、足够热门以值得自动调优的形状，以及启动和启动成本占主导的小内核。
需要与我们合作以推广您的 GitHub 仓库或 Hugging Face 页面或产品发布或网络研讨会等吗？联系地址：https://forms.gle/wbash1wF6efRj8G58
Sana Hassan，是 Marktechpost 的咨询实习生，同时是 IIT 马德拉斯的双学位学生，热衷于应用技术和人工智能解决现实世界的挑战。由于对解决实际问题的浓厚兴趣，他为 AI 与现实解决方案的结合带来了新的视角。
以实践者为先的 AI/ML 新闻与分析，每月有超过 100 万开发者和研究人员阅读。
In this tutorial：https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Deep%20Learning/cudnn_frontend_nvidia_tutorial_Marktechpost.ipynb , we work through the cuDNN Frontend ：https://github.com/NVIDIA/cudnn-frontend‘s graph API from below the framework: we describe a computation as a graph of operations, let cuDNN pick an engine to run it, and then take control of that choice ourselves. Every kernel we build here is expressed the same way: we declare tensors by their dimensions and strides, chain operations onto them, run the five-step build pipeline of validate, build operation graph, create execution plans, check support, and build plans, and then execute against a variant pack of pointers. We run it all on a single Colab GPU, checking each result against a PyTorch reference so we can see both that the fusion is correct and what it costs. The topics build on each other, moving from a single fused convolution to autotuning across engine configs, FP8-style epilogues, attention, plan serialization, dynamic shapes, and CUDA graph capture.
We start by installing nvidia-cudnn-frontend and solving the problem that trips up most first runs: making libcudnn.so visible to the frontend’s dynamic loader. We force PyTorch to load its bundled cuDNN first and then preload the shared objects explicitly, so the frontend’s own dlopen resolves against a library already resident in the process. We then report the compute capability, pick bfloat16 or float16 accordingly, create the cuDNN handle, and define the helpers for tensor description, graph building, workspace allocation, and event-based benchmarking that the rest of the notebook reuses.
We build our first graph, a convolution followed by a bias add and a ReLU, all fused into a single kernel. We keep every tensor in channels_last because that is what gives cuDNN the NHWC strides its tensor-core engines want, and we pin the output dimensions and strides explicitly so the result is written back in the same layout. We validate the output against torch.nn.functional.conv2d, then benchmark the fused graph against PyTorch running the convolution and activation as separate kernels.
We rebuild the same convolution but stop trusting the heuristic, asking for plans from heuristic modes A, B, and FALLBACK and compiling all of them with build_plan_policy.ALL. We then walk the plan list, build each config, allocate its specific workspace, and time it with execute_plan_at_index, printing throughput and workspace size for every candidate. The spread between the fastest and slowest engine is the point of the exercise, because it tells us how much we gain by shipping an autotuned index instead of accepting the default pick.
We move to a batched matmul and hang a full epilogue off it: an alpha scale supplied as a pass-by-value host scalar, a bias add, an activation, and an AMAX reduction over the result. The AMAX in the same kernel is the pattern that FP8 training relies on, since it collects the scale factor for the next quantization step without a second pass over the output. We compare against a PyTorch chain of baddbmm, activation, and amax, which makes clear that the speedup comes from eliminating epilogue memory traffic rather than from a faster GEMM.
We build a fused scaled dot-product attention graph with causal masking and check it against torch.nn.functional.scaled_dot_product_attention, guarding the whole section behind an SM80 check because the fused kernels need Ampere or newer. We write the causal argument with fallbacks, since the frontend has moved from use_causal_mask toward diagonal_alignment and bound arguments across its 1.x releases. We then serialize a built matmul graph to bytes, reload it into a fresh graph object, and execute it via integer UIDs, which lets us skip the compilation cost entirely at process startup.
We finish with two production concerns. First, we share a kernel cache across four graphs that differ only in batch size and time each build, so we can see later shapes reuse an already compiled kernel instead of paying the JIT cost again. Then we capture the convolution plan inside a CUDA graph, setting the cuDNN handle’s stream to the capture stream. So the work lands in the graph, and we measure how much per-iteration launch overhead the replay removes.
In conclusion, what we built here was small in code but broad in scope: a convolution, a matmul, and an attention kernel, each expressed as a graph rather than a library call. Working at that level changed what we could decide. We chose which operations collapsed into a single kernel, so the bias adds, activations, and AMAX reductions we folded into the epilogues never wrote an intermediate to memory. We chose the engine ourselves instead of accepting a heuristic, and timing every candidate config told us what that choice was worth. We also chose when to pay for compilation, pushing it out of the hot path with serialized plans, a kernel cache shared across shapes, and CUDA graph capture. The checks against PyTorch mattered as much as the timings, since the places where we merely matched it were usually places where PyTorch was already calling cuDNN underneath. That marked out where this API earns its keep: fusions with no framework-level equivalent, shapes hot enough to justify autotuning, and small kernels where startup and launch costs dominate.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us ：https://forms.gle/wbash1wF6efRj8G58
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
Practitioner-first AI/ML news and analysis, read by 1M+ developers and researchers every month.
扫码直达文章详情