On July 13, 2026, the Hugging Face team ran a series of attention benchmarks on an NVIDIA A100‑SXM4‑80GB GPU. Their goal? To show exactly how PyTorch attention profiling looks when you peel back the layers of a Transformer’s core operation. The results aren’t just pretty pictures; they expose a memory copy that most developers never see.
Key Takeaways
- Naive attention spawns an unexpected memory copy due to out‑of‑place masking.
- Switching to
masked_fill_eliminates the copy, shaving off kernel launches. - GPU‑lane traces line up perfectly with the CPU‑lane expectations once in‑place ops are used.
- Hugging Face’s scripts make reproducing the traces straightforward on their infrastructure.
Historical Context
Attention mechanisms have been a staple of deep‑learning models since the early 2010s. Early implementations relied on pure Python loops, which made profiling a low‑priority concern. As GPUs grew more powerful, libraries such as PyTorch introduced dedicated kernels for matrix multiplication and softmax, turning attention into a series of predictable GPU calls. By the time the A100 arrived, developers expected a clean mapping: one kernel per logical step. The reality, however, proved messier. Out‑of‑place operations—convenient but costly—started to insert hidden copies that the profiler could finally surface. Hugging Face’s benchmark highlights that gap, turning a long‑standing blind spot into a concrete data point.
That shift matters. In earlier generations, the overhead of a stray copy could be masked by slower kernels or lower batch sizes. On modern hardware, where each kernel launch costs only a few microseconds, the extra copy becomes a noticeable bottleneck. The 2026 experiment therefore sits at the intersection of three trends: mature attention APIs, high‑throughput GPUs, and increasingly fine‑grained profiling tools.
PyTorch Attention Profiling: What the Traces Reveal
First, the team wrote a straightforward NaiveCausalAttention module. It follows the textbook steps: compute scores with a matmul, scale them, mask with masked_fill, apply softmax, then another matmul with the values. We’ve seen those pieces before in the series; what’s new is how the profiler prints them.
CPU Lane vs. GPU Lane
Running uv run 04_a_naive_attention.py and then uvx trace-util -f traces/ -b produced two clear views. The CPU lane, folded for brevity, listed the five operations we expected. The GPU lane, when unfolded, revealed six kernels. Aside from the two matmul kernels, the scaling mul, the masked_fill mask, and the softmax kernel, there was a stray memory copy kernel.
That copy puzzled the authors. PyTorch often creates a temporary tensor for out‑of‑place ops, copies data, applies the operation, and returns the result. In this case, the culprit was the ordinary masked_fill call.
In‑Place Masking Cuts the Extra Copy
To test the hypothesis, the code was tweaked to use masked_fill_—the in‑place variant. The only change in the forward method was swapping the line:
scores = scores.masked_fill(mask, float("-inf"))
for
scores.masked_fill_(mask, float("-inf"))
The rest stayed the same.
Running the updated script (04_b_inplace_ops_attention.py) and re‑examining the trace showed a leaner GPU lane. The memory‑copy kernel vanished, leaving just the five expected kernels. That single underscore saved a kernel launch and a needless data movement.
Why In‑Place Matters
Developers often shy away from in‑place ops because they fear accidental overwrites. Here, the trade‑off is concrete: fewer kernel launches mean lower latency on the same hardware. On an A100, every microsecond counts, especially when the operation repeats thousands of times inside a Transformer.
Beyond the Naive Path: SDPA and Custom Kernels
The series doesn’t stop at in‑place masking. The next scripts—04_c_sdpa_attention.py and 04_d_kernels_attention.py—explore two other common tricks. The first swaps the manual steps for torch.nn.functional.scaled_dot_product_attention (SDPA), which fuses the scaling, masking, and softmax into a single kernel. The second script plugs in hand‑tuned kernels that aim to reduce the quadratic memory footprint.
Although the blog post doesn’t list the exact kernel names for those scripts, the pattern is clear: each optimization collapses multiple steps into fewer kernels, trimming the trace and boosting throughput.
Profiling on Hugging Face Infrastructure
All these experiments ran on Hugging Face’s own cloud. The authors point out that you can spin up an A100 in a few clicks using Dev Mode with Spaces, or fire off a batch job via the Hugging Face Jobs pipeline. The uv and uvx commands are the same across all four scripts, making the workflow reproducible for anyone with a HF account.
Interpreting the Trace: Practical Tips
When you open a trace, start with the CPU lane. It gives you the high‑level order of operations and lets you verify that the profiler caught everything you wrote. Then flip open the GPU lane. Look for any kernels you didn’t anticipate—like the memory copy in the naive run.
- Identify extra kernels that stem from out‑of‑place ops.
- Replace them with in‑place equivalents when safe.
- Consider fused functions (SDPA) to collapse multiple steps.
- Watch for kernel launch overhead; each extra kernel adds latency.
These steps turn a black‑box trace into a concrete checklist for performance gains.
What This Means For You
If you’ve been profiling Transformers on a GPU and seeing more kernels than you’d expect, the answer might be as simple as an out‑of‑place operation. Switching to masked_fill_ could shave off a copy kernel without changing the mathematical outcome. That’s a win for latency‑critical services like real‑time translation or recommendation engines.
Beyond the masking tweak, the series nudges you toward built‑in fused ops like SDPA. Those functions are already optimized for the A100’s tensor cores, so they’ll likely outperform hand‑rolled sequences. By aligning your code with the patterns the profiler highlights, you can squeeze more performance out of the same hardware.
Concrete Scenarios
Consider a chatbot deployed on a server that handles hundreds of requests per second. Each request runs a small Transformer to generate a response. The naive attention path adds a hidden memory copy for every token. Over a burst of traffic, that copy can translate into milliseconds of extra latency, which users notice as a pause. Replacing the mask with its in‑place version removes that pause entirely, keeping the conversation fluid.
Another case involves a recommendation system that scores millions of items in parallel. The scoring routine uses causal attention to respect temporal ordering. The extra copy kernel inflates GPU memory usage, forcing the system to split batches into smaller chunks. Smaller batches mean more kernel launches and the cumulative effect can reduce throughput. An in‑place mask lets the system stay within the original batch size, preserving the intended throughput.
Finally, think about an edge device that runs a distilled Transformer for speech-to-text. The device’s GPU is a low‑power variant with limited memory bandwidth. Every unnecessary copy competes with the core computation for bandwidth. Dropping the copy frees bandwidth for the main matmul kernels, yielding smoother transcription and lower power draw.
Key Questions Remaining
Even with the clear benefit of in‑place masking, several open issues remain for developers who want to push performance further.
- How do in‑place ops interact with autograd when gradients are needed for fine‑tuning? The trade‑off between speed and gradient fidelity isn’t fully mapped.
- Will future PyTorch releases expose a flag that automatically converts eligible out‑of‑place ops to in‑place equivalents? Such a feature could reduce manual code changes.
- What is the impact of in‑place masking on mixed‑precision workflows? The interplay between FP16 tensors and in‑place updates could affect numerical stability.
Answering these questions will require deeper collaboration between library maintainers and hardware vendors. Until then, developers can rely on the profiler to spot hidden copies and apply the simple fix demonstrated here.
Looking Ahead: Will Profiling Keep Up?
As Transformer models grow larger and new attention variants appear, the profiling toolbox will need to evolve. Will future PyTorch releases expose even finer‑grained kernels for sparsity‑aware attention? Will Hugging Face add automated suggestions based on trace patterns? Those questions will shape how developers keep their models both fast and affordable.
For now, the takeaway is clear: a single underscore can make a measurable difference, and the profiler is the best way to see it.
Sources: Hugging Face Blog, PyTorch Documentation
