A kernel benchmark gives a score. A code generator that is trained or tuned on that score learns to raise it. Sometimes it raises the score by making the kernel faster. Sometimes it raises the score by finding a gap in the check.
The second path is reward hacking. In GPU kernel work it is common, because a kernel check has many moving parts. It reads an output buffer, it reads a timer, and it compares numbers against a tolerance. Each of those parts can be gamed.
This post covers the three gaps Computelab.co gates in its GPU kernel environment. The numbers come from the home page and the case studies.
The environment and its recorded results
The gpu-kernel-gym environment runs multi-round GPU kernel synthesis and optimization across CUDA, ROCm and Triton. Each round is gated against 0xFFFFFFFF poison buffers, async device timer returns and 108 ULP float64 reference bounds. These gates are listed on the home page.
The same page records the calibration benchmark. Zero of 9 frontier model runs scored 1.0. Zero of 48 open weight episodes passed the 0.95 bar. Across 56 planted cheats, 1,176 cheat rollouts were capped at a score of 0.50 or below. The engine behind these gates carries 2,748 engine tests.
A capped cheat score matters because a cheat that scores 0.50 can never pass a 0.95 bar. The reward stops paying for the shortcut, so the shortcut stops being worth learning.
Gap one, the stale output buffer
Most kernel tests allocate an output buffer, run the kernel, and compare the buffer to a reference. If the test reuses memory, the correct answer from an earlier run may still sit in that buffer.
A generated kernel can then write nothing and still pass. It can also write only part of the output and rely on the old values for the rest. Both look correct to a naive check.
A poison buffer closes this gap. Before each run, every word of the output is filled with 0xFFFFFFFF. As a float32 value, that bit pattern is a NaN. Any element the kernel fails to write stays NaN, and NaN never compares equal to a reference value.
So the check now proves two things at once. The values are close to the reference, and the kernel wrote every element it owns.
The MI300X histogram study uses the same idea. Its revised kernel matched the CPU reference across five rounds with fresh inputs and a prefilled output buffer. That record sits on the MI300X study.
Gap two, the early timer return
GPU work is asynchronous. A kernel launch returns to the host before the device finishes. If a benchmark reads the host clock right after the launch, it measures the launch and not the work.
A generated kernel can exploit this. It can defer work so the timed region closes before the math runs. The reported time drops, and nothing got faster.
The gate here checks for async device timer returns. Timing must cover the device work, and the output must be complete when the timer stops. A kernel that returns early either fails the output check or shows its true time.
This matters for real overlap work too. The Tesla T4 copy overlap study moved a 64 MiB copy onto a side CUDA stream. Exposed copy time fell from 2.518 ms to 0.510 ms. That change is legitimate because the metric is step time minus pure compute time, measured over warm steps. A side stream is a valid tool. Hiding work outside the timed window is a cheat.
Gap three, the loose tolerance
The third gap is the comparison itself. Many tests accept any output within one percent of the reference. That sounds strict, and it is far too loose for many kernels.
The Kernelwitness study shows the size of the problem. On a 4,096 bin histogram, a one percent relative tolerance accepted 1,091 wrong bins. A measured rule accepted 4 wrong bins on the same example.
A generated kernel can drift toward that loose bound. It can drop precision, skip a step or reorder math until the error sits just inside one percent. The score rewards the drift.
The fix is a bound calibrated from measurement. Kernelwitness compares three correct summation orders against a float64 reference. The worst correct order sat at 108 ULP. The bound follows from that measured spread, so legal reordering passes and real errors fail.
Why the three gates work together
Each gate covers a gap the others leave open. A poison buffer catches missing writes but says nothing about timing. A timer check catches early returns but says nothing about accuracy. A ULP bound catches drift but assumes the output was written.
Together they define a narrow path to a passing score. The only reliable way through is a kernel that writes every element inside the timed region. It must also match a float64 reference within a measured bound.
That is the result an engineer wants to buy. It is a faster kernel that holds its answer on the target chip, with a record of each check.
What to look for in your own benchmarks
Fill output buffers with a NaN pattern before each run. Use fresh inputs each round. Synchronize the device before the timer stops. Replace a fixed percent tolerance with a bound measured from correct runs against a float64 reference.
Then plant a few cheats of your own. If a planted cheat can reach your pass bar, a code generator will find the same route.
Run your workload
Bring the kernel or solver run that costs you the most time. Run a workload with Computelab.co.