Every GPU kernel check ends in a comparison. The kernel output is set beside a reference, and a tolerance decides pass or fail. The tolerance is often a single number picked by habit, such as one percent relative error.
For reductions, that habit fails. This post explains why, using the 4,096 bin histogram example recorded on the Kernelwitness study.
Why reductions drift
A reduction adds many values into fewer values. A sum, a histogram and a softmax denominator are all reductions. On a GPU, thousands of threads add in parallel. The order of those additions changes with the launch shape, the chip and the library version.
Floating point addition does not follow the associative law. Adding a, then b, then c can give a slightly different result than adding c, then b, then a. Both answers are correct for the hardware. Neither matches exact math.
So a correct kernel does not produce one fixed output. It produces a small cloud of outputs, one per legal summation order. A good tolerance must accept that whole cloud and reject everything outside it.
What one percent accepted
The recorded example uses a 4,096 bin histogram with three stated summation orders. It then plants wrong output across the bins and asks each rule which bins it accepts.
A one percent relative tolerance accepted 1,091 of 4,096 wrong bins. More than a quarter of the planted errors passed as correct.
The measured rule accepted 4 of 4,096 wrong bins in the same example. The other 4,092 wrong bins fell outside the bound and failed.
The gap is large because one percent has no link to the real spread of correct answers. For most bins, the legal drift between summation orders is far smaller than one percent. The extra room becomes space where wrong answers hide.
How the bound was calibrated
The method on the Kernelwitness study has three steps.
First, compute a float64 reference. Float64 carries far more precision than the float32 kernel, so it serves as a stable target.
Second, run three correct summation orders and measure each one against the float64 reference in ULP. A ULP is the gap between one floating point number and the next representable number. Measuring in ULP scales the error to the size of each value. Small bins and large bins are then judged fairly.
Third, take the worst observed distance and set the bound from it. In this example the worst correct order sat at 108 ULP. Four times the worst observed distance set the bound.
The margin covers legal orders that the calibration did not sample. The bound still stays tight enough to reject real errors, as the 4 of 4,096 result shows.
The bound belongs to its conditions
A calibrated bound is tied to the conditions that produced it. The study ties its bound to the 4,096 bin histogram, the three stated summation orders and seed 7. The full example is published as evidence-example.json.
When the chip, the data type or the reduction size changes, the spread of correct answers changes too. The calibration should run again on the new conditions. That costs far less than a wrong answer that a loose tolerance approved.
The home page lists the 108 ULP float64 reference bound as a gate in the GPU kernel environment. It sits beside poison buffers and async timer checks. A kernel must write every element, time honestly and stay inside a measured bound.
Absolute bounds for non reduction checks
Not every check needs ULP calibration. The Tesla T4 copy overlap study checked a 4096 by 4096 float32 matmul with nine checks on fresh inputs. Its recorded bound was an absolute 1e-3.
The maximum absolute error against a float64 CPU product was 7.27e-06. The largest difference against the unchunked reference was 1.9e-05. Both sat well inside the bound, and the run report records each measurement.
The lesson is the same in both cases. A tolerance should be written down, tied to a reference, and backed by measurement.
A checklist for your own reductions
Build a float64 reference for the reduction. Run several correct summation orders on the target chip. Measure each one in ULP against the reference. Set the bound from the worst observed distance with a stated margin.
Then plant wrong outputs and count how many each rule accepts. If a one percent rule lets through a quarter of the planted errors, the rule is the bug.
Run your workload
Bring the kernel or solver run that costs you the most time. Run a workload with Computelab.co.