Numerical correctness
Measuring the tolerance in a kernel check
Kernelwitness compares output against a reference and records the evidence behind its tolerance.
Wrong bins accepted
Recorded results. Select a series to inspect it.
Reproducible processor example
A one percent relative tolerance accepted 1,091 wrong bins. The measured rule accepted four wrong bins in this example.
The example uses the seed seven histogram and three stated summation orders. The bound belongs to those calibration conditions.
The problem
Different correct summation orders can produce different floating point answers. A loose tolerance can also accept incorrect output.
The method
Three correct summation orders were compared with a float64 reference. Four times the worst observed distance set the bound.
Validation
The report records the comparison, calibration and environment. The same example can be reproduced from the public repository.
Recorded measurements
| Method | Recorded result |
|---|---|
| Wrong bins accepted by one percent tolerance | 1,091 of 4,096 |
| Wrong bins accepted by measured rule | 4 of 4,096 |
| Measured bound | 432 ULP |
The example uses the seed seven histogram and three stated summation orders. The bound belongs to those calibration conditions.
Source material
Start with a real operating problem