Case study. GPU transfer overlap

Exposed copy time on a Tesla T4, from 2.518 ms to 0.510 ms.

A copy that waited behind the math now runs under it. Every check passed, and the result carries a signed record a buyer checks for themselves.

Book a demo See the engine

Exposed copy time per stepRecorded

Baseline, matmul then copy2.518 msCopy on a side stream0.510 ms

One Tesla T4. Exposed copy time per step.

The pain

Idle GPUs cost the same as busy ones.

Idle time between bursts of math is paid GPU time. On this step the GPU waited 2.518 ms on a copy before the change.

The right overlap and communication settings move with the chip, the driver and the library version. So every upgrade puts the old tuning in doubt.

Before and after, one stepRecorded

Beforecomputecopyexposed 2.518 msAftercomputecopyexposed 0.510 ms

Compute 35.353 ms and copy 5.141 ms in the baseline. Bars to scale.

What changed

The copy moved under the math.

  • Workload. A 4096 by 4096 float32 matmul with a 64 MiB device to host copy, on one GPU.
  • Change. The copy moved to a side CUDA stream through the copy engine, so it overlaps the matmul.
  • Method. Median of warm steps, step time minus pure compute time. PyTorch 2.11.0 with CUDA 13.0 on sm_75.
  • Scope. Exposed copy time per step on one Tesla T4.

CorrectnessRecorded

9 of 9 checks within predefined toleranceslargest difference 1.9e-05 against the reference, bound 1e-03

Fresh inputs each round, checked against the unchunked product and a float64 CPU product.

Signed proof

A result a buyer checks on their own machine.

The record names the chip, the software and the checks behind the result. A buyer checks it on their own machine, and any edit makes the check fail. A lead can hand it to finance or a customer to check for themselves.

From run to signed record

Run on the chipTesla T4, sm_75, PyTorch 2.11.0, CUDA 13.0Checks and timing recordedrun reportRecord signedEd25519, CLV-6656C-166A2-002EF-BC989Buyer checks itany edit makes the check fail

FAQ

Questions buyers ask.

What exactly was measured?

Exposed copy time per step on one Tesla T4, the time the step waits on a 64 MiB copy after the math is done. It fell from 2.518 ms to 0.510 ms.

How was correctness checked?

Nine checks on fresh inputs, compared with the unchunked product and a float64 CPU product. Every check stayed within a 1e-03 bound.

How does tuning hold after a driver or library upgrade?

The engine tunes again on the new version and shows which setting changed.

How can an engineer check a Computelab.co result?

Each result carries a signed record. The engineer checks it on their own machine, and any change to the result makes the check fail.

Next step

Bring the workload that costs the most GPU hours.

The engine measures it on real hardware, tunes it layer by layer and returns a signed record. See how the engine works or book a demo.

Book a demo See the engine