The disappointing result is familiar: you go from one GPU to eight, and training gets four or five times faster instead of eight. Sometimes it barely improves. Before spending on more hardware, it helps to understand where the lost time goes, because the cause decides the fix.

What “Scaling Efficiency” Means

Scaling efficiency compares actual speedup with ideal speedup. If eight devices deliver a speedup of six, the efficiency is three quarters. The gap is time spent on something other than useful computation: waiting for communication, waiting for data, or waiting for the slowest device.

Notice that these are different problems. Communication costs grow as you add devices. Data loading problems are independent of the GPU count but become visible once the compute side gets faster. Imbalance appears when work is uneven. Diagnosing which one you have comes before choosing a remedy.

Where the Time Goes

Gradient synchronisation. In data-parallel training, each device computes gradients on its own slice of the batch, and then the devices must combine them before the next step. That exchange moves a volume of data tied to the model size, every step. If the connection between devices is slow relative to the computation, the GPUs sit idle waiting.

Interconnect topology. Devices within a single server are often linked by a faster connection than devices in different servers. Scaling across the boundary between those two can reveal a sudden drop in efficiency. Check the vendor’s current documentation for the actual topology of your machines.

Small per-device work. If you keep the total batch fixed and add devices, each one gets less work per step, so communication becomes a bigger share of the step time. Strong scaling, where the problem stays the same size, is harder than weak scaling, where the problem grows with the device count.

Input pipeline. If reading, decoding and augmenting data cannot keep up, the GPUs wait on the CPU or storage. Faster GPUs make this worse, not better.

Stragglers. Synchronous training moves at the speed of the slowest device in each step. A single slow node, a noisy neighbour on shared infrastructure or uneven data can hold everyone back.

Large-batch side effects. More devices often means a bigger effective batch. Beyond some point, training may need more steps or tuned hyperparameters to reach the same quality, which eats into the speed gain even if throughput scales well.

Diagnosing Before Spending

A methodical approach saves a lot of money.

1. Measure single-device throughput first. If one GPU is already underused, as discussed in guides on low utilisation, fixing that comes before scaling out.
2. Measure step time at several device counts. Plot throughput against the number of devices and note where the curve bends.
3. Profile a step. Look for gaps between kernels, long communication calls and time waiting on data. Most framework profilers can show this.
4. Compare compute time with communication time. If communication is a large share of the step, the interconnect or the overlap strategy is your target.
5. Check data loading in isolation. Run the input pipeline alone and see whether it can supply data faster than the GPUs consume it.

A Hypothetical Case

Suppose a team trains the same model on one, two, four and eight GPUs inside one server, then moves to sixteen across two servers. Throughput scales respectably up to eight devices, then barely improves when the second server is added.

The profile shows long waits during gradient exchange after the move. The connection between servers is considerably slower than the one inside a server, and the exchange is not overlapped well with computation. Their options: increase the per-device batch so each step has more computation to hide the communication behind, enable overlap of gradient exchange with the backward pass, use gradient accumulation to synchronise less often, or consider sharding strategies that change what gets communicated. They try the first two and measure again.

The lesson is that the knee in the curve points to a specific resource, and that resource is what to improve.

Symptoms and Likely Causes

What you observe	Likely cause	First thing to try
Efficiency drops when crossing servers	Slower inter-server link	Larger per-device batch, overlap communication
GPUs idle between steps, CPU busy	Input pipeline bottleneck	More loader workers, caching, faster storage
Efficiency declines steadily as devices rise	Communication share growing	Gradient accumulation, check topology
Step time varies a lot	Stragglers or uneven data	Inspect per-device timings, balance workload
Throughput scales but time to quality does not	Large-batch effects	Tune learning rate schedule, check convergence
One device far slower than others	Hardware or thermal issue	Monitor clocks and temperature per device

Remedies That Usually Help

• Increase per-device work. More computation per step gives communication more to hide behind.
• Overlap communication and computation. Many frameworks support starting gradient exchange while the backward pass is still running.
• Synchronise less often. Gradient accumulation lets you do several local steps per exchange.
• Fix the input path. Prefetching, parallel workers and local caching are inexpensive compared with hardware.
• Reconsider the parallelism strategy. For very large models, splitting the model itself rather than only the data may be required, with its own communication patterns.

Frequently Asked Questions

Is perfect linear scaling ever realistic? For some workloads and small device counts it can come close, but expecting it at scale is usually unrealistic. Plan around measured efficiency instead.

Does faster networking always fix it? Only if communication is actually the bottleneck. Profile first, because buying interconnect to fix a data-loading problem wastes money.

Why is scaling worse for small models? Small models have little computation per step relative to the gradient exchange, so communication dominates sooner.

Should I always use the largest batch that fits? Not necessarily. Larger batches can improve throughput but may hurt convergence unless you retune. Judge by time to reach your target quality.

How do I know if a straggler is the problem? Compare per-device step times. Consistently one device being slower is a strong sign.

Next Step

Run your training job at one, two and four devices for a short fixed number of steps and chart the throughput. If the curve bends early, take one profile at the bend and identify which wait dominates before changing anything.