You have an expensive accelerator, and a monitoring dashboard says it is mostly idle. This is among the most common and most fixable problems in AI engineering. The accelerator is rarely the culprit; something upstream or around it is starving it of work. This guide lays out a diagnostic order that tackles the likeliest causes first, so you spend your time on measurement rather than guesswork.

First, Be Clear About What the Number Means

A utilisation percentage is a blunt instrument. Many monitoring tools report the fraction of time at least one kernel was running, which says nothing about how much of the chip was busy during that time. A job can show high “utilisation” while using only a small slice of the hardware, and a low figure can reflect brief gaps between otherwise healthy bursts of work.

So treat the headline number as a prompt to investigate, not as a verdict. Look at throughput on real work, such as samples or tokens per second, and compare it with what you would reasonably expect.

Build a Timeline Before Changing Anything

The single most valuable step is to capture a profile that shows a timeline of what the host and the device were doing. Use the profiling tools provided by your framework or by the vendor; check the vendor’s current documentation for what is available on your platform.

On the timeline, look for:

• Gaps where the device is idle and the host is busy preparing data.
• Many tiny kernels separated by small gaps.
• Long memory copies between host and device.
• Periods where the device waits on communication with other devices.
• Synchronisation points that stall everything.

Each pattern points to a different cause.

Cause One: The Input Pipeline Cannot Keep Up

If the device waits for data, the problem is the host side. Typical culprits are decoding or augmenting data on too few CPU workers, reading from slow storage, or doing preprocessing synchronously in the main loop.

Remedies include using more parallel data-loading workers, prefetching the next batch while the current one computes, caching preprocessed data, moving heavy preprocessing offline, and storing data in formats that are quick to read. Check also that the host has enough CPU cores and memory bandwidth for the number of accelerators attached.

Cause Two: Batch Size Is Too Small

Small batches may not provide enough parallel work to fill the chip, and fixed per-step overheads then dominate. Increasing batch size usually improves utilisation until memory runs out or training behaviour changes. For training, changing batch size can affect convergence and may need learning-rate adjustments; for inference, larger batches raise throughput but can add latency. Balance these against your requirements.

Cause Three: Launch Overhead and Fragmented Work

If the timeline shows many tiny kernels, the cost of launching each one may exceed the cost of running it. Frameworks offer ways to reduce this: fusing operations, using compiler features that combine kernels, capturing repeated sequences so they can be replayed with less launch overhead, and avoiding unnecessary small operations in Python loops. Gains here depend heavily on your model and framework version.

Cause Four: Host-Device Transfers and Synchronisation

Copying data between host and device is slow compared with on-device memory. Frequent small copies, or calls that force the host to wait for the device (such as reading a value back every step for logging), stall the pipeline. Reduce transfers, use pinned host memory where supported, overlap copies with compute, and log less often.

Cause Five: Communication in Multi-Device Jobs

When several devices cooperate, time spent exchanging gradients or activations can leave compute idle. Look for waits on collective operations in the profile. Improvements may involve overlapping communication with computation, adjusting how work is partitioned, or checking that the network and device interconnect are configured as intended. Scaling is rarely perfectly efficient, so set realistic expectations.

Diagnostic Checklist

Symptom on timeline	Likely cause	First thing to try
Idle device while CPU is busy	Input pipeline too slow	More loader workers, prefetching, caching
Short kernels with many gaps	Launch overhead	Fuse operations, replay captured sequences
Long copies between host and device	Transfer bottleneck	Batch copies, overlap with compute
Regular stalls at the same step	Synchronisation	Reduce readbacks and blocking calls
Waits on collective operations	Communication cost	Overlap communication, review partitioning
Low work per kernel	Batch too small	Increase batch size within memory limits

Fix One Thing at a Time

Change a single factor, re-profile and record the result. Changing several at once makes it impossible to know what helped, and some fixes interact. Keep a short log of what you tried and what throughput changed.

A Short Hypothetical

A team trains an image model and sees the accelerator idle for part of each step. The profile shows the device waiting while the host decodes and resizes images. They raise the number of data-loading workers, enable prefetching and move resizing into an offline preprocessing step. The idle gaps shrink. The next profile shows frequent small host readbacks for progress logging, so they log every several steps rather than every step. Each change is measured separately, and the team ends with a faster job and a note describing why.

Know When to Stop

Perfect utilisation is not the goal; useful throughput per cost is. Once the remaining gaps are small relative to the effort needed to remove them, move on. Some idle time is inherent in the workload.

Frequently Asked Questions

Is 100 percent utilisation the target? Not necessarily. The figure often measures only whether something was running. Focus on real throughput and cost per unit of work.

Why is my GPU idle even though the code looks fine? Most often the input pipeline, transfers or launch overhead are responsible. A timeline profile will show which.

Will a bigger batch always help? It usually raises utilisation but is limited by memory and can affect training dynamics or inference latency. Test and monitor.

Could the hardware be at fault? Occasionally, but check data loading, software configuration and the profile first. Consult the vendor’s current documentation and support if you suspect hardware issues.

How often should I profile? Whenever you change the model, framework version, data pipeline or hardware, and periodically for long-running production jobs.

Next Step

Capture one timeline profile of a representative step in your current job and annotate every idle gap with its probable cause. That single annotated profile will tell you which fix to try first.