“Will it fit?” is the first question in almost every GPU planning conversation, and the honest answer is usually “roughly, and here is how to check.” You can get a useful estimate with arithmetic you can do on paper, then confirm it with one real run. The skill is knowing which pieces of memory exist and which of them grow when you change something.

The Pieces of GPU Memory

Whether you are training or serving, memory is consumed by a handful of components.

Weights. The model parameters. Their size is the parameter count multiplied by the bytes per parameter at the chosen precision. A model with a billion parameters stored in a 16-bit format needs about two gigabytes for weights alone, because each parameter takes two bytes. At 8 bits it is about one gigabyte. This is simple multiplication, and it is the one number you can compute reliably.

Activations. Intermediate results produced during the forward pass. In training they are kept so the backward pass can use them. They scale with batch size, sequence length and model depth, which makes them the most variable component.

Gradients. One value per trainable parameter, usually stored at a similar size to the weights. Only needed in training.

Optimizer state. Adaptive optimizers keep extra values per parameter, commonly more than one. This often makes optimizer state larger than the weights themselves, and surprises people who only budgeted for the model.

KV cache. In transformer inference, past attention keys and values are stored so they need not be recomputed. This grows with the number of concurrent requests and the length of each sequence, and for long contexts or many users it can rival the weights.

Workspace and fragmentation. Kernels need temporary buffers, and the allocator wastes some space. Leave headroom for these.

A Training Estimate Step by Step

1. Compute weight memory from parameter count and precision.
2. Add gradient memory, roughly equal to the weights’ size.
3. Add optimizer state, which depends on the optimizer; check its documentation for how many values it stores per parameter.
4. Estimate activations by running a tiny batch and measuring, then scaling.
5. Add a safety margin for workspace and fragmentation.

Step four is the one worth measuring rather than calculating, since activation memory depends on implementation details like attention kernels and checkpointing.

An Inference Estimate Step by Step

1. Compute weight memory at the serving precision.
2. Estimate KV cache per token for your model, which depends on layer count, hidden dimensions and cache precision.
3. Multiply by the number of tokens you need to hold at once: concurrent requests times typical sequence length.
4. Add runtime overhead and margin.

The concurrency number is where plans fall apart. A system sized for the average request count can run out of memory during a burst of long conversations.

A Hypothetical Walkthrough

Imagine a team wants to fine-tune a model with several billion parameters. Weights alone fit comfortably on their GPU, so they assume the job fits. Then it fails with an out-of-memory error at the first training step.

The mistake was budgeting for weights only. Adding gradients and optimizer state roughly tripled or quadrupled the requirement, and activations added more on top. They had several options: reduce the batch size and accumulate gradients, enable activation checkpointing to trade compute for memory, use a lower-precision optimizer state, or switch to a parameter-efficient method that trains far fewer parameters. Each lowers a different component, so knowing which component dominates tells them which lever to pull.

Levers That Reduce Memory

Lever	What it reduces	Cost or tradeoff
Lower batch size	Activations	Smaller batches may change training dynamics
Gradient accumulation	Activations, keeps effective batch	Slightly slower steps
Activation checkpointing	Activations	Extra compute from recomputation
Lower-precision weights	Weights, KV cache	Possible accuracy impact; validate
Parameter-efficient fine-tuning	Gradients, optimizer state	Different quality characteristics
Sharding across devices	Weights, gradients, optimizer state	Communication overhead, complexity
Shorter context or fewer concurrent requests	KV cache	Product limits

Confirming With a Measurement

Estimates are for planning; measurements are for decisions. Run a single training step or a short serving test and read the framework’s reported peak memory. Compare that number against your estimate for each component. If they disagree by a lot, the usual culprits are an implementation detail you did not account for, such as unfused attention, extra copies of tensors, or memory held by caching allocators.

Also be careful about what “used” means. Many frameworks reserve memory beyond what is actively in use, so monitoring tools may show higher numbers than the actual requirement. Look at the peak allocated figure as well as reserved memory.

Frequently Asked Questions

Why does my job run out of memory after running fine for a while? Memory use can grow with longer sequences, larger final batches or fragmentation over time. Test with your longest realistic input, not an average one.

Is memory usage the same across GPUs? Your model’s logical requirement is the same, but the runtime may differ in overhead and kernel choices. Always verify on the target hardware.

Does a bigger batch always use proportionally more memory? Activations grow with batch size, but weights, gradients and optimizer state do not. So total memory grows slower than the batch size.

Can I use CPU memory to extend GPU memory? Offloading is possible in many frameworks and can make a job fit, but it is usually much slower because data must cross the connection between CPU and GPU. Use it when fit matters more than speed.

How much headroom should I leave? There is no universal figure. Leave a margin that survives your worst realistic input, then adjust after measuring.

Next Step

Write a one-line memory budget for your target job listing each component in gigabytes, then run one short test and record the peak figure. Where the two differ, find out why before you scale the job up.