For decades the story of better chips was largely a story of shrinking transistors on a single piece of silicon. That story has not ended, but it has been joined by another: assembling several pieces of silicon into one package so they behave like a single, larger device. For AI hardware in particular, this approach has become central. This article explains the idea of chiplets, what advanced packaging does, and where the trade-offs lie.

The Problem Chiplets Address

A monolithic chip puts everything on one die. That keeps communication fast and simple, but it runs into two limits.

The first is size. A manufacturing system can only print a die up to a maximum area, so a single die cannot grow without bound.

The second is yield. As described in most manufacturing overviews, the larger a die is, the more likely it is to contain at least one defect, so a smaller proportion of large dies come out working. Large dies therefore tend to be more expensive per good chip.

Chiplets sidestep both limits by building a system from several smaller dies.

What a Chiplet Design Looks Like

Instead of one large die, the processor is divided into functional pieces: perhaps several compute dies, an input and output die and stacks of memory. Each piece is manufactured separately, tested and then assembled onto a shared base inside one package. Connections between them are made very dense and short so that data can pass at high speed and low energy cost compared with travelling across a circuit board.

A useful side effect is that different pieces can use different manufacturing processes. Compute logic may benefit from the most advanced process, while input and output circuitry, which gains less from shrinking, can use a more mature and less costly one.

How the Pieces Are Joined

Several packaging approaches exist, and vendors use different names for them. In general terms:

• 2D arrangements place dies side by side on an organic substrate with ordinary routing.
• 2.5D arrangements place dies side by side on an intermediate layer, sometimes called an interposer, which provides much denser wiring between them than a standard substrate can.
• 3D arrangements stack dies on top of one another, connected by vertical links, giving very short paths between layers.

High-bandwidth memory is a well-known example of stacking: several memory dies are piled up and placed beside the processor, offering far more bandwidth per unit of board area than conventional memory arrangements. Specific stack heights, link counts and speeds change between generations, so check the manufacturer’s current documentation for details.

Why AI Processors Lean on This Approach

AI workloads put enormous pressure on memory bandwidth and demand large amounts of compute. Advanced packaging lets designers place memory right beside the compute and pack more total silicon into a device than one die could provide. It also lets a vendor reuse a well-tested compute chiplet across several products, mixing and matching pieces to create different tiers.

Trade-Offs

Chiplets are not a free lunch.

• Interconnect cost. Communication between dies, though fast, is slower and more energy hungry than communication inside one die. Software and architecture must account for the extra distance.
• Packaging complexity. Assembling several dies precisely is hard. A fault during assembly can waste expensive, already-tested dies, so test strategy matters.
• Capacity constraints. Advanced packaging lines are specialised, and their capacity can limit how many finished accelerators reach the market.
• Thermal challenges. Dense stacks concentrate heat, so cooling design is critical.
• Design effort. Standard interfaces and careful partitioning are needed to make dies from different sources work together.

Monolithic and Chiplet Designs Compared

Aspect	Monolithic die	Chiplet-based package
Yield on large designs	Lower, since one defect can scrap the whole die	Better, because small dies are tested before assembly
Internal communication	Fastest and simplest	Fast, but with some added latency and energy cost
Process flexibility	One process for everything	Different processes for different pieces
Product variety	Needs a new die per product	Can recombine existing chiplets
Packaging difficulty	Simpler	More complex and capacity-sensitive
Thermal design	Single heat source	More concentrated, requires careful cooling

What This Means for Buyers and Engineers

You rarely choose chiplets directly; you choose a product. But the packaging approach affects things you do care about: how much memory the device can carry, how its bandwidth compares with alternatives, how supply may be constrained and how the device behaves when work crosses internal boundaries. When a datasheet describes a multi-die design, ask how the vendor presents it to software, as one device or several, and what that means for placement of your model.

A Short Hypothetical

Consider a vendor planning two accelerators: a flagship and a smaller model. With a monolithic approach each would need its own large die, doubling design effort and exposing both to yield risk. With a chiplet approach the vendor builds one compute chiplet design and places four on the flagship and two on the smaller part, both sharing memory stacks of different counts. The vendor saves design effort and can salvage some production by assigning lightly defective dies to the smaller product, although assembly and packaging now become the critical production steps.

Frequently Asked Questions

Is a chiplet the same as a core? No. A core is a processing unit inside a die. A chiplet is a whole small die that is combined with others in one package.

Do chiplets make chips cheaper? Often they improve yield for large designs and allow reuse, but packaging adds cost. The overall result depends on the specific design and volumes.

Why is high-bandwidth memory placed next to the processor? Short, dense connections allow much higher bandwidth at lower energy per bit than distant memory could provide.

Can chiplets from different companies be combined? Industry efforts exist to standardise die-to-die interfaces, but the practice is still maturing. Check current specifications from the relevant bodies.

Does a chiplet design change how I program the device? Usually the vendor hides the structure, but performance may be sensitive to how work and data are placed. Consult the vendor’s current documentation.

Next Step

Take the datasheet of any current AI accelerator and identify how many dies and memory stacks it contains. Then ask what that implies for memory capacity, bandwidth and supply, and note any questions to put to the vendor.