Memory & AI Compute Silicon

Source inventory

The technology: memory as the AI-compute bottleneck

AI accelerators are not limited only by multiply-accumulate logic. A large share of energy and latency in neural-network inference and training is spent moving weights and activations between off-chip DRAM and the compute die. Processing-in-memory literature consistently argues that data movement, not arithmetic, dominates energy in data-heavy workloads [S11].

The incumbent solution is High Bandwidth Memory (HBM). HBM is a DRAM architecture that stacks multiple DRAM die vertically using through-silicon vias (TSVs), connects them to a base die, and places the stack on a silicon interposer close to the GPU or accelerator die. JEDEC’s HBM3 standard, JESD238, defines an interface with eight independent channels per stack, each 128 bits wide, for a total bus width of 1,024 bits per stack [S3]. That width allows HBM to deliver much higher bandwidth than a typical DDR interface without pushing per-pin signaling to extreme speeds.

HBM3’s baseline per-pin data rate is 6.4 Gb/s in the JEDEC standard [S3]. Suppliers market HBM3E as a follow-on with higher data rates. Micron’s HBM3E product page advertises 9.2 Gb/s per pin and >1.2 TB/s per stack [S6]. Those are vendor marketing numbers, not the baseline JEDEC HBM3 number. The distinction matters: “HBM3E” is supplier nomenclature layered on top of the HBM3 standard, and system integrators may derate the peak data rate for power, thermal, signal-integrity, and yield reasons.

A parallel research trajectory is compute-in-memory (CIM) or processing-in-memory (PIM), in which logic is placed inside or adjacent to the memory array so that at least some computation occurs without moving data to a host processor. A granted Micron patent, US 9,430,191 B2, claims apparatus and methods for performing logical operations using DRAM sensing circuitry [S1]. The patent describes using sense-amplifier-based circuits to perform operations in the memory array, a form of near-data processing. Separately, a Nature paper reports a resistive-RAM-based compute-in-memory chip [S2]. The paper is peer-reviewed and demonstrates analog in-memory matrix operations, but its measured results should be quoted only from the paper itself; any number pulled from secondary commentary would need primary source validation [S2].

The legal significance of a granted patent is narrower than the engineering significance of a shipping product. A granted patent is recorded evidence that the assignee claims a particular invention; it is not proof that the invention is commercially viable, shipped, or safe at scale. [S1] supports the claim that Micron has protected a compute-in-memory sensing-circuit arrangement. It does not, by itself, support a claim that Micron’s production AI memories perform logic in the array.

Worked quantitative calculation

The core engineering question for memory-and-AI-compute silicon is: how much bandwidth does an HBM stack provide, and how does that translate into system bandwidth?

Formula

\[

\text{BW}_{\text{stack}} = \frac{\text{data rate per pin} \times \text{bus width in bits}}{8}

\]

Where:

• data rate per pin is in Gb/s,
• bus width is in bits,
• the division by 8 converts bits to bytes,
• the result is in GB/s.

Baseline HBM3 calculation using JEDEC JESD238

Inputs:

• HBM3 data rate: 6.4 Gb/s per pin [S3]
• HBM3 bus width: 8 channels × 128 bits = 1,024 bits [S3]

\[

\text{BW}_{\text{stack}} = \frac{6.4 \times 1{,}024}{8} = 819.2~\text{GB/s per stack}

\]

For an 8-stack system:

\[

819.2 \times 8 = 6{,}553.6~\text{GB/s} \approx 6.55~\text{TB/s}

\]

For a 12-stack system:

\[

819.2 \times 12 = 9{,}830.4~\text{GB/s} \approx 9.83~\text{TB/s}

\]

Sensitivity table

The table illustrates why supplier data-rate claims matter: moving from the JEDEC HBM3 baseline of 6.4 Gb/s [S3] to Micron’s marketed 9.2 Gb/s [S6] increases theoretical per-stack bandwidth from 819.2 GB/s to 1,177.6 GB/s, a 43.7% increase.

Cross-check against shipping system specifications

NVIDIA’s H200 product page states 141 GB of HBM3E memory and 4.8 TB/s memory bandwidth [S7]. AMD’s MI300X product page states 192 GB HBM3 and 5.3 TB/s memory bandwidth [S8].

For NVIDIA H200:

\[

\text{Implied per-stack bandwidth} = \frac{4.8~\text{TB/s}}{\text{number of stacks}}

\]

NVIDIA does not disclose the H200 stack count on its product page [S7]. This creates two plausible inferred cases:

• 6-stack case: \(4.8 / 6 = 0.8~\text{TB/s per stack}\). Converted to effective data rate:

\[

\frac{0.8~\text{TB/s} \times 8}{1{,}024} = 6.25~\text{Gb/s}

\]

This is close to the JEDEC HBM3 baseline of 6.4 Gb/s [S3], not the 9.2 Gb/s marketing rate [S6].

• 4-stack case: \(4.8 / 4 = 1.2~\text{TB/s per stack}\). Converted to effective data rate:

\[

\frac{1.2~\text{TB/s} \times 8}{1{,}024} = 9.375~\text{Gb/s}

\]

This is close to Micron’s marketed 9.2 Gb/s [S6].

The H200 capacity of 141 GB [S7] does not cleanly resolve the stack count: 141/24 = 5.875 and 141/36 = 3.92. Vendors disable some cells for repair/spare capacity, so raw capacity is not an exact multiple of stack count. This is a source-bound ambiguity, not a measurement error.

Takeaway: The widely quoted “4.8 TB/s” H200 figure is compatible with either a 4-stack HBM3E system running near Micron’s marketed 9.2 Gb/s, or a 6-stack system running near the HBM3 baseline data rate. The difference is material to suppliers, because it determines how much of the marketed speed actually reaches the system. This is a load-bearing discrepancy and should be resolved with a primary-source board-level specification, not vendor marketing pages. If only one source exists, it is [SINGLE-SOURCE].

Parameter table

Cross-reference against SEC filings and disclosed discrepancies

4.1 Micron’s regulatory filing is consolidated, not HBM-specific

Micron’s fiscal 2024 Form 10-K describes HBM as part of its premium DRAM portfolio and discusses technology transitions, capital intensity, and customer qualification risk in the risk factors and MD&A [S4]. However, the filing does not break out HBM revenue or HBM gross margin as a separate line item. Supplier press releases and earnings-call commentary often say that HBM is “sold out” or “ramping”; those statements are self-reported and may not appear in the annual report. The absence of a separate HBM line in [S4] is a disclosure gap for investors, not proof that HBM is immaterial.

4.2 NVIDIA’s product page vs NVIDIA’s 10-K

NVIDIA’s H200 product page gives specific HBM3E figures: 141 GB capacity and 4.8 TB/s bandwidth [S7]. NVIDIA’s fiscal 2025 10-K discusses supply-chain and component risks, but the filing itself is not a source for granular memory stack count or HBM supplier allocation [S5]. Investors often read the product page as if it resolved the stack-count question; it does not. The product page is a Tier 2 marketing source, not a Tier 1 measured or standards-based source.

4.3 The central discrepancy: peak marketing rate vs system-level bandwidth

Micron markets HBM3E at 9.2 Gb/s and >1.2 TB/s per stack [S6]. NVIDIA’s H200 specification of 4.8 TB/s across an undisclosed number of stacks means the effective per-stack bandwidth is either ~1.2 TB/s (4-stack assumption) or ~0.8 TB/s (6-stack assumption) [S7]. The first assumption matches Micron’s marketing; the second does not. This is a source-bound contradiction that can only be resolved by a primary board-level specification or a third-party teardown. It is not safe to state as fact that H200 uses six stacks or four stacks based on the product page alone.

4.4 Patent filings are not product proof

The granted Micron patent [S1] shows that Micron has claimed sensing-circuitry-based logical operations. But there is no primary-source evidence in this note that a commercial Micron AI memory product performs logic in the sense amplifiers at scale. A patent is a recorded claim, not a benchmark. Similarly, the Nature CIM paper [S2] demonstrates a research chip; it is not a commercial deployment.

Unit economics / business model

The economic structure of memory-and-AI-compute silicon is a multi-layer stack of value capture.

Who monetizes it

Memory suppliers monetize by selling HBM dies and stacks. HBM uses the same underlying DRAM process but adds TSV formation, stacking, known-good-die handling, high-precision thermal management, and extensive test. Micron’s HBM3E page presents the product as a premium memory device with >1.2 TB/s per stack [S6]. The fiscal 2024 Micron 10-K frames HBM within the broader DRAM portfolio and does not separately disclose HBM-specific margin [S4]. Any exact HBM gross margin must therefore be marked needs primary source validation.

Foundry/advanced packaging monetizes the interposer and package integration. TSMC’s CoWoS advanced-packaging technology is the standard route for placing HBM stacks beside a logic die [S10]. The value here is a service/wafer-level packaging business, not a per-bit memory sale.

AI accelerator vendors monetize the final system. NVIDIA sells the H200 as a GPU with HBM3E, and AMD sells the MI300X with HBM3 [S7][S8]. Their margin is on the full system, not on memory alone.

IP/EDA vendors monetize the memory-interface and advanced-packaging design chain. HBM PHYs, 2.5D/3D package design, signaling, and thermal models are licensed or sold as design tools. This note does not cite independent pricing for those IP blocks; specific royalty rates need primary source validation.

Typical margins

This note will not fabricate an HBM margin. The evidence can support only directional claims:

• HBM is a premium DRAM product, not commodity DDR, according to product positioning [S4][S6].
• HBM requires higher capital intensity and testing complexity than standard DRAM because of TSVs and stacking; this is visible in the technical requirements of the product category [S6][S10].
• The system-level margin for HBM-equipped accelerators is captured by the accelerator vendor, not by the memory supplier alone [S7][S8].

Any hard number—for example, “HBM gross margin is 60%” or “HBM is 25% of the GPU bill of materials”—is not supported by the sources retrieved here and should be treated as needs primary source validation.

Who captures the value

The value is captured by:

• The memory supplier with qualified capacity first. Qualification by accelerator vendors is the key gate. Micron’s HBM3E product page emphasizes the product; it does not disclose qualification dates [S6].
• The packaging/foundry partner with available advanced packaging capacity. TSMC’s CoWoS page positions CoWoS as the high-density integration technology for HBM-plus-logic systems [S10].
• The accelerator platform vendor that integrates HBM into a system with a performance advantage [S7][S8].
• The IP/EDA supplier that reduces design risk for HBM PHYs and advanced packages.

The business model is therefore not a single “chip sale”; it is a combination of memory-bit sales, packaging services, EDA/IP licensing, and accelerator system sales.

Disconfirming Evidence

The thesis that memory-and-AI-compute silicon remains a high-value, bandwidth-constrained, and patent-protected segment would be wrong if any of the following appear from primary sources:

1. HBM turns into commodity DRAM. If JEDEC-standard HBM pricing collapses to near DDR pricing, and supplier 10-Ks or earnings calls show HBM as a low-margin commodity, the premium-value thesis fails.

1. System bandwidth catches up without HBM. If a shipping accelerator uses on-chip SRAM, CXL-attached memory, or advanced packaging with conventional DRAM to achieve HBM-class bandwidth at lower cost, the HBM-specific bottleneck thesis weakens.

1. Compute-in-memory goes from research to shipping at expected economics. If a major supplier ships a CIM product with independent benchmark evidence showing large energy-latency gains, the patent-to-product gap described here closes; but that evidence would need to come from a primary benchmark, not a company blog.

1. Stack count and derating are resolved against the marketing claims. If an authoritative teardown or board-level spec shows that H200, for example, runs HBM3E at only ~6.25 Gb/s effective despite 9.2 Gb/s marketing [S6][S7], then the vendor data-rate premium is largely dissipated in the system.

1. Qualification risk and supplier concentration break. If accelerator vendors qualify multiple memory suppliers and capacity overshoots demand, the supplier moat erodes. This would appear first in company 10-K risk factors and capacity disclosures [S4][S5].