Samsung LPDDR5X-PIM: Processing-in-Memory Architecture and Implementation

Samsung's Processing-in-Memory (PIM) implementation for LPDDR5X chips allows compute operations to occur directly within the memory, bypassing the external bus bottleneck to achieve internal bandwidth of 614 GB/s. This architecture is designed to accelerate AI and machine learning workloads by placing multiply-accumulate (MAC) units inside DRAM banks, enabling the chip to function as a constrained SIMD processor while maintaining compatibility with standard memory controllers.

Hardware Architecture and Throughput

Samsung's LPDDR5X-PIM integrates a PIM block into each of the 16 banks of an LPDDR5X-9600 chip. By accessing DRAM banks internally, these blocks avoid the limitations of the chip's external interface, which typically maxes out at 76.8 GB/s.

MAC Unit Specifications

Each PIM block contains a MAC tree, register files, and control logic:

  • Instruction Register File: 1024-bit, holding up to 64 16-bit instructions.
  • Source Register File: 4 kbit for activation vectors.
  • Scale Register: 2 kbit for scaling model weights before computation.
  • Data Flow: Model weights are stored in the attached DRAM block, while activation vectors are supplied via the source register.

Compute Performance

The MAC array supports low-precision formats including INT8 and FP8. A single PIM block can sustain four INT8 or FP8 MAC operations per data clock (eight per cycle). With 4-bit input weights, throughput doubles, resulting in a package-wide compute throughput of 2.4 TOPS. To match the performance of a typical NPU (such as the one in Intel's Meteor Lake), a system would require approximately eight LPDDR5X-PIM chips, totaling 128 GB of system memory.

Protocol Integration and Control

LPDDR5X-PIM remains compatible with the standard LPDDR5X protocol by repurposing specific row addresses as Memory-Mapped I/O (MMIO) triggers to switch between operational modes.

Mode Switching

  • Single-Bank Mode: The standard operating mode for regular DRAM access.
  • Multi-Bank Mode: Applies commands across all 16 banks simultaneously to exploit internal bandwidth.
  • PIM Registers Activated Mode: Triggered by special per-bank rows, this mode redirects read and write commands to PIM registers instead of DRAM bank contents.

SIMD Execution Flow

In multi-bank mode, the chip operates as a SIMD processor. A single write command is broadcast to all 16 banks, ensuring the same operation, scale factor, and source operand are applied across the entire chip. To prevent issues with memory controller reordering, Samsung utilizes an Address Align Mode (AAM), which allows instructions to infer their source register index from the column address being accessed.

Critical Software and System Challenges

Despite the hardware efficiency, integrating PIM into modern computing environments introduces severe architectural conflicts, particularly regarding how CPUs handle memory.

Cache Coherency and Speculation

Because PIM operations modify memory contents and register states internally, they break the fundamental assumptions of the CPU cache hierarchy:

  • Uncacheable Memory: Samsung recommends mapping PIM memory as uncacheable. This eliminates the benefit of CPU caches, significantly increasing latency and stalling cores.
  • Speculative Execution: Modern CPUs use prefetchers and branch prediction to load data speculatively. In a PIM system, a speculative read is not a passive operation—it triggers a computation that modifies the PIM Vector Register File (VRF), potentially corrupting the state of the actual program.

Multitasking and OS Integration

PIM mode switching is a global state change for the memory channel. This creates significant hurdles for multitasking operating systems:

  • Thread Isolation: If one thread uses PIM while another performs a regular memory access, the non-PIM thread could inadvertently trigger a PIM computation or write VRF data to the wrong address.
  • Context Switching: Preempting a PIM thread requires the OS to switch the channel out of PIM mode and manually save the state of instruction, source, scale, and vector register files across all banks.
  • Memory Interleaving: To isolate PIM regions, systems may need to disable address interleaving across channels, which reduces overall memory bandwidth for non-PIM applications.

Technical Perspectives and Alternatives

Industry discussion highlights that while PIM addresses the "Von Neumann bottleneck," it imposes strict constraints on data placement. As noted by community contributors:

"The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well."

To resolve these issues, the author suggests that a more viable path would require systemic hardware changes, such as expanding the DRAM interface with dedicated compute commands, implementing a memory controller that acts as a peer CPU core for cache coherency (using Read-For-Ownership requests), and introducing transparent CPU instructions (e.g., a hypothetical rep macb) to abstract the PIM hardware from the software layer.

Sources

Related