OpenAI Jalapeño Inference Chip: Architecture and Performance Analysis

OpenAI has announced "Jalapeño," a custom inference chip designed from the ground up for Large Language Model (LLM) inference. Developed in partnership with Broadcom, the chip represents an extremely aggressive development cycle, moving from initial team hiring to manufacturing tape-out in approximately 16 months starting in mid-2024.

Performance Benchmarks and Efficiency

Jalapeño demonstrates industry-leading performance-per-watt (perf/W), surpassing Nvidia Blackwell and even the newer Vera Rubin architecture in several scenarios. According to benchmarks conducted using the InferenceX suite, Jalapeño excels in both low-latency and high-throughput scenarios without requiring Multi Token Prediction (MTP) or speculative decoding.

Key Performance Metrics

  • Interactivity: On the DeepSeek R1 model, Jalapeño achieved over 700 tokens per second per user at concurrency 1.
  • Throughput: For models like GPT-OSS, the chip achieved approximately 1,400 tokens per second per user.
  • Energy Efficiency: Jalapeño's output token throughput per megawatt (MW) exceeds Vera Rubin's MTP results and significantly outperforms GB200's 2025 results.
  • TCO: On a performance-per-TCO basis, Jalapeño is currently head-to-head with Vera Rubin, though Jalapeño's results were obtained without speculative decoding, which could potentially reduce costs by an additional 3-5x once implemented.

Hardware Architecture

Jalapeño is a generalized inference chip rather than one specialized for a single model. It is designed to maximize HBM bandwidth and minimize memory movement of weights and KV caches.

Technical Specifications

  • Manufacturing: The B0 stepping is manufactured on TSMC's N3P node.
  • TDP: The chip has a Thermal Design Power (TDP) of 700W, which is lower than Rubin's 900-1,150W.
  • Memory: It utilizes HBM4, providing 15.4 TB/s of memory bandwidth per package, surpassing accelerators using HBM3E.
  • Compute: The B0 stepping delivers 13.4 PFLOPs of MXFP4 on a single reticle-sized compute die.
  • I/O: An N3E I/O chiplet provides 32 lanes of 800G SerDes, with 24 lanes used for local scale-up and 8 lanes for global scale-up.

Architectural Innovations

  • Unified Memory Pool: Unlike some competitors, OpenAI opted not to disaggregate prefill and decode (PD) across separate chip pools. This homogenous pool allows the system to remain flexible as workload mixes (input/output ratios) change over time.
  • Out-of-Order (OoO) Cores: Jalapeño uses OoO cores with L1 caches instead of software-managed scratchpads. This reduces fixed overheads like barrier latencies, allowing the chip to get closer to its theoretical hardware roofline.
  • Matrix Engine: The matrix engine uses MXFP numerical formats and a weight-stationary systolic array that supports smaller shapes and dimensions to avoid performance cliffs common in larger systolic arrays.

Software Stack and Kernel Development

OpenAI has bypassed traditional universal compilers in favor of a high-performance, low-level programming approach.

Gluon and Linear Layouts

OpenAI uses a kernel programming language called Gluon, built on top of Triton. Gluon exposes low-level abstractions and introduces "Linear Layouts," a mathematical formalization of the mapping between hardware resources and tensor elements. This allows for provably correct layout conversions and optimal memory swizzling.

AI-Assisted Kernel Generation

OpenAI utilizes an internal version of Codex to write hand-tuned kernels (some reaching 3,000 lines) as assembly. This AI-driven approach allowed the team to bring up complex kernels, such as MLA kernels for DeepSeek, rapidly and without human intervention.

System-Level Integration

Jalapeño is deployed in a two-rack system consisting of a host rack ("Katsu") and an ASIC rack ("Vindaloo").

  • Scale-Up Network: The system supports a local domain of 128 ASICs per rack and a global domain connecting up to 2,048 ASICs across 16 racks.
  • Interconnects: Local scale-up uses 102.4T Tomahawk 6 switch ASICs via passive copper cables. Global scale-up utilizes a hybrid of copper backplanes and optical circuit switches (OCS) with 1.6T transceivers.
  • Power: A total two-rack system draws approximately 160kW.

Community and Critical Perspectives

While the reported benchmarks are impressive, industry observers and technical commenters have raised several caveats:

"Everyone's silicon beats everyone else's benchmarks until it has to run someone's actual production workload. The real test is six months of your own inference traffic, not a vendor's chart."

Critics have pointed out that the results are based on engineering samples (A0 stepping) and 8k1k workloads, which are simpler to tune for than the multi-turn, long-context workloads found in production agentic workflows. There are also concerns regarding the validity of the comparisons, noting that Jalapeño's performance-per-watt is compared against chips that may be running at different precision levels or TDP limits.

Sources

Related

  • Dispatch
    OpenAI Jalapeño Inference Chip

    OpenAI has unveiled Jalapeño, a custom inference processor developed with Broadcom to reduce reliance on Nvidia GPUs and lower the cost of running AI models.

  • Dispatch
    OpenAI Jalapeño Inference Chip First Results

    OpenAI has introduced Jalapeño, a custom inference chip that delivers 1.5 to 1.9 times more AI work per watt and up to 3.6 times lower latency than existing systems across multiple large-scale models.

  • Dispatch
    OpenAI Announces Jalapeño Custom Inference Chip and Full Stack Strategy

    OpenAI unveiled Jalapeño, its first custom inference chip, demonstrating higher throughput per kilowatt and lower latency on GPT‑OSS 120B, and outlined a full‑stack compute strategy that integrates hardware, software, models, and infrastructure to drive compounding efficiency gains.

  • Dispatch
    OpenAI and Broadcom Unveil Jalapeño LLM Inference Chip

    OpenAI and Broadcom have introduced Jalapeño, a custom-designed AI accelerator optimized specifically for LLM inference to improve performance per watt and reduce compute costs.

  • Dispatch
    OpenAI gpt-oss Release Notes

    OpenAI has released gpt-oss-120b and gpt-oss-20b, two open-weight reasoning models licensed under Apache 2.0 that deliver frontier-level reasoning and tool-use capabilities optimized for consumer hardware.