Why x86‑TSO Emulation Is a Performance Bottleneck on ARM and How Vendors Are Tackling It

TL;DR

Emulating x86‑TSO on ARM’s weak memory model is costly because every load/store must be turned into acquire/release instructions, unaligned accesses trigger costly alignment‑fault handling, and split‑lock atomics require heavyweight kernel mediation; recent ARM extensions (LRCPC, FEAT_LSE2) and Apple’s hardware TSO mode mitigate most of these penalties, enabling near‑native performance on modern ARM CPUs.

What is x86‑TSO and why does it matter?

  • x86‑TSO (Total Store Ordering) guarantees that a store becomes visible to all cores before any later load can observe it. This gives programmers a simple, intuitive model: a write is instantly coherent.
  • ARM’s weak (relaxed) model allows stores to stay in a private cache line and loads may see stale data unless explicit acquire/release instructions are used.
  • When an x86 emulator (e.g., FEX) runs on ARM, it must translate every x86 load into an ARM load‑acquire and every x86 store into an ARM store‑release to preserve TSO semantics.
  • This translation is far more restrictive than ARM’s native model and can dominate runtime performance.

Baseline performance with acquire/release semantics

  • FEX’s early implementation used ARMv8.0‑a’s acquire/load and release/store for all memory operations.
  • Micro‑benchmarks show a 2‑5× slowdown on most CPUs because acquire/release instructions cannot be freely reordered and become the hot path.
  • The AmpereOne server CPU exhibits the worst slowdown: its release‑store unit is especially weak, and the sheer volume of acquire‑loads throttles the pipeline.
  • Even high‑end mobile cores (Cortex‑X4, X925) suffer ~50 % loss on unaligned accesses due to the extra DMB (data‑memory‑barrier) needed when alignment faults occur.

Unaligned accesses and split‑locks

Alignment faults on ARM

  • ARMv8.0 requires natural alignment (address ≡ size). Violating this raises an alignment fault and normally crashes.
  • FEX inserts a patchpoint (a NOP) before each load/store. On fault it patches the instruction to a plain load/store wrapped in a DMB and resumes execution.
  • This fallback works but adds a kernel‑to‑userspace round‑trip for every misaligned access, which is expensive.

Split‑locks on x86

  • x86 permits split‑lock operations: atomic reads/writes that cross a cache‑line boundary. They remain atomic and coherent, never tearing.
  • ARM’s atomic instructions are only single‑copy atomic for naturally aligned accesses. The FEAT_LSE2 extension relaxes this to 16‑byte granules, but still fails for full 64‑byte cache‑line crossings.
  • Consequently, FEX must emulate split‑locks via a sequence of two 64‑bit compare‑exchange (CAS) operations and a kernel‑mediated fault handler. This incurs hundreds of nanoseconds per operation, far slower than native x86 (~1.4 ns).

Architectural extensions that help

LRCPC (Release Consistency processor‑consistent) – ARMv8.3+

  • Introduces LRCPC‑load and LRCPC‑store instructions that implement a memory model matching x86‑TSO requirements.
  • Benchmarks show LRCPC‑loads achieving baseline load performance on all tested CPUs (AmpereOne, Cortex‑X4/X925, Oryon‑3, Apple M1).
  • FEX detects the FEAT_LRCPC feature and switches from acquire‑load to LRCPC‑load, eliminating the costly DMB path for aligned accesses.

FEAT_LSE2 (Large System Extensions 2)

  • Loosens alignment requirements for acquire/LRCPC/Release instructions and for read‑modify‑write atomics to any address within a 16‑byte granule.
  • Provides only a marginal gain for x86 emulation because most x86 code accesses data across full 64‑byte cache lines (true split‑locks).

Apple’s hardware TSO mode

  • Apple Silicon implements a thread‑wide TSO toggle (FEAT_TSO). When enabled, regular ARM loads/stores behave like x86‑TSO.
  • On M1, LRCPC‑loads are simply aliases of normal loads, so both aligned and unaligned accesses have ≤5 % overhead.
  • The trade‑off is a modest (~24 % for stores) slowdown for native ARM code, but the impact is negligible because FEX’s native‑ARM execution fraction is tiny.

Vendor‑specific performance snapshots

CPU Aligned load (GB/s) Unaligned load penalty Aligned store (GB/s) Unaligned store penalty
AmpereOne ~28 (baseline) ≈0 % (alignment handled) ~28 ~8.5 % drop
Cortex‑X4 11.5 ~50 % drop (DMB) 6.7 ~50 % drop
Cortex‑X925 14.2 ~50 % drop 9.1 ~50 % drop
Oryon‑3 18.0 (LRCPC‑load ≈ baseline) ~30 % drop 12.2 (≈68 % of baseline) ~57 % drop
Apple M1 (TSO) ≈baseline ≤5 % ≈76 % of baseline (still fast) negligible

The numbers are derived from the article’s micro‑benchmark graphs; exact values are omitted for brevity.

Uncached (write‑combine) memory challenges

  • Games often write directly to GPU buffers via VK_MEMORY_HOST_CACHED_BIT = 0 (uncached, write‑combine) memory.
  • On ARM, uncached stores still use store‑release, which does not benefit from write‑combine buffering, leading to up to 816× lower bandwidth compared with cached stores.
  • Load bandwidth suffers even more because every load must bypass caches and hit system memory.
  • The result: on PCIe‑GPU systems, emulated x86 games can drop to <1 FPS (e.g., Hollow Knight: Silksong, Subnautica 2).
  • Work‑arounds:
    • On UMA systems (integrated GPUs) force the driver to use cached buffers; NVIDIA Tegra, Snapdragon Adreno, and many Mali GPUs already do this.
    • Apple’s TSO mode automatically avoids the problem because the hardware treats uncached stores as coherent.
    • No current ARM extension solves the write‑combine store issue; a future FEAT_LRCPC4 would be needed.

Split‑lock emulation correctness

  • FEX’s current split‑lock emulation is best‑effort: it may tear data when a non‑split‑lock atomic modifies only one half of the 128‑bit region.
  • A naïve global‑mutex solution would serialize all split‑locks but still allow tears from concurrent aligned atomics.
  • A kernel‑wide “stop‑the‑world” approach is infeasible due to the high frequency of split‑locks in games.
  • The authors propose a 128‑bit CASP that straddles the cache‑line boundary without raising an alignment fault; the operation can fail and be retried, preserving atomicity.
  • ARM’s Transactional Memory Extension (TME) could theoretically help but has been deprecated and never shipped.

Community insights

"Apple solved this problem six years ago by simply adding an x86‑compatible memory ordering mode to their chip when x86 emulation became important. Yet another way Apple’s chips lead the industry." – modeless

"FEX is a translation framework for x86 to ARM much like Apple’s Rosetta2 and Microsoft’s Prism. Valve sponsors development as it’s also the way the new Steam Frame supports x86 games." – dagmx

"The article repeats the common assertion that ARM is the most relaxed and x86 the most strict, but a relaxed model doesn’t necessarily have much benefit." – pdw

These comments reinforce the article’s core claim: hardware‑level support (Apple’s TSO, ARM LRCPC) is the only practical path to high‑performance x86 emulation on ARM.

Outlook

  • ARM vendors are progressively adding LRCPC and LSE2 extensions, narrowing the gap to x86‑TSO.
  • Apple’s hardware TSO mode demonstrates that a dedicated TSO toggle is the cleanest solution.
  • Future proposals such as a split‑lock‑aware 128‑bit CASP or a revived transactional‑memory extension could finally eliminate the remaining correctness and performance gaps.
  • Until such extensions become mainstream, x86 emulators on ARM will continue to favor UMA platforms and rely on driver patches (e.g., Valve’s Steam Frame kernel patch) to mitigate uncached‑memory penalties.

Bottom line: Emulating x86‑TSO on ARM is fundamentally expensive because the two memory models are at opposite ends of the consistency spectrum; however, recent ARM ISA extensions and Apple’s hardware TSO mode dramatically reduce the overhead, making high‑performance x86 gaming on ARM increasingly viable.

Sources

Related