Why x86‑TSO Emulation Is a Performance Bottleneck on ARM and How FEX Tackles It

The Core Problem: Emulating x86‑TSO on a Weakly‑Ordered ARM ISA

Emulating the x86 Total Store Ordering (TSO) memory model on ARM’s weak ordering model incurs large performance penalties because every x86 load must be turned into an ARM load‑acquire and every store into a store‑release. These instructions are far more expensive than regular loads/stores and were never intended to be the dominant instruction class.

Why x86‑TSO Matters for Applications

x86‑TSO guarantees that a store becomes visible to all cores before any later load can observe it, letting programmers reason about memory without extra fences. This strict coherence is baked into the x86 ISA and is relied upon by virtually all PC games and many legacy applications.

ARM’s Weak Consistency Model

ARM’s default loads and stores are not immediately coherent; a store may stay in a private cache line until it is explicitly flushed, and a load may see stale data. To obtain ordering, ARM provides load‑acquire and store‑release instructions (the “RCsc” model) that act as lightweight barriers.

Initial Emulation Strategy: Acquire/Release Everywhere

FEX initially mapped every x86 load to an ARM load‑acquire and every store to a store‑release, which is functionally correct but extremely costly. Micro‑benchmarks on five CPUs showed up to a 50 % slowdown for acquire‑loads and dramatically lower throughput for release‑stores on several cores (e.g., AmpereOne).

The Split‑Lock Challenge

x86 programs frequently perform unaligned accesses that cross cache‑line boundaries (“split‑locks”), which are atomic on x86 but cause alignment faults on ARM. ARM requires natural alignment for acquire/release instructions; a misaligned access triggers a fault, forcing FEX to patch the JIT at runtime, insert a data‑memory‑barrier (DMB), and retry.

"When a alignment fault occurs as one of these patchpoints, FEX will capture the fault, patch the code from a load‑acquire/store‑release instruction to a basic equivalent load‑store, and wraps the instruction in a data memory barrier." – FEX article

The fault‑handling path involves a kernel‑to‑userspace round‑trip for every split‑lock, which can be thousands of times slower than native execution.

ARM Extensions That Help

LRCPC (FEAT_LRCPC, LRCPC2, LRCPC3)

The LRCPC family adds “Release Consistency processor‑consistent” load instructions that match x86‑TSO semantics without the heavy acquire‑load penalty. Benchmarks show LRCPC‑loads achieving near‑baseline throughput on most CPUs.

FEAT_LSE2

FEAT_LSE2 relaxes alignment requirements for acquire/LRCPC/Release instructions to a 16‑byte granule, but still faults on 64‑byte (cache‑line) crossings. This yields only marginal gains for split‑locks because most x86 atomic operations span an entire 64‑byte cache line.

Apple’s Hardware TSO Mode – The Ideal Solution

Apple Silicon implements a hardware toggle that makes regular ARM loads/stores obey x86‑TSO semantics, eliminating the need for acquire/release or LRCPC instructions. On M1, aligned and unaligned accesses have virtually identical performance, and the only overhead is a modest 5 % slowdown for stores when the TSO mode is enabled.

"Apple’s hardware directly added support for the x86‑TSO memory model. When the CPU feature is toggled, their regular load/store ARM instructions change behaviour to match what x86 requires." – FEX article

When FEX detects this feature (e.g., via Asahi Linux), it can enable the hardware TSO mode and gain the same "free" performance boost.

Atomic RMW Instructions: Mapping x86 LOCKs to ARM

ARMv8.1‑a provides one‑to‑one equivalents for all 18 x86 LOCK‑prefixed RMW operations (e.g., LOCK ADD → ldaddal). However, the same alignment‑fault problem applies: unaligned atomic RMWs that cross a cache line still require the costly fault‑handling path.

Uncached (Write‑Combine) Memory – A Show‑Stopper

When games use Vulkan’s VK_MEMORY_HOST_CACHED_BIT cleared (write‑combine buffers), ARM’s LRCPC loads become dramatically slower, and stores can be up to 816× slower than x86. This is because uncached memory forces every load to go to system memory, and LRCPC adds extra barriers.

"The worst case out of all of this is just how badly the store performance is, compared to the performance that Zen gets on the stores, this is basically a showstopper. Up to 816x worse bandwidth!" – FEX article

FEX mitigates the issue on UMA systems by forcing the driver to use cached buffers, but on PCIe‑GPU systems the problem remains unsolved.

Vendor‑Specific Findings

CPU	Aligned Load/Store	Unaligned Penalty (Load)	Unaligned Penalty (Store)
AmpereOne	Baseline ≈ 28 GB/s store	≈ 0 % (load)	~8.5 % slowdown (store)
Cortex‑X4	11.5 GB/s load, 6.7 GB/s store	~50 % slowdown (both)
Cortex‑X925	Similar to X4, higher absolute bandwidth
Oryon‑3	Aligned LRCPC‑load matches baseline; store ≈ 68 % of baseline; unaligned store ≈ 43 %
Apple M1	< 5 % slowdown for stores; loads unchanged – hardware TSO eliminates alignment overhead

Proposed Future Hardware Support for Split‑Locks

A feasible hardware fix would be a 128‑bit CASP that can atomically operate across a 64‑byte boundary without raising an alignment fault. The operation could fail and be retried, leveraging ARM’s LL/SC model to guarantee forward progress.

Takeaways for Developers and Architects

Emulating x86‑TSO on ARM without hardware support is expensive – acquire/release instructions and fault‑handling for split‑locks dominate the overhead.
LRCPC extensions dramatically improve baseline load/store performance but do not solve unaligned atomic or write‑combine cases.
Apple’s hardware TSO mode demonstrates the optimal path: a single toggle that makes regular loads/stores obey x86‑TSO, yielding near‑native performance.
Uncached (write‑combine) memory remains a critical bottleneck for PCIe‑GPU systems; driver‑level workarounds are the only practical mitigation today.
Future ARM extensions (e.g., a split‑lock‑aware CASP) could close the remaining performance gap for atomic operations that cross cache lines.

Community Reactions

"Apple solved this problem six years ago by simply adding an x86‑compatible memory ordering mode to their chip when x86 emulation became important. Yet another way Apple's chips lead the industry." – @modeless

"FEX is amazing; most problems I run into are anti‑cheat related, but the emulator itself is impressively fast given the architectural hurdles." – @sureglymop

"The article repeats the common assertion that ARM is the most relaxed and x86 the most strict; a relaxed model doesn’t necessarily have much benefit." – @pdw (link to external discussion)

Conclusion

Emulating x86‑TSO on ARM is fundamentally hard because the two ISAs enforce opposite memory ordering guarantees; the current best software approach (acquire/release → LRCPC → hardware TSO) yields acceptable performance on most workloads, but split‑locks and uncached memory still cause severe slowdowns. Continued ARM ISA extensions and broader adoption of hardware TSO modes—like Apple’s—are the most promising path to seamless, high‑performance x86 emulation on future ARM platforms.