Anubis WebAssembly and the Challenges of Reproducible Builds

Anubis is introducing WebAssembly-based proof of work (PoW) checks to allow administrators to protect websites using non-SHA256 methods. To ensure consistency between client and server, the check logic is defined in a single location and executed in lockstep. To maintain accessibility for users who have WebAssembly disabled, Anubis recompiles the WebAssembly logic into JavaScript using wasm2js from the binaryen project, ensuring the PoW check can still complete, albeit more slowly.

The Difficulty of Byte-for-Byte Reproducibility

Achieving reproducible builds—where the same input bytes consistently produce the same output bytes—is unexpectedly difficult in C/C++ development. Even when compiler flags and targets are controlled, several factors can introduce non-determinism.

Built-in Macros and Timestamps

One common source of non-determinism is the use of built-in macros such as __DATE__ and __TIME__. These macros stamp the binary with the exact time of execution, ensuring that every build produces a different output even if the source code remains identical.

Implicit Toolchain Dependencies

Compilers often rely on external tools found in the system's $PATH. For example, Clang may silently shell out to wasm-opt (part of the binaryen project) to optimize WebAssembly output. This creates a dependency on the specific version of wasm-opt installed on the host machine. In the case of Anubis, older versions of wasm-opt on certain ARM64 machines failed when encountering WebAssembly Exception handling instructions, which are required by wasi-sdk and binaryen. This was resolved by passing the --no-wasm-opt flag during the linking step to remove the external dependency.

Address-Sensitive Code Generation in Clang

Even after removing external tool dependencies, Clang can produce non-deterministic output due to address-sensitive code generation in its exception handling path. Raw pointer values can leak into the ordering of try_table blocks, causing builds to differ by several bytes across different executions or different CPU architectures (e.g., x86_64 vs. ARM64).

Mitigation Strategies

To achieve determinism within a specific architecture, the following steps were implemented:

1. Disabling ASLR: Using setarch --addr-no-randomize to disable address-space randomization during the build process.
2. Architecture-Specific Checksums: Creating known-good SHA256 checksums for both x86_64 and ARM64 by building on trusted machines.
3. CI Verification: Implementing a CI job that rebuilds the modules and verifies them against the recorded checksums for the respective runner architecture.

Community Perspectives on Determinism and PoW

Technical discussions surrounding these implementation choices highlight a divide between high-level software expectations and low-level engineering realities.

On Compiler Determinism

Some contributors argue that non-deterministic output is a bug that should be fixed upstream in LLVM, specifically by replacing non-deterministic iteration (such as over a DenseMap) with deterministic alternatives like MapVector.

"If Clang generated non-deterministic output due to pointer addresses then that's a bug... The common way to fix that is to switch to a MapVector which pays some additional runtime/memory cost to guarantee deterministic iteration order."

Others suggest that tools like Nix solve these issues by providing a hermetic build environment that catches system calls for time and replaces them with constants (e.g., epoch 0).

On the Use of Proof of Work

The use of PoW for website protection is controversial, with some critics comparing it to "planting a crypto miner" due to the energy consumption required by the client's browser.

"So to avoid those energy-hungry LLM companies from scraping your website, you force each browser to compute a lot of hashes in a necessarily energy-hungry loop, creating, at the same time, all the kind of accessibility problems?"

Additionally, some suggest that PoW could be made more useful by providing a rewarding output, such as contributing to protein folding or a cryptocurrency, rather than performing "useless" computation.