llama-modchip v2.4.0: Running a 284B DeepSeek MoE on a 64GB CPU-Only Machine

llama-modchip v2.4.0 is a patched llama.cpp fork that claims to run the 284.33B-parameter DeepSeek-V4-Flash mixture-of-experts model entirely in RAM on a 64 GB CPU-only machine, at 256K context and roughly 9–15 tokens per second, with zero GPUs. The project's headline claim is that a model whose standard IQ2_XXS GGUF is 76.87 GiB — which out-of-memory crashes on a 64 GB Linux box — has been shrunk to 56.56 GiB on disk and about 57.61–58.9 GiB of resident memory. The Hacker News discussion (387 points, 172 comments) split sharply between people impressed by the engineering and people skeptical that a 2-bit-class model is worth running at all.

What llama-modchip v2.4.0 claims to do

llama-modchip is distributed as a unified CLI (./modchip) plus a set of patches against llama.cpp. Its three advertised pillars are a 284B frontier MoE in a 64 GB CPU box, a 30B MoE running at GPU-like speeds on pure CPU, and a zero-API Claude Code integration.

The project's own summary table maps hardware to recommended models:

Machine	RAM	Recommended model	Claimed speed (0 GPUs)
Developer laptop / mini-PC	16–32 GB	Nemotron-3.5-30B-A3B (14.35 GiB)	25.0–60.4 tok/s
64 GB desktop / server	64 GB	DeepSeek-V4-Flash 284B (56.56 GiB)	44.4 tok/s prefill, 9.1–14.7 tok/s decode
Hybrid desktop (CPU + 16 GB GPU)	32–64 GB	Nemotron-30B / Qwen3-30B	80.80 tok/s
8-channel workstation (Threadripper)	128 GB+	DeepSeek-284B / Nemotron-30B	111.62 tok/s (30B)

All of these figures are self-reported by the project and published in its README and benchmark tables.

How a 284B model fits in 64 GB of RAM

The core mechanism is aggressive quantization combined with a custom tensor format. llama-modchip introduces a block-64, BF16-scale, Gray-coded tensor type it calls Fractal-64 at 3.75 bits per weight, plus a Fractal-MXFP4 variant at 4.25 bits per weight. The project also describes an "Analytical SwiGLU-H2 Second-Order Hessian Compression" step and an "Exact SVD of wo_b * wo_a" reduction that it says retains 99.82% cosine similarity at 25% rank reduction.

The DeepSeek-V4-Flash build is reported at 56.56 GiB on disk, a reduction of about 20.31 GiB from the 76.87 GiB IQ2_XXS baseline. At full 256K context the project reports peak VmRSS of 57.61 GiB, with 58.9 GiB cited elsewhere in the same README.

A critical compatibility caveat: the Fractal-64 models use a custom GGML type ID (GGML_TYPE_Q2_0 = 40) and custom deepseek4 / Stockfish-PVS kernels. The README states plainly that opening these files in unpatched llama.cpp, Ollama, or LM Studio will fail with unknown tensor type 40. Users must run the project's own binary or patched build.

The speculative decoding layer: Stockfish-PVS and RouteCrack

llama-modchip's decode-speed gains lean heavily on a small draft model. The project bundles a 1.64 MB Stockfish-PVS v2 drafter with a RouteCrack component, described as an L2/L3 Zobrist transposition table, and reports 96.4% acceptance on the Nemotron-30B model and 94.5% on DeepSeek-284B.

On DeepSeek-V4-Flash the project reports pure CPU decode of 9.06–10.74 tok/s, rising to 11.5–14.7 tok/s with the PVS drafter enabled. On Nemotron-30B it reports base decode of 25.02 tok/s and 48.39 average / 60.40 peak tok/s with the drafter.

Claude Code integration and warm KV restore

The project ships a ./modchip claude command that wires the Claude Code CLI, with all 40 tools active, to the local engine. The stated problem it solves is the roughly 45-second prompt re-evaluation that a CPU agent hits on every turn. llama-modchip addresses this with a /dev/shm Slot-0 KV warm restore, reported at 18.9 ms on Nemotron-30B and 13.1–32.6 ms on DeepSeek-284B.

Supported hardware

The README lists architecture-specific kernels for AMD Zen 5 (AVX-512 with VPDPBUSD, VPRORVQ, VBMI, BF16), Zen 4 multi-CCD Threadripper and EPYC (dynamic L3 CCD topology discovery, a deterministic 4-cores-per-CCD stride that the project says is 54% faster than -t 64), AMD 3D V-Cache plus Radeon Vulkan hybrids, Intel/NVIDIA Windows hybrids, Apple Silicon via Metal, ARM64 cloud (AWS Graviton3 with SDOT and 256-bit SVE), and a Fedora mini-PC configuration.

Community-submitted leaderboard entries in the README include a Threadripper PRO 7985WX at 111.62 tok/s warm decode on Nemotron-30B, an Apple M4 Pro at 47–48 tok/s, an AWS Graviton3 m7g.2xlarge at 44.97 tok/s, and an AVX2-only EPYC 7402P at 27.29 tok/s.

The central objection: what does 2-bit quantization cost in quality?

The most substantive criticism in the Hacker News thread concerns quantization quality, not speed. Commenter @armcat argued that the relationship between bit-width and model quality is non-linear and that the sweet spot is generally around 4 bits:

"Something to keep in mind with aggressive quantisation, is that it's been empirically proven that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance."

@armcat cited "Reliability Scaling Laws for Quantized Large Language Models" (TUM) and added that in two years of work on speech models, anything below 8-bit was a large quality drop — noting that quantization behavior is not identical across modalities.

@walrus01 was more direct, arguing that shrinking a model this far defeats the purpose:

"Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache... Running a lobotomized model just to say you're doing it is really not accomplishing anything special."

@throw10920 framed the genre succinctly: "'2-bit quant' is the 'in mice' of 'look at how big a model I can fit on this hardware'." @GypsyKing716 made a similar point: "Just because you CAN load it, doesn't mean you SHOULD load it."

Several commenters said the quantization level was not obvious from the README at all. @akie wrote: "At which quantization? That's the only thing I want to know." @boredatoms: "The quantization is probably the most important thing to know and its not clear at all from the dense word salad."

The second objection: does it actually outperform plain llama.cpp?

@aappleby tested the project on a Ryzen 7945HX (16-core Zen 4, 96 GB DDR5) and reported that on Nemotron the performance was roughly the same as stock llama.cpp on CPU, with the exception of prefill:

"Stupid question, but does this AI-generated repo actually do anything? Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in 'modchip setup', so I haven't tested that one yet."

A counterpoint came from @sparse-Matrix, who reported the project working on much weaker hardware: "It works pretty good on my Ryzen 7 5700U/64GB DDR4 Steady 12 to 14 TPS... fast enough to get meaningful personal work done, with a power consumption of 15w."

@robotswantdata raised a separate performance concern: "Prefill of this approach (prompt processing) is far too slow."

The README readability problem

A recurring theme in the comments was the document itself. Multiple readers described the README as AI-generated and hard to parse. @superkickstart: "I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible." @intoXbox called it "quite an obnoxious Readme.md with AI slop" and noted it "wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing." @gtirloni asked: "Do people remember humans read README.md files too or are we all just asking for the TL;DR; from AI slop by default?"

The fork ecosystem problem in llama.cpp

@SillyUsername placed llama-modchip in a broader pattern of specialized llama.cpp forks — naming Bonsai Qwen 3.8 27B, Strata, and Pulsar — and argued the upstream review process is the bottleneck:

"I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers. I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious."

What the discussion says about the direction of local inference

Several commenters read the project as evidence that the hardware floor for usable local inference keeps falling. @kennywinker described a sub-$800 build: "An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant."

@antirez offered a higher-end data point: with 128 GB of unified RAM the model "can run even fast... Like, coding-agent acceptably fast," citing prefill around 300 tok/s.

@nickdothutton drew a historical parallel: "A lot of the 1980s demoscene guys came from countries where hardware was scarce and expensive, and it drove a lot of technical creativity." @aussiegreenie put the economics at roughly "$10 per person per month for enough processing power for useful work with data sovereignty."

Practical caveats before trying it

• The 284B DeepSeek numbers are self-reported by the project author and have not been independently replicated in the thread.
• The Fractal-64 GGUF files require the project's patched binary; stock llama.cpp, Ollama, and LM Studio will reject them with unknown tensor type 40.
• Quality at 3.75 bits per weight is the open question. The README reports a "100/100 Quality Gate" on Nemotron and cosine similarity of 0.99126–1.00000 across DeepSeek profiles, but independent commenters remain skeptical that these metrics capture real-world degradation.
• At least one tester (@aappleby) found no meaningful speedup over stock llama.cpp on Nemotron beyond prefill, while another (@sparse-Matrix) reported usable throughput on older hardware.
• The project is Apache 2.0 licensed and includes an arXiv-style preprint PDF in the paper/ directory.