Memdoor team-solve: Five Qwen3.8-27B Agents Match Fable 5 on LiveCodeBench Hard

Memdoor's team-solve workflow reproduces the GVS5H multi-agent method on Qwen3.8-27B and scores a mean of 9.8 out of 10 across five passes on the first ten LiveCodeBench Hard problems, compared with 9.6 for the original authors' harness and 8.0 for Fable 5 answering alone. The result matters because it shows a small open-weight model, wrapped in a disciplined multi-agent harness, reaching parity with a frontier model on a hard coding benchmark — at roughly $0.39 and a median of 40 minutes of model time per run.

The workflow is a public example in the memdoor repository and was announced on Hacker News, where it drew 385 points and 188 comments. The discussion split between enthusiasm for local-model parity and skepticism about a ten-problem sample.

What team-solve actually is

team-solve is a multi-agent harness for programming problems, written as a Memdoor workflow. Five roles — a planner, two ideators, a solver, and a manager — plus a repair worker each start from a fresh context and communicate only through files on disk. A shell script, not a model, decides whether the run succeeded.

The method comes from GVS5H (Lee, 2026), a paper reporting that five Qwen3.8-27B models match Fable 5 on LiveCodeBench Hard. Memdoor's contribution is a reimplementation of that loop inside its own workflow engine, using the authors' exact prompts.

The workflow files were generated by the coder itself from a description of the method on 2026-10-25; nobody hand-wrote them.

The benchmark numbers

Memdoor rebuilt GVS5H's harness as a Memdoor workflow on the same model — Qwen3.8-27B, a 128K reply cap, the authors' prompts and loop — and ran the first ten problems of their LiveCodeBench Hard list for five passes. Every program was graded on every public and hidden test with the authors' corrected grader.

pass	Memdoor workflow	authors' 27B harness	Fable 5 alone
1	10 / 10	9	7
2	10 / 10	10	8
3	10 / 10	10	10
4	10 / 10	9	8
5	9 / 10	10	7
mean	9.8	9.6	8.0

The authors' columns come from their repository's regraded result files (q38_multiagent_pN, fable5_single_pN) and match the numbers in their paper. All 50 runs are published in bench/results/parity-5pass/, including every request and reply, the generated program, the grader's full output, each call's cost and serving host, and every attempt that did not count.

49 of the 50 runs solved their problem. The 50 counted runs cost $35.79 (15.3M output tokens); including 16 attempts that were cut by a clock, stopped for a fix, or paused, the total was $38.77. A run took a median of 40 minutes of model time, with the longest at 4 hours, and cost about $0.39.

What is identical to the original harness, and what is not

Memdoor's parity test, lib/test_parity.py, runs the authors' multiagent_solve and the Memdoor workflow against one scripted model and compares each request. Four cases were byte-for-byte identical, and a one-character change makes them differ.

The matched settings are the temperatures (plan 0.3, ideation 0.4, manager and worker 0.2), top_p 0.95, top_k 20 (what the authors' vLLM applied from the model's own config), the 128K reply cap, up to 10 rounds, a cut-off worker being summarized, and a reply with no text being answered by its reasoning.

The main difference is the host. The authors ran their own vLLM in FP8. Memdoor ran on OpenRouter, requesting hosts in the order Modal, CoreWeave, Parasail, Wafer until Modal slowed to 14 tokens/s, then restricting to FP8 hosts (Wafer, Parasail, CoreWeave). Of the calls, 203 were served by Modal, 27 by CoreWeave, and 115 by Wafer; every call in the FP8-only ordering went to Wafer. Three runs were moved mid-run with their finished steps preserved, logged in log/host-switch.txt.

The problem text was the hard part

Memdoor's first version wrote its own task file around each problem, and the 27B model planned worse from that than from the authors' exact LiveCodeBench text. On problem arc196_c, the plan succeeded 0 of 2 times with the rewritten task text versus 2 of 2 with the original.

Four changes to Memdoor were needed to let a workflow ask exactly what the authors' harness asks:

• A task answers with the model's reasoning when it produced no text.
• top_p and top_k can be set on a model line.
• mark_cutoff handles a reply cut off at the cap.
• A call can run on its own four-hour clock.

Where the workflow lost

Both failures trace to the same two gaps in the authors' loop, not to the model.

• Pass 5, abc399_e (32 of 44 hidden tests): round 1's program was correct. The manager invented a counterexample with a wrong expected answer, the team rewrote the program, and the new version passed the public samples, which do not cover that case.
• Pass 4, arc196_c and arc196_d: a program passing every hidden test was overwritten by a later round because the manager never declared the work done. Both recovered a correct program before the loop ended — arc196_d at round 8, arc196_c through the final call — at $3.44 and $4.24, the two most expensive runs.

Memdoor notes both are fixable without changing the method: check a manager's counterexample against a brute force on small inputs, and keep the last program that passed the samples. Neither fix is included in the reported numbers.

How the workflow is structured

The step graph is a DAG of YAML files in steps/, using the format of mario, the engine underneath Memdoor:

plan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify
       └── ideate-b ──┘

step	kind	reads	writes
plan	agent	task.md	plan.md, and a notes.md ledger with one anchor per ideator
ideate-a, ideate-b	agent, in parallel	task.md, plan.md, notes.md	a proposal at its own anchor: A the most direct provable algorithm, B one from a different family of technique
solve	agent	both proposals	the decision in notes.md, solution.py
test	command	tests/	test-output.log, test-status.txt (never fails the run)
review	agent	the captured test output, not a model's claim	review.md, repair instructions in notes.md
repair	agent	the manager's instructions	a fixed solution.py, or nothing if the samples passed
verify	command	tests/	final-test-output.log; nonzero exit fails the run

Every agent prompt instructs the model to start from a fresh context, read the files, and not rely on chat history. The paper credits that ledger discipline for the gain. There is one repair round and no human gate. The final verify step is the entire verdict: it runs check-samples.sh, writes final-test-output.log, and exits nonzero if any sample fails.

Running it

curl -fsSL https://memdoor.ai/install.sh | bash
memdoor connect openrouter                 # or anthropic, openai, gemini, chatgpt
# or no key at all: memdoor connect local   # Ollama, llama.cpp, vLLM, LM Studio on this machine
git clone --depth 1 https://github.com/guregodevo/memdoor

mkdir -p ~/.memdoor/workflows
cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/   # the library: any project sees it
mkdir my-problem && cd my-problem
cp -r ../memdoor/examples/workflows/team-solve/sample/* .          # task.md + tests/1.in … 5.out
memdoor workflow run team-solve --wait   # progress on stderr, exit 0 when every sample passes

/workflow in memdoor tui draws the same run live. For a custom problem, replace task.md and add paired tests/N.in and tests/N.out files. The program is solution.py, read from stdin and compared to the samples with diff. It requires Python 3 and bash. To keep the workflow inside a single repository, copy the folder into that project's .memdoor/workflows/; the project's copy takes precedence.

The steps name no model. Whatever /model is pinned to answers the calls, including a local runtime connected via memdoor connect local.

What the first run found

The bundled sample asks for a count of substrings over the alphabet {a, b, c} with equal counts of each letter, for strings up to 200,000 characters. On the default ladder through OpenRouter:

• 8 of 8 steps and 5 of 5 samples passed. The solver chose ideator A's prefix-state method: O(N) with a dictionary of (a−b, a−c) pairs. A brute force over 300 random strings found no mismatch afterward, and the program runs in 0.04 s at N = 200,000. The manager found nothing to repair.
• One ideator's provider dropped the stream mid-answer. The coder had written max_retries: 0, so the run died there. Agent steps now carry max_retries: 1, and the run was resumed with --partition, preserving the planner's and the other ideator's work.
• The ideator prompts originally named the sample's specific methods (A: a streaming prefix-state count, B: sorting prefix states), which would steer any other problem the wrong way. They now ask A for the most direct provable algorithm and B for one from a different family of technique (2026-10-28).
• The first attempts failed every step in 0 seconds with "no agent named coder", because a run looked up its agent only in its own workspace. That was fixed in the gateway the same day in v1.692.0.

Caveats the authors state plainly

The result covers ten problems with five passes each, not the authors' 100 problems. The contests date from 29 March to 6 April 2025, so the model may have seen them during training and contamination cannot be ruled out. Fable 5's column is a single call per problem taken from the authors' result files; it was not run inside this harness. The loop sees only the public samples, while the score is measured on hidden tests.

What the Hacker News discussion added

Skepticism about the sample size

The most common objection was that ten problems cannot support a strong claim. @zihotki wrote:

I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated

@jbellis, who reported spending several weeks on the same problem, framed the generalization risk directly:

My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.

@0xbadcafebee made the compute-tradeoff argument: the result is a small model with many requests and a large token budget beating a large model with one request and a small token budget, which is not surprising on its own.

Why the team structure might work

@topspin argued the interesting variable is the harness, not the model:

My guess would be that the division of labor makes the separate contexts more effective. This is an optimization around the limits of context. Some of that is simply the net amount of context brought to bear: each team member has a context, so the net amount of context is multiplied by the number of team members.

@yurimo described the mechanism as search over solution basins: multiple models and differing opinions can nudge a system toward a better answer even when no single model is strong enough.

@CharlieDigital connected the design to an Emory paper on multi-agent evolution, arguing that role separation keeps noisy thinking tokens and tool calls out of downstream agents' context, so each subsequent agent receives a smaller, more focused input.

@robertkluin reported running a similar file-based architecture in production — planner, researcher, implementer, and reviewer, with the implementer able to spawn helpers — and said colleagues using a frontier model could not tell the difference in output.

@mchusma described a mixed-model review loop in which cheaper models review plans and code produced by a stronger one, and noted how often the smaller models catch something.

The local-model stakes

Several commenters focused on what parity implies for running models on owned hardware. @roenxi estimated that with Bonsai 2 quantizations, Qwen 27B can run at about 50 tokens per second on a reasonably priced gaming machine, citing Steam hardware survey figures of 16GB VRAM and 32GB system memory.

@novaRom reported switching entirely to local models, noting that a 27B model can be served on any modern PC or laptop with 64GB of RAM, with the main drawback being periodic compaction due to a 256K context limit.

@fedpost framed the work as a hedge against provider control:

We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever.

@cloudking pushed back on benchmark-centric enthusiasm, asking whether anyone has built a quality production app or game with real users on local models.

Objections to the write-up itself

@vessenes called the working-journal format hard to read and asked for a short human-written summary. @StilesCrisis argued the text reads as model-generated prose. Both criticisms target presentation rather than the published run artifacts.

The obvious follow-up question

Multiple commenters asked the natural control: if five small models beat one large one, what would five large models do? @conception put it as "a team of five fable models should crush all of its benchmarks," and @amelius asked how anyone knows Fable 5 is not already a swarm. The published experiment does not answer either question, because Fable 5 was only run as a single call per problem.