Memdoor reports 9.8 out of 10 coding score for a Qwen3.8-27B team

A workflow result, not a matched model contest

GVS5H is the original study from which Memdoor takes its Claude Fable 5 baseline. All results discussed here are self-published in GitHub repositories, with no outside reporting independently confirming them.

Memdoor is a GitHub project whose repository includes team-solve, an example programming workflow. The project describes five roles plus a repair worker, with fresh contexts and communication through files. A shell script determines whether sample tests pass. The loop sees public samples; the reported score uses hidden tests.

The project reports a mean of 9.8 problems solved per pass through the ten-problem set, with 50 counted runs overall—not a single ten-problem test. Its comparison lists 9.6 for the GVS5H harness and 8.0 for Fable 5 alone.

Memdoor says it used GVS5H's prompts, loop and corrected grader. It also reports identical requests in four scripted-model parity cases. That establishes a narrower claim than experimental equivalence: serving infrastructure differed, and Fable 5 was not rerun inside Memdoor.

What the price claim does—and does not—show

The GVS5H repository title promises “Fable 5 Level Coding for a Fifth the Price.” Memdoor's figures do not settle that claim: they report its own spending without corresponding Fable 5 costs.

Across 100 hard problems, GVS5H reports that orchestration raised locally served Qwen3.8-27B from 66.8% to 92.4%, alongside 90.4% for Fable 5 under the study's reported pass@1 metric. The authors' own title describes the five-model result as a “match” for Fable 5. GVS5H also reports 93% pass@1 for the distinct Qwen3.8 Flash Next model at 9% of Fable 5's cost. No uncertainty estimates establish a ranking among these scores, and the 9% figure is not a cost comparison with the orchestrated 27B setup.

The problem sets and serving settings differ, so the wider Memdoor–Fable gap cannot be read as a Memdoor improvement over GVS5H. GVS5H also reports that orchestration did not improve every model.

Memdoor reports spending $35.79 on the 50 counted runs and $38.77 including 16 excluded attempts. Counted runs generated 15.3 million output tokens. The reported median run cost was $0.39, with median model time of 40 minutes and a longest run of four hours.

Those measurements describe hosted execution through OpenRouter, not a demonstrated local Memdoor deployment. Although the workflow can use a pinned local model, local support alone does not establish equivalent performance or operating cost.

Where Memdoor's runs went wrong

Memdoor says its sole unsolved run followed a manager-generated counterexample with an incorrect expected answer, prompting a rewrite of an initially correct program. Two other runs temporarily overwrote programs that passed every hidden test because the manager failed to terminate the loop.

Memdoor says proposed safeguards were not included in the published scores.

Who is affected

Developers adapting team-solve must supply a task file and paired sample input/output files, and need Python 3 and bash, according to the workflow documentation. Its underlying Mario engine is described as an experimental Go DAG execution library with an unstable API, a constraint for developers integrating directly with it.

Related coverage

• Apps & BusinessNetherlands plans 25,000 government job cuts by 2035, with AI productivity target
• Apps & BusinessMeta ordered to pay $25 billion in civil penalties in New Mexico Cambridge Analytica case
• Apps & BusinessOpenAI acquires Orchestra Bio, a drug-development coordination startup, for Rosalind