Browse the documentation

Case study: checking what agents say is done

One review rated 11 statements false in what 39 agents wrote in under twelve hours, and the reviewers were wrong twice. See how, and replay the five commands.

On this page
  1. What it took, in numbers
  2. Five steps you can copy
  3. What the reviewers caught, and where they were wrong
  4. Where a person decided
  5. Replay it yourself
  6. What this does not prove, so you can judge it

If an AI agent has ever told you “done” and it was not, this page is for you. It shows what happened when 39 agents rebuilt the landing page, documentation, examples and videos of this site in under twelve hours, between the evening of 2026-10-18 and the morning of 2026-10-19: one review rated 11 statements false, the reviewers were wrong twice, and a person decided what to keep. You will see the five steps, the numbers with their sources, the errors, and the commands to replay on your own project. If you only want to run it, jump to Replay it yourself.

Note Before the numbers: the flaws

Read this first. The agents were launched by Claude Code with its own agent tool, each implementer in its own git worktree, not by the bmad-plus swarm command: that command deals the work, writes the briefs, checks the answers and tallies them, and leaves the launching to your assistant. The reviewers are models too, and two of their findings were wrong. This is one run, and nobody has measured yet whether readers find the result clearer. The pattern is the one BMAD+ ships (ask first with an intake, fan out under a budget, review independently, stop at gates). Every number below was recounted on 2026-10-19 from the repository, the logs or the review reports; a figure that could not be recounted is not on this page.

AI said “done.” Was it? How 39 agents rebuilt this site in under twelve hours, what the reviewers caught, and the five steps and commands to replay it yourself.
Transcript

AI said “done.” Was it? Thirty-nine agents rebuilt a site in under twelve hours. Reviewers found eleven false claims. They were wrong twice, too. You have probably been burned by “done” before. Here is how to check: five steps, five commands. One. How do you stop invented facts? Write the rules first: from scratch, facts from the code, simulated means labelled. Two. What should it ask before it starts? The intake lists what is known and what is missing. Three. How do agents stay out of each other’s way? One brief per page, one copy of the repo each. BMAD+ deals the briefs and caps each wave. Your assistant launches the agents. Four. Who checks the work? Five readers that wrote none of it, replaying commands on a clean server. And the gate says no until every file is accounted for. A promise that your files are kept. A command that never existed. A count off by one. Five. Fix, then prove: nineteen fix commits, 1,879 tests, zero accessibility violations. A person kept the last call. Nobody has measured yet whether readers find it clearer. You get five commands, three swarm files, the review gate and seven mistakes to avoid. Start with step one. “Done” is a claim. Replay it.

Described transcript

What it took, in numbers

What the rebuild produced, and where each figure comes from
FigureValueWhere it comes from
Agents, up to the integrated branch39The saved log of the session’s sub-agents
Commits on the integration branch133 (118 ordinary, 15 merges)git log since master
Files changed328 (72,665 lines added, 650 removed)git diff --shortstat
Pages on the site8 before, 244 after, each with a Markdown mirrorThe live sitemap of 2026-10-18, then the build output
Documentation pages52 per language, in four languagesThe page registry
Examples and terminal transcripts160 examples; 70 transcripts, 19 of them simulated and marked as suchThe page registry
Videos6, narrated in English and French, 24 caption filesThe media manifest
Independent reviewers5, with one written report eachThe review reports of 2026-10-19
Fix commits after the reviews19, on four branchesgit log of the four fix branches
Tests and checks at the end79 site tests, 1,879 tests in the main suite, 91,986 search checks, 2,539 release checksThe gate runs
Browser run976 viewport cases, 300 accessibility runs, 0 violationsThe browser check log
Time11 hours 52 minutes from the request to the integrated branch, 3 hours 21 minutes of it pausedCommit and session timestamps
Media creditsAbout 709 credits for 15 generated clips and the narrationThe credits ledger

Five steps you can copy

  1. Write the rules first

    Write the goal in one sentence and the rules that must never bend. Here the goal was a site that covers every possibility of BMAD+ in four languages, with videos. The rules: everything written from scratch, facts only from the repository, anything simulated says so, and nothing runs on the owner’s own computer (builds and tests ran on a test server, in containers).

  2. Let it ask before it acts

    Before any agent started, the work was listed: what existed and what was missing (the new commands had no example and no reference page). The owner then widened the scope. In BMAD+ this step is the intake: it records what is answered, defaulted or unknown, and asks at most six questions per message.

    In your project terminal
    npx bmad-plus intake plan --intake .agents/skills/pack-design/intake.yaml --answers _bmad-output/design/intake.json

  3. Give each agent its own page and its own copy

    An agent first inventoried the whole product. Then the landing page, the documentation, the screen recordings and the videos were built in parallel, each agent in its own git worktree and its own copy of the project on the test server, with scopes that did not overlap so the branches could be merged. In BMAD+ the same shape is a swarm definition with one unit per page and disjoint scopes.

    In your project terminal
    npx bmad-plus swarm init --definition swarm/page-fanout.json --units swarm/units.json

  4. Make someone else try to break it

    Five reviewers that had written nothing read the result at a fixed commit: one checked facts on the packs, agents and concepts pages, one the reference, guides, security, FAQ and landing, one replayed the examples, one read the French, Spanish and German, one drove a browser for accessibility. They replayed commands on a clean server instead of trusting the text. In BMAD+ the evidence layer is review: it seals the scope and refuses a verdict until every file is accounted for.

    In your project terminal
    npx bmad-plus review scope site --workspace --base HEAD
    npx bmad-plus review gate site

  5. Fix, then prove it

    Seven fix groups and four fix passes corrected what the reviews found (19 fix commits on four branches). One integrator merged 14 branches and re-ran every gate: 79 site tests, 1,879 tests, 91,986 search checks, 2,539 release checks and a browser run with 976 viewport cases and 300 accessibility runs without a violation. Nothing was deployed before the owner said so.

What the reviewers caught, and where they were wrong

You might expect agents that have read the code to get the facts right. They did not always. These seven statements did not survive a replay of the command or a read of the code, and each one was fixed.

Seven claims that did not survive a replay
What the draft saidWhat the reviewer foundThe fix
The installer preserves instruction files you wrote.With install --yes, a CLAUDE.md you wrote is copied to CLAUDE.md.bak and then replaced. The sentence was on four pages.The pages say: not silently. Interactive installs and updates preserve it, --yes backs it up first, --tools none touches no tool configuration.
Three commands on the landing page, shown as runnable.Replayed: intake plan needs --answers, mem recall with evidence ranking needs a project scope, swarm init needed --context pitch=FILE.The commands carry the missing flags and run.
A prompt that began with /animated.No /animated command exists anywhere in the product.The prompt is the sentence the agent listens for: build an animated website from a video.
A Nexus example that printed verification=passed right after create.Replayed, verify right after create fails with exit code 1: it needs completed, unaccepted, reconciled work.The example now starts, records and verifies a real task, with replayed output.
Commands run after the first install start with npx --no-install.In a project that never ran npm install --save-dev bmad-plus, that command stops with "npx canceled due to missing packages".Examples use npx bmad-plus; the --no-install form is explained where it appears, with its prerequisite. The instruction files the installer generates still say --no-install: that needs a release and is not fixed yet.
Two video clips of the installer showed Shield with 26 frameworks.The registry counts 27, and every other page said 27.Both clips were recorded again from the current installer.
Captions: one cue per sentence, and French captions that reworded the narration.Lines of up to 79 characters, cues of up to 11 seconds; French captions did not match the spoken words.All 24 caption files are cut to 2 lines of 42 characters and 6 seconds; the French ones are word for word with the voice.

Warning The reviewers were wrong too

Two findings did not hold up. One review said that 0.23.0 was not on npm; it is, and the commands on this page install it. Another reported English titles burned into the French videos; the video director re-checked the frames, the overlays were already French and the English that remains is real terminal output kept on purpose. A reviewer is a second reader, not an oracle: each finding was checked before it was fixed, and a human kept the final call.

Where a person decided

  • To go, with parallel swarms, and to widen the scope to every possibility of the product and to presentation videos with a voice-over.
  • The narration voice: the first one was rejected, an ElevenLabs voice (Celine) was approved after listening.
  • To start the documentation, the examples and the CLI reference in parallel rather than one after the other.
  • Who is credited for what in the README and the npm description.
  • A full backup of the live site before replacing it.
  • The wording of two public claims: Dev Studio is described as original, and the SEO pack card reads “SEO / GEO”.
  • A pause while the owner was away, and the resume the same night.
  • That nothing is deployed before the owner says so.

Replay it yourself

Five commands carry the method: install, intake plan, swarm init, review scope and review gate. The blocks below run them, with the few commands around them that make them work. You can stop after any block.

You need Node.js 20 or later and a Git project with at least one commit. The blocks below are real output from the published 0.23.0 package. Where sub-agents would write their answers, the swarm part writes three small answer files by hand: no model runs, so the tally counts stand-in answers and says nothing about real pages. The definition, the brief template and the units file are printed on the swarm page.

Install the packs
npx bmad-plus install --yes --lang en --packs core,design,memory --tools claude-code
Output shortened to its last line.
BMAD+ is ready! Talk to Atlas to get started 🚀

Output captured from a real run. Source: Replayed on 2026-10-19 with the published 0.23.0 package, run through npx, in a clean Node 20 container, in a throwaway Git project

Ask first: record answers, then list what is still open
npx bmad-plus intake record --intake .agents/skills/pack-design/intake.yaml --answers _bmad-output/design/intake.json --set surface_kind=landing --set "audience=Urban professionals who buy coffee most days" --set "job=Get a first-time visitor to open a loyalty card"
Recorded surface_kind, audience, job in _bmad-output/design/intake.json.
npx bmad-plus intake plan --intake .agents/skills/pack-design/intake.yaml --answers _bmad-output/design/intake.json
design-read (design): ready — 3 answered, 0 defaulted, 0 unknown, 10 not asked yet.
Ask next (at most 6 per message):
  - brand_assets [choice] Which brand assets exist already? None, a partial set (a logo and a colour or two), or a full kit (logo, palette, type, voice)? (choices: none, partial, full)
Five more questions follow in the real output.

Output captured from a real run. Source: Replayed on 2026-10-19 with the published 0.23.0 package, run through npx, in a clean Node 20 container, in a throwaway Git project

Fan out: one brief per page, then check and tally the answers
npx bmad-plus swarm init --definition swarm/page-fanout.json --units swarm/units.json
site-pages-20261019-kgoi: fanout of 3 units, budget 26, approval not-needed.
Units listed, no seed.
Next: bmad-plus swarm briefs site-pages-20261019-kgoi
npx bmad-plus swarm briefs site-pages-20261019-kgoi
site-pages-20261019-kgoi: 3 of 3 units need a sub-agent, in waves of 10.
Wave 1:
  about  /work/app/.bmad-plus/swarm/runs/site-pages-20261019-kgoi/briefs/about.md
  pricing  /work/app/.bmad-plus/swarm/runs/site-pages-20261019-kgoi/briefs/pricing.md
  contact  /work/app/.bmad-plus/swarm/runs/site-pages-20261019-kgoi/briefs/contact.md
Each sub-agent: Read the brief and follow it exactly; it is your whole instruction.
Disclosure: Each page was written by a separate sub-agent from the same brief; read them together before publishing.
Here the three answer files were written by hand, where each sub-agent would write its own.
npx bmad-plus swarm check site-pages-20261019-kgoi
site-pages-20261019-kgoi: 3 of 3 valid answers.
npx bmad-plus swarm tally site-pages-20261019-kgoi
/work/app/.bmad-plus/swarm/runs/site-pages-20261019-kgoi/tally.md
3 of 3 units counted; 0 skipped.
Disclosure: Each page was written by a separate sub-agent from the same brief; read them together before publishing.

Output captured from a real run. Source: Replayed on 2026-10-19 with the published 0.23.0 package, run through npx, in a clean Node 20 container, in a throwaway Git project

Review: seal the scope, then see the gate refuse
npx bmad-plus review scope site --workspace --base HEAD
/work/app/_bmad-output/review/site/scope.json
1 file(s) to review, 1 changed lines, 1 unit(s) — scope 1ef2d235ea25
checklist: /work/app/_bmad-output/review/site/checklist.md
plan: medium effort, 2 pass(es), refutation pass
Every selected file must end in coverage.json as completed, failed or waived (with a reason).
npx bmad-plus review gate site
site: incomplete — coverage 0/1 completed (0 waived), 0 open finding(s), 0 refuted
  - no coverage.json: which files were reviewed is not established
  - no findings.json: write it even when there is nothing to report
  - 1 selected file(s) never accounted for: site/contact/index.md
Exit code 2: the verdict stays incomplete until the reviewer writes its coverage and its findings.

Output captured from a real run. Source: Replayed on 2026-10-19 with the published 0.23.0 package, run through npx, in a clean Node 20 container, in a throwaway Git project

Rewrite several pages in parallel, then have a stranger attack them

Use when several pages or sections must change at once without agents overwriting each other, and you want a reviewer that wrote none of them. Needs the three swarm files from the replay above and a Git project with one commit.

In your AI assistant
Run the site-pages swarm from swarm/page-fanout.json and swarm/units.json: one sub-agent per page, each limited to its own folder, one wave at a time. When the answers are in, run swarm check and swarm tally and report the disclosure first. Then, with a different sub-agent that wrote none of the pages, run review scope on the working tree against HEAD, read every changed file and replay every command the pages show. Do not accept a page whose command fails, and tell me which claim the reviewer could not verify.

What you should get The briefs are written, the sub-agents answer in waves of at most 10, swarm check reports the valid answers and tally.md opens with the disclosure. The reviewer seals a scope with review scope, and review gate stays incomplete, with exit code 2, until every changed file is covered by a written coverage and findings file. A command that fails when replayed is reported as a finding, not fixed silently.

What this does not prove, so you can judge it

  • One project and one run: it shows that the method can catch mistakes, not how often it will.
  • The reviewers are models too. A finding is a lead to check, and two of them were wrong.
  • The tallies shown on screen and in the examples are demonstrations, and they are marked as such.
  • Nobody has measured whether the site is easier to understand. The checks cover facts, accessibility rules and structure, not comprehension.
  • The French, Spanish and German texts were reviewed by an agent against the English and a shared glossary, not by a native proofreader.
  • The videos were checked by scripts and by an agent looking at frames. How they sound and feel is still a human judgement.

See also