How to Guarantee Quality When AI Mass-Produces Your Code — The 4-Layer Gate System Behind 200 Components

#The Question This Article Answers

When AI mass-produces your code, what actually guarantees quality? This site runs roughly 200 interactive components (197 in the catalog) built AI-first, and every one of them passes through the same quality gates. This article documents the machinery we actually operate, down to the file names.

The full picture first. There are four layers, and each successive layer sits further from human eyes.

The point: none of these layers works by asking the LLM to "please be careful." Every gate is deterministic code that returns the same verdict for the same input.

#Premise: AI Writes More than Humans Can Review

Of the ~200 components, only a fraction have had every line read by a human. AI-driven throughput always outruns human review bandwidth. So instead of adding more review, we invested in a structure that doesn't break without review.

Concretely, that means decomposing "quality" into machine-checkable properties. Did a secret leak? Did a reference break? Does it actually paint pixels? Does the prose smell like AI? Each of these can be checked by code.

#Layer 1: Four Hooks That Block the Commit

.githooks/pre-commit runs four checks in sequence; if any one fails, the commit itself fails.

bun run scripts/repo-guard.ts
bun run scripts/check-llm-client-boundary.ts
bun run scripts/check-generated-files.ts
bun run scripts/humanizer-check.ts --staged

Each has a sharply defined job.

• repo-guard.ts — scans staged diffs for private-note paths and op:// secret references. AI tends to copy context it consulted into its output, and in a public repo that becomes a leak
• check-llm-client-boundary.ts — rejects any file that imports an LLM SDK and a confidential-file loader together, statically severing every path by which sensitive data could reach an LLM API
• check-generated-files.ts — blocks commits that modify generated artifacts (sitemap, feeds, llms.txt) outside the regeneration command. A hand-edited artifact is a time bomb that the next regeneration silently erases
• humanizer-check.ts — see below

#Gating Prose Style — humanizer-check

The fourth hook is the unusual one. humanizer-check.ts detects "AI-typical" Japanese prose with regular expressions. A few of the live patterns:

• Connective constructions AI overuses (Sino-Japanese verbs like 実現/構築 followed by しており)
• Template closers of the 「今後の展開が注目され──」 family
• Inflated modifiers stacking 非常に or 極めて onto 重要/高い
• Three consecutive - **Heading:** description bullets (a formatting habit AI loves)
• Uniform paragraph openings across a section

On a site where AI also writes the blog, prose quality is not something you "catch in review" — it's something the machine rejects. This very article cannot ship without passing that check.

#Layer 2: Freeze Data Contracts as Invariant Tests

All content is static TypeScript data, so structural promises live as vitest invariant tests. Examples:

• Every blog post belongs to exactly one topic cluster (pillars.test.ts) — zero or two memberships both fail
• Demo IDs and related-article references must resolve to real catalog entries (broken-reference detection)
• dateModified must never precede date

When AI adds articles or components, missed registrations, broken references, and duplicates happen at some baseline rate. With invariant tests, those stop being "mistakes review should catch" and become "mistakes that turn CI red." The cost of finding them detaches from human attention.

#Layer 3: Verify "It Painted" in a Real Browser

Canvas components have a signature failure mode: types check, build passes, React renders — and the screen is black. Unit tests never see it.

So the Playwright smoke suite (tests/e2e/site-smoke.spec.ts) reads actual Canvas pixels:

const { data } = context.getImageData(0, 0, node.width, node.height)
// count pixels with alpha > 0 and brightness > 0

It verifies not "a canvas element exists" but "something was actually drawn." The suite also captures every console error during page load and requires zero, and checks that no horizontal overflow appears. CI runs lint → tests → build → this browser smoke, and deployment after merging to main is automated.

#Layer 4: Generated Content Is Stopped Twice — Score, Then PR

The article-generation pipeline (scripts/engine/) has its own publication gates, separate from the code gates.

1. Quality score: every generated article is scored, and anything below qualityScore >= 70 never becomes publishable (AUTO_PUBLISH_MIN_SCORE = 70)
2. PR proposal mode: even articles above the threshold are never pushed to main. The daily pipeline stops at opening a PR; publication happens only when a human merges

Creation and publication are separated. The worst case when automation misbehaves is confined to "one odd PR shows up" — production feels nothing. This separation has already paid for itself: when a defect was found in the pipeline's registry-file output, the broken change stayed inside its PR and production was untouched.

#The Design Principle: Rules Live in Hooks, Not Prompts

One principle runs through all four layers: never put a statically checkable rule into a prompt.

Write "don't include secret paths," "don't break references," "avoid AI-sounding prose" into CLAUDE.md and the LLM obeys only probabilistically — and compliance per instruction drops as instructions accumulate. Put the same rule into a hook, a test, or CI, and compliance is 100%, freeing the prompt for judgments machines can't make.

You can add instructions to the AI, or you can add machines that inspect the AI's output. At production scale, the second one is what worked.

#Summary

• Quality for AI-mass-produced code comes from layered deterministic gates, not more review
• Layer 1: pre-commit stops secrets, boundary violations, artifact edits, and AI-sounding prose
• Layer 2: structural promises are frozen as invariant tests
• Layer 3: "it rendered" is proven by reading real browser pixels
• Layer 4: generated content separates creation from publication via score + PR
• Principle: statically checkable rules belong in code, not prompts

The mass-production war story lives in the pillar article: Bulk-Fixing 200 Components with Claude Code, and the session discipline in One Session, One Task.