4420106f · 2026-07-30The homebase for all MystFlo testing — every layer as a card (what it covers · where it lives · the exact command · what it blocks · where its report is). Read as an honest ledger, not a scoreboard: layers that are stale, unproven, or absent are labelled as such, because a suite nobody trusts is worse than no suite at all.
Every count on this page was taken from
origin/main via git ls-tree, not from a local working
tree — a tree even a few commits behind reports specs that main has already deleted. The globs used
are printed next to each figure so a disagreement can be settled on evidence.
WRITE TIME PER MERGE (/ship) AFTER MERGE / PERIODIC
┌────────────────┐ ┌──────────────────────────────┐ ┌──────────────────────────────┐
│ lessons-learned│ │ scripts/local-checks.sh │ │ /audit whole-repo sweep │
│ PRD read │──►│ 27 sections, numbered 1–19 │──►│ + closure loop (weekly) │
│ chief standards│ │ FAIL = merge blocked │ │ │
└────────────────┘ ├──────────────────────────────┤ │ scripts/prove/ live harness │
prevents │ quality · security · CTO │ │ real EFs + prod DB, Stripe │
│ adversarial review (Step 4) │ │ TEST → PROVEN-LIVE/FAILED │
└──────────────┬───────────────┘ └───────────┬──────────────────┘
│ │
▼ ▼
┌───────────────────────────────────────────────────────────┐
│ NOT A GATE — nothing here blocks a merge today │
│ Playwright E2E (41 specs) · click-through (4 walked, rest draft) │
│ real-browser founder walks · Brain seeded journey rounds │
└───────────────────────────────────────────────────────────┘
· The whole point of the bottom box: the review pipeline never clicks the real
app. Nina's and Rina's demo windows were missed because live functionality failed in real
use even though every PR passed quality / security / CTO review. The real-browser layer is the
answer to that, and it is the least finished layer we have.
· GitHub Actions pr-checks.yml is advisory — five groups, same local-checks.sh --group. Merge still stays on /ship drain; Actions is not a permission slip.
/\
/ \ Real-browser walks 4 walked · activation/wallet still draft
/ \ + Brain seeded journey (3 rounds, in progress)
/──────\
/ \ Live prove harness 9 cases, real EFs + prod DB
/ \ E2E · Playwright 41 specs — not a merge gate
/────────────\
/ \ Edge fn · Deno 133 test files — gated (13d)
/ \ Assistant eval deterministic + opt-in live (13b)
/──────────────────\
/ \ Unit · Vitest 155 test files — gated on the
/______________________\ changed diff only (19)
honesty line ──────────────────────────────────────────────────────────────────
· 27 vitest files fail on a clean main (#2398) — invisible to the changed-file gate
· 1 Brain E2E spec still describes a Brain that no longer exists (#2945, brand-voice)
· no accessibility, visual-regression, load, or cross-browser layer exists at all
The doc docs/testing/testing-strategy.md still draws this pyramid as
"E2E (32 specs) / Edge (expanding) / Unit" and dates itself April 2026 — the counts above are from
origin/main 4420106f and supersede it. Drift list at the bottom of this page.
scripts/local-checks.shThe one thing that can stop a merge. ./scripts/local-checks.sh ·
--quick skips npm install · --fix auto-fixes lint. Summary prints
PASS / FAIL / WARN / SKIP; any FAIL prints BLOCKED and exits non-zero.
origin/main there are 28 numbered headings, one of which (12, Slack webhook
health) only records that the work moved to /ship's final step — so
27 sections actually execute. Numbering runs 1–19 with sub-letters (11b, 12b–12e,
13b–13d, 18b, 18c), carries a duplicate 9 (migration security lint AND metadata
template-id consistency), has no top-level 13 or 16, and the original 19 (vision price
sync) was retired with the number reused. Quote "27 executable sections" or quote nothing —
grep -cE '^# [0-9]+[a-z]?\. ' scripts/local-checks.sh returns 28.
| # | Check | What it blocks |
|---|---|---|
| 1 | Secret scan — TruffleHog | Committed credentials |
| 2 | Dependency audit — npm audit | Known-vulnerable deps |
| 3 | Security anti-pattern scan | Credential logging · n8n API from frontend · select('*') on sensitive tables |
| 4 | Semgrep SAST — diff-scoped, .semgrep/mystflo-rules.yml | 8 custom rules: select-star · credential-logging · error-leakage · missing-rate-limit · frontend-n8n-access · XSS via dangerouslySetInnerHTML · Date.now() in a meter identifier · billing thresholds on add-ons |
| 5 | TypeScript compile — tsc -b | Any TS error (zero-error policy, T374) |
| 6 | Production build | Missing files tsc --noEmit can't see |
| 7 | ESLint | Lint errors |
| 8 | Migration apply dry-run | A migration that won't apply |
| 9 | Migration security lint | RLS / SECURITY DEFINER / grant violations in SQL |
| 9* | Metadata template-id consistency (duplicate number) | Missing/duplicate template_id across workflow metadata.json |
| 10 | n8n workflow validator | Missing Extract Body · strict IF typeValidation · multi-output Code nodes · missing respondToWebhook |
| 11 | Edge-function sprawl guardrail | A new standalone EF with no EF-JUSTIFICATION: tag (100-EF Supabase cap) |
| 11b | Deno type check — changed EF files, baseline-aware | Type errors in touched edge functions |
| 12b | Trigger-gateway status gate (#2166) | checkAssistantActive not imported on a trigger path |
| 12c | Trigger-gateway regression test | The gate imported but not actually enforcing |
| 12d | Shared-engine card identity (#2302) | Card-identity drift across shared-engine assistants |
| 12e | Rate-limit endpoint registration (#2716) | An endpoint missing from ENDPOINT_LIMITS |
| 13b | Assistant eval gate — On-Brand Support golden + voice | Deterministic prompt/answer regression (ONBRAND_EVAL_LIVE=0 in CI) |
| 13c | Required-config contract tests | Server-side activation gate regressions |
| 13d | Full Deno test sweep (#2259) | EF mock-rot across all of supabase/functions/ |
| 14 | PRD spec + pricing zone integrity | Hand-edited auto-gen ## Spec blocks · price drift out of the DB |
| 15 | Platform PRD acceptance contract | A PRD missing machine-checkable AC rows |
| 17 | Billing wiring guardrail | A published paid template with no debitable price_per_run |
| 18 | Documentation hygiene | Doc-governance violations (escape: DOC_HYGIENE_ENFORCE=0) |
| 18b | Doc-sync manifest | Code change with no matching doc update |
| 18c | PRD claims lint (#2201) | Claims in a PRD not backed by code — always exits 0 today |
| 19 | Vitest unit suite (#2375/#2399) | vitest run --changed <base> only. WARNs instead of PASSing when nothing was exercised, so "no test files found" can't read as green |
What runs without a human. Each card: what it covers · where it lives · the exact command · what it gates.
_shared/ prompt builder. Layer 1 deterministic + zero-cost (gates CI); layer 2 LLM-scored, opt-in, needs a model key. RAG retrieval quality is PROVE case B4.local-checks.sh runs Playwright, and GitHub Actions pr-checks.yml does not run E2E — so no E2E result blocks a merge today.The layer that walks the product like a founder. Every scenario reconciles three
oracles: the design frame (docs/design/design-artifacts/*.html) as the visual
oracle · the PRD acceptance criteria (AC-<AREA>-NN) as the behavioural
oracle · the deployed app — real EFs, real DB, Stripe TEST, no mocks.
┌──────────────────────┐ authors ┌────────────────────────┐ executes ┌─────────────────┐
│ Platform QA Agent │──plan──────►│ scenario / test plan │───────────►│ MAIN session │
│ Read·Grep·Glob·Bash │ │ preconditions · steps │ in Chrome │ Claude-in-Chrome│
│ no browser tools │ │ expected frame · pass │ │ against DEPLOYED│
└──────────┬───────────┘ └────────────────────────┘ └────────┬────────┘
│ asserts DB side-effects │ on FAIL
│ (QA test accounts only) ▼
└───────────────────────────────────────────────► structured bug ──► /debug
The split is deliberate and binding: the QA Agent has no browser tools, so it authors and (post-run) asserts backend side effects; the MAIN session drives the clicks. An agent that claims it clicked the app is reporting something that did not happen.
peytonworkprojects@gmail.com test account. Each round started from a wiped Brain
and walked intake → research checklist → identity gate → paid read → Instagram vision →
playbook. Headline: extraction fidelity was consistently high across all three (verbatim
schedules, follower counts, glyph bullets, bilingual structure) — every defect found traces to
the write/confirm/save layer AFTER extraction, not the AI reads themselves. 6 of the resulting
defects are fixed and live in production (#2988, PR #2997) as of 2026-07-31; 2 remain open
follow-ups (see each round report). Reports below link the full raw payloads.docs/testing/ with no automation behind them. Kept
because they encode judgment a script can't, but none has a recorded pass/fail run in-repo —
treat each as a checklist, not evidence.What the per-merge diff can never see.
/ship's diff-scoped reviews — catches drift a
per-merge review can never see (RLS on old tables, dead code, doc-vs-code drift, unattested
EFs) and routes findings to closure instead of an issue graveyard. Reconciles open
findings before filing anything new, auto-closes resolved, escalates anything still
reproducing after 7 days to a Peyton-owned launch blocker./quality-review and /security-review actually
catch real bugs rather than assuming the design works. Mutation seeding hit 5/5 finder
recall; known-miss replay found one live bug (stale main...HEAD diffing — same
root cause as the #1418 near-miss) and one correct non-catch. Live sampling — the check on
false-positive noise — has never run.Every reset script is dry-run by default and user_id-scoped. There is ONE Supabase — every run touches production.
| Asset | What it does | Command |
|---|---|---|
| reset-test-account.sh | E2E baseline reset — user-id-scoped, restores seed connections. Run before an E2E suite so tests start predictable. | scripts/reset-test-account.sh |
| reset-brain-data.sh | Wipes ALL Brain data for one founder — profile, RAG chunks + embeddings, gate verdicts, source snapshots, personas, reputation reads, onboarding chats, uploaded Brain files. Preserves connected apps, WhatsApp customer data, activation state, offerings. | scripts/reset-brain-data.sh --email <addr> then --execute |
| seed-brain-playbook-test-account.ts | Seeds a populated Brain playbook onto workprojects via the real brain-profile EF path (no direct SQL, no AI content). Idempotent. Both test accounts otherwise start with an empty Brain (#2437). | doppler run --project mf-platform --config prd -- deno run -A scripts/seed-brain-playbook-test-account.ts |
| peytonworkprojects@gmail.com | Sandbox account — non-WhatsApp assistants, general testing, full seed + teardown. The seeded-journey rounds run here. | — |
| peyton@mystflo.com | REAL account — holds the real WABA, used only for WhatsApp cases. Prove harness is additive only here and refuses to delete a row it didn't create this run. | — |
| Doppler secrets | TEST_USER_EMAIL / TEST_USER_PASSWORD; anon key is VITE_SUPABASE_PUBLISHABLE_KEY with SUPABASE_ANON_KEY as the server-side alias. | doppler run --project mf-platform --config <env> -- |
A hub that only lists what exists is a flattering hub. This is what is broken, unverified, or simply missing.
Verified against origin/main, not a local working tree — a tree 11 commits behind reported two already-fixed specs as still broken. #2963 (merged 2026-07-30) deleted brain-v2-smoke.spec.ts and repointed brain-read-wallet-empty.spec.ts's selectors from /Read it/i to /Research/i; both are off this list. One Brain spec remains.
| What | State | Detail |
|---|---|---|
| tests/e2e/brain-brand-voice.spec.ts | Likely stale — not individually proven | 3 tests asserting tone presets, gated capture, persona starter line. B5 restructured every voice surface (VoiceRegistersPanel, HowYouSoundSheet, the calibration game). Flagged as likely by inference from the same shape as the proven case, not confirmed. The one Brain spec still unresolved. #2945 |
| Vitest suite on clean main | 27 files failing | No single root cause. The ship gate runs --changed only, so this debt is invisible per-PR and gating on the full suite would block every PR. #2398 |
| Click-through scenarios | 4 QA-walked · 6 guest-blocked 2026-08-15 | Pulse status PASS (2026-08-13). Connect-apps FAIL #3466 (2026-08-14). FAQ journey MIXED (2026-08-09). Research engine is the exemplar. 2026-08-15 guest batch logged BLOCKED (no QA password) — activation Try/preview still never walked as Peyton WP. Index: test-ledger.md. |
| local-checks 18c — PRD claims lint | Always exits 0 | Registered as a check but cannot currently fail. It reports; it does not gate. |
| Review pipeline — live sampling | Never started | The one eval that would tell us whether findings are noisy enough to be ignored. Only meaningful on real post-merge PRs. |
| Category | Status | Why it matters here |
|---|---|---|
| Accessibility | None | No axe-core, jest-axe, pa11y, or Lighthouse anywhere in package.json, tests/, or scripts/. Contrast, focus order, and screen-reader labelling on a dark glassmorphic UI are entirely unverified. |
| Visual regression | None | No toHaveScreenshot / toMatchSnapshot, no Percy / Chromatic / Argos. Design parity is scored by a human or an agent eyeballing screenshots against a frame — nothing catches a silent pixel regression between reviews. Notable given MOCKUP-ARTIFACT-FIRST is binding. |
| Cross-browser / mobile viewport | None | playwright.config.ts defines exactly one test project: chromium, Desktop Chrome. No WebKit, no Firefox, no mobile device project — while the layout-tier contract makes 375px the tier scored first. |
| Load / performance | None | No k6, artillery, or autocannon. Rate limits, EF cold starts, and pgvector query cost under concurrency are untested. |
| E2E as a merge gate | None | Playwright appears in no local-checks.sh section, and pr-checks.yml does not run E2E. Every E2E result is advisory. Merge stays on /ship drain. |
docs/testing/ contradicts the code| Doc says | Code says |
|---|---|
testing-strategy.md — "E2E, 32 specs"; dated April 2026 | 41 spec files (23 root + 18 assistants) |
testing-strategy.md §3 test-structure listing includes tmpl-clfu01, tmpl-rvreq01 | Neither file exists. Missing from the listing: tmpl-cohb01, tmpl-wsbk01, and every brain-*, guest-preview, preview-banner, connects-deeplink, google-picker-migration, t383-data-source-picker, whatsapp-number-picker, waitlist-page, activation-flow spec |
testing-strategy.md §2 — Deno coverage is _shared/; "All others: 0" | 133 Deno test files across brain-ops, assistants, brain-generate, webhooks, and more |
testing-strategy.md §5 — "Full 11-check suite"; Semgrep "7 custom rules" | 27 executable sections; Semgrep has 8 rules. CLAUDE.md's "19 checks / 9 custom rules" is also wrong on both numbers |
testing-strategy.md — prove harness "4 built, 12 planned"; G1/B4/B5/M3 listed ⬜ todo | 9 cases on origin/main: B4, B5, G1, G2, G3, M3, W2, W4, W5 (plus 2 fixtures + index.ts) |
testing-strategy.md §5 — GitHub Actions pre-merge-checks.yml "runs on push to main" | pre-merge-checks.yml is gone. Live workflow is advisory pr-checks.yml (five local-checks.sh --group jobs). Merge stays on /ship. |
test-plan-brain-seeded-journey-2026-07.md pre-flight #1 — "a REAL account, not a QA test account"; #6 — "no wipe mechanism exists, orchestrator must supply one" | Rounds run on the peytonworkprojects@gmail.com QA account, and scripts/reset-brain-data.sh is the wipe mechanism. The plan was authored before both were settled and its pre-flight table has not been updated. #2536 (user-facing Brain deletion) remains open — that is the product gap, not the ops one |
docs/testing/testing-artifacts/testing-hq.html at the same URL, so the board always
shows the latest statedocs/testing/testing-artifacts/README.md · canonical strategy doc:
docs/testing/testing-strategy.md (carries the drift listed above)