Test through the customer’s door

The decision

Tester agents use HYVIRAL with only a customer’s access and knowledge. On 7 September one of them caught the in-app assistant falsely claiming that no social profile was linked.

The path to closure: false account-state report → resolve current state on every turn → recorded deployment → fresh-conversation retest → closed on 7 September.

The next day, I added a dedicated MCP tester that works from customer-facing documentation and client tools, without engineering repositories or specifications. I also retired a reporting repository the tester could not access, after 15 reports had left it empty. Slack threads became the reporting channel, and the engineering register stayed the system of record.

From the source

Closed issue. Account, conversation, tester and deployment identifiers are redacted:

Issue A · S2

Agent says no social profile is linked while [REDACTED: test profile] is linked AND D10-verified and mid-sync. It then continued onboarding instead of answering the question — no error, no Support ID, so the user is told something false rather than shown a failure

Agent chat [REDACTED: conversation route] · [REDACTED: private issue thread] · product-defect

closed (7 Sep) — [REDACTED: customer-side tester] re-verified PASS at 03:27Z in a fresh conversation (the reply led with "Your [REDACTED: test profile] Instagram profile is still syncing…", no false unlinked). Deployed 7 Sep ~02:51Z as BE [REDACTED: backend commit] (rev [REDACTED: backend revision]) / FE [REDACTED: frontend commit] (rev [REDACTED: frontend revision]); awaiting re-verify — the interjection path now receives the profiles loaded on THAT turn and leads with a phase-aware, handle-naming claim (onboardingInterjectionClaim) before the classifier line; 3 new handler cases + 6 data-phase cases, mutation-proven (2 red when the claim is disabled). The 22:07 greeting is a stored message and stays as written; every later turn is truthful. Cause: onboarding.service.ts:308 picks the question FLOW from profiles.length > 0 at conversation-creation time and writes the greeting claim into a stored message; his conversation was created 22:07:05Z, six minutes BEFORE the 22:13:05Z link, and is never re-evaluated. AgentLinkCompletionService exists to close this gap but is post-ingestion and initial-run-only, so it fires only when a run SUCCEEDS — his was watchdog-killed at 22:16:00Z. Real in its own right: asking mid-run would have produced the same false claim even had ingestion succeeded

Tester report. Fix is code: re-resolve the phase per turn, or at minimum stop repeating the claim once a profile exists. Downstream of F-009 only in that the failed run kept the catch-up from ever firing

The row keeps an intermediate “awaiting re-verify” note from before the 03:27Z retest that closed it.

What this shows

A full customer-verified loop: a tester’s report, a root cause traced to state captured at conversation creation, a per-turn fix with mutation-proven tests, a recorded deployment and a fresh-conversation retest. This closure came the day before the MCP tester and the delivery consolidation, so it shows the loop working rather than their effect.