Failure PatternDecision layer

The Control Status Blindspot Trap: Why Agencies Fail With Control When Every Database Reads Serving

Symptom: The Control dashboard shows every client database as serving while a client reports stale data, because data freshness is only tracked on the tables explicitly added to the watched list. Root cause: Control collapses monitoring into four statuses (serving, near limit, down, unreachable), so anything short of a status change is invisible unless the right tables are watched and SLOs are defined.

By InnovaAI ResearchPublished

How do you recognize it?
  • •The Control dashboard shows every client database as serving while a client reports stale data, because data freshness is only tracked on the tables explicitly added to the watched list
  • •Slack stays silent through a partial degradation: connections climb and replication lag grows, but no status flips until the database crosses into near limit or down
  • •Agencies on the Free plan lose the trail after 7 days of history and cannot reconstruct when a client incident actually started
  • •Client-labeled databases (client:acme) are missing from the fleet view because agents were installed without consistent labels, so triage means opening each cluster individually
  • •Recovery notifications fire into a shared Slack channel that mixes several clients, and account managers cannot tell which retainer was affected
Why does it happen?
  • •Control collapses monitoring into four statuses (serving, near limit, down, unreachable), so anything short of a status change is invisible unless the right tables are watched and SLOs are defined
  • •The dbhealth agent reports every minute but only surfaces what the workspace is configured to watch; default installs leave watched tables and endpoints thin, so freshness and replication lag go untracked
  • •Per-cluster plan limits (Free covers 1 cluster and 3 watched tables, Small covers 5 clusters and 25 watched tables each) quietly cap coverage as an agency adds client databases, and nobody revisits the tier
  • •Alert routing is configured once at setup, so a single Slack channel absorbs every client's status changes with no per-client separation for delivery teams
How do you fix it?
  • •Audit the watched tables list on every cluster and add the tables each client's deliverables depend on, then confirm freshness appears in the status view
  • •Define at least one SLO with an error budget per client cluster so near-limit conditions surface before a database goes down
  • •Split Slack alerting into per-client channels or route by client label so the on-call operator sees which retainer is at risk
  • •Check the plan tier against the live cluster count and watched-table totals, and move up from Free or Small before coverage silently truncates