From alert to a verified fix PR — while you sleep.

The 3 AM problem

The fix is rarely the hard part. The first hour is.

“I've spent years as the engineering manager getting paged at 3 AM — the hard part is never the fix, it's the first hour of figuring out what broke.”

• Proof, not guesses

  The host itself checks that the new test fails before the fix and the whole suite is green after it.

  How it works →

• Safe by construction

  Tool permissions are enforced in code, nothing reaches production without a human tap, and an upstream outage gets no fix and no rollback.

  Safety →

• Raw logs stay home

  Redaction runs on the host, and the lab where it reproduces failures has no network.

  Privacy →

What it is not: not an observability platform, not an autonomous production deployer, not multi-cloud. One narrow loop: incident → verified fix → lesson.

How it works

Four steps, from the page to a lesson

1. 01 / 04

   Investigate

   Correlates the alert with deploys, diffs, logs and metrics, checks what it has seen before, and names a suspect commit with evidence.

   Under the hood

   Read-only tools: logs_signatures, logs_query, metrics_query, deploys_list, git_diff, memory_search.

   Veilleur·INC-74+6 s

   🚨 Incident detected: INC-74 · sncf-mcp

   - User impact: 56 of 604 next_departures calls failed (21.47%) in the last 5 min.

   - find_station, plan_journey, and get_disruptions had 0 failures.

   - No non-2xx Navitia responses reported.

   - Last deploy a80001e8c303 was at 19:43:01 UTC, about 1 minute before the alert.

   Investigating.

   Veilleur·INC-74+22 s

   🔎 INC-74 · sncf-mcp: NextSummary indexes an empty departures slice

   Suspect: a80001e8c303 "next_departures: lead with the next train" by Camille Martin, deployed 19:43 UTC

   Evidence:

   - 68 next_departures calls failed on a80001e8c303; 0 of 152 calls failed on previous 26d2a883a3c3.

   - Diff adds NextSummary(deps); it indexes deps[0] at internal/transform/transform.go:168.

   Failing call: next_departures {"station":"Culmont-Chalindrey"} (request lg-368862)

   Next: reproduce this call against a80001e8c303 with a failing test before proposing a fix.

   hypothesis recorded at 19 s

2. 02 / 04

   Reproduce

   Replays the failing call from the logs against the deployed commit in a container with no network, then writes a Go test that fails the same way.

   Under the hood

   sandbox_replay (~1 s) and tests_run (go test, 0.5–1 s) in Docker with --network none.

   package transform
   
   import "testing"
   
   func TestNextSummaryEmpty(t *testing.T) {
   	got := NextSummary(nil)
   	if got != "No upcoming departures" {
   		t.Fatalf("NextSummary(nil) = %q, want %q", got, "No upcoming departures")
   	}
   }

   +	if len(deps) == 0 {+		return "No upcoming departures"+	} 	next := deps[0]

3. 03 / 04

   Propose a fix

   Opens a PR only after the host itself checked it: the new test fails on the deployed commit and the whole suite passes with the fix. You approve with one tap; nothing ships without it.

   Under the hood

   fix_propose re-runs both checks on the host; github_merge_pr is tier T3 and needs a single-use approval.

   Veilleur·INC-74+55 s

   🛠 INC-74 · sncf-mcp: fix proposed, PR #38

   Reproduced: next_departures for Culmont-Chalindrey → panic: index out of range [0] with length 0 on a80001e8c303; TestNextSummaryEmpty fails there with the same panic.

   Fix: NextSummary returns “No upcoming departures” for an empty slice.

   Checks: TestNextSummaryEmpty passes and the full suite is green (52 tests) on 30ec08cb64ec.

   PR: #38

   Approval requested: merging deploys the fix; use the buttons in the approval message.

   PR opened at 47 s

   Approval bot·INC-74+49 s

   🔐 INC-74 · sncf-mcp: approve merging PR #38?

   transform: handle empty departures in NextSummary

   Commit 30ec08cb64ec: the new test fails on the deployed a80001e8c303 and the full suite passes with the fix.

   Approving merges it and deploys to production. Expires at 20:14 UTC.

   ✅ Approve❌ Reject

   Veilleur·INC-74+84 s

   🚀 INC-74 · sncf-mcp: PR #38 merged and deployed as bb629fdcee72

   Approved by the on-call engineer; merge bb629fdcee72.

   Next: the alerts should resolve within a few minutes; then the postmortem.

4. 04 / 04

   Learn

   Writes the postmortem and updates its runbook with a lesson for the whole class of failure. When the same failure comes back, it recognizes it and proposes the known fix in one turn.

   Under the hood

   postmortem_write; the runbook is Markdown in its own git repository; memory_search matches by failure signature.

   Veilleur·INC-74+5 min

   📝 INC-74 · sncf-mcp: postmortem written, incident closed

   Root cause: a80001e8c303 indexed the first departure from an empty upstream result; the fix was deployed as bb629fdcee72.

   Impact: 19:43–19:46 UTC (2 min), 140 failed calls

   Took: diagnosis 19 s, reproduction 31 s, PR 47 s from the page; 21 tool calls

   Runbook: added R-1 "Assuming a collection returned by an upstream is non-empty" (03d6b0616f1b)

   Next time: Check logs_signatures, compare failures across deploys, inspect the top frame, and confirm whether the upstream returned an empty collection.

Demo video

From the page to the lesson, on a phone

What it shows

1. 1🚨 The page
2. 2🔎 The root cause, and what the model saw after redaction
3. 3🛠 The lab, the PR, one tap on Approve, 🚀 deployed
4. 4📝 The postmortem and the runbook diff
5. 5♻️ The same bug again
6. 6🌐 An innocent deploy: no rollback

Proof

16 runs on a live stack. Every root cause right.

correct root causesevery run, cold and warm memory

16/16

host-verified fix PRsin the code-bug runs; the upstream outage correctly gets no PR

12/12

false rollbacksand 0 false fixes, 0 failed agent turns

0

to the root causein every scenario, medians 18–24 s

~20 s

to a fix PR without memorymedians 78–176 s

1–3 min

to a PR when the same failure comes backtool calls 21 → 7 (scenario 1, cold → warm)

78 → 37 s

By scenario

Diagnosis / PR in seconds from the page, then tool calls. Medians of 2 runs.

Read these numbers honestly

• Memory speeds up the same failure, not the diagnosis: the root cause takes ~20 s even with an empty memory.
• For the same class of failure somewhere else, the runbook entry is found and guides the investigation, but the fix is not faster (88 → 80 s).
• 2 runs per cell: small numbers, stated as such.
• Scenario 3 (goroutine leak) was not built; the eval covers 1, 2, 4 and 5.
• The red herring in scenario 2 (it looks like the upstream is slow) fooled the model 0 times out of 4.

Safety

Nothing reaches production without your tap

• A single-use approval

  The approval is a single-use row tied to the incident and the exact commit sha, with a 30-minute TTL, spent atomically on merge. A T3 tool cannot even be registered without the approval gate.

• Buttons the agent never touches

  Approve and Reject come from a separate approval bot that only the host polls. The agent knows only a request id, and a request id alone grants nothing. Approve pressed after the incident was closed does nothing.

• No PR without proof

  Before pushing, the host re-runs both checks itself: the new test fails on the deployed commit, and the full suite passes on the fix.

• Bounded loops

  Hypotheses (3), replays, test runs, fix attempts and total incident time are budgeted in code. When a budget is spent, the incident goes to ESCALATED with the evidence for a human. Reject sends it back to hypothesis and costs one of the three.

• No false rollbacks

  An upstream outage is EXTERNAL_CAUSE: no fix, no rollback. The host refuses EXTERNAL_CAUSE when the logs show a Go change broke calls right after a deploy.

• Survives restarts

  State lives in SQLite. Every step is idempotent, and a restart mid-incident resumes where it stopped.

Privacy

Raw logs never leave the host

{
  "time": "2026-09-26T12:35:20.559143894Z",
  "level": "INFO",
  "msg": "tool call",
  "service": "sncf-mcp",
  "commit": "fc4caadef538",
  "request_id": "lg-74672",
  "tool": "next_departures",
  "args": { "station": "Marseille Saint-Charles" },
  "client_ip": "203.0.113.24",
  "user": "[email protected]",
  "api_key": "sncf_live_d450f297b385e17061c23dca",
  "outcome": "ok",
  "duration_ms": 1
}

{
  "entries": [
    {
      "time": "2026-09-26T12:35:20.559143894Z",
      "level": "INFO",
      "msg": "tool call",
      "service": "sncf-mcp",
      "commit": "fc4caadef538",
      "request_id": "lg-74672",
      "tool": "next_departures",
      "args": { "station": "Marseille Saint-Charles" },
      "client_ip": "[ip]",
      "user": "[email]",
      "outcome": "ok",
      "duration_ms": 1,
      "dropped": ["api_key"]
    }
  ],
  "redactions": { "ipv4": 1, "email": 1 }
}

Real output of make redaction, 26 Sep. The addresses are from documentation ranges, the domains are example ones, the key is made up.

• Layer 1 · allowlist of log fields

  A field the service starts logging tomorrow is dropped by name until someone adds it together with a redaction test.

• Layer 2 · 11 rules on every string

  Values under secret-looking keys are replaced wholesale. Go stack traces shrink to the service's own frames: build paths, goroutines and addresses stay home. Every rule and every allowed field has its own test.

• The lab has no network

  Replays run on an internal Docker network; tests run with --network none.

• The model is swappable

  Any OpenAI-compatible endpoint, including a local model, can be plugged in. The demo uses OpenAI gpt-6-luna. The API key stays on the host, in llm-proxy; the sandbox never sees it.

The 11 rules

• private keys
• JWTs
• bearer tokens
• credentials in URLs
• secret parameters
• emails
• phone numbers
• IPv4
• IPv6
• card numbers (Luhn)
• long opaque tokens

Architecture

The agent reasons. The host enforces.

The incident state machine

State changes go only through it, and every step is recorded in the timeline.

It learns

A postmortem after every incident, a lesson in its own runbook

45ecd53 INC-77: Navitia departures returned intermittent HTTP 503s
e8eb681 INC-76: …
de12c2e INC-75: plan_journey panics on an empty stop date-time list
03d6b06 INC-74: Empty departures slice crashes next-departure summaries

## R-1 · sncf-mcp: Assuming an upstream collection is non-empty

Matches: panic: runtime error: index out of range [N] with length N

- When: A tool panics with an index-out-of-range signature after reading the first item from an upstream collection; the upstream may return HTTP 200 with an empty list while other tools remain healthy.
- Check first: Use `logs_signatures` to identify the affected tool and top in-repo frame; compare failure counts across deploys, inspect the relevant code, then check the failing request for an empty upstream collection.
- Reproduce: Replay a failing request from `logs_signatures` against the deployed revision; add a focused Go test in the affected package with an empty upstream collection and confirm it fails on that revision.
- Fix: Handle an empty collection explicitly before indexing or deriving a summary; return a valid empty-result response, then run the focused regression test and the full Go suite.
- Avoid: Do not infer that a successful HTTP status means the response collection is non-empty; check empty-result cases before investigating unrelated tools.
- Seen: INC-74, INC-75, INC-76

Built with

Built solo, in about ten days

• OpenClaw
• NemoClaw
• OpenShell
• Model Context Protocol (TypeScript SDK)
• Node.js
• SQLite
• Docker
• Prometheus
• Alertmanager
• Go
• Telegram Bot API
• GitHub API
• OpenAI gpt-6-luna

Ilya Seliverstov built Veilleur alone for the NVIDIA Paris Claw Agent Challenge, in the OpenClaw harness category: OpenClaw inside NemoClaw / OpenShell.