Sign up
View stargazersSign up

[Bug]: A copied state database wakes the original install's agents and can touch its worktrees#15513

openedOct 4, 2026byOoliver

Note

🤖 Claude Opus 5.5 on behalf of Oliver

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Summary

A copy of statev2.sqlite booted by a second server, such as a dev server seeded from real data, inherits the original install's automatic work. It does not check which environment that work belongs to. Startup recovery and the background workers then wake the original install's agents and can act on its real worktrees.

The environment ID lives in a file under T3 home (ServerEnvironment.ts), not in the database. The copy therefore can't tell that it isn't the install that wrote the data.

This is the problem behind #12879, which was closed as stale. A fresh fix on current main turned out to need a design decision, so I'm opening this to agree on direction first, per CONTRIBUTING.md.

Steps to reproduce

  1. Run a normal install with agents active, including at least one delegated subagent mid-run.
  2. Snapshot the live DB: VACUUM INTO from ~/.t3/userdata/statev2.sqlite into a fresh home's userdata/.
  3. Optionally set every running run in the copy to interrupted (column and JSON).
  4. Boot a server on that home, e.g. vp run dev in a worktree.

Expected behavior

A database booted by a different environment does not start provider turns, deliver messages to threads, or touch files on behalf of the environment that wrote it. Inherited automatic work stays suspended until someone explicitly adopts it.

Actual behavior

This happened on 2026-10-04. Two disposable dev servers were seeded from a live-DB copy to capture PR screenshots, with every running run set to interrupted and thread continuation turned off. On startup:

  • Startup recovery delivered "subagent interrupted" notifications to the parent orchestrator thread in the copy.
  • The resumed parent, running with full permissions, re-steered duplicate copies of its subagents.
  • For roughly 100 seconds, those duplicates ran real commands in sibling /tmp worktrees. One ran git checkout HEAD -- … and reverted another agent's staged files. Another ran rm on another agent's screenshots. One edited an evidence script.
  • Separately, four resumed claude sessions ran for about 19 minutes in real checkouts of other projects, and left uncommitted edits in one of them.

Nothing was pushed. The only defence was a human noticing.

Paths that wake inherited work today

Each of these acts on a copied database with no ownership check:

PathHow it fires
Restart continuationProviderRuntimeRecoveryService → restartContinuationRun / continueRestartedRun (RestartContinuation.ts) posts "Continue where you left off." into interrupted runs.
Delegated completion delivery#15323's delegation recovery records interrupted or cancelled child results and offers a parent wake through the terminal-run listener (Orchestrator.ts). This is the path that fired in the incident. It ignores continueThreadsAfterServerUpdate, which defaults to off.
Checkpoint-waiting childrenA child run waiting on a pending checkpoint capture is kept by recovery. When CheckpointCaptureService completes it after activation, the parent is woken.
PR watchesPullRequestWatchReactor reads persisted watches and turns check, comment and conflict changes into message.dispatch.
Scheduled tasksScheduledTaskService selects due tasks by enabled state and time only. It launches or sends using the stored project and workspace binding, whose absolute paths can point into the original checkout.
Usage-limit auto-resumePersisted limitRecovery.autoResume is enough for UsageLimitRecoveryWorker to post "Continue where you left off." at reset time, regardless of the copy's settings.
Pending rollbacksprovider-thread.rollback is replayable. CheckpointRollbackService reopens the stored provider session and restores checkpoint files. On the same machine, those are the original worktree's files. CheckpointRestoreSafety only checks threads inside the copied DB.

Held queued runs are already safe: promotion is rejected while any held run exists.

Proposed direction

1. Record which environment owns the database. Add one owner row, orchestration_v2_runtime_owner, in a new migration (057 is currently free). Write it after startup recovery completes.

2. Quarantine inherited automatic work instead of enumerating recovery paths. A one-boot cleanup pass isn't enough: later checkpoint completions, watches, schedules and limit resets can recreate wakes after the owner row has been overwritten. Instead, mark inherited automation as suspended on a known ownership mismatch. Enforce that where it takes effect:

  • automatic message admission (continuations, delegated deliveries, PR-watch wakes, scheduled sends, usage-limit resumes)
  • external-effect execution (rollback and restore)

Suspended work keeps its configuration and data. It isn't deleted.

3. Make adoption explicit and visible. Show suspended work and let the user adopt it ("this is my install") or dismiss it. Never silently drop it.

4. Distinguish three ownership states so existing installs aren't punished on upgrade:

  • Matching owner: behave as today.
  • Known mismatch: suspend all inherited automation.
  • Legacy, no owner recorded (every install's first boot after this ships): the database can't prove where it came from. One proposed heuristic is to record whether the home's environment-id file already existed before this boot. A nonempty database plus a freshly generated identity looks like a copy into a new home, and gets suspended. A pre-existing identity looks like a normal upgrade, and gets adopted. Copying an unowned database into an already-initialized home remains ambiguous without an independent database-to-home binding.

Questions for maintainers

  1. Is a persistent quarantine of inherited automation acceptable, or should ownership live on each record instead? Per-record ownership would mean environmentId on runs, schedules, watches and limit recovery. On the old #12879, putting it on the client OrchestrationV2Run contract was questioned.
  2. Should scheduled tasks and usage-limit resume be included, given that suspending them on a mismatch changes product behaviour?
  3. What should a legacy, unowned database do on first boot: adopt on the identity-file heuristic, suspend and prompt, or something else?
  4. Where should suspended work surface in the UI?

Impact

  • Correctness and safety: a copy can start provider turns against real repos, revert files, and restore checkpoints in the original worktree.
  • Who hits it: anyone who copies a DB for testing, support or migration. That includes maintainers and agents following AGENTS.md's "copy real data" advice, and dev servers seeded from real data. AGENTS.md currently recommends VACUUM INTO snapshots of the live DB for test data. Until this is fixed, that advice is unsafe while agents are running.

Version or commit

main at c5bc98db03 (2026-10-04).

Prior work

  • #12879: original PR, closed as stale.
  • A rebuild on current main exists as three branches on my fork, all reviewed:
  • An adversarial review found the checkpoint, PR-watch, schedule, usage-limit and rollback paths above still open. It concluded that per-path cleanup is the wrong shape, hence this issue.
  • The owner-row branch is reusable as groundwork whatever direction is chosen.

Workaround

Never boot a copy of a live database while agents are running. Use an empty T3 home and create fixtures through the UI.

JJulius MarmingeOct 4, 2026

Note

Grok responding on behalf of Julius.

Triage

Thanks for the careful incident write-up and for opening this to agree on direction first, @flamboh. I could confirm on current main that a statev2.sqlite snapshot booted under a different T3 home can start provider turns and write to the source install's worktrees. The environment id lives only in the file at ServerConfig.environmentIdPath (ServerEnvironmentIdentity in apps/server/src/environment/ServerEnvironment.ts), and nothing in the database records which environment wrote it. 056_RemoveRedundantProjectionIndexes is the latest migration, so 057 is free.

What I found

The incident path. This one ignores continueThreadsAfterServerUpdate (default false in packages/contracts/src/settings.ts).

  • Marking runs interrupted makes those children show up in ProjectionStore.getRecoveryThreadIds("subagent-results") whenever the latest run is terminal and there's no subagent_result transfer.
  • Orchestrator.recoverDelegatedTasks then calls finalizeAppOwnedSubagent, which offers a parent wake. ProviderContinuationService dispatches it as message.dispatch with queue_after_active, and since an interrupted parent has no active run, the wake starts a provider turn on the stored native session and absolute worktree path.
  • continueRestartedRun does re-check the setting and won't post "Continue where you left off." when it's off, but its finalizer still calls recoverDelegatedTask. The DelegatedCompletionDelivery "continuation declines to start" test currently locks that in.

The other automatic paths you listed are real too, and none consult an environment owner:

  • A waiting run with a pending or running checkpoint.capture effect is kept by ProviderRuntimeRecoveryService. That effect is replay-safe (EffectOutbox.REPLAY_SAFE_EFFECT_TYPES_AFTER_PROCESS_LOSS), and when capture completes, handleTerminalRun can finalize the child and wake the parent.
  • provider-thread.rollback replays the same way. CheckpointRestoreSafety only checks other threads in the same database, so a copied thread can restore files at the original cwd.
  • PullRequestWatchReactor sweeps persisted watches and can dispatch a wake on any unsettled thread.
  • ScheduledTaskService.runDueTasks fires from enabled state and next_run_at, using the stored project and workspace binding.
  • limitRecoveryCommand posts "Continue where you left off." from a persisted limitRecovery.autoResume even when the booting server's autoResumeLimitedThreads is off.
  • Held queues are already safe, as you noted: startup reconcile sets queueHeld, and startNextQueuedRun won't promote while a held run exists.

On AGENTS.md: the documented seed is vp run migrate-dev-db with the dev server stopped, not a raw VACUUM INTO into a fresh home. That command snapshots with VACUUM INTO, then drops the families recovery would touch (runtime, queued-runs, subagent-results, delegated-completions, a latest failed run or any limitRecovery) and deletes scheduled_tasks and orchestration_v2_effect_outbox. The incident used a raw copy, which skips that prune. The prune isn't a full ownership check either: an active PR watch on a stopped, unsettled thread survives it and can still wake.

On #12879: it was closed because the branch went stale, not because the problem was rejected. The earlier review note about not adding environmentId to the client OrchestrationV2Run contract still seems relevant; storing it on the run row without publishing it is one way around that.

Likely fix area

Some constraints that came out of the code, for whichever direction is chosen:

  • Timing: the ownership check probably needs to run before startup recovery, delegated recovery, effect replay, and the background workers. Writing orchestration_v2_runtime_owner only after recovery finishes wouldn't protect the boot that writes it, which is the risky one.
  • Upgrades: a pre-existing environment-id plus an unowned database could keep today's behaviour, while a nonempty database with a freshly created home identity looks like a copy. Copying an unowned database into a home that already has an identity file stays ambiguous, and suspending is the more cautious option there.
  • Scope: once an owner row exists, the paths above (including schedules and usage-limit resume) all go through the same unchecked admission, so they look like parts of the same bug.
  • migrate-dev-db pruning seems worth keeping regardless.

Your questions about per-record vs. database-level ownership, legacy first-boot behaviour, and how suspended work surfaces in the UI are the open design choices here.

A maintainer will decide on the fix direction.

Ttheo20d ago

Note

🤖 Claude Opus 5.5 responding on behalf of Theo

Thanks for the report. One part of this is fixed on main in #22744. Please test on the nightly built from main on 2026-10-21, or on the next stable release. One part is still open: Copied databases outside the dev seeding path can still replay automation against the source worktrees. If that part still matters to you, please open a new issue for that part only.