Parallel worktree creation hits the fixed 5-minute git timeout and leaves orphan worktrees that block retry#15541

Triage

Thanks for the traces and for digging into the leftover checkouts, @areidyOTH. I checked this against current main, which has the same worktree code as nightly 73799330, including #15326. The timeout and the leftover worktrees are real. Two details differ from the report: the semaphore isn't what let all seven adds run at once, and Retry only fails the way you describe when the launch named the branch.

What I found

The 5-minute limit is fixed, and nothing caps these adds. GitVcsDriver.createWorktree always passes WORKTREE_ADD_TIMEOUT_MS (300_000) to runGitCommand, which matches the trace: runGitCommand is interrupted at 300001 ms. No setting changes that limit.

gitProcesses is a global semaphore of 8, but only commands with a timeout of 30 seconds or less take a slot. Longer or unlimited timeouts skip it on purpose so long checkouts don't starve short commands like status (#11405). git worktree add uses 300 seconds, so it never takes a slot. All seven adds ran together because nothing limits them, not because the limit is 8.

A timed-out add is missed by the cleanup from #15326. onWorktreeClaimed runs only after executeGit returns, and the launch service removes a worktree only after that callback has stored its path. A timeout ends inside executeGit, so the path is never stored and git worktree remove never runs, even though git has usually registered the worktree by then. Your trees were complete because they were killed near the end of a roughly 5-minute checkout. A kill earlier in the checkout could leave a partial tree that's orphaned the same way.

Retry reuses a worktree only if the thread recorded one. retryPreparation switches to existing_worktree only when both worktreePath and branch are set. Otherwise it runs the original strategy again.

• If the launch named the branch, as your HTTP launches did (t3/…), the retry runs git worktree add -b with the same branch and a path derived from it. Both already exist, so the retry fails. I confirmed that from the code, not in the app.
• If the launch didn't name a branch, the server picks a new t3code/<8 hex> name on each attempt, so Retry creates a second worktree and the first orphan stays behind.

Your --existing-worktree relaunch worked because that strategy never runs git worktree add.

Related

#11735 also leaves a worktree behind, but there the failure happens after git worktree add has returned, so the path is recorded and the #15326 cleanup can remove it. #11099 asks to attach a thread to an existing branch, which wouldn't stop concurrent adds from timing out.

Likely fix area

• Concurrency: one option is to limit or queue git worktree add per repository, separate from the short-command semaphore. Another is to time out on lack of checkout progress instead of total runtime.
• Timeout cleanup: either adopt a finished checkout that matches the requested branch and ref, or remove the worktree and branch git already registered.
• Retry: handle an existing branch and worktree when the failed attempt never recorded a path.

A maintainer will decide on the fix direction.

Reproduced the timeout and blocked retry on current main 89fe509232, using the web dev server on Linux and a 122,629-file fixture. Seven simultaneous orchestration.launchThread calls were accepted, then all seven workspace preparations failed with Git command timed out. The branches remained; retry failed with branch_already_exists.

This VM used ext4 for the source and tmpfs for the worktrees after an ext4 attempt exhausted free inodes. The complete, clean orphan worktrees reported on NTFS were not reproduced. Heavy VM contention stretched total preparation spans beyond five minutes, so this run does not establish the exact timeout timing.

Runtime transcript. Existing work on #17197 covers Git races and #18537 covers fetch timeouts; neither addresses the checkout timeout reported here.

Fix is available in #23802, labeled bugfix. The change removes the fixed five-minute worktree checkout deadline. Recorded real-server verification shows a 310-second checkout succeeding and cancellation stopping Git and its hook; all 30 current-head checks pass. The exact original storage workload remains unverified. The PR remains unmerged.