perf(server): reuse local submodule objects when creating worktrees#9854

closed3
main
GitHub fork: t3code/optimize-first-message-worktree-setup
openedSep 5, 2026byCClowdyffs
View on GitHubTwo-way sync enabledPR last imported 13d ago

fixing since i do a lot of work on a repo with a decent amount of submodule content, so everytime i start a new thread (I have it start a new worktree based on origin for new threads), and send my first message, it would take ~18 seconds to setup the worktree, this does some work to cut that down.

======= below this line is ai slop =========

What Changed

Fresh worktrees clone submodule history again even when the source checkout already has the objects. Initialize each destination submodule using its matching source checkout as a temporary Git --reference, then --dissociate so the destination owns its objects. Repeat for nested submodules. Missing objects still come from the configured remote, and unsuitable references fall back to an ordinary clone.

Git's submodule wrapper retains only the last --reference, so each clone now receives exactly one matching reference. Custom submodule active/update policies and clone concurrency settings retain the ordinary recursive command. If preparation still fails, retry the ordinary recursive update so Git can process remaining siblings before the existing best-effort warning.

The production diff is 98 changed lines in the shared server Git driver; the only other changed file is its focused test suite. No new settings, persistent cache, dependencies, or client/protocol changes.

Closes #9853.

Why

An observed first-message startup spent 0.522 s fetching origin, 0.296 s adding the worktree, and 17.417 s initializing submodules. This change targets the dominant submodule step while preserving automatic population and independent worktrees.

Worktree setup measurements

72.1% less time · 15.82 seconds saved on average

ImplementationRun 1Run 2AverageTime reduction
Before21.19 s22.71 s21.95 s0%
After6.20 s6.05 s6.13 s72.1%

Actual createWorktree() calls, including submodule initialization. Linux/btrfs · Git 2.55 · approximately 1 GiB of packed submodule history. Two runs per version, interleaved, using fresh destinations and the same pinned commit. Excludes origin fetch and provider startup; caches were not flushed.

Measurement method and limits

These measurements were taken on 57a6e533d, replacing the earlier implementation's measurements. The subsequent correction handles global submodule.active=true selection and failure fallback; it does not change the successful path exercised by this benchmark. Baseline: 8faf031c2dac4f147c43fe32e326b76678c20c94. Node 26.8.1, disk-backed destinations, the same remote, pack.threads=2, and nice 10. Execution order was before, after, after, before. The baseline driver was loaded from the original source with only benchmark import paths adjusted.

The measured driver call includes worktree creation, submodule preparation, base-ref configuration and cache invalidation. Each run checked the exact submodule HEAD and absence of persistent alternates. The original ~18-second trace and these benchmarks are separate samples on a shared machine with network variability. Two observations per version establish a local improvement, not a universal speedup; network byte savings were not measured directly.

The large-repository timing above measures one large submodule. The separate edge-case matrix below measures multiple/nested submodules, custom policies, and fallback overhead on Linux/btrfs. Local-path remotes may already benefit from Git's local clone optimizations; the new fixtures use file:// transport to avoid hardlink cloning.

Edge-case measurements on the final commit

The matrix below compares the original driver (8faf031c) with the unchanged PR head (1001b4cc0). 120 measured calls: 12 scenarios, five calls per implementation per scenario. Destination state matched the original driver in every measured call.

ScenarioBefore, meanAfter, meanTime reduction
1 cached submodule122.5 ms100.5 ms+17.9%
4 cached sibling submodules410.4 ms325.5 ms+20.7%
2 parents + 2 nested submodules347.3 ms306.4 ms+11.8%
12 tiny cached submodules240.4 ms404.5 ms-68.2%
Active pathspec: 2 of 4 selected205.3 ms193.1 ms+5.9%
Literal active=true pathspec109.2 ms116.8 ms-6.9%
update=none: 1 of 4 skipped267.5 ms284.9 ms-6.5%
Custom fetchJobs=4227.2 ms227.8 ms-0.3%
Stale + dirty source, 4 modules319.1 ms268.0 ms+16.0%
Shallow sources, 4 modules308.3 ms433.2 ms-40.5%
Uninitialized sources, 4 modules327.1 ms371.5 ms-13.6%
Unreachable first module + 4 healthy siblings326.5 ms347.7 ms-6.5%

Positive percentages mean less elapsed time; negative percentages mean more. These are synthetic local file:// transport fixtures on Linux/btrfs, separate from the approximately 1 GiB repository measurement above. Ordinary modules contain approximately 16 MiB of history each (four revisions of a 4 MiB random payload); the tiny modules contain a README only. Nested modules use spaces in their paths, and submodule names differ from their paths.

The slower cases have small absolute costs in these fixtures, even where the percentage increase is large. The command paths and packet traces explain the likely sources of overhead; the timings measure the whole driver call, not the milliseconds attributable to each subprocess:

  • 12 tiny cached modules: +164.1 ms (240.4 → 404.5 ms). The optimized runs made zero object want requests, so reuse worked. Each module still requires a separate update invocation, reference setup, and dissociation to preserve independence. With only README-sized histories to transfer, this setup costs more than the transfer work it saves.
  • Four shallow sources: +124.9 ms (308.3 → 433.2 ms). Git rejects shallow repositories as clone references, so the attempted reference clones retry as ordinary clones. The fixture shows no reduction in object requests; reference attempts and per-module setup add work. See Git's reference validation.
  • Four uninitialized sources: +44.4 ms (327.1 → 371.5 ms). There are no initialized source checkouts to borrow from. The driver checks for references and invokes updates per module, but each module still needs ordinary cloning.
  • An unreachable first module: +21.1 ms (326.5 → 347.7 ms). The per-module attempt fails before the driver retries the ordinary recursive command, which processes the healthy siblings. This fixture fails immediately at a local missing URL; a slow remote failure can make retries more costly.

Custom active/update/concurrency settings use Git's ordinary recursive path after the configuration checks. Their measured mean differences range from 12.2 ms less to 17.4 ms more; the fetchJobs=4 difference is 0.6 ms more, with substantial sample variation. Those small differences do not establish either an optimization or a precise overhead cost for custom policies.

For interactive first-message setup, the 44–164 ms penalties in the tiny/shallow/uninitialized fixtures are likely minor: they are paid once when creating a worktree, not on every subsequent message in that worktree. This is a practical assessment, not a measured perceptibility result. The separate large-repository sample saved 15.82 seconds; that is context for the tradeoff, not a promise that the same user receives both results. More modules or slower failures can increase the penalty, so the negative rows remain visible rather than being treated as universally negligible.

Method, behavioral checks, and every timing sample

Every sample runs the actual createWorktree() driver call, using a fresh disk-backed destination. Source fixtures are initialized before timing, and each scenario uses the same source and selected commit for both implementations. Execution order is before, after, after, before, before, after, after, before, before, after. Caches were not flushed; runs were serial on a shared machine, at nice 10, with pack.threads=2. Fixture setup, validation, and cleanup are outside the measured interval. Origin fetch and provider startup are excluded. Packet tracing is enabled identically for both implementations. Transport is local, so these fixtures measure no Internet latency or bandwidth savings.

For each destination, compare module initialization, HEAD, tracked-file lists, working-tree status, payload hash, and README presence against the baseline. Source status is checked after every call; the dirty-source scenario additionally checks the original edited file and excludes staged/untracked source files from the destination. Its selected commit is missing from the source reference, and the remote advances beyond that selected commit. Every initialized destination is checked for absence of persistent alternates. Cached single/sibling/nested/tiny cases make no object want requests with the optimization. The existing source-deletion/fsck tests remain separate coverage.

Active selection, update=none, and fetchJobs scenarios retain their selected/skipped modules. In the unreachable-module case, healthy siblings are cloned as with native Git, but Git may abort before checking out their files; successful worktree creation is not a claim of complete submodule population after a failure. All 120 calls matched the corresponding baseline state in these checks.

Times below are milliseconds, listed in execution order within each implementation. Means and percentages above use unrounded samples. Five observations describe these local runs; they do not establish a universal speedup or statistical significance for small differences.

ScenarioBefore: all five samples (ms)After: all five samples (ms)
1 cached submodule144.7, 117.7, 118.3, 116.3, 115.4111.5, 104.6, 96.4, 94.5, 95.5
4 cached sibling submodules374.3, 362.3, 377.4, 354.0, 584.2330.1, 328.8, 333.2, 320.8, 314.8
2 parents + 2 nested submodules348.9, 346.6, 348.3, 343.2, 349.3302.4, 305.0, 305.8, 315.4, 303.2
12 tiny cached submodules235.2, 248.4, 239.3, 244.7, 234.7396.9, 407.0, 398.6, 420.0, 400.2
Active pathspec: 2 of 4 selected185.0, 200.2, 255.2, 196.5, 189.5194.7, 191.5, 193.6, 198.0, 187.7
Literal active=true pathspec112.2, 108.1, 110.0, 104.4, 111.2108.8, 116.5, 117.0, 121.9, 119.5
update=none: 1 of 4 skipped258.3, 262.8, 271.5, 275.4, 269.6271.6, 271.0, 272.5, 274.3, 335.1
Custom fetchJobs=4153.8, 157.9, 161.3, 163.4, 499.7167.2, 417.0, 168.9, 166.4, 219.8
Stale + dirty source, 4 modules333.7, 318.4, 325.1, 310.2, 308.2272.8, 280.9, 268.5, 261.1, 256.6
Shallow sources, 4 modules306.6, 306.1, 311.1, 309.6, 308.1411.2, 409.7, 419.5, 512.1, 413.7
Uninitialized sources, 4 modules309.0, 308.5, 300.8, 361.5, 355.7354.8, 345.2, 380.0, 390.5, 387.0
Unreachable first module + 4 healthy siblings332.9, 326.2, 321.9, 326.7, 325.0351.0, 350.3, 345.7, 345.3, 346.0

Regression coverage

  • 65 tests pass in GitVcsDriverCore.test.ts on Node 24.20.0. The cached fixture uses two siblings and a nested submodule over file:// transport; packet tracing asserts no object want requests when the matching source repositories already contain all objects. This assertion fails on the original PR implementation (462f597) and passes with the correction.
  • Fixtures cover stale/populated, shallow and uninitialized sources; spaces and differing submodule names; a caller in a project subdirectory; a selected commit missing from the source; and a remote newer than the selected commit.
  • Staged, unstaged and untracked source changes remain untouched and are not copied into the destination. Destination submodules retain the selected gitlinks and pass fsck and reset --hard after source object stores are deleted.
  • Active-submodule selection (including the literal submodule.active=true pathspec) and update=none keep their ordinary Git behavior. An unreachable first submodule still allows Git to clone healthy siblings. Both added edge-case tests fail on 57a6e533d, pass against the original baseline, and pass with the correction. The existing best-effort checkout-failure test remains passing.
  • Targeted lint, formatting, server-only typecheck and git diff --check pass. Typecheck reports existing unrelated Effect suggestions, with no errors. No correctness regression was observed in this coverage. The edge-case measurements above record both speedups and timing regressions.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • Focused behavioral tests and before/after measurements are included

Model and harness: GPT-6 / Codex.


Note

Medium Risk Changes Git submodule initialization during worktree creation with new branching and fallbacks; incorrect reference handling could yield wrong commits or slow paths, though failures still degrade to the prior recursive update behavior.

Overview Worktree creation no longer relies on a plain recursive git submodule update when the caller’s checkout already has submodule repos locally. After git worktree add, the driver resolves the source repo top-level from input.cwd and runs new prepareSubmodules, which walks each submodule path, clones with --reference on the matching source checkout when a .git exists there, --dissociate so the new worktree owns its objects, then recurses for nested submodules.

Repos with non-default submodule.active / update / parallel fetch settings, missing references, or failed reference clones still use the original submodule update --init --recursive path (with the same best-effort warning if checkout fails).

Tests replace one happy-path case with a parametrized matrix (cached, populated, shallow, uninitialized, inactive, unreachable, update=none, etc.) covering nested submodules, pinned gitlinks, independence after deleting source object stores, untouched source dirty state, and packet tracing for the fully cached clone case.

Reviewed by Cursor Bugbot for commit 1001b4cc06ac0679c857f00bc1ab7f4bded51d2f. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Reuse local source submodule objects when creating worktrees in GitVcsDriver

Adds a recursive prepareSubmodules helper that clones each configured direct child submodule from the matching source submodule repository with dissociated objects, then recurses into nested children the same way. createWorktree now resolves the top-level source repository from the caller's working directory and invokes this helper before falling back to the ordinary recursive git submodule update.

  • When custom active/fetch-jobs/update policies are set, or when repository/gitmodules inspection fails, the code falls back to ordinary recursive git submodule update for the whole destination.
  • Individual per-submodule reference-clone failures fall back to a normal update for that child only; the overall worktree creation still completes with a warning if both population paths fail.
  • Expands the integration test suite in GitVcsDriverCore.test.ts to eight source-state scenarios (cached, populated, shallow, uninitialized, inactive, explicitly active, unreachable, update-disabled) with direct and nested submodules.
  • Risk: prepareSubmodules inspects .gitmodules and repo config to decide whether reference-based cloning is safe; if that inspection is wrong it could skip the optimization or produce a partial clone — verify behavior in GitVcsDriverCore.ts when custom submodule.active or submodule.<name>.update policies are present.

Macroscope summarized 1001b4c.

Checks

7 successful · 7 skipped · 1 neutral
All checks passed
1001b4c

Added 1 commit

  • perf(server): reuse local submodule objects when creating worktrees462f597
Synced from GitHub

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

ReviewStatusCommitReview trigger
📝 Code Review✅ Completed 2026-09-05T01:49:26.773979Z1001b4cManual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Cchatgpt-codex-connector[bot]reviewed · 1 inline threadSep 5, 2026
Synced from GitHub

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 462f597cbf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

apps/server/src/vcs/GitVcsDriverCore.tsline 2871Outdated · 462f597Open thread
……@@ -2849,19 +2849,46 @@ export const makeGitVcsDriverCore = Effect.fn("makeGitVcsDriverCore")(function*
2868+ const referenceArgs = references.length
2869+ ? [
2870+ "--dissociate",
2871+ ...references.flatMap((ref) => ["--reference", path.resolve(input.cwd, ref)]),

P2 Badge Avoid passing repeated references to submodule update

When the source has multiple initialized submodules, this emits several --reference flags. On Git 2.43, git submodule update -h documents a single --reference <repository>, and tracing this exact invocation shows its wrapper overwrites earlier values and forwards only the final reference to submodule--helper. Consequently every destination clone except, at best, the submodule matching that last reference still transfers its full object set; the tests pass because they verify checkout and dissociation rather than object reuse. Update each submodule with its corresponding reference or use a mechanism that preserves the mappings.

AGENTS.md reference: AGENTS.md:L15-L17

Useful? React with 👍 / 👎.

Addressed in 57a6e533d. Each direct child now receives only its matching source reference, then nested children are initialized with their corresponding source paths. Custom active/update and clone concurrency settings retain the ordinary recursive Git command.

Added a cached two-sibling plus nested-submodule fixture over file:// transport that asserts no object want requests. That assertion fails against 462f597 and passes with this correction; all 63 focused driver tests pass. The PR body now includes fresh measurements of the revised implementation and explicitly separates single-submodule timing from multi-submodule reuse coverage.

Synced from GitHub

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This production-path change replaces ordinary recursive submodule checkout with a new reference-reuse and dissociation algorithm for every worktree containing submodules. Although it has extensive integration coverage and preserves fallback behavior, the implementation is complex and changes existing runtime behavior beyond a small isolated tweak.

Notes:

  • Diff unchanged. Approvability was decided on eligibility alone.

You can add or adjust custom eligibility rules. Learn more.

Added 1 commit

  • fix(server): match submodule clone references by path57a6e53
Synced from GitHub

@codex review

The repeated-reference finding is addressed in 57a6e533d, including a transport-level regression test that fails on the previous implementation. This addresses the same object-reuse coverage gap raised in Macroscope’s approvability comment. The PR description has updated measurements and regression coverage for the revised implementation.

Cchatgpt-codex-connector[bot]reviewed · 1 inline threadSep 5, 2026
Synced from GitHub

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 57a6e533dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

apps/server/src/vcs/GitVcsDriverCore.tslines 2868–2869Outdated · 57a6e53Open thread
……@@ -2831,6 +2831,79 @@ export const makeGitVcsDriverCore = Effect.fn("makeGitVcsDriverCore")(function*
2866+ entries.some(
2867+ (entry) =>
2868+ /^submodule\.(active|fetchjobs|.*\.(active|update))\n/.test(entry) &&
2869+ !entry.endsWith(".active\ntrue"),

P2 Badge Preserve submodule.active=true selection

When repository config contains submodule.active=true, this suffix check treats it like the boolean submodule.<name>.active=true and enters the optimized path, which explicitly updates every path from .gitmodules. However, Git defines the global submodule.active value as a pathspec, so true selects a submodule whose path matches true; an ordinary recursive update leaves other submodules inactive. Repositories using this valid selection therefore clone submodules they intentionally excluded, potentially restoring the startup cost this change is intended to avoid.

Useful? React with 👍 / 👎.

Addressed in 1001b4cc0. The exemption now matches only submodule..active=true, so the global submodule.active=true pathspec uses ordinary Git selection. The new inactive-true fixture fails on 57a6e533d and passes on both the original baseline and the correction.

Ccursor[bot]reviewed · 1 inline threadSep 5, 2026
Synced from GitHub

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 57a6e533dd2d63281df484bf79aa1c0cbbfad3aa. Configure here.

apps/server/src/vcs/GitVcsDriverCore.tsline 2903Outdated · 57a6e53Open thread
……@@ -2831,6 +2831,79 @@ export const makeGitVcsDriverCore = Effect.fn("makeGitVcsDriverCore")(function*
2900+ yield* runGit("GitVcsDriver.cloneSubmodule", destination, args);
2901+ }
2902+ // Dissociation keeps the destination independent of source pruning/deletion.
2903+ yield* prepareSubmodules(reference, path.join(destination, submodule));
Ccursor[bot]on line 2903Sep 5, 2026

Failed submodule skips remaining checkouts

Medium Severity

The per-path prepareSubmodules loop stops at the first submodule update failure and never runs the remaining siblings or the recursive ordinaryUpdate fallback. A single unreachable submodule now leaves later modules empty, whereas git submodule update --init --recursive still continues and the previous driver retried that full command.

Reviewed by Cursor Bugbot for commit 57a6e533dd2d63281df484bf79aa1c0cbbfad3aa. Configure here.

Addressed in 1001b4cc0 by retrying the ordinary recursive update if preparation fails. The new unreachable-first-submodule fixture verifies Git still clones healthy siblings; Git may abort before checking out their files after retrying the failed clone. This test fails on 57a6e533d and passes on both the original baseline and the correction. All 65 focused driver tests pass.

Added 1 commit

  • fix(server): preserve submodule selection and failure behavior1001b4c
Synced from GitHub

@codex review

1001b4cc0 addresses both new findings: the global active pathspec and continuing ordinary Git initialization after a failure. Both regression cases were checked against the original baseline and the reviewed commit; all 65 focused driver tests, targeted lint/formatting, and the server typecheck pass. The full production diff remains 98 changed lines.

Synced from GitHub

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 1001b4cc06

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Synced from GitHub

Updated the edge-case measurement matrix with explanations of the slower cases. Recalculated the published numbers from the raw samples; the existing means, percentages, and absolute differences were correct. No code changed (1001b4cc0).

  • Twelve tiny modules add 164.1 ms despite zero object requests: per-module Git invocations, reference setup, and dissociation outweigh transferring their tiny histories.
  • Four shallow sources add 124.9 ms: unsuitable reference attempts fall back to ordinary cloning, with no reduction in object requests.
  • Four uninitialized sources add 44.4 ms: reference checks and separate updates run despite having no initialized source checkouts to reuse.
  • The immediately failing local-URL case adds 21.1 ms before the ordinary recursive fallback. Slow remote failures can cost more.

These explanations follow the command paths and packet traces; timings do not isolate each subprocess's contribution. The 44–164 ms overhead is likely minor during first-message setup because it occurs once per newly created worktree, not on each subsequent message. User perception was not measured, and larger module counts or slower failures can increase the cost. The separate large-repository sample saved 15.82 seconds; the PR retains both the benefits and the negative rows so reviewers can assess the tradeoff.

All 120 measured calls across 12 scenarios matched the baseline destination state in the documented checks. The PR includes all five samples per implementation and identifies the synthetic local-transport measurements as Linux/btrfs results.

Note

Written by claude-opus-5-5 on behalf of Maria

Hi! We are cleaning up open PRs, and this one appears to have been created with an older model (gpt-6). If this change is really important, we recommend rebuilding the PR with a newer model if possible.