test(evals): guard how agents tell threads from tasks and subagents#27138

Bottom of a stack: the fix that makes tt-background-agent pass is stacked on top of this PR.

Problem

Nothing in evals/ checks whether agents keep top-level threads, delegated tasks and native subagents apart. orchestration.ndjson has a few single cases, such as subagent-ask-no-new-threads and delegates-then-new-thread-with-model. But no case runs a task and a thread side by side, uses indirect thread wordings, has a child launch a thread, or treats "thread", "task" and "agent" as ordinary words in code. A copy change could mix up t3_thread_* and task_* and nothing would catch it.

The field evidence is mostly good behaviour. One month of one user's T3 threads (235 threads, 137 delegate_task calls, 156 native subagents) had 15 t3_thread_launch calls, and every one followed an explicit thread request. So most of these cases are regression guards. Building them turned up one real failure: asked to "spin up an agent in the background", Sonnet 5.5 starts Claude's native Agent with run_in_background:true. Claude Code's Agent description says agents run in the background by default, and T3's "Agent always with run_in_background:false" loses to it.

Change

• cases/threads-vs-tasks.ndjson: 23 cases (17 guards, 5 controls, 1 bad), all on authored Acme data:
  • thread asks become threads, including indirect wordings ("a separate thread I can follow", "another thread and a PR", "a new thread with an agent", a "new chat" on another provider, another project, three at once);
  • task asks stay off threads (a background worker with a PR, "in the background" with and without the word "agent", "another agent in parallel");
  • a running task and a running thread side by side: stop one and keep the other, and ask about each one's status;
  • children don't launch threads unless their task says to;
  • controls where "thread pool", "task queue", "Stop task" and "run the tests in the background" are ordinary code work.
• expect.newThreads counts the top-level threads an agent asks for, across t3_thread_launch calls and create_threads entries. Each value must match a different request, and nothing extra may be requested. Without it, one thread with its link printed three times passed the "three threads" case.
• Runner defaults for launched threads. T3's default t3_thread_launch and create_threads results now give each thread its own id, and t3_thread_read on that id returns its title and brief. Before this, every launch returned one id, and reading it showed an unrelated thread. Codex agents then "verified", saw the wrong thread and launched again: 8 of 12 GPT samples asked for 5–6 threads on a 3-thread request.
• A README section lists the cases, the results on main and one open wording question.

Scope and approval

Julius Marminge asked for evals on how agents distinguish threads from tasks and subagents. This PR changes only evals/. There is no product change, and CI only typechecks the eval package and runs the grader tests.

Verification

• vp run check:changed: format, lint, knip, typecheck and the run.test.ts grader tests pass. There are two new grader tests:
  • newThreads rejects one request, a request reused for two values, and an extra thread; it accepts a failed launch that was retried.
  • The thread defaults return distinct ids, and reading one returns its own brief.
• Paid runs on main's text, 3 samples per case. 1/3 and 2/3 cells were extended to 6 samples, and I read every failing transcript.

A full run of these cases costs about $4.4 on Opus and $2.1 on Sonnet, including the Haiku shell simulator. GPT models go through the CLIProxyAPI provider and aren't priced here.

Audit fixes before these numbers.

• One case required a t3_thread_read before a send. That's stronger than the text, and it's removed.
• Prompts carried two hints that gave the answer away; they're removed.
• One world contradicted its history; it's fixed.
• Status case: on Codex, the link in the answer was flaky on main (Sol 3/6, Astra 4/6), while routing passed in every sample. The case now checks routing only, and thread-mention-without-link-after-read still covers links.

Left out on purpose.

• One case was dropped because main's text doesn't decide it. The user asks to "ask the reviewer in thread <id> to re-check it", and that thread is a top-level thread the user opened.
• Sol and Astra start a new delegate_task review in 6 of 6 samples. They read "Every new delegate review or re-check is a new delegate_task" as covering every re-check. Claude models send to the thread.
• That's a wording question for maintainers, so it's noted in the README rather than kept as a case.

Limits. This is an API replay with a simulated shell and canned T3 results, so no child or thread actually runs. The cases measure routing choices, not delivered work.

Opus 5.5 and GPT-6.1 Sol designed the cases independently. I merged their designs, and both reviewed the merged result; their findings are fixed here.

Prepared with Claude Opus 5.5 (orchestrator) and a GPT-6.1 Sol + Claude Opus 5.5 council via T3 Code, in Claude Code.

🤖 Generated with Claude Code