I have a fix ready on TonybynMp4/t3code@fix/composer-append-findings. Each handoff from the PR panel (fixing a finding, or Add to agent on a diff selection) now appends its text and chips and keeps everything already in the draft. The PR reference chip is added once, not once per handoff.
The replace behaviour was deliberate in #4849, so I'll wait for a maintainer to confirm that appending is the behaviour you want before I open a PR.