We reallocate quite a bunch in this codepath even though we don't need
to, we already roughly know what number of elements we are working with
so we can reduce the required allocations to some degree. This also
reduces the amount of anchor comparisons required.

Came up in profiling for
https://github.com/zed-industries/zed/issues/44503

Release Notes:

- N/A *or* Added/Fixed/Improved ...