fold_point_to_display_point calls to_offset on the fold point, which
panics if it hasn't been clipped.

https://zed-industries.slack.com/archives/C04S6T1T7TQ/p1694850156370919

Release Notes:

- vim: Fix a crash when moving up/down in some circumstances.