Use fallback BPE if the language model doesn't have one (#6848)

Release Notes:

- Added a fallback BPE if the language model doesn't have one.

---------

Co-authored-by: Piotr Osiewicz <24362066+osiewicz@users.noreply.github.com>
Co-authored-by: Marshall Bowers <elliott.codes@gmail.com>
1ab49fdbe6Todsaporn Banjerdkit committed on 1/30/2024, 3:42:03 AM· committed by GitHubparent9cb5a84
2 files changedLine totals unavailable