It turns out the code was not ported correctly from C and produced wrong
results for negative input values. As a bonus fix the NaN codepath by
adding yet another missing piece of code.

Spotted in #9047