Short question: should the resulting quaternion between rotation of two vectors (0, 0, 1) to (0, 0, -1) being a undefined values (nan)? Or this is a bug.
local v1 = vmath.vector3(0, 0, 1)
local v2 = vmath.vector3(0, 0, -1)
local rot = vmath.quat_from_to(v1, v2)
print(rot) -- vmath.quat(-nan, -nan, -nan, 0)
If it’s going to change, I would just return nil or false. It can’t possibly give a specific value because it doesn’t know what plane to rotate around, so you’ll always need to check for it. nil or false would at least be easier to handle than a quat with NaN values, whereas an identity quat would mislead you into thinking that no rotation was necessary.
You can check for nan by comparing it to itself. It will not be equal to itself.