Fix check for QZ eigenvalue criterium

The criterium was previously incorrectly applied to the square absolute value of
eigenvalues. Rather apply it to the absolute value itself (as done in
mjdgges.m and the AIM solver).

Closes #1632

(cherry picked from commit dcac7992)