Otherwise estimation with missing observations and prefilter option will crash (cherry picked from commit 92549e48)