Otherwise estimation with missing observations and prefilter option will crash

(cherry picked from commit 92549e48)