Decide on treatment of prior truncation during estimation
We need to decide on how to proceed, see https://github.com/DynareTeam/dynare/commit/aa503abfa79145efc0840c79210e795e1b76df84
Activity
@JohannesPfeifer I checked your example and Dynare behaves as you say. Nevertheless, I still think that this behaviour is wrong and should be interpreted as a bug. I discussed this issue with @MichelJuillard and he agrees with me.
This behaviour is a bug because it was not intended on our part (I admit that this first reason is a rather weak argument). If users are not happy with the prior domains they must shift, extend or shrink prior domains (when possible). I agree that we should implement true truncation with another syntax (for the gaussian prior for instance). I have done part of this job somewhere (in a branch related to the new estimation interface). It would be nice to have it in the next release.
Another reason is that your argument is correct if and only if the number of estimated parameter is strictly less than two. If we estimate more than two parameters, a truncated parameter will also affect inference about the parameters because the weight of each parameter in the definition of the posterior kernel will change as we change the truncations. The truncation, as currently implemented in Dynare, affects obviously the model comparison but also the posterior distribution (in theory, I do not know if it makes a significant difference).
I will fix the bug asap.
@stepan-a I worry more about backward compatibility of existing mod-files than problems going forward. Whenever something is well-documented, changes should be possible.
But I still don't understand your second argument. What would be an example of incorrect inference that you have in mind? I don't doubt that truncation affects the posterior inference, but I thought in most cases with explicit truncation, this is actually the desired outcome.
Hi Johannes,
the problem is that with the current way truncation is taken into account in Metropolis, the (truncated) prior density doesn't integrate to 1.0 in some dimensions (the multivariate prior is orthogonal, so it must integrate to one in each dimension).
The proper way to introduce truncated priors is to scale up the density in order to integrate to 1.0 after truncation.
Best
Michel
Johannes Pfeifer writes:
@stepan-a I worry more about backward compatibility of existing mod-files than problems going forward. Whenever something is well-documented, changes should be possible.
But I still don't understand your second argument. What would be an example of incorrect inference that you have in mind? I don't doubt that truncation affects the posterior inference, but I thought in most cases with explicit truncation, this is actually the desired outcome.
Reply to this email directly or view it on GitHub: https://github.com/DynareTeam/dynare/issues/750#issuecomment-58178481
Michel Juillard
@MichelJuillard I get the fundamental point about the prior being improper. But intuitively, I have a hard time to see why it matters when tracing out the posterior density via MCMC (in the sense of providing incorrect inference as opposed to just not being the cleanest way to deal with the issue). The proportionality constant is definitly needed for model comparison, but I thought that the posterior distribution derived from the MCMC is still correct. Doesn't truncation just imply that there is a multiplicative constant missing that always cancels when computing the acceptance probability in order to satisfy the reversibility condition? Otherwise, every model in the literature with some prior mass in the indeterminacy/non-existence region (implicit prior truncation) would be providing wrong posterior distributions
The problem is that the truncated prior is only for one (or several) specific parameter. The prior density is therefore underestimated for that particular parameter.
You have a point when you remark that the situation is not much cleaner concerning the additional prior that the parameters must deliver a unique stable trajectory.
However, it is not an argument for multiplying questionable methodological treatments in Dynare when we can avoid it.
Either removing the prior truncation in Metropolis doesn't change much the results and the break in backward compatibility is minor. Or it makes a big difference and I can't see how to defend the procedure currently used in Dynare.
I also believe that some users trusted our initial intent: provide boundaries for helping optimization but without effect on the estimation result and we should deliver on that intent even if it somehow lost its way in the documentation.
Given the backward compatibility issue, we could add a warning whenever a code uses bondaries and Metropolis, pointing to the part of the documentation discussing the change.
Best
Michel
Johannes Pfeifer writes:
@MichelJuillard I get the fundamental point about the prior being improper. But intuitively, I have a hard time to see why it matters when tracing out the posterior density via MCMC (in the sense of providing incorrect inference as opposed to just not being the cleanest way to deal with the issue). The proportionality constant is definitly needed for model comparison, but I thought that the posterior distribution derived from the MCMC is still correct. Doesn't truncation just imply that there is a multiplicative constant missing that always cancels when computing the acceptance probability in order to satisfy the reversibility condition? Otherwise, every model in the literature with some prior mass in the indeterminacy/non-existence region (implicit prior truncation) would be providing wrong posterior distributions
Reply to this email directly or view it on GitHub: https://github.com/DynareTeam/dynare/issues/750#issuecomment-58181968
Michel Juillard
@MichelJuillard I did not try to make a point about implicit truncation. I am just trying to understand why intended truncation for one parameter is a problem incorrectly affecting the results from MCMC. I was just wondering that this problem would occur with implicit truncation as well, but I have never read anything about this in the literature (as opposed to the well-known model comparison issue already discussed in An/Schorfheide). Why is truncating only one parameter problematic? In that dimension there is a multiplicative constant missing, but acceptance ratios should still be correct. Moreover, the mode of the prior still is the mode. Sure, we are truncating part of the posterior as well, but that is intended. I must be missing something, but I honestly don't see what.
I am actually OK with changing the behavior and documenting the change. Most of the disagreement stems from that fact that I am less confident that users relied on your original intent. Was it ever communicated? I only know people who relied on truncation during MCMC and were not aware that it was a numerical trick for mode-finding. My interaction with Dynare users suggests that they rather relied on the implemented feature than on the intended one. But this might be due to me being more in touch with forum users than with the institutional users you are dealing with.
Lastly, I would like to be consistent on this issue. If truncation leads to incorrect posterior results, we should strictly disallow it. Hence, we should convert warnings about prior truncations to errors (#609) and offer ways to specify proper truncated priors.
@JohannesPfeifer, I think that you are right. I was wrongly thinking about prior densities when we are indeed adding log prior densities. Truncating one prior density has only consequences for overall integral, not for the relative weights of the various priors.
I guess that we can wait to have proper truncated densities to change behavior and documentation.
In the mean time, we can issue a warning about model comparison when boundaries are imposed on parameters.
Best
Michel
Johannes Pfeifer writes:
@MichelJuillard I did not try to make a point about implicit truncation. I am just trying to understand why intended truncation for one parameter is a problem incorrectly affecting the results from MCMC. I was just wondering that this problem would occur with implicit truncation as well (shuttung ), but I have never read anything about this in the literature (as opposed to the well-known model comparison issue already discussed in An/Schorfheide). Why is truncating only one parameter problematic? In that dimension there is a multiplicative constant missing, but acceptance ratios should still be correct. Moreover, the mode of the prior still is the mode. Sure, we are truncating part of the posterior as well, but that is intended. I must be missing something, but I honestly don't see what.
I am actually OK with changing the behavior and documenting the change. Most of the disagreement stems from that fact that I am less confident that users relied on your original intent. Was it ever communicated? I only know people who relied on truncation during MCMC and were not aware that it was a numerical trick for mode-finding. My interaction with Dynare users suggests that they rather relied on the implemented feature than on the intended one. But this might be due to me being more in touch with forum users than with the institutional users you are dealing with.
Lastly, I would like to be consistent on this issue. If truncation leads to incorrect posterior results, we should strictly disallow it. Hence, we should convert warnings about prior truncations to errors (#609) and offer ways to specify proper truncated priors.
Reply to this email directly or view it on GitHub: https://github.com/DynareTeam/dynare/issues/750#issuecomment-58190539
Michel Juillard
Just to be sure of the conclusion. The agreement would be that the second and third position arguments have to be interpreted as truncation parameters if a prior density is declared?
I still have difficulties with that (even if our, @MichelJuillard and I, second point was plain wrong). It would mean that:
(1) The first and second positional arguments after the name of the prior are not interpreted as the prior mean and the prior standard deviation. (2) The prior mean and prior standard deviations reported in the tables are wrong. (3) The comparisons of the displayed prior moments and posterior moments (often used to evaluate what we learn from the data) are a total nonsense.
Obviously we could correct the reported prior mean and prior standard deviation, but then we would not be very far from a proper implementation of the truncated priors.
I fully agree with you that having a consistent implementation is very desirable, but from my perspective not a top priority given the many unresolved major bugs. I would not do any short-term fixes until we fully fix the issue.
Thus, unless you, @stepan-a are planning to fix this in the next week or so, we should document the actual behavior of Dynare (just revert your commit) and leave it as is with all the unfortunate consequences you mention.
In the long-run, we transition to a proper truncated prior keeping the current syntax. This would get rid of problems (1) to (3) you mention and is the theoretically advocated solution resulting in proper priors. In terms of backward compatibility, this change would shift the log posterior by an additive constant, but would leave the posterior inference (mostly) as before. This is the closest we can get to backward compatibility and all users should be able to live with this.
Is there a need to offer more than truncated normal distribution?
A truncated gamma? A truncated inverted gamma?
Johannes Pfeifer writes:
I fully agree with you that having a consistent implementation is very desirable, but from my perspective not a top priority given the many unresolved major bugs. I would not do any short-term fixes until we fully fix the issue.
Thus, unless you, @stepan-a are planning to fix this in the next week or so, we should document the actual behavior of Dynare (just revert your commit) and leave it as is with all the unfortunate consequences you mention.
In the long-run, we transition to a proper truncated prior keeping the current syntax. This would get rid of problems (1) to (3) you mention and is the theoretically advocated solution resulting in proper priors. In terms of backward compatibility, this change would shift the log posterior by an additive constant, but would leave the posterior inference (mostly) as before. This is the closest we can get to backward compatibility and all users should be able to live with this.
Reply to this email directly or view it on GitHub: https://github.com/DynareTeam/dynare/issues/750#issuecomment-58555334
Michel Juillard
mentioned in issue #1593 (closed)
