dynare issueshttps://git.dynare.org/Dynare/dynare/-/issues2023-10-02T15:40:12Zhttps://git.dynare.org/Dynare/dynare/-/issues/1859New interface for dynamic and static files representing the model2023-10-02T15:40:12ZSébastien VillemotNew interface for dynamic and static files representing the modelThe preprocessor currently creates `dynamic` and `static` files for representing the model (forgetting about the `bytecode` case which is sligthly different), which have the following two characteristics:
- the Jacobian matrix that they ...The preprocessor currently creates `dynamic` and `static` files for representing the model (forgetting about the `bytecode` case which is sligthly different), which have the following two characteristics:
- the Jacobian matrix that they return is a dense one
- in the `dynamic` file, the column indices for the dynamic (endogenous) variables are organized so as to avoid columns of zeros; said otherwise, if there are `n` variables, there are less than `3×n` columns (for endogenous) in the dynamic Jacobian. This design decision makes sense since the matrix is dense, but it complicates the computations with the Jacobian matrix.
The proposal is to change the design and have the `dynamic` and the `static` files:
- return a sparse Jacobian
- for the `dynamic` file, use exactly `3×n` columns for the endogenous in the Jacobian (and also for higher order derivatives)
It is expected that having a sparse Jacobian will improve performance, at least for large models.
Since the `static` and `dynamic` files are used at many places in the code, the transition cannot be done overnight, and the idea is to have the old and the new interface coexist for some time. The old interface would be removed at some point (unless it turns out that there is a real use case for dense Jacobians).
The interaction with block decomposition also needs to be taken into account when doing this transition. Currently, when block decomposition is requested, the preprocessor outputs `static` and `dynamic` files that have the same name as in the case without block decomposition, but a different interface. This is bad design. The preprocessor should use different names for those files when using block decomposition. We could even go as far as always providing the two versions of those files (with and without block decomposition), so that the `block` option of the `model` block would no longer affect preprocessor output (but only the algorithms used for computations). An even further step could be to always use block decomposition for perfect foresight, and drop the other code paths (but we probably don’t want to do that for the stochastic case). In particular, this could help with large backward models, for which we only need (dynamic) derivatives with respect to contemporaneous variables, an optimization which is already automatically done when using block decomposition.
Steps:
- [x] Change the preprocessor so that it produces the new `static` and `dynamic` files in parallel to the old ones
- [x] Change the preprocessor so that it always outputs the block decomposed files (with the new interface) in parallel to the non-block decomposed ones? (see also preprocessor#41)
- [x] Start using the new interface in the MATLAB/Octave code
- [ ] Drop the perfect foresight algorithms that do not use block decomposition? (i.e. `block` becomes the default, and only choice, for perfect foresight). This would probably require #1858 to be done first.
- [ ] Drop the old interface (if it is confirmed that dense Jacobians have no real use case)Sébastien VillemotSébastien Villemothttps://git.dynare.org/Dynare/dynare/-/issues/1858Use MEX to create stacked Jacobian in two-boundaries blocks with block decomp...2022-09-13T15:50:11ZSébastien VillemotUse MEX to create stacked Jacobian in two-boundaries blocks with block decompositionThe `perfect_foresight_problem` MEX was introduced so as to speed-up the construction of the stacked Jacobian for deterministic problems (when there are both leads and lags).
However, when doing block decomposition, there is no equivale...The `perfect_foresight_problem` MEX was introduced so as to speed-up the construction of the stacked Jacobian for deterministic problems (when there are both leads and lags).
However, when doing block decomposition, there is no equivalent MEX for constructing the stacked Jacobian for two-boundaries blocks. This is an obvious opportunity for optimization.
It remains to be seen how much can be factorized with the existing `perfect_foresight_problem` MEX. The bytecode case may also need special treatment.https://git.dynare.org/Dynare/dynare/-/issues/149Handle missing values in block decomposed version of Kalman filter2021-09-03T16:23:30ZSébastien VillemotHandle missing values in block decomposed version of Kalman filter