New interface for dynamic and static files representing the model

The preprocessor currently creates dynamic and static files for representing the model (forgetting about the bytecode case which is sligthly different), which have the following two characteristics:

• the Jacobian matrix that they return is a dense one
• in the dynamic file, the column indices for the dynamic (endogenous) variables are organized so as to avoid columns of zeros; said otherwise, if there are n variables, there are less than 3×n columns (for endogenous) in the dynamic Jacobian. This design decision makes sense since the matrix is dense, but it complicates the computations with the Jacobian matrix.

The proposal is to change the design and have the dynamic and the static files:

• return a sparse Jacobian
• for the dynamic file, use exactly 3×n columns for the endogenous in the Jacobian (and also for higher order derivatives)

It is expected that having a sparse Jacobian will improve performance, at least for large models.

Since the static and dynamic files are used at many places in the code, the transition cannot be done overnight, and the idea is to have the old and the new interface coexist for some time. The old interface would be removed at some point (unless it turns out that there is a real use case for dense Jacobians).

The interaction with block decomposition also needs to be taken into account when doing this transition. Currently, when block decomposition is requested, the preprocessor outputs static and dynamic files that have the same name as in the case without block decomposition, but a different interface. This is bad design. The preprocessor should use different names for those files when using block decomposition. We could even go as far as always providing the two versions of those files (with and without block decomposition), so that the block option of the model block would no longer affect preprocessor output (but only the algorithms used for computations). An even further step could be to always use block decomposition for perfect foresight, and drop the other code paths (but we probably don’t want to do that for the stochastic case). In particular, this could help with large backward models, for which we only need (dynamic) derivatives with respect to contemporaneous variables, an optimization which is already automatically done when using block decomposition.

Steps:

• Change the preprocessor so that it produces the new static and dynamic files in parallel to the old ones
• Change the preprocessor so that it always outputs the block decomposed files (with the new interface) in parallel to the non-block decomposed ones? (see also preprocessor#41 (closed))
• Start using the new interface in the MATLAB/Octave code
• Drop the perfect foresight algorithms that do not use block decomposition? (i.e. block becomes the default, and only choice, for perfect foresight). This would probably require #1858 to be done first.
• Drop the old interface (if it is confirmed that dense Jacobians have no real use case)