The Forecasting Potential of Complex Models by Antonis Christofides

We are glad to have another guest post by Antonis Christofides (see his earlier post here)

Guest Weblog Post By Antonis Christofides

If you gave me the following paper after replacing the author’s examples of econometric and energy models with climate models, I could not have told it had been written in 1981.

 Ascher, W. (1981). The forecasting potential of complex models. Policy Sciences, 13(3), 247-267. doi:10.1007/BF00138485

Here are some extracts.

On the contrast between bad performance record and large volume of research:

Unless forecasters are completely ignorant of the performance  record, or are attracted solely by the promotional advantages of  the scientific aura of modeling, they can only be attracted to its potential benefits not yet realized.

On the difficulty of retrospective evaluation of model performance when there are competing scenarios:

When no scenario is designated as most likely, the scenarios must  be regarded as exogenous factors, whose likelihoods are not at issue in the modeling exercise. The model produces a set of projections, each posited as correct if the corresponding condition or scenario were to hold, but without implying that any particular one will hold or that some are more likely than others. In this case, the retrospective evaluation of forecast accuracy must proceed by first establishing which condition actually  prevailed, and then measure the discrepancy between the projection tied to that condition and the actual level of the predicted trend. If it is still too early to evaluate a set of conditional forecasts retrospectively, the spread of conditional forecasts of the same trend for the same year can be used as one indication of  uncertainty or minimum error, but only if the conditional is the same for every forecast of the set.

 On using model consensus to judge model validity:

[E]ven the agreement across models need not be an indication of  validity; they could all be wrong. For example, all energy models predicting the 1975 levels of U.S. electricity, petroleum, and  total energy consumption projected these levels higher than they actually turned out to be. This confident consensus was no guarantee that the models were correct then; any consensus among  models’ predictions in the future may be equally misleading.

… [S]imilar models undergoing similar judgmental censorship by  modelers holding similar outlooks on the future can so easily reassure all parties that the future is seen with certainty.

On using the fact that models are physically based as an argument for model correctness:

Complex models are formulated by specifying assumptions and hypothesized relationships as explicit, usually mathematical propositions. While this procedure is often very helpful in uncovering inconsistency and vagueness in the initial ideas or verbal formulations, it cannot establish the correctness of the model’s propositions. Models express assumptions, but do not validate them. If the modeler tries to ensure the validity of the  model’s propositions by focusing on disaggregated behavior of presumably greater regularity, the problem of reaggregating these behaviors to model overall patterns becomes another potential source of error. If the modeler only includes relationships proven  by past experience, there is no guarantee they will hold in the future. There is no procedure or format of model specification that guarantees the validity of this specification.

On the effort required:

Since rigorous, elaborate analysis [of models and their outputs]is time consuming and expensive, there has been a natural tendency  for forecasters to pour their efforts into grand, once-and-for-all projects, carried out only infrequently and yet used long after  they are produced because the immense effort makes them seem definitive.

On the likelihood of modelers to reconsider:     

[A]fter the modeler has spent years developing optimization  routines, apparent violations of … assumptions are more likely  to be accommodated by patchwork modifications, or disregarded altogether as short-term aberrations, than they are to trigger the abandonment of the model altogether.

… [M]odel revision, which seems to the cynic to be an ad hoc effort to keep a fundamentally misspecified model more-or-less in line with reality, is often regarded by the model builder as the normal routine of science.