Hi there, argmin readers! Today’s post is a live blog of Class 12 of my graduate seminar “Forecasting: A Critical Retrospective.” The syllabus and list of past posts are here. It might also be a subtweet of yesterday’s AI discourse?
Twenty-five years ago, Leo Breiman wrote a prophetic warning to his statistical brethren:
“There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.”
Breiman saw that prediction was at the heart of most of his consulting projects. Rather than being asked to determine causation, he was asked to make forecasts. His list of examples is instructive:
Predicting next-day ozone levels.
Using mass spectra to identify halogen-containing compounds.
Predicting the class of a ship from high altitude radar returns.
Using sonar returns to predict the class of a submarine.
Identity of hand-sent Morse Code.
Toxicity of chemicals.
Online prediction of the cause of a freeway traffic breakdown.
Speech recognition.
The sources of delay in criminal trials in state court systems.
If you asked anyone today how they’d solve these tasks, they’d say “collect data and fit a neural net.” You had to be there at the time, but this was quite controversial in 2001. Indeed, even using a decision tree was controversial. Many people thought models built with expert knowledge, with meaningful parameters fit to data, would be far superior for prediction. Moreover, you’d be able to interpret your predictions because you had a mechanistic understanding of reality encoded your predictive stochastic data model.
Given what’s happened since, I don’t think anyone needs convincing that neural nets will work better. Indeed, the reaction in class yesterday confirmed that the zeitgeist has shifted 180 degrees. Of course you’d rather use a neural network than a complicated model. For all of those tasks, everyone would say “use a transformer or do in-context learning and forget about it.” Autoregression has won out by brute force.
Now, given my long history as a machine learning researcher, I don’t think you’ll be surprised to find that I agree with Breiman and struggle to justify stochastic models. However, I think I can capture the motivation of modelers: people want more from their models than predictions.
People like mathematical models because we think that understanding how something works from top to bottom lets us better predict how it will behave in the future. A mechanistic understanding not only lets us play out likely futures, but lets us imagine what might happen under a variety of different adversarial scenarios. These models assure us that we understand the present well, because we can point at all of the subcomponents and build a bottom-up description of macroscale behavior.
It’s so weird how our forecasting culture turns this around: IF we can make good predictions, THEN we understand. You present your Brier Score first. Only if it tops the prediction-market leaderboard are people impressed enough to go read your 25,000-word description of how you prognosticate. Accepting this logical inversion—that accurate predictions indicate understanding—is why our rational brethren venerate superforecasters. But it’s clear that even the most committed forecast bros don’t like this, because they also obsess about mechanistic interpretability.
But when we seek explanations, all we have is the data from which our predictions are made. Data-driven modeling is implicit, leaning on the tacit knowledge of what makes good data and what makes bad data. Almost everything you want to interpret lies in the data and your optimization objective. The compressed storage that are the parameters of your nonparametric model hold this information too, but reading the tea leaves of the weights is like trying to understand a book by looking at the bits in its zip file.
In a wholly data-driven world, better understanding can only come from getting new data. We can either wait for the future to roll in, or we can act to get that data now. That is, to rapidly understand the world you forecast, you have to be able to do things. You need to intervene and act to rule out explanations. This is experimentation.
Forecasts alone are not experimentation. Indeed, a recurring theme in this class is how passive forecasting is. Focusing so hard on prediction makes us forget we can take actions to change the future.
Some think that predictions can shape the world. By warning people of doom, perhaps someone out there will take action. And yet, my impression is that the most dedicated forecasters think their best available action is to place bets. Or write open letters. Abandoning their agency makes their prophecies self-fulfilling.

