Learn from forecasting mistakes: How to conduct effective post-mortems

Conducting post-mortems is a forecasting best practice, which includes examining your rationale, understanding its faults, and developing new strategies to make better forecasts going forward. Here's how...

1. Understand in detail how you approached your forecast.

Start by re-reading your initial forecast, since this will likely represent the main thought processes you used in forecasting this question. Identify the assumptions that went into the rationale, both explicit and implicit. Look at the evidence you cited, and examine the logic you used to connect that evidence to the forecast you submitted. At this point, avoid being overly self-critical, and just try to understand what you were thinking at the time. This can be difficult since you now have the benefit of hindsight, but it will help you later identify areas for improvement.

2. Identify the faults in your forecast.

With hindsight, it might be easier to see where you strayed. Were you forecasting a foreign election, and assumed that the dynamics would be roughly similar to the dynamics in your own country’s politics? Did you over-update on a single piece of information you found instead of more strongly anchoring to a base rate?

When doing this, try to avoid thinking that the thing that derailed your forecast was just one specific factor that you couldn’t have predicted. Accounting for the unexpected (perhaps by forecasting closer to the crowd, or base rates) is a key element of making a forecast. If you aren’t conducting pre-mortems during your forecasting process, adding that can be an area for improvement in your practice.

Additionally, try to think of how you could have updated more frequently or accurately during the question forecasting period. On your forecasting platform, you can set reminders to forecast at a repeated interval (weekly, monthly, etc.) or receive alerts when the crowd forecast changes by a certain amount.

3. Create a plan of action to improve for the future.

Developing a perfect 5-step plan to accurately forecast every time is impossible, so instead you can use heuristics and general guidelines to develop your forecasts. Using the faults you identified in your prior forecasting, you can add to or change the process you typically use. Maybe you made too many unstated assumptions in your forecast, so instead, try writing out a list of assumptions in your rationales going forward to help flag some inaccuracies that will hurt your forecasting if left unchecked. Or, if you just followed base rates without sufficiently updating on new information, in the future you might take a minute to examine how whatever base rate you are using might not apply to the question at hand. This process will be highly specific to your own forecasting style.

Remember that no matter how you perform on a particular question, there will be other questions to practice your skills and improve your ranking. Forecasting is a marathon, not a sprint, and hopefully with some of the steps outlined above, you’ll be able to improve your form if you stumble.

_________________________________________________________

If this article was helpful, check out how one of our team members conducted a post-mortem on a failed forecast in his write up here.