Using AI to improve your forecast rationale

Writing a quality rationale–a clear explanation that gives context behind your probabilistic forecast–can be challenging. In this post, we explore how you can use an AI tool like ChatGPT or Claude to gain a quick understanding of a complex forecast topic as well as improve the logical support behind your rationale.

Research support

AI tools can be valuable references in your initial research before coming up with your first forecast. They can help to summarize underlying context around an issue, give meaning to technical terms you may be unfamiliar with, or provide data on previous similar cases (base rates). Since you can ask follow up questions to the AI, they are often more useful for gaining a deeper understanding of a new topic compared to search engines, which might only direct you to sources that are either overly technical or insufficiently informative.

AI tools can also summarize lengthy, complex source materials. If you provide the AI with reports, academic studies, or news articles relevant to the forecast, then the AI can extract the most salient points and write them in an easily understandable format. This allows you to quickly incorporate key facts and data as evidence to formulate your forecast, without having to pore through dozens of pages of reports yourself.

Refining your argument

Perhaps the greatest value an AI tool provides is in improving the written logic, or argumentation, in your rationale. Argumentation, in general, relies on proving a claim by using reasoning which proves that the claim is true. This reasoning, especially in forecasting, often incorporates evidence from data, news reporting, etc. You can probe the AI about these concepts to improve the argumentative strength and rigor of your rationale.

• For example, these are some of the questions you might ask the AI about your rationale:
• What are the claims included in this rationale, and what reasoning or evidence is used to support each claim?
• What are the most likely circumstances or events that would have to occur for this forecast to be proven wrong?
• What are some logical gaps in my rationale that I may have overlooked?
• Are there any other factors that I failed to consider in my rationale?

Here’s an example of using an AI (Claude, by Anthropic) for rationale feedback.

Author: Below is a forecast value and forecast rationale I will submit in response to the question "Will Vladimir Putin cease to be the president of the Russian Federation before 1 March 2023?" The forecast will be submitted to a crowdsourced forecasting site on May 16, 2022. List the logical gaps in the rationale that I may have overlooked.

Outcome: Yes

Probability: 11%

Rationale for why 11%:

• +7% health: Starting with ~2-3% mortality for a man in Russia his age, adding a bit for rumors of ill health (updates on this may affect the forecast substantially)
• +1% popular uprising: Doesn’t appear to be much antagonism against Putin on the scale required for him to be ousted out of power (needs much broader opposition + ability to overthrow him)
• + (½ * 5%) elite ousting: Russian elites seem to be reaping less of the benefits of a Putin leadership than they are used to, so if some other circumstance (Putin ill health, popular opposition, military catastrophe, etc) allows them to swoop in and replace Putin without overwhelming opposition from the state, they might do so.
• + (½ * 2%) voluntary resignation: This, like the previous outcome, is probably dependent upon some other extreme negative outcome for Putin, but worth mentioning as this might be a way to sidestep one of the above 2 outcomes.

These outcomes are not all mutually exclusive, so I’m adding up the first 2 and adding half of each of the last 2 (as a rough estimate) to generate the forecast probability.

Claude: Here are some potential logical gaps I see in the rationale provided:

• The rationale focuses heavily on health and removal scenarios, but does not consider the possibility that Putin could lose re-election. While he has consolidated power, elections are still held in Russia and he could theoretically lose.
• The popular uprising probability seems low at just 1%. However, the rationale does not provide any evidence to support this number. Public opinion data or analysis of past uprisings in Russia could better justify this probability.
• The rationale for an elite ousting is reasonable, but 5% seems arbitrary without citing evidence of growing elite discontent. Quantifying recent defections, analyzing economic impacts on oligarchs, or surveying expert views could improve this estimate.
• For voluntary resignation, the 2% probability seems speculative without evidence that Putin has personal incentives to step down. Details on his motivations, personality and past behaviors could make this scenario more credible.
• The logic of adding the probabilities is unclear. Some scenarios seem dependent or mutually exclusive, in which case a simple sum may overestimate the total probability. Explicitly mapping out the relationships between scenarios could improve the combined forecast.
• There is no consideration of potential impacts of the Ukraine war on Putin's position. This major external factor seems highly relevant but is not discussed.

Overall the rationale provides a good starting point covering multiple angles, but could be strengthened by citing more evidence, avoiding arbitrary percentages, and mapping out dependencies between scenarios. Explaining the logic behind combining probabilities would also help justify the final 11% forecast.

In this example, Claude provided some actionable feedback, such as saying I should articulate reasons for the probabilities of the various scenarios I thought would be most likely to trigger a positive question resolution, which probably would have improved my model of the question topic. On the other hand, it also provided some weaker feedback; while Claude said that Putin might face electoral defeat, the next presidential election is not scheduled until after the end of the question period. It also included obvious or generic advice that wasn’t usable, like the last paragraph of the output.

Another area where the AI can be useful is in scanning your rationale and helping you to incorporate an argumentative writing technique called "signposting". Functioning like road signs for your writing, these are explicit signals that guide the reader through the structure and progression of the argument. You'll often see top forecasters use signposts like "When looking at base rates…" or "My forecast may be wrong due to…" This helps a reader, and the rationale writer themselves, see how the different pieces of the rationale fit together. An AI could review a draft of the forecaster’s rationale and suggest where signposts might be added to improve the clarity of the argument. By identifying these potential weak spots, the AI allows you to shore up your rationale preemptively.

As you work with AI to improve your rationales, be sure to refer to your forecasting site’s policy on using AI tools. We encourage the use of these tools responsibly, making sure you don’t take all of its information as fact without checking evidence, giving credit appropriately to the AI model used, and they should in no way replace your own unique perspective and authorship. AI tools can be of value as illustrated in this post – both for personal development and for the benefit of the broader forecasting community.