Teaching future government leaders how to assess policy decisions using decomposition
By Vanessa Pineda on April 03, 2024
Earlier this year, I had the privilege alongside my teammate, Henry Tolchard, to give a guest lecture to a graduate class at Georgetown’s Walsh School of Foreign Service about crowd forecasting and its application within the government. As part of that, we ran a mini-workshop using Cultivate’s decomposition methodology to simulate how we work with policy analysts to help them break down a complicated policy issue, like the impacts of the Israel-Hamas War, into forecastable signals that monitor possible future scenarios.
In the workshop, we asked the graduate students in the Forecasting and
Policy Making class, taught by Professor Lisa Bronson,
to tackle the complex question: Are we headed towards a scenario of
de-escalation or escalation of the Mideast conflict?
Students divided into groups to outline the signals that could indicate which direction the war was headed. Group debriefs went something like this:
To monitor if we’re headed towards de-escalation (or a ceasefire), we can look at international pressure, particularly the U.S. calling for a ceasefire, Hamas’ release of more hostages, cessation of hostilities, or negotiation activity.
To track escalation of the war, we could look to signals like Iran’s direct involvement and indirect involvement via proxies, Israel potentially opening up another front to the war with Hezbollah, or direct military support to Israel from the U.S.
From the list of signals they came up with, the students learned how to formulate specific forecast questions that could be posted on a forecast platform (e.g., Will Israel be at war with Hezbollah in the next 6 months?). They prioritized the most critical questions for the issue and practiced making probabilistic forecasts, talking through base rates, and challenging each other’s arguments for why they settled on a specific probability.
Professor Bronson, who is a former Deputy Undersecretary of Defense for Technology Security and Counterproliferation Policy in the U.S. Department of Defense, designed the curriculum and has taught the graduate class on forecasting at Georgetown since 2020. Bronson uses past U.S. intelligence lessons, such as the failure in assessing Iraq’s pre-war weapons of mass destruction program, and current policy challenges, such as the Middle East crisis, to teach students through crowd forecasting methodology how to challenge assumptions, combat bias, and think with more precision around high stakes policy decisions.
Core to the curriculum, students submitted forecasts and rationales over several weeks on questions related to the Mideast conflict on INFER, the crowd forecasting program supporting U.S. policy analysis run by RAND and Cultivate. Students saw firsthand the lifecycle of how their decomposition could be paired with forecasting, and how their contributions aid policy analysts and other stakeholders of INFER.
Influenced by her senior level roles in national security and intelligence, Prof. Bronson is an adamant proponent that crowd forecasting should play a role in intelligence assessments. Explaining why she designed this course, she said:
"A course on forecasting is an excellent vehicle for refining students’ critical thinking skills and encouraging rigorous self-examination of cognitive biases. It exposes students to a tool that could be an important component of an informed decision making process if more widely used within the U.S. government."
---
Vanessa Pineda is Managing Director at Cultivate Labs and helps clients run crowd forecasting efforts, including INFER by RAND.
