Q&A with Cultivate: A look into our forecast question development process
By Cultivate Labs on June 29, 2023
One of the first steps in a crowdsourced forecasting effort involves establishing a process for developing forecast questions that will deliver meaningful signals to analysts and decision-makers. Cultivate works with clients throughout the entire process of question development: scoping an issue, decomposing that issue into forecastable signals, refining the questions that participants will see on the platform, and ultimately "recomposing" all our signals on an ongoing basis to indicate what scenario is more likely to play out.
We wanted to shed some light on the work our teams embark on, so we talked to a few of our team members that focus on developing questions for our client platforms. Meet the team members who participated in our Q&A below…
- Thalia Shamash Baumgarten, manager at Cultivate, has more than five years of experience with crowdsourced forecasting and currently leads the question development team for INFER, a Cultivate engagement supporting U.S. Government stakeholders. In past roles at MITRE and Booz Allen Hamilton, Thalia was part of an IARPA research team that pioneered crowdsourced forecasting techniques for the U.S. Intelligence Community.
- Dante Delaney is a consultant at Cultivate focused on developing forecast questions, managing the question lifecycle, conducting data analysis, and generating comprehensive reports. He was formerly a Research Specialist at Gartner and a Senior Research Support Associate at the MIT Governance Lab.
- Katie Cochran works part-time with Cultivate as an advisor, and is instrumental to the question development process. She has been involved in forecasting research and its application to national security challenges for over a decade, including with the Good Judgement Project, Good Judgement Inc, CSIS, and CNAS. Katie now runs a strategy and analytics consultancy focused on forecasting and nonprofit management.
Q: Let’s start from the top. Can you describe the goals of Cultivate’s question development process?
A: At a high-level, the primary goal of this process is to produce forecasting signals and trends that will aid analysts and decision-makers. In order to get those results, we have to start by asking the right questions that serve 1) those who seek this information to bring rigor to their analysis, and 2) forecasters who need to be engaged enough by the questions to participate. We’re constantly in a balancing act between these two groups.
Q: How do you come up with the questions asked on the forecasting platforms?
A: We typically identify questions in two ways: 1) walking clients through our strategic question decomposition process, in which we break down high-level questions (e.g. “impacts of climate change on coastal communities”) into more concrete signals to forecast. And 2) asking specific questions based on existing priorities that need additional research and refinement.
Q: What are the different elements you have to define about a forecast question besides the question itself?
A: A big part of question development includes drafting background information, or general context and explanation about the question, which is presented alongside the question. For example, we might ask a general audience to forecast on technical topics, such as advanced technologies and international diplomacy, so it’s important to provide context and resources about the question for people new to the topic. The background information also includes resolution criteria, where we define terms and data sources that will be used to resolve and score the question once the answer is known.
Q: Let’s get into question resolution a bit more. How do you ensure that questions have clear resolution criteria and what do you do in more tricky situations where the outcome is “fuzzy”?
A: Since the real world is messy, and answers to important questions aren’t always clear-cut, we do our best to consider all possible outcomes before launching a question. That exercise helps us identify any edge cases or outliers that we can specify in the resolution criteria. We also identify clear time constraints and try to be as specific as possible about the resolution sources we will use to resolve the question.
Numerical data is relatively simple to resolve, but for questions that rely on qualitative reporting, the process can take a little longer to confirm. Sometimes, there are situations where forecasters might interpret an outcome differently, and that’s when our team weighs all evidence and comes to a conclusion that is as close as we can to honoring both the original intent of the question for the decision-maker and what we’ve previously communicated to forecasters through the resolution criteria. We also encourage forecasters to submit requests for clarifications while a question is active, and our team will post those clarifications publicly as needed to minimize any confusion.
Q: You mentioned some forecast questions are formulated from issue decomposition. What is issue decomposition and how is it related?
A: Sometimes the issues that are integral to a client’s decision-making process are high-level and intangible, and include many interrelated components and risks. For example, a U.S. Government client may be focused on understanding an overarching question such as: Will the U.S. gain competitive advantage in the global AI race? We can’t forecast this directly, but we can “decompose” it and break this question down into individual drivers that experts say will influence the outcome. These drivers can then be decomposed further into a portfolio of “signals” that can be used to understand directionally the likelihood of a scenario playing out. Going beyond the forecast questions themselves, the output can be a “recomposition” of forecasted signals, where we show a single, re-aggregated snapshot of directional movement of forecasts over time for the entire portfolio of signals which generates a quantitative view of what was originally a qualitative question.
Q: Given the technical nature of the issue decompositions and the resulting questions, are you all experts on the topics you tackle?
A: No—it’s very hard to be experts on all forecasting topics! The clients and stakeholders we work with have varied interests, so you’d be hard pressed to find any one person that is an expert on all our site topics. For example, in our forecasting programs with government clients, they may want to prioritize national security issues, such as the impact of advanced technologies, international politics and diplomacy, and global supply chains. When we’re working with commercial clients, their forecasting program may be focused on market competitiveness or tracking and monitoring their specific product portfolios. It then becomes much more important for us to have expertise in the question development process itself— getting up to speed quickly on complex topics and understanding what makes an effective forecast question. When there is a need for highly specialized input, we often consult with subject matter experts from client or partner organizations.
Q: In the crowdsourced forecasting space, there is an acknowledgment that forecast questions by their specific nature don’t match up with the “big picture” thinking of decision-makers, so it can be challenging to know what to do with the forecasts generated. How do you bridge the gap between that higher order look of the world and the specificity of forecast questions and signals?
A: That’s something our team thinks about a lot. And, our issue decomposition process is exactly that bridge — not only does it ensure that specific forecastable signals are aligned with the issue or question, but when we put reports together for decision-makers, we will “re-aggregate” all the different individual forecast results so they can more easily see how directionally the “bigger picture” about the future is trending.
Q: Lastly, let's discuss a question we get asked frequently at Cultivate. Some of the most useful questions for decision-makers can be very technical for the participating crowd, who may not find them very engaging or know enough about the topic. How do you balance the technical nature of the forecast questions with ongoing forecaster engagement?
A: This is an interesting challenge that our question development team doesn’t think about in isolation. Instead, it’s also part of the broader engagement strategy for any crowdsourced forecasting program. We know that some forecast questions might seem narrow or esoteric when taken out of context to participants who aren’t privy to the full breadth of the issue decomposition. Decision-makers or analysts may also have technical questions about milestones or thresholds that you would not find being discussed in popular media! Although participants may be excited about forecasting AI’s impact on disinformation, for example, jumping into a forecast question related to specific legislation around large language models can be intimidating and less exciting.
To encourage participation in these kinds of questions, Cultivate often has a dedicated engagement team or works with the client to think about how to “activate” participants around those kinds of technical questions. Leveraging tactics like blog posts, email campaigns, webinars with experts, and themed reward challenges, we can educate participants on a topic, create excitement around it, and reinforce the value proposition to drive participation. Ultimately, this is a consideration that has to be thought of as part of the bigger forecasting ecosystem.
//
If you’re interested in working with Cultivate to leverage the collective intelligence in your organization through a crowdsourced forecasting program, please reach out!
leadership forecasting enterprise crowdsourcing Cultivate Labs innovation crowdsourced forecasting
