Introduction

At the heart of the crowdsourced forecasting process is the aggregation of individual forecasts into a “crowd” or “consensus” forecast. In the mechanism Cultivate uses to elicit forecasts, we collect a probability estimate in response to a forecast question. For example, on January 1, we might ask: “Will Team X win the tournament on February 15th?” and receive the following responses over the time period forecasters can make initial and subsequent updates to their forecast:

Our crowd forecast is the aggregation of those forecasts at any given time, where the aggregation method could be as simple as the mean of those forecasts, or as sophisticated as using decay and weighting strategies based on past accuracy performance.

Whichever aggregation strategy is in place, we only use an individual’s latest forecast. For example, if a user makes an forecast on day 1 that is used in the aggregation calculation, that user’s contribution to the aggregation calculation would change if they update their forecast on day 2+. The day 1 forecast would no longer be included in the calculation.

The following are aggregation options that can be configured for the Cultivate Forecasts platform. In general, anything additional beyond the default strategy is not applicable until your site is averaging at least 40-50+ forecasts/forecasters per question.

Default/Starting Aggregation Strategy

When a platform first launches or doesn’t have the critical mass mentioned above, the following aggregation strategy has proven to be the most optimal approach:

• Mean - average of the most recent forecast from each user;
• Establishing a minimum number of forecasts before applying any forecast decay;
• Setting a maximum number of forecasts to be included in any aggregation;
• Setting a minimum number of days worth of forecasts to include in aggregation, never decaying away very recent forecasts; and
• Once past the minimum number of forecasts, only keep a subset of the latest forecasts for the current aggregation calculation.

Decay Strategies

Given a set of forecasts, a decay strategy will eliminate some subset of those forecasts prior to applying the aggregation strategy to calculate a consensus. Decay options include:

• Constant - weights newer forecasts more heavily and reduces the influence of older forecasts
• N-Recent - keeps the most recent N forecasts, discards the rest
• Percent Recent - keeps the most recent x% of forecasts, discards the rest
• No Decay - keeps all forecasts

Weighting

Forecasts can be weighted more or less heavily in the consensus calculation based on:

• Accuracy - historical accuracy/Brier score of the forecaster
• Frequency - how often the forecaster updates his/her forecast in the question
• Training - whether or not the forecaster has completed a training course

Aggregation strategies

These are different ways to calculate a consensus from a set of forecasts. Options include:

• Mean - average of the forecast values for each answer
• Median - median of the forecast values for each answer
• Voting - take each forecast and round it to 0 or 1. Average the resulting 0’s and 1’s.
• Logit - an extremizing method that uses a weighted geometric mean to aggregate forecasts.