Determining the Accuracy of Prediction Markets
By Brian Leslie on June 12, 2017
So, you asked a prediction market question, and the outcome is now known. The question has been resolved and winnings have been disbursed to the forecasters who held winning positions. Forecasters know how well they did based upon their profits in the question, and you know who your good forecasters were too. But how accurate was your organization at answering the question itself?
There are several things to consider when thinking about accuracy of the prediction market:
Comparing Prediction Markets to Existing Forecast
Do you have an existing forecast to compare your prediction market against? A prediction market isn’t a magic eight ball - it won’t tell you anything with 100% certainty. But if its forecasts are consistently better than your existing process of forecasting, then it’s a success. The good news is, when properly run, a prediction market is almost always going to beat out other forecasting methods.
Timeliness of the Forecast is Crucial
Having an accurate forecast after it’s too late to act upon isn’t going to help you very much. Getting the right signal with enough time to adjust course can mean the difference between success and failure.
Because of this, when evaluating the accuracy of the prediction market we look at what the forecast was over the life of the question, not just at single point in time. One of the great strengths of prediction markets is that the consensus forecast is fluid, and is able to change quickly as new information becomes available and new forecasts are made.
On top of offering accurate forecasts, prediction markets identify the individuals who were accurate in their forecasts. Knowing who within your organization or community to reach out to for insight on challenges can be just as valuable going forward as having good forecasts.
In a future post, I’ll break down exactly how we calculate scores and how this information can be used to improve the forecasting abilities of your organization. If you want a deeper look into the accuracy of prediction markets, check out Measuring Accuracy in Prediction Markets and Opinion Polls/Pools.
