Usually Wrong Can Be Right

By Ben Golden on September 09, 2015

We're developing some new tools to analyze forecasters' performance, biases, and ways to help them improve, and I've been digging into my own forecasting history on Inkling.   I've focused on a set of 3,343 forecasts I've made in questions that use an LMSR algorithm and have already resolved.

The first interesting finding is that most of my forecasts have been wrong.  That is, 56% of the time when I adjust a forecast, I'm moving it in the wrong direction.  (Yikes!)  This is especially likely to happen when I forecast upwards; 68% of the time, the event doesn't occur:

While it seems like my forecasts are making markets less accurate, this turns out not to be the case.  This is because my "incorrect" forecasts tend to make markets slightly less accurate, whereas my "correct" forecasts make markets much more accurate.  To explore this, I looked at how my forecasts affected markets' Brier Scores, our usual metric for forecast accuracy.

Brier scores evaluate markets by aggregating all the forecasts throughout their history, so I compared the Brier component of each market before and after I forecasted and calculated their difference.  The following chart shows the distribution of my forecasts:

While the majority of forecasts result in accuracy declines, the declines are relatively small, mostly resulting in less than a 0.05 decline in Brier Score.  In contrast, there are a substantial number of very large improvements, which more than offset the declines.  273 forecasts improve the Brier Score by at least 0.1, compared to only 76 that decrease it comparably.  In aggregate, my forecasts are making markets more accurate, even though the majority of my forecasts are in the wrong direction.

In upcoming blog posts, I'll dig deeper into the data to better understand which of my forecasts are most valuable, where I might be making mistakes, and how I can improve.

If you liked this post, follow us on Twitter at @cultivatelabs, and sign up for our monthly newsletter.
Ben Golden/@BenGoldn is an Engineer and Data Scientist at Cultivate Labs.

prediction markets crowdsourced forecasting