« The Ultimate Guide to Prediction Markets

A Brier Score is a way to judge and score the accuracy of probabilistic forecasts. For instance, if I say there is a 90% chance that the Cubs will win the World Series in 2017 (a probabilistic forecast), and then they subsequently win the World Series, I was "mostly" correct. But I also implicitly said that there was a 10% chance that the Cubs would not win, so that 10% allocation was "wrong." How do I quantify how "correct" my 90%/10% forecast was?

A Brier Score gives us a way to score these forecasts so that we can compare their accuracy.

You'll often see a Brier Score referred to as the **mean squared
error** of a forecast. Let's break that term down and by showing how a
Brier Score is calculated.

Let's say I made my 90% Cubs win forecast 1 week before the World Series ended (with the Cubs winning). We start by calculating the "squared error" for each of the 7 days that my forecast was valid. To calculate the squared error, we simply square the difference between my forecast and the actual outcome (100% if the Cubs win, 0% if they lose):

(0.9 - 1.0)^2 = 0.01

And for my implicit 10% Cubs lose forecast:

(0.1 - 0.0)^2 = 0.01

So my total error for the day:

0.01 + 0.01 = 0.02

To calculate the mean squared error, calculate the squared error for each
day that my forecast was running and average them:

Day | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 |
---|---|---|---|---|---|---|---|

Forecast (win, lose) | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 |

Error | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 | 0.02 |

We then take the average (aka mean) daily squared error to get a final
score. In this case, since all of the days are the same, my mean squared error
ends up being equal to **0.02**.

What would have happened to my Brier Score if I had updated my forecast mid-way through the week? Maybe the Cubs won a game on Day 3, leading me to update my forecast to 95% on Day 4. For any days where I had a 95% forecast, the squared error would be:

(0.95 - 1.0)^2 = 0.0025

And my new, implicit Cubs-lose forecast of 5%:

(0.05 - 0.0)^2 = 0.0025

So my total error for those days:

0.0025 + 0.0025 = 0.005

So my daily squared error would look something like:

Day | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7 |
---|---|---|---|---|---|---|---|

Forecast | 0.9, 0.1 | 0.9, 0.1 | 0.9, 0.1 | 0.95, 0.05 | 0.95, 0.05 | 0.95, 0.05 | 0.95, 0.05 |

Squared Error | 0.02 | 0.02 | 0.02 | 0.005 | 0.005 | 0.005 | 0.005 |

So my mean squared error would be:

(0.02 + 0.02 + 0.02 + 0.005 + 0.005 + 0.005 + 0.005) / 7 = 0.0114

In the world of Brier Scores, a lower score (ie. less error) is better, so
by updating our forecast mid-way through the week, we improved our score from
**0.02** to **0.0114**.

Go back to The Ultimate Guide to Prediction Markets.

Use prediction markets to learn what your people *really* think is going to happen with your most important projects and metrics.

Use Kickstarter-style internal crowdfunding to prioritize what you should be focusing on at the front-end of your innovation process.