What is a Brier Score and How is it Calculated?

« The Ultimate Guide to Prediction Markets

A Brier Score is a way to judge and score the accuracy of probabilistic forecasts. For instance, if I say there is a 90% chance that the Cubs will win the World Series in 2017 (a probabilistic forecast), and then they subsequently win the World Series, I was "mostly" correct. But I also implicitly said that there was a 10% chance that the Cubs would not win, so that 10% allocation was "wrong." How do I quantify how "correct" my 90%/10% forecast was? 

A Brier Score gives us a way to score these forecasts so that we can compare their accuracy.


Calculating a Brier Score

You'll often see a Brier Score referred to as the mean squared error of a forecast. Let's break that term down and by showing how a Brier Score is calculated.

Let's say I made my 90% Cubs win forecast 1 week before the World Series ended (with the Cubs winning). We start by calculating the "squared error" for each of the 7 days that my forecast was valid. To calculate the squared error, we simply square the difference between my forecast and the actual outcome (100% if the Cubs win, 0% if they lose):

(0.9 - 1.0)^2 = 0.01

And for my implicit 10% Cubs lose forecast:

(0.1 - 0.0)^2 = 0.01

So my total error for the day:

0.01 + 0.01 = 0.02


To calculate the mean squared error, calculate the squared error for each day that my forecast was running and average them:

Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7
Forecast (win, lose) 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.9, 0.1
Error 0.02 0.02 0.02 0.02 0.02 0.02 0.02

We then take the average (aka mean) daily squared error to get a final score. In this case, since all of the days are the same, my mean squared error ends up being equal to 0.02.


Updating a Forecast

What would have happened to my Brier Score if I had updated my forecast mid-way through the week? Maybe the Cubs won a game on Day 3, leading me to update my forecast to 95% on Day 4. For any days where I had a 95% forecast, the squared error would be:

(0.95 - 1.0)^2 = 0.0025

And my new, implicit Cubs-lose forecast of 5%:

(0.05 - 0.0)^2 = 0.0025

So my total error for those days:

0.0025  + 0.0025 = 0.005

So my daily squared error would look something like:

Day Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7
Forecast 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.95, 0.05 0.95, 0.05 0.95, 0.05 0.95, 0.05
Squared Error 0.02 0.02 0.02 0.005 0.005 0.005 0.005

So my mean squared error would be:

(0.02 + 0.02 + 0.02 + 0.005 + 0.005 + 0.005 + 0.005) / 7 = 0.0114


In the world of Brier Scores, a lower score (ie. less error) is better, so by updating our forecast mid-way through the week, we improved our score from 0.02 to 0.0114.


Go back to The Ultimate Guide to Prediction Markets

Cultivate Forecasts logo - prediction market software for enterprises

Crowdsourced Forecasting

Use prediction markets to learn what your people really think is going to happen with your most important projects and metrics.

Cultivate Ignite logo - Kickstarter-style crowdfunding platform for use inside companies

Internal Kickstarter-Style Crowdfunding

Use Kickstarter-style internal crowdfunding to prioritize what you should be focusing on at the front-end of your innovation process.