What are Net Brier Points and How are they Calculated?

« The Ultimate Guide to Prediction Markets

To understand Net Brier Points and their calculation, it is essential to first understand basic Brier scores and how they are calculated. If you are not familiar with Brier scores, you should start by reading our article What is a Brier Score and How is it Calculated?


What are Net Brier Points and what problem do they solve?

In forecasting tournaments, the importance of fair and equitable scoring is paramount. If forecasters can gain an unfair advantage in scoring, then it undermines the goal of identifying the best forecasters. A traditional Brier score can be effective for comparing forecasters if all participate equally. That is to say, they all forecast every day on every question.

But what happens in a forecasting tournament where participants choose the questions and days on which they forecast? In that scenario, a forecaster may choose to wait until late in a question before submitting a forecast, when picking the correct outcome is much easier.

To illustrate the Net Brier Points concept, we can continue the example from What is a Brier Score and How is it Calculated? In that post, we showed a hypothetical forecast on the question "Will the Cubs win the World Series?" and calculated the associated error/score for the scenario where the Cubs win.

n.b. the values in the 'Forecast' row are 'yes,no' forecasts. So '0.9,0.1' would correspond to the forecaster saying there is a 90% chance the Cubs will win the World series and a 10% chance they won't.

Forecaster #1
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Overall Brier Score
Forecast 0.9, 0.1 0.9, 0.1 0.9, 0.1 0.95, 0.05 0.95, 0.05 0.95, 0.05 0.95, 0.05
Score 0.02 0.02 0.02 0.005 0.005 0.005 0.005 0.0114  

Now, let's consider 2 other forecasters in the same question:

Forecaster #2
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Overall Brier Score
Forecast 0.25, 0.75
0.25, 0.75
0.2, 0.8 0.2, 0.8 0.2, 0.8 0.2, 0.8 0.2, 0.8
Score 1.125

1.125

1.28

1.28 1.28 1.28 1.28 1.2357  

And Forecaster #3 (a blank cell indicates no forecast, and thus no score, for that forecaster on that day):

Forecaster #3
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Overall Brier Score
Forecast




0.99, 0.01 0.99, 0.01
Score




0.0002 0.0002 0.0002  

So at 0.0002, Forecaster #3 ends up with the best Brier score (remember, 0.0 is the best possible Brier score and 2.0 is the worst). This seems a little unfair though -- Forecaster #3 waits until Day 6 when the answer might already be obvious. Shouldn't Forecaster #1 be rewarded for making good forecasts early in the question?


Calculating Net Brier Points

Net Brier Points (also known as Relative Brier Scoring) were created to correct this inequity -- a more fair scoring system that rewards early, accurate forecasts. Like traditional Brier Scores, lower is better with Net Brier Points (you can think of it like a golf score, where below par is better).

To calculate Net Brier Points:

  1. Calculate the median daily score across all forecasters
  2. For each day, subtract the median from the forecaster's daily Brier score to get his/her daily Net Brier Score.
  3. For each forecaster, sum the daily Net Brier Scores calculated in step #2
  4. Divide that sum by the total number of days for the question (not just the days the forecaster participated)


To complete our example, let's calculate the median of the daily scores (step #1 above):

Forecaster #1
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7
Forecaster #1 Score 0.02 0.02 0.02 0.005 0.005 0.005 0.005
Forecaster #2 Score
1.125

1.125

1.28

1.28 1.28 1.28 1.28
Forecaster #3 Score





0.0002 0.0002
Median Daily Score
0.5725
0.5725
0.65
0.6425
0.6425
0.005 0.005


Now we can subtract the median from the forecaster's score, sum the forecaster's daily Net Brier Points, and divide by the total number of days for the question (7 days, in our example):

Forecaster #1
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Overall NBP
Forecaster #1 NBP
-0.5525   -0.5525   -0.63   -0.6375   -0.6375   0.0   0.0   -0.43  
Forecaster #2 NBP
0.5525   0.5525   0.63   0.6375   0.6375   1.275   1.275   0.7943  
Forecaster #3 NBP





-0.0048   -0.0048   -0.00137  
Using Net Brier Points, Forecaster #1 is rewarded for forecasting early and accurately, resulting in the best score (-0.43 -- remember, lower is better). Forecaster #3 still receives a score that is better than the median (-0.00137), but no longer beats Forecaster #1, due to entering the question late. The Cubs hater, Forecaster #2, made poor forecasts throughout and received the worst score (0.7943).


If you're interested in learning more about Brier scoring, Net Brier Points, or running a forecasting tournament, feel free to contact us.



Cultivate Forecasts logo - prediction market software for enterprises

Crowdsourced Forecasting

Use prediction markets to learn what your people really think is going to happen with your most important projects and metrics.

Cultivate Ignite logo - Kickstarter-style crowdfunding platform for use inside companies

Internal Kickstarter-Style Crowdfunding

Use Kickstarter-style internal crowdfunding to prioritize what you should be focusing on at the front-end of your innovation process.