To understand Relative Brier Scores and their calculation, it is essential to first understand basic Brier scores and how they are calculated. If you are not familiar with Brier scores, you should start by reading our article What is a Brier Score and How is it Calculated?

Note: in the past, we referred to Relative Brier Scores as Net Brier Points. They are functionally the same -- different names for the same calculations.

What are Relative Brier Scores and what problem do they solve?

In forecasting tournaments, the importance of fair and equitable scoring is paramount. If forecasters can gain an unfair advantage in scoring, then it undermines the goal of identifying the best forecasters. A traditional Brier score can be effective for comparing forecasters if all participate equally. That is to say, they all forecast every day on every question.

But what happens in a forecasting tournament where participants choose the questions and days on which they forecast? In that scenario, a forecaster may choose to wait until late in a question before submitting a forecast, when picking the correct outcome is much easier.

To illustrate the Relative Brier Score concept, we can continue the example from What is a Brier Score and How is it Calculated? In that post, we showed a hypothetical forecast on the question "Will the Cubs win the World Series?" and calculated the associated error/score for the scenario where the Cubs win.

n.b. the values in the 'Forecast' row are 'yes,no' forecasts. So '0.9,0.1' would correspond to the forecaster saying there is a 90% chance the Cubs will win the World series and a 10% chance they won't.

Now, let's consider 2 other forecasters in the same question:

And Forecaster #3 (a blank cell indicates no forecast, and thus no score, for that forecaster on that day):

So at 0.0002, Forecaster #3 ends up with the best Brier score (remember, 0.0 is the best possible Brier score and 2.0 is the worst). This seems a little unfair though -- Forecaster #3 waits until Day 6 when the answer might already be obvious. Shouldn't Forecaster #1 be rewarded for making good forecasts early in the question?

Calculating Relative Brier Scores

Relative Brier Scores (also known as Net Brier Points) were created to correct this inequity -- a more fair scoring system that rewards early, accurate forecasts. Like traditional Brier Scores, lower is better with Relative Brier Scores (you can think of it like a golf score, where below par is better).

To calculate a Relative Brier Score:

1. Calculate the median daily score across all forecasters
2. For each day, subtract the median from the forecaster's daily Brier score to get his/her daily Relative Brier Score.
   
3. For each forecaster, sum the daily Relative Brier Score calculated in step #2
   
4. Divide that sum by the total number of days for the question (not just the days the forecaster participated)

To complete our example, let's calculate the median of the daily scores (step #1 above):

Now we can subtract the median from the forecaster's score, sum the forecaster's daily Relative Brier Scores, and divide by the total number of days for the question (7 days, in our example):