The Brier score calculator measures the accuracy of probability forecasts from three forecast levels, with the decomposition into reliability, resolution and uncertainty and the skill score.
How the calculation works
- Enter, for each of the three levels, the forecast probability, the number of forecasts and the observed frequency of the event.
- The calculator weights the levels by their number.
- It gives the Brier score, reliability, resolution, uncertainty, the skill score and the base rate.
Example
A forecaster issued 100 forecasts of 10 % (event seen 15 %), 50 of 50 % (45 %) and 50 of 90 % (80 %). The base rate is 38.8 %, so the uncertainty is 0.2373. Reliability is 0.0044 and resolution 0.0717, so the Brier score is 0.0044 − 0.0717 + 0.2373 = 0.1700. The skill score is 0.2837: the forecasts add real information over the base rate, with room to improve calibration.
Why it matters
The calculator helps students of forecasting and machine learning, meteorologists, analysts who build probability models for risk or sports, and anyone who keeps score of predictions.
Good to know
Grouping forecasts into a few levels gives a decomposition that is exact only when all forecasts in a group share the same probability; with real data, use the ungrouped Brier score and bins that are wide enough to hold many cases. A good Brier score depends on the base rate: forecasting rare events is easy to score well on by always forecasting a low number, so look at the skill score. The Brier score is a proper score, meaning that honest probabilities give the best expected value. It is not suited to extremely rare events, where the log score may be better. Report the number of forecasts, since small samples are noisy.
Related tools
The confusion matrix calculator evaluates yes-or-no predictions, the logistic regression calculator produces probabilities, the Matthews correlation calculator summarizes binary classification and the KL divergence calculator compares distributions.