| Title: | Evaluation Metrics for Spaced Repetition Schedulers |
|---|---|
| Description: | Calibration and discrimination metrics for spaced-repetition memory models. Provides the sample-weighted binned root mean squared error (RMSE(bins)) used to rank schedulers in the open spaced repetition benchmark, together with log loss, the area under the ROC curve, and calibration curves. |
| Authors: | Christos Longros [aut, cre] (ORCID: <https://orcid.org/0009-0001-2717-0857>) |
| Maintainer: | Christos Longros <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1 |
| Built: | 2026-07-10 10:11:59 UTC |
| Source: | https://github.com/chrislongros/srsbench |
Calibration curve in equal-width bins
calibration_bins(p, y, n = 10)calibration_bins(p, y, n = 10)
p |
Numeric vector of predicted recall probabilities in |
y |
Numeric vector of observed outcomes: |
n |
Number of equal-width probability bins (default 10). |
A data frame with one row per bin: bin index, bin bounds, mean
predicted probability, observed recall rate and review count. Empty bins
have NA means and a count of 0.
calibration_bins(p = c(0.1, 0.2, 0.8, 0.9), y = c(0, 0, 1, 1), n = 5)calibration_bins(p = c(0.1, 0.2, 0.8, 0.9), y = c(0, 0, 1, 1), n = 5)
Log loss (binary cross-entropy)
log_loss(p, y, weights = NULL, eps = 1e-15)log_loss(p, y, weights = NULL, eps = 1e-15)
p |
Numeric vector of predicted recall probabilities in |
y |
Numeric vector of observed outcomes: |
weights |
Optional numeric vector of non-negative per-review weights (default 1). |
eps |
Small value used to clip |
A single non-negative number; lower is better.
log_loss(p = c(0.9, 0.1), y = c(1, 0))log_loss(p = c(0.9, 0.1), y = c(1, 0))
Computes the sample-weighted binned root mean squared error used to rank
schedulers in the open spaced repetition benchmark. Reviews are grouped into
bins along three log-quantised axes – the interval (elapsed_days), the
review number (i) and the number of prior lapses (lapse) – then
the mean predicted and mean observed recall are compared within each bin and
combined, weighting each bin by its number of reviews.
rmse_bins(p, y, elapsed_days, i, lapse, weights = NULL)rmse_bins(p, y, elapsed_days, i, lapse, weights = NULL)
p |
Numeric vector of predicted recall probabilities in |
y |
Numeric vector of observed outcomes: |
elapsed_days |
Numeric vector of review intervals in days. |
i |
Numeric vector giving each review's position in its card's history (1 for the first review, 2 for the second, and so on). |
lapse |
Numeric vector giving the number of lapses before each review. |
weights |
Optional numeric vector of non-negative per-review weights (default 1). |
A single non-negative number; lower is better calibrated.
RMSE(bins) is the metric defined by the open spaced repetition benchmark;
this is an independent R implementation of its rmse_matrix definition.
https://github.com/open-spaced-repetition/srs-benchmark
rmse_bins(p = c(0.9, 0.2), y = c(1, 0), elapsed_days = c(2, 100), i = c(2, 5), lapse = c(0, 1))rmse_bins(p = c(0.9, 0.2), y = c(1, 0), elapsed_days = c(2, 100), i = c(2, 5), lapse = c(0, 1))
A rank-based (Mann-Whitney) estimate of discrimination, with no external
dependencies. Ties in p are handled by mid-ranks.
srs_auc(p, y)srs_auc(p, y)
p |
Numeric vector of predicted recall probabilities in |
y |
Numeric vector of observed outcomes: |
The AUC in [0, 1], or NA if y has only one class.
srs_auc(p = c(0.9, 0.8, 0.2, 0.1), y = c(1, 1, 0, 0))srs_auc(p = c(0.9, 0.8, 0.2, 0.1), y = c(1, 1, 0, 0))