New format_points() reconstructs per-instance scores and labels from a
table of performance values calculated at a set of thresholds - true and
false positive rates, or recall and precision - so that a summary table can
reach evalmod() and mmdata() after all. The rows between two thresholds
become instances that share a score, and the even spread precrec already
applies to a tied run is the non-linear interpolation of Davis and Goadrich
(2006), so nothing new was needed in the curve code and a complete table
round-trips exactly. np and nn are required: rates do not carry the
class totals, and the totals also place the chance-level baseline. The
interpolation is exact only across a gap one instance wide, so an area read
off a coarse table remains an estimate - format_points() says so when it
runs, and Prepare your data says so at more length. Requested in #6 and
#15, and tracked in #32.
Prepare your data replaces its section on what cannot be used as input with one on what a table of summary points can and cannot give you.
Prepare your data gains a section on which direction the score runs.
precrec ranks the highest score first, and a quantity that runs the other
way - a p-value, a distance, an error term - is handled by negating it.
There is no argument for this and there was nothing written down either, so
the question had no answer anywhere on the site. The section also notes the
one place negation does not carry: prob_metrics() reads the values of the
scores rather than their order, so it needs a genuine probability. Reported
in #14.
Prepare your data gains a section on what cannot be used as input.
precrec needs the per-instance scores and labels; precision and recall
already calculated at a set of thresholds, or a table of true and false
positive rates, cannot be used. The counts such a table implies are
recoverable when the totals are known, so the limit is not arithmetic: it
is that the precision-recall interpolation is only correct across a gap one
instance wide, and over the coarse gaps a threshold table leaves it
reintroduces the very error precrec exists to remove. The limit was
deliberate but undocumented, and two people arrived with such a table four
years apart. Reported in #6 and #15.
Correct the pROC argument table in Coming from pROC or ROCR, which
paired direction = with posclass = . The two do different things:
posclass names the positive label, which is pROC's levels, while
direction is the negation above.
metric_table() reported a precision on the row that calls nothing
positive and an NPV on the row that calls everything positive. Neither
has a denominator - precision is TP / (TP + FP) and NPV is
TN / (TN + FN) - and both were filled in from the neighboring row, so
they followed the top-ranked or bottom-ranked instance, which is an
instance the rule does not predict. A threshold above every score
reported precision = 1 beside sensitivity = 0, and 0 instead if the
top-ranked instance happened to be a negative.
Both are now NA, with the metrics derived from them: the false
discovery rate, the false omission rate, and markedness at both ends.
mcc, lift, odds, chisq and the positive likelihood ratio already
reported NA on the same rows.
This matters most for metric_table(at = ), whose purpose is scoring a
threshold chosen somewhere else: a threshold transferred between datasets
lands outside the score range routinely.
The long form as.data.frame(evalmod(mode = "basic")) returns is
unchanged, and so are the curves. The inherited precision is what anchors
the precision-recall curve at recall 0, and it is the right value there
auc(), average_precision() and prbe() are unaffected, which the
tests now assert.best_cutoff() could return the rank 0 row of metric_table(), the
rule that calls nothing positive. There is no threshold that predicts
nothing, so that row's score is NA and it is not an operating point,
which is what the function exists to name. It is no longer a candidate.
It won outright wherever the empty rule is optimal - specificity
always, and accuracy, error and an unweighted cost once positives
are rare, which at 2% positives was 11 runs in 20 - and it won on the
tie-break wherever it merely tied. That last case was not an imbalance
problem at all: precision at rank 0 is the limit from above, so it is
1 whenever the top-ranked instance is a positive, and any classifier
good enough to rank one first came back with no threshold.
Choose an operating point gains the accuracy trap beside the
sensitivity and specificity ones it already covered. On twenty
positives in four hundred, predicting nothing is 0.95 accurate and the
accuracy-optimal cutoff reaches 0.955 by finding two of the twenty;
mcc finds eleven.
prbe() reported a break-even point of 0 whenever the top-ranked
instance was a negative. A precision-recall curve is anchored at recall
0, where precision is 0 in that case, so precision and recall were
trivially equal at the origin and the crossing search counted it - though
nothing has been retrieved there. At a low proportion of positives the
top-ranked instance is nearly always a negative: of 150 random rankings of
a dataset with 20 positives in 1000, 148 had a negative on top and all 148
got the spurious row. It sat in front of the real break-even point, so
taking the first row of the result returned 0 instead of the answer.
A curve that leaves the origin below the diagonal and never catches up now
gets the single NA row the help page has always promised for a curve
that never reaches equal precision and recall.
prbe() gained a baselines column, the proportion of positives. At
chance the curve is flat at the prevalence, so it meets the diagonal at
that recall - a break-even point of 0.2 is chance on data that is 20%
positive and five times chance on data that is 4% positive.
Every table that reports an area now reports what that area is worth by
chance beside it. pauc() gained baselines and sbaselines, one for
each of the two scales it standardizes on, and auc_ci(), auc_boot()
and auc_delong() gained baselines. A partial area is where the chance
level is easiest to get wrong: over false positive rates [x1, x2] a coin
flip covers (x2^2 - x1^2) / 2, so a standardized partial ROC area up
to 0.2 has a chance level of 0.1 and not the 0.5 a standardized
number invites. Both are NA for a ylim other than c(0, 1), which is
the case cpaucs already declines.
Balanced and imbalanced data gains a section on the fact that a baseline
is an asymptote. The prevalence is what a precision-recall area is worth
by chance in the limit; an area measured on a finite sample scatters
around it, and with twenty positives a classifier with no signal averages
1.15 times its baseline over the whole curve and 1.86 times over the first
tenth of recall, where it clears twice the baseline about one run in five.
The median sits below chance, so the error is a skewed distribution rather
than a shifted one. Dividing an area by its baseline is not a test;
auc_boot() is, and it is well calibrated on the same case. ?auc and
?pauc say so too.
pauc() gained a corrected argument, which adds a cpaucs column: the
McClish correction of the ROC partial area, which rescales the span
between chance and perfect onto 0.5 to 1 so that a partial area reads
on the same scale as a full one. It is what
pROC::auc(..., partial.auc.correct = TRUE) reports, to the last digit.
Both this and the spaucs the package has always reported are called
"the standardized partial AUC", and they are not the same number:
spaucs is the area over the area of the region, the fraction of what
was available that the curve covered. A value moved between two tools
looked like a disagreement when it was a choice of convention, and only
one of the two conventions could be had here.
The chance area is taken over the region actually asked for. The formula
usually quoted, and the one Coming from pROC or ROCR published for
readers to apply by hand, is the special case of a region starting at a
false positive rate of 0; it is wrong for a region such as
xlim = c(0.1, 0.3), which is exactly where someone would reach for it.
cpaucs is NA on the precision-recall rows and whenever part() was
given a ylim, both documented in ?pauc. Off by default, so nothing
that reads pauc() today changes.
The help page of pauc() opened by naming auc() as the function it
documents, a line copied from auc() and never corrected.
Balanced and imbalanced data gains a section on why the baseline is reported beside the area rather than folded into it. Precision-Recall-Gain (Flach & Kull 2015) is the published way of folding it in, and on a classifier held fixed while only the prevalence moves, its area climbs from 0.639 to 0.941 - three tenths of its range, in the opposite direction to the precision-recall area, which falls. Normalizing by the baseline does not remove the dependence on prevalence, it reverses it.
auc() and average_precision() gained a baselines column, and the
summary print() shows a Baseline beside each AUC. It is what the area
would be by chance: 0.5 on every ROC row, and the proportion of
positives on every precision-recall row.
A ROC AUC can be read on its own and a PRC AUC cannot, which is the package's own argument and was the one number the package did not report. On one generator at three class balances the ROC AUC holds between 0.81 and 0.85 while the PRC AUC goes 0.801, 0.367, 0.129 - and the last of those, which reads as failure, is six times its baseline of 0.02. The plots have drawn the baseline all along; now the areas carry it too.
The baseline is taken per model and per test dataset, so a fold that does
not hold the classes in the proportions the whole dataset does is read
against its own. On a macro-average row it is averaged over the classes
with the same weights the AUCs were, uniform or by prevalence.
Code that selects columns of these data frames by name or by subset() is
unaffected; code that relies on their column count or position will see
one more.
New metric_table(at = ) reports the metrics of thresholds you name,
rather than of every cutoff. A threshold is rarely one of the observed
scores, so it has no row of its own, but it always names one of the
cutoffs: score >= it calls a certain number of instances positive,
and that count is a rank the table already holds. The row returned is
that rank's row, with at carrying the threshold asked for and score
the observed cutoff realizing it. A threshold above every score gives
the rank = 0 row, and an NA score is never a positive prediction,
as everywhere else in the package.
This is what was missing to score a cutoff on data it was not chosen
on. classification_report(at = ) already took an arbitrary threshold
but reports only precision, recall and F-score;
metric_table() had every metric but only at the cutoffs the data
happens to contain. Choose an operating point now shows the two ends
together - best_cutoff() on nine folds, metric_table(at = ) on the
tenth - and what it costs not to: on data with five percent positives,
mcc reads 0.361 when each fold picks its own cutoff and 0.228 when
the cutoff is chosen without seeing it, held-out below in-sample in
every fold.
best_cutoff() rejects at rather than letting it narrow the search
it does.
Document print(). The six print() methods carried no help page at
all: one of them pointed at a print topic with @rdname, nothing
defined that topic, and so roxygen2 skipped it on every run and
?print.mdat failed. R CMD check did not notice because the methods
are registered rather than exported by name. The topic now exists, and
says which object each method handles and what it shows.
print() returns its argument invisibly, as an S3 print method is
expected to. Five of the six returned the value of the last cat()
call, so x <- print(curves) gave NULL. What the console shows is
unchanged.
Regroup the website. A Compare and decide section now collects the four pages that answer "which model do I ship, and at what threshold" - comparing several models, uncertainty from one test set, choosing an operating point, and the classification report - which were previously split between How-to and Metrics on the grounds of what they returned rather than what they were for. Metrics is left as what it already effectively was, a catalog of what each number means, and Plots is unchanged. No page moved and no URL changed; the grouping is in the menus only.
Add a website page, Choose an operating point, which best_cutoff()
did not have: it was documented inside Get the numbers out, a page about
extracting data, and argued about inside Balanced and imbalanced data, a
page about reading curves. The new page covers the criteria and the two
groups they fall into, weighting the two mistakes with cost_fp and
cost_fn - including why equal costs are a strong claim rather than a
neutral one when positives are rare - tied scores, the one row per model
per test dataset and the spread across them, and the fact that a cutoff
chosen on the data it is scored on is optimistic and that nothing here
corrects for that.
Add a website page, Coming from pROC or ROCR, translating the two
packages call by call now that metric_table() and best_cutoff() make
the table honest. Every equivalence on it was checked against both
packages, including the three places the answers differ: the
precision-recall area, which ROCR reads off straight lines between raw
points; tied scores, which the other two collapse and
evalmod(basic_ties = "hold") matches; the threshold itself, which
pROC names by a midpoint and precrec by an observed score; and the
standardized partial AUC, where spaucs is the fraction of the region
the curve covered and pROC rescales chance to 0.5 instead - with the
conversion between them. What has no equivalent is listed rather than
left to be discovered.
New best_cutoff() picks the cutoff that optimizes one metric, in the
manner of pROC::coords(x, "best"). It returns a row of metric_table()
together with the criterion that chose it, so every other metric is there
to be read at the same cutoff, and one row per model per test dataset.
metric takes any of the metrics evalmod() calculates, plus "youden"
and "topleft" for the two criteria the cutpoint literature names rather
than the metric table does. The direction is a property of the metric and
so is not an argument, and a tie is broken toward the cutoff that calls
the fewest instances positive - except between rows that share a score,
which are one threshold seen several times rather than several cutoffs,
and report the rank that threshold actually produces.
The criteria are documented in two groups rather than as six
interchangeable options. Youden's J and the closest point to the top left
corner are computed from sensitivity and specificity alone, both of which
are conditioned on the true class, so neither knows the prevalence and on
imbalanced data both can choose a cutoff at a precision no one would
deploy. fscore, mcc and a cost weighted by cost_fp and cost_fn
are computed from precision as well. The default is "youden" because
that is what a caller arriving from another package expects, and the
reference page says in the same breath when not to use it.
youden is also accepted as an alias for informedness wherever
evalmod(metrics = ) takes metric names.
New auc_delong() calculates the ROC AUC and its variance analytically,
by DeLong's method, rather than by resampling. auc_ci() reads a normal
interval off it and auc_diff() compares models through its covariance
matrix, so it is the exact counterpart of auc_boot(): same input, same
two functions reading the result, no boot_n, no seed, nothing that
moves between two runs, and no floor under the p-value.
The ROC AUC is a Mann-Whitney U statistic, so its variance follows from
the structural components of that statistic. Those components are read
off midranks rather than off all m * n comparisons, and off the ranks
precrec has already assigned, so ties and NA scores are handled
exactly as they are everywhere else and the AUC reported is the one
auc() reports.
There is no precision-recall counterpart, and the result carries ROC rows
only. The precision-recall AUC is not a U statistic and the interpolated
area is further from being one still, so auc_boot() remains the answer
there - and remains the safer reading on a small or badly imbalanced test
set, since DeLong's variance is an asymptotic one.
auc_diff() is now an S3 generic, dispatching on whichever of the two
objects it is given. Existing calls are unaffected.
New metric_table() returns every basic evaluation metric at every
cutoff, one row per cutoff and one column per metric - what
pROC::coords() and the cutoff slots of a ROCR::performance object
give, and the shape most work downstream of a curve needs: pick an
operating point, hand a threshold to production code, tabulate a report,
compute a metric precrec does not.
Nothing new is calculated. evalmod(mode = "basic") has always kept the
score and the label at every cutoff, and as.data.frame() has always
returned them - as two of the fourteen values of its type column,
alongside the metrics rather than indexing them. The long form is right
for plotting, which is what it was built for, and wrong for the question
"what happens if I cut here". metric_table() is that same data on its
side, through the same converter, so the two cannot drift apart.
It takes the doors evalmod() takes - an mmdata() object, or scores
and labels - and also a mode = "basic" object that has already been
calculated, since reusing one is the point. rank counts the instances
called positive and normalized_rank is that divided by the number of
instances, the x axis of the basic metric plots. Rows are per test
dataset and nothing is averaged across them, because a cutoff belongs to
the dataset it was read off.
The two columns auc_diff() gained in 0.22.1 are renamed. statistic
is now z_values and p_values_norm is now p_values_wald, and the
documentation calls what they come from a Wald test throughout. Neither
column's contents changed.
p_values_norm named the p-value for the distribution it is read off
rather than for the test that produced it, which left it inconsistent
with p_values beside it - named for the percentile method - and
readable as "normalized", which it is not. statistic said nothing about
which distribution it is referred to, and was singular where every other
column in the row is plural.
The reference page and the Uncertainty from one test set article now
give both formulas outright, z_values = diffs / sd(d) and
p_values_wald = 2 * pnorm(-abs(z_values)) against the percentile
count, and answer under its own heading why two p-values are reported at
all: they fail in opposite ways - the percentile one has a floor at
2 / (n + 1), the Wald one has an assumption of normality - and it is
whether they agree that tells you which is failing.
0.22.1 is a day old and was not submitted to CRAN, so the renames are made now rather than carried.
auc_diff() reports a test statistic and a second p-value, and takes a
new alternative argument for the tail they are read from. statistic
divides the observed difference in AUC by the standard deviation of the
resampled differences, so the bootstrap spread stands in for a standard
error, and p_values_norm reads that statistic off the normal
distribution.
The percentile p_values is floored at 2 / (n + 1) on purpose, which
is 0.002 at the default boot_n = 1000; a difference that is not in any
doubt bottoms out there, and reporting p < 0.001 from it needs ten
thousand resamples or more. The normal approximation reads an unbounded
p-value off the resamples that are already there.
It buys that resolution with an assumption. Nothing is shuffled between
the models, so the null is never enforced, and the resampled differences
are taken to be roughly normal - weakest where they are skewed, which is
few positives or either model near the ceiling of the precision-recall
AUC. p_values keeps its meaning, both are reported side by side, and
where the two disagree sharply it is the normal one to distrust.
evalmod(mode = "basic") takes a new basic_ties argument for what the
basic metrics report at the cutoffs inside a run of tied scores. Such a
cutoff separates instances that share a score, so no threshold produces
it. The default "split" spreads the true and false positives of the run
evenly over its cutoffs, which is what precrec has always done and is
the interpolation the ROC and precision-recall curves need. "hold"
gives every cutoff in the run the counts it has once the whole run is
taken, so tied instances share one value of every metric and each metric
becomes a step function that changes only where the score does.
A classifier that is exactly right but scores in a single bit reported
sensitivity climbing across the positives rather than reaching 1 at once,
which is the even spread and not the classifier. basic_ties = "hold"
is the reading that case wants.
The two settings agree whenever the scores are all distinct, "split" is
the default because it is what every published result was computed with,
and the curves are not affected either way.
reduce_points now works for mode = "basic". The basic metrics carry one
point per cutoff, so plotting a large dataset drew a mark per instance: at
a million rows autoplot() spent 6.7 seconds against 0.5 for the
calculation behind it. fortify(), as.data.frame() and autoplot()
accept reduce_points = TRUE for these objects now and keep x_bins
points per metric, which brings that plot to 1.9 seconds.
The default stays FALSE, so no existing output moves unless the
reduction is asked for, and the points kept are the calculated ones - the
reduction selects among them and never interpolates. Nothing is dropped
from an object that already holds x_bins points or fewer.
The argument was accepted and silently discarded before, so a call that passed it was not an error; it simply had no effect.
evalmod(mode = "basic") records x_bins on the object it returns. It
places no supporting points there - the basic metrics have no
interpolation to place them on - but it is the number of points kept when
reduce_points thins them, so the resolution of the drawn output is set
the same way for both modes.
Speed up the C++ hot paths. evalmod() runs about 1.5x faster on a
million rows, and evalmod(mode = "basic") about 1.2x. The results are
unchanged, bit for bit.
Two things were costing the time. First, reading a vector through Rcpp's
operator[] goes through a proxy whose bounds check is a call to
warning(), and in a translation unit the size of precrec_plx.cpp the
compiler leaves that call out of line rather than inlining it away - so
every element access in the innermost loops carried a call, a compare and
a barrier to optimizing the loop around it. The four functions that scan
whole vectors take a raw pointer once instead, which is worth 1.8x in
calc_basic_metrics(), 1.7x in create_prc_curve(), 1.5x in
create_roc_curve() and 1.3x in create_confusion_matrices().
Second, ranking the scores was 84% of get_score_ranks() and all of it
was the sort. A comparison sort over (index, score) pairs is replaced by
a radix sort on an order-preserving key, which is 2.3x faster on distinct
scores and 2.4x on heavily tied ones. Scores that arrive already sorted are
the one shape it loses on, and a monotonicity scan in front of it hands
those to a direct fill rather than to the passes; the scan stops as soon as
the scores are neither ascending nor descending, which on unsorted input is
within the first few elements. Even so, ranking an already sorted vector is
about 0.8x - the comparison sort was unusually good at that one case, and
the scan recovers most of the difference but not all of it.
Ties still rank the way they did. The old comparator broke ties on the input index to keep the permutation from depending on the standard library's choice of introsort; a radix sort is stable pass by pass, so it produces that same permutation from the algorithm instead.
Fix a buffer overrun in create_roc() and create_prc(). Both built the
interpolated points from a grid snapped to each gap's own start, so
neighboring gaps disagreed about where the grid lines fell and together
emitted more points than the x_bins the output buffers are sized for.
The write ran past the end of the vectors, which aborted the session from
x_bins = 5000 up - evalmod(..., x_bins = 10000) on 200 points was
enough to do it. Both now walk one grid shared across the curve, so each
line belongs to exactly one gap and the count cannot exceed x_bins.
The curves move slightly as a result: the spurious extra points are gone, so a curve carries a few points more or fewer than it did, and the ones it carries sit on the grid. AUC, partial AUC and average precision are unchanged - they agree with 0.21.0 to within 2.2e-16, which is the rounding of the arithmetic rather than a change in what is computed.
Cap x_bins at 1e6. Every stage sized by it - the interpolation
buffers, the point reduction and the averaging grid - allocates a vector
of that length per curve, so x_bins = 1e9 asked for tens of gigabytes
and failed on the allocation rather than at the argument. A million
supporting points is finer than a plot resolves and finer than the data
behind it, so the ceiling is well clear of real use.
Skip the interpolation of a gap that spans no grid line at all, which is
almost every gap once the input is large. That was costing a division per
input point to add at most x_bins points to the whole curve, so it grew
with the data while what it produced did not. create_roc() is about
1.8x faster at 1e6 points and create_prc() about 1.3x; evalmod()
itself gains about 4%, the curves being a small part of its work.
Add auc_boot(), which resamples a single test set, so that auc_ci()
can put an interval around an AUC without several test sets to compare.
Every interval in precrec until now was built from the variation
between test sets, and auc_ci() on one dataset raised an error; that
error now names this function.
The resampling is stratified - positives drawn from the positives, negatives from the negatives - so every resample keeps the class balance of the original. An unstratified bootstrap of imbalanced data varies the balance from resample to resample, which moves the precision-recall baseline underneath the quantity being estimated.
auc_ci() on the result gives a percentile interval, read off the
resampled values rather than assumed from a distribution over them, so it
cannot leave [0, 1] and needs none of the clipping the multi-dataset
interval does. dtype is refused there for the same reason.
Add auc_diff(), which compares two models on the same resamples. The
difference is calculated within a resample, so the interval describes the
difference itself. Two separate intervals from auc_ci() cannot be read
that way: they can overlap while the difference is clearly on one side of
zero, because they say nothing about how the two models move together.
It reports a percentile interval and a p-value, floored at 2 / (n + 1)
so that a resample count cannot manufacture significance it does not
support.
Verified rather than asserted: the bootstrap standard error matches the analytic variance of DeLong for the ROC AUC to within 2% at n = 100 to 1000, and the 95% interval covered a known true AUC in 142 of 150 simulations. Both checks are in the test suite, the first as an independent implementation of the DeLong formula.
Add a website page comparing precrec with ROCR, pROC, PRROC,
yardstick, scikit-learn and imbalanced-learn: which to reach for,
what precrec does not do that they do, and the one place the numbers
disagree rather than the interfaces. The step estimator of the
precision-recall area, which most of them report, reads more than a tenth
high at two percent positives against the interpolated area auc()
returns, and the page computes that rather than claiming it.
Add a website page on using precrec from tidymodels. No adapter is
needed: collect_predictions() returns a fold column, a truth column and
one .pred_<class> column per class, which is the shape mmdata()
already takes through nfold_df, and factor labels have always been
accepted.
The page leads with the one difference that is silent when it bites.
yardstick treats the first factor level as the event; precrec treats
the last as the positive class. Pairing yardstick's default score
column with precrec's default inverts the AUC without a warning, and
the page shows a case that comes out 1 one way and 0 the other.
tidymodels stays out of the package entirely, Suggests included, so
the chunks that need it are shown rather than run and the website builds
without it. bench/run_tidymodels_parity.R checks the page against a real
fit instead, and found two of its recipes wrong before it shipped.
Add classification_report(), the per-class table of precision, recall
and F-score that scikit-learn's classification_report prints, with the
macro avg and weighted avg rows under it. It reproduces
scikit-learn's documented examples number for number.
The third summary row follows the rule scikit-learn documents. A binary
problem cut at a threshold puts every observation in exactly one of the
two classes, so the row is accuracy. A multi-class problem is evaluated
one-vs-rest and each class is thresholded on its own, so an observation
can fall into no class or into several; there is then no single-label
accuracy, and the row is micro avg, computed from the true positives,
false positives and false negatives pooled over the classes. Where the
predictions do happen to be single-label the two coincide, which is the
identity scikit-learn gives for printing one and not the other.
at names the operating point and has no default. scikit-learn reports
on y_pred, so its caller has already chosen one; precrec holds scores
and evaluates every cutoff, and no threshold is meaningful on every score
scale, so the choice stays with the caller. It takes one number for all
classes or one per class.
zero_division sets what an empty denominator reports, 0 as in
scikit-learn or NA as in the per-cutoff metrics of evalmod().
Add a website page for it, and trim the metric pages back towards the
length the site was built at. Each release since had appended a section to
them, taking the article average from the 88 lines the site was built at to
99. It is now 93 over 22 articles, one more article than before. Nothing a
page needed was dropped: the cuts are restated prose, a duplicated auc()
chunk on the AUC page, and an unused beta = 2 example on the agreement
page.
Settle on "metric" as the word for the quantities evalmod(metrics = ),
metric_curve(x_metric = , y_metric = ) and prob_metrics() name. The
documentation had called them "measures" throughout while the arguments
that select them had always been called metrics, so a reader had to work
out that the two words meant one thing. Nothing about what precrec
computes has changed.
The website articles that describe them moved from measures-*.html to
metrics-*.html, and plots-basic-measures.html to
plots-basic-metrics.html. The old addresses redirect to the new ones.
print() on a basic-mode object now heads its summary column Metric
rather than Meas., and its banner reads
=== Basic performance evaluation metrics ===.
Correct the count in README.md, which said ten opt-in metrics when there
have been seventeen since 0.18.0.
Add the jaccard, positive_likelihood_ratio and
negative_likelihood_ratio basic measures, opt-in through
evalmod(metrics = ) like the other added measures. They are the
scikit-learn classification metrics precrec did not already have under
a name of its own.
jaccard is the Jaccard index, TP / (TP + FP + FN), also called the
critical success index or threat score. It is the confusion matrix with its
true negative corner left out, which is the same omission precision and
sensitivity make, and it is why it holds still on an imbalanced dataset
where accuracy is mostly counting true negatives.
The two likelihood ratios are sensitivity / fpr and fnr / specificity.
precrec already carried their quotient - the diagnostic odds ratio
odds is LR+ / LR- - but neither one alone, and a classifier can be
worth using on the strength of one of them. Both divide by a rate that is
0 over part of every ranking, and are NA there, as odds and lift
already are where their own denominator vanishes.
Add the D2 scores d2_brier and d2_logloss to prob_metrics() and
prob_metrics_ci(), through a new metrics argument.
A D2 score divides a loss by the loss of the null model - the one that
predicts the observed prevalence for every case and ignores the scores -
and subtracts the result from 1, so it reads like R-squared: 1 is
perfect, 0 is no better than knowing the prevalence, and a negative value
is a model that does worse than that. Negative values are not clipped, and
a dataset holding a single class has a null loss of 0 and so a D2 score
of NA.
The argument defaults to NULL, which is the three metrics
prob_metrics() has always returned, so an existing call still gets three
rows per model and dataset. metrics = "all" asks for all five.
Add the roc_dist and sedi basic measures, opt-in through
evalmod(metrics = ) like the other added measures.
roc_dist is the distance from the point (1 - specificity, sensitivity)
to the perfect corner of ROC space. It is the one measure in the table that
is better when it is smaller, and the one whose maximum is sqrt(2) rather
than 1; minimizing it is one of the standard ways of choosing an
operating point off a ROC curve.
sedi is the symmetric extremal dependence index, a skill score from
forecast verification built to stay informative when the positive class is
rare. The four logarithms in its definition are undefined at both ends of
every dataset, so the rates are clamped away from 0 and 1 first.
Add average_precision(), which returns the step estimator of the area
under a precision-recall curve - the precision at each cutoff weighted by
the recall it gains over the cutoff before it.
It is a different estimator from the one auc() reports, not a different
way of adding up the same one: it joins the raw precision-recall points
with horizontal steps, where auc() measures the area under the curve
evalmod() has interpolated the way Davis and Goadrich showed is correct.
The step estimator reads high wherever the two disagree, and auc()
remains the number to prefer. average_precision() is here because several
other packages report it under this name, and because the size of the gap
between the two is worth being able to see.
It is read off the raw per-cutoff points rather than off the curve, so
x_bins does not change it, and unlike prbe() it survives
evalmod(raw_curves = FALSE).
Add the macro_weight argument to auc(). The macro-average of the
per-class AUCs of a multiclass evaluation has always weighted every class
equally; macro_weight = "prevalence" weights each class by the number of
observations it has instead, and names the added rows
macro-average-weighted to keep the two apart. The default is unchanged,
and the two agree on a balanced dataset.
For a ROC evaluation the two are the measures other packages call
roc_aunu and roc_aunp.
Rewrite the package website. The single long Introduction vignette is now
a short Get started page plus three sets of short articles, reachable
from new How-to, Measures and Plots menus: one page per task, one
page per family of measures, and one page per plot. The vignette shipped
with the package is the Get started page; the rest are on the website
only, so nothing was added to the size of the package.
Update README.md to match, and to describe the measures added in 0.15.0
and 0.16.0.
Shrink the logo. man/figures/logo.png was 3392 x 3392 pixels and 430 KB,
and ships inside the package; it is now 240 x 240 and 13 KB, which is the
size the README and the website actually display. The favicons are
regenerated from it.
label column of as.data.frame(evalmod(mode = "basic")), and the
label panel plot() and autoplot() draw from it, could come out in a
different order on macOS than on Linux for the same data. Tied scores now
keep the order they were given in, on every platform. No evaluation
measure changes; only the order of the tied instances themselves.Add metric_curve(), which takes the name of a measure for the x axis and
the name of a measure for the y axis and draws one against the other, the
way ROCR::performance() does. Every measure evalmod() can calculate is
available on both axes, under its own name or under the ROCR identifier.
It returns an ssxycurves, msxycurves, smxycurves or mmxycurves
object, chosen the way evalmod() chooses between its own, and the object
works with print(), as.data.frame(), fortify(), plot() and
autoplot().
Only two pairs are joined by a line: false positive rate against
sensitivity, which is the ROC curve, and sensitivity against precision,
which is the precision-recall curve. Those two have a defined
interpolation, and for them metric_curve() hands the work to the same
code evalmod(mode = "rocprc") uses, so the two cannot disagree. Every
other pair is drawn as points, because joining raw per-cutoff points with
straight lines is the error this package was written to avoid; pass
type = "l" to join them anyway.
metric_curve() draws one curve per test dataset and does not average
over them. An average needs a rule for interpolating between the points of
each curve, which is what an unregistered pair does not have.
Add the evaluation measures ROCR provides that precrec did not:
fpr, fnr, false_discovery_rate, false_omission_rate,
predicted_positive_rate, predicted_negative_rate, lift, odds,
mi, chisq and cost. Each also answers to the identifier ROCR uses
for it - fall, miss, pcfall, pcmiss, rpp, rnp,
mutual_information - and to its standard abbreviation where it has one,
so a call written against ROCR keeps working. evalmod() gains
cost_fp and cost_fn for the two weights the cost measure takes;
with the default weights of 1 it is the error rate.
They are not calculated unless asked for. evalmod() gains a metrics
argument that names the measures to add, or takes "all"; the default
NULL is the fourteen measures the function has always returned, so an
existing call gets the same object with the same fourteen plot() and
autoplot() panels. A measure that was not calculated cannot be plotted,
and the error says which argument asks for it.
The odds ratio and the chi-square statistic are NA at the top and the
bottom of every dataset, where the 2x2 table has an empty cell and neither
is defined. ROCR reports an infinity or a NaN there. NA is what
precision and npv already do with their own undefined end, and it
keeps an infinity off a shared axis. The mutual information is 0 at
those two points rather than NA, because a cutoff that predicts one
class for everything carries no information about the labels - that value
is defined, and it is zero.
Add prbe(), which finds the points of a precision-recall curve at which
precision and recall are equal. It takes the object evalmod() returns and
gives back a data frame with one row per break-even point, in the manner of
auc().
ROCR::performance(pred, "prbe") interpolates linearly between adjacent
raw precision-recall points to find the crossing, which is not correct and
is the reason this package exists; prbe() reads the crossing off the
curve evalmod() has already interpolated properly.
Add the sar measure, the mean of accuracy, the AUC of the ROC curve, and
one minus the root mean squared error. Like the other added measures it is
opt-in through evalmod(metrics = ). The RMSE reads the values of the
scores rather than their ranks, so sar warns and returns NA when the
scores are not probabilities between 0 and 1; every other measure asked for
in the same call is still returned.
Add the root mean squared error to prob_metrics() and
prob_metrics_ci(), as the "rmse" metric. It is the square root of the
Brier score, so each model and dataset now takes up three rows rather than
two.
Declare stats in Imports. It was used but not listed.
Fix the y axis of plot() for informedness and markedness. Both run from
-1 to 1, and both were drawn on a 0 to 1 axis, which cut off the negative
half of the curve. autoplot() was never affected. The axis range of a
measure now comes from one table rather than from a list of names that had
the internal short names missing from it.
Support datasets with more than two classes, by one-vs-rest decomposition.
Pass a matrix with one score column per class together with the class
labels and mmdata() builds one binary problem per class, carried on the
model axis, so evalmod(), plot(), autoplot(), as.data.frame() and
the averaging all treat the classes as they would several models on one
test set. mmdata() and evalmod() gain a multiclass argument
("none" or "ovr"), detected from the input when it is left unset;
binary input is read exactly as before. auc() gains a macro argument
and reports the macro-average of the per-class AUCs alongside them. The
new C3N150 sample dataset is a 3-class example.
Each one-vs-rest decomposition has its own class balance, so the baseline of a precision-recall curve differs from class to class. The plots now leave that baseline out whenever the datasets do not share a prevalence, rather than drawing one line that fits none of them.
Handle a dataset in which every label belongs to the same class instead of
stopping on it. evalmod(mode = "basic") now warns and calculates what
it can - accuracy and error rate are defined, specificity without
negatives and sensitivity without positives come back as NA - where it
used to stop. ROC and precision-recall curves remain undefined for such a
dataset, so evalmod() still stops by default; the new
on_single_class = "na" asks it to warn and return NA instead, so that
one degenerate fold of an n-fold run no longer aborts the whole
evaluation. auc_ci() and prob_metrics_ci() leave those NAs out of
the interval and report how many datasets it was built from.
Add five confusion-matrix measures to evalmod(mode = "basic"): balanced
accuracy, negative predictive value, informedness (Youden's J), markedness,
and Cohen's kappa. They are calculated in the same pass as the existing
measures and appear as new curvetype values in as.data.frame(),
plot(), autoplot(), fortify() and print(). Code that plots the
default set of measures now gets fourteen panels instead of nine; pass
curvetype to pick a subset. evalmod(mode = "basic") does about a third
more work for them and holds about 40% more memory at peak;
evalmod() in its default mode is unaffected, because the curves are drawn
from three measures and the rest are no longer built for it.
Add a beta argument to evalmod(), which generalizes fscore to the
F-beta score. The default beta = 1 is the F1 score the function has always
returned, so existing results are unchanged.
Add prob_metrics() and prob_metrics_ci(), which calculate the Brier
score and the log loss of prediction scores together with their confidence
intervals over multiple test datasets. Both measures read the values of the
scores rather than their ranks, so the scores must be probabilities;
anything outside the range 0 to 1 is rejected with a
precrec_error_invalid_scores condition.
Replace the vdiffr plot tests with machine-independent snapshots, and
drop vdiffr from Suggests. The tests now record what a plot is made of
-- its panels, their titles and axis labels, and the x/y/group data behind
every layer -- as text, instead of comparing a rendered SVG. Rendering
brought in the local font metrics, so a baseline belonged to one machine,
could not be committed and had to be skipped on CI. The new baselines live
in tests/testthat/_snaps/ under version control, the comparison runs
everywhere including CI, and a failure names what changed.
Stop autoplot() and fortify() warning "Arguments in ... must be used"
on every call. The fortify methods accept raw_curves and
reduce_points for a common interface, but a single test dataset has no
average to contrast a raw curve with and the basic measures have no point
reduction, so those methods never read the argument -- which the ggplot2
fortify generic reported as a possible misspelling. The behavior is
unchanged; the arguments are now consumed explicitly and documented as
having no effect on those objects.
Fix the panel titles of autoplot() in mode = "basic". The measure name
arrives from the plot data as a factor, and the title lookup read it by its
level code instead of its name, so the label panel was titled "MCC" and
the mcc, npv and balanced_accuracy panels were left untitled.
Require R >= 4.1 (was R >= 3.2.1)
Migrate the unit tests to testthat edition 3, and run test files in parallel
Remove the deprecated context() calls from the unit tests
Convert the roxygen2 documentation to markdown. Links to functions now
render with parentheses, such as evalmod(), and links to other packages
are qualified, such as gridExtra::arrangeGrob(). The rendered help pages
are otherwise unchanged.
Replace assertthat with cli and rlang for argument validation.
assertthat is no longer a dependency. Errors raised by argument checks now
carry condition classes, so they can be caught by class rather than by
message: precrec_error_invalid_<argument>, precrec_error_invalid_arg and
precrec_error. The wording of these messages has changed, but the same
inputs are accepted and rejected as before.
Suggest the nearest valid value when an argument is given one that is not
allowed. evalmod(mode = "ROCPRC") now points at rocprc in the error, and
a curvetype with a typo in it is matched against the measure
names.
A value that resembles nothing in the set is reported as before, without a
guess.
Fix the error raised when an argument that must be a whole number is given
an infinite one. evalmod(x_bins = Inf) failed with R's own "missing value
where TRUE/FALSE needed" rather than a precrec condition, because
Inf %% 1 is NaN.
Use checkmate for the argument type checks behind the existing helpers.
The messages and the condition classes are unchanged; checkmate and its
only dependency, backports, are the new Imports.
Replace the apply family with a small internal .map_* family. All 44
lapply, vapply and Filter calls in the package now go through helpers
named after their purrr equivalents, and most of them through a typed one
that states what the call returns. No dependency was added and no behavior
changed: the plot snapshots, the correctness suite and the benchmarks are
the same before and after.
Add the package website to URL in DESCRIPTION, and rebuild the pkgdown
site with the Bootstrap 5 template.
Add inst/WORDLIST and a spell-check test, plus a committed .lintr
configuration.
Fix the ranking of NA scores when na_worst = TRUE. The sentinel value
used for NA was DBL_MIN, the smallest positive double, so NAs
outranked every negative score instead of being ranked last. Results change
only when the scores contain both NAs and negative values: evalmod()
now returns the same curves and AUCs for such input as it does when a
constant is added to every score. na_worst = FALSE was never affected.
Calculate the standard errors of averaged curves and averaged points with
Welford's algorithm instead of E[x^2] - E[x]^2. The previous formula lost
precision through catastrophic cancellation and clamped the resulting
negative variances to zero. Confidence bands from evalmod(calc_avg = TRUE)
change at around the 1e-8 level.
Add as.data.table() methods for the objects evalmod() returns, so the
table precrec builds internally can be had without a conversion. They
accept the same arguments as as.data.frame() and hold the same content.
data.table was already a dependency.
Build the internal tables with data.table. The public contract is
unchanged: as.data.frame(), fortify(), auc(), pauc() and
auc_ci() still return plain data frames, with the same columns, types
and row order as before. Tables reached through the returned frame can no
longer be modified by reference.
Speed up the pure-R conversion path used when use_rcpp = FALSE. It grew
the result with rbind() once per curve, copying everything collected so
far on every pass. Over 20 test datasets it is now around 38 times faster
and allocates around 43 times less memory. auc_ci() and the basic
measure summary had the same pattern and got the same treatment. The
default Rcpp path was never affected.
Speed up as.data.frame() and autoplot() for large curve objects, and
cut the memory they need. The C++ converter filled a C++ buffer and then
copied it into the vectors it returned; it now fills those vectors
directly. Converting a curve object built from 200,000 observations is
around 27% faster, and one pass of evalmod() plus as.data.frame() over
1,000,000 observations peaks at 584 MB instead of 897 MB in
mode = "basic", and 424 MB instead of 470 MB for ROC and
precision-recall curves.
Speed up mmdata() by around 25%. Scores are sorted through a comparison
the sort can inline, rather than through a function pointer, which it
could not. The resulting ranks are unchanged.
Speed up the averaging of basic evaluation measures across datasets,
which evalmod(calc_avg = TRUE, mode = "basic") performs. Collecting the
distinct x values used a std::set and a std::map, each allocating a
node per value and following a pointer per lookup, once for every point
of every dataset; a sorted vector and a binary search replace them. It is
around 1.6 times faster at 100,000 observations and above.
Restructure unit tests for svg comparisons with vdiff
Reformat signatures of S3 methods
Update unit tests to avoid is.atomic(NULL) issue
Update argument names of S3 functions to keep them consistent
Use patchwork to combine multiple plots instead of using grid and gridExtra
Test ggplot results with vdiffr
Fix a bug with as.data.frame when multiple datasets given
Add format_nfold function to convert a dataframe with n-fold data to a list
Add 'aucroc' mode for fast AUC (ROC)
Change how to treat 'show_cb' and 'raw_curves' options
Improve as.data.frame with Rcpp
Create GitHub pages with pkgdown
Add new measures
New generic function
Improved the testing environment
Improved several documents
The first release version of precrec
The package offers five functions