Model Performance and Evaluation Flashcards

Stratified k-fold cross-validation differs from standard k-fold by:

Answer: Preserving the class distribution in each fold

Stratified folds maintain the original class proportions in every fold, important for imbalanced data.

Specificity measures a classifier's ability to correctly identify:

Answer: Actual negatives

Specificity (true negative rate) is the proportion of actual negatives correctly identified.

If two models have identical accuracy but different ROC-AUC, the higher-AUC model is better at:

Answer: Ranking positives above negatives across thresholds

AUC reflects ranking quality across all thresholds, independent of a single decision cutoff.

A learning curve where both training and validation error remain high and close together indicates:

Answer: Underfitting (high bias)

High, converged errors signal underfitting, where the model is too simple for the data.

Cohen's Kappa is preferred over raw accuracy because it:

Answer: Accounts for agreement expected by chance

Cohen's Kappa corrects observed accuracy for the agreement expected by random chance.

When reporting model performance, why is a confidence interval on the metric valuable?

Answer: It quantifies uncertainty in the estimate due to finite test data

Confidence intervals communicate how much the metric might vary given limited test data.

Nested cross-validation is used primarily to:

Answer: Obtain an unbiased performance estimate while tuning hyperparameters

Nested CV separates hyperparameter tuning (inner loop) from performance estimation (outer loop) to avoid optimistic bias.