Data Science Model Performance and Evaluation 2 — Questions and Answers
Question 1: A spam classifier flags 95 of 100 actual spam emails but also flags 40 legitimate emails as spam. Which metric most directly reflects the cost of those false alarms?
- Recall
- Precision (Correct answer)
- Accuracy
- Sensitivity
Correct answer: Precision
Precision penalizes false positives, capturing the cost of legitimate emails wrongly flagged as spam.
Question 2: On a dataset where 99% of cases are negative, a model that always predicts 'negative' achieves 99% accuracy. What does this illustrate?
- Overfitting
- The accuracy paradox on imbalanced data (Correct answer)
- High variance
- Data leakage
Correct answer: The accuracy paradox on imbalanced data
The accuracy paradox shows accuracy can be misleadingly high on imbalanced data despite a useless model.
Question 3: Which metric is the harmonic mean of precision and recall?
- ROC-AUC
- F1 score (Correct answer)
- Log loss
- R-squared
Correct answer: F1 score
The F1 score is the harmonic mean of precision and recall, balancing both.
Question 4: A regression model has a low training error but a much higher test error. This is a sign of:
- Underfitting
- Overfitting (Correct answer)
- High bias
- Class imbalance
Correct answer: Overfitting
A large gap with low training error and high test error indicates overfitting (high variance).
Question 5: The area under the ROC curve (AUC) represents the probability that the model ranks a randomly chosen positive higher than a randomly chosen negative. An AUC of 0.5 means:
- Perfect classifier
- No better than random guessing (Correct answer)
- Worse than random
- Perfectly calibrated
Correct answer: No better than random guessing
An AUC of 0.5 corresponds to random ranking with no discriminative ability.
Question 6: Which evaluation approach gives a more reliable performance estimate on small datasets by averaging across multiple train/test splits?
- Single holdout split
- k-fold cross-validation (Correct answer)
- Using all data for training
- Bootstrap aggregation of features
Correct answer: k-fold cross-validation
k-fold cross-validation averages performance over multiple folds, reducing variance of the estimate.
Question 7: In a confusion matrix, recall (sensitivity) is calculated as:
- TP / (TP + FP)
- TP / (TP + FN) (Correct answer)
- TN / (TN + FP)
- (TP + TN) / Total
Correct answer: TP / (TP + FN)
Recall is true positives divided by all actual positives, TP / (TP + FN).
Data Science Model Performance and Evaluation 2
A spam classifier flags 95 of 100 actual spam emails but also flags 40 legitimate emails as spam.
Which metric most directly reflects the cost of those false alarms?