MS-DS Master of Data science Model Evaluation and Validation 5 — Questions and Answers
Question 1: Which of the following is a consequence of using the test set multiple times during model development?
- Increased irreducible error
- Optimistic bias in generalization estimates (Correct answer)
- Higher training variance
- Slower model convergence
Correct answer: Optimistic bias in generalization estimates
Repeatedly evaluating on the test set allows implicit overfitting to it, making the reported metrics overly optimistic and no longer representative of true generalization.
Question 2: Lift in model evaluation is calculated as:
- (Precision of model) / (Precision of random model) (Correct answer)
- AUC-ROC minus 0.5
- True positive rate minus false positive rate
- F1 score divided by accuracy
Correct answer: (Precision of model) / (Precision of random model)
Lift measures how much better the model performs compared to a random baseline, calculated as the model's response rate in a decile divided by the overall response rate.
Question 3: What is the purpose of a reliability diagram (calibration plot)?
- To show how training loss decreases over epochs
- To visualize whether predicted probabilities match observed event frequencies (Correct answer)
- To compare two models' ROC curves on the same axes
- To display the distribution of residuals across predicted values
Correct answer: To visualize whether predicted probabilities match observed event frequencies
A reliability diagram bins predictions by predicted probability and plots mean predicted probability versus observed frequency, revealing over- or under-confidence.
Question 4: In a multiclass classification problem, macro-averaged F1 differs from weighted-averaged F1 in that macro-averaging:
- Weights each class by its support (number of samples)
- Gives equal weight to each class regardless of class frequency (Correct answer)
- Only considers the top two most frequent classes
- Is equivalent to accuracy when classes are balanced
Correct answer: Gives equal weight to each class regardless of class frequency
Macro-averaging computes F1 for each class independently and takes an unweighted mean, treating all classes equally regardless of how many samples they contain.
Question 5: What does a learning curve showing high training error and high validation error indicate?
- Overfitting — model is too complex
- Underfitting — model lacks capacity to capture the pattern (Correct answer)
- Data leakage in the validation set
- Optimal bias-variance tradeoff achieved
Correct answer: Underfitting — model lacks capacity to capture the pattern
When both training and validation errors are high and converge, the model is underfitting because it cannot capture the underlying pattern in the data.
Question 6: Platt scaling is a technique used to:
- Reduce model overfitting by shrinking coefficients
- Convert a classifier's raw scores into calibrated probability estimates (Correct answer)
- Scale features to unit variance before evaluation
- Adjust predictions for class imbalance via threshold shifting
Correct answer: Convert a classifier's raw scores into calibrated probability estimates
Platt scaling fits a logistic regression on the classifier's output scores to transform them into well-calibrated posterior probabilities.
Question 7: When comparing multiple models across several datasets, the Friedman test followed by a Nemenyi post-hoc test is preferred over pairwise t-tests because:
- It requires fewer hyperparameter assumptions
- It controls the family-wise error rate inflated by multiple comparisons (Correct answer)
- It can handle continuous and categorical metrics simultaneously
- It is faster to compute for large benchmark suites
Correct answer: It controls the family-wise error rate inflated by multiple comparisons
The Friedman-Nemenyi procedure is a non-parametric approach that corrects for multiple comparisons, preventing inflated Type I error rates when simultaneously comparing many models.
MS-DS Master of Data science Model Evaluation and Validation 5
Which of the following is a consequence of using the test set multiple times during model development?