DSE Model Evaluation and Validation 4 — Questions and Answers
Question 1: What is 'data leakage' in the context of model validation?
- Data accidentally deleted during preprocessing
- Information from the test set influencing model training or evaluation (Correct answer)
- Missing values in the training set
- Overfitting due to too many features
Correct answer: Information from the test set influencing model training or evaluation
Data leakage occurs when information from outside the training boundary (e.g., test labels or future data) influences the model, causing overly optimistic results.
Question 2: A model trained on hospital A data is tested on hospital B data and performs much worse. This primarily illustrates:
- Overfitting to training data
- Distribution shift / covariate shift (Correct answer)
- Model underfitting
- Label noise
Correct answer: Distribution shift / covariate shift
Distribution shift occurs when the statistical properties of the test environment differ from training, leading to performance degradation.
Question 3: The G-mean metric for imbalanced classification is calculated as:
- (Precision + Recall) / 2
- √(Sensitivity × Specificity) (Correct answer)
- TP / (TP + FP + FN)
- 2 × (Precision × Recall) / (Precision + Recall)
Correct answer: √(Sensitivity × Specificity)
G-mean is the geometric mean of sensitivity and specificity, balancing performance across both the positive and negative classes.
Question 4: When evaluating a multi-class classifier, 'macro-averaged' F1-score differs from 'weighted-averaged' F1-score in that macro averaging:
- Gives more weight to larger classes
- Gives equal weight to all classes regardless of size (Correct answer)
- Only considers the majority class
- Requires one-vs-rest decomposition
Correct answer: Gives equal weight to all classes regardless of size
Macro-averaging computes the metric for each class independently and takes the unweighted mean, treating all classes equally regardless of support.
Question 5: Which of the following is a sign of high variance (overfitting) when examining learning curves?
- Both training and validation error are high
- Training error is low but validation error is much higher (Correct answer)
- Training error equals validation error and both are high
- Validation error decreases then plateaus
Correct answer: Training error is low but validation error is much higher
A large gap between low training error and high validation error is the classic signature of overfitting (high variance).
Question 6: What is the purpose of the McNemar test in model evaluation?
- Comparing regression model MSEs
- Statistically comparing two classifiers' error rates on the same test set (Correct answer)
- Testing for normality of residuals
- Evaluating calibration quality
Correct answer: Statistically comparing two classifiers' error rates on the same test set
McNemar's test uses a contingency table of paired predictions to determine if two classifiers make statistically different errors on the same samples.
Question 7: In the context of regression evaluation, what does a residual plot where residuals fan out as fitted values increase suggest?
- The model is well-specified
- Heteroscedasticity — non-constant error variance (Correct answer)
- Multicollinearity among predictors
- The model is underfitting
Correct answer: Heteroscedasticity — non-constant error variance
Residuals fanning out (increasing spread) indicate heteroscedasticity, meaning the error variance is not constant across the range of predictions.
DSE Model Evaluation and Validation 4
What is 'data leakage' in the context of model validation?