Data Science Model Performance and Evaluation 3 — Questions and Answers
Question 1: For a regression model, which metric penalizes large errors more heavily because it squares the residuals?
- Mean Absolute Error (MAE)
- Mean Squared Error (MSE) (Correct answer)
- Median Absolute Error
- R-squared
Correct answer: Mean Squared Error (MSE)
MSE squares residuals, so larger errors contribute disproportionately more to the total.
Question 2: R-squared of 0.0 in a regression model indicates the model:
- Perfectly fits the data
- Explains no more variance than predicting the mean (Correct answer)
- Has negative residuals
- Is overfitting
Correct answer: Explains no more variance than predicting the mean
An R-squared of 0 means the model does no better than always predicting the target mean.
Question 3: When tuning hyperparameters, why should you use a separate validation set rather than the test set?
- To increase training size
- To avoid leaking test information and overly optimistic estimates (Correct answer)
- To reduce computation
- Because test sets cannot be reused at all
Correct answer: To avoid leaking test information and overly optimistic estimates
Tuning on the test set leaks information and inflates the final reported performance.
Question 4: A model's predicted probabilities are systematically too confident. Which technique addresses this?
- Feature scaling
- Probability calibration (e.g., Platt scaling) (Correct answer)
- Increasing learning rate
- Dropout
Correct answer: Probability calibration (e.g., Platt scaling)
Calibration methods like Platt scaling or isotonic regression align predicted probabilities with observed frequencies.
Question 5: In the bias-variance tradeoff, a model that is too simple to capture the underlying pattern exhibits:
- High variance, low bias
- High bias, low variance (Correct answer)
- Low bias, low variance
- High variance, high bias
Correct answer: High bias, low variance
An overly simple model underfits, showing high bias and low variance.
Question 6: Which metric is most appropriate when both false positives and false negatives carry costs and classes are imbalanced?
- Accuracy
- F1 score (Correct answer)
- True negative rate
- Mean squared error
Correct answer: F1 score
F1 balances precision and recall, making it suitable for imbalanced data where both error types matter.
Question 7: A precision-recall curve is generally preferred over an ROC curve when:
- Classes are perfectly balanced
- The positive class is rare (highly imbalanced) (Correct answer)
- The model is a regressor
- There are no false positives
Correct answer: The positive class is rare (highly imbalanced)
PR curves are more informative than ROC curves when the positive class is rare.
Data Science Model Performance and Evaluation 3
For a regression model, which metric penalizes large errors more heavily because it squares the residuals?