Data Science FREE Data Science Model Evaluation and Validation Questions and Answers 2 — Questions and Answers
Question 1: Which metric is most appropriate for evaluating a classification model on a highly imbalanced dataset?
- Accuracy
- Precision-Recall AUC (Correct answer)
- Mean Squared Error
- R-squared
Correct answer: Precision-Recall AUC
Precision-Recall AUC is preferred for imbalanced datasets because accuracy can be misleadingly high when the majority class dominates predictions.
Question 2: What does a high variance and low bias in a model typically indicate?
- Underfitting
- Overfitting (Correct answer)
- Optimal generalization
- Insufficient data preprocessing
Correct answer: Overfitting
High variance with low bias means the model fits training data very well but fails to generalize, which is the hallmark of overfitting.
Question 3: In stratified k-fold cross-validation, what is preserved across each fold?
- The number of features
- The proportion of each class label (Correct answer)
- The order of data points
- The ratio of training to test size
Correct answer: The proportion of each class label
Stratified k-fold ensures each fold maintains the same class distribution as the original dataset, which is critical for representative evaluation.
Question 4: What is the primary purpose of a calibration curve (reliability diagram)?
- To measure feature importance
- To assess whether predicted probabilities match actual outcomes (Correct answer)
- To detect multicollinearity
- To optimize hyperparameters
Correct answer: To assess whether predicted probabilities match actual outcomes
A calibration curve plots predicted probabilities against observed frequencies to show whether a model's confidence scores are trustworthy.
Question 5: When using bootstrapping for model evaluation, what is the typical approach?
- Splitting data into exactly two halves
- Sampling with replacement to create multiple training sets (Correct answer)
- Removing outliers before each evaluation
- Using only the first 80% of data chronologically
Correct answer: Sampling with replacement to create multiple training sets
Bootstrapping creates multiple resampled datasets by sampling with replacement from the original data, allowing robust estimation of model performance statistics.
Question 6: What does the Kolmogorov-Smirnov (KS) statistic measure in model evaluation?
- The correlation between predicted and actual values
- The maximum separation between cumulative distributions of positive and negative classes (Correct answer)
- The average log-likelihood of predictions
- The number of misclassified samples
Correct answer: The maximum separation between cumulative distributions of positive and negative classes
The KS statistic measures the maximum distance between the cumulative distribution functions of positive and negative class scores, indicating discriminative power.
Data Science FREE Data Science Model Evaluation and Validation Questions and Answers 2
Which metric is most appropriate for evaluating a classification model on a highly imbalanced dataset?