MS-DS Master of Data science MS-DS Master of Data science Model Evaluation and Validation Questions and Answers 3

Question 1

What is the primary advantage of using the Matthews Correlation Coefficient (MCC) over accuracy for binary classification on imbalanced datasets?

Accepted Answer

MCC accounts for all four confusion matrix categories proportionally

Answer

MCC ignores true negatives entirely

Answer

MCC is always higher than accuracy

Answer

MCC does not require a decision threshold

Question 2

During model validation, what problem does data leakage most commonly cause?

Accepted Answer

Overly optimistic performance estimates that fail to replicate on new data

Answer

Underfitting due to reduced training set size

Answer

Increased computational cost during cross-validation

Answer

Lower precision but higher recall on the test set

Question 3

In the context of learning curves, what does convergence of training and validation scores at a low performance level suggest?

Accepted Answer

High bias — the model is too simple for the data

Answer

High variance — the model is too complex

Answer

The dataset has too many features

Answer

The learning rate is set too high

Question 4

Which validation technique is specifically designed for time-series data to prevent future information from leaking into training?

Accepted Answer

Walk-forward (expanding window) validation

Answer

Stratified k-fold cross-validation

Answer

Leave-one-out cross-validation

Answer

Standard random holdout split

Question 5

What does the area under the Precision-Recall curve (AUPRC) measure that AUROC may fail to capture?

Accepted Answer

Model performance on the minority class in imbalanced settings

Answer

The overall accuracy across all decision thresholds

Answer

The speed of model convergence during training

Answer

The correlation between predicted and actual values

Question 6

When using bootstrap validation (0.632+ bootstrap), what does the 0.632 factor represent?

Accepted Answer

The expected proportion of unique training samples in each bootstrap sample

Answer

The minimum required accuracy for model acceptance

Answer

The ratio of test samples to training samples

Answer

The confidence level of the performance estimate