Data Science Model Evaluation and Validation 3 — Questions and Answers
Question 1: What is the primary purpose of k-fold cross-validation?
- To obtain a more reliable performance estimate by averaging over multiple train/test splits (Correct answer)
- To increase the size of the training set permanently
- To remove outliers from data
- To tune the learning rate automatically
Correct answer: To obtain a more reliable performance estimate by averaging over multiple train/test splits
K-fold averages performance across k splits, reducing variance from any single split.
Question 2: In 5-fold cross-validation, what fraction of data is used for testing in each fold?
- About 20% (Correct answer)
- About 50%
- About 80%
- About 5%
Correct answer: About 20%
With 5 folds, each test fold is 1/5 (20%) while 80% trains the model.
Question 3: When should stratified k-fold cross-validation be preferred?
- When the dataset has imbalanced class distributions (Correct answer)
- When all features are numeric
- When the dataset is extremely large
- When using only regression models
Correct answer: When the dataset has imbalanced class distributions
Stratified folds preserve class proportions in each fold, important for imbalanced classes.
Question 4: Leave-one-out cross-validation (LOOCV) uses how many folds?
- As many folds as there are data points (Correct answer)
- Exactly 2
- Exactly 10
- The square root of the sample size
Correct answer: As many folds as there are data points
LOOCV sets k equal to n, testing on one sample at a time.
Question 5: Why must time-series data use specialized cross-validation like forward chaining?
- Random shuffling would leak future information into training (Correct answer)
- Time-series data cannot be split
- It has no target variable
- Standard folds run faster
Correct answer: Random shuffling would leak future information into training
Temporal order must be preserved so the model never trains on future data to predict the past.
Question 6: A key drawback of LOOCV compared to 10-fold cross-validation is:
- High computational cost since the model is trained n times (Correct answer)
- It cannot estimate variance
- It always overfits
- It requires a separate validation set
Correct answer: High computational cost since the model is trained n times
LOOCV trains n models, which is expensive for large datasets.
Question 7: The validation set in a train/validation/test split is primarily used for:
- Tuning hyperparameters and model selection (Correct answer)
- Final unbiased performance reporting
- Generating new training samples
- Computing feature correlations
Correct answer: Tuning hyperparameters and model selection
The validation set guides hyperparameter tuning while the test set gives the final estimate.
Data Science Model Evaluation and Validation 3
What is the primary purpose of k-fold cross-validation?