MS-DS Master of Data science Supervised Learning Algorithms 5 — Questions and Answers
Question 1: Which of the following is a key difference between bagging and boosting ensemble methods?
- Bagging trains models sequentially focusing on hard examples; boosting trains them in parallel
- Bagging trains models in parallel on bootstrapped samples; boosting trains sequentially correcting prior errors (Correct answer)
- Bagging only works for regression; boosting only works for classification
- Bagging uses weighted voting; boosting uses simple averaging
Correct answer: Bagging trains models in parallel on bootstrapped samples; boosting trains sequentially correcting prior errors
Bagging builds independent models in parallel on bootstrap samples and averages results, while boosting builds models sequentially where each corrects the errors of its predecessor.
Question 2: In a neural network, what does batch normalization do during training?
- It randomly drops connections between layers to prevent co-adaptation
- It normalizes activations of each layer across the mini-batch, stabilizing training (Correct answer)
- It constrains weights to a unit norm to prevent exploding gradients
- It scales the learning rate based on the gradient magnitude
Correct answer: It normalizes activations of each layer across the mini-batch, stabilizing training
Batch normalization standardizes activations within each mini-batch, reducing internal covariate shift and allowing higher learning rates and faster convergence.
Question 3: What is the 'curse of dimensionality' and how does it affect k-NN classifiers?
- More features always improve k-NN because more information is available
- In high dimensions, data becomes sparse and distance metrics lose discriminative power (Correct answer)
- k-NN training time grows linearly with the number of features
- High dimensionality causes k-NN to require fewer neighbors for accurate predictions
Correct answer: In high dimensions, data becomes sparse and distance metrics lose discriminative power
As dimensionality grows, data points become increasingly equidistant, making nearest-neighbor distances meaningless and degrading k-NN performance significantly.
Question 4: Which of the following describes the purpose of cross-validation in supervised learning?
- To train multiple models and take the one with lowest training error
- To estimate a model's generalization performance and tune hyperparameters on limited data (Correct answer)
- To augment the dataset by creating synthetic training examples
- To identify and remove outliers before model training
Correct answer: To estimate a model's generalization performance and tune hyperparameters on limited data
Cross-validation partitions data into multiple folds, training and evaluating the model on different subsets to produce a reliable estimate of generalization error.
Question 5: A decision tree trained to zero training error likely suffers from which problem?
- Underfitting due to insufficient depth
- Overfitting due to memorizing training noise (Correct answer)
- High bias because the model is too simple
- Feature leakage from improper data preprocessing
Correct answer: Overfitting due to memorizing training noise
A tree that perfectly fits training data has memorized noise and specific examples, resulting in high variance and poor generalization to unseen data.
Question 6: In the context of AdaBoost, how are training samples reweighted across iterations?
- Correctly classified samples get higher weights to reinforce learning
- Misclassified samples receive higher weights so subsequent classifiers focus on hard examples (Correct answer)
- All samples are reweighted uniformly to maintain equal contribution
- Weights are assigned based on feature importance scores from previous rounds
Correct answer: Misclassified samples receive higher weights so subsequent classifiers focus on hard examples
AdaBoost increases the weights of misclassified samples after each round, forcing the next weak learner to concentrate on the examples the current ensemble handles poorly.
Question 7: What is the main reason Lasso (L1) regression can produce sparse models while Ridge (L2) typically does not?
- Lasso uses a squared penalty that shrinks all coefficients equally
- The L1 penalty creates corners in the constraint region where coefficients are exactly zero (Correct answer)
- Lasso minimizes absolute error while Ridge minimizes squared error
- Ridge applies a stronger penalty than Lasso for the same regularization parameter
Correct answer: The L1 penalty creates corners in the constraint region where coefficients are exactly zero
The diamond-shaped L1 constraint region has corners aligned with the axes; solutions are likely to occur at these corners where some coefficients are exactly zero, producing sparsity.
MS-DS Master of Data science Supervised Learning Algorithms 5
Which of the following is a key difference between bagging and boosting ensemble methods?