Data Science Model Evaluation and Validation 2 — Questions and Answers
Question 1: In a binary classifier, what does precision measure?
- Of all predicted positives, how many were actually positive (Correct answer)
- Of all actual positives, how many were predicted positive
- The overall fraction of correct predictions
- The area under the ROC curve
Correct answer: Of all predicted positives, how many were actually positive
Precision = TP / (TP + FP), the proportion of positive predictions that are correct.
Question 2: What does recall (sensitivity) measure?
- Of all actual positives, how many were correctly identified (Correct answer)
- Of all predicted positives, how many were correct
- The fraction of true negatives identified
- The harmonic mean of precision and accuracy
Correct answer: Of all actual positives, how many were correctly identified
Recall = TP / (TP + FN), the proportion of actual positives correctly detected.
Question 3: Why is accuracy a poor metric for highly imbalanced datasets?
- A model predicting the majority class always can score high while missing the minority class (Correct answer)
- It is impossible to compute on imbalanced data
- It always equals 50%
- It only works for regression problems
Correct answer: A model predicting the majority class always can score high while missing the minority class
With rare positives, always predicting the majority class yields high accuracy despite no useful detection.
Question 4: The F1 score is best described as the:
- Harmonic mean of precision and recall (Correct answer)
- Arithmetic mean of precision and recall
- Product of accuracy and recall
- Difference between precision and recall
Correct answer: Harmonic mean of precision and recall
F1 = 2·(precision·recall)/(precision+recall), the harmonic mean balancing both.
Question 5: What does the ROC curve plot?
- True positive rate against false positive rate across thresholds (Correct answer)
- Precision against recall
- Accuracy against threshold
- Loss against epoch
Correct answer: True positive rate against false positive rate across thresholds
The ROC curve plots TPR vs FPR as the classification threshold varies.
Question 6: An AUC of 0.5 indicates a classifier that:
- Performs no better than random guessing (Correct answer)
- Is perfect
- Always predicts the positive class
- Has zero false positives
Correct answer: Performs no better than random guessing
AUC = 0.5 corresponds to chance-level discrimination between classes.
Question 7: In a confusion matrix, a false negative occurs when the model:
- Predicts negative for an actual positive case (Correct answer)
- Predicts positive for an actual negative case
- Predicts positive for an actual positive case
- Predicts negative for an actual negative case
Correct answer: Predicts negative for an actual positive case
A false negative is an actual positive that the model labeled negative.
Data Science Model Evaluation and Validation 2
In a binary classifier, what does precision measure?