Data Wrangling and Preprocessing Flashcards

A dataset for customer analysis contains a 'Country' column with entries like "USA", "U.S.", "United States", and "America". Before using this feature in a model, a data scientist combines these into a single category, "USA". What specific data quality issue is being addressed?

Answer: Lack of consistency

The issue is that the same real-world entity is represented in multiple ways. This is a classic example of inconsistent data. [23, 24] Standardizing these entries into a single, uniform format resolves the lack of consistency, ensuring that analyses and models treat all these variations as the same entity. [19]

A data scientist is preparing features for a machine learning model that is sensitive to feature scale. One feature has a distribution that is approximately Gaussian (normal), while another is heavily skewed. Which combination of scaling techniques is generally most appropriate?

Answer: Apply standardization (Z-score scaling) to the Gaussian feature and normalization (Min-Max scaling) to the skewed feature.

Standardization (Z-score scaling) is ideal for features that are normally distributed, as it centers the data at a mean of 0 and scales it to a standard deviation of 1, which aligns with the assumptions of many algorithms. [25, 26] Normalization (Min-Max scaling) is often more suitable for non-normal distributions, as it scales data into a fixed range (e.g., 0 to 1) without assuming a specific distribution. [21]

You are analyzing a dataset of housing prices where the 'Square_Footage' feature contains several legitimate but extreme values that are significantly influencing the mean. To prepare this data for a linear regression model, which of the following is the most robust method to handle the influence of these outliers?

Answer: Capping the feature at the 1st and 99th percentile values.

Capping (also known as winsorizing) is a robust method to handle outliers without losing the data points entirely. It replaces the extreme values with a specified percentile value, thereby reducing their skewing effect on the model. [5] Deleting rows can lead to information loss, and imputing with the mean is not robust as the mean itself is sensitive to outliers. [17]

A data analyst has a dataset where each row represents a student, and columns represent their scores in different subjects: `StudentID`, `Math_Score`, `Science_Score`, `History_Score`. For a specific analysis, the data needs to be restructured so that each row contains a student ID, a subject, and the corresponding score. Which data wrangling operation should be performed?

Answer: Melting

Melting is the process of transforming a dataset from a wide format to a long format. [9, 13] In this case, the multiple subject score columns are "melted" into two new columns: one for the subject name ('variable') and one for the score ('value'), making the data tidy for certain plotting and analysis tasks. [4, 22] Pivoting is the reverse operation.

In a clinical trial dataset, data for a follow-up blood pressure measurement is more likely to be missing for patients whose initial blood pressure was very high, as they may have dropped out of the study due to health complications. This scenario is an example of which type of missing data mechanism?

Answer: Missing Not at Random (MNAR)

This is an example of Missing Not at Random (MNAR) because the reason for the data being missing is directly related to the (unobserved) value of the missing data itself. The probability of a blood pressure reading being missing depends on what that reading would have been (i.e., high). [10, 14] This is the most challenging type of missing data to handle as it introduces significant bias. [12]

A data scientist decides to convert a continuous 'Age' feature into a categorical 'Age_Group' feature (e.g., '18-25', '26-40', '41-60', '61+'). Which of the following is a primary benefit of this technique, known as binning?

Answer: It helps to capture non-linear relationships when using linear models.

Binning (or discretization) can help linear models capture non-linear relationships. [1] For example, the effect of age on a target variable might not be linear. By converting age into bins, a linear model can assign a different weight to each age group, effectively modeling a non-linear pattern without using a more complex model. [16]