ACP ACP Data Engineering & Workflow Automation 2 — Questions and Answers
Question 1: Which Python library is specifically designed for defining, scheduling, and monitoring data pipeline workflows as Directed Acyclic Graphs (DAGs)?
- Luigi
- Apache Airflow
- Prefect
- All of the above (Correct answer)
Correct answer: All of the above
Luigi, Apache Airflow, and Prefect are all Python-native workflow orchestration frameworks that model pipelines as DAGs with scheduling and monitoring capabilities.
Question 2: In pandas, which method is used to apply a custom function to every row or column of a DataFrame?
- df.map()
- df.apply() (Correct answer)
- df.transform()
- df.execute()
Correct answer: df.apply()
`df.apply()` applies a function along an axis (rows with `axis=1`, columns with `axis=0`), enabling custom transformations across the entire DataFrame.
Question 3: Which pandas method efficiently removes duplicate rows from a DataFrame, keeping only the first occurrence by default?
- df.remove_duplicates()
- df.drop_duplicates() (Correct answer)
- df.unique_rows()
- df.deduplicate()
Correct answer: df.drop_duplicates()
`df.drop_duplicates()` returns a DataFrame with duplicate rows removed, with `keep='first'` as default and options for `keep='last'` or `keep=False` to drop all duplicates.
Question 4: When building a data pipeline, what does the term 'idempotency' mean in the context of pipeline task execution?
- A task runs in parallel with zero overhead
- Running a task multiple times produces the same result as running it once (Correct answer)
- A task automatically retries on failure
- A task can process data from any source format
Correct answer: Running a task multiple times produces the same result as running it once
An idempotent pipeline task can be safely re-executed without side effects — running it once or ten times yields the same final state, which is critical for reliable data engineering.
Question 5: Which pandas method stacks a DataFrame from wide format (one column per variable) to long format (one row per observation)?
- pd.wide_to_long()
- df.melt() (Correct answer)
- df.stack()
- df.pivot()
Correct answer: df.melt()
`df.melt()` unpivots a DataFrame from wide to long format by converting specified columns into rows, creating `variable` and `value` columns.
Question 6: In pandas, what does `groupby()` followed by `agg()` allow you to do?
- Sort a DataFrame by multiple columns
- Apply multiple aggregation functions to groups simultaneously (Correct answer)
- Filter rows based on group membership
- Join two DataFrames on a common group key
Correct answer: Apply multiple aggregation functions to groups simultaneously
`df.groupby('col').agg({'col1': 'sum', 'col2': 'mean'})` groups rows and applies different aggregation functions to different columns in one vectorized operation.
ACP ACP Data Engineering & Workflow Automation 2
Which Python library is specifically designed for defining, scheduling, and monitoring data pipeline workflows as Directed Acyclic Graphs (DAGs)?