Data Engineering Orchestrating Data Workflows 2 — Questions and Answers
Question 1: In Apache Airflow, what does a DAG represent?
- A directed acyclic graph defining task dependencies (Correct answer)
- A database connection pool
- A distributed file storage layer
- A message queue partition
Correct answer: A directed acyclic graph defining task dependencies
A DAG (Directed Acyclic Graph) defines tasks and the order in which they run, with no cycles.
Question 2: Which Airflow concept controls how many task instances can run concurrently for a single DAG?
- max_active_runs
- concurrency (max_active_tasks) (Correct answer)
- retries
- schedule_interval
Correct answer: concurrency (max_active_tasks)
The concurrency (max_active_tasks) setting limits how many task instances run at once within a DAG.
Question 3: What is the purpose of an idempotent task in a data pipeline?
- It runs faster on each retry
- Re-running it produces the same result without side effects (Correct answer)
- It never needs to be scheduled
- It bypasses dependency checks
Correct answer: Re-running it produces the same result without side effects
Idempotent tasks can be safely retried because repeated execution yields the same end state.
Question 4: In a workflow scheduler, what is 'backfilling'?
- Deleting old task logs
- Running a DAG for past dates that were missed or newly added (Correct answer)
- Increasing worker memory
- Compressing intermediate data
Correct answer: Running a DAG for past dates that were missed or newly added
Backfilling executes a pipeline over historical intervals to populate missing data.
Question 5: Which statement best describes the difference between scheduling and orchestration?
- They are identical terms
- Scheduling triggers jobs by time; orchestration manages dependencies and coordination (Correct answer)
- Orchestration only runs single tasks
- Scheduling handles error recovery exclusively
Correct answer: Scheduling triggers jobs by time; orchestration manages dependencies and coordination
Scheduling decides when jobs run, while orchestration coordinates dependencies, retries, and data flow across tasks.
Question 6: In Airflow, what does an XCom enable?
- Cross-DAG authentication
- Passing small amounts of data between tasks (Correct answer)
- External cloud storage mounting
- Compression of task logs
Correct answer: Passing small amounts of data between tasks
XComs (cross-communications) let tasks exchange small pieces of data within a DAG.
Question 7: What is a common reason to use a sensor in an orchestration tool?
- To wait for an external condition like a file or partition to appear (Correct answer)
- To compress output files
- To allocate GPU resources
- To encrypt connection strings
Correct answer: To wait for an external condition like a file or partition to appear
Sensors pause a workflow until a specified external event or condition is met.
Data Engineering Orchestrating Data Workflows 2
In Apache Airflow, what does a DAG represent?