Orchestrating Data Workflows Flashcards
7 cards from real Data Engineering practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 7 Orchestrating Data Workflows flashcards as text
In a DAG, what does 'fan-out' followed by 'fan-in' typically represent?
Answer: Splitting work into parallel tasks, then aggregating their results
Fan-out runs tasks in parallel and fan-in collects their outputs into a downstream task.
What is a 'dead-letter queue' used for in workflow systems?
Answer: Storing messages or records that repeatedly fail processing
A dead-letter queue isolates failed items for later inspection instead of blocking the pipeline.
Which is a key consideration when orchestrating tasks across multiple time zones?
Answer: Standardizing schedules and timestamps on UTC
Using UTC consistently avoids ambiguity and daylight-saving errors across regions.
What does 'data lineage' tracking in orchestration provide?
Answer: A record of how data flows and transforms across tasks
Lineage maps the origins and transformations of data, aiding debugging and compliance.
Why is a separate staging environment valuable when deploying pipeline changes?
Answer: It lets you validate DAG changes before they affect production data
Staging environments catch errors in workflow changes before they impact production.
What is the main purpose of a 'catchup=False' setting in Airflow?
Answer: To prevent automatic backfilling of past intervals when a DAG starts
Setting catchup=False stops Airflow from running all missed intervals when the DAG is first enabled.
Which practice best supports recovering a pipeline from a mid-run failure?
Answer: Checkpointing progress and making tasks resumable
Checkpointing and resumable tasks let a pipeline continue from where it failed rather than restarting fully.