Orchestrating Data Workflows Flashcards

What is a 'task instance' in Airflow?

Answer: A specific run of a task for a particular execution date

A task instance is one execution of a task tied to a specific DAG run and date.

Why might you set a task's 'trigger_rule' to 'all_done' instead of the default?

Answer: To run regardless of whether upstream tasks succeeded or failed

The 'all_done' trigger rule runs a task once all upstream tasks finish, regardless of their state.

What is the primary risk of a poorly designed long-running monolithic task in a DAG?

Answer: Failures require re-running everything with no granular recovery

Monolithic tasks lack checkpoints, so any failure forces a full re-run.

In orchestration, what is a 'critical path'?

Answer: The longest dependency chain that determines minimum total runtime

The critical path is the longest sequence of dependent tasks, setting the floor on total execution time.

What does setting a task SLA (Service Level Agreement) in Airflow accomplish?

Answer: It alerts when a task exceeds an expected completion time

An SLA triggers a notification when a task runs longer than its defined expected duration.

Which approach improves observability of a data pipeline?

Answer: Emitting metrics, structured logs, and lineage metadata

Metrics, structured logs, and lineage make pipeline behavior transparent and debuggable.

What is the benefit of separating orchestration logic from business/transformation logic?

Answer: Transformations can be tested independently and reused across pipelines

Decoupling lets transformation code be unit-tested and reused without orchestrator dependencies.