Orchestrating Data Workflows Flashcards

A data engineering team manages a complex data platform. Job B must run only after Job A completes, and Job C must run after Job B. If Job B fails, it should be retried three times before an alert is sent. Which tool is best suited for managing this entire process?

Answer: A workflow orchestration engine like Apache Airflow

Workflow orchestration engines are specifically designed to manage complex dependencies between tasks, handle automatic retries on failure, and provide monitoring and alerting. Cron jobs lack native dependency management and retry logic, while Flink and Trino serve different purposes (stream processing and ad-hoc querying, respectively).

In workflow orchestration frameworks, what is the primary role of a Directed Acyclic Graph (DAG)?

Answer: To define the tasks and their dependencies, showing how they are connected and in what order they should run

A Directed Acyclic Graph (DAG) is a core concept in orchestration that defines the workflow structure. 'Directed' means the edges have a direction, 'Acyclic' means there are no loops, and 'Graph' refers to the nodes (tasks) and edges (dependencies). It serves as the blueprint for how tasks are ordered and related.

Which of the following features is a primary advantage of using a dedicated workflow orchestrator over a series of cron jobs for managing data pipelines?

Answer: Native support for complex dependency management and failure handling with retries

While cron is excellent for simple time-based scheduling, it lacks built-in mechanisms for managing dependencies (e.g., 'run task B only if task A succeeds'), handling retries, backfilling, and providing centralized logging and monitoring, which are core features of orchestrators.

A daily data aggregation pipeline failed to run on May 15th due to a temporary network outage. The data for that specific day is now missing from the summary tables. The data engineering team needs to run the pipeline only for May 15th's data. What is this common orchestration practice called?

Answer: Backfilling

Backfilling is the process of running a pipeline for a specific historical period to process or re-process data that was missed or needs to be corrected. This is a common and critical feature of workflow orchestration systems.

When designing a task for a data workflow, the principle of idempotency is crucial for ensuring reliability. What does it mean for a task to be idempotent?

Answer: The task can be run multiple times with the same input and will produce the same result without causing unintended side effects.

Idempotency ensures that repeatedly executing an operation yields the same result as the first execution. This is vital for data pipelines because tasks often need to be retried after failures. An idempotent task can be safely retried without duplicating data or corrupting the final state.

A data pipeline must be triggered to process new customer reviews immediately after they are submitted and stored as a JSON file in a cloud storage bucket. Which orchestration scheduling pattern is most suitable for this use case?

Answer: Event-driven scheduling based on a file arrival sensor.

Event-driven scheduling allows workflows to be triggered by external events, such as a file appearing in a storage bucket. This is far more efficient and timely for near real-time requirements than polling on a fixed time-based schedule.