ETL and ELT Pipelines Flashcards

Which orchestration tool models pipelines as Directed Acyclic Graphs (DAGs) of tasks?

Answer: Apache Airflow

Airflow defines workflows as DAGs where each node is a task with explicit dependencies.

A pipeline must process events within seconds of arrival. Which paradigm fits best?

Answer: Streaming (real-time) processing

Streaming processes records continuously as they arrive, achieving low end-to-end latency.

In dimensional modeling, a Type 2 Slowly Changing Dimension handles updates by what method?

Answer: Adding a new row to preserve historical versions

SCD Type 2 inserts a new row with validity dates, keeping the full history of the attribute.

What is the main risk of using SELECT * during extraction from a wide source table?

Answer: Pulling unnecessary columns and breaking when schema changes

SELECT * transfers all columns wastefully and makes the pipeline fragile to upstream schema changes.

Which storage layer typically holds raw, unprocessed data in a modern data lakehouse pattern?

Answer: The bronze (raw) layer

The medallion architecture's bronze layer stores raw ingested data before refinement.

A dead-letter queue in a pipeline is used to do what?

Answer: Capture records that fail processing for later inspection

A dead-letter queue isolates malformed or failed messages so the main flow continues uninterrupted.

Which practice helps ensure data quality is verified within the pipeline itself?

Answer: Automated data validation tests on row counts and null thresholds

Embedding automated assertions (row counts, nulls, ranges) catches quality issues before data reaches users.