Data Ingestion Patterns Flashcards

A retail company needs to load order events into a warehouse with at most a few minutes of delay, but does not require sub-second latency. Which ingestion pattern best fits?

Answer: Micro-batch ingestion

Micro-batch ingestion processes small batches every few minutes, balancing freshness with throughput efficiency.

In Change Data Capture (CDC), which approach reads the database transaction log to detect changes?

Answer: Log-based CDC

Log-based CDC reads the database write-ahead/transaction log, capturing changes with minimal load on the source.

What is the primary advantage of a pull-based ingestion model over a push-based one?

Answer: The consumer controls the rate and timing of data retrieval

In pull-based ingestion the consumer polls the source, controlling retrieval rate and avoiding being overwhelmed.

Which scenario most justifies using a streaming ingestion pattern instead of batch?

Answer: Fraud detection that must react within seconds

Fraud detection requires immediate reaction, making low-latency streaming ingestion appropriate.

When ingesting from a REST API that paginates results, what mechanism prevents missing or duplicating records across pages?

Answer: A stable cursor or continuation token

A stable cursor/continuation token gives a consistent position so pages neither overlap nor skip records.

What does 'idempotent ingestion' guarantee?

Answer: Reprocessing the same data does not create duplicate effects

Idempotent ingestion ensures that processing the same record multiple times yields the same result without duplicates.

A pipeline ingests files dropped into a cloud storage bucket and triggers processing on arrival. This is an example of which pattern?

Answer: Event-driven ingestion

Reacting to file-arrival events is event-driven ingestion, where processing starts in response to a triggering event.