Data Engineering Data Ingestion Patterns 2 — Questions and Answers
Question 1: A retail company needs to load order events into a warehouse with at most a few minutes of delay, but does not require sub-second latency. Which ingestion pattern best fits?
- Micro-batch ingestion (Correct answer)
- Pure real-time streaming with per-event processing
- Nightly full-table batch load
- Manual CSV export
Correct answer: Micro-batch ingestion
Micro-batch ingestion processes small batches every few minutes, balancing freshness with throughput efficiency.
Question 2: In Change Data Capture (CDC), which approach reads the database transaction log to detect changes?
- Log-based CDC (Correct answer)
- Trigger-based CDC
- Timestamp polling
- Full snapshot diffing
Correct answer: Log-based CDC
Log-based CDC reads the database write-ahead/transaction log, capturing changes with minimal load on the source.
Question 3: What is the primary advantage of a pull-based ingestion model over a push-based one?
- The consumer controls the rate and timing of data retrieval (Correct answer)
- The source guarantees exactly-once delivery automatically
- It eliminates the need for any scheduling
- It always provides lower latency
Correct answer: The consumer controls the rate and timing of data retrieval
In pull-based ingestion the consumer polls the source, controlling retrieval rate and avoiding being overwhelmed.
Question 4: Which scenario most justifies using a streaming ingestion pattern instead of batch?
- Fraud detection that must react within seconds (Correct answer)
- Monthly financial reporting
- One-time historical data migration
- Annual data archival
Correct answer: Fraud detection that must react within seconds
Fraud detection requires immediate reaction, making low-latency streaming ingestion appropriate.
Question 5: When ingesting from a REST API that paginates results, what mechanism prevents missing or duplicating records across pages?
- A stable cursor or continuation token (Correct answer)
- Random page ordering
- Increasing the request timeout
- Disabling retries
Correct answer: A stable cursor or continuation token
A stable cursor/continuation token gives a consistent position so pages neither overlap nor skip records.
Question 6: What does 'idempotent ingestion' guarantee?
- Reprocessing the same data does not create duplicate effects (Correct answer)
- Data is always loaded in alphabetical order
- Ingestion always completes faster on retry
- The source schema can never change
Correct answer: Reprocessing the same data does not create duplicate effects
Idempotent ingestion ensures that processing the same record multiple times yields the same result without duplicates.
Question 7: A pipeline ingests files dropped into a cloud storage bucket and triggers processing on arrival. This is an example of which pattern?
- Event-driven ingestion (Correct answer)
- Scheduled cron batch
- Synchronous request-response
- Two-phase commit
Correct answer: Event-driven ingestion
Reacting to file-arrival events is event-driven ingestion, where processing starts in response to a triggering event.
Data Engineering Data Ingestion Patterns 2
A retail company needs to load order events into a warehouse with at most a few minutes of delay, but does not require sub-second latency.
Which ingestion pattern best fits?