Data Ingestion Patterns Flashcards

Which schema-handling strategy lets a pipeline continue ingesting when new optional columns appear at the source?

Answer: Schema evolution with backward compatibility

Backward-compatible schema evolution allows new optional fields without breaking existing consumers.

In a Lambda architecture, what is the role of the speed layer?

Answer: Provide low-latency views of recent data before the batch layer catches up

The speed layer delivers real-time, low-latency results for recent data while the batch layer recomputes accurate views.

What distinguishes Kappa architecture from Lambda architecture?

Answer: Kappa uses a single streaming pipeline for both real-time and reprocessing

Kappa architecture treats everything as a stream, using one pipeline and replaying the log for reprocessing.

Why is partitioning ingested data by event time often preferred over ingestion time for analytics?

Answer: It keeps late-arriving events grouped with the period they actually belong to

Event-time partitioning places records in the time bucket they occurred, even if they arrive late, improving analytical correctness.

A watermark in stream processing primarily helps the system to:

Answer: Decide when a time window can be considered complete despite late data

Watermarks track event-time progress, letting the engine close windows and handle late-arriving data sensibly.

Which approach reduces small-file problems when continuously ingesting into a data lake?

Answer: Periodic file compaction into larger files

Compaction merges many small files into fewer larger ones, improving read performance and metadata efficiency.

When ingesting from multiple time zones, the recommended practice is to:

Answer: Normalize timestamps to UTC at ingestion

Normalizing to UTC at ingestion provides a consistent, unambiguous time reference across sources.