Data Engineering Real-Time Streaming Architectures Questions and Answers

Question 1

A financial services company wants to monitor credit card transactions for fraudulent activity.
The system must group all transactions by a specific user that occur closely together, but the time between these bursts of activity is unpredictable.
A new group should start only after a significant period of user inactivity.
Which windowing strategy is most suitable for this scenario?

Accepted Answer

Session Windows

Answer

Tumbling Windows

Answer

Sliding Windows

Answer

Global Windows

Question 2

In a real-time streaming architecture that processes data using event time, what is the primary function of a watermark?

Accepted Answer

To signal to the processing engine a point in time beyond which it is unlikely to receive older events, allowing it to close windows and trigger computations.

Answer

To filter out duplicate events before they enter a time window.

Answer

To apply backpressure to the data source when the processing rate is slower than the ingestion rate.

Answer

To guarantee exactly-once processing semantics by creating transactional micro-batches.

Question 3

A data engineering team is building a real-time dashboard to display the number of unique visitors on their website over the last 15 minutes. Which type of stream processing is fundamentally required to implement this feature?

Accepted Answer

Stateful processing

Answer

Stateless transformation

Answer

Idempotent filtering

Answer

Static data enrichment

Question 4

Which of the following is a defining characteristic of the Kappa Architecture compared to the Lambda Architecture?

Accepted Answer

It simplifies the architecture by using a single stream-processing engine to handle both real-time queries and historical data reprocessing from an append-only log.

Answer

It mandates the use of a separate batch processing layer for historical accuracy and a speed layer for real-time analysis.

Answer

It relies exclusively on batch processing, refreshing views periodically to simulate real-time results.

Answer

It is designed primarily for structured data and requires a relational database as its serving layer.

Question 5

In a streaming data pipeline, a processing stage is consistently unable to keep up with the rate of data it receives from an upstream stage. This leads to growing memory usage, increased latency, and potential system instability. What is the term for the mechanism designed to mitigate this specific problem?

Accepted Answer

Backpressure

Answer

Watermarking

Answer

Fault tolerance

Answer

Windowing

Question 6

An IoT application collects sensor readings from thousands of devices. Each reading is tagged with the precise timestamp when the measurement was taken. Due to variable network conditions, these readings arrive at the central processing system out of order and with significant delays. For accurate time-series analysis (e.g., calculating hourly averages), which time characteristic must the system rely on?

Accepted Answer

Event Time

Answer

Processing Time

Answer

Ingestion Time

Answer

System Wall-Clock Time