MS-DS Master of Data science Big Data 5 — Questions and Answers
Question 1: Which strategy reduces shuffle data volume in Spark by replacing groupByKey with a more efficient alternative?
- Using repartition() before groupByKey()
- Replacing groupByKey() with reduceByKey() (Correct answer)
- Increasing executor memory
- Using broadcast joins
Correct answer: Replacing groupByKey() with reduceByKey()
reduceByKey() applies a combining function locally on each partition before shuffling, dramatically reducing the data transferred across the network.
Question 2: In data lakehouse architecture, what role does a metadata layer like Apache Iceberg or Delta Lake serve?
- It replaces object storage with block storage
- It adds ACID transactions, schema evolution, and time travel to data lake files (Correct answer)
- It compresses Parquet files automatically
- It manages cluster autoscaling
Correct answer: It adds ACID transactions, schema evolution, and time travel to data lake files
Table formats like Iceberg and Delta Lake maintain transactional metadata over raw files, enabling ACID guarantees, schema evolution, and historical snapshots.
Question 3: Which metric best quantifies the throughput of a big data streaming system?
- P99 latency
- Events processed per second (Correct answer)
- Number of brokers
- Replication lag in bytes
Correct answer: Events processed per second
Events (or messages) processed per second directly measures how much data the system handles over time, representing throughput.
Question 4: Apache Beam's unified programming model is designed to allow pipelines to run on multiple execution engines. What is the abstraction called that wraps these engines?
- Connector
- Runner (Correct answer)
- Executor
- Driver
Correct answer: Runner
A Beam Runner (e.g., Dataflow Runner, Flink Runner) translates the portable Beam pipeline into the target engine's native execution model.
Question 5: In a distributed join, which optimization avoids a shuffle by sending a small table to all worker nodes?
- Sort-merge join
- Hash join
- Broadcast join (Correct answer)
- Nested loop join
Correct answer: Broadcast join
A broadcast join replicates the smaller table to every executor so the larger table can be joined locally without shuffling either dataset.
Question 6: Which characteristic of big data refers to the uncertainty and unreliability of data sources?
- Volume
- Velocity
- Variety
- Veracity (Correct answer)
Correct answer: Veracity
Veracity addresses data quality concerns such as noise, bias, and trustworthiness inherent in large heterogeneous data sources.
Question 7: A data engineer needs to process a 10 TB dataset nightly with a 4-hour SLA. Which processing paradigm is most appropriate?
- Real-time stream processing with microsecond latency
- Micro-batch processing with 1-second windows
- Scheduled batch processing (Correct answer)
- Online transaction processing (OLTP)
Correct answer: Scheduled batch processing
A nightly batch job with a 4-hour window fits scheduled batch processing, which optimizes throughput over large datasets without requiring streaming infrastructure.
MS-DS Master of Data science Big Data 5
Which strategy reduces shuffle data volume in Spark by replacing groupByKey with a more efficient alternative?