MS-DS Master of Data science Big Data Technologies 4 — Questions and Answers
Question 1: Which Apache Spark optimization technique avoids shuffling by co-partitioning two datasets on the same key?
- Broadcast join
- Sort-merge join with bucketing (Correct answer)
- Cartesian join
- Hash join without partitioning
Correct answer: Sort-merge join with bucketing
Bucketing pre-partitions data on a join key so that matching partitions are co-located, eliminating the shuffle step during joins.
Question 2: In Google's original MapReduce paper, what is the purpose of the 'combiner' function?
- To merge outputs from multiple reducers into one file
- To perform partial aggregation on the mapper's output before it is sent to the reducer, reducing network traffic (Correct answer)
- To combine multiple input splits into a single mapper task
- To validate schema consistency before the reduce phase
Correct answer: To perform partial aggregation on the mapper's output before it is sent to the reducer, reducing network traffic
The combiner acts as a mini-reducer on each mapper node, aggregating intermediate key-value pairs locally to minimize data shuffled over the network.
Question 3: What is Apache Iceberg's primary advantage over a traditional Hive partitioned table?
- Iceberg tables can only be read by Spark, not Hive
- Iceberg provides ACID transactions, hidden partitioning, and time-travel on large analytic tables without full table scans on metadata (Correct answer)
- Iceberg stores data in row format for faster writes
- Iceberg eliminates the need for any file format like Parquet
Correct answer: Iceberg provides ACID transactions, hidden partitioning, and time-travel on large analytic tables without full table scans on metadata
Iceberg adds a metadata layer that enables ACID guarantees, partition evolution, and time-travel queries without modifying the underlying Parquet/ORC files.
Question 4: Which consistency model does Apache Cassandra use when the consistency level is set to 'QUORUM'?
- All replicas must acknowledge before the operation succeeds
- A majority (more than half) of replicas in the replication factor must acknowledge (Correct answer)
- Only one replica needs to acknowledge
- Only the coordinator node acknowledges the write
Correct answer: A majority (more than half) of replicas in the replication factor must acknowledge
QUORUM requires acknowledgment from a majority of replicas, balancing consistency and availability.
Question 5: In stream processing, what does 'exactly-once semantics' guarantee?
- Each record is processed at least once but may be duplicated
- Each record is processed at most once, tolerating some data loss
- Each record is processed exactly once, with no duplicates and no data loss (Correct answer)
- Records are processed in strict arrival order with no retries
Correct answer: Each record is processed exactly once, with no duplicates and no data loss
Exactly-once semantics ensures every record affects the output exactly one time, combining idempotent writes with transactional commits or distributed snapshots.
Question 6: What problem does Apache Kafka's 'log compaction' feature solve?
- It compresses messages to reduce storage by using zstd encoding
- It retains only the most recent value for each key, enabling changelog-style topics to be replayed to current state (Correct answer)
- It splits large log segments into smaller files to speed up reads
- It removes duplicate messages from producer retries
Correct answer: It retains only the most recent value for each key, enabling changelog-style topics to be replayed to current state
Log compaction ensures that for each unique key, the latest value is always retained, making the topic suitable for state reconstruction.
Question 7: Which of the following best describes the concept of 'schema-on-read' used in data lakes?
- Schema is enforced when data is written, rejecting non-conforming records
- Data is stored raw without enforced schema, and structure is applied at query time (Correct answer)
- Data must be pre-processed through ETL before landing in the lake
- Schema is validated by a central catalog before any read occurs
Correct answer: Data is stored raw without enforced schema, and structure is applied at query time
Schema-on-read allows raw data to be ingested without upfront transformation, deferring schema enforcement to when the data is queried.
MS-DS Master of Data science Big Data Technologies 4
Which Apache Spark optimization technique avoids shuffling by co-partitioning two datasets on the same key?