Data Engineering Distributed Data Processing 2 — Questions and Answers
Question 1: In a MapReduce job, what is the primary purpose of the shuffle phase?
- To group and transfer mapper output to reducers by key (Correct answer)
- To split the input file into blocks
- To compress the final output
- To launch the driver program
Correct answer: To group and transfer mapper output to reducers by key
The shuffle phase sorts mapper output and routes records with the same key to the same reducer.
Question 2: What problem does data skew most directly cause in a distributed join?
- A few overloaded tasks slow the whole stage (Correct answer)
- Loss of data replicas
- Network encryption failures
- Schema corruption
Correct answer: A few overloaded tasks slow the whole stage
Skew concentrates many records on one key, making a few tasks far slower than the rest.
Question 3: Which Spark operation triggers actual computation rather than just building the DAG?
- count() (Correct answer)
- filter()
- map()
- select()
Correct answer: count()
count() is an action, while filter, map, and select are lazy transformations.
Question 4: A broadcast join is most appropriate when:
- One table is small enough to fit in each executor's memory (Correct answer)
- Both tables are extremely large
- The join key is null for most rows
- No join key exists
Correct answer: One table is small enough to fit in each executor's memory
Broadcasting a small table to every node avoids shuffling the large table.
Question 5: What does HDFS block replication primarily provide?
- Fault tolerance against node failure (Correct answer)
- Faster CPU scheduling
- Lower storage cost
- Stronger SQL typing
Correct answer: Fault tolerance against node failure
Multiple replicas let the cluster recover data if a DataNode fails.
Question 6: In Spark, what is a 'narrow' transformation?
- Each output partition depends on one input partition (Correct answer)
- Output depends on all partitions
- It always triggers a shuffle
- It writes directly to disk
Correct answer: Each output partition depends on one input partition
Narrow transformations like map require no data movement across partitions.
Question 7: Why is checkpointing useful in long-running streaming jobs?
- It persists state so the job can recover after failure (Correct answer)
- It compresses messages
- It increases parallelism automatically
- It removes the need for offsets
Correct answer: It persists state so the job can recover after failure
Checkpoints save progress and state, enabling recovery without reprocessing everything.
Data Engineering Distributed Data Processing 2
In a MapReduce job, what is the primary purpose of the shuffle phase?