MS-DS Master of Data science Big Data Technologies 3 — Questions and Answers
Question 1: In the Lambda architecture for big data, what is the purpose of the 'speed layer'?
- To reprocess all historical data in batch
- To provide low-latency views of recent data that the batch layer has not yet processed (Correct answer)
- To store immutable raw data indefinitely
- To manage schema evolution across data stores
Correct answer: To provide low-latency views of recent data that the batch layer has not yet processed
The speed layer handles real-time data streams to fill the latency gap left by the batch layer's processing delay.
Question 2: Which Apache HBase concept ensures that writes are not lost if a RegionServer crashes before flushing to disk?
- Bloom filters
- Write-Ahead Log (WAL) (Correct answer)
- Compaction
- ZooKeeper quorum
Correct answer: Write-Ahead Log (WAL)
HBase's Write-Ahead Log records every mutation before it is applied to the MemStore, allowing recovery after a crash.
Question 3: When using Spark Structured Streaming, what does 'output mode: complete' mean?
- Only the rows added since the last trigger are written
- Only updated rows are written
- The entire result table is written to the sink on every trigger (Correct answer)
- The stream stops after one batch is complete
Correct answer: The entire result table is written to the sink on every trigger
In complete mode, the full aggregated result table is rewritten to the output sink every time a trigger fires.
Question 4: What is the primary function of Apache ZooKeeper in a distributed big data system?
- Storing large binary objects across nodes
- Providing distributed coordination, leader election, and configuration management (Correct answer)
- Compressing data before writing to HDFS
- Scheduling MapReduce jobs across the cluster
Correct answer: Providing distributed coordination, leader election, and configuration management
ZooKeeper offers a centralized service for maintaining configuration, naming, synchronization, and group services in distributed systems.
Question 5: In Apache Spark, what is 'data skew' and what is a common mitigation technique?
- Data skew is uneven partition sizes; a common fix is salting keys to redistribute load (Correct answer)
- Data skew is schema inconsistency; fixed by schema enforcement
- Data skew is slow disk I/O; fixed by adding SSDs
- Data skew is network latency; fixed by co-locating data and compute
Correct answer: Data skew is uneven partition sizes; a common fix is salting keys to redistribute load
Data skew occurs when some partitions are much larger than others, and salting adds a random prefix to keys to spread the load more evenly.
Question 6: Which technique does Apache Kafka use to guarantee message ordering within a topic?
- Global ordering across all partitions via a sequence number
- Ordering is guaranteed only within a single partition (Correct answer)
- Ordering is guaranteed per consumer group, not per partition
- Kafka uses timestamps to reorder messages globally
Correct answer: Ordering is guaranteed only within a single partition
Kafka guarantees message order only within a partition; to maintain order for a key, all messages with that key must be routed to the same partition.
Question 7: What is the role of the 'NameNode' in HDFS?
- It stores actual data blocks on local disk
- It manages the filesystem namespace and metadata, tracking where blocks are stored across DataNodes (Correct answer)
- It replicates data blocks between DataNodes
- It executes MapReduce task attempts
Correct answer: It manages the filesystem namespace and metadata, tracking where blocks are stored across DataNodes
The NameNode holds all filesystem metadata (file names, directory structure, block locations) but does not store actual data.
MS-DS Master of Data science Big Data Technologies 3
In the Lambda architecture for big data, what is the purpose of the 'speed layer'?