MS-DS Master of Data science Big Data Technologies 5 — Questions and Answers
Question 1: In Apache Spark, what is the 'Catalyst optimizer' responsible for?
- Managing memory allocation for RDD caching
- Transforming logical query plans into optimized physical execution plans for DataFrames and Datasets (Correct answer)
- Compressing shuffle data between executors
- Scheduling tasks across YARN worker nodes
Correct answer: Transforming logical query plans into optimized physical execution plans for DataFrames and Datasets
Catalyst is Spark SQL's rule-based and cost-based query optimizer that rewrites logical plans into efficient physical plans.
Question 2: What distinguishes an 'append-only' log in Apache Kafka from a traditional message queue?
- Kafka deletes messages after they are consumed, like a queue
- Kafka retains messages for a configurable retention period, allowing multiple consumers to replay events independently (Correct answer)
- Kafka guarantees strict FIFO order across all topics
- Kafka requires consumers to acknowledge messages before they are removed
Correct answer: Kafka retains messages for a configurable retention period, allowing multiple consumers to replay events independently
Kafka's immutable log persists records beyond consumption, enabling multiple independent consumers and event replay without impacting each other.
Question 3: Which Hadoop ecosystem tool is best suited for running SQL-like queries directly on data stored in HDFS or S3 without moving it to a separate database?
- Apache Sqoop
- Apache Flume
- Apache Hive (Correct answer)
- Apache Oozie
Correct answer: Apache Hive
Apache Hive provides HiveQL, a SQL dialect that translates queries into MapReduce, Tez, or Spark jobs that run directly over data in HDFS or compatible object stores.
Question 4: In the context of big data pipelines, what is 'backpressure' and why does it matter?
- A network protocol that compresses upstream data before delivery
- A flow-control mechanism where a downstream component signals a slower ingestion rate to prevent being overwhelmed (Correct answer)
- A technique to replay missed messages from Kafka to a consumer
- A disk-flush strategy used by HBase MemStore
Correct answer: A flow-control mechanism where a downstream component signals a slower ingestion rate to prevent being overwhelmed
Backpressure propagates downstream capacity constraints upstream, preventing faster producers from overwhelming slower consumers and causing out-of-memory failures.
Question 5: What is the purpose of Delta Lake's transaction log ('_delta_log' directory)?
- It caches frequently accessed Parquet files in memory
- It records every change to the table as a JSON commit entry, enabling ACID transactions and time-travel queries (Correct answer)
- It stores schema definitions separately from data files
- It indexes all columns for sub-second point lookups
Correct answer: It records every change to the table as a JSON commit entry, enabling ACID transactions and time-travel queries
Delta Lake's transaction log is an ordered list of atomic commits that provides ACID properties, optimistic concurrency control, and the ability to query previous table versions.
Question 6: Which partitioning strategy in Apache Kafka ensures that all messages with the same key always go to the same partition?
- Round-robin partitioning
- Random partitioning
- Hash-based key partitioning (Correct answer)
- Sticky partitioning without a key
Correct answer: Hash-based key partitioning
Kafka's default partitioner computes a hash of the message key modulo the number of partitions, routing identical keys to the same partition deterministically.
Question 7: In a distributed system using Apache Spark on a cluster, what is a 'shuffle' operation and why can it be expensive?
- A shuffle randomly samples rows to create a training set; it is expensive due to memory usage
- A shuffle redistributes data across partitions based on a key, requiring network data transfer between all executors (Correct answer)
- A shuffle sorts data within each partition locally; it is expensive due to CPU usage
- A shuffle checkpoints RDDs to disk to recover from executor failure
Correct answer: A shuffle redistributes data across partitions based on a key, requiring network data transfer between all executors
Shuffles involve serializing, transferring, and deserializing data across the network to group records by key, making them the most common performance bottleneck in Spark jobs.
MS-DS Master of Data science Big Data Technologies 5
In Apache Spark, what is the 'Catalyst optimizer' responsible for?