DSE Big Data Technologies 5 — Questions and Answers
Question 1: What does 'exactly-once semantics' mean in a distributed streaming system?
- Each message is produced only once by the source system
- Each message is processed and its effect is reflected exactly once, even in the presence of failures (Correct answer)
- The consumer reads each message before the producer sends the next one
- Messages are deduplicated at the network layer before reaching brokers
Correct answer: Each message is processed and its effect is reflected exactly once, even in the presence of failures
Exactly-once semantics guarantee that despite failures and retries, each message's effect on state or output appears exactly once, requiring idempotency and transactional support.
Question 2: Which of the following is a key advantage of using Apache Iceberg over plain Parquet files on a data lake?
- Iceberg stores data in a binary format unreadable by other tools
- Iceberg provides ACID transactions, hidden partitioning, and schema evolution without full rewrites (Correct answer)
- Iceberg requires a dedicated cluster to function
- Iceberg automatically converts data to columnar format
Correct answer: Iceberg provides ACID transactions, hidden partitioning, and schema evolution without full rewrites
Apache Iceberg adds a transactional metadata layer over Parquet (or ORC) files, enabling ACID operations, time travel, and seamless schema/partition evolution.
Question 3: In HDFS, what is the NameNode's primary function?
- Storing actual data blocks on local disks
- Maintaining the filesystem metadata and directory tree, tracking where blocks are stored (Correct answer)
- Replicating data between DataNodes
- Running MapReduce application master processes
Correct answer: Maintaining the filesystem metadata and directory tree, tracking where blocks are stored
The NameNode is the master server that manages the HDFS namespace, tracking which DataNodes hold which blocks, but it does not store data itself.
Question 4: What is the purpose of Apache Oozie in the Hadoop ecosystem?
- Real-time ingestion of log data into HDFS
- Workflow scheduling and coordination of Hadoop jobs (Correct answer)
- Providing REST APIs for HDFS operations
- Compressing Avro files before storage
Correct answer: Workflow scheduling and coordination of Hadoop jobs
Apache Oozie is a workflow scheduler that chains and coordinates Hadoop jobs (MapReduce, Hive, Pig, Spark) based on time or data availability triggers.
Question 5: Which metric is most important when evaluating the performance of a big data storage system under write-heavy workloads?
- Query response time for SELECT * operations
- Write throughput measured in MB/s or records/second (Correct answer)
- Number of supported SQL dialects
- Replication factor across data centers
Correct answer: Write throughput measured in MB/s or records/second
Write throughput (records or bytes per second) directly measures how quickly the system can ingest data, which is the primary bottleneck in write-heavy workloads.
Question 6: What does 'partition pruning' mean in the context of big data query optimization?
- Removing corrupted partitions from HDFS
- Skipping the scan of partitions that cannot contain data matching the query's filter conditions (Correct answer)
- Splitting large partitions into smaller ones for better parallelism
- Deleting old partitions based on a retention policy
Correct answer: Skipping the scan of partitions that cannot contain data matching the query's filter conditions
Partition pruning allows the query engine to read only the partitions relevant to the filter predicate, dramatically reducing I/O on large partitioned datasets.
Question 7: In a Kappa Architecture, what replaces the batch layer found in Lambda Architecture?
- A dedicated OLAP cube
- Reprocessing historical data through the same stream processing pipeline with a replay mechanism (Correct answer)
- A relational database for serving historical queries
- A separate Spark cluster running nightly batch jobs
Correct answer: Reprocessing historical data through the same stream processing pipeline with a replay mechanism
Kappa Architecture eliminates the separate batch layer by reprocessing historical data by replaying it through the unified stream processing pipeline, simplifying operations.
DSE Big Data Technologies 5
What does 'exactly-once semantics' mean in a distributed streaming system?