MS-DS Master of Data science Master of Data science Big Data 1 — Questions and Answers
Question 1: Which file format is optimized for columnar storage and is commonly used in big data ecosystems for efficient query performance?
- CSV
- JSON
- Parquet (Correct answer)
- XML
Correct answer: Parquet
Parquet is a columnar storage format designed for big data processing. It stores data by column rather than row, enabling efficient compression and query performance when only specific columns need to be read.
Question 2: In the Lambda architecture for big data processing, which layer handles real-time data processing with low latency?
- Batch layer
- Serving layer
- Speed layer (Correct answer)
- Storage layer
Correct answer: Speed layer
The speed layer (also called the streaming layer) in Lambda architecture processes incoming data in real time with low latency to provide up-to-date results, compensating for the high latency of the batch layer.
Question 3: Which algorithm is commonly used for distributed machine learning to split gradient computation across multiple nodes?
- K-Means clustering
- Stochastic Gradient Descent
- Distributed Gradient Boosting (Correct answer)
- Principal Component Analysis
Correct answer: Distributed Gradient Boosting
Distributed Gradient Boosting (as implemented in frameworks like XGBoost and LightGBM) splits the gradient computation and histogram building across multiple nodes, enabling scalable ensemble learning on large datasets.
Question 4: What is the primary role of Apache ZooKeeper in a big data cluster?
- Data serialization
- Distributed coordination and configuration management (Correct answer)
- In-memory caching
- ETL pipeline orchestration
Correct answer: Distributed coordination and configuration management
Apache ZooKeeper provides distributed coordination services such as configuration management, naming, synchronization, and group membership, allowing distributed applications like Hadoop and Kafka to maintain consistency across nodes.
Question 5: Which concept in big data describes the phenomenon where storing and processing costs decrease as the volume of data increases per unit?
- Data gravity
- Economies of scale (Correct answer)
- Data sparsity
- Horizontal sharding
Correct answer: Economies of scale
Economies of scale in big data refers to the cost efficiency gained as data volume grows — distributed systems like Hadoop and cloud storage reduce the per-unit cost of storage and computation at higher volumes.
Question 6: In Apache Spark, which operation triggers the actual execution of a transformation chain on an RDD or DataFrame?
- map()
- filter()
- collect() (Correct answer)
- flatMap()
Correct answer: collect()
Spark uses lazy evaluation, meaning transformations like map() and filter() are not executed immediately. An action such as collect() triggers the actual computation by submitting a job to the Spark cluster.
MS-DS Master of Data science Master of Data science Big Data 1
Which file format is optimized for columnar storage and is commonly used in big data ecosystems for efficient query performance?