Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion 3

Question 1

Which statement BEST describes the COPY INTO SQL command in Databricks?

Accepted Answer

It is an idempotent SQL command that loads files from cloud storage into a Delta table, automatically skipping already-loaded files

Answer

It copies data between two existing Delta tables within the same catalog

Answer

It copies data from a streaming source into a batch Delta table

Answer

It copies schema definitions and constraints between Unity Catalog objects

Question 2

When an Auto Loader stream is started with `trigger(availableNow=True)`, what behavior results?

Accepted Answer

The stream processes all data available at startup across one or more micro-batches, then stops automatically

Answer

The stream runs continuously, processing files as they arrive indefinitely

Answer

The stream processes exactly one micro-batch and then waits for the next scheduled trigger

Answer

The stream ignores the trigger setting and falls back to the default processing interval

Question 3

What does setting `cloudFiles.useNotifications` to true accomplish in Auto Loader?

Accepted Answer

Switches Auto Loader from directory listing mode to cloud file notification mode using services like AWS SNS/SQS

Answer

Configures Auto Loader to send Databricks job alerts when new files arrive

Answer

Enables Databricks notification API integration for downstream alerting

Answer

Causes Auto Loader to notify downstream Delta Live Tables pipelines on file arrival

Question 4

In a production Auto Loader deployment, what information is stored at the `checkpointLocation`?

Accepted Answer

Stream metadata, progress offsets, and the complete record of processed source files

Answer

The ingested data in a compressed intermediate format

Answer

Only the schema evolution history for the stream

Answer

Auto Loader configuration settings and runtime parameters

Question 5

Which of the following is the correct syntax to create an Auto Loader stream reading JSON files from a cloud path?

Accepted Answer

spark.readStream.format('cloudFiles').option('cloudFiles.format', 'json').load(path)

Answer

spark.readStream.format('json').option('cloudFiles', True).load(path)

Answer

spark.readStream.format('autoLoader').option('fileFormat', 'json').load(path)

Answer

spark.readStream.cloudFiles().format('json').load(path)

Question 6

A data engineer needs to ingest millions of small JSON files uploaded daily to S3. Which approach is MOST appropriate?

Accepted Answer

Auto Loader with file notification mode and cloudFiles.maxFilesPerTrigger to control batch size

Answer

spark.read.json() scheduled via a Databricks Job with a full directory scan each run

Answer

COPY INTO run hourly via a SQL task, loading the full directory each time

Answer

A manual Python script using boto3 to move files into Delta format before reading

Question 7

When Auto Loader's `schemaEvolutionMode` is set to `'addNewColumns'` and a new column appears in incoming data, what occurs?

Accepted Answer

The stream stops, updates the persisted schema at schemaLocation to include the new column, then restarts automatically

Answer

The stream throws a schema mismatch exception and halts

Answer

The new column is silently dropped and excluded from the output DataFrame

Answer

The new column's data is captured in the rescuedDataColumn instead of the main schema