AI AI Engineer: Neural Networks and Deep Learning 1 — Questions and Answers
Question 1: What is the vanishing gradient problem in deep neural networks?
- Gradients become too large during backpropagation
- Gradients become extremely small, making early layers learn very slowly (Correct answer)
- The loss function fails to converge
- Weights are initialized to zero
Correct answer: Gradients become extremely small, making early layers learn very slowly
Vanishing gradients occur when backpropagated gradients shrink exponentially through layers, causing weights in early layers to update negligibly.
Question 2: Which activation function is most commonly used in hidden layers of modern deep neural networks to mitigate the vanishing gradient problem?
- Sigmoid
- Tanh
- ReLU (Rectified Linear Unit) (Correct answer)
- Softmax
Correct answer: ReLU (Rectified Linear Unit)
ReLU outputs max(0, x), maintaining gradient magnitude for positive inputs and largely avoiding vanishing gradients compared to sigmoid or tanh.
Question 3: What is the purpose of batch normalization in a neural network?
- To split data into mini-batches
- To normalize activations within a layer to stabilize and accelerate training (Correct answer)
- To regularize weights using L2 penalty
- To reduce the number of parameters
Correct answer: To normalize activations within a layer to stabilize and accelerate training
Batch normalization normalizes layer inputs to zero mean and unit variance, reducing internal covariate shift and enabling higher learning rates.
Question 4: In a convolutional neural network (CNN), what does 'stride' control?
- The depth of the network
- How many pixels the filter moves at each step (Correct answer)
- The number of filters applied
- The size of the pooling window
Correct answer: How many pixels the filter moves at each step
Stride determines how many pixels the convolutional filter shifts at each step, controlling the spatial dimensions of the output feature map.
Question 5: What is dropout regularization in neural networks?
- Removing layers with poor performance
- Randomly setting a fraction of neurons to zero during training to prevent overfitting (Correct answer)
- Reducing learning rate during training
- Pruning weights below a threshold
Correct answer: Randomly setting a fraction of neurons to zero during training to prevent overfitting
Dropout randomly deactivates neurons during each training step, forcing the network to learn redundant representations and reducing overfitting.
Question 6: Which optimizer adapts the learning rate for each parameter individually using estimates of first and second moments of gradients?
- SGD (Stochastic Gradient Descent)
- Momentum
- Adam (Correct answer)
- Adagrad
Correct answer: Adam
Adam combines adaptive learning rates with momentum, maintaining per-parameter moving averages of gradients and their squares.
AI AI Engineer: Neural Networks and Deep Learning 1
What is the vanishing gradient problem in deep neural networks?