L2: MLP, backpropagation and activation functions
Which loss function should be used in a binary classification task?
(Binary) Cross-entropy
Name 5 activation functions
1) Linear 2) Sigmoid 3) ReLU 4) Hyperbolic tangent 5) Softmax
Give three examples of loss functions
1) Mean squared error (MSE) 2) Mean absolute error 3) Cross-entropy
What defines an MLP's output layer for a binary classification task?
A single neuron with sigmoid activation function.
What is Leaky-ReLU?
A variant of ReLU where for negative inputs, a small slope (<1) is used.
In a regression test, what should the output produce?
All possible values (in principle from -∞ to ∞)
What is a solution to the vanishing gradient?
An activation function with a constant or linear derivative.
At which stage does an MLP optimize its parameters?
Backpropagation
Which loss function should be used in a multiclass classification task?
Categorical cross-entropy
How are the layers in a multilayer perceptron or feed-forward network called?
Dense or Fully connected layers
What is meant with the 'vanishing gradient'?
Due to the chain rule, the gradients are smaller layer after layer.
What is meant with 'dying ReLU'?
During training a ReLU unit can fall into a state where its output will be 0 for any input. It is very difficult to recover from such a sate as the gradient will also be 0.
How is the multilayer perceptron (MLP) also called?
Feed-forward network
What defines the output layer of an MLP that is designed for a classification task of five different classes?
Five neurons with softmax activation function.
What is special about the multilayer perceptron (MLP) or feedforward network?
Formed by more than one layer.
At which stage does an MLP calculate the loss/error?
Forward propagation
To which activation function does this the formula belong? tanh(σ) = (e^20 - 1)/(e^20 + 1)
Hyperbolic tangent
What does the chain rule say?
If L = f(a) and a = g(z) than dL/dz = dL/da × da/dz
When does a linear regression activation function work?
In the output layer of a regression test
Give the formula for the cross-entropy loss function that is used in binary classification.
L = -y log(p) - (1-y) log(1-p)
To which activation function does this the formula belong? σ(z) = z
Linear
Which activation function should be used in a regression test?
Linear
How is the feed-forward network also called?
Multilayer perceptron (MLP)
What is special about the XOR?
No single straight line can separate the two classes.
Can the perceptron solve the XOR?
No, the perceptron can only separate the space linearly.
In a multiclass classification test, what should the output produce?
One-hot encoding indicating the correct class
Give an example of an activation function with a constant or linear derivative.
ReLU
To which activation function does this the formula belong? ReLU(z) = max(z, 0)
ReLU
To which activation function does this the formula belong? σ(z) = 1/(1+e^(-z))
Sigmoid
Which activation function should be used in a binary classification test?
Sigmoid or Tanh
To which activation function does this the formula belong? σ(z_i) = e^(z_i)/(∑_i^N e^(z_j))
Softmax
Which activation function should be used in a multiclass classification task?
Softmax
How do you know how each individual parameter contributes to the error?
Take the derivative of the error (cost function) with respect to each parameter.
What does the loss (error/cost) function calculate?
The "cost" or distance between the network's output and a expected one.
What is dL/dw_ij^l?
The derivative of the error (cost function) with respect to each parameter.
What calculates the "cost" or distance between the network's output and a expected one?
The loss (error/cost) function
What happens at the stage 'forward propagation'?
The loss/error is calculated.
What are the outputs of XOR?
The output is 1 if only x1 or x2 is 1, but not both.
For a MLP with n layers, what is the output of the network (y)?
The output of the last layer
What happens at the stage 'backpropagation'?
The parameters (weights and biases) are updated while minimizing the loss (with gradient descent).
For what is a nonlinear activation function needed?
To learn more complex functions.
In a binary classification test, what should the output produce?
Two different values
For a MLP with n layers, the output of each layer (a^l) is defined as: a^l = f(W^l × a^(l-1) + b^l) What is a^0? What is a^n?
a^0 = x a^n = y
How is the output of each layer (a^l) defined, for a MLP with n layers?
a^l = f(W^l × a^(l-1) + b^l)
Suppose a MLP starts with an x value and a weight (W₁). u₁ = W₁x. There is one more hidden layer that leads to a₁ and this value is used to calculate the loss. Give the formula to calculate how much parameter W₁ contributes to the error.
dloss/dW₁ = dloss/da₁ × da₁/du₁ × du₁/dW₁
In the formula for the cross-entropy loss function that is used in binary classification ( L = -y log(p) - (1-y) log(1-p) ), what is p and what is y?
p is the output of the network and y is the target output.
Give the function of the linear activation function
σ(z) = z
