Artificial Neural Networks

What are the 8 components of ANNs?

1. Neurons 2. Activation function 3. Threshold 4. Output function 5. Connections and weights 6. Propagation function 7. Bias 8. Layers 9. Hyperparameters 10. Learning rate 11. Cost function (Also known as loss function) 12. Backpropagation 13. Gradient Descent

How to solve overfitting?

1. Reduce the network's capacity by removing layers or reducing the number of elements in the hidden layers 2. Apply regularization, which comes down to adding a cost to the loss function for large weights 3. Use Dropout layers, which will randomly remove certain features by setting them to zero.

What are hyperparameters in ANNs? What are examples of them? Are values of hyperparameters dependent on anything else?

A hyperparameter is a constant parameter whose value is set before the training process starts. Examples of hyperparameters include learning rate, the number of hidden layers and batch size. The values of some hyperparameters can be dependent on those of other hyperparameters. For example, the size of some layers can depend on the overall number of layers.

How does a neuron work?

ANNs are composed of artificial neurons which retain the biological concept of neurons, which receive input, combine the input with their internal state (activation) and an optional threshold using an activation function, and produce output using an output function.

What is an activation function in an ANN? What are 3 examples and what do they do? What's the difference between the two classic activation functions that range from 0 to 1? (These two are part of the answer to the previous question) What are the formulas for the two classic activation functions?

An activation function of a neuron defines the output of that neuron given an input or set of inputs. The important characteristic of the activation function is that it provides a smooth, differentiable transition as input values change, i.e. a small change in input produces a small change in output. Examples: Relu, which maps all numbers to positive numbers Sigmoid (also known as logistic and soft step), which maps values between 0 and 1 Softmax, which maps values between 0 and 1 Difference: Softmax distributes the probability throughout each output node. In binary classification, sigmoid is same as softmax. In multi-class classification, we use softmax for activation with cross-entropy. Formulas: Relu: f(x) = 0 for x<=0 and x for x>0 Sigmoid: f(x) = 1/[1-e^(-x)]

What is backpropagation in ANNs? How does backpropagation work in ANNs?

Backpropagation is a method to adjust the connection weights to compensate for each error found during training. The goal is to effectively divide the error rate among the network's connections. Technically, backprop calculates the gradient or the derivative of the cost function associated with a given state with respect to the weights. The weight updates can be done via stochastic gradient descent or other methods.

What is an artificial neural network (ANN)?

Computing systems inspired by biological neural networks. The network is a directed, weighted graph.

What are connections and weights in ANNs?

Connections in the network pass the output of one neuron as an input to the next neuron. Each connection is assigned a weight that represents its relative importance. A single neuron can have multiple input and output connections.

Why would your model not converge?

Data is not normalized Imbalanced data across classes

How would you speed up the training time of ANNs?

GPUs Reduce data dimensionality Decrease number of layers Decrease learning rate Decrease number of epochs Parallel neural network training Pruning

What are 2 methods you can use to tune hyperparameters?

Grid Search -It works by searching exhaustively through a specified subset of hyperparameters. Random Search - Random search differs from grid search mainly in that it searches the specified subset of hyperparameters randomly instead of exhaustively. The major benefit being decreased processing time.

What is a threshold in ANNs?

Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold.

What is a cost function in ANNs? (Also known as the loss function) How is it used in ANNs?

The cost function maps an event's output values to a number representing a "cost" associated with the event. The goal of an ANN is to train a model and find weights (also known as parameters) that minimize the cost function.

What is a learning rate in ANNs? How does a high learning rate differ from a low learning rate?

The learning rate defines the size of the corrective steps that the model takes to adjust for errors in each observation. A high learning rate decreases training time and may also decrease accuracy. A lower learning rate increases training time and may increase accuracy.

What are layers in ANNs?

The neurons are typically organized into multiple layers, especially in deep learning. Neurons of one layer connect only to neurons of the immediately preceding and immediately following layers. The layer that receives external data is the input layer. The layer that produces the ultimate result is the output layer. In between them are zero or more hidden layers. Single layer and unlayered networks are also used. Between two layers, multiple connection patterns are possible. They can be fully connected, with every neuron in one layer connecting to every neuron in the next layer. They can be pooling, where a group of neurons in one layer connect to a single neuron in the next layer, thereby reducing the number of neurons in that layer.[44] Neurons with only such connections form a directed acyclic graph and are known as feedforward networks.[45] Alternatively, networks that allow connections between neurons in the same or previous layers are known as recurrent networks.[46]

What is a propagation function in ANNs?

The propagation function computes the input to the current neuron from the outputs of previous neurons and their connections as a weighted sum. An optional bias term is added to the output of propagation.

What are parameters in ANNs?

The values of parameters are derived via learning.