| | Step 1: Initialize the ReLU derivation to create logical neuron |
| | f(z) = max {0,z}: ReLU |
| | Step 2: compute Activation function derivative z = 1 and the neuron operate in the active region z = 0 |
| | Step 3: Compute the Activation value is positive when z = 0.1 to increase input activation is positive |
| | For negative it derivates 0 |
| | When z = 0 it chooses either 1 or 0. |
| | |
| | |
| | Step 4: To Recalls gradient parameters of hidden layers are computed |
| | Step 5: Derivation activate multiplicative factor Trais neurons |
| | Instead of applying f(z) maxout unit divide Z into group of K value. |
| | |
| | G(i) indicates of input for group i, {(i − 1) k + 1… ik} |
| | Training freezes when z < 0 |
| | To overcome this issue, ReLU is proposed |
| | f(z) = max {0,z} + α min {0,z}. |
| | Value α different variant results are given below |
| | α = −1 absolute value is rectification. |
| | α = 0.01 small value is non-linearity called Leaky ReLU |
| | α = left parameter during training |
| | Step 6: The class ŷ ruled to predict the marginal weight fixed condition |
| | ŷ = |
| | The parameters θ weight retains the softmax class at the defined active function |
| | Step 7: To differentiate ReLU based cross-entropy with respect to repose dependencies last link layer form activation rule |
| | ℓ(θ) = −) |
| | H produces the relative output on logic condition with active input X |
| | |
