I made this short video and notebook to explain two classic goto activation functions. I’d highly appreciate constructive feedback!
Video , Notebook , And images (it won’t render images on GitHub)
PS – the “previous video” will be published soon…
EDIT: the code is attached for your reference:
# coding: utf-8 # ### Intro to Deep Learning: Activation Functions # In[1]: import numpy as np #NumPy is the fundamental package for scientific computing in Python. # In[6]: def Sigmoid(x): """ Sigmoid will transform data into a probability between o and 1, based on which a neuron will be activated (or not). """ return 1/ (1+np.exp(-x)) # In[2]: def Sigmoid_derivative(x): """ merely the derivative of the sigmoid, to be used for gradient descent """ return (x*(1-x)) #  # In[3]: def ReLU(x): """ This is the golden standard nowadays, for computational reasons. since computation is within the essence when considering deep learning, let's find the best way to write this function: https://stackoverflow.com/questions/32109319/how-to-implement-the-relu-function-in-numpy """ # return min(o,x) # x * (x > 0) # (abs(x) + x) / 2 # x[x<0]=0 This or the following: return np.maximum(x, 0, x) # In[5]: def ReLU_derivative(x,epslion): #this is not part of the video, left as an exercise for the student. if x <=0: return 1 else: return epslion # In[ ]: #building on the computational hack we've learned, let's use this time the "fancy slicing": def better_relu_de(x, epsilon) x[x<=0] = epsilon x[x>0] = 1 return x #  #  # ### Resources: # # Deriving sigmoid: https://beckernick.github.io/sigmoid-derivative-neural-network/ #