Origins of Neural Networks

  • Single Machine Learning is not useful because it has low expressive power.

  • It must be linearly separable.

    • Classification problem.
  • Method

    • The sign of the dot product between the weight vector (perpendicular to the separation line) and the x vector indicates whether the two vectors are facing the same side.
    • If they are not facing the same side, update the weight vector to x vector + weight vector.
    • Do this for all data.

#getting Started with Machine Learning in Python


  • There are two ways to represent a perceptron.

    • One is like the image above, where the bias is one of the inputs (always with a value of 1), and the weight becomes the bias.
    • The other is a type that holds the bias as an internal value of the perceptron.
    • The former is more commonly used.
  • Even the AND operator can be represented by a perceptron as shown below: image

  • Combining perceptrons like the one above can create XOR, a simple neural network: image #Udacity_Intro_to_Deep_Learning_with_PyTorch


  • Using functions like the sigmoid function, the output must be converted to a continuous value.

  • With the sigmoid function, when the value of x is sufficiently large, the gradient (used for error) becomes almost zero.

    • As a solution, other functions such as ReLU or Tanh can be used.
      • ReLU returns the input value as it is if it is positive, so the gradient becomes 1.
      • Tanh is a bit more complicated, but it returns a better gradient than the sigmoid function. #Deep Learning