Neural Network Processor

• Domain Specific Architecture

• Considering a friendly Deep Learning for Logic Circuits

  • What does it mean to be logic circuit friendly?
    • Instead of multiplication, use addition, etc.
    • Would it be even better with logical operations?

• BRein

  • Point 1: Quantize the weights $w$ of the Perceptron to only +1 or -1 using binarization
    • Furthermore, convert them to 1 or 0 instead of +1 or -1
    • This allows weight calculations using XNOR logic operations
      • This enables NN to be implemented with smaller memory and computational circuits
  • Point 2: Perform computations in memory
    • The structure of the Neural Network is directly mapped to the circuit structure of memory
    • When information of a single-bit neuron arrives, it is parallelly forwarded to the next layer (with different weights) and then collected in a single place
      • Can this be pipelined and performed collectively?

• QUEST

  • Differs from BRein in the quantization part, using quantization instead of binarization
    • The values of $w$ are often close to 0
    • Therefore, it quantizes the parts closer to 0 more finely
  • By performing calculations in logarithmic domain, multiplication can be converted to addition
    • Addition in the “logarithmic domain” is equivalent to multiplication
    • Addition requires smaller circuits compared to multiplication, making it logic circuit friendly
  • When investigating the relationship between the level of quantization (log3, log4, log5, etc.) and recognition accuracy,
    • There is a sharp drop in recognition accuracy for MNIST dataset at some point

• Quantization naturally introduces errors

  • By incorporating these quantization errors into the evaluation function, both recognition and quantization errors can be minimized during training
  • Weights naturally gather around points with lower quantization errors

• Dithering of weights

  • Applying the concept of Dithering in images to the quantization of neural networks
  • This can improve recognition accuracy
  • This allows for increasing recognition accuracy at low cost, as the circuit size remains almost the same

• Dynamic Invalid Neuron Prediction (DNP)

  • ReLU and similar functions set negative values to 0, making 0 remain as it is (= dead neuron)
  • If we can predict these invalid neurons, we can skip the calculations for those parts
    • We don’t want to perform the calculation of $x\times 0=0$
  • How to do it
    • Predict the dying neurons with a lighter prediction mechanism
    • Then create a neural network without those neurons and efficiently perform learning
  • Results
    • Tested with VGG
    • In some cases, there is a decrease of about 5% in accuracy, but computational resources are halved
    • Trade-off between speed and accuracy, tuning is required (Information Science Master)