Building a Perceptron from Scratch in Python

The Perceptron: The Foundation of Neural Networks

A perceptron is the simplest possible artificial neuron, serving as the fundamental building block for all modern neural networks. At its core, a perceptron takes an input, applies a weight and a bias, and produces a binary (yes/no) output based on whether the result exceeds a specific threshold.

How a Perceptron Makes Decisions

A perceptron functions as a linear classifier. It calculates a weighted sum of its inputs and adds a bias to determine the output. The mathematical representation of this decision is:

output = 1 if (weight * input + bias) > 0 else 0

The Role of Weights

Weights represent the importance of a specific input. In a real-world scenario, if a person is deciding whether to take a job offer, they might weigh "extra pay" more heavily than "commute distance." In a perceptron, the weight multiplies the input value, amplifying or diminishing its impact on the final decision.

The Role of Bias

Bias allows the decision boundary to shift away from the origin (zero). Without a bias, the decision boundary is locked at zero, meaning the machine can only classify data that is naturally split at the zero point.

For example, if a machine is tasked with determining if a student passed an exam (where the passing score is 50), a perceptron without bias would fail because it cannot move its boundary to 50; it would either mark everyone as passing or everyone as failing. The bias moves the boundary left or right, allowing the perceptron to fit the data regardless of where the threshold sits.

The Learning Process: Training the Perceptron

A perceptron learns by iteratively adjusting its weight and bias based on the errors it makes during training. This process involves three key components:

1. The Error Calculation

When the perceptron makes a prediction that differs from the actual result, an error is calculated: error = result - prediction

2. The Weight and Bias Update

The machine "nudges" the weight and bias in the direction that reduces the error:

  • weight += learning_rate * error * input
  • bias += learning_rate * error

3. Epochs and Learning Rate

  • Epochs: One epoch is a single full pass over the entire training dataset. Multiple epochs are required because the machine rarely finds the perfect boundary in one pass.
  • Learning Rate: This is a scalar that determines the size of the adjustment made during each update. A learning rate that is too high can cause the model to overshoot the optimal boundary, while one that is too low makes training excessively slow.

Data Normalization

Normalization is the process of scaling input data to a small, consistent range (e.g., 0 to 1). This is critical for two reasons:

  1. Stability: Large input values can cause massive swings in weight updates, leading to unstable training where the accuracy "jumps around."
  2. Feature Parity: When dealing with multiple inputs of different scales (e.g., salary in thousands vs. a binary 0/1 for location), normalization ensures that the larger numbers do not drown out the smaller, more significant features.

Implementation in Python

Below is a complete implementation of a single-input perceptron designed to classify whether a number is positive.

import random

learning_rate = 0.1
EPOCHS = 100

weight = random.uniform(-1, 1)
bias   = random.uniform(-1, 1)

# positive numbers are True, negative numbers are False
data  = [(i * 0.1, True)  for i in range(1, 501)]
data += [(i * 0.1, False) for i in range(-500, 0)]
random.shuffle(data)

for epoch in range(EPOCHS):
    for value, result in data:
        prediction = (weight * value + bias) > 0
        if prediction != result:
            error = result - prediction          # +1 or -1
            weight += learning_rate * error * value
            bias   += learning_rate * error

decision_boundary = -bias / weight
print(f"weight = {weight:.3f}")
print(f"bias   = {bias:.3f}")
print(f"decision boundary = {decision_boundary:.3f}")

Community Insights and Extensions

While the single-input perceptron is a powerful pedagogical tool, community discussions highlight several ways to expand these concepts:

  • Hardware Implementation: Historically, early machine learning was implemented in hardware. Some users noted that early networks like ADALINE were built physically, and today such a "small brain" could be constructed using op-amps on a breadboard.
  • Multi-Input Perceptrons: A single-input perceptron is limited to a 1D line. Real-world applications use multi-input perceptrons where the weighted sum is calculated across multiple features: weighted_sum = bias + sum(input[i] * weight[i])
  • Scaling to Neural Networks: A single perceptron can only solve linearly separable problems (it can only draw one straight line). The transition to deep learning occurs when these neurons are stacked in layers, allowing the network to learn complex, non-linear shapes and patterns.

Sources