Perceptron Learning Rule

Definition

The perceptron learning rule updates the weights and bias when the perceptron makes a mistake.

Basic idea:

if prediction is correct:
  do nothing
if prediction is wrong:
  adjust weights and bias

Perceptron learning changes the decision boundary after errors.

---

Setup

Use labels:

-1
1

For one training example:

input = x
label = y

where:

y is either -1 or 1

The perceptron predicts:

prediction = sign(dot(w, x) + b)

The model compares prediction with label.

---

Correct Prediction

If the prediction matches the label, no update is needed.

Example:

label = 1
prediction = 1

The current weights handled this example correctly.

So:

weights stay the same
bias stays the same

Correct examples do not force an update.

---

Incorrect Prediction

If the prediction differs from the label, the model updates.

Example:

label = 1
prediction = -1

The score was too low for this example.

The weights should move toward the input.

Another case:

label = -1
prediction = 1

The score was too high.

The weights should move away from the input.

The update moves the boundary in a direction that helps the mistaken example.

---

Update Rule

One common form:

w = w + learning_rate * y * x
b = b + learning_rate * y

when the example is misclassified.

Where:

w = weight vector
x = input vector
y = label (-1 or 1)
b = bias

The label decides the update direction.

---

Example Update

Suppose:

w = [0, 0]
b = 0
x = [2, 3]
y = 1
learning_rate = 1

If the model misclassifies this example, update:

w = [0, 0] + 1 * 1 * [2, 3]
w = [2, 3]

Bias:

b = 0 + 1 * 1
b = 1

The model shifts toward classifying this example as positive.

---

What The Update Does Geometrically

The weight vector controls the decision boundary.

Updating weights changes the boundary's orientation.

Updating bias shifts the boundary.

So each mistake changes the boundary slightly.

Perceptron learning adjusts the boundary from errors.

---

When It Converges

If the data is linearly separable, the perceptron learning rule can eventually find a separating boundary.

If the data is not linearly separable, it may keep making updates.

This is a limitation of linear classification.

Perceptron learning works cleanly when a linear separator exists.

---

Final Summary

• The perceptron updates only after mistakes.
• Correct predictions do not require an update.
• With labels -1 and 1, the label controls update direction.
• The weight update is w = w + learning_rate * y * x.
• The bias update is b = b + learning_rate * y.
• Weight updates rotate the boundary.
• Bias updates shift the boundary.
• Convergence depends on linear separability.