How to Implement a Basic Neural Network from Scratch Using Python
Learn to build a basic neural network in Python without high-level libraries.
Neural networks are the backbone of deep learning and have revolutionized fields such as computer vision, natural language processing, and more. While powerful libraries like TensorFlow and PyTorch make it easy to build complex models, understanding how to implement a basic neural network from scratch can provide valuable insights into how these models work under the hood.
In this tutorial, we will walk through the steps to create a simple feedforward neural network using Python, without relying on any deep learning libraries. We'll implement the forward pass, backpropagation, and training loop manually.
Overview of the Neural Network
A basic neural network consists of layers of neurons that are connected by weights. The network we will build will have:
Input Layer: Takes the input features.
Hidden Layer: A single hidden layer with a configurable number of neurons.
Output Layer: Provides the final output predictions.
We will use the sigmoid activation function for the hidden layer and a binary cross-entropy loss function for training on a binary classification problem.
Step 1: Import Libraries
We'll start by importing the necessary libraries. For this implementation, we only need NumPy.
import numpy as np
Step 2: Initialize the Neural Network
Let's define a simple neural network class with an initializer to set up weights and biases.
class SimpleNeuralNetwork:
def __init__(self, input_size, hidden_size, output_size):
# Initialize weights with random values
self.weights_input_hidden = np.random.randn(input_size, hidden_size)
self.weights_hidden_output = np.random.randn(hidden_size, output_size)
# Initialize biases with zeros
self.bias_hidden = np.zeros((1, hidden_size))
self.bias_output = np.zeros((1, output_size))
Here, input_size
is the number of input features, hidden_size
is the number of neurons in the hidden layer, and output_size
is the number of output classes (1 for binary classification).
Step 3: Define Activation and Loss Functions
We'll use the sigmoid function as our activation function and binary cross-entropy for the loss.
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def sigmoid_derivative(x):
return x * (1 - x)
def binary_cross_entropy(y_true, y_pred):
return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
Step 4: Implement the Forward Pass
The forward pass computes the output of the neural network for a given input.
class SimpleNeuralNetwork:
# ... (same __init__ method)
def forward(self, X):
# Compute hidden layer activation
self.hidden_input = np.dot(X, self.weights_input_hidden) + self.bias_hidden
self.hidden_output = sigmoid(self.hidden_input)
# Compute output layer activation
self.output_input = np.dot(self.hidden_output, self.weights_hidden_output) + self.bias_output
self.output = sigmoid(self.output_input)
return self.output
Step 5: Implement Backpropagation
Backpropagation calculates the gradients of the loss with respect to each weight, which will be used to update the weights.
class SimpleNeuralNetwork:
# ... (same __init__ and forward methods)
def backward(self, X, y, learning_rate):
# Calculate the error in the output
output_error = self.output - y
output_delta = output_error * sigmoid_derivative(self.output)
# Calculate the error in the hidden layer
hidden_error = output_delta.dot(self.weights_hidden_output.T)
hidden_delta = hidden_error * sigmoid_derivative(self.hidden_output)
# Update the weights and biases
self.weights_hidden_output -= self.hidden_output.T.dot(output_delta) * learning_rate
self.bias_output -= np.sum(output_delta, axis=0, keepdims=True) * learning_rate
self.weights_input_hidden -= X.T.dot(hidden_delta) * learning_rate
self.bias_hidden -= np.sum(hidden_delta, axis=0, keepdims=True) * learning_rate
Step 6: Train the Neural Network
Now we need a method to train the network using forward and backward passes.
class SimpleNeuralNetwork:
# ... (same __init__, forward, and backward methods)
def train(self, X, y, epochs, learning_rate):
for epoch in range(epochs):
# Forward pass
self.forward(X)
# Backward pass
self.backward(X, y, learning_rate)
# Compute the loss
loss = binary_cross_entropy(y, self.output)
if epoch % 100 == 0:
print(f"Epoch {epoch}, Loss: {loss}")
Step 7: Test the Neural Network
To test the neural network, we'll create some dummy data. Let's create a simple dataset for binary classification.
# Dummy data for XOR problem
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])
# Initialize and train the neural network
nn = SimpleNeuralNetwork(input_size=2, hidden_size=2, output_size=1)
nn.train(X, y, epochs=10000, learning_rate=0.1)
# Test the neural network
output = nn.forward(X)
print("Predictions:")
print(output)