Neural Network

A Neural Network is inspired by the human brain. It consists of artificial neurons that are interconnected and process information through learned weights.

Perceptron

The perceptron is the smallest unit of a neural network:

output = f(w1*x1 + w2*x2 + ... + wn*xn + b)

Where f is the activation function.

Activation Functions

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-5, 5, 100)

# Sigmoid
sigmoid = 1 / (1 + np.exp(-x))

# ReLU
relu = np.maximum(0, x)

# Tanh
tanh = np.tanh(x)

# Leaky ReLU
leaky_relu = np.where(x > 0, x, 0.01 * x)

fig, axes = plt.subplots(2, 2, figsize=(10, 8))
axes[0, 0].plot(x, sigmoid); axes[0, 0].set_title('Sigmoid')
axes[0, 1].plot(x, relu); axes[0, 1].set_title('ReLU')
axes[1, 0].plot(x, tanh); axes[1, 0].set_title('Tanh')
axes[1, 1].plot(x, leaky_relu); axes[1, 1].set_title('Leaky ReLU')
for ax in axes.flat: ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Forward Propagation

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Manual forward propagation
X = np.array([[0.5, 0.3], [0.9, 0.1]])
W1 = np.random.randn(2, 4)
b1 = np.zeros((1, 4))
W2 = np.random.randn(4, 1)
b2 = np.zeros((1, 1))

# Hidden layer
z1 = X.dot(W1) + b1
a1 = sigmoid(z1)

# Output layer
z2 = a1.dot(W2) + b2
a2 = sigmoid(z2)

print(f"Output shape: {a2.shape}")

Backpropagation

Backpropagation computes the gradient of the loss with respect to every weight using the chain rule.

# Loss function (Binary Cross Entropy)
def binary_crossentropy(y_true, y_pred):
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Sigmoid derivative
def sigmoid_derivative(z):
    s = sigmoid(z)
    return s * (1 - s)

Neural Network with Keras

from tensorflow import keras
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder

iris = load_iris()
X = StandardScaler().fit_transform(iris.data)
y = OneHotEncoder(sparse_output=False).fit_transform(iris.target.reshape(-1, 1))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = keras.Sequential([
    keras.layers.Dense(64, activation='relu', input_shape=(4,)),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(3, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

model.summary()

history = model.fit(X_train, y_train, epochs=100, batch_size=16,
                    validation_split=0.2, verbose=0)

loss, acc = model.evaluate(X_test, y_test)
print(f"Test Accuracy: {acc:.4f}")

Loss Functions

Optimizers

Training Tips

• Learning rate: start at 0.001, lower it if it diverges
• Batch size: 32-256 for small-medium data
• Epochs: use early stopping
• Regularization: Dropout, L2 to reduce overfitting

Next chapter: Deep Learning, more complex neural network architectures.

Next: Deep Learning →