Understanding Neural Network Architecture

At its core, a neural network is a computational model inspired by the brain's neural connections. It consists of layers of interconnected nodes, referred to as neurons, which process and transmit information. The architecture is typically structured into three categories, as illustrated in the image below:

1. Input Layer: The first layer that receives raw data, including images, text, or numerical values.
2. Hidden Layers: Intermediate layers that lie between the input and output layers, where each neuron processes information and relays it to subsequent layers.
3. Output Layer: The final layer, which produces the network's predictions or outputs based on the processed data.

In our earlier discussion on perceptrons, we examined a single neuron, which operates similarly to logistic regression. Each neuron takes input values, applies weights, computes a weighted sum, and then utilizes an activation function to produce an output. This process is akin to biological neurons firing in response to stimuli, akin to logistic regression. The activation function enables neural networks to identify complex patterns within data. By stacking multiple neurons, we can create extensive and profound neural network architectures, the essence of neural networks.

Mathematical Foundation of Neural Networks

Now that we've established the theoretical groundwork, let’s delve into the mathematical expressions that underpin neural networks. Neurons in one layer connect to those in the subsequent layer through connections characterized by weights and biases. These weights determine the influence of one neuron's output on another.

The mathematical representation for multiple neurons in a single layer can be expressed as follows:

The weighted sum for multiple neurons in one layer:

Where: j = 1 … M (M denotes the number of neurons in that layer), z_j represents the output of the j-th neuron.

However, a more efficient method for calculating these operations in neural network layers employs vector notation (matrices). This can be mathematically expressed as:

The vector form:

Where z is a column vector of size M (Mx1), x is a column vector of size D (Dx1), w is a DxM matrix, and b is a vector of size M. The operation σ is an element-wise function that does not depend on the size of the matrices.

Input to Output for an L-layer Neural Network

As depicted in the accompanying image, we can mathematically articulate the neural network as follows:

Mathematical expression for hidden neurons:

Where w^(L)T denotes the weights corresponding to the L layer, x^(L-1) is the data from the preceding layer, and b^(L) corresponds to the bias for that layer.

The Hidden Layers

In the equations z² and z³, a subtle distinction from the primary mathematical formula is apparent. The initial equation utilizes X from the previous layer (input layer), while the subsequent equations draw on the outputs from earlier layers, hence the use of z.

It's vital to recognize that the output layer for binary classification differs from that of regression. In binary classification, we maintain the activation function to yield the predicted class probabilities, which can be defined as follows:

Binary Classification

For regression, the understanding is more straightforward:

Regression

Training a Neural Network

Training a neural network involves adjusting its weights and biases to minimize the difference between expected and actual outputs. This is achieved through backpropagation and optimization techniques like gradient descent. The chain rule of calculus is employed to compute gradients, leading to iterative weight adjustments to minimize the loss function. In the following article, we will address backpropagation and the loss function in greater detail.

If you wish to delve deeper into optimization techniques or the fundamental concept of gradient computation via the chain rule, please follow the links below.

Now that we’ve covered the theory, we can transition to the Python implementation.

The video "Bob Friday Talks: Bytes of Brilliance, Unveiling the AI Canvas" provides insights into the architecture of neural networks and their applications in AI.

Implementing Neural Networks in Python

import numpy as np

class FeedforwardNN:

def __init__(self, input_size, hidden_size, output_size):

self.input_size = input_size

self.hidden_size = hidden_size

self.output_size = output_size

self.weights_input_hidden = np.random.rand(self.input_size, self.hidden_size)

self.bias_hidden = np.zeros((1, self.hidden_size))

self.weights_hidden_output = np.random.rand(self.hidden_size, self.output_size)

self.bias_output = np.zeros((1, self.output_size))

def sigmoid(self, x):

return 1 / (1 + np.exp(-x))

def sigmoid_derivative(self, x):

return x * (1 - x)

def forward(self, input_data):

self.hidden_activation = self.sigmoid(np.dot(input_data, self.weights_input_hidden) + self.bias_hidden)

self.output_activation = self.sigmoid(np.dot(self.hidden_activation, self.weights_hidden_output) + self.bias_output)

return self.output_activation

def backward(self, input_data, target, learning_rate):

output_error = target - self.output_activation

output_delta = output_error * self.sigmoid_derivative(self.output_activation)

hidden_error = output_delta.dot(self.weights_hidden_output.T)

hidden_delta = hidden_error * self.sigmoid_derivative(self.hidden_activation)

self.weights_hidden_output += self.hidden_activation.T.dot(output_delta) * learning_rate

self.weights_input_hidden += input_data.reshape(-1, 1).dot(hidden_delta.reshape(1, -1)) * learning_rate

self.bias_output += np.sum(output_delta, axis=0, keepdims=True) * learning_rate

self.bias_hidden += np.sum(hidden_delta, axis=0, keepdims=True) * learning_rate

def train(self, training_data, target_data, epochs, learning_rate):

for epoch in range(epochs):

for input_data, target in zip(training_data, target_data):

self.forward(input_data)

self.backward(input_data, target, learning_rate)

# Example usage

training_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])

target_data = np.array([[0], [1], [1], [0]])

nn = FeedforwardNN(input_size=2, hidden_size=4, output_size=1)

nn.train(training_data, target_data, epochs=10000, learning_rate=0.1)

# Test predictions

for input_data in training_data:

prediction = nn.forward(input_data)

print(f"Input: {input_data}, Prediction: {prediction}")

Implementing Neural Networks with Keras

import numpy as np

from keras.models import Sequential

from keras.layers import Dense

# Generate synthetic training data

training_data = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])

target_data = np.array([[0], [1], [1], [0]])

# Build the neural network model

model = Sequential()

model.add(Dense(units=4, activation='sigmoid', input_dim=2)) # Input layer with 2 input nodes

model.add(Dense(units=1, activation='sigmoid')) # Output layer with 1 output node

# Compile the model

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

# Train the model

model.fit(training_data, target_data, epochs=10, verbose=0)

# Test predictions

for input_data in training_data:

prediction = model.predict(np.array([input_data]))

print(f"Input: {input_data}, Prediction: {prediction[0][0]}")

The video "Neural Network Architectures & Deep Learning" offers a comprehensive overview of various neural network architectures and their implications in deep learning.