Putthing A Neural Network Together

Lesson 7: Neural Network from Scratch - Part 2 (Cost, Backward Propagation & Training Loop) 🧠⚙️📉

Objective:

  • Implement the cost function (Binary Cross-Entropy) for our 2-layer neural network.
  • Understand the principles of backward propagation and derive/state the necessary gradient formulas.
  • Implement the backward propagation steps to calculate gradients for all parameters.
  • Implement the parameter update rule (gradient descent).
  • Combine all parts into a complete training loop for the neural network.
  • Train and evaluate the neural network.

Recap of Lesson 6:

  • Defined a 2-layer neural network architecture (Input -> Hidden (ReLU) -> Output (Sigmoid)).
  • Implemented initialize_parameters_nn to set up W[1],b[1],W[2],b[2].
  • Implemented forward_propagation_nn which computes A[2] (the prediction) and a cache containing intermediate values (Z[1],A[1],Z[2],A[2]).
  • Remembered the importance of input data shape for NN functions: (n_features, m_samples).

1. Computing the Cost Function (Practical - 15 min) 💸

After forward propagation, we get A[2], which is a vector of predicted probabilities (shape (1, m_samples) because ny=1). We need to compare these predictions to the true labels Y (shape (1, m_samples)) to see how well our network is doing.

The cost function (or loss function) quantifies this "error." For binary classification, we use the Binary Cross-Entropy loss, which is the same as we used for Logistic Regression, but now applied to the output of our neural network A[2].

The formula for m training examples is:

J=−1/m∑i=1m(y(i)log⁡(a2)+(1−y(i))log⁡(1−a2))

Where:

  • m is the number of training examples.
  • y(i) is the true label for the i-th example.
  • $a^{[2]}$ is the predicted probability for the i-th example (from A[2]).
# (Ensure numpy is imported: import numpy as np)

def compute_cost_nn(A2, Y):
    """
    Computes the cross-entropy cost given predictions A2 and true labels Y.

Arguments:
    A2 -- The sigmoid output of the second activation, of shape (1, m_samples) (predictions)
    Y -- "true" labels vector of shape (1, m_samples) (e.g., 0 if non-survived, 1 if survived)

Returns:
    cost -- cross-entropy cost (scalar)
    """
    m = Y.shape[1] # Number of examples (Y has shape (1, m_samples))

epsilon = 1e-15

logprobs = np.multiply(np.log(A2 + epsilon), Y) + np.multiply(np.log(1 - A2 + epsilon), (1 - Y))
    cost = - (1/m) * np.sum(logprobs)

cost = np.squeeze(cost) # To make sure cost is a scalar
    assert(isinstance(cost, float))

return cost

2. Backward Propagation: The Intuition (Conceptual - 20 min) ⏪

Backward propagation (or "backprop") is the algorithm used to calculate the gradients of the cost function J with respect to each parameter (W[1],b[1],W[2],b[2]) in the network. These gradients tell us how much a small change in each parameter would affect the cost.

  • Why do we need gradients? To perform gradient descent. We want to update our parameters in the direction that minimizes the cost. Gradients point in the direction of the steepest ascent, so we move in the opposite direction.
  • The Chain Rule: Backprop relies heavily on the chain rule from calculus.

Flow of Backpropagation:

  1. Calculate dZ[2]: dZ[2]=A[2]−Y
  2. Calculate dW[2] and db[2]: dW[2]=1/mdZ[2]A[1]T; db[2]=1/m∑examplesdZ[2]
  3. Calculate dA[1]: dA[1]=W[2]TdZ[2]
  4. Calculate dZ[1]: dZ[1]=dA[1]∗g[1]′(Z[1])
  5. Calculate dW[1] and db[1]: dW[1]=1/mdZ[1]XT; db[1]=1/m∑examplesdZ[1]

(The actual implementation of the backward_propagation_nn function will be in the next part of this lesson).

3. Implementing Backward Propagation (Practical - 30 min) ⚙️⏪

Now, let’s translate the formulas from the conceptual overview into a Python function. We'll need the cache from forward propagation, which contains Z[1],A[1],Z[2],A[2].


def backward_propagation_nn(parameters, cache, X_input_nn, Y_labels_nn):
    """
    Implements the backward propagation for the 2-layer neural network.

Arguments:
    parameters -- parameters (W1, b1, W2, b2)
    cache -- cache (from forward_propagation_nn)
    X_input_nn -- input data of shape (n_x, m_samples)
    Y_labels_nn -- "true" labels vector of shape (1, m_samples)

Returns:
    grads -- gradients with respect to different parameters:
             dW1, db1, dW2, db2
    """
    m = X_input_nn.shape[1] # Number of examples

W2 = parameters["W2"]
    A1 = cache["A1"]
    A2 = cache["A2"]
    Z1 = cache["Z1"]

dZ2 = A2 - Y_labels_nn  
    dW2 = (1/m) * np.dot(dZ2, A1.T) 
    db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True) 
    dA1 = np.dot(W2.T, dZ2) 
    dRelu_Z1 = np.where(Z1 > 0, 1, 0) 
    dZ1 = dA1 * dRelu_Z1 
    dW1 = (1/m) * np.dot(dZ1, X_input_nn.T) 
    db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)

grads = {"dW1": dW1,
             "db1": db1,
             "dW2": dW2,
             "db2": db2}

return grads

Updating Parameters (Practical - 10 min) 🛠️

Once we have the gradients, we update the parameters using gradient descent:


def update_parameters_nn(parameters, grads, learning_rate):
    """
    Updates parameters using the gradient descent update rule.

Arguments:
    parameters -- parameters (W1, b1, W2, b2)
    grads -- gradients (dW1, db1, dW2, db2)
    learning_rate -- the learning rate, alpha.

Returns:
    parameters -- updated parameters
    """
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

W1 = W1 - learning_rate * grads["dW1"]
    b1 = b1 - learning_rate * grads["db1"]
    W2 = W2 - learning_rate * grads["dW2"]
    b2 = b2 - learning_rate * grads["db2"]

parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

return parameters

Building the Neural Network Model (Training Loop) (Practical - 25 min) 🔄

Now we integrate all the pieces: initialize, loop for num_iterations (epochs): forward prop, compute cost, backward prop, update parameters.


def nn_model(X_input_nn, Y_labels_nn, n_h, num_iterations=10000, learning_rate=0.0075, print_cost_every=1000):
    """
    Arguments:
    X_input_nn -- input data of shape (n_x, m_samples)
    Y_labels_nn -- true "label" vector of shape (1, m_samples)
    n_h -- size of the hidden layer
    num_iterations -- Number of iterations in gradient descent loop
    learning_rate -- learning rate for the gradient descent update rule
    print_cost_every -- if True, print the cost every 1000 iterations

Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    costs -- list of costs recorded during training
    """
    if X_input_nn.size == 0 or Y_labels_nn.size == 0:
        print("Error: Input data or labels are empty. Cannot train model.")
        return None, []

np.random.seed(3) 
    n_x = X_input_nn.shape[0] 
    n_y = Y_labels_nn.shape[0]

parameters = initialize_parameters_nn(n_x, n_h, n_y)
    costs = []

for i in range(0, num_iterations):
        A2, cache = forward_propagation_nn(X_input_nn, parameters)
        cost = compute_cost_nn(A2, Y_labels_nn)
        grads = backward_propagation_nn(parameters, cache, X_input_nn, Y_labels_nn)
        parameters = update_parameters_nn(parameters, grads, learning_rate)

if print_cost_every > 0 and i % print_cost_every == 0:
            costs.append(cost)
            print (f"Cost after iteration {i}: {cost:.6f}")
    return parameters, costs

Predictions with the Neural Network (Practical - 10 min) 🔮

Once the model is trained, we can use the learned parameters to make predictions.


def predict_nn(X_input_nn, parameters, threshold=0.5):
    """
    Using the learned parameters, predicts a class for each example in X.

Arguments:
    X_input_nn -- input data of size (n_x, m_samples)
    parameters -- python dictionary containing parameters W1, b1, W2, b2
    threshold -- probability threshold to classify as 1

Returns:
    predictions -- vector of predictions of our model (0 / 1)
    """
    if parameters is None or parameters.get("W1") is None:
        print("Error: Model parameters are not available for prediction.")
        return np.array([])

A2, cache = forward_propagation_nn(X_input_nn, parameters)
    predictions = (A2 >= threshold).astype(int)

return predictions

7. Discussion & Next Steps (Conceptual - 10 min)

  • Performance: How does this simple neural network compare to the Logistic Regression model from scratch?
  • Hyperparameters:
    • n_h: Number of hidden units.
  • Overfitting/Underfitting: If training accuracy is high but validation accuracy is much lower, the model might be overfitting.
  • Improvements (Conceptual):
  • Deeper Networks: Add more hidden layers.
  • Different Activation Functions: Experiment.

Wrap-up & Teaser for Lesson 8:

  • Recap: We’ve built our first neural network from scratch!

  • Teaser for Lesson 8: In the next lesson, “Evaluating and Improving the Neural Network,” we’ll dive deeper into:

    • More robust evaluation metrics.