# Putthing A Neural Network Together

## Lesson 7: Neural Network from Scratch - Part 2 (Cost, Backward Propagation & Training Loop) 🧠⚙️📉

**Objective:**

- Implement the cost function (Binary Cross-Entropy) for our 2-layer neural network.
- Understand the principles of backward propagation and derive/state the necessary gradient formulas.
- Implement the backward propagation steps to calculate gradients for all parameters.
- Implement the parameter update rule (gradient descent).
- Combine all parts into a complete training loop for the neural network.
- Train and evaluate the neural network.

---

### Recap of Lesson 6:

- Defined a 2-layer neural network architecture (Input -> Hidden (ReLU) -> Output (Sigmoid)).
- Implemented `initialize_parameters_nn` to set up W[1],b[1],W[2],b[2].
- Implemented `forward_propagation_nn` which computes A[2] (the prediction) and a `cache` containing intermediate values (Z[1],A[1],Z[2],A[2]).
- Remembered the importance of input data shape for NN functions: `(n_features, m_samples)`.

---

### 1. Computing the Cost Function (Practical - 15 min) 💸

After forward propagation, we get A[2], which is a vector of predicted probabilities (shape `(1, m_samples)` because ny=1). We need to compare these predictions to the true labels Y (shape `(1, m_samples)`) to see how well our network is doing.

The **cost function** (or loss function) quantifies this "error." For binary classification, we use the **Binary Cross-Entropy** loss, which is the same as we used for Logistic Regression, but now applied to the output of our neural network A[2].

The formula for m training examples is:

J=−1/m∑i=1m(y(i)log⁡(a[2](i))+(1−y(i))log⁡(1−a[2](i)))

Where:

- m is the number of training examples.
- y(i) is the true label for the i-th example.
- $a^{[2]}$ is the predicted probability for the i-th example (from A[2]).

```python
# (Ensure numpy is imported: import numpy as np)

def compute_cost_nn(A2, Y):
    """
    Computes the cross-entropy cost given predictions A2 and true labels Y.

Arguments:
    A2 -- The sigmoid output of the second activation, of shape (1, m_samples) (predictions)
    Y -- "true" labels vector of shape (1, m_samples) (e.g., 0 if non-survived, 1 if survived)

Returns:
    cost -- cross-entropy cost (scalar)
    """
    m = Y.shape[1] # Number of examples (Y has shape (1, m_samples))

epsilon = 1e-15

logprobs = np.multiply(np.log(A2 + epsilon), Y) + np.multiply(np.log(1 - A2 + epsilon), (1 - Y))
    cost = - (1/m) * np.sum(logprobs)

cost = np.squeeze(cost) # To make sure cost is a scalar
    assert(isinstance(cost, float))

return cost
```

### 2. Backward Propagation: The Intuition (Conceptual - 20 min) ⏪

Backward propagation (or "backprop") is the algorithm used to calculate the gradients of the cost function J with respect to each parameter (W[1],b[1],W[2],b[2]) in the network. These gradients tell us how much a small change in each parameter would affect the cost.

- **Why do we need gradients?** To perform gradient descent. We want to update our parameters in the direction that _minimizes_ the cost. Gradients point in the direction of the steepest _ascent_, so we move in the opposite direction.
- **The Chain Rule:** Backprop relies heavily on the chain rule from calculus.

**Flow of Backpropagation:**
1. **Calculate dZ[2]:**  dZ[2]=A[2]−Y
2. **Calculate dW[2] and db[2]:**  dW[2]=1/mdZ[2]A[1]T;  db[2]=1/m∑examplesdZ[2]
3. **Calculate dA[1]:**  dA[1]=W[2]TdZ[2]
4. **Calculate dZ[1]:**  dZ[1]=dA[1]∗g[1]′(Z[1])
5. **Calculate dW[1] and db[1]:**  dW[1]=1/mdZ[1]XT; db[1]=1/m∑examplesdZ[1]

_(The actual implementation of the `backward_propagation_nn` function will be in the next part of this lesson)._

---

### 3. Implementing Backward Propagation (Practical - 30 min) ⚙️⏪

Now, let’s translate the formulas from the conceptual overview into a Python function. We'll need the `cache` from forward propagation, which contains Z[1],A[1],Z[2],A[2].

```python

def backward_propagation_nn(parameters, cache, X_input_nn, Y_labels_nn):
    """
    Implements the backward propagation for the 2-layer neural network.

Arguments:
    parameters -- parameters (W1, b1, W2, b2)
    cache -- cache (from forward_propagation_nn)
    X_input_nn -- input data of shape (n_x, m_samples)
    Y_labels_nn -- "true" labels vector of shape (1, m_samples)

Returns:
    grads -- gradients with respect to different parameters:
             dW1, db1, dW2, db2
    """
    m = X_input_nn.shape[1] # Number of examples

W2 = parameters["W2"]
    A1 = cache["A1"]
    A2 = cache["A2"]
    Z1 = cache["Z1"]

dZ2 = A2 - Y_labels_nn  
    dW2 = (1/m) * np.dot(dZ2, A1.T) 
    db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True) 
    dA1 = np.dot(W2.T, dZ2) 
    dRelu_Z1 = np.where(Z1 > 0, 1, 0) 
    dZ1 = dA1 * dRelu_Z1 
    dW1 = (1/m) * np.dot(dZ1, X_input_nn.T) 
    db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)

grads = {"dW1": dW1,
             "db1": db1,
             "dW2": dW2,
             "db2": db2}

return grads
```

### Updating Parameters (Practical - 10 min) 🛠️

Once we have the gradients, we update the parameters using gradient descent:

```python

def update_parameters_nn(parameters, grads, learning_rate):
    """
    Updates parameters using the gradient descent update rule.

Arguments:
    parameters -- parameters (W1, b1, W2, b2)
    grads -- gradients (dW1, db1, dW2, db2)
    learning_rate -- the learning rate, alpha.

Returns:
    parameters -- updated parameters
    """
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]

W1 = W1 - learning_rate * grads["dW1"]
    b1 = b1 - learning_rate * grads["db1"]
    W2 = W2 - learning_rate * grads["dW2"]
    b2 = b2 - learning_rate * grads["db2"]

parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

return parameters
```

### Building the Neural Network Model (Training Loop) (Practical - 25 min) 🔄

Now we integrate all the pieces: initialize, loop for num_iterations (epochs): forward prop, compute cost, backward prop, update parameters.

```python

def nn_model(X_input_nn, Y_labels_nn, n_h, num_iterations=10000, learning_rate=0.0075, print_cost_every=1000):
    """
    Arguments:
    X_input_nn -- input data of shape (n_x, m_samples)
    Y_labels_nn -- true "label" vector of shape (1, m_samples)
    n_h -- size of the hidden layer
    num_iterations -- Number of iterations in gradient descent loop
    learning_rate -- learning rate for the gradient descent update rule
    print_cost_every -- if True, print the cost every 1000 iterations

Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    costs -- list of costs recorded during training
    """
    if X_input_nn.size == 0 or Y_labels_nn.size == 0:
        print("Error: Input data or labels are empty. Cannot train model.")
        return None, []

np.random.seed(3) 
    n_x = X_input_nn.shape[0] 
    n_y = Y_labels_nn.shape[0]

parameters = initialize_parameters_nn(n_x, n_h, n_y)
    costs = []

for i in range(0, num_iterations):
        A2, cache = forward_propagation_nn(X_input_nn, parameters)
        cost = compute_cost_nn(A2, Y_labels_nn)
        grads = backward_propagation_nn(parameters, cache, X_input_nn, Y_labels_nn)
        parameters = update_parameters_nn(parameters, grads, learning_rate)

if print_cost_every > 0 and i % print_cost_every == 0:
            costs.append(cost)
            print (f"Cost after iteration {i}: {cost:.6f}")
    return parameters, costs
```

### Predictions with the Neural Network (Practical - 10 min) 🔮

Once the model is trained, we can use the learned parameters to make predictions.

```python

def predict_nn(X_input_nn, parameters, threshold=0.5):
    """
    Using the learned parameters, predicts a class for each example in X.

Arguments:
    X_input_nn -- input data of size (n_x, m_samples)
    parameters -- python dictionary containing parameters W1, b1, W2, b2
    threshold -- probability threshold to classify as 1

Returns:
    predictions -- vector of predictions of our model (0 / 1)
    """
    if parameters is None or parameters.get("W1") is None:
        print("Error: Model parameters are not available for prediction.")
        return np.array([])

A2, cache = forward_propagation_nn(X_input_nn, parameters)
    predictions = (A2 >= threshold).astype(int)

return predictions
```

### 7. Discussion & Next Steps (Conceptual - 10 min)
- **Performance:** How does this simple neural network compare to the Logistic Regression model from scratch?
- **Hyperparameters:**  
  - `n_h`: Number of hidden units.  
- **Overfitting/Underfitting:** If training accuracy is high but validation accuracy is much lower, the model might be overfitting.
- **Improvements (Conceptual):**  
 - **Deeper Networks:** Add more hidden layers.
 - **Different Activation Functions:** Experiment.

---

### Wrap-up & Teaser for Lesson 8:
- **Recap:** We’ve built our first neural network from scratch!

- **Teaser for Lesson 8:** In the next lesson, “Evaluating and Improving the Neural Network,” we’ll dive deeper into:  
  - More robust evaluation metrics.