# Improving Your Neural Network

Module 8 of 8

Beginner Level

## Lesson 8: Evaluating and Improving the Neural Network 📈🎯

**Objective:**

- Understand and implement more robust evaluation metrics beyond accuracy (Confusion Matrix, Precision, Recall, F1-Score).
- Learn to diagnose overfitting and underfitting in the neural network.
- Conceptually understand techniques to improve model performance and combat overfitting, such as regularization and hyperparameter tuning.

---

### Recap of Lesson 7:

- Successfully built, trained, and made initial predictions with a 1-hidden-layer neural network from scratch.
- Calculated training and validation accuracy for this neural network (`train_accuracy_nn`, `val_accuracy_nn`).
- Our key outputs from the previous lesson for evaluation are:
  - `Y_pred_train_nn`, `Y_pred_val_nn` (predictions from our NN model)
  - `Y_train_for_comp`, `Y_val_for_comp` (true labels, shaped as `(1, m_samples)`)
  - `final_trained_nn_params` (the learned weights and biases)

---

### 1. Beyond Accuracy: More Robust Evaluation Metrics (Conceptual & Practical - 30 min)

While accuracy (percentage of correct predictions) is a common metric, it can be misleading, especially for datasets where classes are imbalanced.

Let’s define:

- **Positive Class:** Typically the class of interest (e.g., Survived = 1).
- **Negative Class:** The other class (e.g., Died = 0).
- **True Positives (TP):** Correctly predicted positive.
- **True Negatives (TN):** Correctly predicted negative.
- **False Positives (FP):** Incorrectly predicted positive.
- **False Negatives (FN):** Incorrectly predicted negative.

#### a) Confusion Matrix

A table that summarizes the performance of a classification model by showing TP, TN, FP, FN.

|  | Predicted Died (0) | Predicted Survived (1) |
| --- | --- | --- |
| **Actual Died (0)** | TN | FP |
| **Actual Survived (1)** | FN | TP |

Let’s calculate these for our validation set predictions (`Y_pred_val_nn`) and true labels (`Y_val_for_comp`).

```python
# (Ensure numpy is imported: import numpy as np)
# (Assume Y_pred_val_nn and Y_val_for_comp are available from Lesson 7)
import numpy as np

# For demonstration, create dummy predictions and labels if not available
if 'Y_pred_val_nn' not in locals() or 'Y_val_for_comp' not in locals() or \ 
   Y_pred_val_nn.size == 0 or Y_val_for_comp.size == 0:
    Y_val_for_comp = np.array([[1, 0, 1, 1, 0, 0, 1, 0, 1, 0]])
    Y_pred_val_nn  = np.array([[1, 1, 0, 1, 0, 0, 1, 1, 0, 1]])
else:
    pass  # Using existing data

TP = np.sum((Y_pred_val_nn == 1) & (Y_val_for_comp == 1))
TN = np.sum((Y_pred_val_nn == 0) & (Y_val_for_comp == 0))
FP = np.sum((Y_pred_val_nn == 1) & (Y_val_for_comp == 0))
FN = np.sum((Y_pred_val_nn == 0) & (Y_val_for_comp == 1))

confusion_matrix_scratch = np.array([[TN, FP], [FN, TP]])
```

Scikit-learn has a convenient function for this too.

b) Precision

Precision answers: “Of all passengers we predicted would survive, how many actually survived?”

```python
if (TP + FP) > 0:
    precision = TP / (TP + FP)
else:
    precision = 0.0
```

c) Recall (Sensitivity)

Recall answers: “Of all the passengers who actually survived, how many did our model correctly identify?”

```python
if (TP + FN) > 0:
    recall = TP / (TP + FN)
else:
    recall = 0.0
```

d) F1-Score

The F1-Score is the harmonic mean of Precision and Recall.

```python
if (precision + recall) > 0:
    f1_score = 2 * (precision * recall) / (precision + recall)
else:
    f1_score = 0.0
```

Discussion:

- Trade-off: Often, there’s a trade-off between precision and recall.
- Which metric is more important? It depends on the problem:
  - Spam detection: High precision is important.
  - Medical diagnosis: High recall is important.

---

### 2. Understanding and Diagnosing Overfitting and Underfitting (Conceptual - 20 min) 📉📈

After evaluating our model, we need to understand if its performance is optimal or if it’s suffering from common issues.

- **Underfitting (High Bias):** The model is too simple. Symptoms include high training error and high validation/test error.

- **Overfitting (High Variance):** The model learns training data too well, performing well on the training set but poorly on validation/test set.

- **Good Fit:** The model generalizes well.

**Diagnosing with Learning Curves:**

- **Cost Curves:** Plotting training cost and validation cost.
- **Accuracy Curves:** Plotting training accuracy and validation accuracy.

---

### 3. Techniques to Improve Model Performance / Combat Overfitting (Conceptual - 25 min)

If our model is underfitting or overfitting, here are some strategies:

**If Underfitting:**
1. **Increase Model Complexity.**
2. **Add More Features.**
3. **Train Longer.**
4. **Decrease Regularization.**
5. **Try a Different Optimization Algorithm.**

**If Overfitting:**
1. **Get More Training Data.**
2. **Reduce Model Complexity.**
3. **Early Stopping.**
4. **Feature Selection.**
5. **Regularization.**

---

### 4. Hyperparameter Tuning Strategies (Conceptual - 15 min) 🎛️

Finding good hyperparameters is crucial for model performance:
1. **Identify Key Hyperparameters.**
2. **Choose a Search Strategy:**
   - Manual Tuning
   - Grid Search
   - Random Search
3. **Use a Validation Set.**
4. **Iterate.**

---

### Wrap-up & Next Steps for Lesson 8:

- **Recap:** Explored evaluation metrics, diagnosed overfitting and underfitting, introduced techniques to combat overfitting, and discussed hyperparameter tuning.
- **Looking Ahead:** Future steps could involve implementing regularization or dropout, experimenting with different architectures, or transitioning to deep learning frameworks like TensorFlow or PyTorch.