Improving Your Neural Network

Module 8 of 8

Beginner Level

Lesson 8: Evaluating and Improving the Neural Network 📈🎯

Objective:

  • Understand and implement more robust evaluation metrics beyond accuracy (Confusion Matrix, Precision, Recall, F1-Score).
  • Learn to diagnose overfitting and underfitting in the neural network.
  • Conceptually understand techniques to improve model performance and combat overfitting, such as regularization and hyperparameter tuning.

Recap of Lesson 7:

  • Successfully built, trained, and made initial predictions with a 1-hidden-layer neural network from scratch.
  • Calculated training and validation accuracy for this neural network (train_accuracy_nn, val_accuracy_nn).
  • Our key outputs from the previous lesson for evaluation are:
    • Y_pred_train_nn, Y_pred_val_nn (predictions from our NN model)
    • Y_train_for_comp, Y_val_for_comp (true labels, shaped as (1, m_samples))
    • final_trained_nn_params (the learned weights and biases)

1. Beyond Accuracy: More Robust Evaluation Metrics (Conceptual & Practical - 30 min)

While accuracy (percentage of correct predictions) is a common metric, it can be misleading, especially for datasets where classes are imbalanced.

Let’s define:

  • Positive Class: Typically the class of interest (e.g., Survived = 1).
  • Negative Class: The other class (e.g., Died = 0).
  • True Positives (TP): Correctly predicted positive.
  • True Negatives (TN): Correctly predicted negative.
  • False Positives (FP): Incorrectly predicted positive.
  • False Negatives (FN): Incorrectly predicted negative.

a) Confusion Matrix

A table that summarizes the performance of a classification model by showing TP, TN, FP, FN.

Predicted Died (0) Predicted Survived (1)
Actual Died (0) TN FP
Actual Survived (1) FN TP

Let’s calculate these for our validation set predictions (Y_pred_val_nn) and true labels (Y_val_for_comp).

# (Ensure numpy is imported: import numpy as np)
# (Assume Y_pred_val_nn and Y_val_for_comp are available from Lesson 7)
import numpy as np

# For demonstration, create dummy predictions and labels if not available
if 'Y_pred_val_nn' not in locals() or 'Y_val_for_comp' not in locals() or \ 
   Y_pred_val_nn.size == 0 or Y_val_for_comp.size == 0:
    Y_val_for_comp = np.array([[1, 0, 1, 1, 0, 0, 1, 0, 1, 0]])
    Y_pred_val_nn  = np.array([[1, 1, 0, 1, 0, 0, 1, 1, 0, 1]])
else:
    pass  # Using existing data

TP = np.sum((Y_pred_val_nn == 1) & (Y_val_for_comp == 1))
TN = np.sum((Y_pred_val_nn == 0) & (Y_val_for_comp == 0))
FP = np.sum((Y_pred_val_nn == 1) & (Y_val_for_comp == 0))
FN = np.sum((Y_pred_val_nn == 0) & (Y_val_for_comp == 1))

confusion_matrix_scratch = np.array([[TN, FP], [FN, TP]])

Scikit-learn has a convenient function for this too.

b) Precision

Precision answers: “Of all passengers we predicted would survive, how many actually survived?”

if (TP + FP) > 0:
    precision = TP / (TP + FP)
else:
    precision = 0.0

c) Recall (Sensitivity)

Recall answers: “Of all the passengers who actually survived, how many did our model correctly identify?”

if (TP + FN) > 0:
    recall = TP / (TP + FN)
else:
    recall = 0.0

d) F1-Score

The F1-Score is the harmonic mean of Precision and Recall.

if (precision + recall) > 0:
    f1_score = 2 * (precision * recall) / (precision + recall)
else:
    f1_score = 0.0

Discussion:

  • Trade-off: Often, there’s a trade-off between precision and recall.
  • Which metric is more important? It depends on the problem:
    • Spam detection: High precision is important.
    • Medical diagnosis: High recall is important.

2. Understanding and Diagnosing Overfitting and Underfitting (Conceptual - 20 min) 📉📈

After evaluating our model, we need to understand if its performance is optimal or if it’s suffering from common issues.

  • Underfitting (High Bias): The model is too simple. Symptoms include high training error and high validation/test error.

  • Overfitting (High Variance): The model learns training data too well, performing well on the training set but poorly on validation/test set.

  • Good Fit: The model generalizes well.

Diagnosing with Learning Curves:

  • Cost Curves: Plotting training cost and validation cost.
  • Accuracy Curves: Plotting training accuracy and validation accuracy.

3. Techniques to Improve Model Performance / Combat Overfitting (Conceptual - 25 min)

If our model is underfitting or overfitting, here are some strategies:

If Underfitting:

  1. Increase Model Complexity.
  2. Add More Features.
  3. Train Longer.
  4. Decrease Regularization.
  5. Try a Different Optimization Algorithm.

If Overfitting:

  1. Get More Training Data.
  2. Reduce Model Complexity.
  3. Early Stopping.
  4. Feature Selection.
  5. Regularization.

4. Hyperparameter Tuning Strategies (Conceptual - 15 min) 🎛️

Finding good hyperparameters is crucial for model performance:

  1. Identify Key Hyperparameters.
  2. Choose a Search Strategy:
    • Manual Tuning
    • Grid Search
    • Random Search
  3. Use a Validation Set.
  4. Iterate.

Wrap-up & Next Steps for Lesson 8:

  • Recap: Explored evaluation metrics, diagnosed overfitting and underfitting, introduced techniques to combat overfitting, and discussed hyperparameter tuning.
  • Looking Ahead: Future steps could involve implementing regularization or dropout, experimenting with different architectures, or transitioning to deep learning frameworks like TensorFlow or PyTorch.