Day 21: Fine-Tune and Evaluate Autoencoder Model for Anomaly Detection

Today, we continued with our autoencoder-based anomaly detection by fine-tuning the model and evaluating its performance on the Credit Card Fraud Detection Dataset. Our objectives were:

  1. Fine-Tune the Autoencoder Model: Improve model performance by adjusting hyperparameters.
  2. Determine Reconstruction Error Threshold: Use reconstruction error to classify normal vs. fraudulent transactions.
  3. Evaluate Performance: Utilize metrics like Precision, Recall, F1-Score, and AUC to understand the model’s effectiveness.

Step 1: Fine-Tuning the Autoencoder Model

In this step, we experimented with different model configurations and hyperparameters to try and reduce the reconstruction error.

Modify the Model Architecture

We experimented with the number of neurons and layers to see if a different configuration would yield better results.

# Adjust the model complexity
model = Sequential()

# Encoder
model.add(Dense(20, activation='relu', input_shape=(X_train.shape[1],)))
model.add(Dense(10, activation='relu'))

# Latent representation
model.add(Dense(5, activation='relu'))  # Smaller latent space to capture key features

# Decoder
model.add(Dense(10, activation='relu'))
model.add(Dense(20, activation='relu'))
model.add(Dense(X_train.shape[1], activation='linear'))

# Compile with modified learning rate
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss='mse')
model.summary()

# Train the model
history = model.fit(X_train_normal, X_train_normal,
                    epochs=100,
                    batch_size=128,
                    validation_split=0.2,
                    verbose=1)
  • Encoder/Decoder Changes: We added more layers and neurons to increase model complexity. A smaller latent space helped in focusing on key patterns.
  • Learning Rate: The learning rate was set to 0.001 to ensure smoother convergence.
  • Training Epochs: Increased the number of epochs to 100 for more training time.

Plot Training Loss

We plotted the training and validation loss to understand if the model was learning effectively.

plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.title('Training and Validation Loss for Fine-Tuned Autoencoder')
plt.show()
  • A decreasing trend in both losses indicated that the model was improving in reconstruction. If there was a large gap between the two, it would indicate overfitting.

Step 2: Set Threshold for Anomaly Detection

After training the autoencoder, we calculated the reconstruction error for all transactions in the test set and used this to classify transactions as either normal or anomalous.

# Predict the reconstructed test data
X_test_pred = model.predict(X_test)

# Calculate reconstruction error
reconstruction_errors = np.mean(np.power(X_test - X_test_pred, 2), axis=1)
  • Reconstruction Error: Calculated using the Mean Squared Error (MSE) between X_test and X_test_pred for each data sample.

Determine the Threshold

We set a threshold based on the 95th percentile of the reconstruction errors for the normal transactions in the training set.

threshold = np.percentile(reconstruction_errors[y_test == 0], 95)
print(f"Threshold for anomaly detection: {threshold}")
  • A high reconstruction error indicates an anomaly since the autoencoder has difficulty reconstructing fraudulent transactions.
  • The 95th percentile was chosen to allow for a balance between false positives and true negatives.

Step 3: Classify and Evaluate

Using the reconstruction error threshold, we classified each test transaction as normal or anomalous.

y_pred = [1 if error > threshold else 0 for error in reconstruction_errors]

# Actual labels for evaluation
print(f"Actual Anomalies: {sum(y_test)}, Detected: {sum(y_pred)}")
  • Threshold-based Classification: Transactions with reconstruction errors above the threshold are labeled as 1 (fraud), while those below are labeled as 0 (normal).

Evaluate Model Performance

We used classification metrics to evaluate the model’s performance:

from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score

# Confusion Matrix
conf_matrix = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(conf_matrix)

# Classification Report
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

# AUC Score
auc_score = roc_auc_score(y_test, reconstruction_errors)
print(f"AUC Score: {auc_score:.2f}")
  • Confusion Matrix: Shows True Negatives, False Positives, False Negatives, and True Positives.
  • Classification Report: Includes Precision, Recall, and F1-Score.
    • Precision tells us how many flagged transactions were actually fraud.
    • Recall tells us how many of the fraudulent transactions were detected.
    • F1-Score is a harmonic mean of precision and recall.
  • AUC Score: Measures the model’s ability to distinguish between positive and negative classes. A higher AUC is better.

Observations from Evaluation

Threshold for anomaly detection is: 1.0146805608893366
Actual Anomalies: 98, Detected: 2932
Confusion Matrix:
 [[54020  2844]
 [   10    88]]
Classification Report:
               precision    recall  f1-score   support

           0       1.00      0.95      0.97     56864
           1       0.03      0.90      0.06        98

    accuracy                           0.95     56962
   macro avg       0.51      0.92      0.52     56962
weighted avg       1.00      0.95      0.97     56962

The metrics showed the following:

  • The model had a high recall, meaning it was good at detecting fraudulent transactions, but it also had a low precision, meaning there were many false positives.
  • High False Positive Rate: Many normal transactions were incorrectly flagged as fraud, which indicates that the threshold may need further fine-tuning to reduce these false positives.

Improving the Model

To improve the model’s performance, we could try:

  1. Adjusting the Threshold: Trying different percentile values or using a validation dataset to determine the best threshold value.
  2. Model Complexity: Adding more neurons or layers, or even using a Variational Autoencoder (VAE) for better results.
  3. Resampling the Data: To handle the imbalance in the dataset, we could use oversampling for the fraud class to help the model learn these patterns better.

Video