PRACTICAL 7: COMPARING CLASSIFICATION MODELS

Dataset: pulsar_stars.csv (target_class = 0 / 1)

STEPS (same for every model):
1. Install the libraries:  pip install pandas numpy scikit-learn matplotlib
2. Load pulsar_stars.csv and separate the features X and the target y (target_class).
3. Scale the features (StandardScaler) and split the data into training and testing sets.
4. Train the model on the training set and predict on the test set.
5. Calculate the metrics (accuracy, confusion matrix, precision, sensitivity, specificity, AUC).
6. Draw the plots and compare the results of all six models.


1. LINEAR REGRESSION

CODE:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score, accuracy_score
import matplotlib.pyplot as plt
import numpy as np
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data.columns = data.columns.str.strip()
X = data[['Skewness_DM_SNR_curve']]
y = data['target_class']
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
lr_model = LinearRegression()
lr_model.fit(X_train, y_train)
y_pred = lr_model.predict(X_test)
y_pred_binary = [1 if pred >= 0.5 else 0 for pred in y_pred]
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
accuracy = accuracy_score(y_test, y_pred_binary)
plt.figure(figsize=(10, 6))
plt.scatter(y_test, y_pred, color='blue', alpha=0.6, edgecolor='k')
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], color='red', lw=2)
plt.xlabel('Actual Values')
plt.ylabel('Predicted Values')
plt.title('Actual vs Predicted Values - Linear Regression')
plt.show()
residuals = y_test - y_pred
plt.figure(figsize=(10, 6))
plt.scatter(y_pred, residuals, color='green', alpha=0.6, edgecolor='k')
plt.axhline(0, color='red', linestyle='--', lw=2)
plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residuals Plot - Linear Regression')
plt.show()
print("Mean Squared Error:", mse)
print("R-squared:", r2)
print("Accuracy:", accuracy)

EXPLANATION:
- Linear regression is used with one feature (Skewness_DM_SNR_curve) to predict the class value.
- The predicted values are converted to 0 / 1 using the 0.5 threshold, so accuracy can be calculated.
- Mean squared error and R-squared tell how close the predictions are to the actual values.
- The Actual vs Predicted plot and the Residuals plot show the errors of the model.
- Linear regression is not really meant for classification, so it is the weakest model here.


2. DECISION TREE

CODE:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import (accuracy_score, confusion_matrix,
precision_score,
recall_score, roc_curve, auc)
import matplotlib.pyplot as plt
import numpy as np
filepath = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(filepath)
# Binarize the target_class
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
X = data.drop(columns=['target_class'])
y = data['target_class']
# Split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
random_state=42)
# Initialize and train the model
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X_train, y_train)
# Get predictions and probabilities
y_pred = clf.predict(X_test)
y_pred_prob = clf.predict_proba(X_test)[:, 1]
# Calculate metrics
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
precision = precision_score(y_test, y_pred)
sensitivity = recall_score(y_test, y_pred)
TN = conf_matrix[0, 0]
FP = conf_matrix[0, 1]
specificity = TN / (TN + FP)
standard_error = np.sqrt((accuracy * (1 - accuracy)) / len(y_test))
# ROC and AUC
fpr, tpr, _ = roc_curve(y_test, y_pred_prob)
roc_auc = auc(fpr, tpr)
# Plot ROC Curve
plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)'% roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
plt.show()
# Plot Decision Tree
plt.figure(figsize=(12, 8))
plot_tree(clf, feature_names=X.columns, class_names=['0', '1'], filled=True,
max_depth=3)
plt.title("Decision Tree (Max Depth = 3)")
plt.show()
# Print Results
print("Predictions:", y_pred)
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", conf_matrix)
print("Standard Error:", standard_error)
print("Precision:", precision)
print("Sensitivity (Recall):", sensitivity)
print("Specificity:", specificity)
print("AUC:", roc_auc)

EXPLANATION:
- The decision tree splits the data again and again on feature conditions until the classes are separated.
- All eight features are used and the data is split 80% train / 20% test.
- The confusion matrix gives precision, sensitivity (recall), specificity and the standard error of the accuracy.
- The ROC curve and AUC value show how well the tree separates pulsar and non-pulsar stars.
- plot_tree() draws the first 3 levels of the tree.


3. NAIVE BAYES

CODE:
#Naive Bayes
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import (
    accuracy_score,
    confusion_matrix,
    precision_score,
    recall_score,
    roc_curve,
    auc
)
import matplotlib.pyplot as plt
import numpy as np
# Load the dataset
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
# Convert target_class to binary if it has continuous values
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
# Define features and target variable
X = data.drop(columns=['target_class'])
y = data['target_class']
# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Split the dataset (20% for test and 80% for train)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)
# Create a Naive Bayes Classifier
nb_model = GaussianNB()
# Train the model
nb_model.fit(X_train, y_train)
# Make predictions
y_pred = nb_model.predict(X_test)
y_pred_prob = nb_model.predict_proba(X_test)[:, 1]
# Calculate metrics
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
precision = precision_score(y_test, y_pred)
sensitivity = recall_score(y_test, y_pred)
# Specificity calculation
TN = conf_matrix[0, 0]
FP = conf_matrix[0, 1]
specificity = TN / (TN + FP)
# Standard error
standard_error = np.sqrt((accuracy * (1 - accuracy)) / len(y_test))
# Calculate ROC curve and AUC
fpr, tpr, _ = roc_curve(y_test, y_pred_prob)
roc_auc = auc(fpr, tpr)
# Plot ROC curve
plt.figure()
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic - Naive Bayes')
plt.legend(loc="lower right")
plt.show()
# Print results
print("Predictions:", y_pred)
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", conf_matrix)
print("Standard Error:", standard_error)
print("Precision:", precision)
print("Sensitivity (Recall):", sensitivity)
print("Specificity:", specificity)
print("AUC:", roc_auc)

EXPLANATION:
- Gaussian Naive Bayes uses Bayes' theorem and assumes the features are independent and follow a normal distribution.
- The features are standardized first and the data is split 80% train / 20% test.
- It is very fast because it only calculates the mean and variance of each feature for each class.
- predict_proba() gives the class probabilities, which are used to draw the ROC curve.
- Accuracy, precision, sensitivity, specificity, standard error and AUC are printed for comparison.


4. PCA (PRINCIPAL COMPONENT ANALYSIS) WITH LOGISTIC REGRESSION

CODE:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt
import numpy as np
# Load the dataset
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
# Convert target_class to binary if it has continuous values
data['target_class'] = data['target_class'].apply(lambda x: 1 if x >= 0.5 else 0)
# Define features and target variable
X = data.drop(columns=['target_class'])
y = data['target_class']
# Standardize the features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Apply PCA to reduce dimensionality
n_components = 7
pca = PCA(n_components=n_components)
X_pca = pca.fit_transform(X_scaled)
# Explained variance ratio for PCA
explained_variance_ratio = pca.explained_variance_ratio_
total_variance_explained = np.sum(explained_variance_ratio)
# Print PCA results
print("Explained Variance Ratio:", explained_variance_ratio)
print("Total Variance Explained:", total_variance_explained)
# Visualize the explained variance ratio
plt.figure(figsize=(8, 5))
plt.bar(range(1, n_components + 1), explained_variance_ratio, alpha=0.7, align='center', label='Individual Explained Variance')
plt.step(range(1, n_components + 1), np.cumsum(explained_variance_ratio), where='mid', label='Cumulative Explained Variance')
plt.xlabel('Principal Component Index')
plt.ylabel('Explained Variance Ratio')
plt.title('PCA Explained Variance')
plt.legend(loc='best')
plt.show()
# Split the dataset using PCA features
X_train, X_test, y_train, y_test = train_test_split(X_pca, y, test_size=0.2, random_state=42)
# Train a Logistic Regression model
model = LogisticRegression()
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
# Print accuracy
print("Accuracy:", accuracy)

EXPLANATION:
- PCA reduces the 8 features to 7 new uncorrelated features called principal components.
- The explained variance ratio tells how much of the information each component keeps.
- The bar and step plot show individual and cumulative explained variance.
- Logistic regression is trained on the reduced features and its accuracy is printed.
- This checks whether the model still works well after reducing the dimensions.


5. SVM (SUPPORT VECTOR MACHINE)

CODE:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
file_path = "C:/Users/Admin/Downloads/pulsar_stars.csv"
data = pd.read_csv(file_path)
data.columns = data.columns.str.strip()
X = data.drop(columns=['target_class'])
y = data['target_class']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
svm_model = SVC(
    kernel='rbf',
    class_weight='balanced',
    probability=True,
    random_state=42
)
svm_model.fit(X_train_scaled, y_train)
# Make Predictions
y_pred = svm_model.predict(X_test_scaled)
y_prob = svm_model.predict_proba(X_test_scaled)[:, 1]
# Evaluate the Model
print("--- Confusion Matrix ---")
print(confusion_matrix(y_test, y_pred))
print("\n--- Classification Report ---")
# Focus heavily on Precision and Recall for Class 1 (Pulsars)
print(classification_report(y_test, y_pred))
print(f"ROC-AUC Score: {roc_auc_score(y_test, y_prob):.4f}")

EXPLANATION:
- SVM finds the best boundary (hyperplane) that separates the two classes with the widest margin.
- The RBF kernel lets it draw a curved boundary, and class_weight='balanced' gives more importance to the rare pulsar class.
- The data is split 70% / 30% with stratify=y so both sets keep the same class ratio.
- The scaler is fitted only on the training data and then applied to the test data.
- Confusion matrix, classification report and ROC-AUC score show how well the rare class is found.


6. RANDOM FOREST

CODE:
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
df = pd.read_csv('C:/Users/Admin/Downloads/pulsar_stars.csv')
df.columns = df.columns.str.strip()
X = df.drop(columns=['target_class'])
y = df['target_class']
# Split into training and testing sets (Stratified to maintain class balance)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
# Feature Scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Initialize and Train Random Forest
# class_weight='balanced' penalizes mistakes on the rare pulsar class heavily.
rf_model = RandomForestClassifier(
    n_estimators=100,
    class_weight='balanced',
    random_state=42,
    n_jobs=-1
)
rf_model.fit(X_train_scaled, y_train)
# Make Predictions
y_pred = rf_model.predict(X_test_scaled)
y_prob = rf_model.predict_proba(X_test_scaled)[:, 1] # Probability estimates for ROC-AUC
# Evaluate the Model
print("--- Confusion Matrix ---")
print(confusion_matrix(y_test, y_pred))
print("\n--- Classification Report ---")
print(classification_report(y_test, y_pred))
print(f"ROC-AUC Score: {roc_auc_score(y_test, y_prob):.4f}")
# Feature Importance
importances = pd.Series(rf_model.feature_importances_, index=X.columns)
print("\n--- Feature Importances ---")
print(importances.sort_values(ascending=False))

EXPLANATION:
- Random forest builds 100 decision trees on random samples of the data and the final class is decided by their majority vote.
- Voting over many trees reduces overfitting compared with a single decision tree.
- class_weight='balanced' penalizes mistakes on the rare pulsar class more.
- The same confusion matrix, classification report and ROC-AUC score are printed for comparison.
- feature_importances_ shows which features contributed most to the prediction.
