================================================================================
  Question CQ4: MQTT Air Quality Streaming & Random Forest Classification [30 Marks]
================================================================================

FULL QUESTION:
CAE 2 IoT Practical Exam - Combined Question:
CQ4. MQTT-Based Air Quality Telemetry and Random Forest Classification [30 Marks]
Develop an industrial air quality monitoring and hazard classification pipeline using MQTT communication and Random Forest model.
Input Parameters: PM2.5, PM10, NO2, SO2, and O3.
Historical Dataset: India National Air Quality dataset (city_day.csv).
Target Variable: AQI_Bucket (Good, Satisfactory, Moderate, Poor, Very Poor, Severe).
The system should:
- Implement an MQTT Publisher that streams simulated/edge particulate and chemical pollutant telemetry in JSON format to a public broker (HiveMQ or Mosquitto). [6 Marks]
- Implement an MQTT Subscriber using paho-mqtt that receives and parses incoming air quality packets in real time. [6 Marks]
- Train a Random Forest Classification model using historical city_day.csv dataset to categorize air quality into standard hazard buckets. Handle missing values via median imputation. [5 Marks]
- Perform real-time inference on every incoming MQTT payload and determine the AQI bucket. [5 Marks]
- Develop a Streamlit web dashboard displaying live pollutant gauges, color-coded AQI hazard status, and Random Forest feature importance. [5 Marks]
- Provide an interactive simulation panel allowing users to test hypothetical pollutant spikes. [3 Marks]
Flow: Air Quality Telemetry -> MQTT Broker (HiveMQ:1883) -> MQTT Subscriber -> Random Forest Model -> Streamlit Dashboard.

SHORT QUESTION:
MQTT Air Quality Publisher & Subscriber with Random Forest Classification and Streamlit Dashboard [30 Marks].

================================================================================
README & SETUP INSTRUCTIONS:
================================================================================
# Question CQ4: MQTT-Based Air Quality Random Forest Classifier

## Problem Statement [30 Marks]
Develop an MQTT publish/subscribe telemetry stream and apply a Random Forest Classifier to categorize air quality based on PM2.5, PM10, NO2, SO2, and O3.

- **Broker**: `broker.hivemq.com` (Port 1883)
- **Topic**: `iot_lab/airquality/telemetry`
- **Dataset**: `city_day.csv` (or synthetic fallback)
- **Target**: `AQI_Bucket` (Good, Satisfactory, Moderate, Poor, Very Poor, Severe)
- **Model**: Random Forest Classifier (`scikit-learn`, `n_estimators=100`)

---

## Architectural Flow
```text
Air Quality Telemetry ──> MQTT Broker (HiveMQ:1883) ──> MQTT Subscriber ──> Random Forest ──> Streamlit UI
```

---

## Files in this Package
- `publisher.py`: Generates and publishes periodic air quality telemetry to the MQTT broker.
- `subscriber.py`: CLI Paho-MQTT subscriber that trains Random Forest on `city_day.csv` and predicts incoming telemetry.
- `app.py`: Streamlit interactive dashboard displaying color-coded AQI hazard alerts and feature importance charts.

---

## How to Run

### Step 1: Install Dependencies
```bash
pip install pandas scikit-learn streamlit paho-mqtt matplotlib
```

### Step 2: Start the MQTT Subscriber OR Streamlit Dashboard
Open Terminal 1:
```bash
python subscriber.py
```

### Step 3: Start the MQTT Publisher
Open Terminal 2:
```bash
python publisher.py
```

### Step 4: Run Streamlit Web Dashboard (Alternative Option)
Open Terminal 3:
```bash
streamlit run app.py
```

================================================================================
SOURCE CODE FILES:
================================================================================

1. publisher.py
--------------------------------------------------------------------------------
import time
import json
import random
import paho.mqtt.client as mqtt

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/airquality/telemetry"

def create_client():
    try:
        client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
    except AttributeError:
        client = mqtt.Client()
    return client

def main():
    client = create_client()
    print("=" * 60)
    print("  CQ4: MQTT Air Quality Telemetry Publisher")
    print("=" * 60)
    print(f"Connecting to Broker: {BROKER}:{PORT}")

    try:
        client.connect(BROKER, PORT, keepalive=60)
        client.loop_start()
        print(f"Publishing air quality data to: {TOPIC}\n")

        cycle = 0
        while True:
            cycle += 1

            mode = (cycle // 4) % 3
            if mode == 0:
                pm25 = round(random.uniform(10.0, 35.0), 1)
                pm10 = round(random.uniform(20.0, 50.0), 1)
                no2 = round(random.uniform(5.0, 20.0), 1)
                so2 = round(random.uniform(4.0, 15.0), 1)
                o3 = round(random.uniform(10.0, 30.0), 1)
            elif mode == 1:
                pm25 = round(random.uniform(55.0, 90.0), 1)
                pm10 = round(random.uniform(90.0, 150.0), 1)
                no2 = round(random.uniform(35.0, 65.0), 1)
                so2 = round(random.uniform(20.0, 45.0), 1)
                o3 = round(random.uniform(35.0, 70.0), 1)
            else:
                pm25 = round(random.uniform(150.0, 320.0), 1)
                pm10 = round(random.uniform(250.0, 480.0), 1)
                no2 = round(random.uniform(80.0, 140.0), 1)
                so2 = round(random.uniform(50.0, 95.0), 1)
                o3 = round(random.uniform(80.0, 150.0), 1)

            payload = {
                "PM2.5": pm25,
                "PM10": pm10,
                "NO2": no2,
                "SO2": so2,
                "O3": o3,
                "timestamp": time.strftime("%Y-%m-%d %H:%M:%S")
            }

            msg = json.dumps(payload)
            client.publish(TOPIC, msg)
            print(f"[MQTT Pub] Packet #{cycle} -> {msg}")
            time.sleep(3)

    except KeyboardInterrupt:
        print("\n[Publisher] Disconnecting...")
        client.loop_stop()
        client.disconnect()

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------

2. subscriber.py
--------------------------------------------------------------------------------
import os
import json
import time
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
import paho.mqtt.client as mqtt

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/airquality/telemetry"
FEATURES = ["PM2.5", "PM10", "NO2", "SO2", "O3"]
TARGET = "AQI_Bucket"
CLASSES = ["Good", "Satisfactory", "Moderate", "Poor", "Very Poor", "Severe"]

def load_or_create_dataset():
    possible_paths = [
        "city_day.csv",
        os.path.join(os.path.dirname(__file__), "city_day.csv"),
        os.path.join("..", "city_day.csv")
    ]
    for p in possible_paths:
        if os.path.exists(p):
            print(f"[Dataset] Loading India Air Quality dataset from {p}...")
            df = pd.read_csv(p)
            if all(col in df.columns for col in FEATURES + [TARGET]):
                df = df[FEATURES + [TARGET]].dropna(subset=[TARGET])

                for col in FEATURES:
                    df[col] = df[col].fillna(df[col].median())
                return df

    print("[Dataset] city_day.csv not found. Synthesizing air quality dataset...")
    np.random.seed(42)
    n = 2400
    buckets = []
    pm25s, pm10s, no2s, so2s, o3s = [], [], [], [], []

    for _ in range(n):
        b = np.random.choice(CLASSES)
        buckets.append(b)
        if b == "Good":
            pm25s.append(np.random.uniform(5, 30))
            pm10s.append(np.random.uniform(10, 50))
            no2s.append(np.random.uniform(5, 20))
            so2s.append(np.random.uniform(2, 15))
            o3s.append(np.random.uniform(5, 35))
        elif b == "Satisfactory":
            pm25s.append(np.random.uniform(30, 60))
            pm10s.append(np.random.uniform(50, 100))
            no2s.append(np.random.uniform(20, 40))
            so2s.append(np.random.uniform(15, 30))
            o3s.append(np.random.uniform(30, 60))
        elif b == "Moderate":
            pm25s.append(np.random.uniform(60, 90))
            pm10s.append(np.random.uniform(100, 150))
            no2s.append(np.random.uniform(40, 70))
            so2s.append(np.random.uniform(30, 50))
            o3s.append(np.random.uniform(50, 90))
        elif b == "Poor":
            pm25s.append(np.random.uniform(90, 150))
            pm10s.append(np.random.uniform(150, 250))
            no2s.append(np.random.uniform(70, 100))
            so2s.append(np.random.uniform(50, 75))
            o3s.append(np.random.uniform(80, 120))
        elif b == "Very Poor":
            pm25s.append(np.random.uniform(150, 250))
            pm10s.append(np.random.uniform(250, 350))
            no2s.append(np.random.uniform(100, 140))
            so2s.append(np.random.uniform(75, 100))
            o3s.append(np.random.uniform(110, 150))
        else:
            pm25s.append(np.random.uniform(250, 450))
            pm10s.append(np.random.uniform(350, 600))
            no2s.append(np.random.uniform(130, 200))
            so2s.append(np.random.uniform(90, 160))
            o3s.append(np.random.uniform(140, 220))

    return pd.DataFrame({"PM2.5": pm25s, "PM10": pm10s, "NO2": no2s, "SO2": so2s, "O3": o3s, "AQI_Bucket": buckets})

print("Fitting Random Forest Classifier on historical air quality records...")
df = load_or_create_dataset()
X_train, X_test, y_train, y_test = train_test_split(df[FEATURES], df[TARGET], test_size=0.20, random_state=42)
model = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
model.fit(X_train, y_train)

acc = accuracy_score(y_test, model.predict(X_test))
print(f"[Model] Random Forest Accuracy Score: {acc * 100:.2f}%\n")

def on_message(client, userdata, message):
    try:
        data = json.loads(message.payload.decode("utf-8"))
        features = [[data["PM2.5"], data["PM10"], data["NO2"], data["SO2"], data["O3"]]]
        pred_bucket = model.predict(features)[0]

        print(f"\n[MQTT Telemetry] PM2.5={data['PM2.5']} | PM10={data['PM10']} | NO2={data['NO2']} | SO2={data['SO2']} | O3={data['O3']}")
        print(f"  ──> PREDICTED HAZARD BUCKET: [{pred_bucket.upper()}]")

    except Exception as e:
        print(f"[Error] {e}")

def main():
    print("=" * 60)
    print("  CQ4: MQTT Subscriber & Random Forest Pipeline")
    print("=" * 60)

    try:
        client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
    except AttributeError:
        client = mqtt.Client()

    client.on_message = on_message
    print(f"Subscribing to {TOPIC} on {BROKER}...")
    client.connect(BROKER, PORT, keepalive=60)
    client.subscribe(TOPIC)

    try:
        client.loop_forever()
    except KeyboardInterrupt:
        print("\n[Subscriber] Disconnecting...")
        client.disconnect()

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------

3. app.py
--------------------------------------------------------------------------------
import os
import json
import time
import numpy as np
import pandas as pd
import streamlit as st
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import paho.mqtt.client as mqtt

st.set_page_config(page_title="CQ4: MQTT Air Quality Classifier", layout="wide", page_icon="🏭")

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/airquality/telemetry"
FEATURES = ["PM2.5", "PM10", "NO2", "SO2", "O3"]
TARGET = "AQI_Bucket"
CLASSES = ["Good", "Satisfactory", "Moderate", "Poor", "Very Poor", "Severe"]

COLOR_MAP = {
    "Good": "#2ecc71",
    "Satisfactory": "#27ae60",
    "Moderate": "#f39c12",
    "Poor": "#e67e22",
    "Very Poor": "#e74c3c",
    "Severe": "#8e44ad"
}

@st.cache_resource
def get_trained_random_forest():
    possible_paths = [
        "city_day.csv",
        os.path.join(os.path.dirname(__file__), "city_day.csv"),
        os.path.join("..", "city_day.csv")
    ]
    df = None
    for p in possible_paths:
        if os.path.exists(p):
            df = pd.read_csv(p)
            if all(col in df.columns for col in FEATURES + [TARGET]):
                df = df[FEATURES + [TARGET]].dropna(subset=[TARGET])
                for col in FEATURES:
                    df[col] = df[col].fillna(df[col].median())
                break

    if df is None:
        np.random.seed(42)
        n = 2400
        buckets, pm25s, pm10s, no2s, so2s, o3s = [], [], [], [], []
        for _ in range(n):
            b = np.random.choice(CLASSES)
            buckets.append(b)
            if b == "Good":
                pm25s.append(np.random.uniform(5, 30)); pm10s.append(np.random.uniform(10, 50))
                no2s.append(np.random.uniform(5, 20)); so2s.append(np.random.uniform(2, 15)); o3s.append(np.random.uniform(5, 35))
            elif b == "Satisfactory":
                pm25s.append(np.random.uniform(30, 60)); pm10s.append(np.random.uniform(50, 100))
                no2s.append(np.random.uniform(20, 40)); so2s.append(np.random.uniform(15, 30)); o3s.append(np.random.uniform(30, 60))
            elif b == "Moderate":
                pm25s.append(np.random.uniform(60, 90)); pm10s.append(np.random.uniform(100, 150))
                no2s.append(np.random.uniform(40, 70)); so2s.append(np.random.uniform(30, 50)); o3s.append(np.random.uniform(50, 90))
            elif b == "Poor":
                pm25s.append(np.random.uniform(90, 150)); pm10s.append(np.random.uniform(150, 250))
                no2s.append(np.random.uniform(70, 100)); so2s.append(np.random.uniform(50, 75)); o3s.append(np.random.uniform(80, 120))
            elif b == "Very Poor":
                pm25s.append(np.random.uniform(150, 250)); pm10s.append(np.random.uniform(250, 350))
                no2s.append(np.random.uniform(100, 140)); so2s.append(np.random.uniform(75, 100)); o3s.append(np.random.uniform(110, 150))
            else:
                pm25s.append(np.random.uniform(250, 450)); pm10s.append(np.random.uniform(350, 600))
                no2s.append(np.random.uniform(130, 200)); so2s.append(np.random.uniform(90, 160)); o3s.append(np.random.uniform(140, 220))
        df = pd.DataFrame({"PM2.5": pm25s, "PM10": pm10s, "NO2": no2s, "SO2": so2s, "O3": o3s, "AQI_Bucket": buckets})

    X_train, X_test, y_train, y_test = train_test_split(df[FEATURES], df[TARGET], test_size=0.20, random_state=42)
    model = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
    model.fit(X_train, y_train)

    acc = accuracy_score(y_test, model.predict(X_test))
    importances = dict(zip(FEATURES, model.feature_importances_))
    return model, acc, importances

def main():
    st.title("🏭 MQTT Air Quality Stream & Random Forest Classifier")
    st.markdown("**CAE 2 Combined Question 4**: MQTT Protocol + Random Forest (`paho-mqtt` + `scikit-learn` + `Streamlit`)")
    st.markdown("---")

    model, acc, importances = get_trained_random_forest()

    with st.sidebar:
        st.header("⚙️ MQTT Settings")
        st.write(f"**Broker:** `{BROKER}`")
        st.write(f"**Topic:** `{TOPIC}`")
        st.markdown("---")
        st.header("📊 Model Metrics")
        st.metric("Model Accuracy", f"{acc * 100:.2f}%")
        st.subheader("Feature Importance")
        st.bar_chart(importances)

    if "latest_aqi" not in st.session_state:
        st.session_state.latest_aqi = {
            "PM2.5": 72.0, "PM10": 125.0, "NO2": 45.0, "SO2": 28.0, "O3": 54.0, "timestamp": time.strftime("%H:%M:%S")
        }

    col_btn, col_info = st.columns([1, 3])
    with col_btn:
        fetch_clicked = st.button("📡 Fetch Latest MQTT Packet", type="primary", use_container_width=True)
    with col_info:
        st.info(f"Make sure `python publisher.py` is running to send continuous messages to `{TOPIC}`.")

    if fetch_clicked:
        msgs = []
        def on_msg(client, userdata, msg):
            msgs.append(msg.payload.decode("utf-8"))

        try:
            temp_cli = mqtt.Client()
            temp_cli.on_message = on_msg
            temp_cli.connect(BROKER, PORT, keepalive=10)
            temp_cli.subscribe(TOPIC)
            temp_cli.loop_start()
            t0 = time.time()
            while time.time() - t0 < 3.5 and len(msgs) == 0:
                time.sleep(0.2)
            temp_cli.loop_stop()
            temp_cli.disconnect()

            if msgs:
                st.session_state.latest_aqi = json.loads(msgs[-1])
                st.success("Fetched latest air quality telemetry via MQTT!")
            else:
                st.warning("No new MQTT message in 3.5s. Displaying cached data.")
        except Exception as e:
            st.error(f"MQTT Error: {e}")

    data = st.session_state.latest_aqi

    st.subheader("1. Live Pollutant Telemetry")
    p1, p2, p3, p4, p5 = st.columns(5)
    p1.metric("PM2.5", f"{data['PM2.5']} µg/m³")
    p2.metric("PM10", f"{data['PM10']} µg/m³")
    p3.metric("NO₂", f"{data['NO2']} µg/m³")
    p4.metric("SO₂", f"{data['SO2']} µg/m³")
    p5.metric("O₃", f"{data['O3']} µg/m³")

    st.subheader("2. Real-Time AQI Classification")
    in_vec = [[data["PM2.5"], data["PM10"], data["NO2"], data["SO2"], data["O3"]]]
    pred_bucket = model.predict(in_vec)[0]
    color = COLOR_MAP.get(pred_bucket, "#34495e")

    st.markdown(
        f"""
        <div style="background-color: {color}; padding: 20px; border-radius: 10px; color: white; text-align: center;">
            <h2 style="margin: 0;">Predicted AQI Category: {pred_bucket.upper()}</h2>
            <p style="margin: 5px 0 0 0; font-size: 16px;">Telemetry timestamp: {data.get('timestamp', 'N/A')}</p>
        </div>
        """,
        unsafe_allow_html=True
    )

    st.markdown("---")

    st.subheader("3. Hypothetical Pollution Simulator")
    s1, s2, s3, s4, s5 = st.columns(5)
    sim_pm25 = s1.slider("Simulate PM2.5", 0.0, 450.0, float(data["PM2.5"]))
    sim_pm10 = s2.slider("Simulate PM10", 0.0, 600.0, float(data["PM10"]))
    sim_no2 = s3.slider("Simulate NO2", 0.0, 200.0, float(data["NO2"]))
    sim_so2 = s4.slider("Simulate SO2", 0.0, 160.0, float(data["SO2"]))
    sim_o3 = s5.slider("Simulate O3", 0.0, 220.0, float(data["O3"]))

    sim_res = model.predict([[sim_pm25, sim_pm10, sim_no2, sim_so2, sim_o3]])[0]
    st.info(f"Simulated AQI Bucket Prediction: **{sim_res}**")

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------
