================================================================================
  Question CQ5: MQTT Environmental Telemetry & Linear Regression Predictor [30 Marks]
================================================================================

FULL QUESTION:
CAE 2 IoT Practical Exam - Combined Question:
CQ5. MQTT-Based Environmental Telemetry and Multiple Linear Regression Temperature Prediction [30 Marks]
Develop an edge-to-cloud environmental monitoring and temperature forecasting system using MQTT communication and Multiple Linear Regression.
Input Parameters: PM2.5, PM10, Atmospheric Pressure (PRES), Dew Point (DEWP), and Wind Speed (WSPM).
Historical Dataset: Beijing Multi-Site Air Quality dataset (PRSA_Data_Aotizhongxin_20130301-20170228.csv).
Target Variable: Ambient Temperature (TEMP in °C).
The system should:
- Implement an MQTT Publisher that streams environmental telemetry in JSON format to a public broker (HiveMQ or Mosquitto). [6 Marks]
- Implement an MQTT Subscriber using paho-mqtt that receives and parses incoming meteorological telemetry packets in real time. [6 Marks]
- Train a Multiple Linear Regression model on historical environmental dataset to predict Temperature (TEMP) from PM2.5, PM10, PRES, DEWP, and WSPM. Compute R2, MAE, and RMSE metrics. [5 Marks]
- Perform real-time regression inference on every incoming MQTT payload and predict the temperature. [5 Marks]
- Develop a Streamlit web dashboard displaying live environmental gauges, predicted temperature vs actual, and regression evaluation curves. [5 Marks]
- Display predicted temperature and residual metrics on the dashboard with interactive manual parameter tuning. [3 Marks]
Flow: Environmental Telemetry -> MQTT Broker (HiveMQ:1883) -> MQTT Subscriber -> Linear Regression Model -> Streamlit Dashboard.

SHORT QUESTION:
MQTT Environmental Publisher & Subscriber with Multiple Linear Regression Temperature Prediction and Streamlit Dashboard [30 Marks].

================================================================================
README & SETUP INSTRUCTIONS:
================================================================================
# Question CQ5: MQTT Environmental Telemetry & Linear Regression Predictor

## Problem Statement [30 Marks]
Develop an MQTT publish/subscribe telemetry stream and apply a Multiple Linear Regression model to forecast temperature based on PM2.5, PM10, PRES, DEWP, and WSPM.

- **Broker**: `broker.hivemq.com` (Port 1883)
- **Topic**: `iot_lab/environment/telemetry`
- **Dataset**: `PRSA_Data_Aotizhongxin_20130301-20170228.csv` (or built-in robust fallback)
- **Target**: `TEMP` (Ambient Temperature in °C)
- **Model**: Multiple Linear Regression (`scikit-learn`)

---

## Architectural Flow
```text
Environmental Telemetry ──> MQTT Broker (HiveMQ:1883) ──> MQTT Subscriber ──> Linear Regression ──> Streamlit UI
```

---

## Files in this Package
- `publisher.py`: Generates and publishes periodic atmospheric telemetry to the MQTT broker.
- `subscriber.py`: CLI Paho-MQTT subscriber that trains Multiple Linear Regression on PRSA data and predicts incoming telemetry.
- `app.py`: Streamlit interactive dashboard displaying predicted temperature gauges and regression metrics.

---

## How to Run

### Step 1: Install Dependencies
```bash
pip install pandas scikit-learn streamlit paho-mqtt matplotlib
```

### Step 2: Start the MQTT Subscriber OR Streamlit Dashboard
Open Terminal 1:
```bash
python subscriber.py
```

### Step 3: Start the MQTT Publisher
Open Terminal 2:
```bash
python publisher.py
```

### Step 4: Run Streamlit Web Dashboard (Alternative Option)
Open Terminal 3:
```bash
streamlit run app.py
```

================================================================================
SOURCE CODE FILES:
================================================================================

1. publisher.py
--------------------------------------------------------------------------------
import time
import json
import random
import paho.mqtt.client as mqtt

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/environment/telemetry"

def create_client():
    try:
        client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
    except AttributeError:
        client = mqtt.Client()
    return client

def main():
    client = create_client()
    print("=" * 60)
    print("  CQ5: MQTT Environmental Telemetry Publisher")
    print("=" * 60)
    print(f"Connecting to Broker: {BROKER}:{PORT}")

    try:
        client.connect(BROKER, PORT, keepalive=60)
        client.loop_start()
        print(f"Publishing environmental telemetry to: {TOPIC}\n")

        cycle = 0
        while True:
            cycle += 1
            pres = round(random.uniform(995.0, 1025.0), 1)
            dewp = round(random.uniform(5.0, 24.0), 1)
            pm25 = round(random.uniform(15.0, 110.0), 1)
            pm10 = round(random.uniform(30.0, 180.0), 1)
            wspm = round(random.uniform(0.8, 6.0), 1)

            actual_temp = round(0.75 * dewp - 0.05 * (pres - 1010) + 15 + random.uniform(-1.5, 1.5), 1)

            payload = {
                "PM2.5": pm25,
                "PM10": pm10,
                "PRES": pres,
                "DEWP": dewp,
                "WSPM": wspm,
                "actual_temp": actual_temp,
                "timestamp": time.strftime("%Y-%m-%d %H:%M:%S")
            }

            msg = json.dumps(payload)
            client.publish(TOPIC, msg)
            print(f"[MQTT Pub] Packet #{cycle} -> {msg}")
            time.sleep(3)

    except KeyboardInterrupt:
        print("\n[Publisher] Disconnecting...")
        client.loop_stop()
        client.disconnect()

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------

2. subscriber.py
--------------------------------------------------------------------------------
import os
import json
import time
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error
import paho.mqtt.client as mqtt

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/environment/telemetry"
FEATURES = ["PM2.5", "PM10", "PRES", "DEWP", "WSPM"]
TARGET = "TEMP"

def load_or_create_dataset():
    possible_paths = [
        "PRSA_Data_Aotizhongxin_20130301-20170228.csv",
        os.path.join(os.path.dirname(__file__), "PRSA_Data_Aotizhongxin_20130301-20170228.csv"),
        os.path.join("..", "PRSA_Data_Aotizhongxin_20130301-20170228.csv")
    ]
    for p in possible_paths:
        if os.path.exists(p):
            print(f"[Dataset] Loading PRSA dataset from {p}...")
            df = pd.read_csv(p)
            if all(c in df.columns for c in FEATURES + [TARGET]):
                return df[FEATURES + [TARGET]].dropna()

    print("[Dataset] PRSA dataset not found. Generating synthetic dataset...")
    np.random.seed(42)
    n = 2000
    pm25 = np.random.uniform(10, 150, n)
    pm10 = pm25 * np.random.uniform(1.2, 1.8, n)
    pres = np.random.uniform(995, 1030, n)
    dewp = np.random.uniform(-5, 26, n)
    wspm = np.random.uniform(0.5, 7.0, n)
    temp = 0.75 * dewp - 0.05 * (pres - 1010) - 0.02 * pm25 + 0.1 * wspm + 15 + np.random.normal(0, 2.5, n)
    return pd.DataFrame({"PM2.5": pm25, "PM10": pm10, "PRES": pres, "DEWP": dewp, "WSPM": wspm, "TEMP": temp})

print("Fitting Multiple Linear Regression on historical environmental records...")
df = load_or_create_dataset()
X_train, X_test, y_train, y_test = train_test_split(df[FEATURES], df[TARGET], test_size=0.20, random_state=42)
model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))

print("-" * 55)
print(f"  Model R2 Score : {r2:.4f}")
print(f"  Model MAE      : {mae:.2f} °C")
print(f"  Model RMSE     : {rmse:.2f} °C")
print("-" * 55)

def on_message(client, userdata, message):
    try:
        data = json.loads(message.payload.decode("utf-8"))
        features = [[data["PM2.5"], data["PM10"], data["PRES"], data["DEWP"], data["WSPM"]]]
        pred_temp = model.predict(features)[0]

        print(f"\n[MQTT Telemetry] PRES={data['PRES']}hPa | DEWP={data['DEWP']}°C | WSPM={data['WSPM']}m/s")
        print(f"  ──> PREDICTED TEMP : {pred_temp:.2f} °C")
        if "actual_temp" in data:
            print(f"  ──> ACTUAL TEMP    : {data['actual_temp']:.2f} °C (Err: {abs(pred_temp - data['actual_temp']):.2f} °C)")

    except Exception as e:
        print(f"[Error] {e}")

def main():
    print("=" * 60)
    print("  CQ5: MQTT Subscriber & Linear Regression Pipeline")
    print("=" * 60)

    try:
        client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
    except AttributeError:
        client = mqtt.Client()

    client.on_message = on_message
    print(f"Subscribing to {TOPIC} on {BROKER}...")
    client.connect(BROKER, PORT, keepalive=60)
    client.subscribe(TOPIC)

    try:
        client.loop_forever()
    except KeyboardInterrupt:
        print("\n[Subscriber] Disconnecting...")
        client.disconnect()

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------

3. app.py
--------------------------------------------------------------------------------
import os
import json
import time
import numpy as np
import pandas as pd
import streamlit as st
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error
import paho.mqtt.client as mqtt

st.set_page_config(page_title="CQ5: MQTT Temperature Predictor", layout="wide", page_icon="🌡️")

BROKER = "broker.hivemq.com"
PORT = 1883
TOPIC = "iot_lab/environment/telemetry"
FEATURES = ["PM2.5", "PM10", "PRES", "DEWP", "WSPM"]
TARGET = "TEMP"

@st.cache_resource
def get_trained_model():
    possible_paths = [
        "PRSA_Data_Aotizhongxin_20130301-20170228.csv",
        os.path.join(os.path.dirname(__file__), "PRSA_Data_Aotizhongxin_20130301-20170228.csv"),
        os.path.join("..", "PRSA_Data_Aotizhongxin_20130301-20170228.csv")
    ]
    df = None
    for p in possible_paths:
        if os.path.exists(p):
            df = pd.read_csv(p)
            if all(c in df.columns for c in FEATURES + [TARGET]):
                df = df[FEATURES + [TARGET]].dropna()
                break

    if df is None:
        np.random.seed(42)
        n = 2000
        pm25 = np.random.uniform(10, 150, n)
        pm10 = pm25 * np.random.uniform(1.2, 1.8, n)
        pres = np.random.uniform(995, 1030, n)
        dewp = np.random.uniform(-5, 26, n)
        wspm = np.random.uniform(0.5, 7.0, n)
        temp = 0.75 * dewp - 0.05 * (pres - 1010) - 0.02 * pm25 + 0.1 * wspm + 15 + np.random.normal(0, 2.5, n)
        df = pd.DataFrame({"PM2.5": pm25, "PM10": pm10, "PRES": pres, "DEWP": dewp, "WSPM": wspm, "TEMP": temp})

    X_train, X_test, y_train, y_test = train_test_split(df[FEATURES], df[TARGET], test_size=0.20, random_state=42)
    model = LinearRegression()
    model.fit(X_train, y_train)

    y_pred = model.predict(X_test)
    metrics = {
        "r2": r2_score(y_test, y_pred),
        "mae": mean_absolute_error(y_test, y_pred),
        "rmse": np.sqrt(mean_squared_error(y_test, y_pred))
    }
    return model, metrics

def main():
    st.title("🌡️ MQTT Environmental Telemetry & Temperature Predictor")
    st.markdown("**CAE 2 Combined Question 5**: MQTT Protocol + Linear Regression (`paho-mqtt` + `scikit-learn` + `Streamlit`)")
    st.markdown("---")

    model, metrics = get_trained_model()

    with st.sidebar:
        st.header("⚙️ MQTT Broker")
        st.write(f"**Broker:** `{BROKER}`")
        st.write(f"**Topic:** `{TOPIC}`")
        st.markdown("---")
        st.header("📊 Model Metrics")
        st.metric("R² Score", f"{metrics['r2']:.4f}")
        st.metric("Mean Absolute Error (MAE)", f"{metrics['mae']:.2f} °C")
        st.metric("Root Mean Squared Error", f"{metrics['rmse']:.2f} °C")

    if "latest_env" not in st.session_state:
        st.session_state.latest_env = {
            "PM2.5": 42.0, "PM10": 85.0, "PRES": 1013.0, "DEWP": 17.5, "WSPM": 2.5, "actual_temp": 26.8, "timestamp": time.strftime("%H:%M:%S")
        }

    col_btn, col_info = st.columns([1, 3])
    with col_btn:
        fetch_clicked = st.button("📡 Fetch Latest MQTT Packet", type="primary", use_container_width=True)
    with col_info:
        st.info(f"Make sure `python publisher.py` is running to send continuous messages to `{TOPIC}`.")

    if fetch_clicked:
        msgs = []
        def on_msg(client, userdata, msg):
            msgs.append(msg.payload.decode("utf-8"))

        try:
            temp_cli = mqtt.Client()
            temp_cli.on_message = on_msg
            temp_cli.connect(BROKER, PORT, keepalive=10)
            temp_cli.subscribe(TOPIC)
            temp_cli.loop_start()
            t0 = time.time()
            while time.time() - t0 < 3.5 and len(msgs) == 0:
                time.sleep(0.2)
            temp_cli.loop_stop()
            temp_cli.disconnect()

            if msgs:
                st.session_state.latest_env = json.loads(msgs[-1])
                st.success("Fetched latest environmental telemetry via MQTT!")
            else:
                st.warning("No new MQTT message in 3.5s. Displaying cached readings.")
        except Exception as e:
            st.error(f"MQTT Error: {e}")

    data = st.session_state.latest_env

    st.subheader("1. Incoming Atmospheric Telemetry")
    p1, p2, p3, p4, p5 = st.columns(5)
    p1.metric("PM2.5", f"{data['PM2.5']} µg/m³")
    p2.metric("PM10", f"{data['PM10']} µg/m³")
    p3.metric("Pressure", f"{data['PRES']} hPa")
    p4.metric("Dew Point", f"{data['DEWP']} °C")
    p5.metric("Wind Speed", f"{data['WSPM']} m/s")

    st.subheader("2. Real-Time Regression Inference")
    in_vec = [[data["PM2.5"], data["PM10"], data["PRES"], data["DEWP"], data["WSPM"]]]
    pred_temp = model.predict(in_vec)[0]

    r1, r2, r3 = st.columns(3)
    r1.metric("Predicted Temperature", f"{pred_temp:.2f} °C")
    act = data.get("actual_temp", 26.0)
    r2.metric("Actual Sensor Temp", f"{act:.2f} °C")
    r3.metric("Residual Error", f"{abs(pred_temp - act):.2f} °C")

    st.markdown("---")

    st.subheader("3. Environmental Parameter Simulator")
    s1, s2, s3, s4, s5 = st.columns(5)
    sim_pm25 = s1.slider("Simulate PM2.5", 0.0, 300.0, float(data["PM2.5"]))
    sim_pm10 = s2.slider("Simulate PM10", 0.0, 400.0, float(data["PM10"]))
    sim_pres = s3.slider("Simulate Pressure", 970.0, 1040.0, float(data["PRES"]))
    sim_dewp = s4.slider("Simulate Dew Point", -10.0, 30.0, float(data["DEWP"]))
    sim_wspm = s5.slider("Simulate Wind Speed", 0.0, 15.0, float(data["WSPM"]))

    sim_res = model.predict([[sim_pm25, sim_pm10, sim_pres, sim_dewp, sim_wspm]])[0]
    st.info(f"Simulated Predicted Temperature: **{sim_res:.2f} °C**")

if __name__ == "__main__":
    main()
--------------------------------------------------------------------------------
