Metadata-Version: 2.4
Name: cryptolabs-proxy
Version: 1.4.0
Summary: Unified reverse proxy and fleet management landing page for CryptoLabs products
Author-email: CryptoLabs <support@cryptolabs.co.za>
License: MIT
Project-URL: Homepage, https://github.com/cryptolabsza/cryptolabs-proxy
Project-URL: Documentation, https://cryptolabs.co.za/docs
Project-URL: Repository, https://github.com/cryptolabsza/cryptolabs-proxy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0
Requires-Dist: rich>=13.0
Requires-Dist: questionary>=2.0
Requires-Dist: jinja2>=3.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.28
Dynamic: license-file

# CryptoLabs Proxy

Unified reverse proxy and fleet management landing page for CryptoLabs products. **This is the central entry point** that manages authentication and routing for all CryptoLabs services.

## Architecture

```
                    ┌─────────────────────────────────┐
                    │      CryptoLabs Proxy           │
                    │   (Landing Page & Auth)         │
                    │                                 │
   User → HTTPS ───►│  ┌─────────────────────────┐   │
                    │  │ Unified Authentication  │   │
                    │  └─────────────────────────┘   │
                    │              │                  │
                    └──────────────┼──────────────────┘
                                   │
              ┌────────────────────┼────────────────────┐
              │                    │                    │
              ▼                    ▼                    ▼
        ┌──────────┐        ┌──────────┐        ┌──────────┐
        │  /ipmi/  │        │   /dc/   │        │/grafana/ │
        │   IPMI   │        │    DC    │        │ Grafana  │
        │ Monitor  │        │ Overview │        │          │
        └──────────┘        └──────────┘        └──────────┘
```

## Features

- **cryptolabs-watchtower** - Auto-updates cryptolabs-proxy (primary) and other labeled containers (dc-overview, ipmi-monitor, etc.)
- **Fleet Management Dashboard** - Landing page showing all CryptoLabs services
- **Unified Authentication** - Single login for all services
- **Site Name Branding** - Customize landing page with your datacenter name
- **Auto-Detection** - Automatically detects running services via Docker
- **Health Checks** - Real-time health status for all containers
- **Cross-Promotion** - Promotes other CryptoLabs products when not installed
- **SSL Support** - Let's Encrypt and self-signed certificate support
- **Subpath Routing** - Route to services via `/ipmi/`, `/dc/`, `/grafana/`, etc.

## Supported Services

| Service | Path | Description |
|---------|------|-------------|
| IPMI Monitor | `/ipmi/` | Server hardware monitoring |
| DC Overview | `/dc/` | Datacenter overview dashboard |
| Grafana | `/grafana/` | Metrics visualization |
| Prometheus | `/prometheus/` | Metrics collection (with auth) |

### CryptoLabs' own public sites

This product is the fleet proxy for customer datacenters. It does **not** route
CryptoLabs' own public sites under `cryptolabs.co.za`. Those are served by the
common proxy owned by `cryptolabs-ai-platform`
([services/nginx-configs](https://github.com/cryptolabsza/cryptolabs-ai-platform/tree/dev/services/nginx-configs),
deployed by its Synchronized Platform Deployment).
Add or change public-site routing there, not here.

## Quick Start

The easiest way to deploy is through **DC Overview** or **IPMI Monitor** quickstart, which automatically sets up cryptolabs-proxy:

### Option 1: Deploy with DC Overview (Full Monitoring Stack)

```bash
# Install from dev branch
pip install git+https://github.com/cryptolabsza/dc-overview.git@dev --break-system-packages

# Run quickstart with config file
sudo dc-overview quickstart -c /path/to/config.yaml -y
```

### Option 2: Deploy with IPMI Monitor (IPMI/BMC Only)

```bash
# Install from dev branch
pip install git+https://github.com/cryptolabsza/ipmi-monitor.git@dev --break-system-packages

# Run quickstart with config file
sudo ipmi-monitor quickstart -c /path/to/config.yaml -y
```

### Option 3: Standalone Installation

```bash
apt install pipx -y && pipx ensurepath
source ~/.bashrc
pipx install cryptolabs-proxy
sudo cryptolabs-proxy setup
```

## Authentication Configuration

Authentication credentials are configured via environment variables or through the quickstart config file:

| Variable | Description | Default |
|----------|-------------|---------|
| `FLEET_ADMIN_USER` | Admin username | `admin` |
| `FLEET_ADMIN_PASS` | Admin password | **Required** |
| `SITE_NAME` | Site name for landing page branding | `DC Overview` |
| `AUTH_SECRET_KEY` | Token signing key | Auto-generated |
| `AUTH_DATA_DIR` | Auth data directory | `/data/auth` |

**Important:** You must set `FLEET_ADMIN_PASS` - there is no default password.

When using the quickstart config file, set these in your YAML:

```yaml
site_name: My Datacenter       # Appears in landing page title
fleet_admin_user: admin
fleet_admin_pass: YOUR_ADMIN_PASSWORD
```

## Manual Docker Deployment

```bash
docker run -d \
  --name cryptolabs-proxy \
  -p 80:80 -p 443:443 \
  -e FLEET_ADMIN_USER=admin \
  -e FLEET_ADMIN_PASS=YOUR_ADMIN_PASSWORD \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v cryptolabs_ssl:/etc/nginx/ssl \
  -v cryptolabs_auth:/data/auth \
  ghcr.io/cryptolabsza/cryptolabs-proxy:latest
```

## Configuration

Config files are stored in `/etc/cryptolabs-proxy/`:

```
/etc/cryptolabs-proxy/
├── nginx.conf          # Nginx configuration
├── services.yaml       # Registered services
└── ssl/                # SSL certificates
```

The image ships a bootstrap Nginx configuration. Service lifecycle commands
regenerate the configuration above and require it to be mounted at
`/etc/nginx/nginx.conf` in the running proxy container. This is how optional
Vast Price Manager routing is enabled and removed; the CLI refuses to claim a
route change when the container is still using the bundled configuration.
The bootstrap configuration intentionally contains no Vast Price Manager route.
Before an installer enables or disables VPM, it must first create the registry
configuration, render `/etc/cryptolabs-proxy/nginx.conf`, bind-mount that file
as `/etc/nginx/nginx.conf`, and verify the proxy is using it. The subsequent
`cryptolabs-proxy register` or `unregister` command then validates and reloads
that mounted configuration atomically with the registry update.

### Existing custom proxy configuration

Do not use the generic CLI alone to migrate a proxy that already runs a custom
unmounted `nginx.conf`: it regenerates the full configuration. Use the reviewed
`scripts/vpm-custom-config-migrate.py` helper first. Its read-only `plan` mode
derives a migration ID from the source container and config hash. Its explicit
`apply` stores root-only backups, mounts a byte-preserved custom baseline with
only the canonical managed blocks added, and can perform a proxy-only
`rollback`. Custom-config mode then permits only normal VPM
`register`/`unregister`; it rejects changes to every other service. Its managed
Fleet auth deny block remains installed in every HTTP server block after VPM is unregistered, so the
always-running internal VPM session and reauthentication endpoints cannot fall
through a preserved public <code>/auth/</code> route. Rollback restores the
original stored baseline exactly.

Supported DC monitoring adoption uses the private durable
[controller adoption guard](docs/controller-adoption-guard.md), including normal
updater and proxy-adoption exclusion during crash recovery.

### Vast Price Manager Fleet sign-in

### Vast Price Manager prerequisite

Before a new Vast Price Manager route can be added, the local Vast.ai exporter
must be running and report at least one connected account. Complete setup at
`/vastai/`. Existing VPM installations remain available for maintenance if the
exporter later loses connectivity; they are never automatically disabled.

The proxy checks the exporter from inside its container using its internal
management token, which must be provisioned by the exporter setup. It records
only whether setup is ready and the count of connected accounts; API keys,
account names, balances, and tokens are not returned.

When Vast Price Manager runs in Fleet mode, it uses the existing Fleet
`fleet_session` cookie and does not create a second VPM account or login. VPM
introspects the current Fleet session on the Docker network at
`http://cryptolabs-proxy:8081/auth/vast-price-manager/session`; the public
Nginx configuration explicitly returns `404` for that endpoint and its
reauthentication companion, including their trailing-slash variants.

The internal session response is limited to the current enabled Fleet admin's
username, role, and two purpose-separated HMAC values derived from the signed
cookie: a subject and a CSRF token. It never returns the cookie, password,
password hash, or signing key. The proxy rereads the user record for every
session, reauthentication, and VPM proxy authorization check, so a disabled,
deleted, demoted, or password-change-required account loses VPM access
immediately. Sensitive VPM confirmation calls
`POST /auth/vast-price-manager/reauth` internally with the Fleet password and
the returned CSRF token; it uses the normal Fleet password throttle.

User authentication data is stored in `/data/auth/`:

```
/data/auth/
├── users.json          # User database (hashed passwords)
└── sessions/           # Active sessions
```

## API Endpoints

| Endpoint | Description |
|----------|-------------|
| `GET /` | Fleet management landing page |
| `GET /api/health` | Health status of all services |
| `GET /api/services` | List of registered services |
| `POST /auth/login` | Authentication endpoint |
| `POST /auth/logout` | Logout endpoint |

## DC Watchdog Integration

Fleet Management provides seamless integration with DC Watchdog for uptime monitoring.

### How It Works

When a user clicks "Enable DC Watchdog" in the Fleet Management UI:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│  First-Time Setup (No API Key)                                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. User clicks "Enable DC Watchdog"                                        │
│     └── Fleet checks /data/auth/watchdog_api_key → empty                   │
│                                                                              │
│  2. Redirect to WordPress signup:                                            │
│     https://cryptolabs.co.za/dc-watchdog-signup/                            │
│       ?redirect_uri=https://your-fleet.local/auth/watchdog/callback         │
│       &source=fleet_management                                               │
│                                                                              │
│  3. User logs in / creates account on WordPress                              │
│     └── Clicks "Start Free Trial"                                           │
│     └── WordPress generates API key (sk-ipmi-xxx)                           │
│                                                                              │
│  4. WordPress redirects back with API key:                                   │
│     https://your-fleet.local/auth/watchdog/callback?api_key=sk-ipmi-xxx     │
│                                                                              │
│  5. Fleet saves API key to /data/auth/watchdog_api_key                      │
│                                                                              │
│  6. Auto-SSO to DC Watchdog dashboard (no manual login needed!)             │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────┐
│  Returning User (API Key Already Saved)                                     │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. User clicks "DC Watchdog" → API key loaded from storage                 │
│  2. Fleet generates signed SSO token (using API key as secret)              │
│  3. Redirect to watchdog.cryptolabs.co.za/auth/sso                          │
│  4. Instant dashboard access (no WordPress redirect!)                       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘
```

### DC Watchdog Endpoints

| Endpoint | Description |
|----------|-------------|
| `GET /auth/watchdog/sso` | Generate SSO URL and redirect to DC Watchdog |
| `GET /auth/watchdog/sso-url` | Get SSO URL as JSON (for JavaScript) |
| `GET /auth/watchdog/callback` | OAuth-style callback from WordPress signup |
| `GET /auth/watchdog/status` | Check DC Watchdog configuration status |
| `POST /auth/watchdog/deploy-agents` | Deploy agents to all servers via dc-overview |

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `WATCHDOG_API_KEY` | Pre-configured API key (optional, requires SSO verification) | (none) |
| `WATCHDOG_URL` | DC Watchdog server URL | `https://watchdog.cryptolabs.co.za` |

### First-Time SSO Requirement

Even if `WATCHDOG_API_KEY` is provided via environment variable (e.g., from dc-overview quickstart), users must complete the SSO flow at least once to verify their account. This is enforced by:

1. **Persistent verification flag** (`/data/auth/watchdog_verified`) - Only set after SSO completion
2. **Status endpoint checks** - Returns `not_configured` until verification is complete
3. **UI enforcement** - Shows "Link Account" button until verified

This ensures users explicitly authorize DC Watchdog integration, even when API keys are pre-provisioned.

### Agent Deployment

Once DC Watchdog is enabled, you can deploy agents to all your servers:

1. Click "Deploy Agents" in the Fleet Management UI
2. Agents are installed via SSH to each configured server
3. Agents send heartbeats every 30 seconds to DC Watchdog
4. If a server stops responding, you get alerts via email, Telegram, push, or app

## Development Status

### Implemented Features ✓

| Feature | Status |
|---------|--------|
| Fleet Management landing page | ✓ Complete |
| Unified authentication | ✓ Complete |
| DC Watchdog SSO integration | ✓ Complete |
| First-time SSO verification enforcement | ✓ Complete |
| "Cloud Service" label for DC Watchdog | ✓ Complete |
| Duplicate card prevention (System Updates) | ✓ Complete |
| `key_invalid` state with "Re-link Account" prompt | ✓ Complete |
| Agent deployment status display | ✓ Complete |

### Pending Features (In Development)

| Feature | Status |
|---------|--------|
| Token renewal UI feedback | ⏳ Future |
| Subscription expiry warning banner | ⏳ Future |

## Related Projects

| Project | Description |
|---------|-------------|
| [DC Overview](https://github.com/cryptolabsza/dc-overview) | Full datacenter monitoring with GPU metrics, Prometheus & Grafana |
| [IPMI Monitor](https://github.com/cryptolabsza/ipmi-monitor) | IPMI/BMC hardware monitoring, SEL logs, ECC tracking |
| [DC Exporter](https://github.com/cryptolabsza/dc-exporter-releases) | Standalone GPU metrics exporter for Prometheus |
| [DC Watchdog](https://github.com/cryptolabsza/dc-watchdog) | External uptime monitoring with multi-channel alerts |

## Changelog

### v1.1.1 (Feb 2026) - DC Watchdog Integration

**SSO & Verification:**
- First-time SSO verification enforcement (even with pre-configured API key)
- Persistent verification flag (`/data/auth/watchdog_verified`)
- "Link Account" button until SSO is completed

**UI Improvements:**
- "Cloud Service" label for CryptoLabs-hosted services
- Fixed duplicate DC Watchdog cards in System Updates
- `key_invalid` state displays "Re-link Account" when API key expires
- `agents_installed` state shows installed agents waiting for heartbeats

**Backend:**
- Prioritized file-based API key storage over environment variables
- Added key validation against WordPress API
- Propagated `keyError` state to frontend for proper UI handling

### v1.0.0 (Jan 2026) - Initial Release

- Fleet Management landing page
- Unified authentication for all services
- DC Watchdog SSO flow
- Agent deployment via dc-overview

## License

MIT License - See [LICENSE](LICENSE) for details.

## Links

- [CryptoLabs](https://cryptolabs.co.za)
- [Documentation](https://cryptolabs.co.za/dc-monitoring/)

## NetBox environment link

NetBox starts in **Available Products** on the Fleet dashboard. An administrator
can select **Set up NetBox**, enter that site's environment URL, and select
**Save NetBox**. The card then moves to **Services**, where **Open NetBox** opens
the saved environment in a new tab. NetBox keeps its own login and permissions.

Use **Configure** to edit the destination. **Remove NetBox**, or saving an empty
URL, returns the card to Available Products. Each installation keeps its own
link in the existing persistent Fleet authentication data volume. Ordinary
Fleet users can open the link; only administrators can change it.

URLs must be absolute HTTP or HTTPS links without embedded credentials. Use
HTTPS for NetBox Cloud. The card indicates a configured external service; it
does not poll NetBox or claim that the environment is healthy.

For API configuration, authenticate to Fleet and fetch `/auth/api/netbox`.
Administrators receive a session-bound `csrf_token` with the public link
metadata. POST JSON containing `netbox_url` to `/auth/api/settings`, with that
token in the `X-CSRF-Token` header. An empty string clears the link; omitting
the field preserves it. Invalid URLs or failed saves preserve existing settings.

SSO is a separate integration. A NetBox environment link does not establish
NetBox authentication or grant inventory permissions.
