Metadata-Version: 2.4
Name: vllm-ipc-cache
Version: 0.2.0
Summary: Zero-copy CUDA/MACA IPC weight pre-load cache for vLLM instant restart
Author: vLLM Community
License-Expression: Apache-2.0
Keywords: vllm,llm,inference,ipc,cache,cuda,zero-copy,metax,benchmark
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: POSIX :: Linux
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Dynamic: license-file

# vLLM IPC Weight Cache (`vllm-ipc-cache`)

[![PyPI version](https://img.shields.io/pypi/v/vllm-ipc-cache.svg)](https://pypi.org/project/vllm-ipc-cache/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python](https://img.shields.io/badge/Python-3.9%2B-green.svg)](https://www.python.org/)

Zero-copy, instant model weight pre-load cache for **vLLM** on CUDA and Heterogeneous GPUs (such as MetaX MACA). 

By decoupling model weight lifecycle from the vLLM engine process, `vllm-ipc-cache` reduces model restart and loading time from **tens of seconds to sub-second (0.47s, 76x speedup)** with **0.0 GiB duplicate memory overhead**.

---

## 🌟 Key Highlights

- **⚡ Sub-Second Weight Attach**: Reduces 27B model loading from 35.8s to **0.47s** (76x faster).
- **🔌 Zero-Intrusive Plugin Architecture**: Packaged as a standard Python wheel using `vllm.general_plugins` entry points. Works out-of-the-box on older or vendor-forked vLLM without modifying a single line of core vLLM code.
- **🛡️ Enterprise Multi-Tenant Security**: Private socket sandbox (`0700` dir / `0600` socket) with Linux kernel `SO_PEERCRED` handshake validation to prevent cross-tenant privilege escalation and handle hijacking.
- **🎯 5-Tuple Fingerprint Guard**: The engine handshake validates model path, tensor parallel size & rank, data type, and vLLM version against the daemon's preloaded config. Any mismatch is rejected with a clear error instead of silently attaching wrong weights.
- **🚀 Ideal for Benchmark Sweeps & High Availability**: Slashes hours of repeated disk I/O in parameter grid searches (concurrency, batch size, block size sweep) down to minutes.

---

## 📊 Benchmark Results

Real-world test on MetaX N300-A GPU (Qwen3.8-27B, Tensor Parallel = 4):

| Metric | Standard Cold Load | With `vllm-ipc-cache` | Improvement |
| :--- | :--- | :--- | :--- |
| **Weight Load Time** | 35.80 s | **0.47 s** | **76.2x Faster** |
| **Duplicate Memory** | N/A | **0.00 GiB** | **Zero VRAM Waste** |
| **50-Round Parameter Sweep** (estimated) | ~30.0 mins (disk I/O) | **~23.5 s** | **98.7% Time Saved** |

> The sweep row is an estimate derived from the measured per-restart saving, not a stopwatch measurement.

---

## 🏗️ Architecture

```
+-------------------------------------------------------------+
|                      GPU Device Memory                      |
|  +-------------------------------------------------------+  |
|  |             Preloaded Weights in VRAM                 |  |
|  +-------------------------------------------------------+  |
|          ^                                     ^            |
|          | (CUDA IPC Export)                   | (Attach)   |
|          |                                     |            |
|   +---------------+                     +----------------+  |
|   | Daemon Worker |                     |  vLLM Engine   |  |
|   +---------------+                     +----------------+  |
|          |                                     |            |
+----------|-------------------------------------|------------+
           |                                     |
           +-----> [Unix Domain Socket] <--------+
                   - 0700/0600 Sandbox
                   - SO_PEERCRED UID Auth
                   - 5-Tuple Fingerprint
```

1. **Daemon Workers** load weights into GPU VRAM once and hold them in memory.
2. For each tensor, a CUDA/MACA IPC memory handle is generated.
3. When the **vLLM Engine** starts with `--load-format ipc_cache`, it connects to the daemon worker via Unix Domain Socket.
4. After UID authentication and fingerprint validation, the engine maps the IPC handles directly into its memory space via zero-copy.

---

## 📦 Installation

```bash
pip install vllm-ipc-cache
```

*Note: `torch` and `vllm` are expected to be pre-installed in your environment (e.g. vendor container image or standard CUDA env).*

---

## 🚀 Quick Start

### Step 1: Start the Weight Cache Daemon

Launch background daemon workers to preload model weights into GPU VRAM:

```bash
# Launch daemon workers (e.g., TP=4 for a 27B model)
python3 -m ipc_cache.daemon \
  --model /path/to/model \
  --tp 4 \
  --master-port 29500
```

Wait until all TP ranks report `READY!`. Weights are now resident in VRAM.

### Step 2: Start vLLM with IPC Cache

Simply pass `--load-format ipc_cache` when launching your vLLM engine or OpenAI API server:

```bash
python3 -m vllm.entrypoints.openai.api_server \
  --model /path/to/model \
  --tensor-parallel-size 4 \
  --load-format ipc_cache \
  --port 8000
```

vLLM will automatically detect the plugin and attach to the cached weights in **< 0.5s**!

---

## 🛠️ Typical Use Cases

1. **Benchmark Parameter Sweep**:
   Sweep `max_num_batched_tokens`, `max_num_seqs`, and `block_size` without repeatedly waiting for multi-gigabyte disk reads on every restart.
2. **High-Availability (HA) Fast Recovery**:
   When an inference worker crashes, the orchestrator restarts the process and restores serving capabilities within a single second.
3. **Model Time-Sharing / Dynamic Switching**:
   Swap between active serving engines with minimal cold-start penalty.

---

## 🔒 Security & Robustness

- **UID Sandboxing**: Daemon sockets are created in `/tmp/vllm_ipc_cache_uid_<UID>` with `0700` permissions. Sockets are restricted to `0600`.
- **Peer Credential Verification**: Before responding to requests, the daemon queries `SO_PEERCRED` on the incoming socket to verify the connecting PID and UID.
- **Fingerprint Verification**: Requests must provide a 5-tuple fingerprint matching the daemon's preloaded configuration:
  - Model Path (realpath-resolved)
  - Tensor Parallel Size & Rank
  - Data Type (`torch_dtype`)
  - vLLM Version

  If any field mismatches, the request is rejected with a clear error message. There is **no** automatic fallback to cold load — the engine fails loudly instead of silently attaching wrong weights.

---

## ⚠️ Limitations & Operational Notes

- **Resident VRAM cost**: The daemon keeps weights in GPU memory at all times — this design trades VRAM for restart latency. Size your `--gpu-memory-utilization` with the daemon's share accounted for (e.g. Qwen3.8-27B @ TP=4 keeps ~15 GB per card resident).
- **Verified scope**:
  - Dense bf16 models (e.g. Qwen3.8-27B): fully verified end-to-end.
  - `compressed-tensors` W8A8 models (e.g. MiniMax-M2.7, DeepSeek-V4-Flash): architecture & format compatibility verified; full-load benchmarks pending.
- **Pipeline parallelism is not supported**: the cache is keyed by tensor-parallel rank only (`pipeline_parallel_size` must be 1).
- **SGLang is not supported**: the loader hooks vLLM's plugin interface only.
- **GPU ordering must match**: sockets are keyed by TP rank, so the daemon and the engine must see GPUs in the same order (identical `CUDA_VISIBLE_DEVICES` mapping).
- **Trust model**: the socket sandbox and `SO_PEERCRED` check protect against *other local users*. Processes running as the same user are within the trust boundary — this matches the upstream vLLM weight-cache threat model.

---

## 📄 License

This project is licensed under the [Apache-2.0 License](LICENSE).
