Metadata-Version: 2.4
Name: compilersutra-perf
Version: 0.2.0
Summary: Alpha CLI for compiler and GPU performance experiments — compile, profile, diff, and export JSON/CSV results
Author-email: Priya Pandey <osc@compilersutra.com>, CompilerSutra <osc@compilersutra.com>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/compilersutra/CompilerSutraPerfTool
Project-URL: Documentation, https://github.com/compilersutra/CompilerSutraPerfTool/blob/main/docs/USAGE.md
Project-URL: Repository, https://github.com/compilersutra/CompilerSutraPerfTool
Project-URL: Issues, https://github.com/compilersutra/CompilerSutraPerfTool/issues
Project-URL: Changelog, https://github.com/compilersutra/CompilerSutraPerfTool/blob/main/CHANGELOG.md
Project-URL: Author, https://www.linkedin.com/in/priya-pandey-54a5a22aa/
Keywords: benchmarking,compiler,gpu,hip,llvm,opencl,performance,profiling,systems,vulkan
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Compilers
Classifier: Topic :: System :: Benchmark
Classifier: Topic :: System :: Hardware
Classifier: Topic :: System :: Systems Administration
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: dev
Requires-Dist: pytest>=9.0.3; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: coverage[toml]>=7.6; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: mypy>=1.13; extra == "dev"
Requires-Dist: bandit>=1.8; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Requires-Dist: urllib3>=2.7; extra == "dev"
Requires-Dist: click>=8.3.3; extra == "dev"
Requires-Dist: cryptography>=50.0.0; extra == "dev"
Requires-Dist: GitPython>=3.1.57; extra == "dev"
Requires-Dist: idna>=3.15; extra == "dev"
Requires-Dist: Pillow>=12.3.0; extra == "dev"
Requires-Dist: Pygments>=2.20.0; extra == "dev"
Requires-Dist: requests>=2.33.0; extra == "dev"
Requires-Dist: tornado>=6.5.7; extra == "dev"
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == "excel"
Provides-Extra: visualize
Requires-Dist: streamlit>=1.33; extra == "visualize"
Requires-Dist: plotly>=5.20; extra == "visualize"
Provides-Extra: pdf
Requires-Dist: weasyprint>=62; extra == "pdf"
Provides-Extra: track
Requires-Dist: mlflow>=2.11; extra == "track"
Provides-Extra: tune
Requires-Dist: optuna>=3.5; extra == "tune"
Requires-Dist: nevergrad>=1.0.5; extra == "tune"
Provides-Extra: distributed
Requires-Dist: ray>=2.10; extra == "distributed"
Requires-Dist: dask>=2024.3.1; extra == "distributed"
Dynamic: license-file

<!-- SPDX-License-Identifier: Apache-2.0 -->
<!-- Copyright (c) 2026 CompilerSutraPerfTool -->
<!-- Author: Priya Pandey | https://www.linkedin.com/in/priya-pandey-54a5a22aa/ | CompilerSutra | https://www.linkedin.com/company/compilersutra/ | https://compilersutra.com -->

# CompilerSutraPerfTool

**Alpha** performance experimentation for C/C++ programs and GPU kernels (OpenCL, HIP, Vulkan shaders).

Compile, run, profile, compare, and export structured JSON/CSV/XLSX artifacts — from the CLI or `pip install compilersutra-perf`.

**Status:** Alpha MVP (0.2.0). Suitable for local experimentation; not yet validated for production CI regression gates. See [limitations](#limitations) and [docs/METHODOLOGY.md](docs/METHODOLOGY.md).

---

## Install

```bash
pip install compilersutra-perf
```

Optional extras:

```bash
pip install 'compilersutra-perf[visualize]'   # HTML reports + Streamlit dashboard
pip install 'compilersutra-perf[excel]'       # XLSX result workbooks
pip install 'compilersutra-perf[pdf]'         # PDF reports (WeasyPrint)
pip install 'compilersutra-perf[dev]'         # pytest
```

**Requirements:** Python 3.11+. Linux wheels ship prebuilt native runners for CPU (and Vulkan, where available), so CPU workloads run out of the box. Other platforms (or exotic Python builds) fall back to building the runner on demand, which needs a C/C++ toolchain and `cmake` on `PATH`; `perf` is also recommended for full CPU profiling on Linux. GPU backends need their respective runtimes (ROCm, OpenCL, Vulkan) installed separately.

**Platform wheels and old distributions:** wheels are tagged for the glibc they were built on (e.g. `linux_x86_64`), so a wheel built on Ubuntu 24.04 may fail to load its bundled runner on Ubuntu 22.04 or older (newer glibc). The loader probes the bundled runner at startup and automatically falls back to an on-demand local build, so only an older toolchain is needed there. Hardware counter events are probed per-CPU, so vendor-specific events never break collection on other vendors (Intel vs AMD).

**Native Windows:** not yet supported. The runner uses POSIX APIs (`fork`, `sched_setaffinity`) and won't build on Windows natively — use [WSL2](docs/WINDOWS_WSL2.md). Result artifacts (JSON/CSV/XLSX/HTML reports) open fine on Windows.

Check that every system tool csperf needs is present, and install whatever is missing:

```bash
csperf quickstart --output-dir results/quickstart   # doctor + example + report
csperf doctor              # report missing tools
csperf doctor --install    # install missing required tools (apt/dnf/pacman/apk/brew)
csperf doctor --install --all   # also install optional GPU/profiling tools
```

See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for perf, energy, and GPU fixes.

Develop from source:

```bash
git clone https://github.com/compilersutra/CompilerSutraPerfTool.git
cd CompilerSutraPerfTool
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
# Native runners are compiled on demand at first run; build them explicitly with:
cmake -S native/runtime -B build/native && cmake --build build/native
```

To produce a platform wheel that embeds prebuilt runners (used by the release pipeline):

```bash
CSPERF_BUNDLE_NATIVE=1 python -m build --wheel
```

---

## Quick start

```bash
csperf list-backends
csperf quickstart --output-dir results/quickstart
csperf list-energy-backends
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \
  --warmup-runs 1 --repeat-runs 3 --output results/cpu.json
csperf profile results/cpu.json
csperf diff results/run-a.json results/run-b.json --csv results/compare.csv
# Auto-compare -O0 / -O1 / -O2 / -O3 runtimes (writes results/optimize/ + comparison report):
csperf run --input examples/cpp/matrix_traversal.cpp --diff-optimize --no-perf --report-format both
# Energy (Linux RAPL / AMD GPU when available):
csperf run --input examples/cpp/matrix_traversal.cpp --backend cpu \
  --energy-backend amd-cpu --output results/cpu-energy.json
```

Each `run` writes `results/<name>.json`, `.csv`, and `.xlsx` (when `openpyxl` is installed).

---

## Backends

| Backend | Status | Notes |
| --- | --- | --- |
| **CPU** | Execute + profile | Native C++ runner, LLVM IR (clang), Linux `perf` counters, warmup/repeat, optional `--cpu-affinity` |
| **HIP** | Execute + profile | `hipcc` compile-and-run, ROCm devices, `rocprofv3` (then v2/v1) |
| **OpenCL** | Execute | Native kernel launch, device selection, warmup/repeat |
| **Vulkan** | Compile + validate | GLSL → SPIR-V, shader-module validation |
| **`gpu` alias** | Auto-route | `.hip` → HIP, `.cl` → OpenCL, shaders → Vulkan |
| **CUDA / Metal** | Planned | Listed in registry but not executable |
| **macOS CPU** | Experimental | Execution works; profiling uses `powermetrics` (needs passwordless `sudo`) |

---

## Compiler selection and flags

Use `--compiler` to select the compiler explicitly, or select the toolchain with environment variables; pass optimization and language flags with `--compiler-flag`.

### Choose compiler (CPU)

```bash
# Default: clang / clang++
csperf run --input program.cpp --backend cpu --output results/clang.json

# GCC
CC=gcc CXX=g++ csperf run --input program.cpp --backend cpu --output results/gcc.json

# Explicit compiler override env vars (highest priority after CXX_COMPILER)
CXX=g++-13 csperf run --input program.cpp --backend cpu --output results/gpp13.json
```

Resolution order: `CXX_COMPILER` / `C_COMPILER` → `CXX` / `CC` → default (`clang++` / `clang`).

HIP always uses `hipcc` when available.

### Add compiler flags

Repeat `--compiler-flag` for each option. `-O3` is added by default unless you pass another `-O*` flag (for example `--compiler-flag=-O0`):

```bash
csperf run --input program.cpp --backend cpu \
  --compiler-flag=-march=native \
  --compiler-flag=-funroll-loops \
  --compiler-flag=-fopenmp \
  --output results/optimized.json

csperf run --input program.cpp --compiler-flag=-std=c++20 --output results/cpp20.json
```

`CXXFLAGS` / `CFLAGS` are **not** read — pass flags explicitly.

### Compare two compilers (batch)

For a folder of `.c`/`.cpp` files, use the batch script with JSON compiler configs:

```bash
python scripts/compiler_diff_batch.py examples/cpp \
  --config1 configs/compiler_gcc.sample.json \
  --config2 configs/compiler_clang.sample.json
```

Sample config (`configs/compiler_gcc.sample.json`):

```json
{
  "compiler_family": "gcc",
  "cc": "gcc",
  "cxx": "g++",
  "compiler_flags": ["-O3", "-march=native"]
}
```

Outputs: per-file JSON/CSV diffs, `summary.csv`, and `summary.xlsx`.

### Compare two saved runs

```bash
csperf diff results/gcc.json results/clang.json \
  --csv results/gcc-vs-clang.csv \
  --derived-config configs/derived_metrics.sample.json
```

---

## Common run options

| Flag | Purpose |
| --- | --- |
| `--warmup-runs N` | Warmup iterations before measurement (default `3`) |
| `--repeat-runs N` | Measured trials with summary stats (default `15`) |
| `--no-perf` | Skip hardware counter collection |
| `--plan-only` | Print compile/run commands without executing |
| `--build-dir PATH` | Output directory for binary and IR |
| `--cpu-affinity 0,1` | Pin CPU execution (Linux) |
| `--device-index N` | GPU device for HIP / OpenCL / Vulkan |
| `--backend gpu` | Auto-select GPU backend from file type |
| `--compiler=PATH` | Select the CPU compiler binary |
| `--compiler-flag=FLAG` | Append compile flag (repeatable) |
| `--program-arg ARG` | Pass an argument to the workload (repeatable) |
| `--stdin-file PATH` | Provide workload stdin |
| `--execution-timeout SECONDS` | Bound each workload execution |
| `--diff-optimize` | Run `-O0`…`-O3`, write per-level results and runtime comparison |
| `--energy` / `--no-energy` | Enable or disable energy measurement (on by default when backends exist) |
| `--energy-backend KEY` | Select backends (`auto`, `amd-cpu`, `amd-gpu`, …); see `list-energy-backends` |
| `--json` | Machine-readable stdout |
| `--version` | Print package version |

OpenCL kernel arguments:

```bash
csperf run --input examples/opencl/saxpy.cl --backend opencl --device-index 0 \
  --kernel-name saxpy \
  --kernel-arg buffer:float:read:4096:1.0 \
  --kernel-arg scalar:uint32:4096 \
  --readback-arg 2
```

Inspect planned commands:

```bash
csperf run --input program.cpp --plan-only
```

---

## Results and visualization

Artifacts per run:

- **JSON** — full structured result (metrics, execution with `host_platform` and `metrics_availability`, hardware, diagnostics)
- **CSV** — sectioned spreadsheet-friendly report with dotted metric paths and units
- **XLSX** — Results + Metadata sheets (requires `[excel]` extra)

```bash
csperf profile results/cpu.json
csperf visualize results/cpu.json results/gpu.json --output report.html
csperf visualize results/cpu.json --format pdf --output report.pdf
csperf visualize results/cpu.json --format both --output report
csperf list-report-formats
pip install 'compilersutra-perf[visualize]' && csperf dashboard results/cpu.json
```

Energy and power domains appear in the HTML/PDF report when the run collected a `power` block. See [docs/ENERGY.md](docs/ENERGY.md) and [docs/REPORT.md](docs/REPORT.md).

Failed runs exit non-zero (`1`) and include actionable hints when tooling is missing. See [exit codes](docs/USAGE.md#exit-codes) in USAGE.

---

## Limitations

Read these before trusting numbers in papers or CI:

- **Alpha** — methodology and backend coverage are still evolving.
- CPU runs support `--execution-timeout` to bound workload execution.
- CPU runs support repeated `--program-arg` values and `--stdin-file`; manifests can define the same settings.
- **`--experiment` / `--opt-config`** — apply compile macros and optional extra flags; plugin entry points remain [#24](https://github.com/compilersutra/CompilerSutraPerfTool/issues/24) / 0.3.0.
- Optional extras `track`, `tune`, and `distributed` are reserved for later releases and are not wired into the CLI.
- **OpenCL** — program may recompile each trial; interpret kernel times carefully ([#38](https://github.com/compilersutra/CompilerSutraPerfTool/issues/38)).
- **CUDA / Metal** — not implemented despite registry entries.
- **Native runners** — PyPI wheel does not bundle C++ binaries; build from source or install runners separately for full execution.

Best supported today: **Linux CPU** compile/run/profile with `perf` configured, plus HIP/OpenCL exploration with manual interpretation.

---

## Documentation

Full guides live in the repository:

| Doc | Topic |
| --- | --- |
| [docs/GETTING_STARTED.md](docs/GETTING_STARTED.md) | First workload |
| [docs/USAGE.md](docs/USAGE.md) | CLI reference |
| [docs/ENERGY.md](docs/ENERGY.md) | Energy / power backends |
| [docs/REPORT.md](docs/REPORT.md) | Modular HTML/PDF reports |
| [docs/METHODOLOGY.md](docs/METHODOLOGY.md) | What metrics mean and platform caveats |
| [docs/TESTING.md](docs/TESTING.md) | Validation |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Design |
| [CONTRIBUTING.md](CONTRIBUTING.md) | Contributions |
| [CHANGELOG.md](CHANGELOG.md) | Release notes |

Repository: https://github.com/compilersutra/CompilerSutraPerfTool

---

## License

Apache License 2.0 — see [LICENSE](LICENSE).

**Author:** [Priya Pandey](https://www.linkedin.com/in/priya-pandey-54a5a22aa/) · [CompilerSutra](https://compilersutra.com)
