Metadata-Version: 2.4
Name: codi-api-agent
Version: 0.3.1
Summary: Read-only, spec-driven natural-language agent over any API (OpenAPI/Swagger, GraphQL, or auto-converted Postman/RAML/API Blueprint), with citations and a faithfulness check.
Author: API Agent
License: MIT
Project-URL: Documentation, https://github.com/your-org/api-agent/blob/main/HOW_IT_WORKS.md
Keywords: openapi,graphql,llm,agent,api,postman,read-only,rag
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: openai>=1.30
Requires-Dist: requests>=2.31
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: graphql-core>=3.2
Provides-Extra: ui
Requires-Dist: streamlit>=1.30; extra == "ui"
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=3.0; extra == "embeddings"
Requires-Dist: numpy>=1.26; extra == "embeddings"
Provides-Extra: sql
Requires-Dist: psycopg[binary]>=3.1; extra == "sql"
Provides-Extra: all
Requires-Dist: streamlit>=1.30; extra == "all"
Requires-Dist: sentence-transformers>=3.0; extra == "all"
Requires-Dist: numpy>=1.26; extra == "all"
Provides-Extra: test
Requires-Dist: pytest>=8; extra == "test"
Provides-Extra: evals
Requires-Dist: evals>=3.0.1; extra == "evals"
Requires-Dist: pyyaml>=6; extra == "evals"

# API Agent

Load an **API description** — OpenAPI/Swagger or GraphQL natively, a Postman
collection / RAML / API Blueprint (auto-converted on load), or even a **prose API
reference doc** — ask a question in natural language, and the agent picks the
right operations, calls them (strictly read-only), and returns an **answer with
citations** plus a multi-signal **evaluation** (grounding, sufficiency,
responsiveness). Ships with a Streamlit UI with live steps, streaming answers,
and token/cost tracking.

```
Load spec(s) / reference doc ──▶ Catalog of operations
                     │
User question ──▶ Router + intent (narrow to relevant ops) ──▶ Executor (call operations)
                     ──▶ Synthesis + Citation ──▶ Self-review ──▶ Evaluator ──▶ Answer
```

The LLM provider is an **OpenAI-compatible endpoint**, so the model is a config
value — Groq, HuggingFace, Ollama and OpenAI are all swappable without code
changes. The **generator**, **judge** and **router** roles are configured separately.
Full internals in [How_It_Works.md](How_It_Works.md).

## Install as a pip package (share it / minimal setup)

```bash
python -m venv .venv && source .venv/bin/activate
pip install "codi-api-agent[all]"                          # from a package index
# or from a wheel someone shared with you:
# pip install "codi_api_agent-0.3.1-py3-none-any.whl[all]"
export LLM_API_KEY="your-groq-or-openai-key"
api-agent                                                  # opens the UI at http://localhost:8501
```

The distribution is `codi-api-agent`; the import name stays `api_agent`
(`from api_agent import Agent`).

Then add a free no-auth demo spec in the sidebar (Countries GraphQL or the SWAPI Postman
collection) and ask away. **Full step-by-step in [TUTORIAL.md](TUTORIAL.md).** Build the wheel
yourself from a checkout with `pip install build && python -m build` (→ `dist/`).

## Setup (from a source checkout)

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"       # editable install with UI + embeddings
cp .env.example .env          # then edit .env with your provider + key
```

| Provider | `LLM_BASE_URL` | Key | Notes |
|---|---|---|---|
| **Groq** (recommended) | `https://api.groq.com/openai/v1` | free API key | fast, good tool-calling |
| **HuggingFace** | `https://router.huggingface.co/v1` | HF token | many open models |
| **Ollama** (local) | `http://localhost:11434/v1` | any non-empty string | offline; pick a tool-calling model |
| **OPENAI** | `https://api.openai.com/v1` | api-key | open ai models |

Set `GENERATOR_MODEL` / `JUDGE_MODEL` to models your provider serves **that support
tool/function calling** (e.g. on Groq, `llama-3.3-70b-versatile`).

**Rate-limit resilience:** give several keys (`LLM_API_KEYS=key1,key2,…`) and/or whole
backends (`LLM_POOL='[{"base_url":…,"api_key":…,"model":…}, …]'`) and the client rotates
over them per query and **fails over on a 429** with a cooldown — useful on free tiers.

**Cost controls (for paid, per-token providers):** set `MAX_RESPONSE_TOKENS` to cap tokens per
response — the agent stops early and returns a partial answer once the ceiling is hit (0 = unlimited).
Each response also shows an estimated **$ cost**; override the built-in per-model prices with
`MODEL_PRICING` (JSON per 1M tokens, e.g. `MODEL_PRICING='{"gpt-4o":[2.5,10]}'`) or the sidebar
**💲 Budget & cost** fields. Unknown/free models simply show no cost.

## Run

```bash
streamlit run app/streamlit_app.py
```

It opens empty — add an API via the sidebar's **📚 Load an API** panel (set
`DEFAULT_SPEC=<url-or-path>` in `.env` to auto-load one on startup). Free, no-auth
demos to try: the **Countries GraphQL** endpoint `https://countries.trevorblades.com/`
(tick **GraphQL API**), or the Petstore spec
`https://petstore.swagger.io/v2/swagger.json` → *“fetch all pets that are sold”*.

## Load any API spec

Sidebar → **📚 Load an API** → paste a spec **URL or file path** (or upload files) →
**➕ Add spec**. OpenAPI/Swagger and GraphQL load natively; a Postman collection,
RAML 0.8, or API Blueprint file is **detected from its content and auto-converted**
to OpenAPI on load. You can add **several sources** (any mix of formats) — each keeps
its own base URL and auth, and one question can span all of them.

Every `GET`/`HEAD` operation (or GraphQL **query** field) becomes a callable tool
(write operations are excluded — the read-only guardrail; they remain *describable*
in documentation mode). Each operation maps: `operationId` → name,
`summary`/`description` → routing text, `parameters` → arguments, `servers`
(or Swagger-2.0 `host`+`basePath`) → base URL.

**Authenticated (private) APIs:** open the **Auth (optional)** expander and set a
header before loading — e.g. `Authorization` = `Bearer <token>`, or `X-API-Key` =
`<key>`. It's attached to every call. If an API needs auth and none is set, the
agent says so instead of guessing.

**No spec?** Write a small one from the
[minimal template](examples/minimal_rest_template.openapi.yaml) — describe just the `GET`
endpoints you care about (copy a block per endpoint), then load it as a file path. You don't
need to be an OpenAPI expert; the `summary`/`description` you write are what the agent routes on.

## Other formats (RAML / API Blueprint / Postman)

The agent's pipeline is format-agnostic — only the *loader* speaks OpenAPI — so these are
**converted to OpenAPI automatically when you load them** (in the UI or via `load_catalog`).
To pre-convert from the terminal instead:

```bash
python scripts/convert_spec.py path/to/api.apib  -o specs/api.openapi.json   # API Blueprint
python scripts/convert_spec.py path/to/api.raml  --base-url https://your-host.com   # RAML 0.8
python scripts/convert_postman.py your.postman_collection.json --base-url https://your-host.com
```

| Input | Converts to | Notes |
|---|---|---|
| **API Blueprint** (`.apib`) | Swagger 2.0 | read natively |
| **RAML 0.8** (`.raml`) | OpenAPI 3.0 | RAML **1.0** has no good free CLI converter — convert it to 0.8/OpenAPI first |
| **Postman collection** | OpenAPI 3.0 | see [scripts/convert_postman.py](scripts/convert_postman.py) |

Requires Node/npx (the converters are npm tools, fetched on first use).

## No spec at all? Load a prose API reference doc

Sidebar → **📚 Load an API** → source type **API reference doc** → point it at a
reference page (URL, file, or upload) → **🔍 Extract endpoints**. The documented
`METHOD /path` lines, curl examples, path and query params are extracted from the
doc's **literal text** (free, deterministic — it cannot invent an endpoint); an
optional checkbox lets the LLM also enrich param types or handle prose-only docs.
You then **review and approve** the extracted endpoints before any become callable —
GETs load as tools, writes as documentation-only.

Inspect what a spec produces from the terminal:

```bash
python scripts/load_openapi_demo.py                          # default: Petstore
python scripts/load_openapi_demo.py --source your_spec.yaml --no-call
```

## Verified working spec URLs (no auth)

| API | Spec URL |
|---|---|
| Petstore v2 (pets) — *default* | `https://petstore.swagger.io/v2/swagger.json` |
| APIs.guru (API directory) | `https://api.apis.guru/v2/specs/apis.guru/2.2.0/openapi.json` |
| ExchangeRate-API (FX rates) | `https://api.apis.guru/v2/specs/exchangerate-api.com/4/openapi.json` |
| Color Name API | `https://api.apis.guru/v2/specs/color.pizza/1.0.0/openapi.json` |

## Query your own GitHub repos

GitHub publishes its OpenAPI spec, so you can ask about *your* account:

1. **Load the public spec** (sidebar → **📚 Load an API** → *URL or file path*):
   `https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json`
   It has ~624 GET operations; the default **Max operations to load** (1000) loads them
   all and the router narrows per query.
2. **Set a token** in the **Auth** expander *before loading*: header `Authorization`,
   value `Bearer <your GitHub PAT>` (a token with `repo` / read scope).
3. **Ask:** *“show my repositories”* → routes to `GET /user/repos` and lists your repos.

More once loaded + authed: *“who am I on GitHub?”* (`GET /user`), *“list my open issues”*.

> The big GitHub spec takes a few seconds to fetch/parse on load. Without a token,
> authed endpoints return 401 and the agent tells you a key is needed.

## Routing (large specs)

When a spec has many operations, the **router** narrows them to the most relevant
per query before the agent runs: **hybrid** recall — lexical (idf-weighted, stemmed)
fused with **local embeddings** (`all-MiniLM-L6-v2`, free/offline; catches paraphrases
with zero shared words) via Reciprocal Rank Fusion — then a fast LLM makes the final
pick **and classifies the request intent** (write/doc/data) in the same call. Falls
back to lexical-only if embeddings are unavailable. Specs with ≤ `router_min_tools`
(default 6) operations skip routing. Configure in the sidebar or via
`ROUTER_ENABLED` / `ROUTER_TOP_K` / `ROUTER_MODEL` / `ROUTER_MIN_TOOLS` /
`EMBEDDING_MODEL`.

## Testing & performance report

Two ways to check the agent is behaving:

```bash
./scripts/run_checks.sh              # deterministic regression suite (no LLM/network) — see tests/README.md
python evals/run_eval.py --offline   # reliability report (from the suite), zero cost
python evals/run_eval.py             # full report: reliability + live accuracy on public APIs (needs LLM_API_KEY)
```

The eval harness writes a stakeholder-facing `evals/report.md` + `report.json` covering guardrail
reliability, per-category accuracy, faithfulness, read-only/PII safety, abstention, latency, and
(with `--repeats N`) run-to-run consistency. Details in [evals/README.md](evals/README.md).

## Project layout

```
api_agent/
  config.py          # env-driven settings (provider, per-component models, router, budget)
  llm.py             # OpenAI-compatible client + multi-LLM rotation + tool-call recovery
  openapi_loader.py  # OpenAPI/Swagger spec -> callable Tools; compaction + PII redaction
  graphql_loader.py  # GraphQL introspection/SDL + the load_catalog format dispatcher
  spec_convert.py    # Postman / RAML / API Blueprint -> OpenAPI (auto, via npx)
  doc_extract.py     # prose API reference doc -> draft OpenAPI (structural; LLM optional)
  catalog.py         # Tool + Catalog (operation registry + result cache + name recovery)
  router.py          # hybrid lexical+embedding routing per query
  schemas.py         # Evidence, Citation, ToolCall, Faithfulness, AgentResult, Usage
  agent.py           # pipeline: cache -> route -> execute -> synthesize -> review -> evaluate
  ui.py              # Streamlit app: live steps, streaming, Stop, token/cost, doc review gate
app/streamlit_app.py # entry point for the UI
scripts/load_openapi_demo.py  # CLI: inspect a spec + one live call
tests/               # deterministic regression suite (no LLM/network)
evals/               # reliability + live-accuracy report harness
```

## Notes & limitations

- **Read-only:** only `GET`/`HEAD` operations are exposed; write endpoints are never called.
- **Honest failures:** if an operation needs a key (401/403), is unreachable, or the
  server returns 5xx, the agent reports that clearly instead of guessing.
- **Not production-hardened:** the loader fetches the given spec URL and calls
  endpoints as-is — SSRF egress controls and per-user credential scoping are
  follow-ups; fine for dev against trusted specs.
- Public demo servers (e.g. Petstore **v3**) are often flaky — prefer v2 / a spec
  whose server you control.
