Metadata-Version: 2.5
Name: ghost-driver
Version: 0.2.0
Summary: A nodriver wrapper for automation that needs to survive behavioral antibot checks, not just fingerprint checks: real X11 input instead of CDP-synthesized events, human trajectory replay, and pluggable captcha solvers.
License: MIT
Requires-Python: >=3.10
Requires-Dist: nodriver==0.50.3
Requires-Dist: requests>=2.31.0
Provides-Extra: cv
Requires-Dist: numpy<2,>=1.24; extra == 'cv'
Requires-Dist: opencv-python-headless<5,>=4.8; extra == 'cv'
Description-Content-Type: text/markdown

# ghost-driver

A [nodriver](https://github.com/ultrafunkamsterdam/nodriver) wrapper for automation that needs to survive **behavioral** antibot checks, not just fingerprint checks.

## Why

CDP's `Input.dispatchMouseEvent`, however carefully its parameters are set — coordinates, pressure, the `buttons` bitmask — does not trigger the same internal Blink engine state a genuine X11 input event does. This was confirmed with a rotate-captcha (drag a slider to un-rotate a photo) that a real human solved manually at ~90-100%, while every synthetic-input variant we tried — careful CDP dispatch, algorithmic "natural" mouse easing, even over real X11 — stayed stuck at 20-36%.

The tell: the slider button's CSS `:active` pseudo-class engages correctly on a CDP `mousePressed`, then flips `false` on the very first subsequent `mousemove` and never recovers — 98% `false` across CDP-driven drags vs 96% `true` for real manual ones. Real X11/XTest events (the same event class VNC's own mouse injection uses, via `xdotool`) don't have this problem.

Driving the **entire drag** through real X11 input, replaying an **actual recorded human trajectory** (not an algorithmic approximation), closed the gap: 15/15 (100%) on a live confirming run.

ghost-driver packages that finding — plus everything that had to be fixed alongside it (WebGL being entirely absent under Xvfb, `document.hasFocus()` never being true, fractional pixel coordinates no real mouse can produce) — into a reusable library on top of `nodriver`.

## Install

```bash
pip install ghost-driver
# for the free CV-based captcha solver (needs opencv/numpy):
pip install ghost-driver[cv]
```

nodriver itself needs two small source patches applied after install (see [`ghost_driver/browser/patches.py`](ghost_driver/browser/patches.py) for exactly what and why — both live deep inside methods too large to cleanly monkeypatch, so this edits the installed package on disk, idempotently, with an assertion that fails loudly if nodriver's source has changed underneath it):

```bash
ghost-driver-patch-nodriver
```

Run it once after every `pip install`/`pip install --upgrade` of `nodriver` — e.g. right after `pip install` in a Dockerfile.

`pip install` can't run this (or install the system packages below) for you automatically — there's no post-install hook in modern Python packaging for either. Forget one and `ghost_driver.launch()` will tell you exactly what's missing and how to fix it (the same "browsers not installed, run `playwright install`" pattern Playwright uses) rather than failing confusingly later, deep inside an actual session — see `ghost_driver.browser.preflight`.

### System requirements

- `xdotool` on `PATH` — this is the actual mechanism, not optional tooling
- A real X11 `DISPLAY` with a window manager in front of it. **Xvfb is fine** (virtual, no GPU needed) — bare Xvfb has no window manager though, so also run something minimal like `fluxbox` alongside it
- Google Chrome (not Debian's `chromium` package — this was developed and tested against `google-chrome-stable`)

Real X11 input has no mapped window to target at all under `--headless=new` — this is an architectural limit, not a bug, and `ghost_driver.input.x11` raises `NoX11DisplayError` rather than silently degrading. See [Headless vs headful](#headless-vs-headful) below.

## Quickstart

```python
import asyncio

import ghost_driver
from ghost_driver.input import trajectories, x11
from ghost_driver.solvers.capguru import CapGuruSolver
from ghost_driver.solvers import correction_to_percent
from ghost_driver import page_state


async def main():
    browser = await ghost_driver.launch(headless=False)  # real X11 input needs headful (Xvfb is fine)
    page = await browser.create_context(url="https://example.com/captcha-page")

    state, _data = await page_state.wait_for_widget_or_content(
        page, timeout_s=30,
        has_widget_js_expr="!!document.querySelector('.your-widget-class')",
    )
    if state != "widget":
        return

    await x11.ensure_real_focus(page)  # a real xdotool click -- CDP's page.activate() alone doesn't flip document.hasFocus()

    solver = CapGuruSolver(api_key="your-cap-guru-key")  # or set CAP_GURU_KEY in the environment
    image_bytes = ...  # capture your widget's already-rendered image (see note below)
    result = await solver.solve(image_bytes)
    target_percent = correction_to_percent(result.correction_deg)

    # geometry is yours to read off your own widget's DOM
    start_x, start_y, max_left = ...
    needed_delta_x = max_left * (target_percent / 100.0)

    pool = trajectories.load_trajectories()  # defaults to the bundled e-disclosure.ru example recordings
    events = trajectories.pick_for_target(pool, needed_delta_x)
    await x11.drag_replay(page, events, start_x, start_y, start_x + needed_delta_x)


asyncio.run(main())
```

Capture the captcha image by reading an already-rendered `<img>` through a `<canvas>` (`drawImage` + `toDataURL`), not a raw `fetch()` of the image URL — an endpoint that serves a fresh random image per request will hand a raw fetch a *different* image than the one actually shown.

`trajectories.load_trajectories()` uses 60 real recorded human trajectories bundled with the package (see [`ghost_driver/examples/e_disclosure_rotate_captcha/`](ghost_driver/examples/e_disclosure_rotate_captcha/)) by default, so drag replay works out of the box without recording anything yourself first. These are calibrated to one specific site's slider though — pass your own `directory` (e.g. wherever `ghost_driver.input.recorder.record_round` saves to) once you're targeting a different widget.

### Quickstart: unknown page, auto-detect + Cloudflare Turnstile

For a URL where you don't know in advance whether a captcha will show (or which one), `navigate_auto()` detects and solves it for you — including Cloudflare Turnstile, which works headless (unlike the rotate-captcha family above, which needs real X11 input):

```python
import asyncio

import ghost_driver
from ghost_driver import session


async def main():
    browser = await ghost_driver.launch(headless=True)  # Cloudflare Turnstile solving works headless
    page = await session.new_page(browser)

    result = await session.navigate_auto(page, "https://example.com")
    print(result.ok, result.state, result.family)  # family is None if nothing was detected at all

    browser.stop()


asyncio.run(main())
```

Real Cloudflare-protected sites *usually* pass `navigate_auto()` with `family=None` — Turnstile's own non-interactive check accepts a well-behaved browser without ever showing a checkbox (confirmed live against 12 real sites, zero interactive challenges). When it does show one, `ghost_driver.fingerprint.cloudflare.apply_defaults()` and `ghost_driver.solvers.cloudflare.solve_turnstile()` are applied automatically. For full runnable examples (a multi-site `navigate_auto()` sweep, and the solve-once-headful/reuse-headless pattern that's the right approach for a site with a genuinely tightened Cloudflare configuration — a real, confirmed exception, not the common case), see `solve_cloudflare_auto.py` and `solve_cloudflare_cookie_reuse.py` — not part of this repo, kept alongside the (unpublished, local-only) project this library was developed against.

## Modules

| Module | What it does |
|---|---|
| `ghost_driver.browser` | `launch()` — thin wrapper around `nodriver.start()` with WebGL (`swiftshader`) and a launch-retry loop already applied |
| `ghost_driver.browser.patches` | Idempotent, assertion-guarded source patches for nodriver's CDP connection-wait retry count and proxy-forwarder timeout |
| `ghost_driver.browser.preflight` | `check()` — raises `SetupError` (with every problem found AND its exact fix) if the nodriver patches, `xdotool`, or a real Chrome binary are missing. Runs automatically at the start of `launch()`; pass `skip_preflight=True` to opt out |
| `ghost_driver.input.x11` | The core mechanism: real X11 input via `xdotool` — `ensure_real_focus`, `window_geometry`, `drag_replay` |
| `ghost_driver.input.cdp_replay` | CDP-based drag replay — a documented, **confirmed weaker** fallback for when there's no real X11 display at all |
| `ghost_driver.input.drag` | `drag_replay()` — picks X11 when available, falls back to CDP automatically |
| `ghost_driver.input.trajectories` | Loading, filtering, and picking recorded human drag trajectories from JSON recordings |
| `ghost_driver.input.recorder` | `record_round()` — captures real human pointer/mouse events during a manual (e.g. VNC) solving session, with correct handling of a challenge that bounces through multiple navigations in one round |
| `ghost_driver.solvers` | `RotationSolver` protocol + `RotationResult` — pluggable rotate-captcha solvers |
| `ghost_driver.solvers.capguru` | `CapGuruSolver` — paid, full 0-360° answer in one shot |
| `ghost_driver.solvers.cv_angle` | `CVAngleSolver` — free, local, purely geometric (needs the `cv` extra); only tells axis-alignment (mod 90°), see `RotationResult.family_period_deg` |
| `ghost_driver.solvers.cloudflare` | `solve_turnstile()` — Cloudflare Turnstile ("Verify you are human") solving: event-driven OOPIF attach, real X11 click headful / session-scoped CDP click headless, multi-round handling |
| `ghost_driver.solvers.oniks_krep` | `solve()` — oniks-krep.ru/tvoysaratov.ru's three challenge layouts: "image" (blur+rotate+correlate matching, confirmed 22/22), "color" (Euclidean RGB distance, confirmed 5/7), "buttons" (nothing to distinguish candidates by — confirmed 0/3, a known open problem, not a bug) |
| `ghost_driver.solvers.image_match` | `rank_candidates()` / `rank_colors()` — the pure image/color comparison logic `solvers.oniks_krep` solves with (needs the `cv` extra); no ghost_driver/nodriver dependency, so also usable standalone |
| `ghost_driver.solvers.detect` | `detect()` — looks at an already-loaded page and reports which known captcha family (if any) is showing: Cloudflare Turnstile, ServicePipe rotate-captcha, or oniks-krep image/color |
| `ghost_driver.page_state` | `wait_for_widget_or_content` / `wait_for_stable_content` — reliable "did the challenge appear / did it actually pass" polling with a debounce against mid-navigation false reads |
| `ghost_driver.fingerprint` | `webrtc` (IP-leak fix), `misc` (window dims, pointer capability, device memory, color depth, languages), `native` (`Function.toString` spoofing helper), `cloudflare` (Turnstile-proven UA/Client-Hints/focus-emulation bundle) |
| `ghost_driver.session.cookies` | `to_cookie_param()` — converting a captured `Cookie` into a `CookieParam` for reuse in a fresh context |
| `ghost_driver.diagnostics` | `dump()` / `save_html()` / `save_json()` — save what actually happened on an attempt, so a confusing result has something to inspect afterward |
| `ghost_driver.session.navigate_auto` | Navigates, runs `detect()`, and dispatches to whichever family's solver applies automatically — for a caller who doesn't know in advance what (if anything) a URL will show |

## Headless vs headful

`ghost_driver.browser.launch(headless=...)` supports both — it's just `nodriver.start()`'s own parameter. But the reason this library exists, `ghost_driver.input.x11`, **only works headful** (Xvfb counts as headful — it's a real, if virtual, X11 display and window). Under true `--headless=new` there is no mapped window at all for `xdotool` to target; calling any `ghost_driver.input.x11` function with `headless=True` raises `NoX11DisplayError` immediately rather than silently grabbing an unrelated X11 surface (confirmed live: a blind `xdotool getactivewindow` under real headless doesn't fail cleanly, it happily returns geometry for *some* unrelated surface, and a drag built on it "succeeds" with no exception while touching nothing).

`ghost_driver.input.drag_replay()` handles this automatically: it tries X11 first (unless you pass `headless=True`) and falls back to `cdp_replay` on `NoX11DisplayError`. The CDP path is real and documented, but it's the ~20-36%-pass-rate path this whole library exists to get away from — treat it as a fallback, not a primary option.

## Known limitations

- **Cookie reuse needs a static IP.** Reusing a verified session's cookies in a fresh context (`ghost_driver.session.cookies`) only works if both contexts share the same exit IP — a verification cookie can be bound server-side to the IP it was issued to, not just its own value. A rotating proxy pool between the solve and reuse phases breaks this even with the cookie set correctly.
- **The free CV solver can't tell orientation.** `CVAngleSolver` only detects axis-alignment (mod 90°) via background texture direction — it can't tell which of the 4 rotationally-symmetric orientations is actually upright. Use `ghost_driver.solvers.candidate_corrections()` and try multiple candidates across retry rounds. `CapGuruSolver` doesn't have this problem (it looks at the actual photographed object) but costs money per solve.
- **Don't add `Emulation.setHardwareConcurrencyOverride`.** Confirmed via an isolated A/B (10/10 passed with it off, 0/8 with it on) that this actively breaks solving — it's presumed to change the real worker-thread pool Chrome allows, not just the reported value, corrupting whatever computation signs a site's behavioral-telemetry payload. Documented in `ghost_driver.fingerprint.misc`'s module docstring; deliberately not shipped as a working helper.

## Credentials

`CapGuruSolver` takes `api_key` as a constructor argument, or reads the `CAP_GURU_KEY` environment variable if not passed explicitly. There is no built-in fallback key — never hardcode a real API key as a source-level default in code that might be published or shared.

## License

MIT
