Metadata-Version: 2.5
Name: sequence-ai
Version: 0.5.0
Summary: Cloud inference for robot policies — a control loop that keeps the connection open and the action buffer full.
Project-URL: Homepage, https://generalsequences.com
Project-URL: Documentation, https://generalsequences.com/docs/
Project-URL: Console, https://app.generalsequences.com
Author: General Sequences
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: inference,manipulation,policy,robotics,vla
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: httpx>=0.24
Provides-Extra: dev
Requires-Dist: numpy>=1.24; extra == 'dev'
Requires-Dist: pillow>=10; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: image
Requires-Dist: pillow>=10; extra == 'image'
Description-Content-Type: text/markdown

# sequence-ai

Cloud inference for robot policies.

```bash
pip install sequence-ai
```

```python
import sequence_ai

with sequence_ai.connect(model="pi05-droid") as policy:
    out = policy.act(
        "pick up the cup",
        observe=robot.read_observation,   # cameras + joints
        act=robot.apply_action,           # one joint target
        validate=robot.is_safe,
        hold=robot.hold,
        max_seconds=30,
    )

policy.interrupt()      # from any thread, at any moment
```

---

## Two brains, one string between them

A VLM looks at a snapshot, decides *what* to do next, and says it in a sentence. This package is
the *how*: it takes that sentence with the current cameras and joints, asks a
vision-language-action model for the next second of joint targets, and plays them out while
fetching the next second.

The two halves run at completely different speeds — the brain thinks between subtasks, the arm
needs a chunk every second — and `instruction` is their entire interface.

| | endpoint | who ships it |
|---|---|---|
| slow brain | `POST /v1/chat` | Messages in, one message out |
| fast brain | this package | `policy.act("pick up the cup", ...)` |

`act()` is shaped to be called as a tool. It overwrites the observation's instruction on every
call, so your eyes never have to know what the brain last decided and a stale sentence cannot
leak into a chunk after the brain has moved on. `max_seconds` is required and has no default:
this drives hardware, and a tool call a language model can start but nothing bounds is one an
arm can be left running by a dropped conversation.

Wiring the two together — retries, failure detection, choosing the next sub-goal — is yours.
This package provides inference and a control loop, not orchestration.

---

## Changing your mind

```python
policy.interrupt()
```

Thread-safe, and the only method meant to be called from a thread other than the one driving.

**It stops within one control period, not one chunk.** The loop checks between every action, so
at 15 Hz the arm stops in about 67 ms. Waiting for the current chunk to finish playing would be
up to a full second of an arm still reaching for something the brain has already given up on.

**It also cuts through a stalled fetch.** If the buffer has run dry and the loop is blocked on a
chunk that is taking five seconds, an interrupt does not wait that out — stop latency is a
property of this loop and never of the service.

Interruption unwinds through `hold()` like any other early stop, and is not raised: a stop you
asked for is an outcome, not an error. `RunOutcome.reason` says which ending happened.

| `reason` | meaning |
|---|---|
| `completed` | ran out of actions to play |
| `until` | your `until()` returned True — you judged the subtask done |
| `timeout` | `max_seconds` elapsed |
| `interrupted` | someone called `interrupt()` |
| `validate` | a `validate` callback refused an action |
| `raised:<Class>` | your callback or the network raised |

---

## What it does for you

Four things, and each of them is something a loop written straight against the HTTP endpoints
has to get right before it behaves properly on a robot:

| | Without it |
|---|---|
| Keeps the connection open | 185 ms of handshake on every call — 60% of a bare request |
| Refills before the buffer empties | The arm stops once per chunk, for a full round trip |
| Classifies errors | No way to tell "wait 118 s" from "stop and page someone" |
| One driver per buffer | Two loops on one handle interleave, and both report success |

None of the four is guesswork. Every one is a defect that existed in this client, was measured,
and was fixed — the numbers below are those measurements, not estimates.

## It keeps the connection open

Measured against the production gateway, six samples each:

| | median | min | max |
|---|---|---|---|
| new connection per call | **305.9 ms** | 246.7 | 766.7 |
| one reused connection | **121.0 ms** | 116.8 | 133.4 |

**184.9 ms per call — 60% of the total — is TCP and TLS handshake.** That is what
`with sequence_ai.connect(...)` removes. Nothing proprietary: any HTTP client that reuses a
connection gets the same result. This package just makes it the default rather than
something you have to remember.

Why it matters: a chunk is a deadline. `pi05-droid` returns 1.00 s of motion per call, and a
warm end-to-end `/v1/act` through this gateway measured **793 ms** — the next chunk has to
arrive before the current one finishes playing, so 185 ms of avoidable handshake is a fifth
of the entire budget.

## Actions come in chunks

One call returns a block of future actions, not a single command — between 0.5 s and 2.1 s
of motion depending on the model. That is why cloud inference works at all: the control loop
does not need a network round trip per control step.

```python
with sequence_ai.connect(model="pi05-droid") as policy:
    pred = policy.predict(observation)
    print(len(pred.action_chunk), "steps covering",
          pred.action_chunk.covers_seconds, "s at",
          pred.action_chunk.control_frequency_hz, "Hz")
```

## The gap between chunks is the hard part

A loop that waits for the buffer to empty before asking for the next chunk stops the arm once
per chunk, every chunk, for a full round trip. It is not an occasional hiccup — it is
structural, and on the faster models it dominates:

| model | chunk covers | refill | arm actually moving |
|---|---:|---:|---:|
| `cosmos3-edge-policy-droid` | 2.13 s | ~0.79 s | 73% |
| `pi05-droid` | 1.00 s | ~0.79 s | 56% |
| `lingbot-va-5b` | 0.64 s | ~0.79 s | 45% |
| `lingbot-vla-v2-6b` | 0.50 s | ~0.79 s | **39%** |

So `run()` starts the next inference *while the current chunk is still playing*, and reports
what happened rather than hoping:

```python
out = policy.run(observe=..., act=..., max_actions=250, on_underrun=robot.hold_position)

print(out)        # RunOutcome(250 actions over 17 chunks in 16.8s, completed)
out.underruns     # times the buffer ran dry before the next chunk arrived
out.underrun_s    # total seconds the arm spent with no command
out.max_seam_jump # largest per-dimension step across a chunk boundary
```

**The trade, stated plainly:** a prefetched chunk is computed from an observation taken
*before* the previous chunk finished, so the overlap window is open-loop. Freshness and
continuity are in direct opposition here and no setting gets both. `prefetch=False` restores
strictly closed-loop behaviour, stall included — right for bench work, wrong for a moving arm.

`max_seam_jump` is a **measurement, not a correction**, and it stays one unless you ask
otherwise. `smooth_seam=N` ramps the first N steps of each new chunk out of the last executed
action, and it is the only setting in this library that changes a number on its way to the
motors — so it is off by default and guarded twice: it applies only when the chunk declares an
*absolute* action space (`action_chunk.action_space`), and never to the first chunk of a run.
Blending deltas is not smoothing; it rescales the increments and moves the arm somewhere the
model never asked for, so a delta chunk is passed through untouched even when you ask.
`max_seam_jump` is measured *before* any blending, so turning it on cannot hide what it smooths.

## Many robots at once

**One `Policy` per control loop.** A `Policy` holds one action buffer, and two loops popping
from it do not take turns — they interleave. Driving the same handle from a second thread
raises, because both quieter options are worse: interleaving sends each robot a shuffled half
of the other's plan while both loops report success, and a blocking lock would make the second
loop run at half rate, underrunning on every chunk.

```python
def drive(robot, model):
    with sequence_ai.connect(model=model) as policy:      # one each
        return policy.run(observe=robot.read, act=robot.apply,
                          validate=robot.is_safe, max_actions=250)

with ThreadPoolExecutor() as pool:
    left, right = pool.map(drive, [arm_l, arm_r], ["pi05-droid"] * 2)
```

This costs nothing to follow. The handshake is paid *once per `Policy`*, not once per call.
Measured with eight loops running concurrently against one gateway: eight connections, ten
requests each, zero sequence breaks, zero underruns. Requests arriving while the gateway is
busy queue on the server rather than displacing work already in flight.

## What the loop says while it runs

`run()` reports lifecycle, not telemetry:

```
  starting…      first chunk on its way, or a cold worker loading — nothing is moving yet
  running        a command has reached the robot
  done
```

Waiting out a cold worker (about 100 s) and losing a few hundred milliseconds at a chunk
boundary are the library's problems, not yours — `startup_timeout_s` defaults to 300 s, so the
first call on a cold worker waits rather than failing. Every number is still in `RunOutcome`
afterwards if you want it.

```python
policy.run(..., on_status=log.info)     # programmatic
policy.run(..., progress=False)         # silent
```

The default `progress="auto"` writes a single line to **stderr** only when stderr is a
terminal — nothing in a script, a pipe, or a log file.

## Cold starts

A worker that is not loaded takes about **100 seconds** to become ready — 12.5 GB of weights and
a JIT compile. A control loop's budget for one chunk is **533 ms**. Those two numbers are why
connecting is a separate call from controlling.

### Connect first, then drive

```python
policy.wait_until_ready()          # once, before the robot needs to move
while running:
    policy.next_action(observe())  # every one of these is warm by construction
```

`wait_until_ready()` needs no observation — a robot should be able to bring its model up before
it is in position, which is exactly when it has no frame worth sending. It is authenticated but
**not billed**: it touches no GPU. It is also what *starts* the worker, so polling it is the
thing that brings the model up, not merely a way to watch.

`ready()` is the non-blocking form, returning `(ready, eta_seconds)`.

**Skipping it does not fail — it silently costs you the loop.** Measured on one run against a
cold endpoint, the client starting anyway:

```
first control request   13,533 ms      <- the cold start, now inside the loop
p50                        415 ms      <- the steady state was always fine
mean                     1,096 ms      -> 0.49 arms sustainable   
mean without that one      441 ms      -> 1.21 arms sustainable   
```

One request that should not have been in the loop is the entire difference between sustainable
and not.

### If you skip it anyway

The gateway does not stall on a cold worker — it answers 503 immediately with the number:

```python
try:
    policy.predict(observation)
except sequence_ai.Unavailable as exc:
    if exc.warming:
        print(f"loading; ready in ~{exc.retry_after_s}s")   # 118
```

`run()` waits that out for you by default (`startup_timeout_s=300`), **and only before the
first action**. It reports `starting…` while it does, so a wait is never mistakable for a hang:

```python
policy.run(..., startup_timeout_s=0)     # opt out: fail immediately on a cold worker
```

That line is the whole design. Before the first action nothing is moving, so waiting is free.
Once the arm is in motion, silently pausing it for two minutes and resuming from a
two-minute-old plan is worse than stopping — so mid-run warming is raised, and `hold` fires.

## How big your frames are is how fast you go

**Latency scales with the bytes you send, at roughly 16 ms per kB.** This is the single largest
thing under your control, and it is larger than the model:

```
observation    p50      mean     sustainable arms
14.6 kB      586 ms    623 ms         0.86        
 3.4 kB      384 ms    433 ms         1.23        
```

Measured through the gateway, alternating A/B against one warm worker over one connection so
that drift in the service cancels. `inference_ms` did not move between the two (119 ms against
111 ms) — every millisecond of the difference was transport. A direct measurement against the
worker agreed: 181 ms for the same 11.2 kB.

**Send JPEG, not arrays.** A frame as a list of integers is roughly 16x the bytes of the same
frame as base64 JPEG, and it is the most common way to land on the slow side of that table.

**Use quality 95. Do not go below 90.** With the sampling noise pinned, on a real DROID frame:

| quality | max\|Δ action\| | share of action amplitude |
|---|---|---|
| 95 | 0.0048 | 0.51% |
| 90 | 0.0019 | 0.20% |
| 85 | 0.0177 | **1.86%** |
| 75 | 0.0269 | 2.86% |

Quality 85 and below is where the error jumps by an order of magnitude. This section previously
read "quality 85 is safe" and quoted the 0.51% figure next to it — but 0.51% is **q95's** number,
and q85 is nearly four times worse. `sequence_ai.encode()` defaults to 95 so the decision is not
yours to get wrong.

**Fidelity below that is not a gentle slope.** Degrading the frame further does not degrade the
policy proportionally; it stops working. Quantising each channel to four levels — layout, edges,
shadows and hues all preserved, a milder change than any sane JPEG setting — took pi0.5-DROID
from placing the cube in the bowl in 5.3 s to never lifting it at all, with its closest approach
20x worse than the successful run's final distance. Measured in simulation, 2026-09-11.

## Four numbers you should never have to remember

Which camera views a model wants, how wide its state vector is, what aspect ratio it trained on,
and what its returned floats mean. All four are per-model, none is guessable, and **every one of
them fails silently** — an unknown view name is not rejected, it arrives as an empty camera; a
short state vector is not rejected, it is zero-padded. The robot moves either way; it just moves
on something other than what you meant.

So read them off the catalogue instead of remembering them:

```python
model = {m.short_id: m for m in sequence_ai.models(endpoint="act")}["pi05-droid"]

obs = sequence_ai.observation(
    model,
    images={"exterior_1": cam.read(), "wrist_left": wrist.read()},   # ndarray or PIL
    joints=arm.joint_positions(), gripper=[arm.gripper_position()],
    instruction="pick up the cup",
)
```

That centre-crops each frame to the declared ratio — cropping, not padding, because the
model's own transform already pads and doing it twice fences the scene with bars on all four
sides — encodes at quality 95, and raises **locally,
before anything is billed**, if the views or the state width do not match what the model
declares. `policy.predict()` runs the same check and warns when you build an observation
yourself; `connect(validate="strict")` makes it raise, `"off"` skips it.

`state_dim` is the input width and is not `action_dim`. Across the served catalogue it is 8, 14, 8 and 0 — the
last meaning that model never reads joints at all, so you may omit them. Read it off the model,
not off this sentence.

`state_layout` says how that width splits — `{"joint_positions": 7, "gripper": 1}` for
pi0.5-DROID — and the halves are not interchangeable. Eight joints with no gripper sums to 8 and
is accepted; the eighth is then dropped and the gripper zero-filled, which holds the hand **open**
for the whole episode. `observation()` checks each half, not the sum.

## What to do with the rows that come back

**Send them raw.** Every model reports `pass_raw: true`, and that is what the checkpoint's own
reference loop does on a real robot — take the row, binarise the gripper, clip, `env.step()`,
nothing added:

```python
env = RobotEnv(action_space="joint_velocity", gripper_action_space="position")   # DROID stack
chunk = policy.predict(obs).action_chunk
for i in range(chunk.open_loop_horizon or len(chunk)):
    env.step(sequence_ai.prepare(chunk, model, i))
```

`model.action_semantics` tells you which controller interface those rows are for, so you can
find out before the first request rather than after. The catalogue spans four action spaces —
`joint_delta`, `joint_absolute`, `ee_absolute`, `ee_delta` — and reading one as another is
invisible in the shape — the array has the same width and dtype either way.

**Convert only if your controller cannot take that interface.** Isaac Lab's DROID scene has a
joint-position action term and no velocity one, which is the case this exists for:

```python
q_ref = arm.joint_positions()                  # ONCE, before the chunk is requested
for target in sequence_ai.to_joint_positions(chunk, q_ref, model):
    arm.move_to_joint_positions(target)
```

Every row offsets the same `q_ref`. They are cumulative displacements, so re-reading the live
pose each step turns the chunk into an integrator and the arm overshoots. `to_joint_positions()`
refuses any model that does not declare a verified conversion rather than inventing one.

`sequence_ai.recipes` holds one module per model — pi0.5-DROID and Cosmos3-Edge — with the
worked loop for that model and a `check()` that compares it against the live catalogue.
`for_model()` returns `None` for the rest.

## Three ways to drive it

From most control to least. They are the same request underneath; the difference is who owns
the loop.

```python
policy.predict(obs)            # the whole chunk, you do everything
policy.next_action(obs)        # we hold the buffer, you own the cadence
policy.run(observe=, act=)     # we own the loop
policy.act("...", observe=, act=)   # we own the loop and the instruction
```

`act()` is `run()` with the instruction pinned and a time bound required — the shape a VLM calls
as a tool. Everything `run()` accepts, `act()` accepts.

## Safety

**An action returned by any model is model output, not a safe robot command.**

This library does not check joint limits, reachability, collisions, or whether a step is safe
at the robot's current velocity. Bounds checking, a watchdog and an e-stop belong between
this library and your motors.

```python
policy.run(
    observe=robot.read_observation,
    act=robot.apply_action,
    validate=robot.is_safe,   # return False to stop the loop
    hold=robot.hold,          # called if it stops early, or if act() raises
    max_actions=250,
)
```

`max_actions` is required and keyword-only. There is no `run_forever()` — an unbounded loop
that moves a robot should not be startable by accident.

`validate=None` is allowed for bench and simulation work, and warns once so it cannot happen
silently on real hardware.

## Errors are typed

Because a controller reacts differently to each:

| | meaning | what to do |
|---|---|---|
| `AuthError` | key missing, revoked, expired | stop; retrying will not help |
| `OutOfCredit` | balance exhausted | stop and hold; top up |
| `InvalidRequest` | bad model, malformed observation, body too large | fix it; deterministic |
| `Unavailable` | upstream blip, or a cold worker | check `.warming` — see below |
| `ChunkExhausted` | asked for an action with an empty buffer and no observation | pass `observation=` every call |

## Configuration

```bash
export SEQUENCES_API_KEY=seq_live_...      # or pass api_key= to connect()
export SEQUENCES_BASE_URL=...              # for staging; defaults to production
```

Get a key at [app.generalsequences.com](https://app.generalsequences.com).

## Install footprint

One dependency: `httpx`. Python 3.9+.

A robot controller is often on a Jetson with a pinned, fragile Python environment, and ROS 2
Humble ships Python 3.10. Every transitive dependency is another chance for the install to
fail on the machine that actually matters.

## If your controller is not Python

The endpoints underneath are public and documented at
[generalsequences.com/docs](https://generalsequences.com/docs/). There is no private control
plane and nothing this package reaches that you cannot:

```bash
curl https://api.generalsequences.com/v1/act \
  -H "Authorization: Bearer $SEQUENCES_API_KEY" \
  -d '{"model":"accounts/sequences/models/pi05-droid","observation":{...}}'
```

Keeping that door open is deliberate — a vendor SDK that is the *only* supported way in locks
you to one language, and robot controllers are very often C++. But it is a door, not the front
entrance. The four items in the table at the top are work your client then has to do itself,
and [the reference](https://generalsequences.com/docs/python/) documents each precisely enough
to reimplement. Reaching for Python first is simply cheaper.
