Metadata-Version: 2.4
Name: voltdbclient
Version: 16.1.0
Summary: VoltDB Python client: the original single-connection client and the topology-aware VoltClient
Author-email: Volt Active Data <support@voltactivedata.com>
License-Expression: MIT
Project-URL: Homepage, https://www.voltactivedata.com/
Project-URL: Documentation, https://docs.voltactivedata.com/
Project-URL: Source, https://github.com/VoltDB/voltdb-client-python
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: extras
Requires-Dist: pyjks; extra == "extras"
Requires-Dist: cryptography; extra == "extras"
Requires-Dist: gssapi; extra == "extras"
Dynamic: license-file

# VoltDB Python Client Libraries

This repository contains two Python client libraries for VoltDB:

| Module            | Main classes                      | Role                                 |
|-------------------|-----------------------------------|--------------------------------------|
| `voltdbclient.py` | `FastSerializer`, `VoltProcedure` | The original client and wire codec   |
| `voltclient.py`   | `VoltClient`, `VoltBulkLoader`    | Topology-aware client built on it    |

`voltdbclient` is the original client: a single connection to a single server, called synchronously, plus the wire-protocol serialization for all of VoltDB's data types. `voltclient` builds on it — `VoltClient` holds a connection to every cluster node, routes each call to the node that will execute it, supports asynchronous calls, and recovers automatically from cluster changes. New applications should generally start with `VoltClient`; existing `FastSerializer` code needs **no changes** and keeps working as-is.

Because `VoltClient` uses `voltdbclient`'s serialization underneath, the two modules share the same type constants and return the same `VoltResponse` / `VoltTable` objects — they interoperate naturally, and you can migrate one call site at a time.

What `VoltClient` gives an application:

- **One client for the whole cluster.** Name any node; the client discovers the rest and keeps a connection to each.
- **Client affinity.** Each single-partition call goes straight to the node that leads its partition.
- **Synchronous and asynchronous calls.** `call` blocks; `call_async` returns a `concurrent.futures.Future`, which `asyncio` code can await.
- **Cluster changes without restarts.** The cluster pushes topology changes to the client, so node failures, rejoins and elastic expansion are followed automatically.
- **Bulk loading.** `VoltBulkLoader` loads a table in per-partition batches, one transaction per batch.
- **A bound on load.** `max_outstanding` caps the calls in flight, which keeps latency predictable.

Both modules are pure Python with no third-party dependencies. `voltclient` and the PyPI package require Python 3.9 or later; `voltdbclient.py` on its own still accepts Python 3.6.

## Installation

Both modules are published to PyPI as one package:

```
pip install voltdbclient
```

Reading Java keystores for TLS and Kerberos authentication need optional third-party packages: `pip install "voltdbclient[extras]"`.

The VoltDB distribution also ships both modules in `lib/python`, where VoltDB's own command-line tools load them from. To use that copy, put the directory on your `PYTHONPATH` (or append it to `sys.path`). `voltclient` imports `voltdbclient`, so if you copy the files rather than the directory, keep the two together.

## Quick start

Connect, call a procedure, and read the result:

```python
from voltclient import VoltClient, VoltType

with VoltClient(hosts=["voltsvr1", "voltsvr2"]) as client:
    response = client.call("Vote",
                           [VoltType.BIGINT, VoltType.INTEGER],
                           [5551234567, 2])
    if response.status == 1:  # ClientResponse.SUCCESS
        for row in response.tables[0].tuples:
            print(row)
```

Give `hosts` any subset of the cluster: the client discovers the full membership at connect time and opens one connection per node. Use the client as a context manager (or call `close()`) so its background threads and sockets are always torn down.

## Connecting

All `VoltClient` constructor arguments are keywords with defaults:

| Argument | Default | Meaning |
|---|---|---|
| `hosts` | | Names or addresses of the nodes to connect to first. Any subset of the cluster will do. |
| `port` | `21212` | The client port, the same on every node. |
| `username`, `password` | `""` | Credentials, for a database with security enabled. |
| `usessl` | `False` | Connect over TLS. |
| `ssl_config_file` | `None` | A properties file naming the keystore and truststore, in the format VoltDB's Java tools use. |
| `default_cacerts` | `True` | With TLS and no `ssl_config_file`, check the server's certificate against the system's default CA certificates. If `False`, the certificate is not checked at all. |
| `kerberos` | `False` | Authenticate with Kerberos (needs `voltdbclient[extras]`). |
| `connect_timeout` | `8` | Seconds to wait for a connection to open. |
| `procedure_timeout` | `120` | Default timeout of a call, in seconds. A `timeout` passed to the call overrides it. |
| `max_outstanding` | `3000` | Calls in flight at once; see [Bounding load](#bounding-load-max_outstanding). |
| `connection_response_timeout` | `30` | A connection idle this many seconds is pinged, and declared dead if the ping goes unanswered as long. |
| `initial_backoff`, `max_backoff`, `reconnect_jitter` | `0.5`, `30`, `0.2` | Reconnect backoff in seconds, and the fraction of random jitter added to each wait. |

For example, a TLS connection with credentials:

```python
client = VoltClient(hosts=["voltsvr1"], username="app", password="secret",
                    usessl=True, ssl_config_file="/etc/voltdb/ssl.properties")
```

## Calls

- `call(proc_name, param_types, params, timeout=None)` — send and block for the `VoltResponse`.
- `call_async(...)` — send and return a `concurrent.futures.Future` immediately. Compose with the standard library:

```python
import concurrent.futures

futures = [client.call_async("STORE.upsert",
                             [VoltType.STRING, VoltType.VARBINARY],
                             [key, value])
           for key, value in batch]
for f in concurrent.futures.as_completed(futures):
    if f.result().status != 1:
        ...
```

`asyncio` applications can bridge a call with `await asyncio.wrap_future(client.call_async(...))`.

One caution: callbacks registered with `Future.add_done_callback` run on the client's network reader threads. Keep them short and never block in them (no `call` or `call_async`, no lock waits) — parse, record, hand off.

- `call_all_partitions(proc_name, param_types=None, params=None, timeout=None)` — run a procedure once on every partition and return one `PartitionCallResult` per partition (sorted by partition id), each carrying either the partition's `response` or the `error` that ended its leg; `ok` summarizes. Works with procedures partitioned on their first parameter and with directed procedures. Leave the partitioning parameter out: the client supplies a key for each partition and prepends it, so `param_types` and `params` carry only the remaining parameters. `call_all_partitions_async` is the Future form. One timeout budget covers the whole fan-out.

```python
# A procedure partitioned on its first parameter, which is its only one:
# no parameters to pass.
for result in client.call_all_partitions("CountRowsInPartition"):
    if result.ok:
        print(result.partition_id, result.response.tables[0].tuples)
    else:
        print(result.partition_id, result.error or result.response.statusString)
```

- `get_partition_for_parameter(vtype, value)` — which partition a value hashes to, using the same elastic hashinator the cluster uses. Useful for batching work by partition before sending it.
- `outstanding_requests` — the number of calls currently in flight.

## Bulk loading

`VoltBulkLoader` loads rows into one table the way the Java client's bulk loader does: rows are grouped by partition and each batch is sent as a single `@LoadSinglepartitionTable` transaction to the partition's leader (`@LoadMultipartitionTable` for a replicated table). One transaction per batch instead of per row cuts both client and server work per row.

```python
from voltclient import VoltBulkLoader

def failed(row, outcome):          # a VoltResponse or an exception
    print("not loaded:", row, outcome)

with VoltBulkLoader(client, "store", batch_size=1000, upsert=True,
                    failure_callback=failed) as loader:
    for key, value in rows:
        loader.insert_row(key, value)   # values in column order
print(loader.completed_row_count, loader.failed_row_count)
```

- The table's columns, types and partition column are read from the catalog when the loader is created. `upsert=True` needs a primary key on the table.
- `insert_row` checks each value against its column (no silent coercion or truncation: a `str` for an `INTEGER` column, an out-of-range number, or a NULL partition key fails that row through `failure_callback` with a `VoltTypeError`). TIMESTAMP accepts `datetime` (naive means local time) or integer microseconds since the epoch; DECIMAL accepts `decimal.Decimal` or `int`; VARBINARY accepts any bytes-like value.
- If the server rejects a batch (a constraint violation, a value too long for its column), none of its rows are inserted, so they are retried one at a time and only the rows that actually fail are reported. A batch whose outcome is unknown (connection lost or timed out) is retried the same way. In insert mode, a row that did load may then be reported as a duplicate-key failure, or inserted twice on a table without a primary key. Use `upsert=True` if that matters.
- `failure_callback` runs on a client network thread. Keep it quick, and don't call the loader (`insert_row`, `flush`, `drain`, `close`) or the client from it. To re-insert a corrected row, hand it to another thread.
- `insert_rows(rows)` inserts an iterable of rows. `timeout` (default: the client's `procedure_timeout`) applies to each batch. `completed_row_count`, `failed_row_count` and `outstanding_row_count` report progress, and `column_names` / `column_types` describe the table.
- `flush()` sends partially filled batches, `drain()` also waits until every row is loaded or failed (including rows other threads insert meanwhile), and `close()` (or leaving the `with` block) stops accepting rows, then drains. `insert_row` may be called from several threads. Without a `failure_callback`, the first failure is logged as a warning and `close()` logs the total.
- A `batch_size` of 500–1,000 is a good default: much larger batches leave too few in flight per partition to keep the cluster busy. `max_batches_in_flight` (default 32) bounds memory and server queue depth.
- Supported column types: TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT, VARCHAR, TIMESTAMP, DECIMAL, VARBINARY.

## Responses and types

Both clients return the same objects, defined in `voltdbclient`:

- `VoltResponse` — `status` (integer; 1 is success — see the VoltDB documentation for the full list), `statusString` (human-readable meaning), `roundtripTime` (milliseconds), `tables` (list of `VoltTable`, or `None`), `exception` (a `VoltException`, or `None`), and `appStatus` / `appStatusString` (the status a procedure set for itself).
- `VoltTable` — `columns` (list of `VoltColumn`, each with `name` and `type`) and `tuples` (list of rows; each row is a list of values already converted to Python types).
- `VoltException` — `type` (one of the `VOLTEXCEPTION_*` constants: `NONE`, `EEEXCEPTION`, `SQLEXCEPTION`, `CONSTRAINTFAILURE`, `GENERIC`) and `message`.

Parameter types are declared with the `VoltType` constants (`VoltType.BIGINT`, `VoltType.STRING`, ...; these alias the original module's `FastSerializer.VOLTTYPE_*` values), and values map between VoltDB and Python as follows:

| VoltDB type | Python type         |
|-------------|---------------------|
| NULL        | `None`              |
| BOOLEAN     | `bool`              |
| TINYINT     | `int`               |
| SMALLINT    | `int`               |
| INTEGER     | `int`               |
| BIGINT      | `int`               |
| FLOAT       | `float`             |
| DECIMAL     | `decimal.Decimal`   |
| STRING      | `str`               |
| TIMESTAMP   | `datetime.datetime` |
| DATE        | `datetime.date`     |
| VARBINARY   | `bytes` to send; `array.array('B')` when read |
| GEOGRAPHY_POINT | a `(longitude, latitude)` tuple of floats |
| GEOGRAPHY   | `Geography` |

Python does not distinguish integer widths, but a value serialized as a given type must fit it — an out-of-range value raises an error. Array parameters are passed as Python lists, declared with their element type. `Geography.geographyFromWKT("POLYGON((...))")` builds a `Geography` from well-known text and `str()` turns one back into it; `Geography.pointFromWKT("POINT(lng lat)")` gives a point tuple. `Geography` and `XYZPoint` are importable from `voltclient` as well.

## Topology awareness

The client keeps its view of the cluster current by **subscription, not polling**: it registers for server push notifications on one of its connections, and the cluster pushes refreshed topology and procedure-catalog data whenever they change. In practice this means:

- **Client affinity.** Single-partition procedure calls are sent directly to the node leading their partition, and multi-partition calls to the multi-partition initiator — skipping a server-side forwarding hop. Calls the client cannot place (unknown procedures, `@AdHoc`, unhashable values) simply round-robin; routing degrades, it never fails a call. `affinity_stats()` reports affinity-routed vs round-robin sends per connection.
- **Self-healing membership.** Node failures, rejoins, and elastic cluster expansions flow into the connection pool and the routing tables without an application restart. A lost connection's in-flight calls fail (see the failure contract below); the pool redials and re-converges in the background with capped exponential backoff.
- **No stale-routing hazard.** A call routed on a topology that just changed is forwarded or transparently restarted by the cluster on the correct node — a small latency cost, never a correctness one.

`client.topology` exposes the current snapshot (hosts, partition leaders, procedure partitioning) and `refresh_topology()` remains available as a manual backstop for applications that want their own schedule on top. If the server declines the subscription (for example while the database is paused), the client keeps working and tries again two minutes later.

## Bounding load: `max_outstanding`

`call_async` admits at most `max_outstanding` calls in flight (default 3000); a caller past the cap blocks until a completion frees a slot, and the wait draws on that call's own timeout. This bound is the client's load governor, and tuning it is the single most useful performance knob:

- Throughput rises with the cap only until the cluster (or the client process) saturates — the "knee" of the curve. Past the knee, extra outstanding calls add **only queue latency**: throughput stays flat while tail latencies grow by orders of magnitude.
- For latency-sensitive applications, set `max_outstanding` near the knee measured on your hardware (often tens, not thousands).
- The 3000 default suits bulk loading, where total throughput matters and per-call latency does not.

## Failure contract

All errors derive from `VoltClientError`:

- `VoltConnectionError` — the connection carrying a call was lost before its response arrived. The call's outcome is **unknown**: it may or may not have executed. The client never resends such a call, because only the application can know whether a retry is safe — retry freely if the operation is idempotent, otherwise look up the outcome first. Sends are written by a background thread, so a call made just as its connection dies may be reported through its `Future` rather than raised by `call_async`; handle both.
- `VoltNoConnectionsError` — no live connections to send on.
- `VoltTimeoutError` — no response within the call's timeout (default 120s), or the call could not obtain an outstanding-request slot in time.
- `VoltTypeError` — a parameter could not be serialized as the declared type (or, for `VoltBulkLoader`, a row value does not fit its column).

Server-side failures (constraint violations, procedure exceptions) are not Python exceptions: they arrive as a normal `VoltResponse` with a non-success `status` and `statusString`.

## Performance characteristics

What to expect, and why:

- A synchronous single-connection client's throughput ceiling is one call per round-trip: `FastSerializer` used directly cannot exceed `1/RTT` calls per second. `VoltClient`'s pipelining removes that wall: its ceiling is the Python interpreter itself — one CPU core's worth of serialization and dispatch, independent of network latency. Writes and reads are batched on the socket, so one process sustains on the order of 50,000 single-row calls per second against a local server (measured with small rows; your numbers depend on row size and hardware).
- Latency at moderate concurrency is competitive with any client: per-call time is dominated by the network and the server, and affinity routing removes a network hop from every single-partition call.
- A single Python process will not saturate a VoltDB cluster row by row — that is a property of CPython, not of this client. For loading tables, `VoltBulkLoader` runs one server transaction per batch instead of per row, which is several times faster; beyond that, run several processes.

These figures come from a benchmark run in VoltDB's own testing. It drives the `voltkv` workload (the schema, keys and default procedures of the voltkv example app), comparing direct `FastSerializer` use with `VoltClient` across a sweep of `max_outstanding` values, then compares loading rows one at a time with `VoltBulkLoader` at several batch sizes.

## Migrating from `voltdbclient`

Existing `FastSerializer` code needs no changes. When you do want a call site on the new client, the mapping is mechanical — the types and response objects are the same:

```python
# before
fs = FastSerializer("voltsvr1", 21212)
proc = VoltProcedure(fs, "Vote", [FastSerializer.VOLTTYPE_BIGINT,
                                  FastSerializer.VOLTTYPE_INTEGER])
response = proc.call([5551234567, 2])

# after
client = VoltClient(hosts=["voltsvr1"])
response = client.call("Vote", [VoltType.BIGINT, VoltType.INTEGER],
                       [5551234567, 2])
```

Differences to be aware of:

- One `VoltClient` replaces a `FastSerializer` *per node* plus your own pooling and failover code.
- Procedures are not pre-declared; pass the parameter types with each call.
- Authentication, TLS, and Kerberos options pass through the `VoltClient` constructor to `FastSerializer` unchanged.

## Using `voltdbclient` directly

The original API remains fully supported and is the right tool for small scripts talking to one server. Create a `FastSerializer` connection, declare each stored procedure as a `VoltProcedure` with its parameter types, and call it:

```python
from voltdbclient import *

client = FastSerializer("voltsvr1", 21212)  # optional: username, password, usessl, ...
proc = VoltProcedure(client, "@SystemInformation",
                     [FastSerializer.VOLTTYPE_STRING])
response = proc.call(["OVERVIEW"])
client.close()
```

Details worth knowing:

- `VoltProcedure.call(params, response=True, timeout=None)` — with `response=False` the call is sent without waiting for the result (returns `None`); `timeout` (seconds) raises `socket.timeout` if the server does not respond in time.
- The `FastSerializer` constructor also accepts `dump_file_path`, which writes all bytes sent and received to a file — occasionally useful for debugging at the wire level.
- **Thread safety differs between the modules.** Nothing in `voltdbclient` is thread-safe: give each thread its own `FastSerializer`. A `VoltClient`, by contrast, is designed to be shared — many threads may call one instance concurrently.

## Logging

The client logs through the standard `logging` module under the logger name `voltclient` (connection lifecycle at INFO/DEBUG, degraded-routing and subscription warnings at WARNING). It adds no handlers of its own.

Full API details are in the module docstrings: `pydoc voltclient`.

## Examples

The [examples](https://github.com/VoltDB/voltdb-client-python/tree/master/examples) directory has small programs for the schemas in the VoltDB kit's tutorials and examples:

- `voltclient_helloworld.py` — Hello World with `VoltClient`: asynchronous inserts, then a query.
- `helloworld.py` — the same with `FastSerializer`.
- `voter.py` — the Voter example's client, with `FastSerializer`.

## Contributing

See [CONTRIBUTING.md](https://github.com/VoltDB/voltdb-client-python/blob/master/CONTRIBUTING.md) for how to build the package and run the tests.
