Metadata-Version: 2.4
Name: hologres-dataframe
Version: 0.1.0.dev0
Summary: Lazy, pandas-style DataFrame API for Hologres
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: hologres-client>=0.1.2.dev0
Requires-Dist: psycopg[binary,pool]>=3.1
Provides-Extra: pandas
Requires-Dist: pandas>=1.3; extra == "pandas"
Requires-Dist: pyarrow>=12.0; extra == "pandas"
Provides-Extra: fc
Requires-Dist: alibabacloud-fc20230330<5,>=4.7.9; extra == "fc"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pandas>=1.3; extra == "dev"
Requires-Dist: pyarrow>=12.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"

# Hologres DataFrame API (Python)

A lazy, pandas-style DataFrame API for [Hologres](https://www.aliyun.com/product/bigdata/hologram).
Chain transformations in Python; the library compiles them into a single
PostgreSQL statement and pushes it down to Hologres, so computation stays in the
engine and only results come back.

Naming and execution model follow [Snowpark for Python](https://docs.snowflake.com/en/developer-guide/snowpark/python/index),
with snake_case only (`group_by`, never `groupBy`) and PostgreSQL semantics.

## Install

```bash
pip install --pre hologres-dataframe
```

The `0.1.0.dev0` pre-release requires `hologres-client>=0.1.2.dev0`, so the
matching client pre-release must be available from the same package index.

For development from this repository:

```bash
pip install -e ".[dev]"
```

`pandas` and `pyarrow` are optional; they are only needed for `to_pandas()`,
`from_pandas()` and local `read_files()`.

Function Compute deployment is also optional:

```bash
pip install "hologres-dataframe[fc]"
```

## Quickstart

```python
import hologres.dataframe as hg

session = hg.connect(
    host="xxx.hologres.aliyuncs.com",
    port=80,
    dbname="my_db",
    user="<access_id>",
    password="<access_key>",
)

(
    session.table("public.orders")
    .filter(hg.col("amount") > 100)  # WHERE amount > 100
    .with_column("tax", hg.col("amount") * 0.06)  # amount * 0.06 AS tax
    .select("order_id", "region", "amount", "tax")
    .sort(hg.col("amount"), ascending=False)  # ORDER BY amount DESC
    .show(10)
)  # <- the only step that runs SQL
```

Everything before `show()` is lazy: transformations just build a logical plan.
Actions (`show`, `collect`, `to_pandas`, `count`, `first`, `take`, `iter_rows`,
`write`, `write_files`, `as_dynamic_table`) are what trigger
execution. `session.refresh_table()` also executes immediately.

Like Snowpark, in-memory DataFrames should normally be created from a Session
so the returned DataFrame already knows where actions and writes should execute:

```python
df = session.from_dict({"id": [1, 2], "name": ["a", "b"]})
df.write("public.target")
```

The top-level `hg.from_dict` and `hg.from_records` functions remain available
for unbound SQL construction and compatibility. The legacy top-level
`hg.from_pandas` entry still accepts `session=...`; new code should use
`session.from_pandas(pdf)`. An unbound DataFrame can be compiled, but it cannot
execute an action or write until it is created through a Session-bound entry
point.

Materialize a lazy query as a Dynamic Table with safe create-if-absent behavior
by default:

```python
summary = session.table("public.orders").select("order_id", "region", "amount")
summary.as_dynamic_table(
    "public.dt_orders",
    freshness="10 minutes",
    refresh_mode="incremental",
    options={"cdc_format": "binlog"},
)
session.refresh_table("public.dt_orders")
```

Use `mode="errorifexists"` to reject an existing target or `mode="replace"` to
drop and recreate it in one transaction. Query the current contents through
`session.table("public.dt_orders")`.

Request Serverless Computing for a query by marking the DataFrame before its
action:

```python
rows = (
    session.table("public.orders")
    .filter(hg.col("amount") > 100)
    .serverless(priority=4, required_cores=64, max_cores=128)
    .collect()
)
```

The marker is lazy and immutable. At execution time the client applies
transaction-scoped `SET LOCAL` settings on the same connection as the query, so
they do not leak through the connection pool. Hologres still decides whether
the statement is eligible for Serverless execution; unsupported Serverless
settings raise an error instead of silently falling back.

## Function Compute UDFs

FC account/region settings belong to the Session. Alibaba Cloud credentials
come from the standard credential chain; do not put AK secrets in decorators:

```python
session = hg.connect(
    host="xxx.hologres.aliyuncs.com",
    port=80,
    dbname="my_db",
    user="<access_id>",
    password="<access_key>",
    fc_config={
        "endpoint": "123.cn-hangzhou-internal.fc.aliyuncs.com",
        "region": "cn-hangzhou",
        "role_arn": "acs:ram::123:role/fc-execution-role",
    },
)
```

Decorators only validate and retain Python definitions. The first action using
the function packages its source/imports, creates or updates the FC function,
and registers `LANGUAGE function_compute` in Hologres. Later actions in the
same Session reuse the registration.

```python
@hg.udf(packages=["numpy"])
def grade(n: int) -> str:
    return "high" if n >= 2 else "low"


scored = session.table("public.scores").select("id", grade("score").alias("grade"))
```

UDTFs are called directly in `select`; no lateral-join API is required:

```python
@hg.udtf(
    output_schema=hg.StructType(
        [
            hg.StructField("word", hg.StringType()),
        ]
    )
)
class SplitWords:
    def process(self, text: str):
        for word in text.split():
            yield (word,)


words = session.table("public.docs").select("id", SplitWords("text"))
```

Hologres Remote UDX cannot return PostgreSQL `record`, so `output_schema` must
contain exactly one field. FC endpoints must be internal endpoints in the same
region as Hologres. Third-party `packages` are installed as Linux wheels for
the selected `runtime`; package names without compatible wheels fail during
deployment instead of producing an incompatible ZIP.

## Runnable Examples

The [examples](examples/README.md) directory contains scripts that connect
directly to Hologres and exercise the implemented APIs:

- [basic_query.py](examples/basic_query.py): filter, derive, project, and sort.
- [join_and_aggregate.py](examples/join_and_aggregate.py): join and grouped
  aggregation inside Hologres.
- [write_rows.py](examples/write_rows.py): append or UPSERT in-memory records
  into an existing table.
- [oss_files.py](examples/oss_files.py): read and export OSS files through
  `EXTERNAL_FILES`.
- [dynamic_table.py](examples/dynamic_table.py): create and refresh a Dynamic
  Table.

Connection values and example table names come from environment variables; see
[examples/README.md](examples/README.md) for the required schemas and commands.

## Data types

Types are named after Snowpark/Spark and each knows the PostgreSQL type it
compiles to:

| Python API | PostgreSQL |
| --- | --- |
| `BooleanType()` | `boolean` |
| `ByteType()` | `"char"` (1 byte; character semantics, needs `::int4` for arithmetic) |
| `ShortType()` / `IntegerType()` / `LongType()` | `smallint` / `integer` / `bigint` |
| `FloatType()` / `DoubleType()` | `real` / `double precision` |
| `DecimalType(38, 2)` | `numeric(38,2)` |
| `StringType()` / `StringType(64)` | `text` / `varchar(64)` |
| `BinaryType()` | `bytea` |
| `DateType()` / `TimeType()` | `date` / `time` |
| `TimestampType()` / `TimestampType(TimestampTimeZone.TZ)` | `timestamp` / `timestamptz` |
| `JsonbType()` | `jsonb` |
| `GeographyType()` / `GeometryType()` | `geography` / `geometry` (PostGIS) |
| `ArrayType(StringType())` | `text[]` |
| `VectorType(float, 768)` | `vector(768)` (pgvector) |
| `StructType([StructField("word", StringType())])` | row shape, e.g. `RETURNS TABLE (word text)` |

`StructType` is a client-side row descriptor, not a column type. It is what
`df.schema` returns and what `@hg.udtf(output_schema=...)` and
`session.read_files(schema=...)` accept.

## Packaging

This distribution is named `hologres-dataframe` and installs into the
`hologres` namespace as `hologres.dataframe`, alongside `hologres-client`
(`../holo-client-py`), which it depends on for high-throughput write channels.

## Implementation status

| Step | Scope | Done |
| --- | --- | --- |
| C1 | Project scaffolding, data types, `Row`, exceptions | ✅ |
| C2 | Column expressions: `col`/`lit`, operators, `alias`/`asc`/`desc` | ✅ |
| C3 | Logical plan + PostgreSQL SQL generator | ✅ |
| C4 | Transformations: `select`/`filter`/`sort`/`limit`/... | ✅ |
| C5 | `Session`, `hg.connect`, actions, `df.schema`/`df.columns` | ✅ |
| C6 | `group_by`/`agg`, aggregate functions, `hg.function`/`hg.expr` | ✅ |
| C7a | `join` (USING/ON forms, relation-qualified columns) | ✅ |
| C7b | `union`/`union_all`, `explode` | ✅ |
| C8 | Catalog: `use_schema`/`use_database`, `current_*`, `list_*` | ✅ |
| C9a | `from_dict`/`from_records` (inline VALUES), `object_table` | ✅ |
| C9b | `from_pandas` (stage / scratch-table materialization) | ✅ |
| C9c | `read_files` for local csv/json/parquet | ✅ |
| C10 | `df.write`: SQL append/upsert, native overwrite + legacy fallback | ✅ |
| C11 | `read_files`/`write_files` (OSS `EXTERNAL_FILES`) | ✅ |
| C12 | AI functions (`hg.ai.*`) | ✅ |
| C13 | Vector search: distance methods, `search_vector` | |
| C14 | Dynamic Table (`as_dynamic_table`, `refresh_table`) | ✅ |
| C15 | `@hg.udf` / `@hg.udtf` (Function Compute remote UDF) | ✅ |
| C16 | `df.apply_agent` (Holo Agent bridge) | |
| C17 | Runnable Hologres examples | ✅ |
| C19 | DataFrame Serverless execution (`df.serverless(...)`) | ✅ |

## Tests

```bash
python3 -m pytest        # line coverage gate of 80% is enforced via pytest.ini
```

Tests do not need a live Hologres instance: the compiler is a pure
plan-to-SQL function and is verified by asserting on generated SQL, while the
execution layer is covered with fake connections.
