Metadata-Version: 2.4
Name: StructureCloud
Version: 0.0.3
Summary: Standard formating and easy access to 3D structural datasets for machine learning. currently under development...
Author-email: Ty Perez <ty.jperez@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/TyJPerez/StructureCloud
Project-URL: Issues, https://github.com/TyJPerez/StructureCloud/issues
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch
Requires-Dist: torch_geometric
Requires-Dist: huggingface_hub>=0.20
Requires-Dist: numpy>=1.21
Requires-Dist: tqdm>=4.60
Requires-Dist: matplotlib>=3.5
Requires-Dist: plotly>=5.0
Provides-Extra: dev
Requires-Dist: matplotlib>=3.5; extra == "dev"
Requires-Dist: scipy>=1.7; extra == "dev"
Requires-Dist: rdkit>=2022.3; extra == "dev"
Requires-Dist: datasets>=2.0; extra == "dev"
Dynamic: license-file

# StructureCloud

A collection of 3D atomistic (point-cloud) datasets for machine learning on molecules and
materials — property prediction, representation learning, and generative modeling. Every
dataset is exposed through a single, uniform loader that returns
[PyTorch Geometric](https://pytorch-geometric.readthedocs.io/) `Data` objects, and is stored
in a **memory-mapped, collated format** so that many large datasets can be **stacked and
trained on together without exhausting RAM**.

## Why StructureCloud

Atomistic datasets ship in many incompatible formats (jsonl, LMDB, parquet, per-source
schemas) and decoding them one row at a time is slow and RAM-hungry. StructureCloud solves
this with three conventions shared across every dataset:

- **One uniform sample schema.** Each item is a PyG `Data` with `pos` `[N,3]` (Å) and `z`
  `[N]` (atomic numbers); periodic crystals add `cell` `[1,3,3]` and `pbc` `[1,3]`. Labels
  carry short, descriptive keys — never a bare `y`/`dy`.
- **Collated + memory-mapped storage.** Each split is preprocessed once into a single
  `.pt` file where *everything is a tensor* (variable-length atom arrays are concatenated
  with offset tables; strings are packed). `torch.load(mmap=True)` pages tensors in from
  disk through the shared OS page cache, so stacking N datasets costs the combined working
  set, not the sum of their sizes.
- **Download-only loaders.** Preprocessed files live on the Hugging Face Hub under
  `StructureCloud/{dataset}`; the loader downloads the split it needs and maps it — no
  building from raw at load time.

## Install

```bash
pip install StructureCloud
```

Or from source (editable), e.g. for development or to preprocess new datasets:

```bash
git clone https://github.com/TyJPerez/StructureCloud.git
cd StructureCloud
pip install -e .
```

**Dependencies.** The core install pulls in everything needed to load, stack, and plot
datasets: `torch`, `torch_geometric`, `huggingface_hub`, `numpy`, `tqdm`, `matplotlib`, and
`plotly`. Split files download automatically from the Hugging Face Hub on first use and are
cached locally.

> **Install `torch` and `torch_geometric` first, matched to your platform.** Their wheels are
> tied to your CUDA/CPU setup, so they are intentionally left unpinned here — follow the
> official [PyTorch](https://pytorch.org/get-started/locally/) and
> [PyG](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html)
> install instructions for your system, then install StructureCloud.

One optional extra covers dataset *building* — preprocessing a new dataset into the collated
format and generating its figures/`stats.json`. It is not needed to load or plot the datasets
that already ship:

```bash
pip install "StructureCloud[dev]"   # matplotlib, scipy, rdkit, datasets
```

## Quickstart

```python
from StructureCloud.Datasets import QM9

# Downloads and memory-maps StructureCloud/QM9 preprocessed 'train' split
ds = QM9(split='train')          # pass mmap=False to load fully into RAM
print(len(ds))                   # number of structures

data = ds[0]                     # a torch_geometric.data.Data
data.pos                         # [N, 3] float32 positions (Angstrom)
data.z                           # [N]    int64 atomic numbers
data.smiles                      # str metadata
data.y                           # (19,) DFT property vector (QM9-specific)
```

Every loader takes the common arguments `split`, `transform`, `mmap`, and `root` (custom
cache location); dataset-specific arguments (`label`, `subset`, `components`, `sample`, …)
are documented in each loader's docstring and in that dataset's `README.md`. Batch with a
standard PyG `DataLoader`:

```python
from torch_geometric.loader import DataLoader
loader = DataLoader(ds, batch_size=64, shuffle=True)
```

## Available datasets

Import any of these from `StructureCloud.Datasets`. Each dataset name links to its full
documentation (schema, labels, splits, statistics, and citations).

### Small organic molecules (molecular, non-periodic)

| Dataset | Size | Description | Splits |
|---|---|---|---|
| [`QM9`](src/StructureCloud/Datasets/QM9/README.md) | 130,831 molecules | CHONF, ≤9 heavy atoms; relaxed geometry + 19 DFT properties | `train` |
| [`PCQM4Mv2`](src/StructureCloud/Datasets/PCQM4Mv2/README.md) (alias `PCQ`) | 3,378,550 molecules | DFT-relaxed geometry + HOMO–LUMO gap (OGB-LSC / PubChemQC) | `train` |
| [`GEOM`](src/StructureCloud/Datasets/GEOM/README.md) | ~37M conformers / ~450k molecules | Conformer **ensemble** (energies + Boltzmann weights); variants `GEOM`/`GEOM64`/`GEOM32`/`GEOM10`/`GEOM1`, collections `drugs`/`qm9`/`molnet` | per-collection |
| [`SPICE1`](src/StructureCloud/Datasets/SPICE/README.md) / [`SPICE2`](src/StructureCloud/Datasets/SPICE/README.md) | 19,238 mol / 1.11M conf · 113,999 mol / 2.0M conf | QC single-points with per-atom **forces** + per-conformer energies (ωB97M-D3(BJ)/def2-TZVPPD); conformer ensemble | `all` |
| [`MD17`](src/StructureCloud/Datasets/MD17/README.md) | 10 molecules × ~100k frames | rMD17 MD trajectories; energy + forces at PBE/def2-SVP; each molecule is its own split | per-molecule |
| [`ANI1x`](src/StructureCloud/Datasets/ANI1x/README.md) | 3,114 formula buckets / 4.96M conformers | Off-equilibrium H/C/N/O conformers; DFT energy + **forces** (ωB97x) with a CCSD(T) subset (ANI-1ccx); conformer ensemble | `all` |
| [`OrbNetDenali`](src/StructureCloud/Datasets/OrbNetDenali/README.md) | 212,920 molecules / 2.34M conformers | Off-equilibrium drug-like conformers spanning protomers/tautomers/salt complexes/counterions (17 elements, charge −8…+6); **two** energies — DFT ωB97X-D3/def2-TZVP **and** GFN1-xTB — for Δ-learning; **no forces**; conformer ensemble | `all` |
| [`OMol25`](src/StructureCloud/Datasets/OMol25/README.md) | 3.99M (`4m`) – 34.3M (`neutral`) structures | Large molecular DFT single-points at fixed theory; energy + **forces**, variable charge/spin; biomolecules, metal complexes, electrolytes | `4m`/`neutral`/`val`/`val_neutral`/`test` |
| [`MISATO_QM`](src/StructureCloud/Datasets/MISATO_QM/README.md) | 19,413 ligands | Protein-bound ligands with **GFN2-xTB** QM properties: 7 per-molecule scalars + 23 per-atom descriptors + bond graph (aromatic orders included) | `all` |

### Materials (periodic crystals)

| Dataset | Size | Description | Splits |
|---|---|---|---|
| [`MP20`](src/StructureCloud/Datasets/MP20/README.md) | 45,229 crystals | ≤20-atom Materials Project crystals; standard generation benchmark | `train`/`val`/`test` |
| [`AlexMP20`](src/StructureCloud/Datasets/AlexMP20/README.md) | 675,204 crystals | ≤20-atom crystals from Alexandria + MP; stability, band gap, elastic/magnetic properties | `train`/`val`/`test`/`all` |
| [`Carbon24`](src/StructureCloud/Datasets/carbon_24/README.md) | 10,153 crystals | Carbon-only crystals from ab-initio random structure search; energy/atom label | `train`/`val`/`test` |
| [`Perov5`](src/StructureCloud/Datasets/perov_5/README.md) | 18,928 crystals | Perovskite ABX₃, fixed 5-atom cell; formation energy + band gap | `train`/`val`/`test` |
| [`WBM`](src/StructureCloud/Datasets/WBM/README.md) | 256,963 crystals | Matbench Discovery test set; unrelaxed→relaxed formation energy / stability (85 elements) | `test` |
| [`Matbench`](src/StructureCloud/Datasets/matbench/README.md) | 9 tasks | The 9 structure-based Matbench property tasks; fixed 5-fold CV (`mp_gap`, `mp_e_form`, `perovskites`, `phonons`, `dielectric`, `jdft2d`, `log_gvrh`, `log_kvrh`, `mp_is_metal`) | per-task |
| [`MPtrj`](src/StructureCloud/Datasets/MPtrj/README.md) | 454,594 trajectories / 1.58M frames | CHGNet training set; MP relaxation-trajectory frames; energy + **forces** + stress + magmoms; universal MLIP training | `all` |
| [`OMat24`](src/StructureCloud/Datasets/OMat24/README.md) | ~102M single-points | Largest inorganic-materials DFT set (VASP PBE/PBE+U); non-equilibrium configs; energy + **forces** + stress; MLIP pretraining | `train`/`val`/`1m`/`1m_val`/`1m_test` |
| [`sAlex`](src/StructureCloud/Datasets/sAlex/README.md) | 10.4M / 553k rows | Matbench-Discovery-compliant Alexandria PBE slice; OMat24 fine-tuning corpus; energy + **forces** + stress | `train`/`val` |
| [`MatPES`](src/StructureCloud/Datasets/MatPES/README.md) | 433,189 (PBE) / 386,544 (r²SCAN) | Foundational PES dataset; MD-sampled crystal single-points; energy + **forces** + stress at two functionals | `pbe`/`r2scan` |
| [`QMOF`](src/StructureCloud/Datasets/QMOF/README.md) | 20,372 MOFs | DFT properties of periodic metal–organic frameworks; band gaps, charges/spins, pore descriptors (79 elements) | `all` |
| [`OMC25`](src/StructureCloud/Datasets/OMC25/README.md) | 218,841 crystals / 26.26M frames | Organic **molecular crystals** — packings of whole molecules, a modality no other set here covers (OMat24/sAlex/MatPES are inorganic, QMOF is frameworks, OMol25 is isolated molecules); PBE-D3 relaxation-trajectory frames; energy + **forces** + stress; 12 elements; conformer ensemble. `split='val'` is the default — `train` is a ~79 GB chunked download | `val`/`train` |
| [`COD`](src/StructureCloud/Datasets/COD/README.md) | 530,868 crystals | **Experimental** crystal structures (organic + inorganic + organometallic + mineral) from the Crystallography Open Database, CC0 — the measured counterpart to the computed inorganic sets. 97 elements, 161M atom rows. **No energies/forces/stress**, so no `y`. Disorder is **kept and filtered at load** (`ordered_only=True` gives 358,646); occupancy stored as parsed, never clamped | `all` (COD ships no canonical split) |

### Proteins and biomolecules

| Dataset | Size | Description | Splits |
|---|---|---|---|
| [`LBA`](src/StructureCloud/Datasets/LBA/README.md) | 4,463 complexes | Protein–ligand binding affinity (Atom3D / PDBbind); full protein preserved, available as `protein`/`pocket`/`ligand` components | `train`/`val`/`test`/`all` |
| [`HiQBind`](src/StructureCloud/Datasets/HiQBind/README.md) | 32,275 complexes | Protein–ligand binding affinity (open PDBbind successor); `protein`/`ligand`/`water`/`additives` components, load-time `pocket_cutoff` | no split ships |
| [`PLINDER`](src/StructureCloud/Datasets/PLINDER/README.md) | 311,008 systems | Protein–ligand structures with **leakage-controlled splits** (docking / pose / cofolding); `protein`/`ligand`/`apo`/`pred` components, ~750 annotations, **no affinity target** | `train`/`val`/`test`/`all` |
| [`MISATO_MD`](src/StructureCloud/Datasets/MISATO_MD/README.md) | 16,972 complexes / 1.7M frames | Protein–ligand **MD trajectory ensemble** (100 snapshots over 8 ns); per-atom flexibility + per-frame MMGBSA/RMSD/distance/bSASA, subsamples `full`/`md10`/`md1`, **no affinity target** | `train`/`val`/`test`/`all` |

### RNA (nucleic-acid structures)

| Dataset | Size | Description | Splits |
|---|---|---|---|
| [`RNA3DB`](src/StructureCloud/Datasets/RNA3DB/README.md) | 15,441 chains | Single-chain RNA 3D structures from the PDB with a **leakage-aware split**: 2,199 sequence clusters grouped into 142 structurally-dissimilar components (141 Rfam-connected, plus `component_0` for the chains whose cluster representative matched no family). Per-chain sequence + Rfam family; **no energy/force/affinity target** — the coordinates are the prediction target | `train`/`test`/`all` |

## Documentation

Every other README and doc in the repo, reachable from here:

**Guides**
- [Datasets usage guide](src/StructureCloud/Datasets/README.md) — loading, batching, **stacking**, **conformer samplers**, **transforms**, and **loading-speed / memory** tuning (including multi-GPU / DDP).
- [Models](src/StructureCloud/models/README.md) — model architectures (early / experimental).
- [Equivariant Transformer (ET) architecture](src/StructureCloud/models/arch/ET/README.md) — build/compile notes for the on-the-fly neighbor-list C++ extension.

**Per-dataset docs** — one README per dataset, linked from each name in
[Available datasets](#available-datasets) above:
[QM9](src/StructureCloud/Datasets/QM9/README.md) ·
[PCQM4Mv2](src/StructureCloud/Datasets/PCQM4Mv2/README.md) ·
[GEOM](src/StructureCloud/Datasets/GEOM/README.md) ·
[SPICE](src/StructureCloud/Datasets/SPICE/README.md) ·
[MD17](src/StructureCloud/Datasets/MD17/README.md) ·
[ANI1x](src/StructureCloud/Datasets/ANI1x/README.md) ·
[OrbNetDenali](src/StructureCloud/Datasets/OrbNetDenali/README.md) ·
[OMol25](src/StructureCloud/Datasets/OMol25/README.md) ·
[MP20](src/StructureCloud/Datasets/MP20/README.md) ·
[AlexMP20](src/StructureCloud/Datasets/AlexMP20/README.md) ·
[Carbon24](src/StructureCloud/Datasets/carbon_24/README.md) ·
[Perov5](src/StructureCloud/Datasets/perov_5/README.md) ·
[WBM](src/StructureCloud/Datasets/WBM/README.md) ·
[Matbench](src/StructureCloud/Datasets/matbench/README.md) ·
[MPtrj](src/StructureCloud/Datasets/MPtrj/README.md) ·
[OMat24](src/StructureCloud/Datasets/OMat24/README.md) ·
[sAlex](src/StructureCloud/Datasets/sAlex/README.md) ·
[MatPES](src/StructureCloud/Datasets/MatPES/README.md) ·
[QMOF](src/StructureCloud/Datasets/QMOF/README.md) ·
[OMC25](src/StructureCloud/Datasets/OMC25/README.md) ·
[COD](src/StructureCloud/Datasets/COD/README.md) ·
[LBA](src/StructureCloud/Datasets/LBA/README.md) ·
[HiQBind](src/StructureCloud/Datasets/HiQBind/README.md) ·
[PLINDER](src/StructureCloud/Datasets/PLINDER/README.md) ·
[MISATO_MD](src/StructureCloud/Datasets/MISATO_MD/README.md) ·
[MISATO_QM](src/StructureCloud/Datasets/MISATO_QM/README.md) ·
[RNA3DB](src/StructureCloud/Datasets/RNA3DB/README.md)

**Contributing / internals** (under `.agents/docs/`)
- [About: dataset conventions & the collated / memory-mapped format](.agents/docs/about.md)
- [Adding a dataset — guidelines](.agents/docs/adding_dataset/guidelines.md)
- [Adding a dataset — planning](.agents/docs/adding_dataset/planning.md)
- [Dataset README template](.agents/docs/adding_dataset/dataset_readme_template.md)
- [Validation-tests reference](.agents/docs/validation_tests.md)

## Stacking datasets

The core feature: concatenate arbitrary datasets into one memory-mapped training set with a
uniform key schema. `StackedDataset` tags each sample's origin (`AddTag`) and standardizes
keys (`StandardizeKeys` fills a `cell` bounding box / `pbc` for molecular samples and a
unified `str_id`), so a mixed molecular + periodic batch collates through a single PyG
`DataLoader`.

```python
from StructureCloud.Datasets import AlexMP20, GEOM, PCQM4Mv2, StackedDataset

stk = StackedDataset(
    [AlexMP20('all', label=None), GEOM('GEOM10', 'drugs'), PCQM4Mv2()],
    tags=['alexmp20', 'geom10', 'pcq'],
    keep_keys=['pos', 'z', 'cell', 'pbc', 'natoms', 'str_id', 'tag'],
)
```

Because every dataset is memory-mapped, a stack that is several GB on disk stays at a small
resident footprint, with cold pages evicted under memory pressure and shared across
DataLoader workers.

## Conformer-ensemble datasets

`GEOM`, `SPICE1`/`SPICE2`, `ANI1x`, `OrbNetDenali`, `MPtrj`, `OMC25`, and `MISATO_MD` store many
conformers/frames per entity.
Their loaders return one conformer per access with selectable sampling: `sample=True`
(random conformer, default), `sample=False` (flatten — index every entity/conformer pair),
or `sample_fn=…` with a shared sampler from `StructureCloud.Datasets.samplers`
(`sample_by_relative`, `sample_by_boltzmann`). See the
[Datasets usage guide](src/StructureCloud/Datasets/README.md#conformer-ensemble-samplers)
for the full sampler reference and examples.


## Also in this repo

Beyond the datasets, the package includes early/experimental modules that are not the focus
of this README: `StructureCloud.models` (an equivariant-transformer architecture),
`StructureCloud.utils` (graph construction, augmentation, structure viewing), and
`StructureCloud.chem_tools` (3D atom/material plotting, atomic properties).

## Contributing a dataset

Dataset conventions, the collated/memory-mapped format, and the step-by-step process for
adding a new dataset are documented under `.agents/docs/` — start with
[`about.md`](.agents/docs/about.md) and
[`adding_dataset/guidelines.md`](.agents/docs/adding_dataset/guidelines.md). New loaders
subclass `CollatedStructureDataset` (`Datasets/collated.py`), ship a `README.md`, a
distribution figure, and a `stats.json`, and register in `Datasets/__init__.py` and the
verification harness (`scripts/verify_dataset.py`).

## Citation

If StructureCloud is useful in your work, please cite the library:

```bibtex
@software{perez_structurecloud_2026,
  author  = {Perez, Ty},
  title   = {{StructureCloud}: uniform, memory-mapped 3D atomistic datasets for machine learning},
  year    = {2026},
  version = {0.0.3},
  url     = {https://github.com/TyJPerez/StructureCloud}
}
```

**Also cite the datasets you use.** StructureCloud repackages existing datasets; it does not
replace their original sources. Every dataset ships a `Sources & citation` section in its own
README (linked from [Available datasets](#available-datasets)) naming the paper and data
release to cite, and any license terms that come with it.

## License

MIT — see [LICENSE](LICENSE).
