Metadata-Version: 2.4
Name: pytesprocess
Version: 0.1.1
Summary: Detector data processing for pytesdaqx acquisitions
Author-email: Bruno Serfass <serfass@berkeley.edu>, Samuel Watkins <samwatkins@berkeley.edu>
License: MIT License
        
        Copyright (c) 2022 Samuel Watkins
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/spice-herald/pytesprocess
Project-URL: Repository, https://github.com/spice-herald/pytesprocess
Keywords: detector,TES,DAQ,signal processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.26
Requires-Dist: scipy
Requires-Dist: matplotlib
Requires-Dist: PyYAML
Requires-Dist: qetpy>=1.8.6
Requires-Dist: pandas
Requires-Dist: pytesdaqx
Requires-Dist: humanfriendly
Requires-Dist: vaex
Requires-Dist: pyarrow
Requires-Dist: lmfit
Requires-Dist: cloudpickle
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Dynamic: license-file

# pytesprocess

`pytesprocess` is the detector-processing layer used with raw acquisitions written
by `pytesdaqx`. It provides random-event selection, software triggering,
feature extraction, salting, IV/dIdV processing, noise/filter generation, and
persistence of processed event products as Vaex HDF5 dataframes.

The current code targets Python 3.11+ and the modern `pytesdaqx` acquisition
model (`acquisition` + `stream`).

## Installation

For a development checkout:

```bash
pip install -e .
```

Runtime dependencies and the installed CLI entry point are defined entirely in
`pyproject.toml`.

## Command-line interface

Installing the package provides:

```bash
pytesprocess --help
```

The normal event-processing workflow is selected directly from the top-level
command:

```bash
pytesprocess ACQUISITION \
    --config processing.yaml \
    --steps randoms trigger feature
```

`--steps` is mandatory. It accepts either whitespace-separated or
comma-separated values, including mixed input:

```bash
--steps randoms trigger feature
--steps randoms,trigger,feature
--steps randoms,trigger feature
```

All are equivalent. The execution order is normalized internally to:

```text
randoms -> salting -> trigger -> feature
```

The separate detector-characterization/filter workflows remain explicit:

```bash
pytesprocess filter ACQUISITION --config processing.yaml
pytesprocess ivsweep ACQUISITION
```

Example:

```bash
pytesprocess acquisition_I2_D20260807_T170432.zarr \
    --config continuous_data_processing_v2.yaml \
    --steps randoms,trigger,feature \
    --nrandoms 300 \
    --nevents 300 \
    --ncores 4
```

See [docs/user/cli.md](docs/user/cli.md) for the current CLI behavior.

## Configuration

New processing configuration uses `config_version: 2` and separates workflow
sections explicitly:

```yaml
config_version: 2

resources:
  filter_file: /path/to/filterdata.hdf5

trigger:
  global: {}
  channels: {}

feature:
  global:
    trace_length_msec: 20
    pretrigger_length_msec: 10
  presets: {}
  channels: {}

salting:
  global: {}
  channels: {}

filter: {}
```

Feature trace lengths are specified in milliseconds in v2 configuration. The
resolver converts them to samples using the selected acquisition sample rate.
Channel selectors include `all`, shell-style globs such as `Z1P*`, comma groups
for applying one block to several independent channels, and explicit
multi-channel expressions such as `A|B`.

See [docs/user/configuration.md](docs/user/configuration.md).

## Processed dataframe identity

Each Vaex processing product has one dataframe-group identity:

```text
dataframe_group_name
dataframe_group_id
dataframe_group_number
dataframe_file_index
processing_label
```

For example:

```text
trigger_I2_D20260908_T123456/
  trigger_I2_D20260908_T123456_F0001.hdf5
  trigger_I2_D20260908_T123456_F0002.hdf5
```

`F####` is a processing task/output shard index, not a second stream or series
identifier. Raw provenance uses canonical stream fields such as `stream_id`,
`stream_number`, and `stream_trigger_index`.

See [docs/user/outputs.md](docs/user/outputs.md).

## Package structure

The current package layout intentionally separates reusable analysis objects
from acquisition-processing workflows:

```text
pytesprocess/
  core/       reusable analysis/data objects and algorithms
  process/    raw/dataframe processing executors
  config/     YAML loading, selection, resolution, validation
  salting/    salt metadata generation and waveform injection
  workflows/  multi-step orchestration used by the CLI
  cli/        command-line parsing and dispatch
  utils/      shared utilities and HDF5/dataframe helpers
```

See [docs/developer/architecture.md](docs/developer/architecture.md) for the
current developer-oriented architecture.

## Notes on multiprocessing

The existing Vaex/PyArrow and numerical-library thread limits are intentional.
They were introduced to avoid thread oversubscription and unstable/slower
multicore processing. They should not be removed or relocated without dedicated
multicore benchmarking.
