Metadata-Version: 2.5
Name: polystring
Version: 1.0.0
Summary: Span-level language detection for mixed-language text
Project-URL: Homepage, https://github.com/saadlohani/polystring
Project-URL: Documentation, https://github.com/saadlohani/polystring#readme
Project-URL: Bug Tracker, https://github.com/saadlohani/polystring/issues
Project-URL: Changelog, https://github.com/saadlohani/polystring/blob/main/CHANGELOG.md
Author-email: Saad Khan <saadlohani@yahoo.com>
License: MIT License
        
        Copyright (c) 2026 Saad Khan
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: code-switching,language-detection,language-identification,mixed-language,multilingual,nlp,romanization,script-detection
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: mypy>=1.9; extra == 'dev'
Requires-Dist: pre-commit>=3.5; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.16; extra == 'dev'
Description-Content-Type: text/markdown

# polystring

[![PyPI](https://img.shields.io/pypi/v/polystring.svg)](https://pypi.org/project/polystring/)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](https://pypi.org/project/polystring/)
[![CI](https://github.com/saadlohani/polystring/actions/workflows/ci.yml/badge.svg)](https://github.com/saadlohani/polystring/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Span-level language detection for mixed-language text.

Most language detectors return one label for a whole string. polystring
returns a labelled span for every part of it, with character offsets, a
confidence and the script it is written in, so a sentence that switches
language mid-way is not flattened into a single wrong guess.

```python
import polystring

for span in polystring.detect("je ne peux pas venir demain, can you call me later?").spans:
    print(f"{span.lang:8} {span.confidence:.2f}  {span.text!r}")
```

```
fr       1.00  'je ne peux pas venir demain, '
en       0.94  'can you call me later?'
```

## Install

```bash
pip install polystring
```

Python 3.9 or newer. The model ships with the package as data files, so the
first call works offline.

## Usage

```python
import polystring

detection = polystring.detect("Здравствуйте, my friend and 你好")

for span in detection.spans:
    print(span.start, span.end, span.lang, span.script, round(span.confidence, 2))
```

```
0   14  ru  Cyrl  0.74
14  28  en  Latn  0.66
28  30  zh  Hani  0.55
```

Reuse a `Detector` when you are calling it repeatedly; the model is loaded
once and shared.

```python
detector = polystring.Detector(abstain_below=0.5)
results = [detector.detect(line) for line in lines]
```

### Detection

| attribute   | meaning                                          |
| ----------- | ------------------------------------------------ |
| `text`      | the input, NFC-normalized unless you disabled it |
| `spans`     | every span, in order, covering the text exactly  |
| `languages` | the set of languages found                       |
| `is_mixed`  | whether more than one language is present        |

### Span

| attribute      | meaning                                                            |
| -------------- | ------------------------------------------------------------------ |
| `start`, `end` | Unicode code-point offsets; `span.text == text[start:end]`         |
| `text`         | the span's own text                                                |
| `kind`         | `LANG`, `SPECIAL`, `NEUTRAL` or `UND`                              |
| `lang`         | BCP-47 tag such as `en`, `ur-Latn`; `None` unless `kind` is `LANG` |
| `language`     | the language alone, without the script subtag                      |
| `orthography`  | `"native"` or `"romanized"`                                        |
| `script`       | ISO 15924 code such as `Latn`, `Arab`, `Deva`                      |
| `confidence`   | calibrated probability in `[0, 1]`                                 |
| `candidates`   | ranked alternatives, for genuinely ambiguous spans                 |

Spans always tile the input: they are sorted, non-overlapping, and together
cover `[0, len(text))`. URLs, mentions, hashtags, emoji and bare numerals
come back as `SPECIAL` spans and are never given a language. Where the
evidence does not support an answer, the span is `UND` rather than a guess.

### Detector options

| option          | default | meaning                                     |
| --------------- | ------- | ------------------------------------------- |
| `abstain_below` | `0.40`  | confidence below which a span becomes `UND` |
| `normalize`     | `True`  | apply NFC normalization before detecting    |

### Errors

All exceptions derive from `polystring.PolystringError`.

| exception            | meaning                                                                                            |
| -------------------- | -------------------------------------------------------------------------------------------------- |
| `InputError`         | bad input, such as a non-string or an unrecognised option                                          |
| `ModelNotFoundError` | the packaged model data is missing or unreadable, so the install is damaged                        |
| `ContractViolation`  | the output failed its own tiling invariant, which is a bug in polystring rather than in your input |
| `InvalidTagError`    | an emitted tag failed BCP-47 validation, also a bug in polystring                                  |

## How it works

Three stages, and which one answers explains most of the accuracy.

**Extraction** claims the parts of the text that are not language: URLs,
mentions, hashtags, emoji, numerals.

**Script segmentation** splits the rest into runs of a single script. A
script used by only one language is answered here, with no model call,
because there is nothing to disambiguate. A script shared by several
languages becomes a window for the next stage, carrying the language it
resolves to on script evidence alone. Perso-Arabic and Cyrillic have
marker-codepoint tables, since some of their languages have exclusive
letters.

**Disambiguation** decides the windows that remain. It shortlists the
languages plausibly present in the window, scores each token against that
shortlist, and decodes the result as a sequence, so a token's neighbours can
outweigh its own weak evidence and a genuine one-word switch still survives.

The classifier is a multinomial Naive Bayes model over character n-grams,
with two further sources of evidence folded into the same score: whole-word
probabilities, and an orthographic penalty for a language that does not write
a letter present in the text. A separate term asks whether the input is a
language at all, which is what makes keyboard mash come back as `UND`.

## Coverage

394 language and script pairs across 30 scripts. Labels are canonical
`<ISO 639-3>_<ISO 15924>` internally and BCP-47 tags at the API boundary, and
the two round-trip:

```python
from polystring import inventory

inventory.normalize("ur-Latn")  # 'urd_Latn'
inventory.to_legacy("urd_Latn")  # 'ur-Latn'
inventory.scripts_for("urd")  # ('Arab', 'Latn')
```

Because language and script are separate axes, Urdu in Nastaliq and Roman
Urdu are different labels with different difficulty, and `span.orthography`
tells them apart without string-matching the tag.

### Romanized text

Romanized labels are detected and allocated to their language, covering the
informal writing and typing patterns people use in several languages:

```python
polystring.detect("hava cok guzel").spans[0].lang  # 'tr'
polystring.detect("kya tum mere saath chaloge").spans[0].lang  # 'hi-Latn'
```

### Confusion groups

Some labels cannot be reliably told apart, and splitting a dialect continuum
finely would otherwise make the detector abstain more, not less. Abstention
is decided on the pooled confidence of the winner's group, while the reported
confidence stays with the individual label:

```python
from polystring import confusion

sorted(confusion.group_of("hin_Latn"))  # ['hif_Latn', 'hin_Latn', 'urd_Latn']
```

## Accuracy

Measured on held-out text with no training overlap, one sample per language
so that a common language cannot carry the average.

|                              |       |
| ---------------------------- | ----- |
| exact                        | 88.6% |
| right family, wrong member   | 4.5%  |
| wrong                        | 5.5%  |
| abstained                    | 1.4%  |
| median per-language accuracy | 97.7% |

Against the other Python detectors, using the same held-out text for each.
Every system is scored only on the languages it actually supports, so a
missing label counts against coverage rather than against accuracy:

|            | languages | single language | finds both |
| ---------- | --------- | --------------- | ---------- |
| polystring | 394       | 96.3%           | 91.2%      |
| langdetect | 55        | 94.2%           | n/a        |
| lingua     | 75        | 93.5%           | 70.0%      |
| langid     | 97        | 82.5%           | n/a        |

The test set is drawn from the 74 languages both lingua and polystring
support. langdetect has labels for 50 of them and langid for 66, and each is
scored on its own subset.

"Finds both" splices two languages into one line and asks for both back.
langid and langdetect return one label per document by design, so the column
does not apply to them.

## Development

```bash
git clone https://github.com/saadlohani/polystring
cd polystring
pip install -e ".[dev]"
pytest
```

Linting and typing are enforced in CI:

```bash
ruff format --check src/ tests/
ruff check src/ tests/
mypy --strict src/polystring/
```

The same formatting and lint checks run locally as pre-commit hooks:

```bash
pre-commit install
```

The corpus and the scripts that build the model from it are not in this
repository; the trained artifacts in `src/polystring/data/` are what the
library needs, and they are versioned with it.

## License

MIT. See [LICENSE](LICENSE).
