112 lines
5.1 KiB
Markdown
112 lines
5.1 KiB
Markdown
# @suaveplan/i18n-formatter-segmenter - Intl.Segmenter wrapper for the @suaveplan/i18n family — locale-aware text segmentation by grapheme, word, or sentence (compound-emoji-aware grapheme boundaries, ICU word-boundary rules, sentence segmentation). Per-(locale, granularity) cache, Zod-validated options, OTel histogram + cache-hit counter, structured I18nError mapping. Brand-new formatter — closes the gap from the original i18n-formatters change scope. Universal leaf package.
|
|
|
|
## Summary
|
|
|
|
Wrapper around `Intl.Segmenter` for the `@suaveplan/i18n` family —
|
|
locale-aware text segmentation by grapheme, word, or sentence.
|
|
Compound emoji count as a single grapheme; word boundaries follow
|
|
ICU rules (handles non-space-separated scripts like Thai / Khmer /
|
|
Japanese correctly); sentence boundaries follow ICU sentence-break
|
|
rules.
|
|
|
|
This is a **brand-new formatter** in the i18n family — `Intl.Segmenter`
|
|
was listed in the original `i18n-formatters` change scope but was
|
|
never implemented in the monolith. This package closes that gap.
|
|
|
|
<!-- begin:auto-docs -->
|
|
## Documentation tree
|
|
|
|
- [formatter](./src/formatter/formatter.md) — `SegmenterFormatter` and the `segmentText` / `formatSegments` standalone helpers expose `Intl.Segmenter` for the `@suav…
|
|
- [formatter](./src/formatter/formatter.md) — `SegmenterFormatter` and the `segmentText` / `formatSegments` standalone helpers expose `Intl.Segmenter` for the `@suav…
|
|
- [helpers](./src/helpers/helpers.md) — The three ergonomic helpers (`countGraphemes`, `splitWords`, `splitSentences`) pre-bind `segmentText` and `formatSegmen…
|
|
- [helpers](./src/helpers/helpers.md) — The three ergonomic helpers (`countGraphemes`, `splitWords`, `splitSentences`) pre-bind `segmentText` and `formatSegmen…
|
|
- [observability](./src/observability/observability.md) — The Segmenter formatter exports two OpenTelemetry instruments and a small in-memory state ring so spec-conformance test…
|
|
- [observability](./src/observability/observability.md) — The Segmenter formatter exports two OpenTelemetry instruments and a small in-memory state ring so spec-conformance test…
|
|
<!-- end:auto-docs -->
|
|
|
|
## Why this exists
|
|
|
|
The 14-package i18n modular split (cross-cutting design
|
|
`2026-05-05-i18n-modular-split.md`) lifts each `Intl.*` wrapper into
|
|
its own sibling package and adds the two formatters
|
|
(`Collator` + `Segmenter`) that the original change scope listed but
|
|
never shipped. Locale-aware segmentation is the right primitive for
|
|
character-limited inputs (tweet-style counters), full-text search
|
|
tokenisation, and sentence-aware UI truncation.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
bun add @suaveplan/i18n-formatter-segmenter
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```ts
|
|
import {
|
|
SegmenterFormatter,
|
|
segmentText,
|
|
formatSegments,
|
|
countGraphemes,
|
|
splitWords,
|
|
splitSentences,
|
|
} from "@suaveplan/i18n-formatter-segmenter";
|
|
|
|
segmentText("ab", "en-GB"); // ["a", "b"]
|
|
segmentText("hello world", "en-GB", { granularity: "word" });
|
|
// ["hello", " ", "world"]
|
|
|
|
countGraphemes("👨👩👧", "en-GB"); // 1 (single grapheme)
|
|
splitWords("hello, world!", "en-GB"); // ["hello", "world"]
|
|
splitSentences("Hi. How are you?", "en-GB");
|
|
// ["Hi.", "How are you?"]
|
|
|
|
const f = new SegmenterFormatter({ granularity: "word" });
|
|
f.segmentWithMetadata("hi!", "en-GB");
|
|
// [
|
|
// { segment: "hi", index: 0, isWordLike: true },
|
|
// { segment: "!", index: 2, isWordLike: false },
|
|
// ]
|
|
```
|
|
|
|
## Modules
|
|
|
|
| Subpath | Description |
|
|
|---|---|
|
|
| `@suaveplan/i18n-formatter-segmenter` | Root barrel — class + standalones + helpers + observability |
|
|
| `@suaveplan/i18n-formatter-segmenter/formatter` | `SegmenterFormatter`, `segmentText`, `formatSegments`, types, schema |
|
|
| `@suaveplan/i18n-formatter-segmenter/helpers` | `countGraphemes`, `splitWords`, `splitSentences` |
|
|
|
|
## Modules Documentation
|
|
|
|
- [formatter.md](src/formatter.md) — `SegmenterFormatter`, `segmentText`, `formatSegments`, `clearSegmenterCache`
|
|
- [helpers.md](src/helpers.md) — three ergonomic helpers
|
|
- [observability.md](src/observability.md) — OTel histogram + cache-hit counter
|
|
|
|
## Public Surface (snapshot)
|
|
|
|
| Export | Kind | Origin |
|
|
|---|---|---|
|
|
| `SegmenterFormatter` | class | `formatter.ts` |
|
|
| `segmentText` | function | `formatter.ts` |
|
|
| `formatSegments` | function | `formatter.ts` |
|
|
| `clearSegmenterCache` | function | `formatter.ts` |
|
|
| `SegmenterFormatterOptionsSchema` | Zod schema | `formatter.schemas.ts` |
|
|
| `SegmenterFormatterOptions` | type | `formatter.types.ts` |
|
|
| `SegmenterGranularity` | type | `formatter.types.ts` |
|
|
| `SegmentData` | type | `formatter.types.ts` |
|
|
| `SegmenterFormatterCacheKey` | type | `formatter.types.ts` |
|
|
| `countGraphemes` | function | `helpers.ts` |
|
|
| `splitWords` | function | `helpers.ts` |
|
|
| `splitSentences` | function | `helpers.ts` |
|
|
| `recordSegmenterFormat` | function | `observability.ts` |
|
|
| `recordSegmenterCacheHit` | function | `observability.ts` |
|
|
| `getSegmenterMetricsState` | function | `observability.ts` |
|
|
| `resetSegmenterMetricsState` | function | `observability.ts` |
|
|
|
|
## Versioning
|
|
|
|
Part of the 14-package i18n modular-split rollout. Stays at `0.1.0`
|
|
until the slim core (Wave 6) cuts the family `1.0.0` release.
|
|
|