first commit

This commit is contained in:
2026-05-29 08:20:56 +01:00
commit 949299f58d
6368 changed files with 527176 additions and 0 deletions

View File

@@ -0,0 +1,13 @@
# `search`
Full-text search, BM25, fuzzy matching, and faceted search.
<!-- begin:auto-docs -->
## Packages in `search`
1 package.
| Package | Description |
|---|---|
| [`@suaveplan/search`](./search/README.md) | Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support. |
<!-- end:auto-docs -->

View File

@@ -0,0 +1 @@
Full-text search, BM25, fuzzy matching, and faceted search.

View File

@@ -0,0 +1,54 @@
# @suaveplan/search - Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support
## Summary
Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support.
<!-- begin:auto-docs -->
## Documentation tree
- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
<!-- end:auto-docs --> - [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ - [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ - [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ - [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_
## Modules
- [MemorySearchEngine](src/engine/memory-engine/memory-engine.md) — in-memory BM25 search engine
- [Tokenizer](src/text/tokenizer/tokenizer.md) — text tokenization and stemming
- [Trigram](src/text/trigram/trigram.md) — trigram-based fuzzy similarity
- [Scorer](src/text/scorer/scorer.md) — BM25 and TF-IDF scoring functions
- [SearchError](src/errors/search-error.md) — structured errors with stable codes
- [Schemas](src/schemas/search.schemas.md) — Zod schemas for all query types
- [Types](src/types/search.types.md) — shared TypeScript interfaces
## Quick Start
```ts
import { createMemorySearchEngine } from "@suaveplan/search";
interface Product { id: string; name: string; price: number }
const engine = createMemorySearchEngine<Product>({
fields: [
{ name: "name", type: "text", searchable: true },
{ name: "price", type: "number", sortable: true },
],
});
await engine.index([
{ id: "1", name: "Laptop Pro", price: 999 },
{ id: "2", name: "Laptop Air", price: 799 },
]);
const result = await engine.search({ text: "laptop" });
console.log(result.hits.length); // 2
```

View File

@@ -0,0 +1,85 @@
# memory-engine
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides `createMemorySearchEngine`, an in-memory implementation of `SearchEngine` backed by a term-frequency inverted index and BM25 scoring. Designed for single-process use, testing, and small-to-medium datasets (up to ~100k documents). Supports full-text search, structured filters, fuzzy trigram matching, faceted aggregations, and keyword highlighting.
## Features
- Inverted index built from tokenized searchable fields
- BM25 scoring (k1=1.2, b=0.75) for relevance ranking
- Trigram fuzzy matching with configurable threshold
- Faceted search: bucket counts for `facetable` fields
- Highlight extraction with configurable tag (default `<mark>`)
- Structured filters: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`
- Multi-field sorting with direction
- Cursor and offset pagination
- `suggest(prefix)` for autocomplete from indexed terms
- OTel spans on `index`, `search`, and `suggest`
## Usage: Basic Example
```ts
import { createMemorySearchEngine } from "@suaveplan/search";
interface Product { id: string; name: string; price: number }
const engine = createMemorySearchEngine<Product>({
fields: [
{ name: "name", type: "text", searchable: true },
{ name: "price", type: "number", sortable: true },
],
});
await engine.index([{ id: "1", name: "Laptop Pro", price: 999 }]);
const result = await engine.search({ text: "laptop" });
```
## Usage: Advanced Example
```ts
const engine = createMemorySearchEngine<Product>({
fields: [
{ name: "name", type: "text", searchable: true, facetable: false },
{ name: "category", type: "keyword", facetable: true },
],
fuzzyThreshold: 0.4,
stemming: true,
});
await engine.index(products);
const result = await engine.search({
text: "laptops",
filters: [{ field: "price", op: "lte", value: 1000 }],
sort: [{ field: "price", direction: "asc" }],
pagination: { type: "offset", offset: 0, limit: 20 },
fuzzy: true,
});
console.log(result.facets?.["category"]);
```
## API Reference
### `createMemorySearchEngine<T>(options)`
- `options.fields: FieldConfig<T>[]` — field schema with type and searchability flags
- `options.stemming?: boolean` — enable simple English stemming
- `options.stopWords?: string[]` — custom stop-word list
- `options.fuzzyThreshold?: number` — trigram similarity threshold for fuzzy (default 0.3)
### `SearchEngine<T>`
- `index(documents, options?)` — adds documents; replaces if `replace !== false`
- `search(query, options?)` — returns `SearchResult<T>` with hits, total, facets
- `suggest(prefix, options?)` — returns terms from inverted index starting with prefix
- `remove(ids)` — removes documents by ID
- `clear()` — empties the entire index
## Implementation Notes
The inverted index maps `term → Map<docId, tf>`. Field lengths per document are tracked for BM25 length normalization. Fuzzy matching runs after BM25 scoring and assigns a 50% score weight to fuzzy hits so they rank below exact matches.

View File

@@ -0,0 +1,55 @@
# search-error
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Defines the structured error type and stable error-code registry for `@suaveplan/search`. All runtime failures from indexing, querying, cursor pagination, and field validation surface as `SearchError` instances with a machine-readable `code` field.
## Features
- `SEARCH_CODES` registry via `@suaveplan/error`'s `defineCodes` — codes namespaced `SEARCH.*`
- `SearchError` extends `SuaveplanError` for cause-chain traversal
- Five codes: `INDEX_FAILED`, `QUERY_INVALID`, `ENGINE_NOT_READY`, `FIELD_UNKNOWN`, `CURSOR_INVALID`
- `SearchCode` union type for exhaustive switch statements
## Usage: Basic Example
```ts
import { SearchError, SEARCH_CODES } from "@suaveplan/search";
throw new SearchError(SEARCH_CODES.FIELD_UNKNOWN, `Field "price" is not indexed`);
```
## Usage: Advanced Example
```ts
import { SearchError, SEARCH_CODES } from "@suaveplan/search";
function handleSearchError(err: unknown): void {
if (err instanceof SearchError) {
if (err.code === SEARCH_CODES.CURSOR_INVALID) {
// Reset pagination cursor
}
}
}
```
## API Reference
### `SEARCH_CODES`
- `SEARCH_CODES.INDEX_FAILED` — indexing operation failed
- `SEARCH_CODES.QUERY_INVALID` — query syntax or parameter is invalid
- `SEARCH_CODES.ENGINE_NOT_READY` — engine has no documents indexed
- `SEARCH_CODES.FIELD_UNKNOWN` — referenced field not in schema
- `SEARCH_CODES.CURSOR_INVALID` — pagination cursor is malformed or expired
### `SearchError`
Extends `SuaveplanError<SearchCode>`. Constructor: `(code, message, options?)`.
## Implementation Notes
Codes are prefixed with `"SEARCH."` by `defineCodes`, producing strings like `"SEARCH.QUERY_INVALID"`.

View File

@@ -0,0 +1,63 @@
# search.schemas
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides Zod validation schemas for all public search query and configuration types in `@suaveplan/search`. Used at API boundaries to validate incoming queries, engine configurations, and filter specifications before they reach the in-memory engine.
## Features
- `SearchQuerySchema` — validates the full search query structure including text, filters, sort, pagination, fuzzy, and highlightTag
- `SearchFilterSchema` — validates individual filter clauses with the 8-operator enum
- `SortSpecSchema` — validates sort field and direction
- `PaginationSpecSchema` — validates cursor/offset pagination with positive limit
- `FieldConfigSchema` — validates field definitions
- `MemorySearchEngineOptionsSchema` — validates engine construction options
## Usage: Basic Example
```ts
import { SearchQuerySchema } from "@suaveplan/search";
const query = SearchQuerySchema.parse({
text: "laptop",
filters: [{ field: "price", op: "lte", value: 1000 }],
pagination: { type: "offset", offset: 0, limit: 20 },
});
```
## Usage: Advanced Example
```ts
import { MemorySearchEngineOptionsSchema } from "@suaveplan/search";
const opts = MemorySearchEngineOptionsSchema.parse({
fields: [{ name: "title", type: "text", searchable: true }],
fuzzyThreshold: 0.4,
stemming: true,
});
```
## API Reference
### `SearchQuerySchema`
Validates `{ text?, filters?, sort?, pagination?, fuzzy?, highlightTag? }`.
### `SearchFilterSchema`
Validates `{ field: string, op: FilterOp, value: unknown }`.
### `PaginationSpecSchema`
Validates `{ type: "cursor" | "offset", cursor?, offset?, limit: positive }`.
### `MemorySearchEngineOptionsSchema`
Validates `{ fields, stemming?, stopWords?, fuzzyThreshold? }`. Passthrough for injectable clock.
## Implementation Notes
The `value` field in `SearchFilterSchema` uses `z.unknown()` since filter values can be any type depending on the field type. The `fuzzy` field accepts either a boolean or `{ threshold: number }` via a Zod union.

View File

@@ -0,0 +1,69 @@
# scorer
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides BM25 and TF-IDF scoring functions used by the in-memory search engine to rank search results by relevance. BM25 (Best Match 25) is the de facto standard ranking algorithm used by Elasticsearch, Solr, and Lucene. TF-IDF is provided as a simpler alternative.
## Features
- `bm25Score(options)` — Okapi BM25 with configurable k1 (default 1.2) and b (default 0.75)
- `tfIdfScore(tf, idf)` — classic TF × IDF product
- `computeIdf(df, n)` — IDF computation with logarithmic smoothing
- All functions are pure and stateless; no external dependencies
## Usage: Basic Example
```ts
import { bm25Score } from "@suaveplan/search";
const score = bm25Score({
tf: 3, // term appears 3 times in document
df: 10, // term appears in 10 documents
n: 100, // total documents in corpus
fieldLength: 50,
avgFieldLength: 30,
});
```
## Usage: Advanced Example
```ts
import { bm25Score, tfIdfScore, computeIdf } from "@suaveplan/search";
// Custom k1/b for short-field optimization
const titleScore = bm25Score({
tf: 1, df: 5, n: 1000, fieldLength: 5, avgFieldLength: 8,
k1: 0.9, b: 0.4,
});
// TF-IDF comparison
const idf = computeIdf(5, 1000);
const tfidf = tfIdfScore(2, idf);
```
## API Reference
### `bm25Score(options: Bm25Options)`
- `tf` — term frequency
- `df` — document frequency
- `n` — corpus size
- `fieldLength` — length of field being scored
- `avgFieldLength` — average field length across corpus
- `k1?` — saturation parameter (default 1.2)
- `b?` — length normalization (default 0.75)
### `tfIdfScore(tf, idf)`
Returns `tf * idf`.
### `computeIdf(df, n)`
Returns `log(1 + n/df)`. Returns `0` for zero inputs.
## Implementation Notes
BM25 IDF uses the Robertson-Sparck Jones formula: `log(1 + (N - df + 0.5) / (df + 0.5))`. This variant is always non-negative. The field-length normalization parameter `b = 0.75` slightly penalises longer documents, which improves precision for titles and short-text fields.

View File

@@ -0,0 +1,59 @@
# tokenizer
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides text tokenization and normalization utilities for the in-memory search engine. Tokenization converts raw text into a stream of normalized tokens used to build the inverted index and to process search queries. The tokenizer handles stop-word removal, case normalization, and optional simple English stemming.
## Features
- `tokenize(text, options?)` — splits text on non-word characters, removes stop words, optionally stems
- `normalizeToken(token)` — lowercases and trims a single token
- `simpleStem(word)` — strips common English suffixes: `-tion`, `-ing`, `-est`, `-ed`, `-er`, `-ly`, `-s`
- Default stop-word list of 40+ common English words
- Case sensitivity and stop-word list are configurable per call
## Usage: Basic Example
```ts
import { tokenize } from "@suaveplan/search";
const tokens = tokenize("The quick brown fox jumps over the lazy dog");
console.log(tokens); // ["quick", "brown", "fox", "jump", ...]
```
## Usage: Advanced Example
```ts
import { tokenize, simpleStem } from "@suaveplan/search";
const tokens = tokenize("Running faster than expected", {
stemming: true,
stopWords: ["than"],
});
console.log(simpleStem("computing")); // "comput"
```
## API Reference
### `tokenize(text, options?)`
- `options.caseSensitive?: boolean` (default `false`)
- `options.stopWords?: string[]` — replaces default stop-word list
- `options.stemming?: boolean` — applies `simpleStem` to each token (default `false`)
- Returns `string[]`
### `normalizeToken(token)`
Lowercases and trims a single token. Used internally.
### `simpleStem(word)`
Returns the stem of an English word by stripping common suffixes. Words ≤3 characters are returned unchanged.
## Implementation Notes
The stemmer is intentionally simple (suffix-stripping, no Snowball/Porter) to avoid external dependencies. For production use with high-accuracy stemming requirements, a custom tokenizer implementing the same interface can be passed via `MemorySearchEngineOptions`.

View File

@@ -0,0 +1,51 @@
# trigram
<!-- begin:auto-docs -->
_(no submodules)_
<!-- end:auto-docs -->
## Purpose
Provides trigram-based fuzzy matching utilities for the in-memory search engine. Trigrams are overlapping 3-character substrings that capture structural similarity between strings — they are language-agnostic, robust to minor typos, and require no external libraries.
## Features
- `buildTrigrams(text)` — generates a `Set<string>` of all 3-character substrings, padded with `##` for start/end boundary capture
- `trigramSimilarity(a, b)` — computes Jaccard similarity (intersection / union) over the trigram sets, returning a value in `[0, 1]`
- Case-insensitive: both functions normalize input to lowercase before processing
- Exact matches always return `1.0`
## Usage: Basic Example
```ts
import { trigramSimilarity } from "@suaveplan/search";
const score = trigramSimilarity("hello", "helo");
console.log(score); // ~0.71
```
## Usage: Advanced Example
```ts
import { buildTrigrams, trigramSimilarity } from "@suaveplan/search";
const threshold = 0.5;
const candidates = ["hello", "help", "world", "dog"];
const matches = candidates.filter(
(c) => trigramSimilarity("helo", c) >= threshold
);
console.log(matches); // ["hello", "help"]
```
## API Reference
### `buildTrigrams(text)`
Returns `Set<string>` of all trigrams in `text` (case-insensitive, `##`-padded).
### `trigramSimilarity(a, b)`
Returns `number` in `[0, 1]`. Uses Jaccard coefficient: `|intersection| / |union|`. Returns `1` for identical strings or empty trigram sets.
## Implementation Notes
The `##` padding ensures that short strings (1-2 characters) still generate meaningful trigrams that capture their start and end positions. The Jaccard coefficient is symmetric: `trigramSimilarity(a, b) === trigramSimilarity(b, a)`. Complexity is O(n + m) where n and m are the trigram set sizes.

View File

@@ -0,0 +1,70 @@
# search.types
## Purpose
Shared TypeScript type definitions for the `@suaveplan/search` package. Defines all public contracts: search documents, queries, filters, results, field configurations, and engine interfaces. Centralising types here prevents circular imports between the engine, text utilities, and schema modules.
## Features
- `SearchDocument` — base interface requiring `id: string` plus arbitrary fields
- `SearchQuery` — full query structure with text, filters, sort, pagination, and fuzzy options
- `SearchFilter` — structured field filter with 8 operators (`eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`)
- `SearchResult<T>` / `SearchHit<T>` — typed result container with score, highlights, and facets
- `FieldConfig<T>` — per-field schema: type, searchable, sortable, facetable, weight
- `MemorySearchEngineOptions<T>` — full configuration for the in-memory engine
- `SearchEngine<T>` — interface for `index`, `search`, `suggest`, `remove`, `clear`
- `Bm25Options` — parameters for BM25 scoring function
## Usage: Basic Example
```ts
import type { SearchDocument, SearchQuery, FieldConfig } from "@suaveplan/search";
interface Product extends SearchDocument {
name: string;
price: number;
}
const field: FieldConfig<Product> = {
name: "name",
type: "text",
searchable: true,
facetable: false,
};
```
## Usage: Advanced Example
```ts
import type { SearchQuery, SearchFilter } from "@suaveplan/search";
const query: SearchQuery = {
text: "laptop",
filters: [{ field: "price", op: "lte", value: 1000 }],
sort: [{ field: "price", direction: "asc" }],
pagination: { type: "offset", offset: 0, limit: 20 },
fuzzy: true,
};
```
## API Reference
### `SearchDocument`
Base interface: `{ id: string; [field: string]: unknown }`.
### `SearchQuery`
Full query: `text?`, `filters?`, `sort?`, `pagination?`, `fuzzy?`, `highlightTag?`.
### `FieldConfig<T>`
`name: keyof T & string`, `type: FieldType`, `searchable?`, `sortable?`, `facetable?`, `weight?`.
### `SearchEngine<T>`
`index(docs, opts?)`, `search(query, opts?)`, `suggest(prefix, opts?)`, `remove(ids)`, `clear()`.
## Implementation Notes
The `FilterOp` union and `FieldType` union are intentionally narrow — they define the supported operation set for the in-memory engine. Engines backed by external systems may support additional operators through extension.