Move docs/ package mirror into document-navigator plugin
Relocate the ~6k-file docs tree from the repo root into plugins/document-navigator/docs/ so the plugin is self-contained and portable. docs/superpowers/ (specs and plans) stays at the repo root. Moved via cp+rm rather than git mv because directory rename() fails with EPERM on this WSL /mnt DrvFs mount; git rename-detection pairs the identical-content files at commit time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
13
plugins/document-navigator/docs/persistence/search/README.md
Normal file
13
plugins/document-navigator/docs/persistence/search/README.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# `search`
|
||||
|
||||
Full-text search, BM25, fuzzy matching, and faceted search.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Packages in `search`
|
||||
|
||||
1 package.
|
||||
|
||||
| Package | Description |
|
||||
|---|---|
|
||||
| [`@suaveplan/search`](./search/README.md) | Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support. |
|
||||
<!-- end:auto-docs -->
|
||||
@@ -0,0 +1 @@
|
||||
Full-text search, BM25, fuzzy matching, and faceted search.
|
||||
@@ -0,0 +1,54 @@
|
||||
# @suaveplan/search - Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support
|
||||
|
||||
## Summary
|
||||
|
||||
Full-text and fuzzy search primitives with in-memory BM25 engine, trigram fuzzy matching, faceted search, and pagination support.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
<!-- end:auto-docs --> - [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_- [memory-engine](./src/engine/memory-engine/memory-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_ - [search-error](./src/errors/search-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_ - [search.schemas](./src/schemas/search.schemas.md) — <!-- begin:auto-docs --> _(no submodules)_- [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [scorer](./src/text/scorer/scorer.md) — <!-- begin:auto-docs --> _(no submodules)_- [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_ - [tokenizer](./src/text/tokenizer/tokenizer.md) — <!-- begin:auto-docs --> _(no submodules)_- [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_ - [trigram](./src/text/trigram/trigram.md) — <!-- begin:auto-docs --> _(no submodules)_
|
||||
## Modules
|
||||
|
||||
- [MemorySearchEngine](src/engine/memory-engine/memory-engine.md) — in-memory BM25 search engine
|
||||
- [Tokenizer](src/text/tokenizer/tokenizer.md) — text tokenization and stemming
|
||||
- [Trigram](src/text/trigram/trigram.md) — trigram-based fuzzy similarity
|
||||
- [Scorer](src/text/scorer/scorer.md) — BM25 and TF-IDF scoring functions
|
||||
- [SearchError](src/errors/search-error.md) — structured errors with stable codes
|
||||
- [Schemas](src/schemas/search.schemas.md) — Zod schemas for all query types
|
||||
- [Types](src/types/search.types.md) — shared TypeScript interfaces
|
||||
|
||||
## Quick Start
|
||||
|
||||
```ts
|
||||
import { createMemorySearchEngine } from "@suaveplan/search";
|
||||
|
||||
interface Product { id: string; name: string; price: number }
|
||||
|
||||
const engine = createMemorySearchEngine<Product>({
|
||||
fields: [
|
||||
{ name: "name", type: "text", searchable: true },
|
||||
{ name: "price", type: "number", sortable: true },
|
||||
],
|
||||
});
|
||||
|
||||
await engine.index([
|
||||
{ id: "1", name: "Laptop Pro", price: 999 },
|
||||
{ id: "2", name: "Laptop Air", price: 799 },
|
||||
]);
|
||||
|
||||
const result = await engine.search({ text: "laptop" });
|
||||
console.log(result.hits.length); // 2
|
||||
```
|
||||
@@ -0,0 +1,85 @@
|
||||
# memory-engine
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides `createMemorySearchEngine`, an in-memory implementation of `SearchEngine` backed by a term-frequency inverted index and BM25 scoring. Designed for single-process use, testing, and small-to-medium datasets (up to ~100k documents). Supports full-text search, structured filters, fuzzy trigram matching, faceted aggregations, and keyword highlighting.
|
||||
|
||||
## Features
|
||||
|
||||
- Inverted index built from tokenized searchable fields
|
||||
- BM25 scoring (k1=1.2, b=0.75) for relevance ranking
|
||||
- Trigram fuzzy matching with configurable threshold
|
||||
- Faceted search: bucket counts for `facetable` fields
|
||||
- Highlight extraction with configurable tag (default `<mark>`)
|
||||
- Structured filters: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`
|
||||
- Multi-field sorting with direction
|
||||
- Cursor and offset pagination
|
||||
- `suggest(prefix)` for autocomplete from indexed terms
|
||||
- OTel spans on `index`, `search`, and `suggest`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { createMemorySearchEngine } from "@suaveplan/search";
|
||||
|
||||
interface Product { id: string; name: string; price: number }
|
||||
|
||||
const engine = createMemorySearchEngine<Product>({
|
||||
fields: [
|
||||
{ name: "name", type: "text", searchable: true },
|
||||
{ name: "price", type: "number", sortable: true },
|
||||
],
|
||||
});
|
||||
|
||||
await engine.index([{ id: "1", name: "Laptop Pro", price: 999 }]);
|
||||
const result = await engine.search({ text: "laptop" });
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
const engine = createMemorySearchEngine<Product>({
|
||||
fields: [
|
||||
{ name: "name", type: "text", searchable: true, facetable: false },
|
||||
{ name: "category", type: "keyword", facetable: true },
|
||||
],
|
||||
fuzzyThreshold: 0.4,
|
||||
stemming: true,
|
||||
});
|
||||
|
||||
await engine.index(products);
|
||||
|
||||
const result = await engine.search({
|
||||
text: "laptops",
|
||||
filters: [{ field: "price", op: "lte", value: 1000 }],
|
||||
sort: [{ field: "price", direction: "asc" }],
|
||||
pagination: { type: "offset", offset: 0, limit: 20 },
|
||||
fuzzy: true,
|
||||
});
|
||||
|
||||
console.log(result.facets?.["category"]);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `createMemorySearchEngine<T>(options)`
|
||||
|
||||
- `options.fields: FieldConfig<T>[]` — field schema with type and searchability flags
|
||||
- `options.stemming?: boolean` — enable simple English stemming
|
||||
- `options.stopWords?: string[]` — custom stop-word list
|
||||
- `options.fuzzyThreshold?: number` — trigram similarity threshold for fuzzy (default 0.3)
|
||||
|
||||
### `SearchEngine<T>`
|
||||
|
||||
- `index(documents, options?)` — adds documents; replaces if `replace !== false`
|
||||
- `search(query, options?)` — returns `SearchResult<T>` with hits, total, facets
|
||||
- `suggest(prefix, options?)` — returns terms from inverted index starting with prefix
|
||||
- `remove(ids)` — removes documents by ID
|
||||
- `clear()` — empties the entire index
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The inverted index maps `term → Map<docId, tf>`. Field lengths per document are tracked for BM25 length normalization. Fuzzy matching runs after BM25 scoring and assigns a 50% score weight to fuzzy hits so they rank below exact matches.
|
||||
@@ -0,0 +1,55 @@
|
||||
# search-error
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Defines the structured error type and stable error-code registry for `@suaveplan/search`. All runtime failures from indexing, querying, cursor pagination, and field validation surface as `SearchError` instances with a machine-readable `code` field.
|
||||
|
||||
## Features
|
||||
|
||||
- `SEARCH_CODES` registry via `@suaveplan/error`'s `defineCodes` — codes namespaced `SEARCH.*`
|
||||
- `SearchError` extends `SuaveplanError` for cause-chain traversal
|
||||
- Five codes: `INDEX_FAILED`, `QUERY_INVALID`, `ENGINE_NOT_READY`, `FIELD_UNKNOWN`, `CURSOR_INVALID`
|
||||
- `SearchCode` union type for exhaustive switch statements
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { SearchError, SEARCH_CODES } from "@suaveplan/search";
|
||||
|
||||
throw new SearchError(SEARCH_CODES.FIELD_UNKNOWN, `Field "price" is not indexed`);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { SearchError, SEARCH_CODES } from "@suaveplan/search";
|
||||
|
||||
function handleSearchError(err: unknown): void {
|
||||
if (err instanceof SearchError) {
|
||||
if (err.code === SEARCH_CODES.CURSOR_INVALID) {
|
||||
// Reset pagination cursor
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `SEARCH_CODES`
|
||||
|
||||
- `SEARCH_CODES.INDEX_FAILED` — indexing operation failed
|
||||
- `SEARCH_CODES.QUERY_INVALID` — query syntax or parameter is invalid
|
||||
- `SEARCH_CODES.ENGINE_NOT_READY` — engine has no documents indexed
|
||||
- `SEARCH_CODES.FIELD_UNKNOWN` — referenced field not in schema
|
||||
- `SEARCH_CODES.CURSOR_INVALID` — pagination cursor is malformed or expired
|
||||
|
||||
### `SearchError`
|
||||
|
||||
Extends `SuaveplanError<SearchCode>`. Constructor: `(code, message, options?)`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Codes are prefixed with `"SEARCH."` by `defineCodes`, producing strings like `"SEARCH.QUERY_INVALID"`.
|
||||
@@ -0,0 +1,63 @@
|
||||
# search.schemas
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides Zod validation schemas for all public search query and configuration types in `@suaveplan/search`. Used at API boundaries to validate incoming queries, engine configurations, and filter specifications before they reach the in-memory engine.
|
||||
|
||||
## Features
|
||||
|
||||
- `SearchQuerySchema` — validates the full search query structure including text, filters, sort, pagination, fuzzy, and highlightTag
|
||||
- `SearchFilterSchema` — validates individual filter clauses with the 8-operator enum
|
||||
- `SortSpecSchema` — validates sort field and direction
|
||||
- `PaginationSpecSchema` — validates cursor/offset pagination with positive limit
|
||||
- `FieldConfigSchema` — validates field definitions
|
||||
- `MemorySearchEngineOptionsSchema` — validates engine construction options
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { SearchQuerySchema } from "@suaveplan/search";
|
||||
|
||||
const query = SearchQuerySchema.parse({
|
||||
text: "laptop",
|
||||
filters: [{ field: "price", op: "lte", value: 1000 }],
|
||||
pagination: { type: "offset", offset: 0, limit: 20 },
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { MemorySearchEngineOptionsSchema } from "@suaveplan/search";
|
||||
|
||||
const opts = MemorySearchEngineOptionsSchema.parse({
|
||||
fields: [{ name: "title", type: "text", searchable: true }],
|
||||
fuzzyThreshold: 0.4,
|
||||
stemming: true,
|
||||
});
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `SearchQuerySchema`
|
||||
|
||||
Validates `{ text?, filters?, sort?, pagination?, fuzzy?, highlightTag? }`.
|
||||
|
||||
### `SearchFilterSchema`
|
||||
|
||||
Validates `{ field: string, op: FilterOp, value: unknown }`.
|
||||
|
||||
### `PaginationSpecSchema`
|
||||
|
||||
Validates `{ type: "cursor" | "offset", cursor?, offset?, limit: positive }`.
|
||||
|
||||
### `MemorySearchEngineOptionsSchema`
|
||||
|
||||
Validates `{ fields, stemming?, stopWords?, fuzzyThreshold? }`. Passthrough for injectable clock.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The `value` field in `SearchFilterSchema` uses `z.unknown()` since filter values can be any type depending on the field type. The `fuzzy` field accepts either a boolean or `{ threshold: number }` via a Zod union.
|
||||
@@ -0,0 +1,69 @@
|
||||
# scorer
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides BM25 and TF-IDF scoring functions used by the in-memory search engine to rank search results by relevance. BM25 (Best Match 25) is the de facto standard ranking algorithm used by Elasticsearch, Solr, and Lucene. TF-IDF is provided as a simpler alternative.
|
||||
|
||||
## Features
|
||||
|
||||
- `bm25Score(options)` — Okapi BM25 with configurable k1 (default 1.2) and b (default 0.75)
|
||||
- `tfIdfScore(tf, idf)` — classic TF × IDF product
|
||||
- `computeIdf(df, n)` — IDF computation with logarithmic smoothing
|
||||
- All functions are pure and stateless; no external dependencies
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { bm25Score } from "@suaveplan/search";
|
||||
|
||||
const score = bm25Score({
|
||||
tf: 3, // term appears 3 times in document
|
||||
df: 10, // term appears in 10 documents
|
||||
n: 100, // total documents in corpus
|
||||
fieldLength: 50,
|
||||
avgFieldLength: 30,
|
||||
});
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { bm25Score, tfIdfScore, computeIdf } from "@suaveplan/search";
|
||||
|
||||
// Custom k1/b for short-field optimization
|
||||
const titleScore = bm25Score({
|
||||
tf: 1, df: 5, n: 1000, fieldLength: 5, avgFieldLength: 8,
|
||||
k1: 0.9, b: 0.4,
|
||||
});
|
||||
|
||||
// TF-IDF comparison
|
||||
const idf = computeIdf(5, 1000);
|
||||
const tfidf = tfIdfScore(2, idf);
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `bm25Score(options: Bm25Options)`
|
||||
|
||||
- `tf` — term frequency
|
||||
- `df` — document frequency
|
||||
- `n` — corpus size
|
||||
- `fieldLength` — length of field being scored
|
||||
- `avgFieldLength` — average field length across corpus
|
||||
- `k1?` — saturation parameter (default 1.2)
|
||||
- `b?` — length normalization (default 0.75)
|
||||
|
||||
### `tfIdfScore(tf, idf)`
|
||||
|
||||
Returns `tf * idf`.
|
||||
|
||||
### `computeIdf(df, n)`
|
||||
|
||||
Returns `log(1 + n/df)`. Returns `0` for zero inputs.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
BM25 IDF uses the Robertson-Sparck Jones formula: `log(1 + (N - df + 0.5) / (df + 0.5))`. This variant is always non-negative. The field-length normalization parameter `b = 0.75` slightly penalises longer documents, which improves precision for titles and short-text fields.
|
||||
@@ -0,0 +1,59 @@
|
||||
# tokenizer
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides text tokenization and normalization utilities for the in-memory search engine. Tokenization converts raw text into a stream of normalized tokens used to build the inverted index and to process search queries. The tokenizer handles stop-word removal, case normalization, and optional simple English stemming.
|
||||
|
||||
## Features
|
||||
|
||||
- `tokenize(text, options?)` — splits text on non-word characters, removes stop words, optionally stems
|
||||
- `normalizeToken(token)` — lowercases and trims a single token
|
||||
- `simpleStem(word)` — strips common English suffixes: `-tion`, `-ing`, `-est`, `-ed`, `-er`, `-ly`, `-s`
|
||||
- Default stop-word list of 40+ common English words
|
||||
- Case sensitivity and stop-word list are configurable per call
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { tokenize } from "@suaveplan/search";
|
||||
|
||||
const tokens = tokenize("The quick brown fox jumps over the lazy dog");
|
||||
console.log(tokens); // ["quick", "brown", "fox", "jump", ...]
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { tokenize, simpleStem } from "@suaveplan/search";
|
||||
|
||||
const tokens = tokenize("Running faster than expected", {
|
||||
stemming: true,
|
||||
stopWords: ["than"],
|
||||
});
|
||||
|
||||
console.log(simpleStem("computing")); // "comput"
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `tokenize(text, options?)`
|
||||
|
||||
- `options.caseSensitive?: boolean` (default `false`)
|
||||
- `options.stopWords?: string[]` — replaces default stop-word list
|
||||
- `options.stemming?: boolean` — applies `simpleStem` to each token (default `false`)
|
||||
- Returns `string[]`
|
||||
|
||||
### `normalizeToken(token)`
|
||||
|
||||
Lowercases and trims a single token. Used internally.
|
||||
|
||||
### `simpleStem(word)`
|
||||
|
||||
Returns the stem of an English word by stripping common suffixes. Words ≤3 characters are returned unchanged.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The stemmer is intentionally simple (suffix-stripping, no Snowball/Porter) to avoid external dependencies. For production use with high-accuracy stemming requirements, a custom tokenizer implementing the same interface can be passed via `MemorySearchEngineOptions`.
|
||||
@@ -0,0 +1,51 @@
|
||||
# trigram
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides trigram-based fuzzy matching utilities for the in-memory search engine. Trigrams are overlapping 3-character substrings that capture structural similarity between strings — they are language-agnostic, robust to minor typos, and require no external libraries.
|
||||
|
||||
## Features
|
||||
|
||||
- `buildTrigrams(text)` — generates a `Set<string>` of all 3-character substrings, padded with `##` for start/end boundary capture
|
||||
- `trigramSimilarity(a, b)` — computes Jaccard similarity (intersection / union) over the trigram sets, returning a value in `[0, 1]`
|
||||
- Case-insensitive: both functions normalize input to lowercase before processing
|
||||
- Exact matches always return `1.0`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { trigramSimilarity } from "@suaveplan/search";
|
||||
|
||||
const score = trigramSimilarity("hello", "helo");
|
||||
console.log(score); // ~0.71
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { buildTrigrams, trigramSimilarity } from "@suaveplan/search";
|
||||
|
||||
const threshold = 0.5;
|
||||
const candidates = ["hello", "help", "world", "dog"];
|
||||
const matches = candidates.filter(
|
||||
(c) => trigramSimilarity("helo", c) >= threshold
|
||||
);
|
||||
console.log(matches); // ["hello", "help"]
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `buildTrigrams(text)`
|
||||
|
||||
Returns `Set<string>` of all trigrams in `text` (case-insensitive, `##`-padded).
|
||||
|
||||
### `trigramSimilarity(a, b)`
|
||||
|
||||
Returns `number` in `[0, 1]`. Uses Jaccard coefficient: `|intersection| / |union|`. Returns `1` for identical strings or empty trigram sets.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The `##` padding ensures that short strings (1-2 characters) still generate meaningful trigrams that capture their start and end positions. The Jaccard coefficient is symmetric: `trigramSimilarity(a, b) === trigramSimilarity(b, a)`. Complexity is O(n + m) where n and m are the trigram set sizes.
|
||||
@@ -0,0 +1,70 @@
|
||||
# search.types
|
||||
|
||||
## Purpose
|
||||
|
||||
Shared TypeScript type definitions for the `@suaveplan/search` package. Defines all public contracts: search documents, queries, filters, results, field configurations, and engine interfaces. Centralising types here prevents circular imports between the engine, text utilities, and schema modules.
|
||||
|
||||
## Features
|
||||
|
||||
- `SearchDocument` — base interface requiring `id: string` plus arbitrary fields
|
||||
- `SearchQuery` — full query structure with text, filters, sort, pagination, and fuzzy options
|
||||
- `SearchFilter` — structured field filter with 8 operators (`eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`)
|
||||
- `SearchResult<T>` / `SearchHit<T>` — typed result container with score, highlights, and facets
|
||||
- `FieldConfig<T>` — per-field schema: type, searchable, sortable, facetable, weight
|
||||
- `MemorySearchEngineOptions<T>` — full configuration for the in-memory engine
|
||||
- `SearchEngine<T>` — interface for `index`, `search`, `suggest`, `remove`, `clear`
|
||||
- `Bm25Options` — parameters for BM25 scoring function
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import type { SearchDocument, SearchQuery, FieldConfig } from "@suaveplan/search";
|
||||
|
||||
interface Product extends SearchDocument {
|
||||
name: string;
|
||||
price: number;
|
||||
}
|
||||
|
||||
const field: FieldConfig<Product> = {
|
||||
name: "name",
|
||||
type: "text",
|
||||
searchable: true,
|
||||
facetable: false,
|
||||
};
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import type { SearchQuery, SearchFilter } from "@suaveplan/search";
|
||||
|
||||
const query: SearchQuery = {
|
||||
text: "laptop",
|
||||
filters: [{ field: "price", op: "lte", value: 1000 }],
|
||||
sort: [{ field: "price", direction: "asc" }],
|
||||
pagination: { type: "offset", offset: 0, limit: 20 },
|
||||
fuzzy: true,
|
||||
};
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `SearchDocument`
|
||||
|
||||
Base interface: `{ id: string; [field: string]: unknown }`.
|
||||
|
||||
### `SearchQuery`
|
||||
|
||||
Full query: `text?`, `filters?`, `sort?`, `pagination?`, `fuzzy?`, `highlightTag?`.
|
||||
|
||||
### `FieldConfig<T>`
|
||||
|
||||
`name: keyof T & string`, `type: FieldType`, `searchable?`, `sortable?`, `facetable?`, `weight?`.
|
||||
|
||||
### `SearchEngine<T>`
|
||||
|
||||
`index(docs, opts?)`, `search(query, opts?)`, `suggest(prefix, opts?)`, `remove(ids)`, `clear()`.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The `FilterOp` union and `FieldType` union are intentionally narrow — they define the supported operation set for the in-memory engine. Engines backed by external systems may support additional operators through extension.
|
||||
Reference in New Issue
Block a user