first commit
This commit is contained in:
49
docs/foundation/data/dedup/README.md
Normal file
49
docs/foundation/data/dedup/README.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# @suaveplan/dedup - Entity deduplication engine with pluggable match keys, scoring rules, Jaro-Winkler/Levenshtein similarity, and merge strategies
|
||||
|
||||
## Summary
|
||||
|
||||
Entity deduplication engine for detecting and merging duplicate records in any data set. Given a list of objects with varying representations of the same real-world entity (e.g. `"John Smith"` vs `"J. Smith"`, or the same email in different capitalisation), the `DedupEngine` scores pairs using a weighted combination of field-level similarity algorithms — Jaro-Winkler, Levenshtein, exact match, or phonetic — groups records into connected components via union-find, and merges each group with a pluggable strategy (`latestWins`, `manual`, or `scoredMerge`). Use this package for ETL pipelines, contact deduplication, and data cleansing. For domain-name-specific normalisation and alias resolution use `@suaveplan/domain-normalizer`; for string-similarity primitives only, the scoring module is exported as a standalone subpath.
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
## Documentation tree
|
||||
|
||||
- [engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
|
||||
<!-- end:auto-docs --> - [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_ - [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_ - [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_ - [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ - [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_
|
||||
## Modules
|
||||
|
||||
| Module | Description |
|
||||
|---|---|
|
||||
| [scoring](src/core/scoring/similarity.md) | Levenshtein, Jaro, Jaro-Winkler, exact, and phonetic similarity algorithms |
|
||||
| [engine](src/core/engine/dedup-engine.md) | `DedupEngine` — detect duplicate groups and merge them |
|
||||
| [merge](src/core/merge/merge-strategies.md) | Built-in merge strategies: latest-wins, manual, scored |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```ts
|
||||
import { DedupEngine } from "@suaveplan/dedup/engine";
|
||||
import { latestWinsStrategy } from "@suaveplan/dedup/merge";
|
||||
|
||||
const engine = new DedupEngine({
|
||||
matchKeys: [
|
||||
{ field: "name", weight: 0.6, strategy: "jaro-winkler" },
|
||||
{ field: "email", weight: 0.4, strategy: "exact" },
|
||||
],
|
||||
scoringRules: [],
|
||||
mergeThreshold: 0.85,
|
||||
});
|
||||
|
||||
const groups = engine.detect(records);
|
||||
for (const group of groups) {
|
||||
const result = engine.merge(group, latestWinsStrategy);
|
||||
console.log(result.merged);
|
||||
}
|
||||
```
|
||||
114
docs/foundation/data/dedup/src/core/engine/dedup-engine.md
Normal file
114
docs/foundation/data/dedup/src/core/engine/dedup-engine.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# dedup-engine
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`DedupEngine` is the central orchestrator for entity deduplication. It accepts a validated `DedupConfig`, performs an O(n²) pairwise similarity scan using pluggable match keys and scoring rules, groups near-duplicate entities into connected components via union-find, and exposes a `merge()` method that delegates to caller-supplied merge strategies. The engine is generic over the entity type `T` so it can operate on any record shape.
|
||||
|
||||
## Features
|
||||
|
||||
- **Schema-validated configuration** — `DedupConfig` is parsed with Zod at construction time; invalid configs throw a `DedupError` with code `DEDUP.INVALID_CONFIG`.
|
||||
- **Weighted field scoring** — each `MatchKey` contributes its `weight × similarity(field_a, field_b)` to the pair score.
|
||||
- **Pluggable scoring rules** — `ScoringRule` instances can override or supplement the default key-based scoring for custom business logic.
|
||||
- **Connected-component grouping** — union-find algorithm aggregates transitive duplicate relationships so A≈B and B≈C implies A, B, C are in the same group.
|
||||
- **`maxGroupSize` guard** — prevents unbounded memory growth by capping the number of duplicates per group.
|
||||
- **OTel instrumentation** — emits `dedup.detect.total` counter, `dedup.detect.duration_ms` histogram, and per-span attributes on every `detect()` and `merge()` call.
|
||||
- **Safe merge delegation** — `merge()` wraps strategy execution in a try/catch and re-throws as `DedupError(STRATEGY_ERROR)` to preserve error code stability.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { DedupEngine } from "@suaveplan/dedup/engine";
|
||||
|
||||
const engine = new DedupEngine({
|
||||
matchKeys: [
|
||||
{ field: "name", weight: 0.6, strategy: "jaro-winkler" },
|
||||
{ field: "email", weight: 0.4, strategy: "exact" },
|
||||
],
|
||||
scoringRules: [],
|
||||
mergeThreshold: 0.85,
|
||||
});
|
||||
|
||||
const records = [
|
||||
{ id: 1, name: "Alice Smith", email: "alice@example.com" },
|
||||
{ id: 2, name: "Alice Smyth", email: "alice@example.com" },
|
||||
{ id: 3, name: "Bob Jones", email: "bob@example.com" },
|
||||
];
|
||||
|
||||
const groups = engine.detect(records);
|
||||
// groups[0].canonical → { id: 1, name: "Alice Smith", email: "alice@example.com" }
|
||||
// groups[0].duplicates → [{ id: 2, name: "Alice Smyth", email: "alice@example.com" }]
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { DedupEngine } from "@suaveplan/dedup/engine";
|
||||
import { manualStrategy } from "@suaveplan/dedup/merge";
|
||||
|
||||
const engine = new DedupEngine<{ id: number; name: string; updatedAt: string }>({
|
||||
matchKeys: [
|
||||
{ field: "name", weight: 1, strategy: "jaro-winkler", caseSensitive: false },
|
||||
],
|
||||
scoringRules: [
|
||||
{
|
||||
name: "domain-boost",
|
||||
evaluate(a, b, _keys) {
|
||||
// Boost score when both entities are from the same domain
|
||||
const ra = a as Record<string, string>;
|
||||
const rb = b as Record<string, string>;
|
||||
const domainA = ra["email"]?.split("@")[1] ?? "";
|
||||
const domainB = rb["email"]?.split("@")[1] ?? "";
|
||||
return domainA && domainA === domainB ? 0.95 : 0.0;
|
||||
},
|
||||
},
|
||||
],
|
||||
mergeThreshold: 0.8,
|
||||
maxGroupSize: 50,
|
||||
});
|
||||
|
||||
const groups = engine.detect(records);
|
||||
for (const group of groups) {
|
||||
const result = engine.merge(group, manualStrategy);
|
||||
console.log("merged:", result.merged);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `new DedupEngine<T>(config: DedupConfig)`
|
||||
|
||||
Constructs a new engine. `config` is validated against `DedupConfigSchema`; throws `DedupError(INVALID_CONFIG)` on failure.
|
||||
|
||||
### `detect(entities: T[]): DuplicateGroup<T>[]`
|
||||
|
||||
Scans all entity pairs, builds a similarity graph, finds connected components, and returns one `DuplicateGroup` per multi-entity component. Singletons (no duplicates) are omitted.
|
||||
|
||||
**Params:**
|
||||
|
||||
- `entities` — array of entities to deduplicate
|
||||
|
||||
**Returns:** `DuplicateGroup<T>[]` — possibly empty if no duplicates found
|
||||
|
||||
### `merge(group: DuplicateGroup<T>, strategy: MergeStrategy<T>): MergeResult<T>`
|
||||
|
||||
Applies `strategy` to `group` and returns the merged result. Wraps thrown errors as `DedupError(STRATEGY_ERROR)`.
|
||||
|
||||
**Params:**
|
||||
|
||||
- `group` — a group returned by `detect()`
|
||||
- `strategy` — a `MergeStrategy<T>` function
|
||||
|
||||
**Returns:** `MergeResult<T>`
|
||||
|
||||
**Throws:** `DedupError` with code `DEDUP.STRATEGY_ERROR` if the strategy throws
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The pairwise scan is O(n²) in both time and the number of potential edges. For large corpora (> 10 000 entities) callers should pre-filter candidates using blocking keys (e.g. first three characters of a name) before passing them to `detect()`. A future blocking API may be added as a separate module.
|
||||
|
||||
The union-find implementation uses path-halving rather than full path compression to keep the code concise while still achieving near-linear amortised complexity. For groups larger than `maxGroupSize`, only the first `maxGroupSize − 1` duplicates (by original array index) are retained; this is intentional to bound memory and downstream merge cost.
|
||||
|
||||
When multiple `scoringRules` are present, each successive rule _replaces_ the accumulated score for that pair, not adds to it. This allows a single rule to act as a "veto" or "boost" override. If additive composition is needed, callers should encode that logic within a single rule.
|
||||
@@ -0,0 +1,92 @@
|
||||
# merge-strategies
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides three built-in merge strategies for combining a `DuplicateGroup` into a single entity. Merge strategies are pure functions that conform to the `MergeStrategy<T>` type and can be composed or replaced with custom implementations. They are passed to `DedupEngine.merge()` as callbacks rather than configured up-front, giving callers full control over how and when merging occurs.
|
||||
|
||||
## Features
|
||||
|
||||
- **`latestWinsStrategy`** — selects whichever entity (canonical or duplicate) has the most recent `updatedAt` or `createdAt` timestamp; ISO 8601 strings sort correctly as plain strings so no date parsing is needed.
|
||||
- **`manualStrategy`** — no-op that returns the canonical entity unchanged, useful when a human reviewer should approve merges before they are committed to a data store.
|
||||
- **`scoredMergeStrategy`** — assembles a "best-of-breed" entity by choosing, for each field, the value from the entity with the highest similarity score; the canonical entity acts as the fallback with score 1.0.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { DedupEngine } from "@suaveplan/dedup/engine";
|
||||
import { latestWinsStrategy } from "@suaveplan/dedup/merge";
|
||||
|
||||
const engine = new DedupEngine({
|
||||
matchKeys: [{ field: "name", weight: 1, strategy: "jaro-winkler" }],
|
||||
scoringRules: [],
|
||||
});
|
||||
|
||||
const groups = engine.detect([
|
||||
{ name: "Alice", updatedAt: "2024-01-01T00:00:00Z" },
|
||||
{ name: "Alicia", updatedAt: "2024-06-15T12:00:00Z" },
|
||||
]);
|
||||
|
||||
for (const group of groups) {
|
||||
const result = engine.merge(group, latestWinsStrategy);
|
||||
// result.merged → { name: "Alicia", updatedAt: "2024-06-15T12:00:00Z" }
|
||||
// result.strategy → "latest-wins"
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { DedupEngine } from "@suaveplan/dedup/engine";
|
||||
import { scoredMergeStrategy, manualStrategy } from "@suaveplan/dedup/merge";
|
||||
import type { MergeStrategy } from "@suaveplan/dedup";
|
||||
|
||||
type Person = { id: number; name: string; email: string; phone?: string };
|
||||
|
||||
const engine = new DedupEngine<Person>({
|
||||
matchKeys: [
|
||||
{ field: "name", weight: 0.5, strategy: "jaro-winkler" },
|
||||
{ field: "email", weight: 0.5, strategy: "exact" },
|
||||
],
|
||||
scoringRules: [],
|
||||
mergeThreshold: 0.8,
|
||||
});
|
||||
|
||||
const groups = engine.detect(persons);
|
||||
|
||||
// Pick strategy based on group size
|
||||
for (const group of groups) {
|
||||
const strategy: MergeStrategy<Person> =
|
||||
group.duplicates.length > 5 ? manualStrategy : scoredMergeStrategy;
|
||||
const result = engine.merge(group, strategy);
|
||||
console.log(result.strategy, result.merged);
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `latestWinsStrategy<T>(group: DuplicateGroup<T>): MergeResult<T>`
|
||||
|
||||
Picks the entity with the lexicographically largest `updatedAt` or `createdAt` field value. If neither field is present, returns the canonical unchanged. All entities (canonical + duplicates) participate in the comparison.
|
||||
|
||||
**Returns:** `MergeResult<T>` with `strategy: "latest-wins"`
|
||||
|
||||
### `manualStrategy<T>(group: DuplicateGroup<T>): MergeResult<T>`
|
||||
|
||||
Returns the canonical as the merged entity with no field-level changes. Suitable as a placeholder when merge logic will be determined externally.
|
||||
|
||||
**Returns:** `MergeResult<T>` with `strategy: "manual"`
|
||||
|
||||
### `scoredMergeStrategy<T>(group: DuplicateGroup<T>): MergeResult<T>`
|
||||
|
||||
Constructs a merged object by iterating every field key that appears in any entity and assigning the value from the entity with the highest similarity score. The canonical entity always has score `1.0`. Fields with `null` or `undefined` values are skipped when selecting the best value. The result is cast to `T` — callers should ensure the resulting shape is compatible.
|
||||
|
||||
**Returns:** `MergeResult<T>` with `strategy: "scored-merge"`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
All three strategies are stateless functions with no side effects. They do not mutate the `group` argument. The `scoredMergeStrategy` creates a new plain object (`{}`) and populates it field-by-field, which means prototype-based instances (classes) will lose their prototype chain in the merged result. If the entity type uses class instances, callers should wrap `scoredMergeStrategy` and reconstruct the class after merging.
|
||||
|
||||
`latestWinsStrategy` relies on ISO 8601 string sort order. Timestamps in other formats (Unix seconds as numbers, locale-formatted strings) will not sort correctly. Callers storing dates in non-ISO formats should either normalise before dedup or supply a custom `MergeStrategy` that parses timestamps explicitly.
|
||||
94
docs/foundation/data/dedup/src/core/scoring/similarity.md
Normal file
94
docs/foundation/data/dedup/src/core/scoring/similarity.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# similarity
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Provides string similarity algorithms used by the dedup engine to compare entity field values. Three well-known distance / similarity metrics are implemented from scratch: Levenshtein edit distance, Jaro similarity, and Jaro-Winkler similarity. A phonetic normalisation path is also included for fuzzy name matching. All functions are pure and side-effect-free, operating only on their arguments.
|
||||
|
||||
## Features
|
||||
|
||||
- **Levenshtein distance** — classic Wagner–Fischer dynamic-programming algorithm with O(min(|a|, |b|)) space optimisation (two-row rolling array).
|
||||
- **Jaro similarity** — accounts for matching characters within a dynamic window and transposition pairs; returns a score in [0, 1].
|
||||
- **Jaro-Winkler similarity** — extends Jaro by boosting strings that share a common prefix (up to 4 characters) using the standard p = 0.1 scaling factor.
|
||||
- **Exact match** — binary 1 / 0 equality check for deterministic field comparison.
|
||||
- **Phonetic match** — lightweight vowel-removal normalisation followed by Jaro-Winkler, useful for names that sound alike but are spelled differently.
|
||||
- **`computeSimilarity` dispatcher** — unified entry point that routes to the right algorithm based on the `MatchKey.strategy` discriminant.
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { levenshteinDistance, jaroWinklerSimilarity, computeSimilarity } from "@suaveplan/dedup/scoring";
|
||||
|
||||
levenshteinDistance("kitten", "sitting"); // 3
|
||||
jaroWinklerSimilarity("MARTHA", "MARHTA"); // ≈ 0.961
|
||||
|
||||
// Use the dispatcher for engine integration
|
||||
const score = computeSimilarity("Alice", "Alicia", "jaro-winkler"); // ≈ 0.947
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { computeSimilarity } from "@suaveplan/dedup/scoring";
|
||||
import type { MatchKey } from "@suaveplan/dedup";
|
||||
|
||||
const keys: MatchKey[] = [
|
||||
{ field: "name", weight: 0.5, strategy: "jaro-winkler", caseSensitive: false },
|
||||
{ field: "email", weight: 0.3, strategy: "exact", caseSensitive: false },
|
||||
{ field: "city", weight: 0.2, strategy: "phonetic", caseSensitive: false },
|
||||
];
|
||||
|
||||
function pairScore(a: Record<string, string>, b: Record<string, string>): number {
|
||||
return keys.reduce((sum, k) => {
|
||||
const va = (a[k.field] ?? "").toLowerCase();
|
||||
const vb = (b[k.field] ?? "").toLowerCase();
|
||||
return sum + k.weight * computeSimilarity(va, vb, k.strategy);
|
||||
}, 0);
|
||||
}
|
||||
|
||||
const score = pairScore(
|
||||
{ name: "John Smith", email: "j@example.com", city: "New York" },
|
||||
{ name: "Jon Smyth", email: "j@example.com", city: "New York" },
|
||||
);
|
||||
// score ≈ 0.93 — high confidence duplicate
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `levenshteinDistance(a, b): number`
|
||||
|
||||
Returns the minimum number of single-character edits (insertions, deletions, substitutions) required to transform `a` into `b`. Always ≥ 0. Runs in O(|a| × |b|) time with O(min(|a|, |b|)) space.
|
||||
|
||||
### `jaroSimilarity(s1, s2): number`
|
||||
|
||||
Jaro metric. Returns a value in [0, 1]. The matching window is `floor(max(|s1|, |s2|) / 2) - 1`. Identical strings return 1; completely different strings return 0.
|
||||
|
||||
### `jaroWinklerSimilarity(s1, s2): number`
|
||||
|
||||
Jaro-Winkler metric. Applies a prefix bonus: `jaro + prefix * 0.1 * (1 - jaro)` where `prefix` is the length of the shared prefix capped at 4. Returns a value in [0, 1].
|
||||
|
||||
### `exactMatch(a, b): number`
|
||||
|
||||
Returns `1` if `a === b`, `0` otherwise. No normalisation is applied; callers should lower-case inputs if case-insensitive comparison is required.
|
||||
|
||||
### `computeSimilarity(a, b, strategy): number`
|
||||
|
||||
Routes to the correct algorithm. For `"levenshtein"` the raw distance is normalised to [0, 1] by dividing by `max(|a|, |b|)` and subtracting from 1. For `"phonetic"`, both strings are first passed through a vowel-removal normaliser before Jaro-Winkler scoring.
|
||||
|
||||
**Params:**
|
||||
|
||||
- `a` — first string
|
||||
- `b` — second string
|
||||
- `strategy` — `"exact" | "jaro-winkler" | "levenshtein" | "phonetic"`
|
||||
|
||||
**Returns:** `number` in [0, 1]
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The two-row rolling array in `levenshteinDistance` avoids allocating an (|a|+1)×(|b|+1) matrix; only the previous and current rows are kept in memory. The shorter string is always placed on the column axis to minimise allocations.
|
||||
|
||||
The Jaro matching window formula uses `Math.max(|s1|, |s2|) / 2 - 1` and is floored, matching the original Winkler (1990) paper. Some implementations use `Math.ceil`; this library matches the floor variant which produces the widely-cited benchmark values (MARTHA/MARHTA ≈ 0.944 Jaro, ≈ 0.961 Jaro-Winkler).
|
||||
|
||||
The phonetic normaliser is intentionally lightweight — it is not a full Soundex or Metaphone implementation. It removes vowels and collapses consecutive duplicate consonants, which is sufficient for the common use case of matching first/last names with minor spelling variations. For production-grade phonetic matching, callers should supply a custom `ScoringRule`.
|
||||
82
docs/foundation/data/dedup/src/errors/dedup-error.md
Normal file
82
docs/foundation/data/dedup/src/errors/dedup-error.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# DedupError
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
`DedupError` is the error substrate for the `@suaveplan/dedup` package. It extends the built-in `Error` class with a stable `code` field so callers can branch on specific failure modes without parsing message strings. Every error thrown by the dedup engine or merge strategies uses this class.
|
||||
|
||||
## Features
|
||||
|
||||
- Typed `code` field for reliable programmatic error handling
|
||||
- Two stable error codes: `DEDUP.INVALID_CONFIG` and `DEDUP.STRATEGY_ERROR`
|
||||
- Lightweight — no external dependencies; usable in any runtime environment
|
||||
- `name` set to `'DedupError'` for readable stack traces and `instanceof` checks
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { DedupError, DEDUP_CODES } from '@suaveplan/dedup/errors';
|
||||
|
||||
try {
|
||||
engine.detect(entities);
|
||||
} catch (err) {
|
||||
if (err instanceof DedupError && err.code === DEDUP_CODES.INVALID_CONFIG) {
|
||||
console.error('Configuration is invalid:', err.message);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { DedupError, DEDUP_CODES } from '@suaveplan/dedup/errors';
|
||||
|
||||
function safeMerge(engine: DedupEngine, group: DuplicateGroup, strategy: MergeStrategy) {
|
||||
try {
|
||||
return engine.merge(group, strategy);
|
||||
} catch (err) {
|
||||
if (err instanceof DedupError) {
|
||||
if (err.code === DEDUP_CODES.STRATEGY_ERROR) {
|
||||
// Fall back to manual strategy
|
||||
return engine.merge(group, manualStrategy);
|
||||
}
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `DEDUP_CODES`
|
||||
|
||||
```ts
|
||||
const DEDUP_CODES: {
|
||||
INVALID_CONFIG: 'DEDUP.INVALID_CONFIG';
|
||||
STRATEGY_ERROR: 'DEDUP.STRATEGY_ERROR';
|
||||
}
|
||||
```
|
||||
|
||||
- `INVALID_CONFIG` — thrown when `DedupConfig` fails Zod validation (missing required fields, invalid weights, etc.)
|
||||
- `STRATEGY_ERROR` — thrown when a merge strategy throws an unexpected error
|
||||
|
||||
### `DedupError`
|
||||
|
||||
```ts
|
||||
class DedupError extends Error {
|
||||
readonly code: string;
|
||||
constructor(message: string, code: string);
|
||||
}
|
||||
```
|
||||
|
||||
- `message` — human-readable description of the failure
|
||||
- `code` — one of the `DEDUP_CODES` constants
|
||||
- `name` — always `'DedupError'`
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
This package is a leaf (no `@suaveplan/*` runtime dependencies), so it cannot depend on `@suaveplan/error`. The local `DedupError` class provides the same `code`-based discriminator pattern used throughout the project without introducing a circular dependency. If `@suaveplan/error` is added as a dependency in the future, this class should be migrated to extend `SuaveplanError`.
|
||||
|
||||
All errors thrown by the engine and merge strategies include a `code` value drawn from `DEDUP_CODES`. Callers should always `instanceof DedupError` before reading `.code` to avoid narrowing mistakes when third-party errors propagate through the dedup boundary. The code constants are exported as a plain `const` object rather than an enum to keep the module tree-shakeable and to avoid TypeScript enum pitfalls in strict ESM environments.
|
||||
97
docs/foundation/data/dedup/src/schemas/dedup.schema.md
Normal file
97
docs/foundation/data/dedup/src/schemas/dedup.schema.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Dedup Schemas
|
||||
|
||||
<!-- begin:auto-docs -->
|
||||
_(no submodules)_
|
||||
<!-- end:auto-docs -->
|
||||
## Purpose
|
||||
|
||||
Zod schemas for the `@suaveplan/dedup` package's configuration types. These schemas serve as the single source of truth for runtime validation of `DedupConfig`, `MatchKey`, and `ScoringRule` objects supplied by callers. All engine constructors run inputs through these schemas before use, ensuring invalid configurations fail fast with clear error messages.
|
||||
|
||||
## Features
|
||||
|
||||
- `MatchKeySchema` — validates field path, weight (0–1), strategy, and optional case-sensitivity flag
|
||||
- `ScoringRuleSchema` — validates the name string of a scoring rule
|
||||
- `DedupConfigSchema` — composes the above with threshold and max-group-size defaults
|
||||
- Coerced defaults: `mergeThreshold` defaults to `0.85`, `maxGroupSize` to `100`
|
||||
- All schemas export inferred TypeScript types via `z.infer`
|
||||
|
||||
## Usage: Basic Example
|
||||
|
||||
```ts
|
||||
import { DedupConfigSchema } from '@suaveplan/dedup';
|
||||
import type { DedupConfig } from '@suaveplan/dedup';
|
||||
|
||||
const raw = {
|
||||
matchKeys: [{ field: 'email', weight: 1, strategy: 'exact' }],
|
||||
scoringRules: [],
|
||||
};
|
||||
|
||||
const config: DedupConfig = DedupConfigSchema.parse(raw);
|
||||
```
|
||||
|
||||
## Usage: Advanced Example
|
||||
|
||||
```ts
|
||||
import { DedupConfigSchema } from '@suaveplan/dedup';
|
||||
|
||||
// Validate at API boundary before constructing engine
|
||||
function createEngine(raw: unknown) {
|
||||
const result = DedupConfigSchema.safeParse(raw);
|
||||
if (!result.success) {
|
||||
throw new Error(`Invalid dedup config: ${result.error.message}`);
|
||||
}
|
||||
return new DedupEngine(result.data);
|
||||
}
|
||||
|
||||
// Multi-key config with custom thresholds
|
||||
const config = DedupConfigSchema.parse({
|
||||
matchKeys: [
|
||||
{ field: 'email', weight: 0.6, strategy: 'exact' },
|
||||
{ field: 'firstName', weight: 0.2, strategy: 'jaro-winkler' },
|
||||
{ field: 'lastName', weight: 0.2, strategy: 'jaro-winkler' },
|
||||
],
|
||||
scoringRules: [],
|
||||
mergeThreshold: 0.75,
|
||||
maxGroupSize: 50,
|
||||
});
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### `MatchKeySchema`
|
||||
|
||||
```ts
|
||||
z.object({
|
||||
field: z.string().min(1),
|
||||
weight: z.number().min(0).max(1),
|
||||
strategy: z.enum(['exact', 'jaro-winkler', 'levenshtein', 'phonetic']),
|
||||
caseSensitive: z.boolean().optional(),
|
||||
})
|
||||
```
|
||||
|
||||
### `ScoringRuleSchema`
|
||||
|
||||
```ts
|
||||
z.object({
|
||||
name: z.string().min(1),
|
||||
})
|
||||
```
|
||||
|
||||
### `DedupConfigSchema`
|
||||
|
||||
```ts
|
||||
z.object({
|
||||
matchKeys: z.array(MatchKeySchema).min(1),
|
||||
scoringRules: z.array(ScoringRuleSchema),
|
||||
mergeThreshold: z.number().min(0).max(1).optional().default(0.85),
|
||||
maxGroupSize: z.number().int().positive().optional().default(100),
|
||||
})
|
||||
```
|
||||
|
||||
Inferred types (`MatchKey`, `ScoringRule`, `DedupConfig`) are exported from the package root alongside the schemas.
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
Schemas are defined in a dedicated `.schema.ts` file and excluded from coverage reporting (per project convention). The `ScoringRuleSchema` currently only validates `name` because the `evaluate` function is a runtime interface — Zod cannot validate function types. Callers providing custom scoring rules must implement the `ScoringRule` interface directly.
|
||||
|
||||
The `weight` field on `MatchKeySchema` is validated to the range `[0, 1]` but the engine does not enforce that all weights sum to exactly 1.0 — this is intentional. Weights below 1 produce a score ceiling below the default threshold, which is a valid configuration for partial-match scenarios. Callers should set `mergeThreshold` accordingly when using sub-unity weight sums.
|
||||
Reference in New Issue
Block a user