Initial Commit

packages/foundation/data/dedup/README.md

@@ -0,0 +1,49 @@
+# @suaveplan/dedup - Entity deduplication engine with pluggable match keys, scoring rules, Jaro-Winkler/Levenshtein similarity, and merge strategies
+
+## Summary
+
+Entity deduplication engine for detecting and merging duplicate records in any data set. Given a list of objects with varying representations of the same real-world entity (e.g. `"John Smith"` vs `"J. Smith"`, or the same email in different capitalisation), the `DedupEngine` scores pairs using a weighted combination of field-level similarity algorithms — Jaro-Winkler, Levenshtein, exact match, or phonetic — groups records into connected components via union-find, and merges each group with a pluggable strategy (`latestWins`, `manual`, or `scoredMerge`). Use this package for ETL pipelines, contact deduplication, and data cleansing. For domain-name-specific normalisation and alias resolution use `@suaveplan/domain-normalizer`; for string-similarity primitives only, the scoring module is exported as a standalone subpath.
+
+<!-- begin:auto-docs -->
+## Documentation tree
+
+- [engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+  - [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+  - [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+  - [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+  - [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+  - [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_ <!-- end:auto-docs -->
+<!-- end:auto-docs -->  - [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_- [dedup-engine](./src/core/engine/dedup-engine.md) — <!-- begin:auto-docs --> _(no submodules)_- [merge](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_  - [merge-strategies](./src/core/merge/merge-strategies.md) — <!-- begin:auto-docs --> _(no submodules)_- [scoring](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_  - [similarity](./src/core/scoring/similarity.md) — <!-- begin:auto-docs --> _(no submodules)_- [errors](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_  - [dedup-error](./src/errors/dedup-error.md) — <!-- begin:auto-docs --> _(no submodules)_- [schemas](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_  - [dedup.schema](./src/schemas/dedup.schema.md) — <!-- begin:auto-docs --> _(no submodules)_
+## Modules
+
+| Module | Description |
+|---|---|
+| [scoring](src/core/scoring/similarity.md) | Levenshtein, Jaro, Jaro-Winkler, exact, and phonetic similarity algorithms |
+| [engine](src/core/engine/dedup-engine.md) | `DedupEngine` — detect duplicate groups and merge them |
+| [merge](src/core/merge/merge-strategies.md) | Built-in merge strategies: latest-wins, manual, scored |
+
+## Quick Start
+
+```ts
+import { DedupEngine } from "@suaveplan/dedup/engine";
+import { latestWinsStrategy } from "@suaveplan/dedup/merge";
+
+const engine = new DedupEngine({
+  matchKeys: [
+    { field: "name",  weight: 0.6, strategy: "jaro-winkler" },
+    { field: "email", weight: 0.4, strategy: "exact" },
+  ],
+  scoringRules: [],
+  mergeThreshold: 0.85,
+});
+
+const groups = engine.detect(records);
+for (const group of groups) {
+  const result = engine.merge(group, latestWinsStrategy);
+  console.log(result.merged);
+}
+```