6.4 KiB
6.4 KiB
@suaveplan/dedup - Entity deduplication engine with pluggable match keys, scoring rules, Jaro-Winkler/Levenshtein similarity, and merge strategies
Summary
Entity deduplication engine for detecting and merging duplicate records in any data set. Given a list of objects with varying representations of the same real-world entity (e.g. "John Smith" vs "J. Smith", or the same email in different capitalisation), the DedupEngine scores pairs using a weighted combination of field-level similarity algorithms — Jaro-Winkler, Levenshtein, exact match, or phonetic — groups records into connected components via union-find, and merges each group with a pluggable strategy (latestWins, manual, or scoredMerge). Use this package for ETL pipelines, contact deduplication, and data cleansing. For domain-name-specific normalisation and alias resolution use @suaveplan/domain-normalizer; for string-similarity primitives only, the scoring module is exported as a standalone subpath.
Documentation tree
- engine — (no submodules)
- dedup-engine — (no submodules)
- merge — (no submodules)
- merge-strategies — (no submodules)
- scoring — (no submodules)
- similarity — (no submodules)
- errors — (no submodules)
- dedup-error — (no submodules)
- schemas — (no submodules)
- dedup.schema — (no submodules)
Modules
| Module | Description |
|---|---|
| scoring | Levenshtein, Jaro, Jaro-Winkler, exact, and phonetic similarity algorithms |
| engine | DedupEngine — detect duplicate groups and merge them |
| merge | Built-in merge strategies: latest-wins, manual, scored |
Quick Start
import { DedupEngine } from "@suaveplan/dedup/engine";
import { latestWinsStrategy } from "@suaveplan/dedup/merge";
const engine = new DedupEngine({
matchKeys: [
{ field: "name", weight: 0.6, strategy: "jaro-winkler" },
{ field: "email", weight: 0.4, strategy: "exact" },
],
scoringRules: [],
mergeThreshold: 0.85,
});
const groups = engine.detect(records);
for (const group of groups) {
const result = engine.merge(group, latestWinsStrategy);
console.log(result.merged);
}