# Sitemap completion — extensions, incremental builder, gzip compressor **Package**: `@suaveplan/sitemap` **Status**: PROPOSED **Type**: feature + spec-conformance ## Why `@suaveplan/sitemap` (`packages/api/sitemap/`) currently ships only the core builder, index-builder, parser, and URL utilities. Four upstream surfaces are absent: the Google image-sitemap extension, the Google video-sitemap extension, the incremental on-disk builder, and the gzip compressor. Without them, callers must re-implement Google-spec XML by hand (image / video) or stream URLs through the in-memory builder until they hit Google's 50,000-URL / 50 MiB hard cap. All four legacy modules already exist (`packages/universal/sitemap/src/extensions/image-sitemap.ts` 111 L; `extensions/video-sitemap.ts` 182 L; `incremental/incremental-builder.ts` 213 L; `compression/sitemap-compressor.ts` 91 L) and pass the genesis quality bar — porting them closes the parity gap. ## What changes ### `@suaveplan/sitemap` - ADDED: `SitemapImageExtension` — `addImage(urlEntry, image)` mutates a `SitemapUrl` to include a `` block per Google's image-sitemap spec; supports `{ caption, location, license, title }`. - ADDED: `SitemapVideoExtension` — `addVideo(urlEntry, video)` mutates a `SitemapUrl` to include a `` block; supports `{ thumbnail, title, description, contentLoc, playerLoc, duration, expiration, rating, viewCount, publicationDate, familyFriendly, requiresSubscription, uploader, live }`. - ADDED: `IMAGE_NS` / `VIDEO_NS` namespace constants and Zod schemas `SitemapImageExtensionSchema`, `SitemapVideoExtensionSchema`. - ADDED: `IncrementalSitemapBuilder` class — append-only builder that streams URLs in batches to disk through an injectable `WritableStorageLike` adapter; preserves URL order; rolls over to a new sitemap file at 50,000 URLs OR 50 MiB (whichever first); emits a `SitemapIndex` entry per rollover. - ADDED: `IncrementalSitemapBuilderOptions` interface — `outputDir`, `baseFilename` (default `sitemap`), `urlsPerSitemap` (default 50_000, max 50_000), `maxBytes` (default 50 *1024* 1024), `storage` (`WritableStorageLike`), `compress` (boolean — when true, each rollover output is gzipped). - ADDED: `compressSitemap(xml: string): Promise` and `compressSitemapIndex(xml: string): Promise` async helpers — gzip with valid `gzip` magic bytes (`1f 8b`), DEFLATE compression level 6 (default), ISO-8859-1 OS field, mtime stamped from injectable `Clock`. - ADDED: `decompressGzipSitemap(bytes: Uint8Array): Promise` round-trip helper for tests. - ADDED: `SITEMAP_CODES` extensions — `IMAGE_VALIDATION_FAILED`, `VIDEO_VALIDATION_FAILED`, `INCREMENTAL_ROLLOVER_FAILED`, `COMPRESSION_FAILED` registered via existing `defineCodes` block. - ADDED: subpath exports `./extensions/image-sitemap`, `./extensions/video-sitemap`, `./incremental/incremental-builder`, `./compression/sitemap-compressor`. - MODIFIED: root `src/index.ts` re-exports the new extensions, builder, and compressor helpers. ## Out of scope - **News sitemap extension** (`` block) — separate change. The Google news-sitemap is a different schema with publication-name / publication-language fields and a strict 30-day TTL window; folding it into this change dilutes review. - **Mobile sitemap extension** — deprecated by Google in 2015; do not port. - **Streaming gzip via `CompressionStream`** — keep the synchronous `fflate` path for now; once the gzip surface is stable a follow-up change can swap to a `WritableStream` pipeline. - **`SitemapBuilder.compress()` instance method** — out of scope to keep the core builder side-effect free; callers gzip the output explicitly via `compressSitemap`. ## Source mapping This change derives from `../suaveplan/enterprise/sitemap.md` and the upstream package source. | Requirement | Source file | Source lines | Notes | |---|---|---|---| | `SitemapImageExtension` + Zod schema | `packages/universal/sitemap/src/extensions/image-sitemap.ts` | L1-L111 | Verbatim port; existing `SitemapImageSchema` in `schemas/sitemap.schema.ts` reused | | `SitemapVideoExtension` + Zod schema | `packages/universal/sitemap/src/extensions/video-sitemap.ts` | L1-L182 | Verbatim port; existing `SitemapVideoSchema` reused | | `IncrementalSitemapBuilder` | `packages/universal/sitemap/src/incremental/incremental-builder.ts` | L1-L213 | Verbatim port; replace `Date.now()` and `process.env.HOME` paths with injectable `Clock` and `WritableStorageLike` | | `compressSitemap` / `compressSitemapIndex` / `decompressGzipSitemap` | `packages/universal/sitemap/src/compression/sitemap-compressor.ts` | L1-L91 | Verbatim port | ### Requirements with no source line | Requirement | Justification | |---|---| | Injectable `Clock` for gzip mtime field | AGENTS.md determinism rule (no `Date.now()` in hot paths). Upstream stamps `Date.now()`, breaking deterministic test fixtures. | | Injectable `WritableStorageLike` adapter for `IncrementalSitemapBuilder` | AGENTS.md §11 ("shared types belong in `@suaveplan/types`"). Upstream writes directly via `fs.promises.writeFile`, coupling the builder to Node. | | `INCREMENTAL_ROLLOVER_FAILED` / `COMPRESSION_FAILED` error codes | AGENTS.md §13 row 3. Upstream throws plain `Error`. | ## Verification approach Each requirement carries a `**Verified by:**` line in the spec delta. - `packages/api/sitemap/src/extensions/image-sitemap/image-sitemap.test.ts` — covers Image extension scenarios - `packages/api/sitemap/src/extensions/video-sitemap/video-sitemap.test.ts` — covers Video extension scenarios - `packages/api/sitemap/src/incremental/incremental-builder/incremental-builder.test.ts` — covers append, rollover at 50k URLs, rollover at 50 MiB, index entry emission - `packages/api/sitemap/src/incremental/incremental-builder/incremental-builder.spec-conformance.test.ts` — pins injected `WritableStorageLike` contract and Clock determinism - `packages/api/sitemap/src/compression/sitemap-compressor/sitemap-compressor.test.ts` — covers gzip round-trip, magic-byte assertions, mtime determinism ## Impact ### Affected dependents Run `bunx turbo ls --filter=...@suaveplan/sitemap` to enumerate. - No dependents (verified by `grep -rl "@suaveplan/sitemap" /home/ubuntu/web/genesis/packages/ --include="package.json"` → only `packages/api/sitemap/package.json` itself). Pure additive change. ## Migration impact None — additive only. Existing `SitemapBuilder`, `SitemapIndexBuilder`, `parseSitemap`, `parseSitemapIndex` calls compile unchanged. The `SitemapUrl` type already carries optional `images?: SitemapImage[]` and `videos?: SitemapVideo[]` fields (in `schemas/sitemap.schema.ts`); the new extensions populate those fields rather than introducing a new shape.