75 lines
6.7 KiB
Markdown
75 lines
6.7 KiB
Markdown
# Sitemap completion — extensions, incremental builder, gzip compressor
|
|
|
|
**Package**: `@suaveplan/sitemap`
|
|
**Status**: PROPOSED
|
|
**Type**: feature + spec-conformance
|
|
|
|
## Why
|
|
|
|
`@suaveplan/sitemap` (`packages/api/sitemap/`) currently ships only the core builder, index-builder, parser, and URL utilities. Four upstream surfaces are absent: the Google image-sitemap extension, the Google video-sitemap extension, the incremental on-disk builder, and the gzip compressor. Without them, callers must re-implement Google-spec XML by hand (image / video) or stream URLs through the in-memory builder until they hit Google's 50,000-URL / 50 MiB hard cap. All four legacy modules already exist (`packages/universal/sitemap/src/extensions/image-sitemap.ts` 111 L; `extensions/video-sitemap.ts` 182 L; `incremental/incremental-builder.ts` 213 L; `compression/sitemap-compressor.ts` 91 L) and pass the genesis quality bar — porting them closes the parity gap.
|
|
|
|
## What changes
|
|
|
|
### `@suaveplan/sitemap`
|
|
|
|
- ADDED: `SitemapImageExtension` — `addImage(urlEntry, image)` mutates a `SitemapUrl` to include a `<image:image>` block per Google's image-sitemap spec; supports `{ caption, location, license, title }`.
|
|
- ADDED: `SitemapVideoExtension` — `addVideo(urlEntry, video)` mutates a `SitemapUrl` to include a `<video:video>` block; supports `{ thumbnail, title, description, contentLoc, playerLoc, duration, expiration, rating, viewCount, publicationDate, familyFriendly, requiresSubscription, uploader, live }`.
|
|
- ADDED: `IMAGE_NS` / `VIDEO_NS` namespace constants and Zod schemas `SitemapImageExtensionSchema`, `SitemapVideoExtensionSchema`.
|
|
- ADDED: `IncrementalSitemapBuilder` class — append-only builder that streams URLs in batches to disk through an injectable `WritableStorageLike` adapter; preserves URL order; rolls over to a new sitemap file at 50,000 URLs OR 50 MiB (whichever first); emits a `SitemapIndex` entry per rollover.
|
|
- ADDED: `IncrementalSitemapBuilderOptions` interface — `outputDir`, `baseFilename` (default `sitemap`), `urlsPerSitemap` (default 50_000, max 50_000), `maxBytes` (default 50 *1024* 1024), `storage` (`WritableStorageLike`), `compress` (boolean — when true, each rollover output is gzipped).
|
|
- ADDED: `compressSitemap(xml: string): Promise<Uint8Array>` and `compressSitemapIndex(xml: string): Promise<Uint8Array>` async helpers — gzip with valid `gzip` magic bytes (`1f 8b`), DEFLATE compression level 6 (default), ISO-8859-1 OS field, mtime stamped from injectable `Clock`.
|
|
- ADDED: `decompressGzipSitemap(bytes: Uint8Array): Promise<string>` round-trip helper for tests.
|
|
- ADDED: `SITEMAP_CODES` extensions — `IMAGE_VALIDATION_FAILED`, `VIDEO_VALIDATION_FAILED`, `INCREMENTAL_ROLLOVER_FAILED`, `COMPRESSION_FAILED` registered via existing `defineCodes` block.
|
|
- ADDED: subpath exports `./extensions/image-sitemap`, `./extensions/video-sitemap`, `./incremental/incremental-builder`, `./compression/sitemap-compressor`.
|
|
- MODIFIED: root `src/index.ts` re-exports the new extensions, builder, and compressor helpers.
|
|
|
|
## Out of scope
|
|
|
|
- **News sitemap extension** (`<news:news>` block) — separate change. The Google news-sitemap is a different schema with publication-name / publication-language fields and a strict 30-day TTL window; folding it into this change dilutes review.
|
|
- **Mobile sitemap extension** — deprecated by Google in 2015; do not port.
|
|
- **Streaming gzip via `CompressionStream`** — keep the synchronous `fflate` path for now; once the gzip surface is stable a follow-up change can swap to a `WritableStream` pipeline.
|
|
- **`SitemapBuilder.compress()` instance method** — out of scope to keep the core builder side-effect free; callers gzip the output explicitly via `compressSitemap`.
|
|
|
|
## Source mapping
|
|
|
|
This change derives from `../suaveplan/enterprise/sitemap.md` and the upstream package source.
|
|
|
|
| Requirement | Source file | Source lines | Notes |
|
|
|---|---|---|---|
|
|
| `SitemapImageExtension` + Zod schema | `packages/universal/sitemap/src/extensions/image-sitemap.ts` | L1-L111 | Verbatim port; existing `SitemapImageSchema` in `schemas/sitemap.schema.ts` reused |
|
|
| `SitemapVideoExtension` + Zod schema | `packages/universal/sitemap/src/extensions/video-sitemap.ts` | L1-L182 | Verbatim port; existing `SitemapVideoSchema` reused |
|
|
| `IncrementalSitemapBuilder` | `packages/universal/sitemap/src/incremental/incremental-builder.ts` | L1-L213 | Verbatim port; replace `Date.now()` and `process.env.HOME` paths with injectable `Clock` and `WritableStorageLike` |
|
|
| `compressSitemap` / `compressSitemapIndex` / `decompressGzipSitemap` | `packages/universal/sitemap/src/compression/sitemap-compressor.ts` | L1-L91 | Verbatim port |
|
|
|
|
### Requirements with no source line
|
|
|
|
| Requirement | Justification |
|
|
|---|---|
|
|
| Injectable `Clock` for gzip mtime field | AGENTS.md determinism rule (no `Date.now()` in hot paths). Upstream stamps `Date.now()`, breaking deterministic test fixtures. |
|
|
| Injectable `WritableStorageLike` adapter for `IncrementalSitemapBuilder` | AGENTS.md §11 ("shared types belong in `@suaveplan/types`"). Upstream writes directly via `fs.promises.writeFile`, coupling the builder to Node. |
|
|
| `INCREMENTAL_ROLLOVER_FAILED` / `COMPRESSION_FAILED` error codes | AGENTS.md §13 row 3. Upstream throws plain `Error`. |
|
|
|
|
## Verification approach
|
|
|
|
Each requirement carries a `**Verified by:**` line in the spec delta.
|
|
|
|
- `packages/api/sitemap/src/extensions/image-sitemap/image-sitemap.test.ts` — covers Image extension scenarios
|
|
- `packages/api/sitemap/src/extensions/video-sitemap/video-sitemap.test.ts` — covers Video extension scenarios
|
|
- `packages/api/sitemap/src/incremental/incremental-builder/incremental-builder.test.ts` — covers append, rollover at 50k URLs, rollover at 50 MiB, index entry emission
|
|
- `packages/api/sitemap/src/incremental/incremental-builder/incremental-builder.spec-conformance.test.ts` — pins injected `WritableStorageLike` contract and Clock determinism
|
|
- `packages/api/sitemap/src/compression/sitemap-compressor/sitemap-compressor.test.ts` — covers gzip round-trip, magic-byte assertions, mtime determinism
|
|
|
|
## Impact
|
|
|
|
### Affected dependents
|
|
|
|
Run `bunx turbo ls --filter=...@suaveplan/sitemap` to enumerate.
|
|
|
|
- No dependents (verified by `grep -rl "@suaveplan/sitemap" /home/ubuntu/web/genesis/packages/ --include="package.json"` → only `packages/api/sitemap/package.json` itself).
|
|
|
|
Pure additive change.
|
|
|
|
## Migration impact
|
|
|
|
None — additive only. Existing `SitemapBuilder`, `SitemapIndexBuilder`, `parseSitemap`, `parseSitemapIndex` calls compile unchanged. The `SitemapUrl` type already carries optional `images?: SitemapImage[]` and `videos?: SitemapVideo[]` fields (in `schemas/sitemap.schema.ts`); the new extensions populate those fields rather than introducing a new shape.
|