# Pipeline

## Normal path

```text
smk_all_da.json + smk_all_en.json
        ↓
source reconnaissance + SHA-256 manifest
        ↓
streaming ingest
        ↓
language-aware canonical merge
        ↓
weighted palette parsing + colour conversion
        ↓
per-artwork feature extraction
        ↓
quality checks
        ↓
pilot diagnostics
        ↓
full feature build
        ↓
select/freeze normalization corpus
        ↓
normalized features
        ↓
similarity / sonification experiments
        ↓
compact web catalog + read-only application
```

## Recommended CLI stages

The exact CLI can evolve, but aim for commands conceptually like:

```bash
smk inspect-source
smk ingest
smk canonicalize
smk features
smk quality
smk normalize --profile smk-public-visual-v1
smk sonify --profile art-tone-v1
smk build-all
```

Each stage should expose explicit input/output paths and be independently rerunnable.

## Stage 0 — source reconnaissance

Before writing the final record parser:

- inspect the actual DA and EN top-level structure,
- count records without fully materializing the files in RAM,
- compute SHA-256 + byte sizes,
- sample representative records,
- compare identity sets between languages,
- report duplicates/missing IDs,
- create `source-manifest.json`.

## Stage 1 — audited source ingest

Milestone 1 streams `$[]` with `ijson`, rejects source-ID collisions, preserves every source row in
manifest-bound Parquet and produces duplicate, dimension and language-disagreement audits. It does
not yet collapse duplicate `object_number` groups or produce canonical artwork rows.

## Stage 2 — canonical entities + corpus freeze

Pair DA/EN by identical SMK `id`, preserve non-unique object-number groups, apply field-specific
provenance/conflict rules, resolve image bundles atomically, normalize supported dimension
observations to centimetres and write `artworks.parquet`. Freeze `smk-public-visual-v1` from
explicit rights/image/colors criteria and emit membership reasons. Colour feature extraction is a
later stage.

## Stage 3 — deterministic visual features

For each frozen corpus entity, deduplicate exact source palette colours, assign explicitly uniform
analytical weights, convert sRGB to OKLab/OKLCH, calculate compact palette statistics, carry source
visual values and derive eligible geometry. Emit scalar feature and long-form palette Parquets plus
diagnostics. Do not perform corpus-relative normalization or similarity here.

## Stage 4 — versioned corpus normalization

Profile all candidate scalars over the frozen 27,254-row corpus, freeze P01/P99, median and MAD,
and emit raw plus robust-01, empirical-midrank percentile and robust-z representations. Derive log
aspect ratios and circular hue diagnostics, preserve nulls, and publish separate unweighted color,
visual-character and optional geometry families. Similarity and final weights remain later stages.

## Stage 5 — deterministic exact similarity

Calibrate component scales from a fixed 500,000-pair sample, freeze separate color,
visual-character and geometry profiles, and perform exact full-corpus ranking with explainable
contributions. Missing optional fields renormalize shared weights; required fields determine
eligibility. No ANN index, universal score or palette transport is introduced.

## Stage 6 — enrichment-backed weighted color

Fetch only unique frozen-corpus object numbers into an append-only resumable cache, freeze a
separate source manifest, parse repeated palette tokens into explicit weights, emit weighted
palette/features v2, and add exact OKLab palette transport under a new similarity profile. Existing
bulk v1 corpus, features, normalization and profiles remain byte-independent inputs.

## Stage 7 — exploratory application

Join only the fields needed for interactive browsing into `web-catalog-v1.parquet`, freeze the
sonification profile manifest, and serve the built React client and FastAPI routes from one
container. Runtime startup validates artifacts but never triggers ingestion, enrichment, or
normalization. Processed inputs remain read-only; deep weighted-similarity responses use the
separate invalidatable cache.

## Ingest performance rule

The source files are large. Use bounded-memory streaming where practical. A one-time conversion to Parquet/Arrow as an interim representation is encouraged if it simplifies all later stages.

Avoid repeatedly reparsing both ~900 MB JSON snapshots for every downstream command when a manifest-bound interim artifact can be reused safely.

## Pilot

Before the full normalization build, process 100–500 representative records and inspect distributions and quality flags.

Include examples that are:

- colourful / near-monochrome,
- dark / bright,
- small / large,
- missing dimensions,
- 2D / non-2D,
- odd/missing palette data,
- present in only one language snapshot if such cases exist.

The pilot checks correctness; it is not artistic tuning.

## Full build

After a successful pilot:

1. canonicalize every valid identity,
2. emit deterministic features or explicit nulls/flags,
3. report actual corpus counts instead of expecting a hardcoded number,
4. define exact membership of `smk-public-visual-v1`,
5. compute and freeze corpus statistics,
6. emit normalized features,
7. generate summary counts/distributions and exclusions.