smk-playground/docs/SOURCE_RECONNAISSANCE.md

Source reconnaissance — Milestone 0

Generated from the local snapshots on 2026-10-08 (Europe/Copenhagen). The raw files were read sequentially and were not modified. The detailed machine-readable scan is written locally to data/reports/source-reconnaissance.json; it is intentionally ignored as generated output.

Files and exact structure

Language	Bytes	SHA-256	Records	Unique object_number
DA	910,209,972	8b47b7af20e0450d6a4bbc1f507edbc6f7479f28a8a100e627c6d533db8d9856	202,728	202,394
EN	860,820,598	b8343ee9b12034f5e0aea5ac31c3eed5d3822e576cedcbe87e8234726d34e374	203,023	202,688

Both files are a JSON top-level array with no envelope. Every item observed was an object, so artwork records are reached at $[]. The files are not interchangeable snapshots: their row counts and identity sets differ, and their last records differ. Row position must not be used to join them.

The scan used tools/source-reconnaissance.mjs under Node.js v24.12.0. It streams 1 MiB chunks, materializes only the current array element, and retains compact identity/fingerprint indexes. It does not call json.load() or materialize either source file as one object.

Record and identity summary

Measure	DA	EN
Raw records	202,728	203,023
Unique nonblank object_number	202,394	202,688
Missing/blank object_number	0	0
IDs occurring more than once	305	306
Extra rows beyond one per ID	334	335
public_domain == true	152,216	152,511
has_image == true	54,423	54,426
Non-empty dimensions	199,141	199,442
Non-empty colors	52,249	52,249
All five SMK visual fields present	45,024	45,024

The unique-ID union contains 202,689 identities:

• 202,393 occur in both snapshots;
• 1 is DA-only: KKSgb81877;
• 295 are EN-only (the first include KKS2026-53, KKS2026-54, KKS2026-55, KKS2026-56, and KKS2026-57).

The full duplicate and language-only lists are in the local machine report. Duplicate IDs must be resolved explicitly from record provenance in Milestone 1; selecting the first row silently would be arbitrary.

Dataset v1 field locations

All paths below are relative to one $[] record.

Dataset concern	Observed source path/shape	DA presence	EN presence	Notes
Identity	object_number: string	202,728	203,023	Present and nonblank in every row, but not unique.
Public domain	public_domain: boolean	202,728	203,023	Values agree on all overlapping unique IDs in the selected-field comparison.
Titles	titles[].title, plus type and often language	202,724 arrays	203,015 arrays	The EN snapshot is not guaranteed to contain English titles; the first EN record still has a Danish title/language tag.
Object type/name	object_names[].name	202,699 arrays	202,933 arrays	Localized (Grafik vs Print in the first record).
Physical dimensions	dimensions[].{part,type,unit,value}	199,141 arrays	199,442 arrays	Values are strings; labels/units are localized (højde/millimeter vs height/mm). Select meaningful parts such as Netto only after vocabulary inspection.
Image geometry	image_width, image_height	54,272 each	54,275 each	Numeric pixels; not physical dimensions.
Image availability	has_image	every row	every row	Seven overlapping IDs disagree between snapshots.
Unweighted palette	colors: array<string>	52,249	52,249	Selected-field fingerprints agree across overlap.
Suggested background	suggested_bg_color: array<string>	52,248	52,248	Selected-field fingerprints agree across overlap.
SMK visual values	top-level brightness, contrast, saturation, colortemp, entropy	see below	see below	colortemp has mixed numeric/string types.

Visual-field presence is identical between languages: brightness 52,171, contrast 52,171, saturation 45,024, colortemp 52,249, and entropy 52,249. colortemp is numeric in 52,237 records and a string in 12 records in each file. No field whose name contains palette was observed at any depth. In particular, colors_palette_weighted_s is absent from both bulk snapshots; Milestone 2 cannot claim source weights from these files.

Representative record KKS5261 demonstrates the localization boundary: identity, image geometry, colours and visual numbers match, while dimensions use DA højde/bredde with millimeter versus EN height/width with mm, and object names differ. Record KKS370 demonstrates a normal record with physical dimensions but has_image == false and no palette or visual values.

Cross-language discrepancies

For the 202,393 overlapping unique IDs, compact fingerprints of selected nominally language-independent top-level fields found these mismatch counts:

• rights: 50,468; modified: 334;
• image_width and image_height: 15 each;
• image_iiif_id and image_iiif_info: 11 each; image_size: 9;
• has_image: 7; image_mime_type: 6; image_hq: 6;
• image_orientation: 8; image_cropped: 1; on_display: 2.

Selected colors, suggested_bg_color, the five visual values, public_domain, id, creation date, acquisition date and the stable URLs had zero mismatches in this comparison. Differences on duplicate IDs require duplicate-aware reinspection because the compact comparison retains one fingerprint per identity; they must not be interpreted as a final canonical precedence decision.

Ambiguities carried into Milestones 1–3

1. object_number is complete but non-unique. Duplicate records need comparison by source id, timestamps and content before a deterministic keep/merge/error policy can be selected.
2. The snapshots are temporally asymmetric (295 EN-only versus one DA-only identity) and some nominally language-independent image metadata differs. Missing one language must be retained, and DA precedence alone is not safe for every image field.
3. The exact physical-dimension vocabulary and choice of meaningful Netto/Brutto/part variants still need a bounded vocabulary profile before unit canonicalization. Values are strings and labels/units are localized.
4. The bulk files have no explicit weighted palette. A later enrichment input, separately manifested and provenance-labelled, is needed for source weights; unweighted colors must not be presented as weighted SMK data.
5. colortemp needs explicit validation/coercion for its 12 string-valued cases. Titles in the EN file cannot be treated as English merely because the file is the EN snapshot.
6. Corpus membership remains undefined. Counts above describe raw snapshots and observable field availability, not membership in smk-public-visual-v1.

Recommended next parser/canonicalization approach

For Milestone 1, use ijson.items(stream, "item") against the proven $[] path and write a manifest-bound interim Parquet dataset per language in batches. Preserve source row number, source id, object_number, snapshot language and source-manifest hash. Build a duplicate audit grouped by object_number before canonical merge; do not silently deduplicate.

Canonicalize DA and EN independently first, then full-outer-join by resolved object_number. Keep localized fields separately and choose each language-independent field through an explicit field-level rule that records agreement/disagreement and provenance. Profile dimension part/type/unit vocabularies before mapping Netto width/height/depth to centimetres. Preserve mixed or unknown values as null plus diagnostics. Treat top-level colors as an unweighted source palette only; defer weighted palette parsing until an actual manifested weight-bearing source has been established.