diff --git a/AGENTS.md b/AGENTS.md index f191a8c..9d7d7f6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -167,6 +167,10 @@ Parquet is the portable source of truth for derived tabular output. DuckDB may p - Use `smk::` as source-row identity for the audited snapshots. Never use array position. Treat duplicate source-row IDs as errors and repeated `object_number` values as unresolved groups until an explicit canonical rule handles them. +- Use the top-level SMK `id` without language as `smk_entity_id`. Pair DA/EN only when that ID is + identical. `object_number` is explicitly non-unique and group members remain separate entities. +- Resolve image metadata as an atomic bundle; never combine image identifiers from one source with + pixel dimensions or URLs from the other. ## Canonical artwork rules diff --git a/README.md b/README.md index 2724344..4092b66 100644 --- a/README.md +++ b/README.md @@ -121,10 +121,9 @@ See: ## Current status -Milestones 0–1 complete: the real source structure is manifested, both snapshots have -bounded-memory source Parquets, and duplicate identity, dimensions, palette/visual quality, -rights and DA/EN disagreements are audited. Canonical artwork generation is next; visual feature -extraction has not started. +Milestones 0–2 complete: source structure and audits are manifested, the language snapshots have +bounded-memory source Parquets, DA/EN rows are paired into stable SMK entities, and the exact +`smk-public-visual-v1` membership rule is frozen. Visual feature extraction has not started. Audit results are documented in: @@ -133,9 +132,12 @@ Audit results are documented in: - [`docs/SOURCE_DISAGREEMENTS.md`](docs/SOURCE_DISAGREEMENTS.md) - [`docs/PALETTE_VISUAL_AUDIT.md`](docs/PALETTE_VISUAL_AUDIT.md) - [`docs/RIGHTS_AUDIT.md`](docs/RIGHTS_AUDIT.md) +- [`docs/CANONICALIZATION.md`](docs/CANONICALIZATION.md) +- [`docs/CORPUS_V1.md`](docs/CORPUS_V1.md) Run the audited ingest with: ```bash smk milestone1 +smk milestone2 ``` diff --git a/docs/CANONICALIZATION.md b/docs/CANONICALIZATION.md new file mode 100644 index 0000000..1617625 --- /dev/null +++ b/docs/CANONICALIZATION.md @@ -0,0 +1,89 @@ +# Canonicalization v1 + +Milestone 2 builds `data/processed/artworks.parquet` from the two manifest-bound source Parquets. +It produces one row per SMK entity and does not calculate OKLab, palette weights or colour-derived +features. + +## Identity levels + +1. **Source row** — `source_row_id = smk::`. +2. **SMK entity** — `smk_entity_id = `, without language. +3. **Artwork-facing identifier** — `object_number`, retained as a non-unique accession/business + identifier. + +DA and EN rows are paired only when their top-level SMK `id` is identical. Different SMK IDs are +never paired merely because `object_number` matches. The audited result has 203,023 entities: +202,728 paired DA+EN entities and 295 EN-only entities. No entity is DA-only. + +One paired entity, `1170314109_object`, has conflicting object numbers: DA `KKSgb81877` and EN +`KKSgb82299`. Its canonical `object_number` is null and it is flagged +`CANONICALIZATION_UNRESOLVED`. + +## Field rules + +For source-independent scalar/list fields: + +- exact DA/EN agreement → canonical value with provenance `both`; +- value in only one language → canonical value with provenance `da` or `en`; +- genuine disagreement → both originals retained, canonical value null, status `conflict` unless + an audited field rule exists. + +Localized titles, object names/classifications, artists, production data, techniques and materials +remain in separate DA/EN columns. Source row IDs, source-file hashes and manifest hashes are +retained on every entity. + +## Image bundle + +Image fields are resolved atomically. A complete bundle requires `has_image == true`, positive +integer width and height, a non-empty IIIF image ID, and at least one thumbnail or native/public +image URL. + +Resolution order is exact matching complete bundle, one complete source against an absent or +incomplete source, then unresolved conflict if both sources have different complete bundles. No +canonical bundle mixes fields from DA and EN. + +| Status | Entities | +|---|---:| +| `IMAGE_BUNDLE_OK` | 39,956 | +| `IMAGE_BUNDLE_SINGLE_SOURCE` | 8 | +| `IMAGE_BUNDLE_CONFLICT` | 3 | +| `IMAGE_BUNDLE_INCOMPLETE` | 163,056 | + +The three conflicts are `KMSsp451`, `KKSbog14617c` and `KKS1984-179`. Both source bundles remain +available for inspection. + +## Dimensions + +Every source observation is preserved as a nested typed struct containing canonical part/type/unit, +centimetre value where supported, original language/value fields, source-row ID and mapping status. +Non-geometric weight, duration and register-tonnage observations are preserved but not converted. + +Candidate slots exist for width, height, depth, diameter and length under `net`, `gross`, `sheet`, +`plate`, `mount`, `image` and `paper`. Repeated equal observations agree. Multiple distinct values +for one slot make that slot null and retain all candidates. + +- 197,517 entities have at least one normalized physical dimension; +- 9,902 have canonical Netto width and 10,331 have Netto height; +- 9,897 have a complete conflict-free Netto width × height representation; +- 1,798 additionally have Netto depth; +- 227 entities have at least one dimension-slot conflict. + +`has_2d_physical_geometry` means conflict-free Netto width and height. It does not claim every +entity is rectangular or a 2D artwork. `has_3d_physical_geometry` additionally requires Netto depth. + +## Rights and visual data + +Known rights URLs map to `PDM`, `CC0` or `SMK_RESTRICTED_POLICY`; originals remain in `rights_da` +and `rights_en`. Counts are 151,913 PDM, 598 CC0 and 50,512 restricted-policy entities. +`public_domain` remains a separately resolved explicit source field. + +The 12 source `"Infinity"` colortemp strings become canonical null with +`COLORTEMP_NONFINITE_SOURCE`. Finite values are preserved without clipping. `COLORTEMP_EXTREME` +uses absolute robust z-score greater than 10, calculated with median 3.4118087 and MAD 0.1786611 +over 52,237 entity values; 481 entities are flagged. This is a statistical diagnostic, not a +scientific validity claim. + +Canonical rows contain deterministic sorted quality flags, including language missingness, shared +object number, dimension conflict, image status, colortemp diagnostics, partial visual fields and +unresolved canonicalization. + diff --git a/docs/CORPUS_V1.md b/docs/CORPUS_V1.md new file mode 100644 index 0000000..2b457fe --- /dev/null +++ b/docs/CORPUS_V1.md @@ -0,0 +1,70 @@ +# Corpus: `smk-public-visual-v1` + +Milestone 2 freezes the first corpus for openly reusable visual/colour analysis. Membership is +stored on every canonical entity and materialized in +`data/processed/smk-public-visual-v1.parquet`. + +## Exact inclusion predicate + +An entity is included only when all conditions hold: + +1. canonicalization is resolved; +2. explicit canonical `public_domain == true`; +3. canonical rights semantic is `PDM` or `CC0`; +4. the atomic canonical image bundle is complete; +5. image width and height are positive (part of bundle completeness); +6. canonical `colors` is a non-empty list; +7. there is no unresolved image-bundle conflict. + +The rule does **not** require physical dimensions, English metadata, visual scalar fields, +weighted palettes or enrichment data. + +## Membership + +| Measure | Count | +|---|---:| +| Canonical entities | 203,023 | +| Included | 27,254 | +| Excluded | 175,769 | + +Exclusion reasons are non-exclusive: + +| Code | Count | +|---|---:| +| `NO_USABLE_IMAGE` | 163,059 | +| `NO_COLORS` | 150,774 | +| `NOT_PUBLIC_DOMAIN` | 50,512 | +| `RIGHTS_NOT_OPEN` | 50,512 | +| `IMAGE_CONFLICT` | 3 | +| `CANONICALIZATION_UNRESOLVED` | 1 | + +Milestone 1 observed about 39.5k public-domain rows with an image/public URL. Recomputed at entity +level, that broad candidate definition contains 39,508 entities. All 27,254 corpus members overlap +it. The stricter corpus is smaller because it additionally requires a complete atomic image bundle +and non-empty `colors`; it is not forced to an expected count. + +## Included-data diagnostics + +- 321 included entities have no normalized physical dimension; +- 25,065 lack a complete Netto width × height pair; +- 871 have partial visual fields; +- missing visual values: saturation 871, brightness 62, contrast 62, entropy 1, colortemp 1; +- 143 included entities have a colortemp quality flag. + +Physical and visual missingness does not exclude a work because those properties are independent +of whether its primary image and source colours are usable. + +## Shared object numbers + +There are 306 `object_number` groups containing more than one canonical SMK entity; the largest +contains 24 entities. Nineteen such groups contain corpus members, representing 35 included +entities. Members remain separate entities. `data/processed/object-number-groups.parquet` provides +the relationship for later analysis/UI without destructive merging. + +## Artifacts + +- `data/processed/artworks.parquet` — every canonical SMK entity; +- `data/processed/smk-public-visual-v1.parquet` — included entities; +- `data/reports/corpus-membership.parquet` — decision and reason codes for every entity; +- `data/processed/object-number-groups.parquet` — non-destructive accession-number grouping; +- `data/reports/milestone2-summary.json` — machine-readable counts and rule parameters. diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index e476921..dd73c25 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -20,7 +20,9 @@ Values calculated relative to a named, frozen corpus/profile. Identifies exact input snapshots by filename, size, SHA-256, observed JSON structure and record counts. ### `artworks.parquet` -One canonical row per artwork identity. +One canonical row per SMK entity, produced by pairing DA/EN source rows with the same top-level +SMK `id`. It includes localized values, field-level provenance/conflict status, atomic image +resolution, typed dimension observations/candidates, quality flags and corpus membership. Suggested fields include: @@ -76,11 +78,23 @@ content and report the fallback. Array index is never permanent identity. A dupl `source_row_id` is a hard ingest error. Keep `smk_source_id`, `object_number`, language, source-file hash and logical record hash alongside -the internal ID. Canonical artwork identity is deliberately deferred until duplicate groups have -an explicit resolution; it must not silently equate every repeated `object_number` row. +the internal ID. The canonical entity key is the language-independent top-level SMK `id`. +Repeated `object_number` values remain separate canonical entities and are represented in +`object-number-groups.parquet`; they are never collapsed implicitly. If duplicate IDs exist, inspect them and implement an explicit deterministic policy with provenance; do not keep an arbitrary “first row”. +### Canonical identity + +```text +source_row_id = smk:: +smk_entity_id = +object_number = non-unique accession/business identifier +``` + +DA and EN rows pair only by identical `smk_entity_id`. A disagreement in `object_number` becomes +an unresolved field conflict, not a cross-ID pairing rule. + ## Manifest-bound source Parquet `source_da.parquet` and `source_en.parquet` contain one row per source row, including selected typed @@ -99,3 +113,7 @@ Zero is a real value and must never stand in for missing data. Canonical ingestion and feature extraction may include records that are not eligible for `smk-public-visual-v1`. Normalization membership is a separate, explicit decision. Store inclusion/exclusion reasons rather than deleting records merely because a downstream profile cannot use them. + +`smk-public-visual-v1` is now frozen as an analyzable source corpus, not a normalization profile. +Its exact predicate and counts are documented in `CORPUS_V1.md`. Later corpus-relative feature +normalization remains independently versioned. diff --git a/docs/IMPLEMENTATION_PLAN.md b/docs/IMPLEMENTATION_PLAN.md index 00d1945..124806f 100644 --- a/docs/IMPLEMENTATION_PLAN.md +++ b/docs/IMPLEMENTATION_PLAN.md @@ -28,10 +28,22 @@ Done when a representative fixture/sample becomes deterministic canonical rows. Status: source loading and pre-canonical audits complete. Stable source-row identity, lossless logical Parquet, duplicate taxonomy, dimension vocabulary, palette/visual quality and DA/EN -precedence recommendations are documented. Canonical artwork generation remains blocked on an -explicit duplicate-group resolution policy and is intentionally not performed in this milestone. +precedence recommendations are documented. Canonical entities are implemented in Milestone 2 +without collapsing duplicate `object_number` groups. -## Milestone 2 — Weighted palette + colour conversion +## Milestone 2 — Canonical entities + first analyzable corpus + +- pair language rows only by SMK `id` +- preserve `object_number` as non-unique grouping metadata +- canonicalize fields with explicit provenance and conflicts +- resolve image bundles atomically +- preserve and normalize dimension observations without choosing conflicts +- freeze `smk-public-visual-v1` with machine-readable exclusion reasons + +Status: complete. See `CANONICALIZATION.md` and `CORPUS_V1.md`. Visual/colour feature extraction +has not started. + +## Milestone 3 — Weighted palette + colour conversion - locate ordinary/weighted palette fields in the actual snapshot/enrichment data - parse `colors_palette_weighted_s` when present @@ -41,7 +53,7 @@ explicit duplicate-group resolution policy and is intentionally not performed in Done when known palette fixtures produce stable colour rows and valid weights. -## Milestone 3 — Feature extraction +## Milestone 4 — Feature extraction Implement: @@ -56,7 +68,7 @@ Implement: Done when one artwork can produce a complete feature record independently of the corpus. -## Milestone 4 — Quality layer +## Milestone 5 — Quality layer Implement documented flags and source/language/aspect checks. @@ -73,18 +85,18 @@ Produce `quality-report.json` with: - palette-size distribution - obvious extreme values -## Milestone 5 — Pilot build +## Milestone 6 — Pilot build Run 100–500 representative artworks. Review distributions and manually inspect selected records. Fix parser/source/math errors only; do not artistically tune features. -## Milestone 6 — Corpus definition + normalization +## Milestone 7 — Corpus-relative normalization -Do not hardcode 39,508. +Use the frozen `smk-public-visual-v1` entity membership from Milestone 2. Do not redefine corpus +membership from an expected count. -- define the exact inclusion predicate for `smk-public-visual-v1` -- record included/excluded counts and reasons +- reference included/excluded counts and reasons from the frozen corpus artifact - compute count/null/min/max/quantiles - median + MAD - implement `robust_01` @@ -92,7 +104,7 @@ Do not hardcode 39,508. - implement robust z-score - freeze profile with source-manifest reference -## Milestone 7 — Full dataset build +## Milestone 8 — Full dataset build Run the complete local source snapshots. @@ -100,7 +112,7 @@ Definition of done: Every canonical artwork has valid applicable features or explicit nulls/flags explaining missing/invalid prerequisites. Every normalization result references a frozen profile and exact source manifest. -## Milestone 8 — Similarity +## Milestone 9 — Similarity Implement separate profiles rather than one magic similarity score: @@ -110,7 +122,7 @@ Implement separate profiles rather than one magic similarity score: Use fast feature-vector candidate retrieval followed optionally by weighted-palette optimal-transport reranking. Do not precompute all pairs. -## Milestone 9 — Art Tone v1 +## Milestone 10 — Art Tone v1 Implement versioned, explainable sonification mapping separate from objective features. @@ -126,7 +138,7 @@ Initial mapping: Emit exact inputs alongside every mapped result. -## Milestone 10 — Exploration +## Milestone 11 — Exploration Only after Dataset v1 is trustworthy, start experimental interfaces/notebooks/apps: diff --git a/docs/PIPELINE.md b/docs/PIPELINE.md index c2b4d6c..201e70d 100644 --- a/docs/PIPELINE.md +++ b/docs/PIPELINE.md @@ -63,6 +63,14 @@ Milestone 1 streams `$[]` with `ijson`, rejects source-ID collisions, preserves manifest-bound Parquet and produces duplicate, dimension and language-disagreement audits. It does not yet collapse duplicate `object_number` groups or produce canonical artwork rows. +## Stage 2 — canonical entities + corpus freeze + +Pair DA/EN by identical SMK `id`, preserve non-unique object-number groups, apply field-specific +provenance/conflict rules, resolve image bundles atomically, normalize supported dimension +observations to centimetres and write `artworks.parquet`. Freeze `smk-public-visual-v1` from +explicit rights/image/colors criteria and emit membership reasons. Colour feature extraction is a +later stage. + ## Ingest performance rule The source files are large. Use bounded-memory streaming where practical. A one-time conversion to Parquet/Arrow as an interim representation is encouraged if it simplifies all later stages. diff --git a/src/smk_playground/canonical/milestone2.py b/src/smk_playground/canonical/milestone2.py new file mode 100644 index 0000000..c7a71fe --- /dev/null +++ b/src/smk_playground/canonical/milestone2.py @@ -0,0 +1,598 @@ +"""Build the Milestone 2 canonical entity layer and frozen visual corpus.""" + +from __future__ import annotations + +import argparse +import json +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + +import duckdb +import pyarrow as pa +import pyarrow.parquet as pq + +from smk_playground.canonical.rules import ( + IMAGE_KEYS, + SUPPORTED_AXES, + SUPPORTED_PARTS, + corpus_membership, + dimension_slots, + image_bundle, + image_bundle_complete, + is_extreme_colortemp, + normalize_dimension_observations, + resolve_image_bundle, + resolve_scalar, + resolve_smk_entity_id, + rights_semantic, + robust_colortemp_parameters, + visual_number, +) +from smk_playground.ingest.source import logical_json + +CANONICAL_VERSION = "1.0.0" +VISUAL_FIELDS = ("brightness", "contrast", "saturation", "entropy", "colortemp") +DIMENSION_STRUCT = pa.struct([ + ("part_canonical", pa.string()), + ("type_canonical", pa.string()), + ("unit_canonical", pa.string()), + ("value_canonical", pa.float64()), + ("original_language", pa.string()), + ("original_part", pa.string()), + ("original_type", pa.string()), + ("original_unit", pa.string()), + ("original_value", pa.string()), + ("source_row_id", pa.string()), + ("status", pa.string()), +]) + + +def _schema() -> pa.Schema: + fields: list[tuple[str, pa.DataType]] = [ + ("canonical_schema_version", pa.string()), + ("smk_entity_id", pa.string()), + ("object_number", pa.string()), + ("object_number_da", pa.string()), + ("object_number_en", pa.string()), + ("object_number_provenance", pa.string()), + ("object_number_status", pa.string()), + ("source_row_id_da", pa.string()), + ("source_row_id_en", pa.string()), + ("present_in_da", pa.bool_()), + ("present_in_en", pa.bool_()), + ("source_manifest_sha256_da", pa.string()), + ("source_manifest_sha256_en", pa.string()), + ("source_file_sha256_da", pa.string()), + ("source_file_sha256_en", pa.string()), + ("titles_da_json", pa.string()), + ("titles_en_json", pa.string()), + ("object_names_da_json", pa.string()), + ("object_names_en_json", pa.string()), + ("classifications_da_json", pa.string()), + ("classifications_en_json", pa.string()), + ("artists_da_json", pa.string()), + ("artists_en_json", pa.string()), + ("production_da_json", pa.string()), + ("production_en_json", pa.string()), + ("production_dates_da_json", pa.string()), + ("production_dates_en_json", pa.string()), + ("techniques_da_json", pa.string()), + ("techniques_en_json", pa.string()), + ("materials_da_json", pa.string()), + ("materials_en_json", pa.string()), + ("public_domain", pa.bool_()), + ("public_domain_da", pa.bool_()), + ("public_domain_en", pa.bool_()), + ("public_domain_provenance", pa.string()), + ("public_domain_status", pa.string()), + ("rights_da", pa.string()), + ("rights_en", pa.string()), + ("rights_semantic", pa.string()), + ("rights_semantic_da", pa.string()), + ("rights_semantic_en", pa.string()), + ("rights_provenance", pa.string()), + ("rights_status", pa.string()), + ("colors", pa.list_(pa.string())), + ("colors_da", pa.list_(pa.string())), + ("colors_en", pa.list_(pa.string())), + ("colors_provenance", pa.string()), + ("colors_status", pa.string()), + ] + for field in VISUAL_FIELDS: + fields.extend([ + (field, pa.float64()), + (f"{field}_da_json", pa.string()), + (f"{field}_en_json", pa.string()), + (f"{field}_provenance", pa.string()), + (f"{field}_status", pa.string()), + ]) + fields.extend([ + ("image_bundle_status", pa.string()), + ("image_bundle_provenance", pa.string()), + ("image_bundle_complete", pa.bool_()), + ("image_bundle_da_json", pa.string()), + ("image_bundle_en_json", pa.string()), + ("has_image", pa.bool_()), + ("image_iiif_id", pa.string()), + ("image_iiif_info", pa.string()), + ("image_width", pa.int64()), + ("image_height", pa.int64()), + ("image_thumbnail", pa.string()), + ("image_native", pa.string()), + ("image_mime_type", pa.string()), + ("image_cropped", pa.bool_()), + ("image_orientation", pa.string()), + ("image_hq", pa.bool_()), + ("dimension_observations", pa.list_(DIMENSION_STRUCT)), + ("dimension_conflict_slots", pa.list_(pa.string())), + ("dimension_conflict_candidates", pa.list_(DIMENSION_STRUCT)), + ]) + for part in SUPPORTED_PARTS: + for axis in SUPPORTED_AXES: + fields.append((f"{part}_{axis}_cm", pa.float64())) + fields.extend([ + ("has_net_width", pa.bool_()), + ("has_net_height", pa.bool_()), + ("has_2d_physical_geometry", pa.bool_()), + ("has_3d_physical_geometry", pa.bool_()), + ("has_any_physical_dimension", pa.bool_()), + ("quality_flags", pa.list_(pa.string())), + ("previous_public_visual_candidate", pa.bool_()), + ("corpus_smk_public_visual_v1", pa.bool_()), + ("corpus_exclusion_reasons", pa.list_(pa.string())), + ]) + return pa.schema(fields) + + +MEMBERSHIP_SCHEMA = pa.schema([ + ("smk_entity_id", pa.string()), + ("object_number", pa.string()), + ("included", pa.bool_()), + ("exclusion_reasons", pa.list_(pa.string())), + ("quality_flags", pa.list_(pa.string())), +]) + +GROUP_SCHEMA = pa.schema([ + ("object_number", pa.string()), + ("canonical_entity_ids", pa.list_(pa.string())), + ("group_size", pa.int64()), + ("group_classification", pa.string()), + ("ambiguity_flag", pa.bool_()), + ("corpus_member_count", pa.int64()), +]) + + +def _raw_value(record: dict[str, Any] | None, key: str) -> Any: + return record.get(key) if record else None + + +def _json_value(record: dict[str, Any] | None, key: str) -> str | None: + value = _raw_value(record, key) + return logical_json(value) if value is not None else None + + +def _localized_fields(row: dict[str, Any], da: dict[str, Any] | None, en: dict[str, Any] | None) -> None: + mappings = { + "titles": "titles", + "object_names": "object_names", + "classifications": "classification", + "artists": "artist", + "production": "production", + "production_dates": "production_date", + "techniques": "techniques", + "materials": "materials", + } + for output, source in mappings.items(): + row[f"{output}_da_json"] = _json_value(da, source) + row[f"{output}_en_json"] = _json_value(en, source) + + +def _visual_fields( + row: dict[str, Any], da: dict[str, Any] | None, en: dict[str, Any] | None, +) -> list[str]: + flags: list[str] = [] + for field in VISUAL_FIELDS: + da_raw = _raw_value(da, field) + en_raw = _raw_value(en, field) + da_number, da_status = visual_number(da, field) + en_number, en_status = visual_number(en, field) + canonical, provenance, status = resolve_scalar(da_number, en_number) + if da_raw is not None and da_number is None: + status = f"invalid_da:{da_status}" + if en_raw is not None and en_number is None: + status = f"invalid_en:{en_status}" if status == "missing" else f"{status}|invalid_en:{en_status}" + row[field] = canonical + row[f"{field}_da_json"] = logical_json(da_raw) if da_raw is not None else None + row[f"{field}_en_json"] = logical_json(en_raw) if en_raw is not None else None + row[f"{field}_provenance"] = provenance + row[f"{field}_status"] = status + if field == "colortemp" and ( + (isinstance(da_raw, str) and da_raw.casefold() == "infinity") + or (isinstance(en_raw, str) and en_raw.casefold() == "infinity") + ): + flags.append("COLORTEMP_NONFINITE_SOURCE") + if any(row[field] is None for field in VISUAL_FIELDS): + flags.append("VISUAL_FIELDS_PARTIAL") + return flags + + +def _canonical_row( + pair: dict[str, Any], shared_object_numbers: set[str], colortemp_median: float, + colortemp_mad: float, +) -> dict[str, Any]: + da = json.loads(pair["da_record_json"]) if pair["da_record_json"] else None + en = json.loads(pair["en_record_json"]) if pair["en_record_json"] else None + da_entity_id = da.get("id") if da else None + en_entity_id = en.get("id") if en else None + entity_id = resolve_smk_entity_id(da_entity_id, en_entity_id) + if pair["smk_entity_id"] != entity_id: + raise ValueError("Joined SMK entity id does not match source record id") + row: dict[str, Any] = { + "canonical_schema_version": CANONICAL_VERSION, + "smk_entity_id": entity_id, + "source_row_id_da": pair["da_source_row_id"], + "source_row_id_en": pair["en_source_row_id"], + "present_in_da": da is not None, + "present_in_en": en is not None, + "source_manifest_sha256_da": pair["da_manifest_sha"], + "source_manifest_sha256_en": pair["en_manifest_sha"], + "source_file_sha256_da": pair["da_file_sha"], + "source_file_sha256_en": pair["en_file_sha"], + } + da_object_number = _raw_value(da, "object_number") + en_object_number = _raw_value(en, "object_number") + object_number, provenance, status = resolve_scalar(da_object_number, en_object_number) + row.update({ + "object_number": object_number, + "object_number_da": da_object_number, + "object_number_en": en_object_number, + "object_number_provenance": provenance, + "object_number_status": status, + }) + _localized_fields(row, da, en) + + da_public = _raw_value(da, "public_domain") + en_public = _raw_value(en, "public_domain") + public_domain, public_provenance, public_status = resolve_scalar(da_public, en_public) + row.update({ + "public_domain": public_domain, + "public_domain_da": da_public, + "public_domain_en": en_public, + "public_domain_provenance": public_provenance, + "public_domain_status": public_status, + }) + + da_rights = _raw_value(da, "rights") + en_rights = _raw_value(en, "rights") + da_semantic = rights_semantic(da_rights) + en_semantic = rights_semantic(en_rights) + if da is not None and en is not None and da_semantic == en_semantic: + rights_value, rights_provenance, rights_status = da_semantic, "both", "semantic_agreement" + elif da is not None and en is None: + rights_value, rights_provenance, rights_status = da_semantic, "da", "single_source" + elif en is not None and da is None: + rights_value, rights_provenance, rights_status = en_semantic, "en", "single_source" + else: + rights_value, rights_provenance, rights_status = None, "conflict", "conflict" + row.update({ + "rights_da": da_rights, + "rights_en": en_rights, + "rights_semantic": rights_value, + "rights_semantic_da": da_semantic if da is not None else None, + "rights_semantic_en": en_semantic if en is not None else None, + "rights_provenance": rights_provenance, + "rights_status": rights_status, + }) + + da_colors = _raw_value(da, "colors") if isinstance(_raw_value(da, "colors"), list) else None + en_colors = _raw_value(en, "colors") if isinstance(_raw_value(en, "colors"), list) else None + colors, colors_provenance, colors_status = resolve_scalar(da_colors, en_colors) + row.update({ + "colors": colors, + "colors_da": da_colors, + "colors_en": en_colors, + "colors_provenance": colors_provenance, + "colors_status": colors_status, + }) + + quality_flags = _visual_fields(row, da, en) + if is_extreme_colortemp(row["colortemp"], colortemp_median, colortemp_mad): + quality_flags.append("COLORTEMP_EXTREME") + + da_bundle = image_bundle(da) + en_bundle = image_bundle(en) + canonical_bundle, image_provenance, image_status = resolve_image_bundle(da_bundle, en_bundle) + row.update({ + "image_bundle_status": image_status, + "image_bundle_provenance": image_provenance, + "image_bundle_complete": image_bundle_complete(canonical_bundle), + "image_bundle_da_json": logical_json(da_bundle) if da_bundle is not None else None, + "image_bundle_en_json": logical_json(en_bundle) if en_bundle is not None else None, + }) + for key in IMAGE_KEYS: + row[key] = canonical_bundle.get(key) if canonical_bundle else None + + observations = normalize_dimension_observations(da, "da", pair["da_source_row_id"]) + observations += normalize_dimension_observations(en, "en", pair["en_source_row_id"]) + slots, dimension_conflicts, conflict_candidates = dimension_slots(observations) + row.update(slots) + row["dimension_observations"] = observations + row["dimension_conflict_slots"] = dimension_conflicts + row["dimension_conflict_candidates"] = conflict_candidates + row["has_net_width"] = row["net_width_cm"] is not None + row["has_net_height"] = row["net_height_cm"] is not None + row["has_2d_physical_geometry"] = row["has_net_width"] and row["has_net_height"] + row["has_3d_physical_geometry"] = ( + row["has_2d_physical_geometry"] and row["net_depth_cm"] is not None + ) + row["has_any_physical_dimension"] = any(value is not None for value in slots.values()) + + def previous_candidate(record: dict[str, Any] | None) -> bool: + return bool( + record and record.get("public_domain") is True and record.get("has_image") is True + and (record.get("image_native") or record.get("image_thumbnail") or record.get("image_iiif_id")) + ) + + row["previous_public_visual_candidate"] = previous_candidate(da) or previous_candidate(en) + + if da is None or en is None: + quality_flags.append("SOURCE_LANGUAGE_MISSING") + if object_number in shared_object_numbers: + quality_flags.append("OBJECT_NUMBER_SHARED") + if dimension_conflicts: + quality_flags.append("DIMENSION_CONFLICT") + quality_flags.append(image_status) + unresolved = any(status_value == "conflict" for status_value in ( + status, public_status, rights_status, colors_status, + *(row[f"{field}_status"] for field in VISUAL_FIELDS), + )) + if unresolved: + quality_flags.append("CANONICALIZATION_UNRESOLVED") + row["quality_flags"] = sorted(set(quality_flags)) + included, reasons = corpus_membership(row) + row["corpus_smk_public_visual_v1"] = included + row["corpus_exclusion_reasons"] = reasons + return row + + +def _colortemp_parameters(source_paths: tuple[Path, Path]) -> tuple[float, float, int]: + values_by_entity: dict[str, float] = {} + for source_path in source_paths: + parquet = pq.ParquetFile(source_path) + for batch in parquet.iter_batches(columns=["smk_source_id", "visual_json"], batch_size=10_000): + for row in batch.to_pylist(): + if row["smk_source_id"] in values_by_entity: + continue + visual = json.loads(row["visual_json"]) + number, _ = visual_number(visual, "colortemp") + if number is not None: + values_by_entity[row["smk_source_id"]] = number + values = list(values_by_entity.values()) + median, mad = robust_colortemp_parameters(values) + return median, mad, len(values) + + +def _shared_object_numbers(connection: duckdb.DuckDBPyConnection, da_path: str, en_path: str) -> set[str]: + query = f""" + SELECT object_number + FROM ( + SELECT smk_source_id, object_number FROM read_parquet('{da_path}') + UNION + SELECT smk_source_id, object_number FROM read_parquet('{en_path}') + ) + WHERE object_number IS NOT NULL + GROUP BY object_number + HAVING count(DISTINCT smk_source_id) > 1 + """ + return {row[0] for row in connection.execute(query).fetchall()} + + +def run(root: Path) -> dict[str, Any]: + da_path = (root / "data" / "interim" / "source_da.parquet").resolve() + en_path = (root / "data" / "interim" / "source_en.parquet").resolve() + processed = root / "data" / "processed" + reports = root / "data" / "reports" + processed.mkdir(parents=True, exist_ok=True) + reports.mkdir(parents=True, exist_ok=True) + connection = duckdb.connect() + da_sql = da_path.as_posix().replace("'", "''") + en_sql = en_path.as_posix().replace("'", "''") + shared_object_numbers = _shared_object_numbers(connection, da_sql, en_sql) + colortemp_median, colortemp_mad, colortemp_count = _colortemp_parameters((da_path, en_path)) + + schema = _schema() + da_metadata = pq.ParquetFile(da_path).schema_arrow.metadata or {} + en_metadata = pq.ParquetFile(en_path).schema_arrow.metadata or {} + manifest_sha = da_metadata[b"smk.source_manifest_sha256"].decode() + if en_metadata[b"smk.source_manifest_sha256"].decode() != manifest_sha: + raise ValueError("DA/EN source Parquets reference different source manifests") + metadata = { + b"smk.canonical_schema_version": CANONICAL_VERSION.encode(), + b"smk.source_manifest_sha256": manifest_sha.encode(), + b"smk.corpus_rule": b"smk-public-visual-v1", + } + artwork_writer = pq.ParquetWriter(processed / "artworks.parquet", schema.with_metadata(metadata), compression="zstd") + corpus_writer = pq.ParquetWriter( + processed / "smk-public-visual-v1.parquet", schema.with_metadata(metadata), compression="zstd", + ) + membership_writer = pq.ParquetWriter( + reports / "corpus-membership.parquet", MEMBERSHIP_SCHEMA.with_metadata(metadata), compression="zstd", + ) + + query = f""" + SELECT + coalesce(da.smk_source_id, en.smk_source_id) AS smk_entity_id, + da.source_row_id AS da_source_row_id, + en.source_row_id AS en_source_row_id, + da.source_manifest_sha256 AS da_manifest_sha, + en.source_manifest_sha256 AS en_manifest_sha, + da.source_file_sha256 AS da_file_sha, + en.source_file_sha256 AS en_file_sha, + da.source_record_json AS da_record_json, + en.source_record_json AS en_record_json + FROM read_parquet('{da_sql}') da + FULL OUTER JOIN read_parquet('{en_sql}') en + ON da.smk_source_id = en.smk_source_id + ORDER BY smk_entity_id + """ + reader = connection.execute(query).to_arrow_reader(1_000) + counters: dict[str, Counter[str]] = { + "pairing": Counter(), "images": Counter(), "rights": Counter(), "quality": Counter(), + "exclusions": Counter(), "dimension_slots": Counter(), + } + object_groups: dict[str, list[str]] = defaultdict(list) + corpus_entities: set[str] = set() + total = 0 + included_count = 0 + corpus_missing_2d = 0 + corpus_visual_partial = 0 + corpus_colortemp_flags = 0 + previous_candidate_count = 0 + previous_candidate_corpus_overlap = 0 + corpus_missing_any_physical = 0 + corpus_missing_visual_fields: Counter[str] = Counter() + try: + for batch in reader: + artwork_rows: list[dict[str, Any]] = [] + corpus_rows: list[dict[str, Any]] = [] + membership_rows: list[dict[str, Any]] = [] + for pair in batch.to_pylist(): + row = _canonical_row(pair, shared_object_numbers, colortemp_median, colortemp_mad) + total += 1 + if row["present_in_da"] and row["present_in_en"]: + counters["pairing"]["both"] += 1 + elif row["present_in_da"]: + counters["pairing"]["da_only"] += 1 + else: + counters["pairing"]["en_only"] += 1 + if row["object_number_status"] == "conflict": + counters["pairing"]["object_number_conflict"] += 1 + counters["images"][row["image_bundle_status"]] += 1 + counters["rights"][row["rights_semantic"] or "CONFLICT"] += 1 + for flag in row["quality_flags"]: + counters["quality"][flag] += 1 + for reason in row["corpus_exclusion_reasons"]: + counters["exclusions"][reason] += 1 + for part in SUPPORTED_PARTS: + for axis in SUPPORTED_AXES: + if row[f"{part}_{axis}_cm"] is not None: + counters["dimension_slots"][f"{part}_{axis}_cm"] += 1 + if row["object_number"]: + object_groups[row["object_number"]].append(row["smk_entity_id"]) + if row["previous_public_visual_candidate"]: + previous_candidate_count += 1 + if row["corpus_smk_public_visual_v1"]: + included_count += 1 + corpus_entities.add(row["smk_entity_id"]) + corpus_rows.append(row) + if not row["has_2d_physical_geometry"]: + corpus_missing_2d += 1 + if not row["has_any_physical_dimension"]: + corpus_missing_any_physical += 1 + if "VISUAL_FIELDS_PARTIAL" in row["quality_flags"]: + corpus_visual_partial += 1 + for field in VISUAL_FIELDS: + if row[field] is None: + corpus_missing_visual_fields[field] += 1 + if any(flag.startswith("COLORTEMP_") for flag in row["quality_flags"]): + corpus_colortemp_flags += 1 + if row["previous_public_visual_candidate"]: + previous_candidate_corpus_overlap += 1 + artwork_rows.append(row) + membership_rows.append({ + "smk_entity_id": row["smk_entity_id"], + "object_number": row["object_number"], + "included": row["corpus_smk_public_visual_v1"], + "exclusion_reasons": row["corpus_exclusion_reasons"], + "quality_flags": row["quality_flags"], + }) + artwork_writer.write_table(pa.Table.from_pylist(artwork_rows, schema=schema)) + membership_writer.write_table(pa.Table.from_pylist(membership_rows, schema=MEMBERSHIP_SCHEMA)) + if corpus_rows: + corpus_writer.write_table(pa.Table.from_pylist(corpus_rows, schema=schema)) + finally: + artwork_writer.close() + corpus_writer.close() + membership_writer.close() + connection.close() + + group_rows: list[dict[str, Any]] = [] + corpus_shared_groups = 0 + corpus_shared_entities = 0 + for object_number, entity_ids in sorted(object_groups.items()): + sorted_ids = sorted(entity_ids) + corpus_members = sum(entity_id in corpus_entities for entity_id in sorted_ids) + if len(sorted_ids) > 1 and corpus_members: + corpus_shared_groups += 1 + corpus_shared_entities += corpus_members + group_rows.append({ + "object_number": object_number, + "canonical_entity_ids": sorted_ids, + "group_size": len(sorted_ids), + "group_classification": ( + "multiple_smk_entities_same_object_number" if len(sorted_ids) > 1 else "unique" + ), + "ambiguity_flag": len(sorted_ids) > 1, + "corpus_member_count": corpus_members, + }) + pq.write_table( + pa.Table.from_pylist(group_rows, schema=GROUP_SCHEMA), + processed / "object-number-groups.parquet", compression="zstd", + ) + summary = { + "canonical_schema_version": CANONICAL_VERSION, + "source_manifest_sha256": manifest_sha, + "canonical_entity_count": total, + "pairing": dict(counters["pairing"]), + "object_number_groups": { + "total": len(group_rows), + "shared_groups": len(shared_object_numbers), + "largest_group": max(len(ids) for ids in object_groups.values()), + "shared_groups_with_corpus_members": corpus_shared_groups, + "corpus_members_in_shared_groups": corpus_shared_entities, + }, + "dimensions": { + "entities_with_conflicts": counters["quality"]["DIMENSION_CONFLICT"], + "entities_with_net_width": counters["dimension_slots"]["net_width_cm"], + "entities_with_net_height": counters["dimension_slots"]["net_height_cm"], + "slot_success_counts": dict(counters["dimension_slots"]), + }, + "images": dict(counters["images"]), + "rights": dict(counters["rights"]), + "quality_flags": dict(counters["quality"]), + "colortemp_extreme_rule": { + "rule": "absolute robust z-score > 10 using median and MAD", + "entity_numeric_count": colortemp_count, + "median": colortemp_median, + "mad": colortemp_mad, + }, + "corpus": { + "name": "smk-public-visual-v1", + "included": included_count, + "excluded": total - included_count, + "exclusion_reason_counts": dict(counters["exclusions"]), + "previous_public_visual_candidate_count": previous_candidate_count, + "overlap_with_previous_public_visual_candidates": previous_candidate_corpus_overlap, + "included_without_any_normalized_physical_dimension": corpus_missing_any_physical, + "included_without_2d_net_geometry": corpus_missing_2d, + "included_with_partial_visual_fields": corpus_visual_partial, + "included_missing_visual_field_counts": dict(corpus_missing_visual_fields), + "included_with_colortemp_quality_flags": corpus_colortemp_flags, + }, + } + (reports / "milestone2-summary.json").write_text( + json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8", + ) + return summary + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, default=Path.cwd()) + args = parser.parse_args() + print(json.dumps(run(args.root.resolve()), indent=2)) + + +if __name__ == "__main__": + main() diff --git a/src/smk_playground/canonical/rules.py b/src/smk_playground/canonical/rules.py new file mode 100644 index 0000000..ca15383 --- /dev/null +++ b/src/smk_playground/canonical/rules.py @@ -0,0 +1,246 @@ +"""Pure, deterministic Milestone 2 canonicalization rules.""" + +from __future__ import annotations + +import math +import statistics +import unicodedata +from collections import defaultdict +from typing import Any + +from smk_playground.ingest.source import parse_strict_number + +PDM_URL = "https://creativecommons.org/publicdomain/mark/1.0/" +CC0_URL = "https://creativecommons.org/publicdomain/zero/1.0/" +SMK_POLICY_URLS = { + "https://www.smk.dk/section/brug-af-museets-materiale/", + "https://www.smk.dk/en/section/use-of-smk-material/", +} + +PART_ALIASES = { + "netto": "net", + "nettomål": "net", + "brutto": "gross", + "bruttomål": "gross", + "bladmål": "sheet", + "plademål": "plate", + "monteringsmål": "mount", + "billedmål": "image", + "papirmål": "paper", +} +TYPE_ALIASES = { + "bredde": "width", + "width": "width", + "højde": "height", + "height": "height", + "dybde": "depth", + "depth": "depth", + "diameter": "diameter", + "længde": "length", + "length": "length", +} +UNIT_TO_CM = { + "millimeter": 0.1, + "milimeter": 0.1, + "mm": 0.1, + "centimeter": 1.0, + "cm": 1.0, +} +SUPPORTED_PARTS = ("net", "gross", "sheet", "plate", "mount", "image", "paper") +SUPPORTED_AXES = ("width", "height", "depth", "diameter", "length") +IMAGE_KEYS = ( + "has_image", "image_iiif_id", "image_iiif_info", "image_width", "image_height", + "image_thumbnail", "image_native", "image_mime_type", "image_cropped", + "image_orientation", "image_hq", +) + + +def resolve_smk_entity_id(da_id: str | None, en_id: str | None) -> str: + """Pair languages only when their stable SMK source entity IDs agree.""" + if da_id and en_id and da_id != en_id: + raise ValueError(f"Cannot pair different SMK ids: {da_id!r} != {en_id!r}") + entity_id = da_id or en_id + if not entity_id: + raise ValueError("Canonical entity requires an SMK id") + return entity_id + + +def normalize_vocabulary_token(value: Any) -> str | None: + if not isinstance(value, str) or not value.strip(): + return None + token = unicodedata.normalize("NFC", value.strip()).casefold() + return token.replace("maal", "mål") + + +def rights_semantic(value: Any) -> str: + if value == PDM_URL: + return "PDM" + if value == CC0_URL: + return "CC0" + if value in SMK_POLICY_URLS: + return "SMK_RESTRICTED_POLICY" + return "UNKNOWN" + + +def resolve_scalar(da_value: Any, en_value: Any) -> tuple[Any, str, str]: + """Return canonical value, provenance, and status without global language precedence.""" + if da_value == en_value and da_value is not None: + return da_value, "both", "agreement" + if da_value is not None and en_value is None: + return da_value, "da", "single_source" + if en_value is not None and da_value is None: + return en_value, "en", "single_source" + if da_value is None and en_value is None: + return None, "none", "missing" + return None, "conflict", "conflict" + + +def image_bundle(record: dict[str, Any] | None) -> dict[str, Any] | None: + if record is None: + return None + return {key: record.get(key) for key in IMAGE_KEYS} + + +def image_bundle_complete(bundle: dict[str, Any] | None) -> bool: + if not bundle or bundle.get("has_image") is not True: + return False + width = bundle.get("image_width") + height = bundle.get("image_height") + return ( + isinstance(width, int) and not isinstance(width, bool) and width > 0 + and isinstance(height, int) and not isinstance(height, bool) and height > 0 + and isinstance(bundle.get("image_iiif_id"), str) and bool(bundle["image_iiif_id"]) + and bool(bundle.get("image_thumbnail") or bundle.get("image_native")) + ) + + +def resolve_image_bundle( + da_bundle: dict[str, Any] | None, en_bundle: dict[str, Any] | None, +) -> tuple[dict[str, Any] | None, str, str]: + da_complete = image_bundle_complete(da_bundle) + en_complete = image_bundle_complete(en_bundle) + if da_bundle == en_bundle and da_bundle is not None: + status = "IMAGE_BUNDLE_OK" if da_complete else "IMAGE_BUNDLE_INCOMPLETE" + return da_bundle, "both", status + if da_complete and not en_complete: + return da_bundle, "da", "IMAGE_BUNDLE_SINGLE_SOURCE" + if en_complete and not da_complete: + return en_bundle, "en", "IMAGE_BUNDLE_SINGLE_SOURCE" + if da_complete and en_complete: + return None, "conflict", "IMAGE_BUNDLE_CONFLICT" + if da_bundle is not None and en_bundle is None: + return da_bundle, "da", "IMAGE_BUNDLE_INCOMPLETE" + if en_bundle is not None and da_bundle is None: + return en_bundle, "en", "IMAGE_BUNDLE_INCOMPLETE" + return None, "none", "IMAGE_BUNDLE_INCOMPLETE" + + +def normalize_dimension_observations( + record: dict[str, Any] | None, language: str, source_row_id: str | None, +) -> list[dict[str, Any]]: + if record is None: + return [] + dimensions = record.get("dimensions") + if not isinstance(dimensions, list): + return [] + output: list[dict[str, Any]] = [] + for item in dimensions: + if not isinstance(item, dict): + continue + original_part = item.get("part") + original_type = item.get("type") + original_unit = item.get("unit") + original_value = item.get("value") + part = PART_ALIASES.get(normalize_vocabulary_token(original_part) or "") + axis = TYPE_ALIASES.get(normalize_vocabulary_token(original_type) or "") + unit_token = normalize_vocabulary_token(original_unit) + factor = UNIT_TO_CM.get(unit_token or "") + number, number_status = parse_strict_number(original_value) + if part and axis and factor is not None and number is not None: + value_cm = number * factor + status = "normalized" + elif axis is None and normalize_vocabulary_token(original_type) in { + "vægt", "weight", "spilletid", "playing time", "bruttoregisterton", + }: + value_cm = None + status = "non_geometric" + else: + value_cm = None + status = f"unmapped_or_{number_status}" + output.append({ + "part_canonical": part, + "type_canonical": axis, + "unit_canonical": "cm" if value_cm is not None else None, + "value_canonical": value_cm, + "original_language": language, + "original_part": None if original_part is None else str(original_part), + "original_type": None if original_type is None else str(original_type), + "original_unit": None if original_unit is None else str(original_unit), + "original_value": None if original_value is None else str(original_value), + "source_row_id": source_row_id, + "status": status, + }) + return output + + +def dimension_slots( + observations: list[dict[str, Any]], +) -> tuple[dict[str, float | None], list[str], list[dict[str, Any]]]: + candidates: dict[str, set[float]] = defaultdict(set) + candidate_rows: dict[str, list[dict[str, Any]]] = defaultdict(list) + for observation in observations: + part = observation["part_canonical"] + axis = observation["type_canonical"] + value = observation["value_canonical"] + if part in SUPPORTED_PARTS and axis in SUPPORTED_AXES and value is not None: + slot = f"{part}_{axis}_cm" + candidates[slot].add(round(float(value), 12)) + candidate_rows[slot].append(observation) + slots: dict[str, float | None] = {} + conflicts: list[str] = [] + for part in SUPPORTED_PARTS: + for axis in SUPPORTED_AXES: + slot = f"{part}_{axis}_cm" + values = candidates.get(slot, set()) + slots[slot] = next(iter(values)) if len(values) == 1 else None + if len(values) > 1: + conflicts.append(slot) + conflict_candidates = [row for slot in conflicts for row in candidate_rows[slot]] + return slots, conflicts, conflict_candidates + + +def visual_number(record: dict[str, Any] | None, field: str) -> tuple[float | None, str]: + if record is None: + return None, "missing" + return parse_strict_number(record.get(field)) + + +def robust_colortemp_parameters(values: list[float]) -> tuple[float, float]: + finite = [value for value in values if math.isfinite(value)] + median = statistics.median(finite) + mad = statistics.median(abs(value - median) for value in finite) + return median, mad + + +def is_extreme_colortemp(value: float | None, median: float, mad: float) -> bool: + if value is None or not math.isfinite(value) or mad == 0: + return False + return abs(value - median) / (1.4826 * mad) > 10.0 + + +def corpus_membership(row: dict[str, Any]) -> tuple[bool, list[str]]: + reasons: list[str] = [] + if "CANONICALIZATION_UNRESOLVED" in row["quality_flags"]: + reasons.append("CANONICALIZATION_UNRESOLVED") + if row["public_domain"] is not True: + reasons.append("NOT_PUBLIC_DOMAIN") + if row["rights_semantic"] not in {"PDM", "CC0"}: + reasons.append("RIGHTS_NOT_OPEN") + if row["image_bundle_status"] == "IMAGE_BUNDLE_CONFLICT": + reasons.append("IMAGE_CONFLICT") + if not row["image_bundle_complete"]: + reasons.append("NO_USABLE_IMAGE") + if not row["colors"]: + reasons.append("NO_COLORS") + return not reasons, reasons + diff --git a/src/smk_playground/cli.py b/src/smk_playground/cli.py index 06175d8..6f894fd 100644 --- a/src/smk_playground/cli.py +++ b/src/smk_playground/cli.py @@ -28,5 +28,14 @@ def milestone1(root: Path | None = None) -> None: typer.echo(f"Milestone 1 complete: {summary['source_rows']}") +@app.command("milestone2") +def milestone2(root: Path | None = None) -> None: + """Build canonical SMK entities and freeze smk-public-visual-v1.""" + from smk_playground.canonical.milestone2 import run + + summary = run((root or Path.cwd()).resolve()) + typer.echo(f"Milestone 2 complete: {summary['canonical_entity_count']} entities") + + if __name__ == "__main__": app() diff --git a/tests/test_canonical_rules.py b/tests/test_canonical_rules.py new file mode 100644 index 0000000..8cb71df --- /dev/null +++ b/tests/test_canonical_rules.py @@ -0,0 +1,90 @@ +from smk_playground.canonical.rules import ( + corpus_membership, + dimension_slots, + image_bundle_complete, + normalize_dimension_observations, + resolve_image_bundle, + resolve_smk_entity_id, + rights_semantic, + visual_number, +) + + +def complete_bundle(image_id: str = "image") -> dict: + return { + "has_image": True, + "image_iiif_id": image_id, + "image_iiif_info": f"{image_id}/info.json", + "image_width": 100, + "image_height": 200, + "image_thumbnail": "thumb", + "image_native": "native", + "image_mime_type": "image/tiff", + "image_cropped": False, + "image_orientation": "portrait", + "image_hq": True, + } + + +def test_da_en_pairing_uses_smk_id_only() -> None: + assert resolve_smk_entity_id("117_object", "117_object") == "117_object" + assert resolve_smk_entity_id(None, "118_object") == "118_object" + try: + resolve_smk_entity_id("117_object", "118_object") + except ValueError: + pass + else: + raise AssertionError("different SMK ids must not be paired") + + +def test_image_bundle_is_atomic() -> None: + da = complete_bundle("da") + en = complete_bundle("en") + canonical, provenance, status = resolve_image_bundle(da, en) + assert canonical is None + assert provenance == "conflict" + assert status == "IMAGE_BUNDLE_CONFLICT" + + canonical, provenance, status = resolve_image_bundle(da, {"has_image": False}) + assert canonical == da + assert provenance == "da" + assert status == "IMAGE_BUNDLE_SINGLE_SOURCE" + assert image_bundle_complete(canonical) + + +def test_dimension_aliases_convert_to_cm_and_conflicts_stay_null() -> None: + da = {"dimensions": [{"part": "Netto", "type": "højde", "unit": "millimeter", "value": "100"}]} + en = {"dimensions": [{"part": "netto", "type": "height", "unit": "cm", "value": "11"}]} + observations = normalize_dimension_observations(da, "da", "da-id") + observations += normalize_dimension_observations(en, "en", "en-id") + slots, conflicts, candidates = dimension_slots(observations) + assert slots["net_height_cm"] is None + assert conflicts == ["net_height_cm"] + assert len(candidates) == 2 + + +def test_rights_semantics_are_language_independent() -> None: + assert rights_semantic("https://creativecommons.org/publicdomain/mark/1.0/") == "PDM" + assert rights_semantic("https://www.smk.dk/section/brug-af-museets-materiale/") == "SMK_RESTRICTED_POLICY" + assert rights_semantic("https://www.smk.dk/en/section/use-of-smk-material/") == "SMK_RESTRICTED_POLICY" + + +def test_infinity_colortemp_is_missing_with_explicit_status() -> None: + assert visual_number({"colortemp": "Infinity"}, "colortemp") == (None, "malformed_text") + + +def test_corpus_rule_returns_machine_readable_reasons() -> None: + row = { + "quality_flags": [], "public_domain": True, "rights_semantic": "PDM", + "image_bundle_status": "IMAGE_BUNDLE_OK", "image_bundle_complete": True, + "colors": ["#ffffff"], + } + assert corpus_membership(row) == (True, []) + row["colors"] = None + row["image_bundle_complete"] = False + assert corpus_membership(row) == (False, ["NO_USABLE_IMAGE", "NO_COLORS"]) + + +def test_rules_are_deterministic() -> None: + bundle = complete_bundle() + assert resolve_image_bundle(bundle, dict(bundle)) == resolve_image_bundle(bundle, dict(bundle))