Complete Milestone 2 canonical corpus
This commit is contained in:
parent
1114ccd8a7
commit
57ae7b780b
11 changed files with 1167 additions and 21 deletions
|
|
@ -167,6 +167,10 @@ Parquet is the portable source of truth for derived tabular output. DuckDB may p
|
||||||
- Use `smk:<language>:<id>` as source-row identity for the audited snapshots. Never use array
|
- Use `smk:<language>:<id>` as source-row identity for the audited snapshots. Never use array
|
||||||
position. Treat duplicate source-row IDs as errors and repeated `object_number` values as
|
position. Treat duplicate source-row IDs as errors and repeated `object_number` values as
|
||||||
unresolved groups until an explicit canonical rule handles them.
|
unresolved groups until an explicit canonical rule handles them.
|
||||||
|
- Use the top-level SMK `id` without language as `smk_entity_id`. Pair DA/EN only when that ID is
|
||||||
|
identical. `object_number` is explicitly non-unique and group members remain separate entities.
|
||||||
|
- Resolve image metadata as an atomic bundle; never combine image identifiers from one source with
|
||||||
|
pixel dimensions or URLs from the other.
|
||||||
|
|
||||||
## Canonical artwork rules
|
## Canonical artwork rules
|
||||||
|
|
||||||
|
|
|
||||||
10
README.md
10
README.md
|
|
@ -121,10 +121,9 @@ See:
|
||||||
|
|
||||||
## Current status
|
## Current status
|
||||||
|
|
||||||
Milestones 0–1 complete: the real source structure is manifested, both snapshots have
|
Milestones 0–2 complete: source structure and audits are manifested, the language snapshots have
|
||||||
bounded-memory source Parquets, and duplicate identity, dimensions, palette/visual quality,
|
bounded-memory source Parquets, DA/EN rows are paired into stable SMK entities, and the exact
|
||||||
rights and DA/EN disagreements are audited. Canonical artwork generation is next; visual feature
|
`smk-public-visual-v1` membership rule is frozen. Visual feature extraction has not started.
|
||||||
extraction has not started.
|
|
||||||
|
|
||||||
Audit results are documented in:
|
Audit results are documented in:
|
||||||
|
|
||||||
|
|
@ -133,9 +132,12 @@ Audit results are documented in:
|
||||||
- [`docs/SOURCE_DISAGREEMENTS.md`](docs/SOURCE_DISAGREEMENTS.md)
|
- [`docs/SOURCE_DISAGREEMENTS.md`](docs/SOURCE_DISAGREEMENTS.md)
|
||||||
- [`docs/PALETTE_VISUAL_AUDIT.md`](docs/PALETTE_VISUAL_AUDIT.md)
|
- [`docs/PALETTE_VISUAL_AUDIT.md`](docs/PALETTE_VISUAL_AUDIT.md)
|
||||||
- [`docs/RIGHTS_AUDIT.md`](docs/RIGHTS_AUDIT.md)
|
- [`docs/RIGHTS_AUDIT.md`](docs/RIGHTS_AUDIT.md)
|
||||||
|
- [`docs/CANONICALIZATION.md`](docs/CANONICALIZATION.md)
|
||||||
|
- [`docs/CORPUS_V1.md`](docs/CORPUS_V1.md)
|
||||||
|
|
||||||
Run the audited ingest with:
|
Run the audited ingest with:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
smk milestone1
|
smk milestone1
|
||||||
|
smk milestone2
|
||||||
```
|
```
|
||||||
|
|
|
||||||
89
docs/CANONICALIZATION.md
Normal file
89
docs/CANONICALIZATION.md
Normal file
|
|
@ -0,0 +1,89 @@
|
||||||
|
# Canonicalization v1
|
||||||
|
|
||||||
|
Milestone 2 builds `data/processed/artworks.parquet` from the two manifest-bound source Parquets.
|
||||||
|
It produces one row per SMK entity and does not calculate OKLab, palette weights or colour-derived
|
||||||
|
features.
|
||||||
|
|
||||||
|
## Identity levels
|
||||||
|
|
||||||
|
1. **Source row** — `source_row_id = smk:<language>:<SMK id>`.
|
||||||
|
2. **SMK entity** — `smk_entity_id = <SMK id>`, without language.
|
||||||
|
3. **Artwork-facing identifier** — `object_number`, retained as a non-unique accession/business
|
||||||
|
identifier.
|
||||||
|
|
||||||
|
DA and EN rows are paired only when their top-level SMK `id` is identical. Different SMK IDs are
|
||||||
|
never paired merely because `object_number` matches. The audited result has 203,023 entities:
|
||||||
|
202,728 paired DA+EN entities and 295 EN-only entities. No entity is DA-only.
|
||||||
|
|
||||||
|
One paired entity, `1170314109_object`, has conflicting object numbers: DA `KKSgb81877` and EN
|
||||||
|
`KKSgb82299`. Its canonical `object_number` is null and it is flagged
|
||||||
|
`CANONICALIZATION_UNRESOLVED`.
|
||||||
|
|
||||||
|
## Field rules
|
||||||
|
|
||||||
|
For source-independent scalar/list fields:
|
||||||
|
|
||||||
|
- exact DA/EN agreement → canonical value with provenance `both`;
|
||||||
|
- value in only one language → canonical value with provenance `da` or `en`;
|
||||||
|
- genuine disagreement → both originals retained, canonical value null, status `conflict` unless
|
||||||
|
an audited field rule exists.
|
||||||
|
|
||||||
|
Localized titles, object names/classifications, artists, production data, techniques and materials
|
||||||
|
remain in separate DA/EN columns. Source row IDs, source-file hashes and manifest hashes are
|
||||||
|
retained on every entity.
|
||||||
|
|
||||||
|
## Image bundle
|
||||||
|
|
||||||
|
Image fields are resolved atomically. A complete bundle requires `has_image == true`, positive
|
||||||
|
integer width and height, a non-empty IIIF image ID, and at least one thumbnail or native/public
|
||||||
|
image URL.
|
||||||
|
|
||||||
|
Resolution order is exact matching complete bundle, one complete source against an absent or
|
||||||
|
incomplete source, then unresolved conflict if both sources have different complete bundles. No
|
||||||
|
canonical bundle mixes fields from DA and EN.
|
||||||
|
|
||||||
|
| Status | Entities |
|
||||||
|
|---|---:|
|
||||||
|
| `IMAGE_BUNDLE_OK` | 39,956 |
|
||||||
|
| `IMAGE_BUNDLE_SINGLE_SOURCE` | 8 |
|
||||||
|
| `IMAGE_BUNDLE_CONFLICT` | 3 |
|
||||||
|
| `IMAGE_BUNDLE_INCOMPLETE` | 163,056 |
|
||||||
|
|
||||||
|
The three conflicts are `KMSsp451`, `KKSbog14617c` and `KKS1984-179`. Both source bundles remain
|
||||||
|
available for inspection.
|
||||||
|
|
||||||
|
## Dimensions
|
||||||
|
|
||||||
|
Every source observation is preserved as a nested typed struct containing canonical part/type/unit,
|
||||||
|
centimetre value where supported, original language/value fields, source-row ID and mapping status.
|
||||||
|
Non-geometric weight, duration and register-tonnage observations are preserved but not converted.
|
||||||
|
|
||||||
|
Candidate slots exist for width, height, depth, diameter and length under `net`, `gross`, `sheet`,
|
||||||
|
`plate`, `mount`, `image` and `paper`. Repeated equal observations agree. Multiple distinct values
|
||||||
|
for one slot make that slot null and retain all candidates.
|
||||||
|
|
||||||
|
- 197,517 entities have at least one normalized physical dimension;
|
||||||
|
- 9,902 have canonical Netto width and 10,331 have Netto height;
|
||||||
|
- 9,897 have a complete conflict-free Netto width × height representation;
|
||||||
|
- 1,798 additionally have Netto depth;
|
||||||
|
- 227 entities have at least one dimension-slot conflict.
|
||||||
|
|
||||||
|
`has_2d_physical_geometry` means conflict-free Netto width and height. It does not claim every
|
||||||
|
entity is rectangular or a 2D artwork. `has_3d_physical_geometry` additionally requires Netto depth.
|
||||||
|
|
||||||
|
## Rights and visual data
|
||||||
|
|
||||||
|
Known rights URLs map to `PDM`, `CC0` or `SMK_RESTRICTED_POLICY`; originals remain in `rights_da`
|
||||||
|
and `rights_en`. Counts are 151,913 PDM, 598 CC0 and 50,512 restricted-policy entities.
|
||||||
|
`public_domain` remains a separately resolved explicit source field.
|
||||||
|
|
||||||
|
The 12 source `"Infinity"` colortemp strings become canonical null with
|
||||||
|
`COLORTEMP_NONFINITE_SOURCE`. Finite values are preserved without clipping. `COLORTEMP_EXTREME`
|
||||||
|
uses absolute robust z-score greater than 10, calculated with median 3.4118087 and MAD 0.1786611
|
||||||
|
over 52,237 entity values; 481 entities are flagged. This is a statistical diagnostic, not a
|
||||||
|
scientific validity claim.
|
||||||
|
|
||||||
|
Canonical rows contain deterministic sorted quality flags, including language missingness, shared
|
||||||
|
object number, dimension conflict, image status, colortemp diagnostics, partial visual fields and
|
||||||
|
unresolved canonicalization.
|
||||||
|
|
||||||
70
docs/CORPUS_V1.md
Normal file
70
docs/CORPUS_V1.md
Normal file
|
|
@ -0,0 +1,70 @@
|
||||||
|
# Corpus: `smk-public-visual-v1`
|
||||||
|
|
||||||
|
Milestone 2 freezes the first corpus for openly reusable visual/colour analysis. Membership is
|
||||||
|
stored on every canonical entity and materialized in
|
||||||
|
`data/processed/smk-public-visual-v1.parquet`.
|
||||||
|
|
||||||
|
## Exact inclusion predicate
|
||||||
|
|
||||||
|
An entity is included only when all conditions hold:
|
||||||
|
|
||||||
|
1. canonicalization is resolved;
|
||||||
|
2. explicit canonical `public_domain == true`;
|
||||||
|
3. canonical rights semantic is `PDM` or `CC0`;
|
||||||
|
4. the atomic canonical image bundle is complete;
|
||||||
|
5. image width and height are positive (part of bundle completeness);
|
||||||
|
6. canonical `colors` is a non-empty list;
|
||||||
|
7. there is no unresolved image-bundle conflict.
|
||||||
|
|
||||||
|
The rule does **not** require physical dimensions, English metadata, visual scalar fields,
|
||||||
|
weighted palettes or enrichment data.
|
||||||
|
|
||||||
|
## Membership
|
||||||
|
|
||||||
|
| Measure | Count |
|
||||||
|
|---|---:|
|
||||||
|
| Canonical entities | 203,023 |
|
||||||
|
| Included | 27,254 |
|
||||||
|
| Excluded | 175,769 |
|
||||||
|
|
||||||
|
Exclusion reasons are non-exclusive:
|
||||||
|
|
||||||
|
| Code | Count |
|
||||||
|
|---|---:|
|
||||||
|
| `NO_USABLE_IMAGE` | 163,059 |
|
||||||
|
| `NO_COLORS` | 150,774 |
|
||||||
|
| `NOT_PUBLIC_DOMAIN` | 50,512 |
|
||||||
|
| `RIGHTS_NOT_OPEN` | 50,512 |
|
||||||
|
| `IMAGE_CONFLICT` | 3 |
|
||||||
|
| `CANONICALIZATION_UNRESOLVED` | 1 |
|
||||||
|
|
||||||
|
Milestone 1 observed about 39.5k public-domain rows with an image/public URL. Recomputed at entity
|
||||||
|
level, that broad candidate definition contains 39,508 entities. All 27,254 corpus members overlap
|
||||||
|
it. The stricter corpus is smaller because it additionally requires a complete atomic image bundle
|
||||||
|
and non-empty `colors`; it is not forced to an expected count.
|
||||||
|
|
||||||
|
## Included-data diagnostics
|
||||||
|
|
||||||
|
- 321 included entities have no normalized physical dimension;
|
||||||
|
- 25,065 lack a complete Netto width × height pair;
|
||||||
|
- 871 have partial visual fields;
|
||||||
|
- missing visual values: saturation 871, brightness 62, contrast 62, entropy 1, colortemp 1;
|
||||||
|
- 143 included entities have a colortemp quality flag.
|
||||||
|
|
||||||
|
Physical and visual missingness does not exclude a work because those properties are independent
|
||||||
|
of whether its primary image and source colours are usable.
|
||||||
|
|
||||||
|
## Shared object numbers
|
||||||
|
|
||||||
|
There are 306 `object_number` groups containing more than one canonical SMK entity; the largest
|
||||||
|
contains 24 entities. Nineteen such groups contain corpus members, representing 35 included
|
||||||
|
entities. Members remain separate entities. `data/processed/object-number-groups.parquet` provides
|
||||||
|
the relationship for later analysis/UI without destructive merging.
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
- `data/processed/artworks.parquet` — every canonical SMK entity;
|
||||||
|
- `data/processed/smk-public-visual-v1.parquet` — included entities;
|
||||||
|
- `data/reports/corpus-membership.parquet` — decision and reason codes for every entity;
|
||||||
|
- `data/processed/object-number-groups.parquet` — non-destructive accession-number grouping;
|
||||||
|
- `data/reports/milestone2-summary.json` — machine-readable counts and rule parameters.
|
||||||
|
|
@ -20,7 +20,9 @@ Values calculated relative to a named, frozen corpus/profile.
|
||||||
Identifies exact input snapshots by filename, size, SHA-256, observed JSON structure and record counts.
|
Identifies exact input snapshots by filename, size, SHA-256, observed JSON structure and record counts.
|
||||||
|
|
||||||
### `artworks.parquet`
|
### `artworks.parquet`
|
||||||
One canonical row per artwork identity.
|
One canonical row per SMK entity, produced by pairing DA/EN source rows with the same top-level
|
||||||
|
SMK `id`. It includes localized values, field-level provenance/conflict status, atomic image
|
||||||
|
resolution, typed dimension observations/candidates, quality flags and corpus membership.
|
||||||
|
|
||||||
Suggested fields include:
|
Suggested fields include:
|
||||||
|
|
||||||
|
|
@ -76,11 +78,23 @@ content and report the fallback. Array index is never permanent identity. A dupl
|
||||||
`source_row_id` is a hard ingest error.
|
`source_row_id` is a hard ingest error.
|
||||||
|
|
||||||
Keep `smk_source_id`, `object_number`, language, source-file hash and logical record hash alongside
|
Keep `smk_source_id`, `object_number`, language, source-file hash and logical record hash alongside
|
||||||
the internal ID. Canonical artwork identity is deliberately deferred until duplicate groups have
|
the internal ID. The canonical entity key is the language-independent top-level SMK `id`.
|
||||||
an explicit resolution; it must not silently equate every repeated `object_number` row.
|
Repeated `object_number` values remain separate canonical entities and are represented in
|
||||||
|
`object-number-groups.parquet`; they are never collapsed implicitly.
|
||||||
|
|
||||||
If duplicate IDs exist, inspect them and implement an explicit deterministic policy with provenance; do not keep an arbitrary “first row”.
|
If duplicate IDs exist, inspect them and implement an explicit deterministic policy with provenance; do not keep an arbitrary “first row”.
|
||||||
|
|
||||||
|
### Canonical identity
|
||||||
|
|
||||||
|
```text
|
||||||
|
source_row_id = smk:<language>:<SMK id>
|
||||||
|
smk_entity_id = <SMK id>
|
||||||
|
object_number = non-unique accession/business identifier
|
||||||
|
```
|
||||||
|
|
||||||
|
DA and EN rows pair only by identical `smk_entity_id`. A disagreement in `object_number` becomes
|
||||||
|
an unresolved field conflict, not a cross-ID pairing rule.
|
||||||
|
|
||||||
## Manifest-bound source Parquet
|
## Manifest-bound source Parquet
|
||||||
|
|
||||||
`source_da.parquet` and `source_en.parquet` contain one row per source row, including selected typed
|
`source_da.parquet` and `source_en.parquet` contain one row per source row, including selected typed
|
||||||
|
|
@ -99,3 +113,7 @@ Zero is a real value and must never stand in for missing data.
|
||||||
Canonical ingestion and feature extraction may include records that are not eligible for `smk-public-visual-v1`.
|
Canonical ingestion and feature extraction may include records that are not eligible for `smk-public-visual-v1`.
|
||||||
|
|
||||||
Normalization membership is a separate, explicit decision. Store inclusion/exclusion reasons rather than deleting records merely because a downstream profile cannot use them.
|
Normalization membership is a separate, explicit decision. Store inclusion/exclusion reasons rather than deleting records merely because a downstream profile cannot use them.
|
||||||
|
|
||||||
|
`smk-public-visual-v1` is now frozen as an analyzable source corpus, not a normalization profile.
|
||||||
|
Its exact predicate and counts are documented in `CORPUS_V1.md`. Later corpus-relative feature
|
||||||
|
normalization remains independently versioned.
|
||||||
|
|
|
||||||
|
|
@ -28,10 +28,22 @@ Done when a representative fixture/sample becomes deterministic canonical rows.
|
||||||
|
|
||||||
Status: source loading and pre-canonical audits complete. Stable source-row identity, lossless
|
Status: source loading and pre-canonical audits complete. Stable source-row identity, lossless
|
||||||
logical Parquet, duplicate taxonomy, dimension vocabulary, palette/visual quality and DA/EN
|
logical Parquet, duplicate taxonomy, dimension vocabulary, palette/visual quality and DA/EN
|
||||||
precedence recommendations are documented. Canonical artwork generation remains blocked on an
|
precedence recommendations are documented. Canonical entities are implemented in Milestone 2
|
||||||
explicit duplicate-group resolution policy and is intentionally not performed in this milestone.
|
without collapsing duplicate `object_number` groups.
|
||||||
|
|
||||||
## Milestone 2 — Weighted palette + colour conversion
|
## Milestone 2 — Canonical entities + first analyzable corpus
|
||||||
|
|
||||||
|
- pair language rows only by SMK `id`
|
||||||
|
- preserve `object_number` as non-unique grouping metadata
|
||||||
|
- canonicalize fields with explicit provenance and conflicts
|
||||||
|
- resolve image bundles atomically
|
||||||
|
- preserve and normalize dimension observations without choosing conflicts
|
||||||
|
- freeze `smk-public-visual-v1` with machine-readable exclusion reasons
|
||||||
|
|
||||||
|
Status: complete. See `CANONICALIZATION.md` and `CORPUS_V1.md`. Visual/colour feature extraction
|
||||||
|
has not started.
|
||||||
|
|
||||||
|
## Milestone 3 — Weighted palette + colour conversion
|
||||||
|
|
||||||
- locate ordinary/weighted palette fields in the actual snapshot/enrichment data
|
- locate ordinary/weighted palette fields in the actual snapshot/enrichment data
|
||||||
- parse `colors_palette_weighted_s` when present
|
- parse `colors_palette_weighted_s` when present
|
||||||
|
|
@ -41,7 +53,7 @@ explicit duplicate-group resolution policy and is intentionally not performed in
|
||||||
|
|
||||||
Done when known palette fixtures produce stable colour rows and valid weights.
|
Done when known palette fixtures produce stable colour rows and valid weights.
|
||||||
|
|
||||||
## Milestone 3 — Feature extraction
|
## Milestone 4 — Feature extraction
|
||||||
|
|
||||||
Implement:
|
Implement:
|
||||||
|
|
||||||
|
|
@ -56,7 +68,7 @@ Implement:
|
||||||
|
|
||||||
Done when one artwork can produce a complete feature record independently of the corpus.
|
Done when one artwork can produce a complete feature record independently of the corpus.
|
||||||
|
|
||||||
## Milestone 4 — Quality layer
|
## Milestone 5 — Quality layer
|
||||||
|
|
||||||
Implement documented flags and source/language/aspect checks.
|
Implement documented flags and source/language/aspect checks.
|
||||||
|
|
||||||
|
|
@ -73,18 +85,18 @@ Produce `quality-report.json` with:
|
||||||
- palette-size distribution
|
- palette-size distribution
|
||||||
- obvious extreme values
|
- obvious extreme values
|
||||||
|
|
||||||
## Milestone 5 — Pilot build
|
## Milestone 6 — Pilot build
|
||||||
|
|
||||||
Run 100–500 representative artworks.
|
Run 100–500 representative artworks.
|
||||||
|
|
||||||
Review distributions and manually inspect selected records. Fix parser/source/math errors only; do not artistically tune features.
|
Review distributions and manually inspect selected records. Fix parser/source/math errors only; do not artistically tune features.
|
||||||
|
|
||||||
## Milestone 6 — Corpus definition + normalization
|
## Milestone 7 — Corpus-relative normalization
|
||||||
|
|
||||||
Do not hardcode 39,508.
|
Use the frozen `smk-public-visual-v1` entity membership from Milestone 2. Do not redefine corpus
|
||||||
|
membership from an expected count.
|
||||||
|
|
||||||
- define the exact inclusion predicate for `smk-public-visual-v1`
|
- reference included/excluded counts and reasons from the frozen corpus artifact
|
||||||
- record included/excluded counts and reasons
|
|
||||||
- compute count/null/min/max/quantiles
|
- compute count/null/min/max/quantiles
|
||||||
- median + MAD
|
- median + MAD
|
||||||
- implement `robust_01`
|
- implement `robust_01`
|
||||||
|
|
@ -92,7 +104,7 @@ Do not hardcode 39,508.
|
||||||
- implement robust z-score
|
- implement robust z-score
|
||||||
- freeze profile with source-manifest reference
|
- freeze profile with source-manifest reference
|
||||||
|
|
||||||
## Milestone 7 — Full dataset build
|
## Milestone 8 — Full dataset build
|
||||||
|
|
||||||
Run the complete local source snapshots.
|
Run the complete local source snapshots.
|
||||||
|
|
||||||
|
|
@ -100,7 +112,7 @@ Definition of done:
|
||||||
|
|
||||||
Every canonical artwork has valid applicable features or explicit nulls/flags explaining missing/invalid prerequisites. Every normalization result references a frozen profile and exact source manifest.
|
Every canonical artwork has valid applicable features or explicit nulls/flags explaining missing/invalid prerequisites. Every normalization result references a frozen profile and exact source manifest.
|
||||||
|
|
||||||
## Milestone 8 — Similarity
|
## Milestone 9 — Similarity
|
||||||
|
|
||||||
Implement separate profiles rather than one magic similarity score:
|
Implement separate profiles rather than one magic similarity score:
|
||||||
|
|
||||||
|
|
@ -110,7 +122,7 @@ Implement separate profiles rather than one magic similarity score:
|
||||||
|
|
||||||
Use fast feature-vector candidate retrieval followed optionally by weighted-palette optimal-transport reranking. Do not precompute all pairs.
|
Use fast feature-vector candidate retrieval followed optionally by weighted-palette optimal-transport reranking. Do not precompute all pairs.
|
||||||
|
|
||||||
## Milestone 9 — Art Tone v1
|
## Milestone 10 — Art Tone v1
|
||||||
|
|
||||||
Implement versioned, explainable sonification mapping separate from objective features.
|
Implement versioned, explainable sonification mapping separate from objective features.
|
||||||
|
|
||||||
|
|
@ -126,7 +138,7 @@ Initial mapping:
|
||||||
|
|
||||||
Emit exact inputs alongside every mapped result.
|
Emit exact inputs alongside every mapped result.
|
||||||
|
|
||||||
## Milestone 10 — Exploration
|
## Milestone 11 — Exploration
|
||||||
|
|
||||||
Only after Dataset v1 is trustworthy, start experimental interfaces/notebooks/apps:
|
Only after Dataset v1 is trustworthy, start experimental interfaces/notebooks/apps:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -63,6 +63,14 @@ Milestone 1 streams `$[]` with `ijson`, rejects source-ID collisions, preserves
|
||||||
manifest-bound Parquet and produces duplicate, dimension and language-disagreement audits. It does
|
manifest-bound Parquet and produces duplicate, dimension and language-disagreement audits. It does
|
||||||
not yet collapse duplicate `object_number` groups or produce canonical artwork rows.
|
not yet collapse duplicate `object_number` groups or produce canonical artwork rows.
|
||||||
|
|
||||||
|
## Stage 2 — canonical entities + corpus freeze
|
||||||
|
|
||||||
|
Pair DA/EN by identical SMK `id`, preserve non-unique object-number groups, apply field-specific
|
||||||
|
provenance/conflict rules, resolve image bundles atomically, normalize supported dimension
|
||||||
|
observations to centimetres and write `artworks.parquet`. Freeze `smk-public-visual-v1` from
|
||||||
|
explicit rights/image/colors criteria and emit membership reasons. Colour feature extraction is a
|
||||||
|
later stage.
|
||||||
|
|
||||||
## Ingest performance rule
|
## Ingest performance rule
|
||||||
|
|
||||||
The source files are large. Use bounded-memory streaming where practical. A one-time conversion to Parquet/Arrow as an interim representation is encouraged if it simplifies all later stages.
|
The source files are large. Use bounded-memory streaming where practical. A one-time conversion to Parquet/Arrow as an interim representation is encouraged if it simplifies all later stages.
|
||||||
|
|
|
||||||
598
src/smk_playground/canonical/milestone2.py
Normal file
598
src/smk_playground/canonical/milestone2.py
Normal file
|
|
@ -0,0 +1,598 @@
|
||||||
|
"""Build the Milestone 2 canonical entity layer and frozen visual corpus."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import duckdb
|
||||||
|
import pyarrow as pa
|
||||||
|
import pyarrow.parquet as pq
|
||||||
|
|
||||||
|
from smk_playground.canonical.rules import (
|
||||||
|
IMAGE_KEYS,
|
||||||
|
SUPPORTED_AXES,
|
||||||
|
SUPPORTED_PARTS,
|
||||||
|
corpus_membership,
|
||||||
|
dimension_slots,
|
||||||
|
image_bundle,
|
||||||
|
image_bundle_complete,
|
||||||
|
is_extreme_colortemp,
|
||||||
|
normalize_dimension_observations,
|
||||||
|
resolve_image_bundle,
|
||||||
|
resolve_scalar,
|
||||||
|
resolve_smk_entity_id,
|
||||||
|
rights_semantic,
|
||||||
|
robust_colortemp_parameters,
|
||||||
|
visual_number,
|
||||||
|
)
|
||||||
|
from smk_playground.ingest.source import logical_json
|
||||||
|
|
||||||
|
CANONICAL_VERSION = "1.0.0"
|
||||||
|
VISUAL_FIELDS = ("brightness", "contrast", "saturation", "entropy", "colortemp")
|
||||||
|
DIMENSION_STRUCT = pa.struct([
|
||||||
|
("part_canonical", pa.string()),
|
||||||
|
("type_canonical", pa.string()),
|
||||||
|
("unit_canonical", pa.string()),
|
||||||
|
("value_canonical", pa.float64()),
|
||||||
|
("original_language", pa.string()),
|
||||||
|
("original_part", pa.string()),
|
||||||
|
("original_type", pa.string()),
|
||||||
|
("original_unit", pa.string()),
|
||||||
|
("original_value", pa.string()),
|
||||||
|
("source_row_id", pa.string()),
|
||||||
|
("status", pa.string()),
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def _schema() -> pa.Schema:
|
||||||
|
fields: list[tuple[str, pa.DataType]] = [
|
||||||
|
("canonical_schema_version", pa.string()),
|
||||||
|
("smk_entity_id", pa.string()),
|
||||||
|
("object_number", pa.string()),
|
||||||
|
("object_number_da", pa.string()),
|
||||||
|
("object_number_en", pa.string()),
|
||||||
|
("object_number_provenance", pa.string()),
|
||||||
|
("object_number_status", pa.string()),
|
||||||
|
("source_row_id_da", pa.string()),
|
||||||
|
("source_row_id_en", pa.string()),
|
||||||
|
("present_in_da", pa.bool_()),
|
||||||
|
("present_in_en", pa.bool_()),
|
||||||
|
("source_manifest_sha256_da", pa.string()),
|
||||||
|
("source_manifest_sha256_en", pa.string()),
|
||||||
|
("source_file_sha256_da", pa.string()),
|
||||||
|
("source_file_sha256_en", pa.string()),
|
||||||
|
("titles_da_json", pa.string()),
|
||||||
|
("titles_en_json", pa.string()),
|
||||||
|
("object_names_da_json", pa.string()),
|
||||||
|
("object_names_en_json", pa.string()),
|
||||||
|
("classifications_da_json", pa.string()),
|
||||||
|
("classifications_en_json", pa.string()),
|
||||||
|
("artists_da_json", pa.string()),
|
||||||
|
("artists_en_json", pa.string()),
|
||||||
|
("production_da_json", pa.string()),
|
||||||
|
("production_en_json", pa.string()),
|
||||||
|
("production_dates_da_json", pa.string()),
|
||||||
|
("production_dates_en_json", pa.string()),
|
||||||
|
("techniques_da_json", pa.string()),
|
||||||
|
("techniques_en_json", pa.string()),
|
||||||
|
("materials_da_json", pa.string()),
|
||||||
|
("materials_en_json", pa.string()),
|
||||||
|
("public_domain", pa.bool_()),
|
||||||
|
("public_domain_da", pa.bool_()),
|
||||||
|
("public_domain_en", pa.bool_()),
|
||||||
|
("public_domain_provenance", pa.string()),
|
||||||
|
("public_domain_status", pa.string()),
|
||||||
|
("rights_da", pa.string()),
|
||||||
|
("rights_en", pa.string()),
|
||||||
|
("rights_semantic", pa.string()),
|
||||||
|
("rights_semantic_da", pa.string()),
|
||||||
|
("rights_semantic_en", pa.string()),
|
||||||
|
("rights_provenance", pa.string()),
|
||||||
|
("rights_status", pa.string()),
|
||||||
|
("colors", pa.list_(pa.string())),
|
||||||
|
("colors_da", pa.list_(pa.string())),
|
||||||
|
("colors_en", pa.list_(pa.string())),
|
||||||
|
("colors_provenance", pa.string()),
|
||||||
|
("colors_status", pa.string()),
|
||||||
|
]
|
||||||
|
for field in VISUAL_FIELDS:
|
||||||
|
fields.extend([
|
||||||
|
(field, pa.float64()),
|
||||||
|
(f"{field}_da_json", pa.string()),
|
||||||
|
(f"{field}_en_json", pa.string()),
|
||||||
|
(f"{field}_provenance", pa.string()),
|
||||||
|
(f"{field}_status", pa.string()),
|
||||||
|
])
|
||||||
|
fields.extend([
|
||||||
|
("image_bundle_status", pa.string()),
|
||||||
|
("image_bundle_provenance", pa.string()),
|
||||||
|
("image_bundle_complete", pa.bool_()),
|
||||||
|
("image_bundle_da_json", pa.string()),
|
||||||
|
("image_bundle_en_json", pa.string()),
|
||||||
|
("has_image", pa.bool_()),
|
||||||
|
("image_iiif_id", pa.string()),
|
||||||
|
("image_iiif_info", pa.string()),
|
||||||
|
("image_width", pa.int64()),
|
||||||
|
("image_height", pa.int64()),
|
||||||
|
("image_thumbnail", pa.string()),
|
||||||
|
("image_native", pa.string()),
|
||||||
|
("image_mime_type", pa.string()),
|
||||||
|
("image_cropped", pa.bool_()),
|
||||||
|
("image_orientation", pa.string()),
|
||||||
|
("image_hq", pa.bool_()),
|
||||||
|
("dimension_observations", pa.list_(DIMENSION_STRUCT)),
|
||||||
|
("dimension_conflict_slots", pa.list_(pa.string())),
|
||||||
|
("dimension_conflict_candidates", pa.list_(DIMENSION_STRUCT)),
|
||||||
|
])
|
||||||
|
for part in SUPPORTED_PARTS:
|
||||||
|
for axis in SUPPORTED_AXES:
|
||||||
|
fields.append((f"{part}_{axis}_cm", pa.float64()))
|
||||||
|
fields.extend([
|
||||||
|
("has_net_width", pa.bool_()),
|
||||||
|
("has_net_height", pa.bool_()),
|
||||||
|
("has_2d_physical_geometry", pa.bool_()),
|
||||||
|
("has_3d_physical_geometry", pa.bool_()),
|
||||||
|
("has_any_physical_dimension", pa.bool_()),
|
||||||
|
("quality_flags", pa.list_(pa.string())),
|
||||||
|
("previous_public_visual_candidate", pa.bool_()),
|
||||||
|
("corpus_smk_public_visual_v1", pa.bool_()),
|
||||||
|
("corpus_exclusion_reasons", pa.list_(pa.string())),
|
||||||
|
])
|
||||||
|
return pa.schema(fields)
|
||||||
|
|
||||||
|
|
||||||
|
MEMBERSHIP_SCHEMA = pa.schema([
|
||||||
|
("smk_entity_id", pa.string()),
|
||||||
|
("object_number", pa.string()),
|
||||||
|
("included", pa.bool_()),
|
||||||
|
("exclusion_reasons", pa.list_(pa.string())),
|
||||||
|
("quality_flags", pa.list_(pa.string())),
|
||||||
|
])
|
||||||
|
|
||||||
|
GROUP_SCHEMA = pa.schema([
|
||||||
|
("object_number", pa.string()),
|
||||||
|
("canonical_entity_ids", pa.list_(pa.string())),
|
||||||
|
("group_size", pa.int64()),
|
||||||
|
("group_classification", pa.string()),
|
||||||
|
("ambiguity_flag", pa.bool_()),
|
||||||
|
("corpus_member_count", pa.int64()),
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def _raw_value(record: dict[str, Any] | None, key: str) -> Any:
|
||||||
|
return record.get(key) if record else None
|
||||||
|
|
||||||
|
|
||||||
|
def _json_value(record: dict[str, Any] | None, key: str) -> str | None:
|
||||||
|
value = _raw_value(record, key)
|
||||||
|
return logical_json(value) if value is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def _localized_fields(row: dict[str, Any], da: dict[str, Any] | None, en: dict[str, Any] | None) -> None:
|
||||||
|
mappings = {
|
||||||
|
"titles": "titles",
|
||||||
|
"object_names": "object_names",
|
||||||
|
"classifications": "classification",
|
||||||
|
"artists": "artist",
|
||||||
|
"production": "production",
|
||||||
|
"production_dates": "production_date",
|
||||||
|
"techniques": "techniques",
|
||||||
|
"materials": "materials",
|
||||||
|
}
|
||||||
|
for output, source in mappings.items():
|
||||||
|
row[f"{output}_da_json"] = _json_value(da, source)
|
||||||
|
row[f"{output}_en_json"] = _json_value(en, source)
|
||||||
|
|
||||||
|
|
||||||
|
def _visual_fields(
|
||||||
|
row: dict[str, Any], da: dict[str, Any] | None, en: dict[str, Any] | None,
|
||||||
|
) -> list[str]:
|
||||||
|
flags: list[str] = []
|
||||||
|
for field in VISUAL_FIELDS:
|
||||||
|
da_raw = _raw_value(da, field)
|
||||||
|
en_raw = _raw_value(en, field)
|
||||||
|
da_number, da_status = visual_number(da, field)
|
||||||
|
en_number, en_status = visual_number(en, field)
|
||||||
|
canonical, provenance, status = resolve_scalar(da_number, en_number)
|
||||||
|
if da_raw is not None and da_number is None:
|
||||||
|
status = f"invalid_da:{da_status}"
|
||||||
|
if en_raw is not None and en_number is None:
|
||||||
|
status = f"invalid_en:{en_status}" if status == "missing" else f"{status}|invalid_en:{en_status}"
|
||||||
|
row[field] = canonical
|
||||||
|
row[f"{field}_da_json"] = logical_json(da_raw) if da_raw is not None else None
|
||||||
|
row[f"{field}_en_json"] = logical_json(en_raw) if en_raw is not None else None
|
||||||
|
row[f"{field}_provenance"] = provenance
|
||||||
|
row[f"{field}_status"] = status
|
||||||
|
if field == "colortemp" and (
|
||||||
|
(isinstance(da_raw, str) and da_raw.casefold() == "infinity")
|
||||||
|
or (isinstance(en_raw, str) and en_raw.casefold() == "infinity")
|
||||||
|
):
|
||||||
|
flags.append("COLORTEMP_NONFINITE_SOURCE")
|
||||||
|
if any(row[field] is None for field in VISUAL_FIELDS):
|
||||||
|
flags.append("VISUAL_FIELDS_PARTIAL")
|
||||||
|
return flags
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_row(
|
||||||
|
pair: dict[str, Any], shared_object_numbers: set[str], colortemp_median: float,
|
||||||
|
colortemp_mad: float,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
da = json.loads(pair["da_record_json"]) if pair["da_record_json"] else None
|
||||||
|
en = json.loads(pair["en_record_json"]) if pair["en_record_json"] else None
|
||||||
|
da_entity_id = da.get("id") if da else None
|
||||||
|
en_entity_id = en.get("id") if en else None
|
||||||
|
entity_id = resolve_smk_entity_id(da_entity_id, en_entity_id)
|
||||||
|
if pair["smk_entity_id"] != entity_id:
|
||||||
|
raise ValueError("Joined SMK entity id does not match source record id")
|
||||||
|
row: dict[str, Any] = {
|
||||||
|
"canonical_schema_version": CANONICAL_VERSION,
|
||||||
|
"smk_entity_id": entity_id,
|
||||||
|
"source_row_id_da": pair["da_source_row_id"],
|
||||||
|
"source_row_id_en": pair["en_source_row_id"],
|
||||||
|
"present_in_da": da is not None,
|
||||||
|
"present_in_en": en is not None,
|
||||||
|
"source_manifest_sha256_da": pair["da_manifest_sha"],
|
||||||
|
"source_manifest_sha256_en": pair["en_manifest_sha"],
|
||||||
|
"source_file_sha256_da": pair["da_file_sha"],
|
||||||
|
"source_file_sha256_en": pair["en_file_sha"],
|
||||||
|
}
|
||||||
|
da_object_number = _raw_value(da, "object_number")
|
||||||
|
en_object_number = _raw_value(en, "object_number")
|
||||||
|
object_number, provenance, status = resolve_scalar(da_object_number, en_object_number)
|
||||||
|
row.update({
|
||||||
|
"object_number": object_number,
|
||||||
|
"object_number_da": da_object_number,
|
||||||
|
"object_number_en": en_object_number,
|
||||||
|
"object_number_provenance": provenance,
|
||||||
|
"object_number_status": status,
|
||||||
|
})
|
||||||
|
_localized_fields(row, da, en)
|
||||||
|
|
||||||
|
da_public = _raw_value(da, "public_domain")
|
||||||
|
en_public = _raw_value(en, "public_domain")
|
||||||
|
public_domain, public_provenance, public_status = resolve_scalar(da_public, en_public)
|
||||||
|
row.update({
|
||||||
|
"public_domain": public_domain,
|
||||||
|
"public_domain_da": da_public,
|
||||||
|
"public_domain_en": en_public,
|
||||||
|
"public_domain_provenance": public_provenance,
|
||||||
|
"public_domain_status": public_status,
|
||||||
|
})
|
||||||
|
|
||||||
|
da_rights = _raw_value(da, "rights")
|
||||||
|
en_rights = _raw_value(en, "rights")
|
||||||
|
da_semantic = rights_semantic(da_rights)
|
||||||
|
en_semantic = rights_semantic(en_rights)
|
||||||
|
if da is not None and en is not None and da_semantic == en_semantic:
|
||||||
|
rights_value, rights_provenance, rights_status = da_semantic, "both", "semantic_agreement"
|
||||||
|
elif da is not None and en is None:
|
||||||
|
rights_value, rights_provenance, rights_status = da_semantic, "da", "single_source"
|
||||||
|
elif en is not None and da is None:
|
||||||
|
rights_value, rights_provenance, rights_status = en_semantic, "en", "single_source"
|
||||||
|
else:
|
||||||
|
rights_value, rights_provenance, rights_status = None, "conflict", "conflict"
|
||||||
|
row.update({
|
||||||
|
"rights_da": da_rights,
|
||||||
|
"rights_en": en_rights,
|
||||||
|
"rights_semantic": rights_value,
|
||||||
|
"rights_semantic_da": da_semantic if da is not None else None,
|
||||||
|
"rights_semantic_en": en_semantic if en is not None else None,
|
||||||
|
"rights_provenance": rights_provenance,
|
||||||
|
"rights_status": rights_status,
|
||||||
|
})
|
||||||
|
|
||||||
|
da_colors = _raw_value(da, "colors") if isinstance(_raw_value(da, "colors"), list) else None
|
||||||
|
en_colors = _raw_value(en, "colors") if isinstance(_raw_value(en, "colors"), list) else None
|
||||||
|
colors, colors_provenance, colors_status = resolve_scalar(da_colors, en_colors)
|
||||||
|
row.update({
|
||||||
|
"colors": colors,
|
||||||
|
"colors_da": da_colors,
|
||||||
|
"colors_en": en_colors,
|
||||||
|
"colors_provenance": colors_provenance,
|
||||||
|
"colors_status": colors_status,
|
||||||
|
})
|
||||||
|
|
||||||
|
quality_flags = _visual_fields(row, da, en)
|
||||||
|
if is_extreme_colortemp(row["colortemp"], colortemp_median, colortemp_mad):
|
||||||
|
quality_flags.append("COLORTEMP_EXTREME")
|
||||||
|
|
||||||
|
da_bundle = image_bundle(da)
|
||||||
|
en_bundle = image_bundle(en)
|
||||||
|
canonical_bundle, image_provenance, image_status = resolve_image_bundle(da_bundle, en_bundle)
|
||||||
|
row.update({
|
||||||
|
"image_bundle_status": image_status,
|
||||||
|
"image_bundle_provenance": image_provenance,
|
||||||
|
"image_bundle_complete": image_bundle_complete(canonical_bundle),
|
||||||
|
"image_bundle_da_json": logical_json(da_bundle) if da_bundle is not None else None,
|
||||||
|
"image_bundle_en_json": logical_json(en_bundle) if en_bundle is not None else None,
|
||||||
|
})
|
||||||
|
for key in IMAGE_KEYS:
|
||||||
|
row[key] = canonical_bundle.get(key) if canonical_bundle else None
|
||||||
|
|
||||||
|
observations = normalize_dimension_observations(da, "da", pair["da_source_row_id"])
|
||||||
|
observations += normalize_dimension_observations(en, "en", pair["en_source_row_id"])
|
||||||
|
slots, dimension_conflicts, conflict_candidates = dimension_slots(observations)
|
||||||
|
row.update(slots)
|
||||||
|
row["dimension_observations"] = observations
|
||||||
|
row["dimension_conflict_slots"] = dimension_conflicts
|
||||||
|
row["dimension_conflict_candidates"] = conflict_candidates
|
||||||
|
row["has_net_width"] = row["net_width_cm"] is not None
|
||||||
|
row["has_net_height"] = row["net_height_cm"] is not None
|
||||||
|
row["has_2d_physical_geometry"] = row["has_net_width"] and row["has_net_height"]
|
||||||
|
row["has_3d_physical_geometry"] = (
|
||||||
|
row["has_2d_physical_geometry"] and row["net_depth_cm"] is not None
|
||||||
|
)
|
||||||
|
row["has_any_physical_dimension"] = any(value is not None for value in slots.values())
|
||||||
|
|
||||||
|
def previous_candidate(record: dict[str, Any] | None) -> bool:
|
||||||
|
return bool(
|
||||||
|
record and record.get("public_domain") is True and record.get("has_image") is True
|
||||||
|
and (record.get("image_native") or record.get("image_thumbnail") or record.get("image_iiif_id"))
|
||||||
|
)
|
||||||
|
|
||||||
|
row["previous_public_visual_candidate"] = previous_candidate(da) or previous_candidate(en)
|
||||||
|
|
||||||
|
if da is None or en is None:
|
||||||
|
quality_flags.append("SOURCE_LANGUAGE_MISSING")
|
||||||
|
if object_number in shared_object_numbers:
|
||||||
|
quality_flags.append("OBJECT_NUMBER_SHARED")
|
||||||
|
if dimension_conflicts:
|
||||||
|
quality_flags.append("DIMENSION_CONFLICT")
|
||||||
|
quality_flags.append(image_status)
|
||||||
|
unresolved = any(status_value == "conflict" for status_value in (
|
||||||
|
status, public_status, rights_status, colors_status,
|
||||||
|
*(row[f"{field}_status"] for field in VISUAL_FIELDS),
|
||||||
|
))
|
||||||
|
if unresolved:
|
||||||
|
quality_flags.append("CANONICALIZATION_UNRESOLVED")
|
||||||
|
row["quality_flags"] = sorted(set(quality_flags))
|
||||||
|
included, reasons = corpus_membership(row)
|
||||||
|
row["corpus_smk_public_visual_v1"] = included
|
||||||
|
row["corpus_exclusion_reasons"] = reasons
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def _colortemp_parameters(source_paths: tuple[Path, Path]) -> tuple[float, float, int]:
|
||||||
|
values_by_entity: dict[str, float] = {}
|
||||||
|
for source_path in source_paths:
|
||||||
|
parquet = pq.ParquetFile(source_path)
|
||||||
|
for batch in parquet.iter_batches(columns=["smk_source_id", "visual_json"], batch_size=10_000):
|
||||||
|
for row in batch.to_pylist():
|
||||||
|
if row["smk_source_id"] in values_by_entity:
|
||||||
|
continue
|
||||||
|
visual = json.loads(row["visual_json"])
|
||||||
|
number, _ = visual_number(visual, "colortemp")
|
||||||
|
if number is not None:
|
||||||
|
values_by_entity[row["smk_source_id"]] = number
|
||||||
|
values = list(values_by_entity.values())
|
||||||
|
median, mad = robust_colortemp_parameters(values)
|
||||||
|
return median, mad, len(values)
|
||||||
|
|
||||||
|
|
||||||
|
def _shared_object_numbers(connection: duckdb.DuckDBPyConnection, da_path: str, en_path: str) -> set[str]:
|
||||||
|
query = f"""
|
||||||
|
SELECT object_number
|
||||||
|
FROM (
|
||||||
|
SELECT smk_source_id, object_number FROM read_parquet('{da_path}')
|
||||||
|
UNION
|
||||||
|
SELECT smk_source_id, object_number FROM read_parquet('{en_path}')
|
||||||
|
)
|
||||||
|
WHERE object_number IS NOT NULL
|
||||||
|
GROUP BY object_number
|
||||||
|
HAVING count(DISTINCT smk_source_id) > 1
|
||||||
|
"""
|
||||||
|
return {row[0] for row in connection.execute(query).fetchall()}
|
||||||
|
|
||||||
|
|
||||||
|
def run(root: Path) -> dict[str, Any]:
|
||||||
|
da_path = (root / "data" / "interim" / "source_da.parquet").resolve()
|
||||||
|
en_path = (root / "data" / "interim" / "source_en.parquet").resolve()
|
||||||
|
processed = root / "data" / "processed"
|
||||||
|
reports = root / "data" / "reports"
|
||||||
|
processed.mkdir(parents=True, exist_ok=True)
|
||||||
|
reports.mkdir(parents=True, exist_ok=True)
|
||||||
|
connection = duckdb.connect()
|
||||||
|
da_sql = da_path.as_posix().replace("'", "''")
|
||||||
|
en_sql = en_path.as_posix().replace("'", "''")
|
||||||
|
shared_object_numbers = _shared_object_numbers(connection, da_sql, en_sql)
|
||||||
|
colortemp_median, colortemp_mad, colortemp_count = _colortemp_parameters((da_path, en_path))
|
||||||
|
|
||||||
|
schema = _schema()
|
||||||
|
da_metadata = pq.ParquetFile(da_path).schema_arrow.metadata or {}
|
||||||
|
en_metadata = pq.ParquetFile(en_path).schema_arrow.metadata or {}
|
||||||
|
manifest_sha = da_metadata[b"smk.source_manifest_sha256"].decode()
|
||||||
|
if en_metadata[b"smk.source_manifest_sha256"].decode() != manifest_sha:
|
||||||
|
raise ValueError("DA/EN source Parquets reference different source manifests")
|
||||||
|
metadata = {
|
||||||
|
b"smk.canonical_schema_version": CANONICAL_VERSION.encode(),
|
||||||
|
b"smk.source_manifest_sha256": manifest_sha.encode(),
|
||||||
|
b"smk.corpus_rule": b"smk-public-visual-v1",
|
||||||
|
}
|
||||||
|
artwork_writer = pq.ParquetWriter(processed / "artworks.parquet", schema.with_metadata(metadata), compression="zstd")
|
||||||
|
corpus_writer = pq.ParquetWriter(
|
||||||
|
processed / "smk-public-visual-v1.parquet", schema.with_metadata(metadata), compression="zstd",
|
||||||
|
)
|
||||||
|
membership_writer = pq.ParquetWriter(
|
||||||
|
reports / "corpus-membership.parquet", MEMBERSHIP_SCHEMA.with_metadata(metadata), compression="zstd",
|
||||||
|
)
|
||||||
|
|
||||||
|
query = f"""
|
||||||
|
SELECT
|
||||||
|
coalesce(da.smk_source_id, en.smk_source_id) AS smk_entity_id,
|
||||||
|
da.source_row_id AS da_source_row_id,
|
||||||
|
en.source_row_id AS en_source_row_id,
|
||||||
|
da.source_manifest_sha256 AS da_manifest_sha,
|
||||||
|
en.source_manifest_sha256 AS en_manifest_sha,
|
||||||
|
da.source_file_sha256 AS da_file_sha,
|
||||||
|
en.source_file_sha256 AS en_file_sha,
|
||||||
|
da.source_record_json AS da_record_json,
|
||||||
|
en.source_record_json AS en_record_json
|
||||||
|
FROM read_parquet('{da_sql}') da
|
||||||
|
FULL OUTER JOIN read_parquet('{en_sql}') en
|
||||||
|
ON da.smk_source_id = en.smk_source_id
|
||||||
|
ORDER BY smk_entity_id
|
||||||
|
"""
|
||||||
|
reader = connection.execute(query).to_arrow_reader(1_000)
|
||||||
|
counters: dict[str, Counter[str]] = {
|
||||||
|
"pairing": Counter(), "images": Counter(), "rights": Counter(), "quality": Counter(),
|
||||||
|
"exclusions": Counter(), "dimension_slots": Counter(),
|
||||||
|
}
|
||||||
|
object_groups: dict[str, list[str]] = defaultdict(list)
|
||||||
|
corpus_entities: set[str] = set()
|
||||||
|
total = 0
|
||||||
|
included_count = 0
|
||||||
|
corpus_missing_2d = 0
|
||||||
|
corpus_visual_partial = 0
|
||||||
|
corpus_colortemp_flags = 0
|
||||||
|
previous_candidate_count = 0
|
||||||
|
previous_candidate_corpus_overlap = 0
|
||||||
|
corpus_missing_any_physical = 0
|
||||||
|
corpus_missing_visual_fields: Counter[str] = Counter()
|
||||||
|
try:
|
||||||
|
for batch in reader:
|
||||||
|
artwork_rows: list[dict[str, Any]] = []
|
||||||
|
corpus_rows: list[dict[str, Any]] = []
|
||||||
|
membership_rows: list[dict[str, Any]] = []
|
||||||
|
for pair in batch.to_pylist():
|
||||||
|
row = _canonical_row(pair, shared_object_numbers, colortemp_median, colortemp_mad)
|
||||||
|
total += 1
|
||||||
|
if row["present_in_da"] and row["present_in_en"]:
|
||||||
|
counters["pairing"]["both"] += 1
|
||||||
|
elif row["present_in_da"]:
|
||||||
|
counters["pairing"]["da_only"] += 1
|
||||||
|
else:
|
||||||
|
counters["pairing"]["en_only"] += 1
|
||||||
|
if row["object_number_status"] == "conflict":
|
||||||
|
counters["pairing"]["object_number_conflict"] += 1
|
||||||
|
counters["images"][row["image_bundle_status"]] += 1
|
||||||
|
counters["rights"][row["rights_semantic"] or "CONFLICT"] += 1
|
||||||
|
for flag in row["quality_flags"]:
|
||||||
|
counters["quality"][flag] += 1
|
||||||
|
for reason in row["corpus_exclusion_reasons"]:
|
||||||
|
counters["exclusions"][reason] += 1
|
||||||
|
for part in SUPPORTED_PARTS:
|
||||||
|
for axis in SUPPORTED_AXES:
|
||||||
|
if row[f"{part}_{axis}_cm"] is not None:
|
||||||
|
counters["dimension_slots"][f"{part}_{axis}_cm"] += 1
|
||||||
|
if row["object_number"]:
|
||||||
|
object_groups[row["object_number"]].append(row["smk_entity_id"])
|
||||||
|
if row["previous_public_visual_candidate"]:
|
||||||
|
previous_candidate_count += 1
|
||||||
|
if row["corpus_smk_public_visual_v1"]:
|
||||||
|
included_count += 1
|
||||||
|
corpus_entities.add(row["smk_entity_id"])
|
||||||
|
corpus_rows.append(row)
|
||||||
|
if not row["has_2d_physical_geometry"]:
|
||||||
|
corpus_missing_2d += 1
|
||||||
|
if not row["has_any_physical_dimension"]:
|
||||||
|
corpus_missing_any_physical += 1
|
||||||
|
if "VISUAL_FIELDS_PARTIAL" in row["quality_flags"]:
|
||||||
|
corpus_visual_partial += 1
|
||||||
|
for field in VISUAL_FIELDS:
|
||||||
|
if row[field] is None:
|
||||||
|
corpus_missing_visual_fields[field] += 1
|
||||||
|
if any(flag.startswith("COLORTEMP_") for flag in row["quality_flags"]):
|
||||||
|
corpus_colortemp_flags += 1
|
||||||
|
if row["previous_public_visual_candidate"]:
|
||||||
|
previous_candidate_corpus_overlap += 1
|
||||||
|
artwork_rows.append(row)
|
||||||
|
membership_rows.append({
|
||||||
|
"smk_entity_id": row["smk_entity_id"],
|
||||||
|
"object_number": row["object_number"],
|
||||||
|
"included": row["corpus_smk_public_visual_v1"],
|
||||||
|
"exclusion_reasons": row["corpus_exclusion_reasons"],
|
||||||
|
"quality_flags": row["quality_flags"],
|
||||||
|
})
|
||||||
|
artwork_writer.write_table(pa.Table.from_pylist(artwork_rows, schema=schema))
|
||||||
|
membership_writer.write_table(pa.Table.from_pylist(membership_rows, schema=MEMBERSHIP_SCHEMA))
|
||||||
|
if corpus_rows:
|
||||||
|
corpus_writer.write_table(pa.Table.from_pylist(corpus_rows, schema=schema))
|
||||||
|
finally:
|
||||||
|
artwork_writer.close()
|
||||||
|
corpus_writer.close()
|
||||||
|
membership_writer.close()
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
group_rows: list[dict[str, Any]] = []
|
||||||
|
corpus_shared_groups = 0
|
||||||
|
corpus_shared_entities = 0
|
||||||
|
for object_number, entity_ids in sorted(object_groups.items()):
|
||||||
|
sorted_ids = sorted(entity_ids)
|
||||||
|
corpus_members = sum(entity_id in corpus_entities for entity_id in sorted_ids)
|
||||||
|
if len(sorted_ids) > 1 and corpus_members:
|
||||||
|
corpus_shared_groups += 1
|
||||||
|
corpus_shared_entities += corpus_members
|
||||||
|
group_rows.append({
|
||||||
|
"object_number": object_number,
|
||||||
|
"canonical_entity_ids": sorted_ids,
|
||||||
|
"group_size": len(sorted_ids),
|
||||||
|
"group_classification": (
|
||||||
|
"multiple_smk_entities_same_object_number" if len(sorted_ids) > 1 else "unique"
|
||||||
|
),
|
||||||
|
"ambiguity_flag": len(sorted_ids) > 1,
|
||||||
|
"corpus_member_count": corpus_members,
|
||||||
|
})
|
||||||
|
pq.write_table(
|
||||||
|
pa.Table.from_pylist(group_rows, schema=GROUP_SCHEMA),
|
||||||
|
processed / "object-number-groups.parquet", compression="zstd",
|
||||||
|
)
|
||||||
|
summary = {
|
||||||
|
"canonical_schema_version": CANONICAL_VERSION,
|
||||||
|
"source_manifest_sha256": manifest_sha,
|
||||||
|
"canonical_entity_count": total,
|
||||||
|
"pairing": dict(counters["pairing"]),
|
||||||
|
"object_number_groups": {
|
||||||
|
"total": len(group_rows),
|
||||||
|
"shared_groups": len(shared_object_numbers),
|
||||||
|
"largest_group": max(len(ids) for ids in object_groups.values()),
|
||||||
|
"shared_groups_with_corpus_members": corpus_shared_groups,
|
||||||
|
"corpus_members_in_shared_groups": corpus_shared_entities,
|
||||||
|
},
|
||||||
|
"dimensions": {
|
||||||
|
"entities_with_conflicts": counters["quality"]["DIMENSION_CONFLICT"],
|
||||||
|
"entities_with_net_width": counters["dimension_slots"]["net_width_cm"],
|
||||||
|
"entities_with_net_height": counters["dimension_slots"]["net_height_cm"],
|
||||||
|
"slot_success_counts": dict(counters["dimension_slots"]),
|
||||||
|
},
|
||||||
|
"images": dict(counters["images"]),
|
||||||
|
"rights": dict(counters["rights"]),
|
||||||
|
"quality_flags": dict(counters["quality"]),
|
||||||
|
"colortemp_extreme_rule": {
|
||||||
|
"rule": "absolute robust z-score > 10 using median and MAD",
|
||||||
|
"entity_numeric_count": colortemp_count,
|
||||||
|
"median": colortemp_median,
|
||||||
|
"mad": colortemp_mad,
|
||||||
|
},
|
||||||
|
"corpus": {
|
||||||
|
"name": "smk-public-visual-v1",
|
||||||
|
"included": included_count,
|
||||||
|
"excluded": total - included_count,
|
||||||
|
"exclusion_reason_counts": dict(counters["exclusions"]),
|
||||||
|
"previous_public_visual_candidate_count": previous_candidate_count,
|
||||||
|
"overlap_with_previous_public_visual_candidates": previous_candidate_corpus_overlap,
|
||||||
|
"included_without_any_normalized_physical_dimension": corpus_missing_any_physical,
|
||||||
|
"included_without_2d_net_geometry": corpus_missing_2d,
|
||||||
|
"included_with_partial_visual_fields": corpus_visual_partial,
|
||||||
|
"included_missing_visual_field_counts": dict(corpus_missing_visual_fields),
|
||||||
|
"included_with_colortemp_quality_flags": corpus_colortemp_flags,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
(reports / "milestone2-summary.json").write_text(
|
||||||
|
json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8",
|
||||||
|
)
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--root", type=Path, default=Path.cwd())
|
||||||
|
args = parser.parse_args()
|
||||||
|
print(json.dumps(run(args.root.resolve()), indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
246
src/smk_playground/canonical/rules.py
Normal file
246
src/smk_playground/canonical/rules.py
Normal file
|
|
@ -0,0 +1,246 @@
|
||||||
|
"""Pure, deterministic Milestone 2 canonicalization rules."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
import statistics
|
||||||
|
import unicodedata
|
||||||
|
from collections import defaultdict
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from smk_playground.ingest.source import parse_strict_number
|
||||||
|
|
||||||
|
PDM_URL = "https://creativecommons.org/publicdomain/mark/1.0/"
|
||||||
|
CC0_URL = "https://creativecommons.org/publicdomain/zero/1.0/"
|
||||||
|
SMK_POLICY_URLS = {
|
||||||
|
"https://www.smk.dk/section/brug-af-museets-materiale/",
|
||||||
|
"https://www.smk.dk/en/section/use-of-smk-material/",
|
||||||
|
}
|
||||||
|
|
||||||
|
PART_ALIASES = {
|
||||||
|
"netto": "net",
|
||||||
|
"nettomål": "net",
|
||||||
|
"brutto": "gross",
|
||||||
|
"bruttomål": "gross",
|
||||||
|
"bladmål": "sheet",
|
||||||
|
"plademål": "plate",
|
||||||
|
"monteringsmål": "mount",
|
||||||
|
"billedmål": "image",
|
||||||
|
"papirmål": "paper",
|
||||||
|
}
|
||||||
|
TYPE_ALIASES = {
|
||||||
|
"bredde": "width",
|
||||||
|
"width": "width",
|
||||||
|
"højde": "height",
|
||||||
|
"height": "height",
|
||||||
|
"dybde": "depth",
|
||||||
|
"depth": "depth",
|
||||||
|
"diameter": "diameter",
|
||||||
|
"længde": "length",
|
||||||
|
"length": "length",
|
||||||
|
}
|
||||||
|
UNIT_TO_CM = {
|
||||||
|
"millimeter": 0.1,
|
||||||
|
"milimeter": 0.1,
|
||||||
|
"mm": 0.1,
|
||||||
|
"centimeter": 1.0,
|
||||||
|
"cm": 1.0,
|
||||||
|
}
|
||||||
|
SUPPORTED_PARTS = ("net", "gross", "sheet", "plate", "mount", "image", "paper")
|
||||||
|
SUPPORTED_AXES = ("width", "height", "depth", "diameter", "length")
|
||||||
|
IMAGE_KEYS = (
|
||||||
|
"has_image", "image_iiif_id", "image_iiif_info", "image_width", "image_height",
|
||||||
|
"image_thumbnail", "image_native", "image_mime_type", "image_cropped",
|
||||||
|
"image_orientation", "image_hq",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_smk_entity_id(da_id: str | None, en_id: str | None) -> str:
|
||||||
|
"""Pair languages only when their stable SMK source entity IDs agree."""
|
||||||
|
if da_id and en_id and da_id != en_id:
|
||||||
|
raise ValueError(f"Cannot pair different SMK ids: {da_id!r} != {en_id!r}")
|
||||||
|
entity_id = da_id or en_id
|
||||||
|
if not entity_id:
|
||||||
|
raise ValueError("Canonical entity requires an SMK id")
|
||||||
|
return entity_id
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_vocabulary_token(value: Any) -> str | None:
|
||||||
|
if not isinstance(value, str) or not value.strip():
|
||||||
|
return None
|
||||||
|
token = unicodedata.normalize("NFC", value.strip()).casefold()
|
||||||
|
return token.replace("maal", "mål")
|
||||||
|
|
||||||
|
|
||||||
|
def rights_semantic(value: Any) -> str:
|
||||||
|
if value == PDM_URL:
|
||||||
|
return "PDM"
|
||||||
|
if value == CC0_URL:
|
||||||
|
return "CC0"
|
||||||
|
if value in SMK_POLICY_URLS:
|
||||||
|
return "SMK_RESTRICTED_POLICY"
|
||||||
|
return "UNKNOWN"
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_scalar(da_value: Any, en_value: Any) -> tuple[Any, str, str]:
|
||||||
|
"""Return canonical value, provenance, and status without global language precedence."""
|
||||||
|
if da_value == en_value and da_value is not None:
|
||||||
|
return da_value, "both", "agreement"
|
||||||
|
if da_value is not None and en_value is None:
|
||||||
|
return da_value, "da", "single_source"
|
||||||
|
if en_value is not None and da_value is None:
|
||||||
|
return en_value, "en", "single_source"
|
||||||
|
if da_value is None and en_value is None:
|
||||||
|
return None, "none", "missing"
|
||||||
|
return None, "conflict", "conflict"
|
||||||
|
|
||||||
|
|
||||||
|
def image_bundle(record: dict[str, Any] | None) -> dict[str, Any] | None:
|
||||||
|
if record is None:
|
||||||
|
return None
|
||||||
|
return {key: record.get(key) for key in IMAGE_KEYS}
|
||||||
|
|
||||||
|
|
||||||
|
def image_bundle_complete(bundle: dict[str, Any] | None) -> bool:
|
||||||
|
if not bundle or bundle.get("has_image") is not True:
|
||||||
|
return False
|
||||||
|
width = bundle.get("image_width")
|
||||||
|
height = bundle.get("image_height")
|
||||||
|
return (
|
||||||
|
isinstance(width, int) and not isinstance(width, bool) and width > 0
|
||||||
|
and isinstance(height, int) and not isinstance(height, bool) and height > 0
|
||||||
|
and isinstance(bundle.get("image_iiif_id"), str) and bool(bundle["image_iiif_id"])
|
||||||
|
and bool(bundle.get("image_thumbnail") or bundle.get("image_native"))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_image_bundle(
|
||||||
|
da_bundle: dict[str, Any] | None, en_bundle: dict[str, Any] | None,
|
||||||
|
) -> tuple[dict[str, Any] | None, str, str]:
|
||||||
|
da_complete = image_bundle_complete(da_bundle)
|
||||||
|
en_complete = image_bundle_complete(en_bundle)
|
||||||
|
if da_bundle == en_bundle and da_bundle is not None:
|
||||||
|
status = "IMAGE_BUNDLE_OK" if da_complete else "IMAGE_BUNDLE_INCOMPLETE"
|
||||||
|
return da_bundle, "both", status
|
||||||
|
if da_complete and not en_complete:
|
||||||
|
return da_bundle, "da", "IMAGE_BUNDLE_SINGLE_SOURCE"
|
||||||
|
if en_complete and not da_complete:
|
||||||
|
return en_bundle, "en", "IMAGE_BUNDLE_SINGLE_SOURCE"
|
||||||
|
if da_complete and en_complete:
|
||||||
|
return None, "conflict", "IMAGE_BUNDLE_CONFLICT"
|
||||||
|
if da_bundle is not None and en_bundle is None:
|
||||||
|
return da_bundle, "da", "IMAGE_BUNDLE_INCOMPLETE"
|
||||||
|
if en_bundle is not None and da_bundle is None:
|
||||||
|
return en_bundle, "en", "IMAGE_BUNDLE_INCOMPLETE"
|
||||||
|
return None, "none", "IMAGE_BUNDLE_INCOMPLETE"
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_dimension_observations(
|
||||||
|
record: dict[str, Any] | None, language: str, source_row_id: str | None,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
if record is None:
|
||||||
|
return []
|
||||||
|
dimensions = record.get("dimensions")
|
||||||
|
if not isinstance(dimensions, list):
|
||||||
|
return []
|
||||||
|
output: list[dict[str, Any]] = []
|
||||||
|
for item in dimensions:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
original_part = item.get("part")
|
||||||
|
original_type = item.get("type")
|
||||||
|
original_unit = item.get("unit")
|
||||||
|
original_value = item.get("value")
|
||||||
|
part = PART_ALIASES.get(normalize_vocabulary_token(original_part) or "")
|
||||||
|
axis = TYPE_ALIASES.get(normalize_vocabulary_token(original_type) or "")
|
||||||
|
unit_token = normalize_vocabulary_token(original_unit)
|
||||||
|
factor = UNIT_TO_CM.get(unit_token or "")
|
||||||
|
number, number_status = parse_strict_number(original_value)
|
||||||
|
if part and axis and factor is not None and number is not None:
|
||||||
|
value_cm = number * factor
|
||||||
|
status = "normalized"
|
||||||
|
elif axis is None and normalize_vocabulary_token(original_type) in {
|
||||||
|
"vægt", "weight", "spilletid", "playing time", "bruttoregisterton",
|
||||||
|
}:
|
||||||
|
value_cm = None
|
||||||
|
status = "non_geometric"
|
||||||
|
else:
|
||||||
|
value_cm = None
|
||||||
|
status = f"unmapped_or_{number_status}"
|
||||||
|
output.append({
|
||||||
|
"part_canonical": part,
|
||||||
|
"type_canonical": axis,
|
||||||
|
"unit_canonical": "cm" if value_cm is not None else None,
|
||||||
|
"value_canonical": value_cm,
|
||||||
|
"original_language": language,
|
||||||
|
"original_part": None if original_part is None else str(original_part),
|
||||||
|
"original_type": None if original_type is None else str(original_type),
|
||||||
|
"original_unit": None if original_unit is None else str(original_unit),
|
||||||
|
"original_value": None if original_value is None else str(original_value),
|
||||||
|
"source_row_id": source_row_id,
|
||||||
|
"status": status,
|
||||||
|
})
|
||||||
|
return output
|
||||||
|
|
||||||
|
|
||||||
|
def dimension_slots(
|
||||||
|
observations: list[dict[str, Any]],
|
||||||
|
) -> tuple[dict[str, float | None], list[str], list[dict[str, Any]]]:
|
||||||
|
candidates: dict[str, set[float]] = defaultdict(set)
|
||||||
|
candidate_rows: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
||||||
|
for observation in observations:
|
||||||
|
part = observation["part_canonical"]
|
||||||
|
axis = observation["type_canonical"]
|
||||||
|
value = observation["value_canonical"]
|
||||||
|
if part in SUPPORTED_PARTS and axis in SUPPORTED_AXES and value is not None:
|
||||||
|
slot = f"{part}_{axis}_cm"
|
||||||
|
candidates[slot].add(round(float(value), 12))
|
||||||
|
candidate_rows[slot].append(observation)
|
||||||
|
slots: dict[str, float | None] = {}
|
||||||
|
conflicts: list[str] = []
|
||||||
|
for part in SUPPORTED_PARTS:
|
||||||
|
for axis in SUPPORTED_AXES:
|
||||||
|
slot = f"{part}_{axis}_cm"
|
||||||
|
values = candidates.get(slot, set())
|
||||||
|
slots[slot] = next(iter(values)) if len(values) == 1 else None
|
||||||
|
if len(values) > 1:
|
||||||
|
conflicts.append(slot)
|
||||||
|
conflict_candidates = [row for slot in conflicts for row in candidate_rows[slot]]
|
||||||
|
return slots, conflicts, conflict_candidates
|
||||||
|
|
||||||
|
|
||||||
|
def visual_number(record: dict[str, Any] | None, field: str) -> tuple[float | None, str]:
|
||||||
|
if record is None:
|
||||||
|
return None, "missing"
|
||||||
|
return parse_strict_number(record.get(field))
|
||||||
|
|
||||||
|
|
||||||
|
def robust_colortemp_parameters(values: list[float]) -> tuple[float, float]:
|
||||||
|
finite = [value for value in values if math.isfinite(value)]
|
||||||
|
median = statistics.median(finite)
|
||||||
|
mad = statistics.median(abs(value - median) for value in finite)
|
||||||
|
return median, mad
|
||||||
|
|
||||||
|
|
||||||
|
def is_extreme_colortemp(value: float | None, median: float, mad: float) -> bool:
|
||||||
|
if value is None or not math.isfinite(value) or mad == 0:
|
||||||
|
return False
|
||||||
|
return abs(value - median) / (1.4826 * mad) > 10.0
|
||||||
|
|
||||||
|
|
||||||
|
def corpus_membership(row: dict[str, Any]) -> tuple[bool, list[str]]:
|
||||||
|
reasons: list[str] = []
|
||||||
|
if "CANONICALIZATION_UNRESOLVED" in row["quality_flags"]:
|
||||||
|
reasons.append("CANONICALIZATION_UNRESOLVED")
|
||||||
|
if row["public_domain"] is not True:
|
||||||
|
reasons.append("NOT_PUBLIC_DOMAIN")
|
||||||
|
if row["rights_semantic"] not in {"PDM", "CC0"}:
|
||||||
|
reasons.append("RIGHTS_NOT_OPEN")
|
||||||
|
if row["image_bundle_status"] == "IMAGE_BUNDLE_CONFLICT":
|
||||||
|
reasons.append("IMAGE_CONFLICT")
|
||||||
|
if not row["image_bundle_complete"]:
|
||||||
|
reasons.append("NO_USABLE_IMAGE")
|
||||||
|
if not row["colors"]:
|
||||||
|
reasons.append("NO_COLORS")
|
||||||
|
return not reasons, reasons
|
||||||
|
|
||||||
|
|
@ -28,5 +28,14 @@ def milestone1(root: Path | None = None) -> None:
|
||||||
typer.echo(f"Milestone 1 complete: {summary['source_rows']}")
|
typer.echo(f"Milestone 1 complete: {summary['source_rows']}")
|
||||||
|
|
||||||
|
|
||||||
|
@app.command("milestone2")
|
||||||
|
def milestone2(root: Path | None = None) -> None:
|
||||||
|
"""Build canonical SMK entities and freeze smk-public-visual-v1."""
|
||||||
|
from smk_playground.canonical.milestone2 import run
|
||||||
|
|
||||||
|
summary = run((root or Path.cwd()).resolve())
|
||||||
|
typer.echo(f"Milestone 2 complete: {summary['canonical_entity_count']} entities")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
app()
|
app()
|
||||||
|
|
|
||||||
90
tests/test_canonical_rules.py
Normal file
90
tests/test_canonical_rules.py
Normal file
|
|
@ -0,0 +1,90 @@
|
||||||
|
from smk_playground.canonical.rules import (
|
||||||
|
corpus_membership,
|
||||||
|
dimension_slots,
|
||||||
|
image_bundle_complete,
|
||||||
|
normalize_dimension_observations,
|
||||||
|
resolve_image_bundle,
|
||||||
|
resolve_smk_entity_id,
|
||||||
|
rights_semantic,
|
||||||
|
visual_number,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def complete_bundle(image_id: str = "image") -> dict:
|
||||||
|
return {
|
||||||
|
"has_image": True,
|
||||||
|
"image_iiif_id": image_id,
|
||||||
|
"image_iiif_info": f"{image_id}/info.json",
|
||||||
|
"image_width": 100,
|
||||||
|
"image_height": 200,
|
||||||
|
"image_thumbnail": "thumb",
|
||||||
|
"image_native": "native",
|
||||||
|
"image_mime_type": "image/tiff",
|
||||||
|
"image_cropped": False,
|
||||||
|
"image_orientation": "portrait",
|
||||||
|
"image_hq": True,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_da_en_pairing_uses_smk_id_only() -> None:
|
||||||
|
assert resolve_smk_entity_id("117_object", "117_object") == "117_object"
|
||||||
|
assert resolve_smk_entity_id(None, "118_object") == "118_object"
|
||||||
|
try:
|
||||||
|
resolve_smk_entity_id("117_object", "118_object")
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
raise AssertionError("different SMK ids must not be paired")
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_bundle_is_atomic() -> None:
|
||||||
|
da = complete_bundle("da")
|
||||||
|
en = complete_bundle("en")
|
||||||
|
canonical, provenance, status = resolve_image_bundle(da, en)
|
||||||
|
assert canonical is None
|
||||||
|
assert provenance == "conflict"
|
||||||
|
assert status == "IMAGE_BUNDLE_CONFLICT"
|
||||||
|
|
||||||
|
canonical, provenance, status = resolve_image_bundle(da, {"has_image": False})
|
||||||
|
assert canonical == da
|
||||||
|
assert provenance == "da"
|
||||||
|
assert status == "IMAGE_BUNDLE_SINGLE_SOURCE"
|
||||||
|
assert image_bundle_complete(canonical)
|
||||||
|
|
||||||
|
|
||||||
|
def test_dimension_aliases_convert_to_cm_and_conflicts_stay_null() -> None:
|
||||||
|
da = {"dimensions": [{"part": "Netto", "type": "højde", "unit": "millimeter", "value": "100"}]}
|
||||||
|
en = {"dimensions": [{"part": "netto", "type": "height", "unit": "cm", "value": "11"}]}
|
||||||
|
observations = normalize_dimension_observations(da, "da", "da-id")
|
||||||
|
observations += normalize_dimension_observations(en, "en", "en-id")
|
||||||
|
slots, conflicts, candidates = dimension_slots(observations)
|
||||||
|
assert slots["net_height_cm"] is None
|
||||||
|
assert conflicts == ["net_height_cm"]
|
||||||
|
assert len(candidates) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_rights_semantics_are_language_independent() -> None:
|
||||||
|
assert rights_semantic("https://creativecommons.org/publicdomain/mark/1.0/") == "PDM"
|
||||||
|
assert rights_semantic("https://www.smk.dk/section/brug-af-museets-materiale/") == "SMK_RESTRICTED_POLICY"
|
||||||
|
assert rights_semantic("https://www.smk.dk/en/section/use-of-smk-material/") == "SMK_RESTRICTED_POLICY"
|
||||||
|
|
||||||
|
|
||||||
|
def test_infinity_colortemp_is_missing_with_explicit_status() -> None:
|
||||||
|
assert visual_number({"colortemp": "Infinity"}, "colortemp") == (None, "malformed_text")
|
||||||
|
|
||||||
|
|
||||||
|
def test_corpus_rule_returns_machine_readable_reasons() -> None:
|
||||||
|
row = {
|
||||||
|
"quality_flags": [], "public_domain": True, "rights_semantic": "PDM",
|
||||||
|
"image_bundle_status": "IMAGE_BUNDLE_OK", "image_bundle_complete": True,
|
||||||
|
"colors": ["#ffffff"],
|
||||||
|
}
|
||||||
|
assert corpus_membership(row) == (True, [])
|
||||||
|
row["colors"] = None
|
||||||
|
row["image_bundle_complete"] = False
|
||||||
|
assert corpus_membership(row) == (False, ["NO_USABLE_IMAGE", "NO_COLORS"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_rules_are_deterministic() -> None:
|
||||||
|
bundle = complete_bundle()
|
||||||
|
assert resolve_image_bundle(bundle, dict(bundle)) == resolve_image_bundle(bundle, dict(bundle))
|
||||||
Loading…
Add table
Reference in a new issue