smk-playground/START_PROMPT.md

Codex start prompt

Read AGENTS.md, README.md and every file under docs/ before changing code.

The two real SMK source snapshots are already present locally:

data/raw/smk_all_da.json
data/raw/smk_all_en.json

Start with Milestone 0 only. Do not jump directly into a guessed parser or full feature build.

Your first job is source reconnaissance:

1. verify both files exist,
2. compute file byte sizes and SHA-256 hashes,
3. inspect their actual top-level JSON/envelope structure using bounded-memory techniques,
4. determine how artwork records are reached,
5. count/summarize records without loading an entire ~900 MB file into one Python object,
6. inspect representative DA/EN records,
7. compare identity sets using object_number (including duplicates/missing IDs and DA-only/EN-only cases),
8. identify where the fields relevant to Dataset v1 actually occur,
9. write source-manifest.json and a concise source-reconnaissance report.

Important constraints:

• Treat data/raw/ as immutable. Never rewrite the source JSON files.
• Do not use live API calls to rebuild information already present locally.
• Live SMK API/enrichment calls are acceptable for a few explicit verification spot-checks only.
• Do not assume DA and EN files have the same row order or identical non-language fields.
• Do not hardcode 39,508 as an expected count. Report actual counts and later define corpus membership explicitly.
• Prefer streaming (ijson) / bounded-memory inspection; avoid json.load() on the entire source dump.
• Keep the solution small. No web UI, API server, PostgreSQL, job queue or distributed infrastructure.
• Add only focused tests. Do not repeatedly run expensive full-file/full-corpus operations while making small edits.
• Make a coherent Milestone 0 commit when the source structure is understood and documented.

STOP after Milestone 0 and report:

1. exact structure of both files,
2. record/identity counts and DA/EN overlap,
3. important field locations and discrepancies,
4. source-manifest path,
5. any ambiguities that affect Milestones 1–3,
6. the concrete parser/canonicalization approach you recommend next.

Do not implement Milestones 1–3 until the actual source structure has been established from the files.