models/ConformanceEvidence
0 records · yaml · accepts undeclared fields
no records found at /Users/eshao/.config/mnt/data/models-catalog/conformance/*/*/*.json
Fields
| Field | Type | Required | Meaning |
|---|---|---|---|
schemaVersion |
exactly 1 | required | Evidence grammar version. |
createdAt |
string matching /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/ | required | UTC timestamp of the probe run that wrote this evidence. |
mode |
one of: probe | soft | required | probe = deterministic checks; soft = two-artifact judgement bundle. |
modality |
one of: text | image | video | music | required | Media kind exercised by the probe. |
model |
string matching /^[a-z0-9-]+\/[A-Za-z0-9][A-Za-z0-9._:-]*$/ | required | Canonical creator-vendor model id, e.g. openai/sora-2. |
family |
non-empty string | required | Evidence directory segment; sanitized model id for current-writer records, hand-named slugs on legacy ones. |
endpoint |
non-empty string | required | Model endpoint the probe dispatched through; default unless a named endpoint was probed. |
param |
non-empty string | required | Probed parameter; equals the evidence file basename. |
value |
string | optional | Probed parameter value as passed on the command line; numbers are stringified. |
prompt |
string | optional | Prompt used for the probe. Always written by the current writer; absent only on legacy records. |
runIds |
list of string matching /^\d{4}-\d{2}-\d{2}\/\d{2}_\d{2}_\d{2}_\d{3}$/ | required | query-llm run-registry ids backing this evidence. |
status |
one of: passed | failed | blocked | required | Aggregate verdict: blocked = probe could not determine an answer (quota, missing tool). |
reason |
string | optional | First failure or block reason. |
commands |
list of object (additional keys allowed) | required | Slim per-attempt execution rows; verbose stdout/stderr stays out of evidence. |
↳ runId |
string | optional | query-llm run id for this attempt. |
↳ status |
number OR exactly null | required | Process exit code, or the provider HTTP status on legacy rows; null when the attempt died before exiting. |
↳ classification |
one of: passed | failed | blocked | required | Per-attempt verdict. |
↳ reason |
string | optional | Classification detail, e.g. the provider rejection message. |
artifacts |
list of object (additional keys allowed) | required | Generated artifact references with hash proof; bytes live in the run registry, not here. |
↳ path |
non-empty string | required | Artifact path at write time; may no longer exist on disk. |
↳ exists |
boolean | required | Whether the path existed when the evidence was written. |
↳ sha256 |
string matching /^[a-f0-9]{64}$/ | optional | SHA-256 of the artifact bytes. |
↳ label |
string | optional | Role label on legacy multi-reference rows. |
↳ firstFrame |
string | optional | Extracted first-frame path on legacy video rows. |
↳ ffprobe |
any shape (not yet constrained by the schema) | optional | Legacy per-artifact ffprobe capture; flat or raw ffprobe JSON. |
↳ bytes |
number | optional | Artifact byte size on legacy rows. |
↳ dimensions |
any shape (not yet constrained by the schema) | optional | Legacy per-artifact dimension capture. |
↳ quality |
string | optional | Requested quality tier on legacy rows. |
checks |
object (additional keys allowed) | required | Deterministic check payloads keyed by check family; empty when the param has no deterministic check. |
↳ aspect |
list of object (additional keys allowed) | optional | Dimension inspections for aspect probes. |
↳ path |
non-empty string | optional | Inspected artifact path. |
↳ width |
number | optional | Pixel width. |
↳ height |
number | optional | Pixel height. |
↳ durationSeconds |
number | optional | Video duration in seconds, when the artifact is a video. |
↳ duration |
list of object (additional keys allowed) | optional | Duration inspections for duration probes. |
↳ path |
non-empty string | optional | Inspected artifact path. |
↳ width |
number | optional | Pixel width. |
↳ height |
number | optional | Pixel height. |
↳ durationSeconds |
number | optional | Video duration in seconds, when the artifact is a video. |
↳ video |
list of object (additional keys allowed) | optional | Legacy video inspections. |
↳ path |
non-empty string | optional | Inspected artifact path. |
↳ width |
number | optional | Pixel width. |
↳ height |
number | optional | Pixel height. |
↳ durationSeconds |
number | optional | Video duration in seconds, when the artifact is a video. |
↳ repeatHashes |
list of object (additional keys allowed) | optional | Artifact hashes across repeated generations. |
↳ path |
non-empty string | optional | |
↳ sha256 |
string matching /^[a-f0-9]{64}$/ | required | |
↳ allIdentical |
boolean | optional | Whether every repeated generation hashed identically (temperature probes). |
↳ sameSeedIdentical |
boolean | optional | Whether two same-seed generations hashed identically. |
↳ differentSeedDifferent |
boolean | optional | Whether a different-seed generation hashed differently. |
↳ imageInput |
object (additional keys allowed) | optional | Source-vs-output comparison produced by the deterministic SSIM/frame checks. |
↳ status |
one of: passed | blocked | candidate | required | passed/blocked from the current comparator; candidate marks a legacy result awaiting human review. |
↳ reason |
string | optional | Why the comparison could not complete. |
↳ detail |
string | optional | Tool error detail behind a blocked comparison. |
↳ source |
string | optional | Reference artifact path or URL. |
↳ output |
string | optional | Generated artifact path compared against the source. |
↳ firstFrame |
string | optional | Extracted first-frame image path (legacy frame comparisons). |
↳ lastFrame |
string | optional | Extracted last-frame image path (legacy frame comparisons). |
↳ statsPath |
string | optional | ffmpeg SSIM stats file path. |
↳ statsPreview |
list of string | optional | First lines of the SSIM stats file. |
↳ dimensions |
object (additional keys allowed) | optional | Normalized comparison dimensions. |
↳ width |
number | required | |
↳ height |
number | required | |
↳ imageFirst |
object (additional keys allowed) | optional | Source-vs-output comparison produced by the deterministic SSIM/frame checks. |
↳ status |
one of: passed | blocked | candidate | required | passed/blocked from the current comparator; candidate marks a legacy result awaiting human review. |
↳ reason |
string | optional | Why the comparison could not complete. |
↳ detail |
string | optional | Tool error detail behind a blocked comparison. |
↳ source |
string | optional | Reference artifact path or URL. |
↳ output |
string | optional | Generated artifact path compared against the source. |
↳ firstFrame |
string | optional | Extracted first-frame image path (legacy frame comparisons). |
↳ lastFrame |
string | optional | Extracted last-frame image path (legacy frame comparisons). |
↳ statsPath |
string | optional | ffmpeg SSIM stats file path. |
↳ statsPreview |
list of string | optional | First lines of the SSIM stats file. |
↳ dimensions |
object (additional keys allowed) | optional | Normalized comparison dimensions. |
↳ width |
number | required | |
↳ height |
number | required | |
↳ imageLast |
object (additional keys allowed) | optional | Source-vs-output comparison produced by the deterministic SSIM/frame checks. |
↳ status |
one of: passed | blocked | candidate | required | passed/blocked from the current comparator; candidate marks a legacy result awaiting human review. |
↳ reason |
string | optional | Why the comparison could not complete. |
↳ detail |
string | optional | Tool error detail behind a blocked comparison. |
↳ source |
string | optional | Reference artifact path or URL. |
↳ output |
string | optional | Generated artifact path compared against the source. |
↳ firstFrame |
string | optional | Extracted first-frame image path (legacy frame comparisons). |
↳ lastFrame |
string | optional | Extracted last-frame image path (legacy frame comparisons). |
↳ statsPath |
string | optional | ffmpeg SSIM stats file path. |
↳ statsPreview |
list of string | optional | First lines of the SSIM stats file. |
↳ dimensions |
object (additional keys allowed) | optional | Normalized comparison dimensions. |
↳ width |
number | required | |
↳ height |
number | required | |
↳ legacy |
object (additional keys allowed) | optional | Imported pre-harness seed evidence with its original source path. |
↳ sourcePath |
non-empty string | required | |
↳ models |
list of object (additional keys allowed) | required | |
↳ model |
non-empty string | required | |
↳ route |
non-empty string | required | |
↳ rawDir |
non-empty string | required | |
↳ sameSeedOutputs |
list of string | required | |
↳ controlOutput |
string | required | |
↳ sameSeedIdentical |
boolean | required | |
↳ controlDiffers |
boolean | required | |
↳ softJudgement |
object (additional keys allowed) | optional | Legacy soft-mode judgement summary. |
↳ expectedEffect |
non-empty string | required | |
↳ artifactCount |
integer | required | |
↳ frames |
list of string | optional | Extracted frame paths behind a legacy soft judgement. |
↳ agentJudgement |
non-empty string | optional | Legacy agent verdict on a soft bundle. |
↳ judgementReason |
non-empty string | optional | Legacy agent reasoning behind agentJudgement. |
↳ contactSheet |
non-empty string | optional | Legacy contact-sheet image path. |
↳ providerError |
object (additional keys allowed) | optional | Provider rejection captured by a legacy probe. |
↳ status |
non-empty string | required | |
↳ message |
non-empty string | required | |
↳ status |
exactly "blocked" | optional | Written only when deterministic inspection itself failed. |
↳ reason |
string | optional | Why deterministic inspection failed. |
↳ detail |
string | optional | Tool error behind a blocked inspection. |
softBundle |
object (additional keys allowed) | optional | Two-artifact judgement bundle written by soft-mode probes. |
↳ artifactA |
non-empty string | required | |
↳ artifactB |
non-empty string | required | |
↳ expectedEffect |
non-empty string | required | |
↳ judgement |
object (additional keys allowed) | required | |
↳ question |
non-empty string | required | |
↳ acceptWhen |
list of string | required | |
↳ rejectWhen |
list of string | required | |
↳ result |
non-empty string | optional | Recorded human/agent judgement outcome. |
↳ reason |
non-empty string | optional | Reasoning behind the recorded judgement. |
Rules
This schema declares no additional rules beyond its field shape.
Defects
No defects in the 0 record(s) checked.
Example record
This store has no records to show.
Where it lives
| Declared in | /Users/eshao/.config/mnt/mdr/skills/models-catalog/assets/schemas/conformance-evidence.ts |
|---|---|
| Binding | conformanceEvidenceDataStore |
| Directory | /Users/eshao/.config/mnt/data/models-catalog/conformance |
| Files | */*/*.json |