Conformance
FerroEHR makes a measured claim: it is an openEHR-spec-conformant Clinical Data Repository, and that claim is backed by a test run you can reproduce. This chapter explains what conformance means here, how to run the suite against FerroEHR, and how to read the artefacts it produces.
- What is measured
- The current result
- The party set
- Running the suite yourself
- Reading the artefacts
- What conformance does not cover
What is measured
Conformance is checked by Veredictum, a separate CNF 2.0 reference runner. It lives in its own repository, has the same maintainer as FerroEHR, and is consumed here at a pinned version. Its catalogue is a set of Arazzo workflows, one document per behaviour, grouped in capability folders. Every workflow cites the specification sentence its expectation rests on, quoted from the vendored openEHR text. Where the release leaves a behaviour to the implementation, the catalogue names an option with its arms and the sentence that leaves it open, and the party statement selects one arm. Where the release is silent or contradicts itself, no workflow asserts the behaviour, and the citation names the outbound report instead: a specification silence is never resolved privately. Every expectation traces to specification text, never to any server’s observed behaviour. The catalogue groups its folders into chapters:
| Chapter | Scope |
|---|---|
| EHR | EHR, EHR_STATUS, COMPOSITION, DIRECTORY, CONTRIBUTION and item tags |
| Definitions | ADL 1.4 and ADL 2 templates and stored queries |
| Query | AQL query execution with committed result-set grounds |
| Demographic | Parties and their versions |
| Content validation | Reference-Model and archetype-constraint accept and reject tables |
| Simplified formats | FLAT and STRUCTURED commit and read |
| Admin | The administrative delete operations |
| Security and privacy | Authenticated access and authorization separation |
| SMART on openEHR | SMART App Launch discovery and resource-scope enforcement |
| System | The OPTIONS capability and conformance manifest |
A run judges every row of every case and counts it in one of two parts. BASE holds what every server that claims a capability must do. OPTIONS holds the cases of an option the specification leaves to the implementation, where the party statement selects one arm, and the cases that rest on TRIAL text. A row of an arm FerroEHR did not select is skipped with the citation that makes it not apply, so a skipped row is never a silent pass.
Each row ends in one of five outcomes. A failed row counts against the
server. A warning row is a SHOULD the server did not follow and never
counts against it. An errored row could not be judged: it counts against
the server unless its record names the instrument as the cause. A skipped
row was not run, and names its reason. A released run also skips every case
that rests on development text alone (development-only), and a development
run skips every case whose cited text the development snapshot drops or
contradicts (cited-text-absent, contested-in-development).
Note
The runner publishes a JSON Schema for every artefact family it writes, so a consumer can validate a record without trusting the tool that produced it. The instrument is built from the currently pinned specifications (AQL 1.1.0, Terminology 3.1.0, ITS-REST 1.1.0, and the Reference Model generation the system under test runs: for FerroEHR’s development run the default
developmentprofile, RM 1.2.0, and for its released run thestableprofile; the composed stack takes the profile from configuration, so astable-profile run is the same catalogue against the released generation).
The current result
Every run on main draws its own badges and pushes them to a branch of the
repository, so these show the latest run, the same badges the README carries.
The released profile reads every citation against the frozen openEHR release
and is the record. The server it judges runs spec_profile = "stable", the
generation set that release defines; the development run keeps the default
development profile:
The development profile reads the same catalogue against the pinned master
of each openEHR repository, the text FerroEHR builds on. It is shown beside the
record and never counted toward it:
The committed record, docs/conformance/ferroehr/results.json, holds one
released-profile run. Its counts:
The record of 2026-11-14: FerroEHR 5.6.0-rc10, judged by veredictum 0.4.1 under the
released profile.
| Part | Passed | Failed | Warning | Errored | Skipped | Total |
|---|---|---|---|---|---|---|
| BASE | 2619 | 0 | 0 | 0 | 81 | 2700 |
| OPTIONS | 256 | 0 | 0 | 0 | 118 | 374 |
| Every row | 2875 | 0 | 0 | 0 | 199 | 3074 |
summary.md
beside it breaks the same run down by chapter, band and section, and lists
every skipped row with its reason.
The party set
The runner is a separate project with its own release line, so the instrument
and the system it judges do not share a build. It assesses any openEHR CDR
reachable over HTTP; this repository runs it against FerroEHR alone. What it
reads about FerroEHR is the party set under docs/conformance/party/ferroehr/:
statement.jsonclaims the capability folders of the catalogue FerroEHR serves and selects one arm of every option the catalogue leaves to the implementation (canonical XML on each resource, the Simplified Formats, version signing, the administrative delete, and the others). Each arm states what the server does, andveredictum validate --statementholds the file to the published statement schema and the catalogue before a run.ixit.jsondescribes the deployment: thesystem_idthe server stamps into commit audits and the version ids it mints, theenvironmenta measured run happened on, andexclusive_server, which lets the workflows that delete every EHR run.deviations.jsonlists the rows FerroEHR fails on purpose, each with the clause it breaks, the decision behind it and the upstream report (see Known deviations).
Credentials are in none of these files. The run reads them from the environment.
Known deviations
Veredictum offers no way to declare a deviation: a row that breaks a MUST is
a failed row, and it stays failed in the results, the badges and the committed
record. FerroEHR carries one deviation on purpose. The server writes the
committer of every version from the authenticated, bound caller, and refuses
with 403 a stated committer that names someone else (RM Common
AUDIT_DETAILS.committer, “Identity […] of user who committed the item”).
ITS-REST asks that whatever committer a client supplies “MUST be merged”
(Requests_and_responses.md §openehr-version and openehr-audit-details), so
the catalogue rows that state another committer and expect it recorded fail.
The gate of the conformance workflow reads deviations.json
(scripts/conformance-deviations.sh). A released run passes when its only red
rows are the listed ones, and fails on any other red row, on a listed row that
passes, and on a listed case the run does not hold, so the list only shrinks.
The job summary prints each listed row with its status and grounds. The four
rows are the three that state another committer
(contribution_create/commit_contribution-committer_related,
composition_delete/delete_composition-audit_details,
ehr_committal/create_ehr-committal_headers) and the refused-commit row whose
control is the first of them. The decision is
#5002, and the upstream
report is #5019.
Running the suite yourself
The suite runs against a real, deployed server (the same container image and stack a deployment uses), so the wire under test is always the production artefact. From a checkout with Docker or Podman available:
bash scripts/conformance.sh
The script reviews the statement, composes one FerroEHR deployment from the
current sources on fresh volumes, runs veredictum test through the pinned
release image with the Basic users of docker/ferroehr.dev.toml and the SMART
Platform base URL, writes results.json, recording.har and the JUnit, CTRF
and Markdown views export renders to docs/conformance/ferroehr/, and tears
the stack down. The recording holds the server’s response bodies, so it is
never committed.
CONF_PROFILE picks the specification text the case citations are read
against. released, the default, is the frozen openEHR release: it is the
record and alone backs a released claim. development is the pinned master
of each openEHR repository, the text FerroEHR builds on (RM 1.2.0, BASE 1.3.0);
its views go to docs/conformance/ferroehr-development/ and are never mixed
into the record.
Useful knobs: CONF_DOCUMENTS to narrow the run to named documents (a working
tool, never the record), CONF_OUT for a different artefact root,
CONF_NO_COMPOSE with CONF_BASE_URL to run against an already-deployed
FerroEHR, SKIP_BUILD to compose a published image instead of building from
source, and CONTAINER_ENGINE to choose between Docker and Podman.
The script also takes one stage: review runs veredictum validate --statement alone, up composes the deployment and leaves it serving, and
down removes it with its volumes. The Conformance workflow uses these stages
to run the same deployment in CI. It compiles no server of its own: it judges
the server another lane built for the commit, runs up, and drives the
catalogue through the Veredictum GitHub Action, pinned by
commit SHA, on the same release image by digest, twice: once under the
released profile and once, on fresh volumes, under the development
profile. Both runs include the SMART documents. The released run decides the
job: it fails on a failed or errored row there. Each run’s Markdown summary
goes on the job summary under its own heading, and each profile’s results,
JUnit and CTRF views are kept as an artifact of their own.
Which server a run judges depends on the event:
| Event | Server under test | Called from |
|---|---|---|
| A pull request or the merge queue, when the change touches the server, the party set, the conformance compose files or the workflow | the binary CI compiled for the change, packaged into the runtime image | the CI workflow |
A push to main | the server and database images the Containers workflow pushed for that commit, by digest (:main for an image the push did not rebuild) | the Containers workflow |
| A release tag | the release pipeline’s own images, built cold, by digest | the Release workflow |
| Twice a week, or by hand | ghcr.io/ferrohealth/ferroehr:main, or the image a manual run names | the Conformance workflow |
The README badges come from the runs on main: every push and the twice-weekly run.
The posture a run covers
The one deployment runs the SMART resource-server role with a trusted test issuer, digest version signing, and FerroTERM as its terminology server, in the fail-open posture.
- SMART is part of the posture. The tokens the SMART documents present are signed by a committed test issuer: public test key material for the harness, never usable for anything else.
- External terminology is part of the posture too. An archetype can
constrain a coded element to a value set only an external terminology query
server can resolve, so the pipeline composes a real FHIR R4B server beside the
CDR, seeded with synthetic test code systems and value sets. That covers AQL
TERMINOLOGY()resolved through the routed server, and commit-time validation of a bound value set. What a deployment does when the value set cannot be resolved at all is not decided by any openEHR text, so it is a declared option arm rather than a verdict.
A measured performance run (scripts/conformance.sh perf) changes two things,
both in docker/sut-measurement.yml: rate limiting is off, because the scout
ramp offers load past the server’s knee and a throttled request would measure
the limiter instead of the server, and response compression is off, because the
posture the run declares has it off. veredictum perf writes no record if the
server answered any 429, so a measurement can never be silently
limiter-shaped.
Reading the artefacts
A run writes results.json and three views veredictum export renders from
it: junit.xml, ctrf.json and summary.md. Each view is a pure function of
results.json, so the same results give the same bytes.
results.jsonis the record: the instrument and its version, the specification profile, the server’s ownOPTIONSanswer, an outline of every case with its chapter, band and part, and one row per workflow row with its outcome, its inputs and, for a skipped row, the reason and the citation.summary.mdis the view for a person: the counts per part, then chapter by chapter, with each red row opened up to the step, what was checked, expected against received, and the cited sentence.junit.xmlandctrf.jsonare the views a CI system shows. The Conformance workflow keeps them as run artifacts and does not commit them.
results.json and summary.md of one released-profile run are committed under
docs/conformance/ferroehr/. The website and this page read their numbers from
that record at build time, and the build fails when summary.md and
results.json disagree.
What conformance does not cover
The catalogue measures the openEHR platform surface, including the simplified (FLAT/STRUCTURED) formats chapter of the ITS-REST specification. It deliberately does not stand in for a performance benchmark: durations recorded during the functional run are telemetry only, and the measured performance record owns that claim (Performance).
The other honest boundary is the gap between openEHR’s service model and its released REST wire. Several service operations were never surfaced as endpoints (listing an EHR’s contributions, counting stored templates or queries, deleting a template or a stored query) so a case that addresses one has no wire to drive on this technology profile. FerroEHR does serve routes of its own design for several of them (see Admin & messaging APIs and the archetype routes in Templates & validation).