Skip to content

tests: extend the conformance completeness test and close known gaps - #953

Merged
ndeloof merged 4 commits into
compose-spec:mainfrom
ndeloof:conformance-completeness
Oct 22, 2026
Merged

ndeloof merged 4 commits into
compose-spec:mainfrom
ndeloof:conformance-completeness

Conversation

@ndeloof

@ndeloof ndeloof commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

What this PR does, in one sentence

Makes the conformance completeness test guard the top-level elements of the schema and verify that a declared test file really uses the element, and closes six of the seven known gaps with new conformance files.

Context

loader/tests/completeness_test.go is what keeps the conformance matrix honest, but its guarantee was narrower than it looked. It only walked the service attributes, so networks, volumes, secrets, configs, models, jobs, include, name... and the properties of their structures (a network's internal, a secret's driver...) could change in the schema without anything asking where they are tested. And for the attributes it did walk, it only checked that the declared file exists: a typo, a moved file or a copy-pasted entry went unnoticed. Seven service attributes were listed as known gaps: mem_limit, mem_reservation, mem_swappiness, memswap_limit, pull_refresh_after, read_only and shm_size.

What the PR brings

  • Wider scope. The test now also covers the ten top-level elements of the schema and the properties of the structured definitions behind them (network, volume, secret, config, model, include, job, schedule, healthcheck). Each is either declared with its file or listed in a known-gaps table with the reason: services (no test of the map itself) and version (obsolete, only exercised by a concurrency regression test), plus healthcheck.disable. The check deliberately stops there: it does not descend into deeper structures such as deploy or build, and says so in its header.
  • Content check. A declared file must now use the element as a YAML key, which is how a conformance test exercises it. The check is shallow on purpose (a guard against stale declarations, not a proof that an assertion is meaningful), and documented as such.
  • Fewer gaps. Service-attribute gaps go from 7 to 1. mem_limit, mem_reservation, mem_swappiness, memswap_limit, read_only and shm_size get a file each, following the repository convention (spec link and quote, YAML-in/YAML-out, interpolation, override, invalid values; the deploy consistency rule for mem_limit/mem_reservation; the -1 and 0 values of memswap_limit). pull_refresh_after remains: it is absent from the specification and compose-go passes the string through, so there is no behavior to lock.

Only tests change: no production code is modified.

🤖 Generated with Claude Code

The completeness test only walked the service attributes, and only checked
that the declared test file exists. Two gaps in that guarantee:

- the top-level elements of the schema (services, networks, volumes,
  secrets, configs, models, jobs, include, name, version) could be added or
  removed without any test being declared for them, and so could the
  properties of the structured definitions behind them (network, volume,
  secret, config, model, include, job, schedule, healthcheck);
- a declaration could point at a file that has nothing to do with the
  element, since only the existence of the file was checked.

The test now covers the top-level elements and the properties of those
definitions, each either declared with its file or listed in a known-gaps
table with the reason (services, version, healthcheck.disable). It also
requires the declared file to use the element as a YAML key, which catches a
typo, a moved file or a copy-pasted entry. The check is shallow on purpose
and says so; it stops at the listed definitions and does not descend into
deeper structures such as deploy or build.

Known gaps for service attributes become a map carrying the reason, as the
other tables do.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
mem_limit, mem_reservation, mem_swappiness and memswap_limit had no
conformance file of their own: they were only exercised together in a
catch-all test.

Each now has its file, with the spec section it locks. The byte-valued ones
(mem_limit, mem_reservation, memswap_limit) check every notation of the
specification's byte values, interpolation, override by a later file and
rejection of invalid values; mem_limit and mem_reservation also check that
they must agree with deploy.resources.limits.memory and
deploy.resources.reservations.memory. memswap_limit locks the two special
values of the specification, -1 for unlimited swap and 0 which is treated as
unset. mem_swappiness locks the percentage notations, and states in its
header what is deliberately not locked.

The vocabulary shared by these files (the byte value notations, a one-service
document builder, a load with the consistency checks enabled) is added to the
test helpers.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
read_only and shm_size had no conformance file of their own.

read_only locks the boolean and its string form, interpolation, override by
a later file in both directions, and the rejection of values that are not
booleans. shm_size locks every notation of the specification's byte values,
interpolation, override and rejection of invalid values.

pull_refresh_after stays the only service attribute without a conformance
test: the specification does not describe it and compose-go passes the string
through without interpreting it, so there is no behavior to lock yet. The
known-gaps table now says so.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
@ndeloof
ndeloof requested a review from glours as a code owner October 22, 2026 15:33
TESTING.md still said the completeness test only walks the service
attributes.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
@ndeloof
ndeloof merged commit 4b8c5f0 into compose-spec:main Oct 22, 2026
12 checks passed
@ndeloof
ndeloof deleted the conformance-completeness branch October 22, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant