[Artifact cache] Add S3 artifact access and shared metadata lookup #9529

Goal

Make artifacts already uploaded to S3 discoverable and downloadable through the artifact cache.

First, support local SQLite metadata lookup with verified downloads from S3. Later, introduce a shared metadata database/API.

Reuse the existing artifact production and upload process.

Phase A — Existing S3 artifacts with local SQLite

This phase can proceed before the shared metadata service is implemented.

Work

1. Define the additional metadata required for existing S3 artifacts.

   Record:

   • Artifact identity, raw CMake fingerprint and cache epoch.
   • Exact component membership for bundles.
   • Existing S3 object locations.
   • Archive formats, checksums and sizes.
   • Inventory information required for verified extraction.
   • Producing repository, workflow, run/attempt and source revisions.
   • Build validation information.

   Fingerprints must come from the same build that produced the artifacts.
   Artifacts without valid fingerprints remain unavailable for fingerprint-based reuse.

2. Implement standalone metadata publication.

   Reference the existing artifact archives instead of uploading another copy.

   Write metadata to a documented, predictable location for each production.
   Use immutable or versioned object references where available.

   Make a completed manifest available only after the corresponding build,
   artifact uploads and required verification have succeeded.

   An uploaded fingerprint alone must not make an artifact reusable.

3. Import remote metadata into local SQLite.

   Accept an explicit producing-run manifest or metadata location.

   Store full ArtifactKeys, producer information and remote object references
   without assuming artifact locations are local directories.

   Make repeated imports idempotent and preserve multiple productions per key.

   Apply eligibility checks to imported records. Metadata lookup alone does not
   prove that the referenced artifact files are present or valid.

   This phase does not require a shared database service or a global S3 key index.

4. Download and verify artifacts through the local cache client.

   Look up the current build's ArtifactKeys in local SQLite.

   Fetch matching archives from their recorded S3 locations, using the archive
   formats already produced by TheRock.

   Verify checksums, sizes, required bundle completeness and safe extraction
   before exposing restored output to a consumer.

   Missing, incompatible, incomplete or corrupt artifacts must be reported as
   unavailable for reuse.

5. Add timing and request diagnostics.

   Measure:

   • Remote metadata retrieval and local import.
   • SQLite lookup.
   • S3 request counts, latency, retries and failures.
   • Bytes transferred and total download duration.
   • Verification and extraction.
   • Overall elapsed time.

   Avoid double-counting request latency already included in download time.
   Account for parallel downloads when reporting overall duration.

Acceptance criteria

• Existing artifact archives are reused at their current S3 locations.
• Published metadata identifies the artifacts from the same producing build.
• A separate machine imports a manifest into its own SQLite database.
• That machine finds matching artifacts by ArtifactKey and downloads and verifies them.
• Repeated imports preserve correct key and producer relationships.
• Incomplete publications and invalid artifacts cannot be accepted for reuse.
• Remote fetching and verification costs are visible in timing reports.

Completing Phase A proves remote discovery, downloading and verification.
Actual use of restored outputs by a TheRock build depends on the separate
local prebuilt-integration task.

Phase B — Shared metadata database/API

Implement this after the local reuse flow and metadata contracts have been validated.

Work

• Implement the agreed shared metadata backend and API.
• Preserve batched lookup using the full ArtifactKey.
• Support multiple productions per key and policy-based candidate selection.
• Preserve existing S3 storage and remote verification behavior.
• Preserve objects-first, ready-metadata-last publication ordering.
• Handle concurrent producers, retries and repeated publication requests.
• Define failure behavior when the metadata service is unavailable.
• Reuse backend contract tests against the shared implementation.

If ClickHouse is selected, demonstrate how it satisfies the required behavior.
Do not assume SQLite transaction semantics transfer directly.

Acceptance criteria

• Separate machines publish and query metadata through the shared service.
• Readers cannot select incomplete publications.
• Concurrent publication preserves key and producer relationships.
• Retries are safe and have documented outcomes.
• Lookup remains batched and independent of GitHub history traversal.
• Downloading and verification preserve the local integrity guarantees.

Dependencies and related work

Phase A builds on the existing local SQLite cache and artifact upload process.
Its metadata contract can be developed alongside the RFC.

Phase B depends on the accepted contracts from the RFC task and evidence from
the local reuse experiments.

#9532 adds CI steps that call the metadata-publication functionality after
existing artifact uploads complete. Initially, publication is limited to
approved main push and scheduled producers.

The local prebuilt-integration task connects verified downloads to TheRock's
build mechanism so producers can actually avoid recompilation.

The performance task compares complete builds, including S3 fetching overhead,
to demonstrate actual time saved.

Scope

This issue implements standalone functionality and backend support.

CI metadata publication belongs to #9532.
CI consumer integration and stage skipping remain later work.