Design Publishing Queues That Survive Image Provider Outages

Sun, 18 Oct 2026 11:52:14 +0000

A publishing queue should not forget an article because a photo API had a bad afternoon.

Quick solution:

if (providerError.isTransient) {
  return queue.defer({ state: "MEDIA_PENDING", retryAfter: "1h" });
}

The retry design must respect Cloudflare Workflow limits and the provider's Pexels API documentation; provider failure does not decide whether the article itself is valid.

Yet that is exactly what happens when a pipeline treats every missing hero image as "invalid content."

The article may be complete. The metadata may be valid. The internal links may pass. The only failure is an external media provider returning no usable result, a rate-limit response, or a temporary server error.

That is not the same class of failure as a malformed article.

The queue should preserve the article and defer the media stage.

Separate source validity from media availability

A clean pipeline asks two different questions:

Is the article publishable?
Is a valid hero asset available right now?

The first question is editorial.

The second depends partly on external systems.

If you merge them into one boolean, this happens:

article valid
photo provider unavailable
=> source rejected
=> next topic generated

Now a temporary outage has changed editorial history.

A better state model is:

VALIDATED
MEDIA_PENDING
MEDIA_SELECTED
READY_TO_PUBLISH
PUBLISHED

The article identity stays the same while the media stage retries.

Classify media failures

Not every visual failure deserves a retry.

Retryable

provider 429 rate limit;
temporary 5xx response;
network timeout;
provider returns zero acceptable results for one query;
asset download temporarily fails;
deployment check is still pending.

Deterministic repair

selected asset already used when uniqueness is required;
wrong dimensions;
missing attribution;
stale manifest hash;
unsupported media type.

Terminal for that candidate

license does not allow the intended use;
source URL cannot be established;
raster is corrupt;
asset content is clearly irrelevant;
provider explicitly rejects the request.

A terminal candidate is not necessarily a terminal article. Select another asset.

Bound provider retries

External APIs need retry budgets.

Pexels’ API documentation currently documents default API limits of 200 requests per hour and 20,000 per month. A publisher that retries without bounds can turn temporary scarcity into a quota problem. Cloudflare’s Workflow limits are a separate runtime ceiling, so keep provider quotas and orchestration limits distinct.

Use a fixed strategy such as:

query 1 -> retry transient failure up to 2 times
query 2 -> try one alternate search phrase
reviewed catalog -> use only if a genuinely relevant asset exists
otherwise -> MEDIA_PENDING

Do not keep changing the query forever inside one Workflow instance.

The purpose of retries is to absorb transient noise, not to search the entire internet before the run can finish.

Keep the source queued

When media is unavailable, persist the article in a state that makes the next run resume it.

Do not delete the source, mark it duplicate, move to a new topic, create another draft with the same intent, or reset the article ID.

A durable record needs enough identity to resume:

article_id: art_example
slug: example-article
state: MEDIA_PENDING
revision: 2
visual_attempts: 3

Whether this lives in Git, Workflow state, or another durable store depends on the architecture. The invariant is the important part: the article survives.

Reuse reviewed assets carefully

A local catalog can reduce provider dependence.

If the catalog already contains an unused or reusable raster that is genuinely relevant and its license permits reuse, it may be a better fallback than another external request.

But "available" is not enough.

Check topic relevance, provider/source URL, license, attribution, byte hash, dimensions, reuse rules, and whether the same visual is already overused.

Never reuse an unrelated image simply to make the pipeline green.

Bind provenance when selection succeeds

Once an asset is selected, record the evidence immediately.

Useful fields include:

provider: Pexels
source_url: https://www.pexels.com/photo/...
license_url: https://www.pexels.com/license/
attribution: Photo by ...
article_id: art_example

After downloading the final raster, add the MIME type, width, height, byte count, and SHA-256 of the transformed file.

The first block explains where the image came from. The second explains exactly which bytes the publication uses.

Do not invent file metadata before the file exists.

Avoid duplicate work after uncertain responses

Media providers and storage APIs can fail after doing the work.

Example:

download completes
network layer times out before caller records success
retry starts
same image downloaded again under another asset ID

Use stable stage identity.

If an asset selection has already been recorded for article A revision 3, the retry should inspect that record before searching again.

Likewise, if the final raster already exists and its hash matches the intended manifest, verify it instead of creating another copy.

This is the same idempotency principle used throughout Building a Reliable Batch Publishing Workflow.

Verify the actual raster, not provider metadata

A provider response can describe the original photograph, while your pipeline may crop or resize it.

The publication should bind to the final transformed bytes, not merely the provider metadata.

After transformation:

decode the file;
read actual dimensions;
compute byte length;
compute SHA-256;
update the asset manifest;
verify the article points to that asset.

The provider page remains provenance. The transformed file becomes publication evidence.

Keep provider outages out of editorial scoring

A photo API outage says nothing about whether an article has information gain.

Do not lower article quality, factual confidence, duplicate status, or rank-potential merely because an image API is unavailable.

Instead report the failure in the stage that owns it:

editorial: PASS
research: PASS
visual: DEFERRED_PROVIDER
publication: PENDING

This makes the blocker understandable to both humans and automation.

Verify after deployment

A queue is not finished when the image reaches Git.

After deployment, fetch the public hero URL and verify a successful response, correct content type, decodable bytes, expected dimensions, and that the expected article actually references it.

If the article is live but the asset is stale, keep the same publication pending until the intended raster is visible.

That exact-version discipline is part of Designing a Reliable Serverless AI Publishing Workflow.

Design for a provider-independent future

Do not let one provider's response schema leak across the whole publishing system.

Normalize a candidate:

type ImageCandidate = {
  provider: string;
  sourceUrl: string;
  downloadUrl: string;
  attribution: string;
  licenseUrl: string;
  width?: number;
  height?: number;
};

Then the rest of the pipeline can perform relevance checks, dedupe, transforms, manifest binding, and deployment verification without caring where the candidate came from.

That makes adding a second provider a local adapter change instead of a workflow rewrite.

A practical recovery algorithm

When an article reaches the visual stage:

1. inspect existing bound asset
2. if healthy -> continue
3. inspect reviewed catalog for relevant candidate
4. if found -> bind, transform, verify
5. otherwise query provider within request budget
6. reject duplicates or invalid provenance
7. transform final raster
8. validate bytes and dimensions
9. if provider unavailable -> MEDIA_PENDING
10. retry same article later

The crucial line is number 10.

When maintaining these pipelines, teams should also review policies for preventing search intent duplicates in automated systems to ensure safe retries.

Verification and execution checklist

Inspect the active article state to confirm MEDIA_PENDING is recorded instead of deleting the draft.
Verify that external API calls respect rate-limit thresholds and orchestration timeouts.
Confirm that transformed asset hashes match the recorded manifest before deployment.
Check public staging endpoints to ensure the final raster is accessible and correctly linked.

Coder Legion - Recent posts tagged workflow-reliability