Preserve Captions in Markdown Publishing

Sat, 17 Oct 2026 15:04:18 +0000

Markdown looks simple because it is easy to type. That simplicity can be deceptive when you build an automated publisher.

Quick solution:

def extract_hero_and_caption(ast_nodes):
    hero_index = find_hero_image_node(ast_nodes)
    if hero_index is not None and is_valid_caption_block(ast_nodes[hero_index + 1]):
        return ast_nodes[hero_index + 2:]
    return ast_nodes[hero_index + 1:]

CommonMark 0.31.2 defines link and image syntax; caption handling is an application-level convention that the transform must preserve explicitly.

A regular expression removes a hero image. Another inserts an internal link. A third strips a caption. The pipeline passes unit tests, but the rendered article suddenly contains an orphan caption or an unnatural sentence where one word has been replaced by an entire destination title.

The problem is not Markdown itself. The problem is treating structured editorial text as an unstructured string.

Two transformations deserve particular care:

extracting an image and its visible caption;
inserting contextual links without rewriting the author's voice.

Standard Markdown does not give you a universal caption node

CommonMark defines links and images, but there is no universal native image-plus-visible-caption construct.

Publishers therefore use conventions such as:

![Architecture overview](hero.jpg)

*The pipeline keeps content and deployment state separate.*

or raw HTML:

`<figure>`
  `<img src="hero.jpg" alt="Architecture overview">`
  `<figcaption>`The pipeline keeps content and deployment state separate.`</figcaption>`
`</figure>`

The first form is just an image block followed by a paragraph. Your pipeline may decide that the paragraph is a caption, but that interpretation is application-specific.

That means caption extraction needs an explicit rule.

The greedy whitespace bug

Consider this input:

![A deployment workflow](hero.jpg)

*The publication path has three independent gates.*

The article begins here with a normal paragraph.

A regex designed to remove the image may accidentally consume too much or too little whitespace.

If it removes only the image token, the caption remains at the top of the article as ordinary prose.

If it greedily absorbs multiple following paragraphs, the first real paragraph disappears too.

The dangerous pattern is matching the image, then matching arbitrary whitespace and text.

The safer rule is structural:

locate the image block;
identify exactly the next block;
classify that block as a caption only if it matches your caption convention;
remove those exact blocks;
preserve all other whitespace and prose.

Prefer an AST when transforms become important

For one tightly controlled format, a carefully tested parser can be enough.

As soon as your pipeline performs several Markdown transformations, an abstract syntax tree becomes easier to reason about than stacked regular expressions.

The conceptual flow becomes:

Markdown
  -> parse blocks/nodes
  -> identify hero image node
  -> inspect immediate next paragraph
  -> classify caption
  -> transform nodes
  -> render Markdown or HTML

Now the next paragraph is an actual node, not a substring discovered by a greedy expression.

An AST also helps distinguish links inside paragraphs, headings, code fences, image alt text, and existing link nodes.

That matters for internal-link insertion.

Contextual links should preserve an existing phrase

Imagine the article contains:

A reliable serverless publishing workflow needs explicit retry boundaries.

You want to link serverless publishing workflow to a relevant internal article.

The good result is:

A reliable [serverless publishing workflow](https://raylabs.app/articles/designing-a-reliable-serverless-ai-publishing-workflow/) needs explicit retry boundaries.

A bad automation finds only the word workflow and replaces it with the destination page title:

A reliable serverless publishing [Designing a Reliable Serverless AI Publishing Workflow](https://raylabs.app/articles/designing-a-reliable-serverless-ai-publishing-workflow/) needs explicit retry boundaries.

The URL may be correct, but the sentence no longer sounds authored.

The rule should be simple: link an existing natural phrase, and do not rewrite prose to manufacture an anchor.

Score candidate phrases instead of replacing tokens

A practical link inserter can work from phrase candidates.

For each destination:

collect phrases already present in the article;
reject phrases inside headings, code, existing links, and captions;
prefer multiword phrases with clear semantic overlap;
choose the first contextually useful occurrence;
wrap exactly that phrase;
do not alter surrounding words.

A basic score can favor three-to-six-word phrases, exact topic terms, and body paragraphs over navigation-like text.

The scoring system does not need artificial intelligence. It needs predictable editorial constraints.

Never link inside code, image syntax, or an existing link

String replacement is especially risky because the same phrase may appear in code, image alt text, headings, or a link that already exists.

A syntax-aware transform can exclude those node types.

A regex stack tends to solve this by adding exceptions until maintenance becomes harder than parsing Markdown correctly.

Link insertion should be idempotent

Run your publishing transform twice.

The second run should not:

add a second copy of the same link;
nest Markdown links;
change the anchor phrase;
reinsert a removed caption;
move the hero;
change unrelated whitespace.

Idempotency is one of the strongest tests for editorial automation because production jobs are retried.

If the same draft can be transformed twice with a different output, recovery becomes dangerous.

This is part of the same design problem discussed in Automating Medium Draft Workflows: retries should continue the same artifact rather than mutate it unpredictably.

Separate editorial transformation from privacy validation

A Markdown transformer may also need to reject private repository URLs or internal hostnames.

Keep that as a separate validator.

Caption extraction answers which blocks belong to the hero presentation. Internal-link insertion answers which existing phrase can safely become an anchor. Privacy validation answers whether any forbidden URL or identifier remains in the final document.

RayLabs treats private-link detection as its own boundary in Preventing Private Repository Links in Public Articles.

Verify the rendered result, not only the source

A Markdown file can look correct and still render incorrectly under the target parser.

Your final test should render the transformed document through the same stack used in production and inspect the result.

For caption handling, confirm that exactly one hero exists, the caption appears in the intended location, and the caption does not appear again as the first article paragraph.

For contextual links, verify that anchor text matches the intended phrase, the destination is the expected existing URL, no nested anchors exist, and code and headings remain unchanged.

Execution Checklist

[ ] Confirm caption extraction isolates hero image and caption blocks without consuming subsequent paragraphs.
[ ] Verify contextual link insertion wraps exact natural phrases without altering surrounding text.
[ ] Execute the publishing transform twice to guarantee idempotency.
[ ] Render the final output through the production parser to validate HTML structure.

Separating SEO Hypotheses From Verified Research

Sat, 03 Oct 2026 05:27:38 +0000

How should an AI content pipeline distinguish an SEO opportunity hypothesis from verified search research? When an automated content system suggests a target query, search intent, and potential content gap, those outputs represent educated editorial guesses rather than proof of a live search engine results page gap. Treating these hypotheses as factual measurements creates a risk of publishing content with unproven ranking potential.

The Problem of Unverified Opportunity Fields

Language models excel at structuring metadata, but they generate plausible text with the same confidence whether they analyze live search data or rely entirely on parametric memory. In a typical publishing workflow, a model might return a specific search query, estimated difficulty, and target audience intent. Without a strict separation between hypothesis and verification, downstream systems may treat these generated fields as validated search metrics.

Consider a pipeline scenario where a team generates an article about technical documentation standards. The model suggests the target query "ai seo research evidence validation" and rates the ranking potential as promising based on internal training patterns. If the editorial system automatically accepts this rating, the publishing record implies that a real competitor analysis occurred when no actual search results were fetched or evaluated.

Distinguishing Hypotheses From Verified Research

To maintain editorial integrity, content systems must enforce an explicit trust boundary between model generation and factual validation. This boundary relies on distinct status states for research evidence, keeping unverified ideas separate from confirmed findings.

Hypothesis State: The model proposes target queries and content gaps based on general knowledge. The research status remains pending, and rank potential stays in reserve.
Verified State: A separate, deterministic process records actual competing pages, dated observations, and concrete evidence supporting a gap.

By separating these concerns, the publishing pipeline allows editors to proceed with content creation based on editorial merit without fabricating competitive metrics.

Implementing an Evidence-State Contract

You can enforce this boundary programmatically by defining a schema that requires explicit status flags before an opportunity can be marked as verified. Below is a configuration example illustrating how to structure these states in a content metadata schema.

{
  "target_query": "ai seo research evidence validation",
  "research_status": "pending",
  "rank_potential": "reserve",
  "evidence_source": null,
  "verification_timestamp": null
}

When a model generates the initial content proposal, the schema defaults the research status to pending and rank potential to reserve. A separate worker or human reviewer must populate the evidence source and timestamp before the system promotes the record to a verified state.

Common Failure Modes in Automated SEO

Failing to separate hypotheses from verified data often leads to predictable pipeline failures. Stale search engine observations can persist indefinitely if pipelines do not expire unverified metadata. Models may also fabricate competitor names or traffic volumes when forced to fill required SEO fields without access to real-time search tools. Furthermore, user interfaces that display model-generated hypotheses alongside measured analytics can trick operators into treating guesses as verified performance metrics.

Conclusion

Preserving a clear distinction between editorial hypotheses and verified search research protects your publishing workflow from ungrounded assumptions. By enforcing explicit status transitions and keeping research states separate from generation, content teams can leverage AI assistance safely while maintaining factual accuracy. For a related implementation, see Preserving Citations In Publishing Pipelines.

Coder Legion - Recent posts tagged content-pipelines