Skip to content

ci: deploy the documentation to the gh-pages branch - #4323

Merged
ariostas merged 1 commit into
scikit-hep:mainfrom
ariostas:ariostas/deploy-docs-to-gh-pages
Sep 8, 2026
Merged

ariostas merged 1 commit into
scikit-hep:mainfrom
ariostas:ariostas/deploy-docs-to-gh-pages

Conversation

@ariostas

@ariostas ariostas commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

🤖 AI text below 🤖

The documentation site has moved off the S3 bucket and CloudFront distribution: awkward-array.org is now served by GitHub Pages from the gh-pages branch, which already holds a copy of the previous bucket contents. This replaces the deployment steps in CI accordingly.

What changed

A new deploy-to-gh-pages composite action publishes a file or a directory to a path on gh-pages. It fetches the branch shallowly, adds it as a detached worktree, replaces the target path, and commits and pushes as github-actions[bot]. An empty source deletes the target instead, and because several workflows write to the branch, a rejected push is retried up to three times, re-applying the change on the new tip each time.

The aws s3 sync/aws s3 cp and aws cloudfront create-invalidation steps are replaced by that action. The layout of the site is unchanged:

Trigger Before After
push to main s3://awkward-array.org/doc/main/ doc/main
release doc/$version/, doc/stable/, doc/switcher.json same paths
docs/switcher.json changed doc/switcher.json same path
pull request preview.awkward-array.org/PR<number> doc/pr/<number>/

The only user-visible change is the preview URL, which is now https://awkward-array.org/doc/pr/<number>/.

Two workflows are added:

  • docs-preview-cleanup.yml removes doc/pr/<number> when a pull request closes. Previews were never removed from the bucket, but every deployment now fetches this branch, and a published GitHub Pages site may be no larger than 1 GB.
  • docs-squash.yml is dispatched manually and squashes gh-pages into a single commit holding the same tree, so that superseded copies of the docs can be dropped from its history. It requires typing gh-pages in a confirmation field, and no-ops if the branch is already a single commit.

Deploy jobs now request contents: write rather than id-token: write, and the AWS_ACCOUNT_ID and AWS_DEPLOY_ROLE secrets are no longer referenced anywhere and can be removed. Because a called workflow inherits the permissions of its caller, the jobs of reusable-docs.yml that build code from the pull request are pinned to contents: read, leaving contents: write to the deploy job alone, which only runs on push and release. The PR number read from the preview artifact is now validated, since it is used as a path on a branch of this repository rather than a key in a separate bucket.

Notes for review

  • .nojekyll has already been pushed to gh-pages. GitHub Pages builds that branch with Jekyll, which excludes the _static and _images directories that Sphinx generates. The composite action keeps the file in place on every deployment.
  • Squashing makes the old objects unreachable, but GitHub reclaims the space on its own schedule, so the reported repository size will lag. It also invalidates local clones of gh-pages, which need a git reset --hard origin/gh-pages.

Previews now share an origin with the site

Previews used to be served from the separate preview.awkward-array.org bucket, and are now served from awkward-array.org/doc/pr/<number>/. Their contents come from the artifact built by the pull request, so anyone able to run CI on a pull request can serve arbitrary HTML and JavaScript from the same origin as the documentation. That was equally true of the bucket, but the origin was a different one, so this is a deliberate trade-off rather than a side effect and is worth agreeing on before merging. Modifying docs/ is enough to do it; modifying a workflow is not required.

What a pull request cannot reach is unchanged. The token of a pull request from a fork is read-only and receives no secrets, whatever permissions the pull request declares. docs-preview.yml and docs-preview-cleanup.yml run the copy of themselves on the default branch, and check out the default and base branch respectively, so the composite action they call is always the reviewed one. The target path is doc/pr/<validated integer> and the artifact is copied into it, so a preview cannot touch doc/main, doc/stable, the root of the site, or the repository.

The exposure is therefore a convincing page on a trusted domain rather than access to anything. Two settings bound it, neither of them part of this pull request:

  • Fork pull requests are approved under the first_time_contributors policy, so a contributor who has already landed a pull request runs CI, and so publishes a preview, without approval. all_external_contributors would require a maintainer to approve each run.
  • default_workflow_permissions for the repository is write. Every workflow here declares permissions explicitly, so nothing relies on it today, but a workflow added without one would get a write token by default.

@github-actions github-actions Bot added the type/ci PR title type: ci (set automatically) label Sep 3, 2026
@ariostas
ariostas force-pushed the ariostas/deploy-docs-to-gh-pages branch from 9a3b2a3 to 74e4e3b Compare September 3, 2026 18:31
@ariostas
ariostas requested a review from TaiSakuma September 3, 2026 19:03
@codecov

codecov Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.40%. Comparing base (6062e76) to head (132cff7).

Additional details and impacted files

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR4323

@ariostas

ariostas commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

(The preview url is still the old one because that workflow uses the workflow file on main)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The release version parsing can produce an empty target that risks overwriting doc/ on gh-pages, and the deploy action needs basic target-path validation to prevent path traversal deletion.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR migrates documentation deployment from S3/CloudFront to GitHub Pages served from the gh-pages branch, introducing a reusable composite action to publish (or delete) site paths and updating CI workflows to use it with appropriate GitHub token permissions.

Changes:

  • Added a deploy-to-gh-pages composite action that publishes a file/directory into a target path on gh-pages, with retry-on-push-race handling.
  • Replaced AWS-based docs deployments (S3 sync/cp + CloudFront invalidation) with GitHub Pages deployments for main, releases, and previews.
  • Added workflows to clean up PR previews on close and to optionally squash gh-pages history into a single commit.
File summaries
File Description
.github/workflows/reusable-docs.yml Switch deploy job from AWS to gh-pages publishing; constrain build jobs to contents: read.
.github/workflows/docs-version.yml Deploy docs/switcher.json to gh-pages instead of S3/CloudFront.
.github/workflows/docs-squash.yml Add a manual workflow to squash gh-pages history (force-push).
.github/workflows/docs-preview.yml Deploy PR preview artifacts to gh-pages and comment the new preview URL.
.github/workflows/docs-preview-cleanup.yml Add PR-close cleanup to remove doc/pr/<number> from gh-pages.
.github/workflows/ci.yml Update reusable docs workflow invocation to use contents: write (no AWS secrets).
.github/actions/deploy-to-gh-pages/action.yml New composite action implementing the gh-pages publish/delete operation.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/reusable-docs.yml Resolved
The site moved off the S3 bucket and CloudFront distribution to GitHub
Pages, which serves `awkward-array.org` from the `gh-pages` branch.

Add a `deploy-to-gh-pages` composite action that publishes a file or a
directory to a path on that branch through a worktree, and use it in
place of the `aws s3 sync`/`cp` and cache invalidation steps. The layout
of the site is unchanged, except that pull request previews now live at
`/doc/pr/<number>/` rather than on a separate bucket.

Previews were never removed from the bucket, but the branch is fetched
by every deployment, so remove a preview once its pull request closes,
and add a manually dispatched workflow that squashes the history of the
branch when it grows too large.

Assisted-by: claude-code:claude-opus-5
@ariostas
ariostas force-pushed the ariostas/deploy-docs-to-gh-pages branch from 74e4e3b to 132cff7 Compare September 3, 2026 20:35
@ariostas
ariostas requested a review from ianna September 4, 2026 17:50

@ianna ianna left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ariostas - thanks for taking care of it! Since I didn't attend the meeting where it was discussed I'd ask @TaiSakuma to double check if that is what we've agreed upon. Thanks!

@TaiSakuma

Copy link
Copy Markdown
Member

@ariostas - thank you for the PR.
The docs in gh-pages are now 650 MB. GitHub's hard limit is 1 GB. Each doc set is about 44 MB. So we can add only about eight more doc sets. I'm planning to release more frequently. We'll hit the limit very soon. I think we need to think about it.

@ariostas

ariostas commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

@TaiSakuma yeah, let's keep talking about it to see if there's a better alternative. There would only be a new doc set per new minor version, and I'm guessing that minor version bumps won't be as frequent. The more likely issue is with PRs, but hopefully the PR removal workflow keeps that under control.

I'll merge now so that the docs stay up to date and we can preview PRs, but let's definitely keep discussing this.

@ariostas
ariostas merged commit a33c94b into scikit-hep:main Sep 8, 2026
45 checks passed
ianna added a commit that referenced this pull request Sep 18, 2026
* ci: deploy the documentation to a Cloudflare R2 bucket

The site moves off GitHub Pages to an R2 bucket served through Cloudflare,
reached over the S3 interface, so revert the `gh-pages` deployment added
in #4323.

Add a `purge-cloudflare-cache` composite action in place of the CloudFront
invalidation, and drop the `deploy-to-gh-pages` action along with the
workflow that squashed the branch history, which only existed because every
deployment committed a full copy of the rendered site.

Pull request previews return to their own bucket at `/PR<number>`. They are
replaced in place on every push, so upload them with a `Cache-Control` that
makes the edge revalidate rather than purging the whole zone for each one;
Cloudflare only offers purge-by-prefix on Enterprise plans. Previews are
still removed once their pull request closes, since nothing else would ever
overwrite them.

R2 needs an access key rather than OIDC, so the deployment jobs take
`R2_ACCOUNT_ID`, `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY`, and the
purge takes `CLOUDFLARE_ZONE_ID` and `CLOUDFLARE_API_TOKEN`.

Assisted-by: claude-code:claude-opus-5

* fix(ci): link the deployed documentation with a trailing slash

An R2 custom domain serves objects by exact key and has no index document,
unlike the S3 website endpoint it replaces, so `/doc/main` and `/PR<number>`
are both a 404. Only the directory itself can be rewritten to the
`index.html` it holds, and that rewrite keys off the trailing slash.

Assisted-by: claude-code:claude-opus-5

* fix(ci): name the R2 buckets that hold the documentation

A bucket name is no longer the domain it is served on, since R2 bucket
names cannot contain a dot.

Assisted-by: claude-code:claude-opus-5

* fix(ci): remove stale pages when syncing the documentation

The documentation is generated with one page per callable, so a renamed or
deleted function used to leave an orphan behind that the site kept serving
and that counted against the storage of the bucket. Sync with `--delete`,
which also restores what the `gh-pages` deployment did by rewriting its
target. A preview now ends in a slash as well, so that the prefix it
deletes under cannot reach the previews of other pull requests.

Retry the cache purge, since it is idempotent and a transient failure would
otherwise fail a deployment whose upload had already succeeded, and report
the reason the API gives for a refused purge.

Assisted-by: claude-code:claude-opus-5

---------

Co-authored-by: Ianna Osborne <ianna.osborne@cern.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/ci PR title type: ci (set automatically)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants