IndexNow deployment discovery

Current implementation and verified gap

The public site already exposes robots.txt, a generated sitemap.xml, and the IndexNow key file 7c5062acbc32b8a88d2e6b627d65cbfa.txt. Before this change, no deployment workflow submitted updated URLs to an IndexNow endpoint, so participating search engines had to discover changes passively.

Goal

Notify participating search engines about every canonical URL in the generated sitemap after each successful publication commit to the site repository.

Boundaries

• The sitemap remains the source of truth for submitted URLs.
• The existing public IndexNow key and key-location file remain unchanged.
• Search notification is best-effort and must never block or roll back a successful site publication.
• The source workflow waits for the site-repository publish action, not for the downstream GitHub Pages deployment in that separate repository. The notification may therefore arrive while GitHub Pages is still processing the same content.
• Google Search Console verification and sitemap submission remain separate because Google does not consume IndexNow.

Protocol decision

The workflow uses the IndexNow batch endpoint after the site repository publish action succeeds. A Python standard-library module reads _publish/sitemap.xml, extracts unique HTTPS URLs owned by openprogram.io, and emits the documented JSON request. The request includes the host, public key, key location, and URL list. More than 10,000 URLs are split into multiple requests, matching the protocol's per-request limit.

Trust boundary and data flow

1. The trusted documentation builder generates _publish/sitemap.xml.
2. The payload builder reads only exact sitemap 0.9 loc elements and accepts only HTTPS URLs whose exact host is openprogram.io; empty and foreign-host inputs fail before network submission.
3. The workflow writes validated request batches to an ephemeral runner directory.
4. The standard-library HTTP client sends every batch to https://api.indexnow.org/indexnow with a bounded timeout and retries. A failed batch is recorded without skipping later batches; the step reports failure only after all batches have been attempted.

Alternatives considered

Failure handling

The submission step uses a bounded HTTP timeout and continue-on-error. Invalid sitemap URLs, payload construction errors, and endpoint failures fail the notification step without affecting the already completed site-repository publication. No repository secret is required because IndexNow ownership is proven by the public key file.

Compatibility and exclusions

• The implementation uses Python 3.12 standard-library modules; no package dependency is added.
• Existing sitemap, robots, Google verification, and GitHub Pages behavior are unchanged.
• Waiting for the downstream openprogram-site Pages deployment is excluded because it is a separate repository workflow and notification is non-blocking discovery metadata.
• Search ranking, crawl timing, and indexing are outside the system's control and are not acceptance criteria.

Verification

• A regression test requires publication to precede notification.
• Executable tests require the documented JSON fields, stable deduplication, 10,001-URL batching, exact sitemap tags, stale-batch cleanup, continued submission after an earlier batch fails, and rejection of HTTP or foreign-host URLs.
• The workflow regression requires the official endpoint, generated sitemap input, and non-blocking failure policy.
• The live key URL and every submitted sitemap URL use HTTPS on openprogram.io.
• A manual batch submission of the current 499 sitemap URLs returned HTTP 200 on 2026-08-13.

Implementation evidence

Reviewed implementation commit f458791e8c245e642b563a5ffc9074c9758683ab adds the standard-library payload builder, batch submission, deployment hook, and eight focused regression cases. Independent specification and quality reviews both reached PASS after their recorded repair waves. Commands and final-gate results are recorded in the task ledger.