Show HN：WordPress 的 AI 爬蟲存取控制（允許、拒絕、預覽摘要）

Plugin Directory

OpenBotAuth – AI Crawler Access Control

OpenBotAuth – AI Crawler Access Control

Description

OpenBotAuth helps publishers control automated access from AI crawlers and agents. It verifies requests using RFC 9421 HTTP Message Signatures (via a configurable verifier) and applies per-site or per-post policies like allow, deny, teaser previews, and 402 payment-required responses. It also publishes AI-friendly endpoints like llms.txt, a JSON feed, and per-post Markdown.

Instead of blocking all bots or allowing unrestricted access, you can:

OpenBotAuth provides machine-readable endpoints for AI systems:

Configure which post types to include (posts, pages, or custom types) and set the feed limit (up to 500 items). All data is served locally from your WordPress database. No external tracking or telemetry. Only published, non-password-protected posts are exposed.

This plugin connects to an external verifier service. When a signed bot request is received, the plugin sends the following data to your configured verifier URL via wp_remote_post:

Privacy protection: Sensitive headers (cookies, authorization, proxy-authorization, www-authenticate) are NEVER forwarded, even if present in the request. If a bot’s signature covers a sensitive header, verification will fail with a clear error.

No WordPress user accounts or personal data is transmitted. Only the headers explicitly covered by the bot’s signature are forwarded to enable cryptographic verification. Note that the URL may include query parameters depending on your site’s structure.

You can:

• Use the hosted verifier at https://verifier.openbotauth.org/verify
• Self-host the verifier service (see documentation)
• The verifier service may log requests server-side depending on your configuration

Analytics are local-only. Decision counts (allow/teaser/deny/pay/rate_limit) and bot traffic observations (User-Agent based) are stored in your WordPress database. No analytics data is sent to external servers.

For more information, please review our Terms of Service and Privacy Policy.

Developer Hooks

openbotauth_policy
Modify policy before applying:

openbotauth_verified
Triggered when a bot is verified:

openbotauth_payment_required
Triggered when 402 is returned:

openbotauth_should_serve_llms_txt
Disable llms.txt endpoint (e.g., when using Yoast):

openbotauth_should_serve_feed
Disable JSON feed endpoint:

openbotauth_should_serve_markdown
Disable markdown endpoints:

openbotauth_feed_item
Modify feed items:

openbotauth_markdown_content
Post-process markdown output:

Installation

FAQ

Do I need to run my own verifier service?

No, you can enable the hosted verifier in Settings by checking “Use hosted OpenBotAuth verifier”. For privacy requirements or custom configurations, you can self-host the verifier service. The plugin does not contact any external service until you explicitly configure it.

Will this block normal human visitors?

No. The plugin only applies to requests that include RFC 9421 signature headers. Normal browser requests without signature headers see full content and bypass OpenBotAuth entirely.

What is a teaser?

A teaser shows the first N words of your content to unverified bots, with a notice that authenticated bots can access full content. You can configure the word count globally or per-post.

Does the 402 payment feature process actual payments?

No. The 402 response is a stub that returns the configured price and payment URL. Actual payment processing requires custom integration.

What data does the plugin send externally?

Only signature verification requests are sent to your configured verifier URL. The request includes the URL being accessed and the signature headers. No personal data, cookies, or user information is transmitted.

Does the plugin send any data to external servers?

The only external call is signature verification (if configured). All analytics are stored locally in your WordPress database. No telemetry or tracking data is sent to any external server.

Does OpenBotAuth work with Yoast SEO?

Yes. OpenBotAuth works alongside Yoast SEO without conflicts. By default, OpenBotAuth serves llms.txt (works standalone). If Yoast is installed and you’ve enabled Yoast’s llms.txt feature, use the “Use Yoast llms.txt” toggle in AI Endpoints settings to let Yoast handle it. OpenBotAuth’s unique feed and markdown endpoints remain active either way.

How is the admin interface organized?

The settings page has three tabs:

Do AI endpoints bypass membership or paywall plugins?

The AI endpoints (llms.txt, JSON feed, markdown) serve content directly on early request interception and only check if content is published and not password-protected. If you use a membership or paywall plugin that restricts content via later WordPress hooks, that content may still be accessible via AI endpoints.

To control this, you can:

Reviews

There are no reviews for this plugin.

Contributors & Developers

“OpenBotAuth – AI Crawler Access Control” is open source software. The following people have contributed to this plugin.

Translate “OpenBotAuth – AI Crawler Access Control” into your language.

Interested in development?

Browse the code, check out the SVN repository, or subscribe to the development log by RSS.

Changelog

Meta

Ratings

No reviews have been submitted yet.

Add my review

See all reviews

Contributors

Support

Got something to say? Need help?

View support forum