# Allow AI crawlers through to ADN (pay-per-crawl) - Agent Delivery Network | Agent Delivery Network

<!-- adn:geo-lead -->
> Pay-per-crawl only works when named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and similar) reach **ADN** on your agents hostname or gateway path. If an **upstream edge** retur

<!-- /adn:geo-lead -->

Back to Help Pay-per-crawl only works when named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and similar) reach ADN on your agents hostname or gateway path. If an upstream edge returns HTTP 403 or a challenge page first, crawlers never see HTTP 402 and cannot pay.
 ADN does not assume you use Cloudflare on your apex or www . Many publishers use Vercel, Netlify, or other DNS (for example UK2, Route 53, Cloudflare only on a subdomain). Use response headers from a probe or browser devtools to see which layer blocked the request.
## How to tell which layer blocked the request
 Request your agents hostname (for example agents.example.com ) or the URL you price, with a GPTBot user agent. Inspect response headers:
 Signal Likely layer
 cf-ray , cf-mitigated , server: cloudflare Cloudflare (customer zone or ADN SaaS hostname)
 x-vercel-id , x-vercel-mitigated , server: Vercel Vercel Firewall / Bot Protection
 x-nf-request-id , server: Netlify Netlify
 HTTP 403 with no ADN headers ( x-adn-outcome , x-adn-traffic-class ) Generic WAF or host rule
 If you see `x-adn-outcome` (including payment_required on HTTP 402), the crawler reached ADN - pay-per-crawl is working at the edge.
## Cloudflare (only if Cloudflare is in the path)
 Applies when Cloudflare proxies the hostname (orange cloud) or when traffic passes through Cloudflare Bot Management.
 1. Security → Bots (or AI Crawl Control on newer dashboards): review Block AI bots and related actions.
 2. Bot Fight Mode / Super Bot Fight Mode : may challenge or block verified AI crawlers before they reach your origin or ADN.
 3. WAF custom rule : add a Skip rule for the agents hostname (for example http.host eq "agents.example.com" ) or for verified bots you intend to charge.
 If your agents hostname is on ADN DNS connect / SaaS and headers show Cloudflare but not ADN, the block may be on ADN’s SaaS zone - contact ADN support; you should not need to change your Vercel or UK2 DNS for that layer.
## Vercel (common for sites without Cloudflare)
 If you see x-vercel-id or Vercel-branded block pages:
 1. Project → Settings → Firewall : review bot rules and IP blocks.
 2. Bot Protection / Attack Challenge Mode : allow verified AI crawlers or exclude the agents hostname path if you terminate agents traffic elsewhere.
 3. Ensure the agents hostname CNAME targets ADN (grey-cloud / DNS-only at any intermediate CDN) so Vercel is not the first hop for agents.* .
## Netlify
 If you see x-nf-request-id :
 1. Review Netlify → Site → Access & security (WAF / bot rules).
 2. Add exceptions for your agents hostname or for crawler user agents you monetise.
## Generic / other hosts
 • Pause or narrow WAF , mod_security , or rate limits on the agents hostname.
 • Allow user agents you price (GPTBot, ClaudeBot, etc.) to reach ADN without a challenge page.
 • Confirm DNS for agents.<domain> points at ADN, not directly at a host that blocks bots.
## ADN connector vs agents hostname
 • Agents hostname (recommended): one CNAME to ADN; pay-per-crawl and HTTP 402 are served by the ADN gateway.
 • Cloudflare Snippet / Next middleware : ensure skip rules forward named AI crawlers to ADN, not to a block page on the customer zone.
 After changes, re-test with GPTBot against https://agents.<your-domain>/robots.txt or a priced path. You should see ADN headers or HTTP 402 when billing is enabled.
 Ask ADN