Skip to content
Cloudflare Docs

Monitor activity

graph LR
A[Enable in<br>account settings] --> B[Set a pay per <br/>crawl price ]
B --> C[Select crawlers<br>to charge]
C --> D[Monitor<br>activity]:::highlight
D --> E[Manage<br>payouts]
classDef highlight fill:#F6821F,color:white

click A "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/enable-in-account-settings/"
click B "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/set-a-pay-per-crawl-price/"
click C "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/"
click E "/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/manage-payouts/"

After configuring pay per crawl, monitor crawler activity to understand how AI crawlers interact with your content, and track your earnings.

View crawler activity

  1. Go to AI Crawl Control.

    Go to AI Crawl Control

  2. Go to the Metrics tab to view detailed analytics.

The metrics help you understand:

  • Which crawlers are accessing your content
  • How often they are being charged
  • Request patterns and trends
  • Robots.txt violations

For detailed information about available metrics, refer to View AI Crawl Control metrics.

Additional considerations

Robots.txt management

Consider updating your robots.txt file to clearly indicate which pages should remain off-limits, even if AI crawlers are willing to pay for access.

Ongoing optimization

Do the following to ensure you are using pay per crawl most effectively:

  • Review crawler activity regularly to identify patterns
  • Adjust pricing based on demand and content value
  • Modify crawler actions (charge, allow, block) as needed
  • Monitor for any unusual or unwanted crawler behavior

Additional resources