AI Crawler Logging: See Which Bots Visit Your Site

AI Crawler Logging tracks visits from AI models, search engines, and other bots so you can see who is accessing your content. The logs show which crawlers visited, when they visited, and what pages they requested — helping you understand AI and search engine interest in your site.

Where to find it

Go to SEOvault AI > General > llms.txt and look for the “Crawler Logging” section.

How it works

When enabled, crawler logging captures requests from known bots and crawlers. The plugin detects the crawler type from the user agent string and logs the visit with a timestamp, requested URL, and HTTP method. For privacy, IP addresses and user agents are hashed before storage — only the detected crawler label (like “Google” or “ChatGPT”) is stored in plain text.

Enable crawler logging

Go to SEOvault AI > General > llms.txt
In the “Crawler Logging” section, check Enable Crawler Logging
Click Save Changes

The plugin now logs crawler visits. Logs are stored in WordPress options and can be viewed in the admin interface or via the REST API.

Detected crawlers

The plugin detects common crawlers by their user agent strings:

Google — Googlebot
Bing — Bingbot
Yahoo — Slurp
DuckDuckGo — DuckDuckBot
Baidu — BaiduSpider
Yandex — YandexBot
Facebook — FacebookExternalHit
Twitter — TwitterBot
LinkedIn — LinkedInBot
Pinterest — Pinterest
Apple — AppleBot
SEMrush — SemrushBot
Ahrefs — AhrefsBot
Majestic — MJ12bot

Crawlers that don’t match known patterns are labeled as “Unknown”.

Log data

Each log entry includes:

Timestamp — When the crawler visited
Crawler — Detected crawler name (Google, Bing, etc.)
URL — The page requested
Method — HTTP method (GET, POST, etc.)
User Agent — Hashed user agent string (privacy-safe)
IP — Hashed IP address (privacy-safe)

IP addresses and user agents are hashed using SHA-256 before storage, matching the privacy handling used in the API audit log. This means you can see that a crawler visited without storing the actual IP or raw user agent.

View crawler logs

Crawler logs are displayed in the admin interface with:

Log count — Total number of logged visits
Recent visits — Most recent crawler activity
Crawler breakdown — Which crawlers are visiting most frequently

Logs are shown newest first and automatically trimmed to a maximum of 1,000 entries to prevent storage bloat.

Clear logs

To clear all crawler logs:

Go to SEOvault AI > General > llms.txt
In the “Crawler Logging” section, click Clear Logs
Confirm the action

All log entries are deleted. New visits will be logged if logging remains enabled.

Disable crawler logging

To stop logging crawler visits:

Go to SEOvault AI > General > llms.txt
Uncheck Enable Crawler Logging
Click Save Changes

Existing logs are preserved but no new visits are recorded.

Privacy considerations

Crawler logging is designed with privacy in mind:

IP addresses are hashed before storage
User agents are hashed before storage
Only the detected crawler label is stored in plain text
Logs are automatically trimmed to 1,000 entries
Logs can be cleared at any time

This approach lets you track crawler activity without storing identifying information about individual requests.

Use cases

Use crawler logging to:

Monitor AI interest — See which AI crawlers are visiting your site
Track search engine crawling — Verify Google, Bing, and other search engines are accessing your content
Debug indexing issues — Check if crawlers are requesting specific pages
Measure AI visibility — Track how often AI systems access your llms.txt or other AI-focused content
Identify unknown bots — Spot unfamiliar crawlers that might be scraping your content

API access

Crawler logs are also available via the REST API at seovaultai/v1/crawler-logs for integration with external tools or custom dashboards. The API requires site key authentication.