Robots.txt Policy Controls

SEOvault AI lets you manage your robots.txt file directly from the WordPress admin. You can choose from preset crawl policies or write custom robots.txt rules. The plugin injects your policy into the virtual robots.txt output using BEGIN/END markers, so it works without writing to the filesystem.

Where to find robots.txt settings

Go to SEOvault AI > General > Robots in your WordPress admin. Scroll to the robots.txt Crawl Policy section at the bottom of the tab.

How robots.txt injection works

WordPress generates robots.txt dynamically unless you have a physical robots.txt file in your site root. SEOvault AI hooks into the robots_txt filter and appends your crawl policy to the output.

The plugin wraps your policy in markers:

# BEGIN SEOVault AI Crawler Policy
User-agent: *
Allow: /
Crawl-delay: 1
# END SEOVault AI Crawler Policy

When you change the policy, the plugin removes the old block and injects the new one. If you switch to None (default), the block is removed entirely.

Preset crawl policies

The plugin includes three preset policies that control crawl delay:

Balanced – User-agent: *, Allow: /, Crawl-delay: 1. Adds a 1-second delay between requests. Recommended for most sites.
Maximum Visibility – User-agent: *, Allow: /, Crawl-delay: 0. No crawl delay. Use this if you want search engines to crawl as fast as possible.
Strict – User-agent: *, Allow: /, Crawl-delay: 5. Adds a 5-second delay. Use this if your server has limited resources or you want to slow down aggressive crawlers.

All presets allow all user agents and allow crawling of the entire site. The only difference is the crawl delay.

Custom robots.txt rules

If the presets do not meet your needs, select Custom and write your own robots.txt rules in the textarea.

Examples of custom rules:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Crawl-delay: 2

Sitemap: https://yourdomain.com/seovault-sitemap.xml

The custom content is injected exactly as you write it. You can include multiple user-agent blocks, disallow rules, allow rules, crawl delays, and sitemap directives.

Step-by-step: configure a robots.txt policy

Go to SEOvault AI > General > Robots.
Scroll to the robots.txt Crawl Policy section.
Select a Crawl Policy:
- None (default) – no SEOvault AI policy is injected
- Balanced – 1-second crawl delay
- Maximum Visibility – no crawl delay
- Strict – 5-second crawl delay
- Custom – write your own rules
If you selected Custom, enter your robots.txt rules in the textarea.
Click Preview robots.txt to see what the full robots.txt output will look like.
Click the View your live robots.txt link to verify the live output.
Click Save Changes at the bottom of the page.

Previewing robots.txt

The Preview robots.txt button shows you the combined output of your policy and any other robots.txt content from WordPress or other plugins. The preview appears in a code block below the button.

After previewing, a link to your live robots.txt at /robots.txt appears. Click this to verify that search engines see the expected output.

Notes and edge cases

Physical robots.txt file: If you have a physical robots.txt file in your site root, WordPress does not use the virtual robots.txt and the plugin injection does not apply. Delete the physical file to use the plugin’s policy.
Autopilot mode only: The robots.txt policy is only applied when the plugin is in Autopilot output mode. In Manual or Disabled mode, the robots.txt filter is not registered.
Other plugins: If another plugin also modifies robots.txt, the order of filter execution determines which content appears first. SEOvault AI uses priority 10, which is the default.
Sitemap directive: You can include a Sitemap: directive in your custom policy to point search engines to your sitemap. For example: Sitemap: https://yourdomain.com/seovault-sitemap.xml.
Crawl delay support: Not all search engines respect the Crawl-delay directive. Bing and Yandex honor it, but Google does not. Google uses its own crawl rate based on server response time and site authority.
User-agent wildcards: The presets use User-agent: * to apply to all crawlers. In custom rules, you can target specific bots like Googlebot or Bingbot.
Allow vs Disallow: The presets use Allow: / to explicitly allow crawling. You can also use Disallow: to block specific paths. Remember that Disallow: / blocks the entire site, while Disallow: with no value allows everything.