AI coaching bots from OpenAI, Anthropic, Amazon, and a dozen different corporations at the moment are hitting manufacturing internet servers with the identical aggression as a DDoS assault, and robots.txt isn’t stopping them. This information walks by way of how InMotion’s techniques group makes use of ModSecurity to implement per-bot price limiting on the server stage, with out chopping off your web site’s…
The Downside: AI Bots That Don’t Comply with the Guidelines
robots.txt has been the de facto settlement between web sites and internet crawlers for many years. A directive like Crawl-delay: 10 tells compliant bots to attend 10 seconds between requests. Google offers you a method to configure crawl price by way of Google Search Console. Conventional search crawlers have operated inside these boundaries lengthy sufficient that the majority sysadmins by no means thought a lot about them.
LLM coaching crawlers are a special story.
Beginning in 2024, InMotion’s techniques administration groups started seeing a sample of unusually heavy site visitors throughout shared and devoted infrastructure. The supply wasn’t a single bot operating wild. It was a number of bots, every operated by a special AI firm, concurrently crawling the identical servers with no delay between requests and no respect for Crawl-delay directives. None of them coordinated with one another. None of them wanted to. The mixed load of GPTBot, ClaudeBot, Amazonbot, and their friends hitting the identical server concurrently produces useful resource exhaustion that appears functionally an identical to an unintentional distributed denial-of-service assault.
That surprises a number of web site homeowners who assume robots.txt is binding. It isn’t. It’s a conference, and these bots aren’t observing it.
Two Choices, One Clear Tradeoff
The blunt instrument is a full block by way of .htaccess. You possibly can deny entry by Person-Agent and the bots cease hitting your server fully. Downside solved, besides it isn’t: your web site additionally disappears from AI-driven discovery techniques. For companies that need to seem in AI-generated solutions or LLM-powered search options, blocking coaching crawlers fully carries an actual long-term value.
Price limiting is the higher path. You sluggish the bots right down to a tempo your server can take up. They nonetheless index your content material. You continue to keep visibility. And when a bot refuses to respect the speed restrict you’ve set, you block that particular request relatively than the bot completely.
How ModSecurity Price Limiting Works
ModSecurity is an open-source Net Utility Firewall that operates inside Apache or Nginx, inspecting HTTP site visitors in actual time. It’s the identical instrument that blocks SQL injection makes an attempt and cross-site scripting assaults on correctly hardened servers. What makes it helpful right here is its means to trace request frequency by Person-Agent and deny requests that exceed an outlined threshold.
The strategy works in two steps:
- Establish the incoming request by Person-Agent string and increment a per-host counter.
- If that counter exceeds the allowed restrict earlier than it expires, deny the request with a 429 Too Many Requests response and set a Retry-After header.
That Retry-After header issues. It explicitly tells the bot how lengthy to attend earlier than its subsequent request. A well-behaved crawler will honor it. One which doesn’t get blocked on its subsequent try.
The ModSecurity Guidelines
Beneath are the rate-limiting guidelines InMotion Internet hosting’s techniques group developed and presently deploys. Every rule set targets a selected bot by Person-Agent and enforces a most of 1 request per 3 seconds per hostname.
GPTBot (OpenAI)
# Restrict GPTBot hits by person agent to 1 hit per 3 secondsSecRule REQUEST_HEADERS:Person-Agent "@pm GPTBot" "id:13075,phase:2,nolog,pass,setuid:%{request_headers.host}, setvar:user.ratelimit_gptbot=+1,expirevar:user.ratelimit_gptbot=3"SecRule USER:RATELIMIT_GPTBOT "@gt 1" "chain,id:13076,phase:2,deny,status:429,setenv:RATELIMITED_GPTBOT, log,msg:'RATELIMITED GPTBOT'" SecRule REQUEST_HEADERS:Person-Agent "@pm GPTBot"Header at all times set Retry-After "3" env=RATELIMITED_GPTBOTErrorDocument 429 "Too Many Requests"ClaudeBot (Anthropic)
# Restrict ClaudeBot hits by person agent to 1 hit per 3 secondsSecRule REQUEST_HEADERS:Person-Agent "@pm ClaudeBot" "id:13077,phase:2,nolog,pass,setuid:%{request_headers.host}, setvar:user.ratelimit_claudebot=+1,expirevar:user.ratelimit_claudebot=3"SecRule USER:RATELIMIT_CLAUDEBOT "@gt 1" "chain,id:13078,phase:2,deny,status:429,setenv:RATELIMITED_CLAUDEBOT, log,msg:'RATELIMITED CLAUDEBOT'" SecRule REQUEST_HEADERS:Person-Agent "@pm ClaudeBot"Header at all times set Retry-After "3" env=RATELIMITED_CLAUDEBOTErrorDocument 429 "Too Many Requests"Amazonbot
# Restrict Amazonbot hits by person agent to 1 hit per 3 secondsSecRule REQUEST_HEADERS:Person-Agent "@pm Amazonbot" "id:13079,phase:2,nolog,pass,setuid:%{request_headers.host}, setvar:user.ratelimit_amazonbot=+1,expirevar:user.ratelimit_amazonbot=3"SecRule USER:RATELIMIT_AMAZONBOT "@gt 1" "chain,id:13080,phase:2,deny,status:429,setenv:RATELIMITED_AMAZONBOT, log,msg:'RATELIMITED AMAZONBOT'" SecRule REQUEST_HEADERS:Person-Agent "@pm Amazonbot"Header at all times set Retry-After "3" env=RATELIMITED_AMAZONBOTErrorDocument 429 "Too Many Requests"Adapting the Guidelines for Different Bots
The construction is similar for each bot. So as to add protection for a brand new crawler, copy any rule set and make two adjustments:
- Exchange the Person-Agent string (e.g., GPTBot) with the brand new bot’s identifier.
- Assign distinctive id values and distinctive env variable names to keep away from conflicts with present guidelines.
The id subject have to be distinctive throughout your total ModSecurity configuration. If you happen to’re including these to an present ruleset, verify what IDs are already in use earlier than assigning new ones. Collisions trigger guidelines to fail silently.
For reference, a rising listing of recognized AI crawler Person-Agent strings contains Bytespider, CCBot, Google-Prolonged, Meta-ExternalAgent, and PerplexityBot, amongst others. The Darkish Guests undertaking maintains a fairly present catalogue of recognized AI agent identifiers.
What Occurs After You Deploy
As soon as these guidelines are lively, a bot that makes two requests to the identical hostname inside a 3-second window receives a 429 on the second request. The Retry-After: 3 header tells it to attend earlier than making an attempt once more.
From there, habits splits into two classes:
Bots that respect the header decelerate routinely. They proceed indexing your content material at a tempo your server can deal with. Assets are conserved, and your web site stays accessible to the crawlers value caring about.
Bots that ignore the header hold hitting the deny rule on each subsequent request till their inside retry logic kicks in or they transfer on. Both method, they’re consuming a fraction of the assets they might have with out price limiting in place.
You received’t repair the underlying drawback of AI corporations deploying aggressive crawlers with out consent. However you cease absorbing the price of their indexing operations in your {hardware}.
Stipulations and The place to Apply These Guidelines
These guidelines require ModSecurity to be put in and enabled in your server. On InMotion Internet hosting Devoted Servers and VPS plans, ModSecurity is obtainable by way of cPanel’s WHM interface beneath Safety Middle > ModSecurity. The principles will be added as customized guidelines by way of WHM or immediately in your server’s ModSecurity configuration listing.
If you happen to’re on a managed devoted server, InMotion Internet hosting’s Superior Product Help group can help with customized ModSecurity rule deployment. Clients with Premier Care have entry to InMotion Options for precisely this sort of customized server configuration work.
Shared internet hosting environments don’t help customized ModSecurity guidelines on the account stage. If aggressive bot site visitors is an issue on shared internet hosting, the choices are restricted to .htaccess blocks or upgrading to a VPS or devoted server the place you will have full WAF configurability.
A Word on robots.txt
None of this replaces a well-structured robots.txt file. Maintaining crawl-delay directives in place for compliant bots stays worthwhile, and explicitly itemizing AI crawlers you need to limit provides a documented sign of intent, even when some bots ignore it. The ModSecurity guidelines deal with enforcement for those that received’t self-regulate.
robots.txt for bots that respect conventions; ModSecurity price limiting for those that don’t. The 2 layers work collectively.
Abstract
AI coaching crawlers don’t observe robots.txt the way in which conventional search bots do, and the mixed load from a number of simultaneous indexing operations can degrade server efficiency for respectable site visitors. ModSecurity’s Person-Agent-based price limiting offers you server-side management over how regularly these bots can request assets, with out requiring you to dam them from indexing your web site fully.
The principles are easy to deploy, lengthen to any bot by copying the template, and supply specific signaling by way of Retry-After headers for crawlers which are able to honoring them.
If you happen to’re seeing unexplained spikes in server load or HTTP request quantity that don’t correlate with actual person site visitors, verify your entry logs for AI crawler Person-Brokers earlier than assuming you’re coping with one thing extra complicated.