Gain control over automated bot traffic

1. 1. A large portion of your website traffic can be bots. AI crawlers and other bots can now cause a significant percentage of all server requests.
   2. Not all bots are bad. Search engines and AI services such as Google, ChatGPT and Claude can contribute to findability. So it's not about blocking everything, but about making a distinction.
   3. Unwanted bots pollute data, cost capacity and money. A lot of bot traffic can cause unreliable visitor data, extra server load, higher hosting costs and in extreme cases a slower website for real visitors.
   4. Measuring comes before blocking. First, analyse server logs to see which bots are passing by, how much traffic they are causing, and what parts of the website they are loading.
   5. Bot management becomes part of website management. With robots.txt, rate limiting, caching, a WAF/CDN, and targeted blocks, you can limit unwanted traffic and keep capacity available for real visitors.

A regular visitor might look at five or ten pages during a visit. A crawler can request hundreds or thousands of URLs in the same amount of time. When multiple bots do that at the same time, the load can quickly add up.

This can have several consequences:

Five reasons why unwanted bot traffic is a problem

1. 1. Clouding of data
      Bot traffic can artificially increase visitor numbers, page views, traffic sources, and engagement. As a result, analytics no longer reliably reflect how real visitors use the website.

   2. Wrong conclusions and decisions
      Polluted data makes it more difficult to determine which content is performing well and which campaigns are delivering results. As a result, optimisations and marketing decisions can be based on incorrect assumptions.

   3. Unnecessary load on the website
      Bots use the same server capacity as real visitors. Search functions, filters, calendars and other dynamic components in particular can require a lot of computing power. With large amounts of bot traffic, this can slow down the website or make it temporarily less accessible. In some hosting environments, it can also lead to additional costs.

   4. Additional security risks
      Malicious bots automatically look for outdated software, old URLs, login pages, and other vulnerable components. They can also try to guess passwords or abuse forms.

   5. Unwanted content collection
      Crawlers can collect articles, product information, knowledge bases, and manuals. Organizations may want to decide for themselves which parties are allowed to use this valuable content, for example to train or feed AI systems.

For example, let's say a website normally processes 12,083 requests per day. Suddenly this rises to 20,833 visitors. Of these, 8,750 appear to come from AI crawlers and other spam bots. 2,083 can be identified as good spam traffic.

The server then uses a significant portion of its capacity for visitors who are unlikely to ever become customers, fill out a form, or make a purchase.

During quiet moments, this does not have to be a problem. But during peak times, these bots do compete with real visitors for the same capacity.

It is then more efficient to limit bot traffic than to just add extra server capacity and keep your data clean.

No. It is important to distinguish between different types of bots.

Google and other regular search engines are important for the findability of a website. AI platforms can also increasingly redirect visitors to websites.

That is why a selective approach is usually wiser than a general blockade.

The question then becomes: Which bots do we want to allow, which ones do we want to restrict and which ones do we want to block altogether?

The answer can vary from one organisation to another and from one website to another.

There are several levels at which bot traffic can be managed.

1. Measure and analyse first
   Before taking any measures, it is wise to analyse server logs. This can be used to determine how much bot traffic there actually is, which bots are responsible and which pages they visit.
2. Robots.txt Setup
   robots.txt can specify which parts of a website crawlers can and cannot visit. Many reliable crawlers respect these instructions. However, it is not technical security: bots can ignore the rules.
3. Limit crawl rate
   Rate limiting can be used to determine how many requests a bot is allowed to make within a certain period of time. The bot does not have to be completely blocked, but it also cannot load the server indefinitely.
4. Block known unwanted bots
   Bots that demonstrably do not add value or cause problems can be banned based on various characteristics.
5. Using a CDN or Web Application Firewall
   A CDN/WAF can recognise and stop a lot of automated traffic before it reaches the web server. This reduces the burden on the infrastructure behind the website.
6. Extra protection for vulnerable or valuable parts
   Not every URL is equally weighty. For example, search functions, filters, forms, APIs and large downloads can receive extra protection or stricter limits.
7. Optimize caching
   When many bots visit the same pages, good caching can ensure that not every request has to be processed by the application and database again.

The best approach is therefore not to simply block as many bots as possible. The goal is bot management: determining which automated traffic is valuable and how much capacity you want to spend on it.

For example, you can distinguish between:

• search engines that are important for SEO;
• AI crawlers that may provide visibility;
• AI crawlers that primarily collect content;
• unknown crawlers;
• aggressive bots that use a lot of server capacity;
• Malicious bots that look for vulnerabilities.

In this way, the website remains accessible for useful services, while unnecessary burden is limited.

AI bots are becoming an increasingly normal part of internet traffic. That doesn't mean that every website has to block them. It does mean that organisations must be increasingly aware of who is allowed to visit their website automatically and under what conditions.

By measuring bot traffic, limiting unwanted crawlers, and reserving server capacity for real visitors, a website remains fast, available, secure, and cost-efficient.

Bot management is therefore increasingly becoming a regular part of good website management