Are AI crawlers reading your website? We measured it on our own site

• 22 October 2026 |
• CrazyPoint

Are AI crawlers reading your website? We measured it on our own site

Short answer: yes. AI crawlers from ChatGPT, Claude and Meta AI visit websites to answer people’s questions.

So instead of guessing, we measured. On the day our new site went live, we switched on the measurement feature of our own plugin, CrazyPoint SEO. Soon the crawlers started to arrive. This post covers who came, what they read and what it means for site owners.

516 requests in two days: who came?

Counting started on the afternoon of 21 October 2026. By the next evening, 516 requests had arrived. Most of them came from just three crawlers.

Of the remaining 26 requests, 24 came from a smaller but more valuable group: search crawlers and bots that open a page on behalf of a user. The last 2 came from the Amazon and Common Crawl crawlers.

Do all AI crawlers do the same job?

No. AI crawlers fall into three groups, and each group affects your site differently.

Training crawlers do not bring visitors directly. Still, your brand name and what you do can become part of a model’s general knowledge. The number of search crawler visits is normal for a new site, and we expect it to grow as the site gets known.

Each of those 9 user-triggered requests is the trace of a real person who asked an AI about our link.

The companies behind these crawlers explain the difference too. For example, OpenAI, Anthropic and Meta each list their crawlers and what they are for.

What did the crawlers read?

The crawlers did not wander at random. In a single day, GPTBot read all 16 of our Turkish project pages. It also went through 6 service pages and 11 English app pages. Meta’s crawler took a similar tour. On top of that, the crawlers asked for the sitemap 25 times.

We took two lessons from this. First, a clean sitemap shows crawlers the way. Second, the two-language structure works, because the English pages got as much attention as the Turkish ones.

What do 404s and redirects tell you?

Crawlers also try old addresses, not only today’s pages.

These old addresses most likely come from links to the previous site and the staging address. First, we checked our side. Then we opened all 54 addresses in the sitemap one by one, and every one of them worked. Redirects are harmless: the crawler arrives at the old address and is sent on to the new one.

So when you rebuild your site, remember to redirect important old addresses to the new pages. Otherwise crawlers and visitors land on an empty page.

Should you block AI crawlers?

It depends on what you want. Blocking each group has a different result.

Our choice is to keep them all open, because as an agency we want our name and our work to be known.

Also, some security and CDN services block these crawlers by default. That is why we suggest you check your own settings once.

What does CrazyPoint SEO do here?

Every number in this post comes from the measurement screen of the CrazyPoint SEO plugin. We use the plugin on our own site first.

For AI visibility, it brings these jobs together in one place:

Of course, no plugin can guarantee that a site will appear in AI answers. Search engines and AI tools make that call. However, removing the technical barriers is the first step on that road.

Where should you start on your own site?

You can start with these five checks:

• Look at your robots.txt file. You may be blocking a crawler by mistake.
• Check your security and CDN settings. Some services stop AI crawlers by default.
• Keep your sitemap up to date. It should list only working addresses.
• Write key information as plain text. Your services and contact details should be readable on the page.
• Measure before you decide. Seeing which crawler comes and how often makes the right choice easier.