Can AI search engines see your website? 76 Harghita company sites measured

I measured how 76 company websites in Harghita County respond when the ChatGPT, Claude or Perplexity crawler knocks. Almost nobody blocks them in robots.txt, but the server turns one of them away on every fifth site. I also cover llms.txt.

In short: Of 76 Harghita company websites, 16 (21%) refuse at least one AI crawler with a 403 error, in 15 cases even though robots.txt would allow it. Only 3 sites block AI crawlers in robots.txt. Claude's crawler is shut out on 15 sites, ChatGPT search on 2. llms.txt exists on 7 sites (9%).

AI search engines (ChatGPT, Claude, Perplexity) read websites with their own crawlers. If a site turns them away, the AI cannot say anything accurate about it, and will not recommend it. I wanted to know whether AI search engines can see company websites in Harghita County, so I measured on 31 October 2026.

What I measured

I used the sample from the September survey: the county's 50 largest companies and 50 small businesses. Of these, 76 had a working site on a distinct domain. For each I checked three things:

  • what robots.txt says about AI crawlers
  • what the server answers when the request identifies as an AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot)
  • whether there is an llms.txt file

As before, I publish no company names, only aggregate numbers.

Robots.txt allows almost everywhere

Of 76 sites, only 3 block an AI crawler in robots.txt, and just one of them shuts out ChatGPT search or another crawler that reads for answers. Ten sites have no valid robots.txt at all, which means every crawler may enter.

Looking at robots.txt alone, you would think Harghita sites are wide open to AI.

The server turns crawlers away on every fifth site

When I requested the same page as an AI crawler, 16 sites (21%) refused with a 403 error for at least one crawler. On 15 of the 16, robots.txt would have allowed it. The owner most likely does not know: the block comes from a hosting security rule or a security plugin.

By crawler:

  • ClaudeBot (Anthropic): 15 sites (20%)
  • GPTBot (OpenAI): 6 sites (8%), with 4 more slowing it down with a 429 error
  • PerplexityBot: 4 sites (5%)
  • OAI-SearchBot (ChatGPT search): 2 sites (3%)

Only 2 of the 16 sit behind Cloudflare. On the rest, the hosting itself filters. Both large companies and small businesses are among the sites that refuse (5 and 11).

Which crawler you block matters

The crawlers fall into two groups. GPTBot and ClaudeBot mainly collect text for training AI models. Blocking them can be a deliberate choice and does not hurt your search visibility. OAI-SearchBot and PerplexityBot, however, read your site so they can cite it in an answer. Block these, and ChatGPT or Perplexity cannot show you as a source.

OpenAI documents publicly what each of its crawlers is for.

Llms.txt: on 7 sites

Llms.txt is a text file in the site root that summarises for AI what the site is about and where the important content is. 7 of the 76 sites have one (9%), all real, filled-in files. Three of the seven are hotels.

My site has one too, with 271 addresses. I know of no public confirmation, though, that the major AI search engines use it. It takes fifteen minutes and does no harm, but on its own it will not make your site more visible. Turning crawlers away matters far more.

Limits of the measurement

  • I sent the requests from my own machine, identifying as an AI crawler. Real crawlers come from their provider's addresses, and some firewalls treat them differently based on address. The figure shows how a site treats a request that identifies as an AI crawler.
  • I treated a 429 error as a rate limit and re-measured with four-second pauses. The 403 results held on the second run.
  • The small-business half only includes companies with a working site, as in the September survey.

What to check on your own site

  • Open yourdomain.com/robots.txt and see whether GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot appears next to Disallow.
  • Ask your host whether AI crawler blocking or a "bad bots" filter is switched on, and which crawlers it covers.
  • If you use Cloudflare, look for the AI crawler switch in the Security settings.

On my site, all four crawlers get through on individual requests. Under rapid requests, though, my own host slows the two training crawlers (GPTBot, ClaudeBot) with a 429 error, while the search crawlers are not affected. If you do not know how yours behaves, send me the address and I will run the same measurement.

Sárosi Zoltán

Written by

Sárosi Zoltán

Founder, WordPress developer – Eagle Solutions

I have been building websites and online stores from Harghita County since 2011, for small businesses, associations and agency partners. Everything I write here comes from my own projects.

More about me →

Have a question about your own website?

Tell me in a few sentences what you need. I reply within one business day, and the first consultation is free.

Already have a website? Put its address in the message and I'll send you a free 10-point list of what I would improve.

Not sure yet what you need? Work out your price range in two minutes →

Download the 10-point website checklist (PDF) →

← Back to the blog