Autonomous AI agents are currently considered one of the hottest topics in the industry. But the open web isn't playing along everywhere: several large websites are locking these software helpers out or only letting them in under certain conditions. This matters for anyone who wants to use or build agents, because the promise "the agent handles that on the web for me" depends on whether the target sites will serve it at all.

What happened

According to The Verge, companies such as OpenAI, Meta, Google and Instinct promise that their agents will browse the internet on users' behalf and complete tasks. The examples range from booking a table at a restaurant to cleaning up an inbox to buying new shoes.

In recent weeks, however, reports of blocks have been piling up. Users of services such as Meta's Muse, xAI's Grok, OpenAI's Dots and Instinct describe their agents being stopped in their tracks on tasks involving important websites.

The report names two concrete examples:

  • Yelp told TechCrunch that the platform only allows AI agents that pay for the company's data licensing program.
  • Delta Air Lines prevents AI agents from booking tickets.

We only have an excerpt of the original text. Further details, such as which other sites are affected or what technical means are used to implement the blocks, cannot be derived from it.

Background

From search robot to action robot

Websites have long had to deal with automated access. For decades, search engines have been sending out crawlers that read and index pages. There are established rules for this: site operators can use a file called robots.txt to specify which areas automated programs may enter. The trade-off was clear for a long time: the search engine brings visitors, and the site allows itself to be read.

AI agents shift this relationship. They don't just read, they act: they fill out forms, click their way through booking flows and trigger purchases. And they do so on behalf of a person who may never visit the site themselves. This raises the question for the operator of who is actually standing at the door and what they are allowed to do there.

Why websites might slow agents down

The report does not describe the motives of the individual companies in detail, but the two examples suggest possible interests.

Data as a business foundation. A review platform like Yelp lives off its content. If an agent pulls this content and summarizes it for the user in its own interface, the visit to the site may be eliminated. The licensing program is the way to regulate and compensate access. That Yelp only admits paying agents fits this logic.

Control over sales. For an airline, it's all about bookings. Issues of customer relationships, fraud protection and liability may play a role here: who is responsible if an agent books the wrong connection? This is an interpretation, not a statement from the report. All that is certain is that Delta blocks agents from booking tickets.

Who is involved

On one side are the providers of the agents, who market their products as universal helpers for the entire web. On the other side are the operators of large websites, who decide whether and under what conditions they accept these helpers. In between are the users, who depend on both sides.

More on this topic:

What This Means

For Regular Users

Anyone who wants an agent to take work off their hands should not assume that every task will work on every site. If the agent fails at a booking, that doesn't necessarily mean the agent made a mistake. It may be a deliberate block by the site. If you need to get something important done, such as a flight, you should have a Plan B and always check the result yourself.

For Developers

Anyone building agents cannot count on free access. It is worth looking into the official interfaces and licensing models of the target sites instead of relying on merely operating the user interface. The Yelp example shows that access can be tied to conditions. There is also the question of how an agent honestly identifies itself to a website. Whether a provider does so can be decisive for acceptance.

For Small Businesses and Site Operators

Anyone who runs their own website with content, appointments, or shop functions also has to take a position. Possible questions:

  • Should agents be allowed to read content, and under what conditions?
  • Should they be able to trigger bookings or purchases, or only humans?
  • Is there a controlled route, such as an interface or a licensing model?

Those who block agents across the board could lose potential customers who delegate their shopping. Those who admit them without checks risk losing control over content and processes. A deliberate decision is better than a chance outcome.

Open Questions

The report shows a development, not a settled situation. It is unclear whether uniform rules for agents will prevail or whether each major site will set its own conditions. Both are conceivable. All that is certain is that the image of the agent navigating freely through the web is currently being shaken by the reality of some large websites.

Fre­quently asked questions

What are autonom­ous AI agents?

They are AI programs meant to act on the internet on behalf of users, for example booking a table, tidying an inbox or buying shoes. Providers such as OpenAI, Meta and Google promise such features.

Which agents are affected by the blocks?

Users of services such as Meta Muse, xAI Grok, OpenAI Dots and Instinct have reported blocks on major websites in recent weeks.

How does Yelp deal with AI agents?

According to Yelp, speaking to TechCrunch, only agents whose providers take part in the company's data licensing program and pay for it are allowed access.

What is Delta Air Lines doing?

The airline prevents AI agents from booking tickets for users.