The Web Is Closing Its Doors to AI Agents

The pitch for personal AI agents is simple: tell it what you want, and it goes and does it on the web for you. Book the flight. Reorder the groceries. Find a table for Friday.

This week it became clear the web has a say in that. TechCrunch rounded up a growing list of sites where agents like Meta's Muse are hitting a wall: Amazon, Delta, Yelp, and others. Some of it is deliberate. Some of it is anti-bot machinery that can't tell an agent acting for a customer from a scraper. And on Tuesday, Meta and Sierra proposed a protocol meant to sort that out.

Back in June I wrote about AWS letting sites charge AI agents at the door. This is the other half of that story: what happens when the door just stays shut.

What happened

It started with Amazon. Since September 20, Muse users trying to buy on Amazon.com have seen this message: "Continued access by an unauthorized AI agent violates Amazon's Conditions of Use, to which our customers have agreed."

Then the list grew. TechCrunch's Oct 6 roundup:

• Delta says it's protecting customers from unauthorized automated activity, and that it "does not currently have a partnership or integration" with third-party agents.
• United didn't confirm a block, but pointed to terms that forbid "any robot, spider, other automatic device" without written permission.
• Yelp doesn't permit non-human traffic unless the agent's maker pays through its data licensing program.
• eBay says it targets "unauthorized agents and actions," like scraping, not all shopping agents.
• Walmart, a Muse partner, says failures on its site "weren't intentional." The likely culprit is a "press and hold" human-verification button that fails if interrupted.

That last one is my favourite detail, because it shows the real problem. Walmart wants the agent in. Its own bot defenses keep it out.

How the blocking actually works

As I understand it, there are three layers, and agents trip over all of them:

1. Terms of service. The legal layer. Many sites already ban automated access without permission. Agents didn't change the rules; they made them matter.
2. Bot management. CDNs and anti-bot services score every request: browser fingerprint, behaviour, IP reputation, challenges like that press-and-hold button. An agent driving a real browser looks a lot like a sophisticated scraper.
3. Site-level policy at the CDN. On September 15, Cloudflare split its old "Block AI Bots" setting into separate Search, Training and Agent controls. Its definition of "Agent" is user-directed traffic: chat fetchers and browser-use agents visiting pages for a human. Sites with the legacy block setting were mapped to Agent = Block on pages with ads, which is also the default for new ad-supported sites. Cloudflare's reasoning is blunt: "Ad revenue depends on a human actually seeing the page."

Some users suspect that change is behind part of the Muse trouble. Cloudflare told TechCrunch it didn't have specific data to share. I can't tell from the outside, and neither can the person whose grocery order failed. That ambiguity is the core problem: a block and a bug look identical to the user.

The fix everyone is reaching for: identity

Meta's line to TechCrunch: "Turning away a personal agent means turning away the customer behind it." That's true, and it's also exactly what a site can't verify today.

On Oct 6, Meta and Sierra announced the Personal Agent Protocol, with Genesys, Rocket, Shopify, Stripe and Walmart as partners. Based on Sierra's announcement:

• An agent starts on the company's website and discovers what it offers and how to reach it.
• It can begin as a guest, enough to check availability or a returns policy.
• For account tasks, the customer signs in, and sessions are built on OAuth, the same standard behind "Sign in with" buttons.
• "The customer is always in control, deciding whether the agent has read-only or write access." The company decides which actions agents can take, through its website, its APIs (MCP and OpenAPI are named), or its own conversational agent.

There's no spec yet. Sierra says v0.1 is coming later in October.

With my systems hat on, this is the right shape. The current approach, an agent pretending to be a person clicking a browser, is screen scraping with extra steps, and screen scraping has always been fragile and adversarial. What replaced it in every previous era was an API with delegated auth. OAuth gave us "this app may read your calendar, not send email." The same move for agents is "this agent may check my booking, not cancel it."

Why it matters if you build software

• If you build agents, plan for the door. Treat each site as a dependency that can say no at any moment. Build failure paths that tell the user why ("this site doesn't allow agents") instead of failing silently.
• If you run a site or API, you now have a policy decision. Block, charge, or allow agents that identify themselves. Doing nothing means your CDN defaults decide for you.
• Watch who's missing. The partner list doesn't include Amazon, Google, OpenAI or Anthropic. A standard with one camp is just a protocol.

Quick hits

• TypeSafe raises $870M at a $7.5B valuation (Oct 9). Bloomberg reports Andreessen Horowitz led the round for the company behind Jev, the decision model I wrote about three weeks ago, and a16z's Martin Casado joins the board. That's less than a month after a $40M seed. TypeSafe's adoption claims are its own and unverified.
• Cloudflare acquires Deno (Oct 9). The whole Deno team is joining Cloudflare. Deno says the runtime gets monthly bug-fix and security releases for about a year before development ends, Deno Deploy shuts down in six months, and JSR keeps running. If you deploy on Deno Deploy, start planning the move.
• Claude Haiku 5.5 (Oct 7). Anthropic says it costs about 75% less to run on average than Haiku 4.5, and it's the first Haiku with an adjustable effort setting. Anthropic pitches it as a subagent next to Opus and Sonnet, which is how I'd try it first.
• Python 3.15 is out (Oct 9). A week later than planned. UTF-8 is now the default encoding, which quietly changes behaviour for any code that opened files without an explicit encoding.

What I'm watching

Whether Amazon, Google or the big labs show up for the Personal Agent Protocol spec, or answer with their own.

Sources

AI agents and blocked websites

• Sierra, introducing the Personal Agent Protocol (Oct 6, 2026)
• Cloudflare, new Search, Training and Agent controls (Sep 15, 2026)
• TechCrunch, the next hurdle for AI agents: getting websites to let them in (Oct 6, 2026)
• TechCrunch, Meta's AI agent blocked from Amazon.com (Sep 21, 2026)

Quick hits

• a16z, investing in TypeSafe AI (Oct 9, 2026)
• Bloomberg Government, report on TypeSafe's $7.5B valuation (Oct 9, 2026)
• Deno, Deno is joining Cloudflare (Oct 9, 2026)
• Anthropic, Claude Haiku 5.5 (Oct 7, 2026)
• Python.org, Python 3.15.0 release (Oct 9, 2026)