Big Technology Podcast
Big Technology Podcast

Sep 2, 2026 · 56 min

AI agents threaten the economics of the open web

Cloudflare CEO: We're Ready To Block Millions of Websites From AI — With Matthew Prince

As AI systems absorb and mediate online information, publishers face a choice between blocking access and surrendering the traffic that funds them.

3 key takeaways
  1. 1AI-generated traffic already exceeds human browsing and could multiply as agents handle more knowledge work and commerce.
  2. 2Cloudflare is building controls that let publishers restrict AI crawlers, but compensation systems and business models remain unsettled.
  3. 3Matthew Prince argues Google should pay for content access as traditional SEO and link-based discovery lose influence.

Don't miss

Matthew Prince predicts that millions of websites could disappear from Google’s radar unless AI companies and publishers reach a new economic arrangement.

The brief

Cloudflare CEO Matthew Prince says automated AI traffic has already overtaken human browsing, challenging the assumption that people remain the web’s primary users.

The growth comes from model training, new AI providers, and agents interacting with sites; Prince expects automated requests to expand dramatically as agents handle work and purchases.

That traffic threatens publishers because AI systems can extract reporting and information without sending readers back, weakening advertising, subscriptions, and the incentive to create.

Cloudflare plans tools and defaults that let ad-supported and subscription sites control AI access, while the industry still lacks a workable micropayment system.

Prince says Google should face the same access rules as other AI companies and predicts traditional SEO and ten-blue-link search will fade.

The episode’s sharpest warning is that millions of sites could vanish from Google’s radar unless AI access produces a sustainable market for content.

What was said on this episode

47 statements · 19 positive · 22 negative · 6 neutral

  1. Matthew Princeon Automated web trafficPositive3:52

    Automated web traffic will reach 1,000 times human traffic within five years

    “if you just fast forward five years from now, we estimate now that automated traffic online will be a thousand times what human traffic is”

    Listen at 3:52

  2. Matthew Princeon Internet infrastructureNegative4:14

    Rising automated traffic will place enormous strain on internet infrastructure

    “The internet is going to put enormous strain on the infrastructure.”

    Listen at 4:14

  3. The internet is currently improving despite increased AI-generated content

    “I actually think the internet's getting better right now.”

    Listen at 6:23

  4. The internet is currently improving despite increased AI-generated content

    “I actually think the internet's getting better right now”

    Listen at 6:23

  5. Matthew Princeon AI-assisted online creationPositive7:36

    AI has increased the number of valuable things available online overall

    “I think that the net has been that we're getting more things out there”

    Listen at 7:36

  6. Matthew Princeon App-store distribution modelNegative7:40

    The internet will increasingly move away from the conventional app-store model

    “I do think that we're going to move more and more away from kind of an app store model”

    Listen at 7:40

  7. Matthew Princeon AI-assisted human creativityPositive8:52

    AI assistance is making humans more creative than before

    “I think it's actually making it so that humans assisted with AI are more creative than they've ever been.”

    Listen at 8:52

  8. Matthew Princeon AI-assisted human creativityPositive8:54

    AI assistance is making humans more creative than before

    “humans assisted with AI are more creative than they've ever been”

    Listen at 8:54

  9. AI will enable more people to create and pursue interesting projects

    “ai is no different it's going to enable more people to be able to be creative and do interesting things”

    Listen at 10:07

  10. Matthew Princeon Internet content creationPositive10:39

    Correct internet incentives could produce a golden age of content creation

    “if we get that right i think it's it's we're about to be on a golden age of content creation”

    Listen at 10:39

  11. Matthew Princeon Frontier AI model providersPositive12:18

    The world should have 500,000 frontier model providers

    “the world shouldn't be five, you know, frontier model providers. It should be 500,000.”

    Listen at 12:18

  12. Autonomous agent-to-agent commerce will become a large source of AI traffic

    “I think that will grow and become a pretty large category”

    Listen at 12:30

  13. Agent-to-agent commerce is currently small but expected to become significant later

    “that sort of agent-to-agent kind of commerce and things is less. That's more of a future thing than a present thing.”

    Listen at 12:34

  14. An agent shopping for a camera could visit 1,000 websites and every page

    “My agent is going to visit... thousand websites and visit every page on all of them”

    Listen at 13:10

  15. Matthew Princeon AI companies using web contentNeutral13:48

    AI companies must return value when using websites and creators' resources

    “there's got to be some exchange of value”

    Listen at 13:48

  16. Matthew Princeon Internet infrastructure capacityNegative15:41

    Internet infrastructure has finite capacity and can be overwhelmed by traffic growth

    “there's a finite capacity to all this stuff”

    Listen at 15:41

  17. Matthew Princeon Internet capacityNegative15:46

    A thousandfold traffic increase would fundamentally change internet capacity needs

    “at a thousand x it's going to be completely different”

    Listen at 15:46

  18. Matthew Princeon WordPress hostingNegative16:26

    A thousandfold traffic increase could raise WordPress hosting costs above $3,000 per site

    “if traffic went up a thousand X, what would happen? And they're like, well, then it would cost us over $3,000 to host that website.”

    Listen at 16:26

  19. Matthew Princeon Web infrastructure technologiesNeutral16:40

    Web technologies must change to support projected AI traffic

    “we've got to change some of these fundamental technologies”

    Listen at 16:40

  20. One agent per knowledge worker would require over 40 times current annual CPU production

    “you would need more than 40 times the current CPU production in a calendar year in order to support that level”

    Listen at 18:39

  21. Matthew Princeon Lightweight execution environmentsPositive19:33

    Lightweight execution technologies will be necessary to scale agent traffic

    “we think that those are the sorts of things that the internet's going to have to have if it's going to be able to continue to scale”

    Listen at 19:33

  22. Matthew Princeon Automated web trafficPositive20:35

    The 1,000-fold automated traffic forecast may underestimate five-year growth

    “a thousand X is not in five years is not only inconceivable, but it might actually be a low estimate”

    Listen at 20:35

  23. Matthew Princeon Automated web trafficPositive20:39

    The 1,000-times automated-traffic forecast may be an underestimate

    “it might actually be a low estimate”

    Listen at 20:39

  24. Matthew Princeon Ad- and subscription-supported publishersNeutral22:50

    Ad- and subscription-supported publishers should control access unless AI companies compensate them

    “those publishers want to have the controls to say you can only get my content if you compensate me for it”

    Listen at 22:50

  25. Matthew Princeon Publisher crawl-to-click ratiosNegative24:24

    Publisher crawl-to-click ratios have approximately doubled over the past year

    “everything has gotten basically, assuming you're a publisher, from the publisher's perspective, it's gotten... twice as bad”

    Listen at 24:24

  26. Matthew Princeon Publisher crawl-to-click ratiosNegative24:28

    Publisher crawl-to-click ratios have doubled over the past year

    “it's gotten... twice as bad. They just double every one of those numbers over the last year.”

    Listen at 24:28

  27. Matthew Princeon Web business modelNegative25:26

    Advertising and subscriptions will not support the web over the next 30 years

    “the business model, which has supported the web for the last 30 years, which has been advertising and subscription based, isn't going to be the business model that supports it for the next 30 years”

    Listen at 25:26

  28. Matthew Princeon Agent micropaymentsPositive26:31

    One to ten percent of Cloudflare transactions could initially monetize through agent micropayments

    “we think that somewhere between 1% and 10% of those will, day one, be monetizable through agents with a micropayments transaction”

    Listen at 26:31

  29. Matthew Princeon Agent micropayments infrastructurePositive26:42

    Agent micropayments infrastructure must scale beyond 100 million transactions per second

    “you've got to build a payments infrastructure that can day one support somewhere around 10 and then be able to scale to over 100 million financial transactions per second”

    Listen at 26:42

  30. Matthew Princeon Google web crawlingPositive28:37

    Google crawls over twice OpenAI's web volume and more than four times Anthropic's

    “Google, to give you some sense, he's more than two times as much of the web as OpenAI, more than four times as much as Anthropic, more than five times as much as Microsoft.”

    Listen at 28:37

  31. Cloudflare will treat Google like other AI crawlers by mid-September

    “By mid-September, we're going to treat Google the same way”

    Listen at 31:13

  32. Matthew Princeon Google ten-blue-link searchNegative32:07

    Traditional ten-blue-link search results will disappear

    “10 blue links are going away.”

    Listen at 32:07

  33. Matthew Princeon Google ten-blue-links search interfaceNegative32:07

    Traditional ten-blue-links search results will disappear

    “10 blue links are going away”

    Listen at 32:07

  34. Matthew Princeon Ad- and subscription-supported businessesNegative35:46

    AI search poses an existential threat to ad- and subscription-supported businesses

    “if you're app supported or subscription supported this is an existential threat to your business”

    Listen at 35:46

  35. Matthew Princeon Content creatorsPositive36:47

    Content creators will begin receiving payment in 2026

    “2026 is going to be the year that in the media space in the content space that you're going to see a real shift and then content creators are going to get paid”

    Listen at 36:47

  36. Matthew Princeon Google payments for publisher contentPositive37:37

    Google will eventually pay for publisher content

    “I think it's inevitable that they'll pay up”

    Listen at 37:37

  37. Matthew Princeon CloudflareNegative38:15

    Millions of ad- and subscription-supported sites will leave Google's index

    “we expect that the vast, vast majority of people that have ad or subscription supported businesses will do this. So there will be millions of sites that will drop off of Google's radar.”

    Listen at 38:15

  38. Matthew Princeon Google search visibility for websitesNegative38:21

    Millions of sites will disappear from Google’s radar

    “there will be millions of sites that will drop off of Google's radar”

    Listen at 38:21

  39. Matthew Princeon GoogleNegative39:13

    Google is using its existing monopoly to build a future monopoly

    “Google is leveraging their monopoly of yesterday in order to generate a monopoly tomorrow.”

    Listen at 39:13

  40. Matthew Princeon Content-creation incentivesNegative39:26

    Without correcting Google’s market power, incentives for content creation will disappear

    “the incentives for content creation just go away”

    Listen at 39:26

  41. The AI industry should contain 500,000 companies rather than five

    “The world that I'm trying to play for is one where there's not five AI companies, there's 500,000.”

    Listen at 44:03

  42. Agentic commerce could cause major-company consolidation if new entrants cannot enter easily

    “if we can't be in a place where new entrants can easily enter the market then what happens over time is that the big companies will acquire each other will have massive consolidation”

    Listen at 47:21

  43. Agentic commerce could consolidate markets in favor of large companies

    “agentic commerce becomes this massive tool of consolidation and that's going to favor the big companies over the little companies”

    Listen at 48:15

  44. Without intervention, agentic commerce will drive market consolidation

    “left to its own devices agentic commerce is going to be a consolidating factor”

    Listen at 51:30

  45. AI could substantially widen the gap between rich and poor

    “AI is this massive divider between the rich and the poor.”

    Listen at 53:51

  46. AI could substantially widen the gap between rich and poor

    “AI is this massive divider between the rich and the poor”

    Listen at 53:51

  47. Matthew Princeon AI agents and global inequalityNegative54:08

    Wealthier populations will gain major advantages from AI agents

    “the wealthy parts of the world. are going to have just huge advantages over the poorer parts of the world.”

    Listen at 54:08

Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.

Weekly email

State of the Pod, every Monday

The week's most-discussed names and stories from podcasts, with the moments behind them. Free, five minutes, unsubscribe in one click.

Listen to the full episode and explore every guest, topic, and moment on PodLume.