Publishers Weigh Blocking Google as AI Search Cuts Traffic

by

Published July 30, 2029

Several large news organisations are weighing whether to block Google’s web crawlers entirely, a step that would remove them from search results in exchange for keeping their journalism out of Google’s AI systems. The Wall Street Journal reported on July 22 that Reddit, USA Today, Reuters, Politico, The Economist and People Inc are all considering some version of it. USA Today Co is already suing Google over its advertising business and says it may cut off access if referral traffic keeps falling.

The reason a threat this drastic is even on the table is technical. Google uses one crawler for two jobs: indexing pages for search and gathering material that feeds its AI features. Publishers can refuse the crawler, but until recently refusing it meant disappearing from search as well. Blocking the AI meant blocking the audience.

The numbers that changed the calculation

Semrush data cited by the Journal put year-on-year declines between June 2025 and June 2026 at close to half for USA Today, 23 percent for Politico, 25 percent for CNN and more than 85 percent for Business Insider, according to Nieman Lab’s summary of the reporting. The direction is not new. Chartbeat figures in the Reuters Institute’s annual trends report had already put global publisher search traffic down by roughly a third in the year to November 2025.

Cloudflare has now moved the deadline into the open. From September 15 the company will sort crawlers by purpose, distinguishing search from AI training and autonomous agents, and will block training and agent crawling by default on ad-supported pages for newly onboarding domains. Because Googlebot serves more than one purpose, the strictest rule a site has chosen will apply to it.

German technical coverage has been sharper than the English-language version on what this actually does. The trade site Dr. Web noted that the new defaults apply only to domains newly arriving at Cloudflare and not to existing ones, a distinction it said news coverage keeps blurring, and warned that operators who switched on a blanket AI block months ago may find they have switched off Google. That is a correction to the story rather than a detail.

Publishers are not in the same position

The industry reads as unified only from a distance. Chartbeat data reported in March showed small publishers losing far more search traffic proportionally than large ones, and the outlets weighing a full block are mostly large enough to have somewhere else to go.

Some are not losing at all. The Guardian and the BBC gained traffic over the period the Journal charted. In Nieman Lab’s quarterly figures, several American nonprofit newsrooms reported search-driven growth during the spring primaries, with the Washington cooperative The 51st rising from 120,000 site visits in May to 179,000 in June, most of it from organic search. Cofounder Maddie Poore attributed it to material that “can’t be broken down at Google’s summary length.”

That points at what the aggregate numbers hide. The losses concentrate in content that answers a question in a sentence, which is what an AI summary is built to replace. Voter guides, candidate profiles and fact-checks apparently survive contact with it.

The Economist has been discussing opting out of AI search while ruling out leaving Google altogether, on the reasoning that declining traffic is still traffic. Time has gone in a stranger direction, placing text advertisements that human readers never see but AI crawlers might pick up. Reddit, meanwhile, is reconsidering a 60 million dollar annual licensing deal with Google on the grounds that the AI features the deal enabled are cannibalising the traffic the deal was supposed to compensate for.

Google’s account of the same data

Google’s position has been consistent: search behaviour is changing, AI Overviews help people find new things, and the clicks that disappeared were low-value. The German search analyst Tom Brigl has tracked the company’s use of “bounce clicks” for those losses, meaning visits from people who only wanted to check a fact. Whether that reframing holds is exactly what publishers dispute, and the independent measurements cited above do not support it.

Regulators are splitting the crawler in two

The most consequential developments have not come from either publishers or Google. They have come from two European jurisdictions, and they are pulling in different directions.

On June 3 the UK’s Competition and Markets Authority imposed a conduct requirement on Google under the country’s digital markets regime, ordering it to let publishers withhold content from AI features while staying in ordinary search. The regulator found Google holds more than 90 percent of UK search. Main publisher controls are due by December 2026, with page-level controls by March 2027, administered through Search Console. This is the first binding instrument anywhere to separate the right to display content from the right to train on it, and Google has not extended it beyond the UK.

Germany has approached the same technology as a liability question. On May 28 the Regional Court of Munich I enjoined Google from repeating false claims its AI Overviews had made about two Munich publishing companies, treating the summaries as Google’s own statements rather than as search results, with penalties of up to 250,000 euros. The court held that generating new statements by combining third-party sources is a different act from making third-party pages findable, and that only Google can check the result. It is a first-instance decision and appealable, so its reach is uncertain.

There is a further German layer that English-language coverage has largely skipped. Under section 44b of German copyright law and the EU AI Act, AI training counts as text and data mining, which rights holders can refuse only if the refusal is machine-readable. That makes a robots.txt signal a legal reservation of rights rather than a preference, as Dr. Web put it. A configuration setting in a Cloudflare dashboard becomes evidence in a European copyright dispute.

How the story travels

Japanese coverage has treated this almost entirely as an American story, relaying the Journal’s reporting and the statements of US executives, with Nikkei’s technology arm framing the question as whether charging for crawler access can rescue media companies. The Japanese framing centres on business model adaptation. The German framing centres on rights and liability. The British framing centres on market power and remedies.

All three are describing the same crawler. What differs is which lever each media system believes is available, and that difference is largely a function of what regulators in each place have already done.

What to watch

The immediate test is whether anyone actually blocks. Threatening to leave Google is cheap and reversible; a publisher that follows through will produce the first real evidence of what search discoverability is now worth, and other newsrooms will read that result closely. The Cloudflare deadline forces a smaller version of the same decision onto every site owner using the service, most of whom are not publishers and many of whom will not notice the default until their traffic moves.

The second test is whether the UK model spreads. If Google builds an opt-out for one market, extending it is a policy choice rather than an engineering problem, and every other regulator now knows that. The European Commission, which has its own digital markets instrument, is the obvious place to look next.

Specific milestones:

  • September 15, when Cloudflare’s purpose-based crawler defaults take effect for newly onboarding domains
  • December 2026, the CMA deadline for Google’s main publisher controls, and March 2027 for page-level controls
  • Any appeal by Google against the Munich injunction, and whether a higher German court accepts the reclassification of AI summaries as the platform’s own speech
  • Whether Reddit renews or abandons its Google licensing arrangement
  • The first major publisher to actually block Googlebot, and its traffic six months later
  • Whether Google offers AI opt-out controls outside the UK without being ordered to

Sources

This post was produced with AI assistance and human editorial review. German-language and Japanese-language coverage was reviewed alongside English-language reporting, and the German material supplied the copyright and liability framing that English coverage of the same events largely omits. The Wall Street Journal’s original July 22 report sits behind a paywall and is cited here through Nieman Lab’s summary of it. The Munich ruling is a first-instance decision under appeal, and traffic estimates from Semrush, Chartbeat and Similarweb are third-party measurements rather than publisher-reported figures.