How Google forces news companies to use its crawlers

Ethics and big business go together like water and sodium. We can’t deny that, in order to ensure continued growth, a business has to pull some shady and messed-up practices, and this means that someone needs to be screwed over in the process. Enter, Google! The journalism industry is not happy with this company, and this is because Google forces news companies to use its crawlers. Why is this shady, and how does it screw over publications?

To catch you up, Google uses crawlers to extract information from different websites. This is the reason why websites show up on Google Search. However, the company has also been using its crawlers to extract data from websites to train its LLMs. At first, this was to train Bard, but this practice is now feeding Gemini.

Google eventually gave publications the ability to block the crawlers from scraping data from their sites to train Gemini. However, there was another threat looming, Google’s AI Overviews. This is the company’s tool that allows people to see AI-generated summaries of what they searched for. It basically stops traffic from reaching many of the sites that people would have otherwise gone to. We’ve already seen the results of this tactic, as numerous sites are reporting a drastic dip in traffic.

This is how Google forces news sites to use its crawlers

Companies that didn’t want their data being scraped received a saving grace in the form of Google Extended. This is a tool that blocks Google from using its crawler to scrape website data to train Gemini. There’s a catch, however. This tool can’t stop Google from accessing sites to use in AI Overviews. So, while your website data isn’t feeding Gemini, Google can still access information on your site to show in AI Overviews.

The only way to stop Google from doing this is to block Googlebot. This is the crawler that Google uses to index your website. However, if you disable that crawler, then your site won’t appear in Google Search results at all. Your site will be invisible to the majority of the world.

So, site owners are left with an impossible choice. They can either have their site reluctantly contribute to AI Overviews or be scraped from the largest search platform in the world. This isn’t the kind of choice that any business should make.

One of the biggest news sites speaks out

Obviously, this infuriates news sites around the globe, and one of them spoke out about this tactic. The director of public policy and platform strategy at The Financial Times, Matt Rogerson, wrote a letter to Baroness Stowell, chair of the House of Lords Communications and Digital Committee. This is a committee that operates in the UK. In the letter, Rogerson says that Google’s practice “leaves website owners with an unenviable choice.” It’s an ultimatum.

The letter continues, “To opt-out of the Google Search crawler entirely, and become invisible to the 90%+ of the UK population that currently uses Google Search, or allow scraping to continue in ways that both extract value without compensation, and undermine nascent commercial licensing markets for the use of high quality IP to build and enable the AI models of the future.”

What makes this worse if the fact that Google is looking to display ads on its AI Overviews. This means that, not only is this tool redirecting traffic from sites, but it’s going to profit from the Overviews. Rather than splitting the revenue from site traffic, Google uses the information from websites for its Overviews and doesn’t pay any money back to the site.

The question is: what’s the point of writing if Google uses what you write, steals your traffic, and profits off of it? At this point, there’s nothing in the way of this taking off except for Google’s own screw-ups. The company struggled with AI Overviews when they launched because of the company’s rushed release. However, the company is patching up the technology and distributing AI Overviews to more places around the world.

Google’s claims

According to the report, Google says that clicks on AI Overviews are higher-quality. It says that this is because people are more likely to stay on the promoted sites. However, AI Overviews shows a short list of sites. While the clicks are higher-quality (this is according to Google, and it didn’t offer any evidence to support this), it blocks out the majority of the other sites that are reporting on the same story.

This means that only the top news sites will have a chance of getting clicks, which leaves out smaller, yet still valid, sites. Additionally, it makes it nearly impossible for up-and-coming news sites to have a chance to thrive. Along with that, AI Overviews will decrease the traffic to the bigger sites because people are more likely to just read the overviews and go on with their day.

The journalism industry is going to change because of AI Overviews and other AI-powered tools. We’re talking about ChatGPT Search, Perplexity, and others. There’s no telling if there will even be a journalism industry in a few years.