AI-powered search engines (e.g., OpenAI’s ChatGPT, Perplexity) are failing to drive meaningful traffic to publishers while their web scraping activities increase. That’s one big takeaway from a recent report from TollBit, a platform that says it helps publishers monetize their content.
CTR comparison. Google’s average search click-through rate (CTR) was 8.63%, according to the report. However, the CTR for AI search engines was 0.74% and 0.33% CTR for AI chatbots. That means AI search sends 91% fewer referrals and chatbots send 96% less than traditional search.
Why we care. This is bad news for publishers because it shows AI search won’t replace traditional search traffic. As AI-generated answers replace direct website visits, you should expect to see this trend continue.
By the numbers. AI bot scraping doubled (+117%) between Q3 and Q4 2024. Also:
- The average number of scrapes from AI bots per website for Q4 was 2 million, with another 1.89 million done by hidden AI scrapers.
- 40% more AI bots ignored robots.txt in Q4 than in Q3.
- ChatGPT-User bot activity skyrocketed by 6,767.60%, making it the most aggressive scraper.
- Top AI bots by share of scraping activity:
- ChatGPT-User (15.6%)
- Bytespider (ByteDance/TikTok) (12.44%)
- Meta-ExternalAgent (11.34%)
- PerplexityBot continued sending referrals to sites that had explicitly blocked it, raising concerns about undisclosed scraping.
Context. One company, Chegg, is attempting to sue Google over AI Overviews. Chegg claims Google’s search feature has severely damaged its traffic and revenue.
About the data. There’s no methodology section, so it’s not entirely clear how many websites were analyzed, just that it’s based on “all onboarded ToolBit sites in Q4.” Toolbit says it “helps over 500 publisher sites.”
The report. TollBit State of the Bots – Q4 2024 (registration required)
New on Search Engine Land
About the author
Danny Goodwin is Editorial Director of Search Engine Land & Search Marketing Expo – SMX. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.
Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.