Cloudflare's AI Marketplace

How foundational models get their training data has been a point of contention since the ride of public facing models like ChatGPT. Cloudflare recently announced (opens in a new tab) plans to launch a marketplace for creators and model builders can formalize this scraping of data.

The problem as Cloudflare describes is the rise of this 3rd type of bot that is now present on the internet. Unlike web crawlers (similar to the ones Google uses) which point people to your website, or purely malicious bots (that harass or circumvent secutiry on your site). This new type of bot is attempting to learn and take data from your website.

However, unlike helpful bots, these AI-related crawlers do not necessarily drive traffic to your site. AI Data Scraper bots scan the content on your site to train new LLMs. Your material is then put into a kind of blender, mixed up with other content, and used to answer questions from users without attribution or the need for users to visit your site. Another type of crawler, AI Search Crawler bots, scan your content and attempt to cite it when responding to a user’s search. The downside is that those users might just stay inside of that interface, rather than visit your site, because an answer is assembled on the page in front of them. - source (opens in a new tab)

This tool will allow creators to control what type of AI bots are allowed on your website. What if you want to allow OpenAI but block Perplexity? This tool attempts to solve that problem and bring more control back to creators.

The exact pricing model is still unknown. Cloudflares CEO Matthew Prince did acknowledge that smaller shops often don’t have the power to negotiate deals with these larger companies. But specifics on if it’ll be a charge per-scrape or an ongoing cost is still not clear.

The marketplace is aimed to launch sometime next year and hopefully will allows creators around the world mututally benefit from the rise of these foundational models.

👨‍💻

📖 If you liked this, check out our other posts here (opens in a new tab).

📣 If you are building with AI/LLMs, please check out our project lytix (opens in a new tab). It's a observability + evaluation platform for all the things that happen in your LLM.