Bill Hunt

The era of indiscriminate web crawling is ending. As infrastructure costs mount and legal frameworks tighten, both search engines and AI companies are shifting from “crawl everything” to “crawl selectively.” This transformation, catalyzed by initiatives like Cloudflare’s Pay Per Crawl and growing resource constraints, demands that content creators optimize for crawl-worthiness or risk digital invisibility.

Earlier this week, I posted a LinkedIn recap of three articles I’d written, each one circling the same conclusion from a different angle. Search Was the Detour argued that search engines were never the destination — just one way of connecting people to information, and AI is simply another interface doing the same job. One

The previous article in this series asked a smaller, more embarrassing question than “should we invest in a knowledge graph.” It asked whether the right people were even in the same room, and whether that room could sit with an unanswered question long enough to be useful — instead of grabbing the nearest concrete task

How to author a canonical record that renders your site, your feeds, and your machine layer from one governed source, and why the page is now an output rather than the original The first two articles in this series built the case for this one. “The Friction Is the Code, Not the Feeling” argued that