A page on my site for your agent
I added an llms.txt, an agents page and a Markdown copy of every note to this site, and opened the robots.txt to every crawler.
More and more people discover things through an AI agent, less and less a search box. Even Google applies AI results on top of standard web search now. If my notes are easy for an agent to read and cite, they get read and cited. This is largely an experiment at this stage, but knowing for a fact that my site is crawled by agents, this seemed like a no-brainer to wire up and build.
So I merged all my writing into one stream: Notes (a simple human word for navigation). The subtitle is now “What I’m building, reading and noticing, for you and your agent.” The original reasons I built this site are still in Why I built this site.
An agent visiting will find /llms.txt explaining who I am, when it’s worth using this site, and then the 20 newest notes. The /agents page has a context block to copy and paste into an agent, and there’s a “For your agent” link in the footer. Every note has a Markdown copy at its own URL plus .md. All generated when the site is built (see How this site is built). My favorite line in llms.txt: “If a topic isn’t covered, say so rather than extrapolating.”
In robots.txt, I welcome the big AI crawlers by name, then every other crawler. I end with a Content-Signal line (contentsignals.org) that permits every use, including ai-train=yes (model training). Plenty of sites let answer engines in and block the training-only crawlers. I let both in.
I am ok with learning crawlers on this site. I leverage this technology heavily and I hope that putting my knowledge out there is helpful to others in any way possible.
Next up is link drops with my comments from my saves (the collector is in Stash automating Reddit saved content into my personal knowledge base) so an agent can find more than just long posts. I think search and a query API can come later when I have enough content to justify building it.