Crawled — a first-party ledger of which machines read this website, which pages they take, and how long after publication they arrive

For thirty years the question was where a page ranked. It is becoming a different question: whether a machine reads the page at all, and whether what it takes away is what the page actually said. Almost everything written about that second question is advice. Very little of it is measurement, because measuring it requires a site, a control group, and a willingness to publish the result when it is boring.

This is the smallest honest instrument for it. Every request to a tracked page is matched against a table of known crawlers, and one row is written: which agent, which path, when. From that come three things that are hard to get any other way — which operators actually fetch a small site, how long a page waits between being published and being read, and whether the answer layer behaves at all differently from the search engines crawling the same pages in the same window.

Half the experiment pages publish a machine-readable brief. Half do not, on purpose, by a rule fixed before there was anything to see. That holdout is the only reason any sentence on this page can be a finding rather than an anecdote, and it costs something real: half the portfolio is deliberately published in the worse condition, if the worse condition turns out to be worse.

A fetch is not a citation. A citation is not a recommendation. A recommendation is not an agent choosing to use something. Those are four different events and only the first is observable from a server log.

It would be easy to invent scores for the other three — a retrievability index, a citation-worthiness rating, an agent-readiness percentage out of a hundred. They would look authoritative, they would be screenshot-friendly, and they would be fabricated, because there is no ground truth on this site to check them against. Anything on this page that is a number came from a request that actually arrived.

The honest version of the missing half is a second instrument that asks real models real questions on a schedule and records whether this site comes back — mention, citation, and whether the description is accurate. That is measurable, it is not cheap, and it is not built yet. It is the obvious next thing, and it stays unbuilt rather than being simulated in the meantime.

• Where the count happens. Edge middleware, before the cache. A story page is served from the edge for sixty seconds at a time, so a crawler fetch never reaches a route handler — anywhere later in the stack, most of this traffic is invisible.
• Who is counted. Only agents in a published table, matched on a substring of the user-agent. Unrecognised agents are not counted at all: an unknown-bot bucket would be mostly scrapers and uptime checks, and putting that beside GPTBot would flatter the result.
• What is stored. Agent, path, timestamp. No IP address, no headers, no request body, and nothing that could identify a person — everything recorded here was sent by a program.
• What is not a zero. A page nothing has asked for reports null. “Nobody came” and “we have not watched long enough” are different facts and the tables above keep them apart.
• The arms. Even experiment numbers publish a brief, odd ones do not. The first version of this was a hash of the slug, which was unarguable and also landed 7 against 15 with almost every recent live page in one arm — a comparison between new pages and old ones with the treatment painted on top. Numbers run in time order, so alternating them balances the arms and stratifies them by age at once.
• When a comparison is allowed. Not until each arm has 3 pages read by the answer layer. The threshold is in the source, so the experiment cannot start claiming a result the moment somebody feels like it has one.

One row per request to a tracked public page whose user-agent matches a fixed table of known crawlers. The count happens in edge middleware, before any cache, because a page served from the edge cache never reaches a route handler and so is invisible everywhere else. Nothing about the request is stored beyond the agent name, the path and the time — no IP, no header dump, no body.

We do not. A user-agent is a claim. Nothing here is verified against published IP ranges or reverse DNS, so an agent that lies is recorded as whatever it claimed to be. Verification is possible for some operators and is the obvious next thing to add; until it exists, every number on this page carries that caveat rather than a footnote apologising for it later.

Because a measurement with no control group is a chart, not evidence. Half the experiment pages publish a brief at /e/<slug>/brief.json and are listed in /llms.txt; the other half deliberately publish nothing of the kind. The rule is the experiment number: even numbers publish a brief, odd numbers do not. Experiment numbers run in time order, so the arms come out the same size and stratified by age, and anybody can check an assignment without running anything. It was fixed before any data existed and is not adjustable once a result starts to look promising.

Whether any of it reaches an answer. Nothing here observes a citation, a recommendation, or an agent deciding to use something. Those need a channel that does not currently exist for a site like this, and inventing a score for them would be exactly the confident nonsense this experiment was built to argue against. What is measurable today is retrieval — who came, what they took, and how long they took to arrive.

Probably not much, and that is a real risk to the whole design. A page can be crawled a hundred times and never appear in a single answer. The reason to measure it anyway is that retrieval is necessary even if it is not sufficient: nothing that was never fetched was ever cited, so the floor is worth knowing before anybody argues about the ceiling.

Easily, and that is why the arms exist. Publishing more pages, or pinging crawlers, would raise every total here without answering anything. The only question this page can settle is a difference between two groups of pages on the same site in the same window, and that difference is indifferent to how big the totals get.

Yes. The whole ledger is at /api/experiments/crawled/ledger as JSON, under CC BY 4.0, with the method and the taxonomy attached. If you cite a number from it, link the endpoint rather than a screenshot — the number will have moved by then, and the endpoint will be right.