AI Crawlers, Shopping Agents and Human Visitors Need Different Measurement

A crawler reading a product page, an agent checking it for a shopper and a person comparing two products can all appear in website activity. They do not represent the same thing.

Cloudflare's July 1 update makes that distinction explicit. Its new controls separate AI traffic into Search, Agent and Training behaviors. For a business reviewing website performance, the useful lesson is broader than one set of controls: decide what kind of activity a number represents before calling it a visitor or a buying signal.

Keep the automated activity you need to understand. Keep it out of reports where it would answer the wrong question.

Understand the three behaviors

In Cloudflare's categories, Search covers automated collection or indexing used to answer questions later. Agent covers automated activity acting in real time on a person's behalf, including fetching information or using a browser. Training covers collection used to train or fine-tune a model.

Those labels describe purposes. They do not tell a store owner that a purchase happened, that a named person visited or that the traffic is necessarily useful to that business.

A product page fetched by a shopping assistant may be part of someone's research. It is still a different observation from a person opening the page and completing the site's checkout.

Use the distinction when reviewing website measurement features. Ask what is being observed and what business conclusion the observation can support.

Start with the question your report answers

Different reports should include different activity.

If you are investigating infrastructure load, automated requests matter. If you are evaluating whether your content is being collected by outside services, excluding crawlers would remove part of the subject you want to study.

If you are reviewing how people navigate a product page, known automated fetching should not quietly become a shopper session. If you are investigating agent-assisted shopping, create a view that describes that activity on its own terms.

Name the report after its question. “Requests to product pages” describes a different measure from “observed shopper visits that reached checkout.” Neither needs to pretend it is the other.

Write the inclusion rule next to the number. That is more useful than a broad label such as “quality traffic,” which can hide several different decisions.

Do not turn requests into people

A request is an interaction with the site. Your logs may record many requests around one task, and a single automated process may request many pages.

Consider a hypothetical review. A site's logs contain 50,000 requests during a period, of which 40,000 are classified as a known automated fetcher. The remaining 10,000 requests are not a count of 10,000 human visitors. They are the requests left after that particular classification.

The team needs another method to describe the browser activity relevant to its buying journey. It also needs an uncertain category when the evidence does not support a confident classification.

Do not subtract one bot count from a request total and label the remainder “buyers.” You would be changing both the unit and the conclusion without establishing either.

Keep agent-assisted activity visible

An agent acting for a person deserves more thought than a simple “bot equals useless” rule.

The agent may be finding dimensions, checking availability or comparing information. Its request can matter to your business even when nobody sees your page in a normal browser during that step.

But the presence of an agent does not identify the person behind the task. It does not tell you that the task ended in a purchase. It does not create a reason to contact someone whose identity you have guessed.

For an online store's measurement, define the next event you would need to establish an outcome. That might be a completed order your store can confirm or another explicit action in your own records. Keep the intermediate agent activity separate until you can support the connection.

Ask how the classification was made

A useful classification comes with evidence and limits.

Ask the person responsible for the traffic tools what signals produced the label, how the classification changes over time and what happens when a service has more than one purpose. Check whether the label describes a particular request, a known operator or a rule your own team configured.

Keep the date and source of the classification where they matter to the report. If a provider changes a category or you change an access rule, a movement in the chart may reflect the new rule.

Allow “unknown” to remain a real answer. A tidy report that forces everything into human or bad bot can hide the very uncertainty the team needs to manage.

Separate access decisions from sales decisions

Your security team may decide whether to allow, block or challenge a request. Your marketing team may decide whether an observation belongs in a campaign report. Your sales team may decide whether a known contact needs a response.

Those decisions can share information without becoming one decision.

A request allowed through an access control is not automatically a qualified lead. A request classified as automated is not automatically malicious. A named contact associated with a separate visit still needs relationship context before outreach.

Use the visitor-to-action workflow to make those handoffs explicit. Record the observation, the decision it supports and the person responsible for that decision.

Treat September's defaults as a future change

As of July 7, the September 15 default changes described in Cloudflare's announcement are still ahead. The stated new-domain defaults concern Training and Agent traffic on pages displaying ads, while Search remains allowed.

Do not describe those future defaults as already applied to every website. Check the conditions of your own account and pages before making an access decision.

If a future change is relevant to your site, assign someone to review its effect on real tasks before the date arrives. Test product information, enquiry forms and purchase paths, including the automated uses your business intends to support.

A category name is not enough to decide whether blocking it would help your customers.

Check collection choices and report boundaries

Separating automated requests does not remove the need to review the rest of your measurement.

Document which tools observe browser activity, which collect server requests and how visitor choices affect the available records. Review consent and data handling with the owner of that setup.

Avoid placing personal details into general event labels or campaign links to make a connection easier. Use the information necessary for the question and preserve an unknown outcome when you cannot establish the link appropriately.

For example, an order record can confirm a transaction in your store. It does not automatically establish which earlier automated request caused it.

Give each kind of activity a useful next step

Choose one report to fix first. Write its unit, included activity, uncertain group and business purpose. Then inspect a few examples with the people who own the data and the decision.

A content team might review what information an agent can retrieve accurately. A store team might make a product specification clearer. An operations team might investigate excessive automated load. A sales team might do nothing because no known relationship needs action.

That is a stronger result than labeling every extra request a visitor. You can understand the growing range of activity on your website without confusing collection, assistance, browsing and buying.