Your CRO tests are lying: how bot traffic warps conversion data

If your conversion rate fell after a “winning” homepage test, the variant may not have failed. Your denominator may have.

Author: StJohn Krog — LinkedIn

If your conversion rate fell after a “winning” homepage test, the variant may not have failed. Your denominator may have.

Bot traffic — automated requests that are not shoppers — does not only steal inventory or scrape prices. It inflates sessions, dilutes conversion rate, creates junk events, and can unevenly bucket across experiment variants when one URL pattern attracts more scrapers than another. Cloudflare puts the scale in context: it is widely cited that over 40% of internet traffic is bots, with a material share malicious or simply noisy. For CRO, the precise global percentage matters less than whether your store’s analytics and tests are human.

How bots break CRO maths

Cloudflare’s learning centre is blunt: unauthorised bots distort pageviews, bounce rate, session duration, geo mix and conversions. That noise cripples A/B testing and optimisation because you are measuring a population that never intended to buy.

Typical ecommerce symptoms:

• Sessions rise while revenue is flat
• Bounce rate spikes on PDPs that scrapers hit
• “Direct” or odd geo traffic surges overnight
• Form spam and gibberish account creations
• Experiment lift that vanishes when you filter non-humans

Optimizely documents the experimentation side: Web Experimentation filters known bots/spiders (IAB/ABC list) from results when User-Agent is present; Feature Experimentation bot filtering must be enabled before a test if you want clean history — turning it on mid-flight does not scrub past contamination.

Cloudflare’s angle: stop the junk before it becomes “data”

Analytics filters clean yesterday’s report. Edge bot management cleans tomorrow’s experiment.

Cloudflare’s bot solutions range from Bot Fight Mode / Super Bot Fight Mode to Bot Management with per-request bot scores (1–99) for Enterprise. The CRO-relevant pattern for UK SMEs and scale-ups:

1. Detect automated traffic at the edge (scores, verified bots allowlists for good crawlers).
2. Challenge or block abusive classes before they hit origin, cart and checkout. Prefer friction that humans barely notice — Cloudflare’s Managed Challenge / Turnstile path exists so you are not training customers on puzzle CAPTCHAs.
3. Pass signals downstream — so GA4, warehouse and experiment tools can segment or exclude.
4. Protect limited inventory and flash paths — Waiting Room plus bot exclusion keeps humans from queuing behind scrapers.

A clean-test playbook for UK ecommerce

Week 1 — Diagnose Compare unfiltered vs human-filtered conversion (Shopify from Oct 2025 data onward). In GA4, hunt spikes by country, hostname and page path. Cross-check Cloudflare bot analytics if enabled.

Week 2 — Edge hygiene Enable appropriate bot fight / WAF rules; allowlist verified search bots; challenge suspicious automated clients on login, account create and checkout start. Avoid blunt blocks that hit corporate proxies or in-app browsers without review.

Week 3 — Experiment hygiene Confirm Optimizely (or VWO equivalent) bot filtering is on before launch. Exclude internal QA. Prefer metrics rooted in purchases or server-confirmed events, not pageviews alone. Annotate the date you started human-only baselines.

Week 4 — Rebaseline Do not celebrate a YoY conversion “win” that straddles a filter change. Compare unfiltered-to-unfiltered for continuity; treat human-only as a new series.

Practical takeaway

CRO without bot hygiene is storytelling with polluted data. Filter humans in analytics, enable experiment bot exclusion before tests, and use Cloudflare-class edge controls so scrapers and credential-stuffers never become sessions in your denominator. Then your next A/B result is about shoppers — not scripts.

Why this matters for Clarity Growth

Conversion programmes fail quietly when the measurement layer is wrong. Clarity Growth helps UK ecommerce brands combine Cloudflare bot management, analytics hygiene and experiment design so CRO tests, paid media and Shopify reporting reflect human demand. Engineering and growth share one goal: decisions you can defend in a board pack.

Sources

1. Cloudflare Learning — What is bot traffic? (bots >40% of internet traffic; analytics and CRO distortion)
2. Cloudflare Docs — Bot solutions overview (Bot Fight Mode, Super Bot Fight Mode, Bot Management)
3. Cloudflare Docs — Bot scores (1–99 scoring for rules/Workers)
4. Optimizely — Filter bot and spider traffic from results (IAB/ABC list; enable before experiments on Feature Experimentation)
5. Shopify Help / Analytics context via industry explainers on Human or bot session (from 7 Oct 2025) (human/bot dimension; reporting vs pixel timing caveats — verify in Shopify Admin docs for your plan)
6. Naniza — Bot traffic and conversion benchmarks (Shopify filter timing; A/B contamination discussion; illustrative 3.5%→4% example)