How we test

What gets tested

Claims that are testable on a site we control, and that matter enough to be worth the effort. That mostly means crawling and indexing behaviour, which is observable, rather than ranking behaviour, which usually is not. A test that cannot produce a clear negative result is not run.

The standing method

1. State the prediction first. Written down before the test runs, including what result would falsify it. A test with no falsifying outcome is not a test.
2. Change one thing. One variable per test. Where several changes ship together, the results are reported as inconclusive, because they are.
3. Use sites we control. Never a client site without written permission, and never a change that could harm a site's performance for the sake of an experiment.
4. Measure from primary evidence. Server access logs for anything crawl-related, Search Console for indexing, and both together where they disagree. Third-party estimates are not evidence.
5. Wait long enough. Crawl and index changes are not instant. A result read after two days is noise.
6. Publish the raw shape of the data. Sample size, date range, site type, and what was excluded.

What a single-site test can show

It can show that a behaviour occurred at least once, which is enough to disprove a universal claim. If someone says a search engine never does X and we observe X, the claim is dead.

What it cannot show

It cannot establish causation for ranking, it cannot establish weights, and it cannot be generalised to every site. One site is one sample under conditions we only partly observe. Where a result is suggestive rather than conclusive, the article says so, in those words.

Correlation studies

Ranking correlation studies are easy to produce and almost never establish cause. Where this site cites one, it says what the study measured, what it sampled, and what it cannot conclude. Where this site runs one, the same disclosure applies to our own work.

Errors

Tests that produce a result we did not expect are published anyway, including the ones that contradict something published here earlier. The editorial policy covers how corrections are made and dated.