Methodology

Below are the rules that decide every score: which texts are tested, how each humanizer and AI detector is used, and how the numbers are combined. They apply to every tool in the same way, and no step involves a judgment call. The scoring code is MIT-licensed and the cycle data is released under CC BY 4.0. This page describes methodology v1.1.0.

Sample selection

Test texts come from seven writing categories, chosen to match what people typically paste into an AI humanizer:

Each category has one or more prompt templates. A template contains placeholders such as [TOPIC], [ROLE] or [FIELD], and each placeholder is filled from a version-controlled bank of possible values. A deterministic procedure decides which value goes into which slot. It is seeded with a random nonce that stays secret until the cycle ends.

Since October 2026, the input texts are written by several language models instead of one. The nonce that fills the templates also decides which model writes each prompt, and the list of models is published with the commitment before the run. October 2026 used 6: Gemini 2.5 Flash, Claude Sonnet 5, GPT-5.6 Luna, Gemini 3.5 Flash, Claude Haiku 4.5, and GPT-5.6 Terra.

Three files in the cycle bundle cover this step: templates.json holds the prompt templates, banks.json the frozen value banks, and samples.json the AI-written input texts every humanizer received. You can read any cycle's sample count straight from that file.

Automated execution and evidence

Nobody operates a tool by hand during a cycle. The runner program calls every humanizer, so each one goes through the same steps, and the published files are enough to re-derive the results. Tools that can only be used by hand are not included, because nobody could repeat or check their runs. Scores are averaged over all samples, not taken from a single output.

A cycle leaves behind three kinds of evidence:

• The records. results.jsonl holds one row per test with the source text, the tool's output, the settings it ran with, and each detector's score. The scorer reads the same data from tests.json and detector-scores.json. We keep only the content of each exchange, not the raw request and response traffic. That traffic would add nothing checkable and would show vendors what benchmark requests look like.
• The cryptographic checks. The commit-reveal sequence and the hash manifest described below prevent choosing prompts after the fact and changing files after publication.
• The manual check. All seven detectors are public products, so you can paste any published output into one and compare its answer with the value we stored. The evidence pages list each prompt, rewrite and verdict to make this easy.

Humanizer configuration

Humanizers offer different modes, models and strength controls. To avoid giving any tool a better setup than the others, one set of rules covers all of them:

• Default settings. A tool runs with the mode, model and strength it uses when a user changes nothing. We do not look for a stronger option for some tools, and we do not turn on premium or “maximum stealth” modes. When a tool requires a choice and has no default, we pick a neutral option and record it in settings_used so the same setup can be run again.
• API calls. Every tool is run through API calls on its default settings. We do not turn on premium models or buy add-ons to improve a result. How each tool was run is stored with each test as plan_tier_used.

For each humanizer, leaderboard.json stores the settings_used and plan_tier_used that actually ran, next to its score.

Detectors

Seven commercial AI detectors judge every output: GPTZero, Winston AI, Originality.ai, ZeroGPT, Grammarly, QuillBot and Copyleaks. The benchmark does not run, tune or coordinate with any of them. Each one receives the same text, and its answer is stored as a human-likelihood score from 0 to 1, where 1 means human-written and 0 means AI-generated.

The only change we make is converting every answer to that scale. An AI probability is flipped to 1 − ai, a percentage is divided by 100, and the result is clamped to [0, 1]. There is no calibration or reweighting. Each cycle's detector-scores.json holds every stored value, with one scores object per test and a key for each detector.

Scoring formula

Four sub-scores, each between 0 and 1, are multiplied by fixed weights that add up to 100. Penalty points are then subtracted, the result is kept between 0 and 100, and it is rounded to one decimal:

weights:  bypass 42 | meaning 32 | readability 16 | consistency 10

composite_raw = 42 × bypass_rate
              + 32 × meaning_preservation
              + 16 × readability
              + 10 × consistency_across_categories

penalty   = sum over penalty codes of min(occurrences × per, cap)
composite = clamp(composite_raw − penalty, 0, 100)

The weights and penalty values are constants inside the cycle's own scoring.js, so each leaderboard ships with the exact numbers that produced it. All of a tool's tests count toward the averages, whatever their status. A refused, errored or unchanged output still counts and can also draw a penalty. Leaving failures out would let a tool gain points by failing.

bypass_rate

For each test, take the median of the detector scores. A null value means that detector returned no verdict for the text (for example, it was unreachable or the text was below its minimum length). It is left out of the median instead of being counted as 0. The bypass rate is the average of the per-test medians.

meaning_preservation

For each test, we embed the input and the output and compute their cosine similarity, clipped to the range 0 to 1. That raw value is stored in tests.json, and the meaning-drift penalty checks this same value.

readability

A language model rates each output once for clarity, fluency and naturalness, on a scale converted to 0–1. The rating is stored per test in tests.json, and the published scorer averages those stored numbers. So checking the leaderboard needs no access to any model.

consistency_across_categories

Average the per-test bypass medians within each writing category. Consistency is max(0, 1 - stddev(category_means)), so a tool that performs the same in every category gets 1, and uneven results across categories lower it.

Penalties

Some output failures cost points. Each penalty code subtracts a fixed amount per occurrence, up to 10 points per code. With five codes, a tool can lose at most 50 points, and no single failure type can decide a ranking alone. The penalties applied to each humanizer are listed one by one in leaderboard.json.

Unavailability is excluded, not penalized

A test gets the status error when the tool could not be used at all: the service was down, the quota ran out, or requests were blocked. If a tool gets this status on 50% or more of its tests, the cycle says little about that tool. It is left out of that cycle's ranking and recorded under excluded in leaderboard.json with its attempt counts. It is not ranked on the few tests that worked, and it is not penalized for the outage. Poor output is handled differently. When a tool runs and returns a bad result, the penalties above apply.

Transparency and verification

A cycle is named after its month. If the prompt draw used that name as its seed, a vendor could work out next month's prompts in advance and optimize for them. So the seed is kept secret, and a commit-reveal sequence proves it was fixed before the run:

1. 1
   Commit

   A random 32-byte nonce is generated first. Only its SHA-256 hash is published, in commit.json, before any tool runs.

2. 2
   Run

   The runner sends the source texts to every humanizer and scores each output with every detector. It keeps the input, output and settings used, not the surrounding network traffic.

3. 3
   Reveal

   At the end of the cycle the nonce is published, along with the templates, value banks, selection code and resulting prompts as they were used.

4. 4
   Seal

   manifest.json lists a SHA-256 hash for every file in the bundle plus one hash covering all of them, so a later change to any file can be detected.

One command runs every check. Each check guards against a different kind of tampering:

• Hashing the published nonce gives the value in commit.json, and running the stored selection code with it reproduces prompts.json. The prompts were not re-drawn after the run.
• Each source text maps to a committed prompt, and each test gave the tool exactly the published source text. The test set was not replaced with easier material.
• The stored scoring code, run on the published per-test rows, reproduces every field of the leaderboard within 1e-4. No figure was changed after scoring.
• Every humanizer has a test for every sample, and every test has an entry for every detector. No rows were removed to improve an average, which the recompute check alone would not catch.
• Every file's hash matches manifest.json. The bundle has not been modified since it was published.

Run it yourself:

git clone https://github.com/AIHumanizerBenchmark/AIHumanizerBenchmark.git
cd AIHumanizerBenchmark
npm run verify                      # checks all published cycles
npm run verify -- "October 2026"  # checks a single cycle

Nothing needs to be installed. The verifier uses only Node's built-in modules and makes no network calls, because a checker that depends on packages you have to trust would prove little.

Recomputing every score from the raw data

The leaderboard is computed entirely from the published files. Each cycle bundle includes its own dependency-free scoring script, and one function call turns the per-test data into the full ranking. From a clone of the repository:

import { readFileSync } from "node:fs";
import { computeLeaderboard } from "./data/cycles/2026-10/scoring.js";

const dir = "./data/cycles/2026-10/";
const load = (name) => JSON.parse(readFileSync(dir + name, "utf8"));

// Ranked tools first, highest composite at index 0.
const ranked = computeLeaderboard(load("tests.json"), load("detector-scores.json"));
console.log(ranked[0]);

You can also check one sub-score by hand. For a humanizer's bypass rate:

1. Collect every row for that humanizer in tests.json, including rows with error, refusal or identical status. Removing them would inflate the result.
2. Find each row's entry in detector-scores.json and compute the median of its scores, ignoring any null. A missing verdict is not a score of 0.
3. Take the mean of those medians to get the bypass rate.
4. Check it against the value in leaderboard.json, allowing an absolute difference of up to 1e-4.

Versioning

leaderboard.json has three version fields. Scores from two cycles can be compared directly only when all three match:

The repository's CHANGES.md has one entry per cycle that says which of these fields changed and why. So when a tool's score shifts between cycles, you can tell whether the tool changed or the rules did.

A published cycle is never edited. Its bundle includes the scoring.js it was scored with, and the verifier checks each cycle against that file, so a new formula cannot change an old result. If a published result proves wrong, the fix is recorded as a new entry in CHANGES.md and a note on that cycle, not by editing the bundle.

Methodology integrity

Rankings are produced only by running the published scoring code on the published data. You do not have to take our word for it. A leaderboard produced any other way would fail the public verifier, which anyone can run.

How to contribute or dispute

If you think a score is wrong or have found a mistake in the data, file a dispute. We reply within 14 business days and publish the outcome. To have a tool considered for a later cycle, submit a humanizer or suggest a detector. Send corrections, data questions, security reports about the verifier and anything else to contact@aihumanizerbenchmark.com.