One Token Is Enough: Verify the Model Behind an AI API

When an API claims to serve a particular large language model, the model name in the response is often the only evidence we get. But a model name is just a string. Aggregators, resellers, routing layers, quantization, and deployment choices can all make the model you reach different from the one you expect.

ChatHub’s new Model Fingerprint tool offers a practical black-box check: compare a trusted reference endpoint with an API under test by looking at distributions of very short answers.

Why can one short answer reveal anything?

Ask a model to “name a random number between 1 and 100” once and the answer tells you almost nothing. Ask the same question repeatedly and a pattern appears. Models are not uniformly random: they tend to favor particular numbers, colors, animals, cities, and words.

Those small but repeatable preferences form a behavioral fingerprint. The idea comes from the paper One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions. It does not require model weights, logits, fine-tuning, or cooperation from the provider. It only needs ordinary API calls and short outputs.

What the ChatHub tool measures

The audit compares two OpenAI-compatible HTTPS endpoints:

• Reference API: a trusted official or self-hosted deployment whose model identity you can verify.
• Target API: the endpoint you want to test.

The full probe battery crosses 10 everyday tasks with four languages—English, Russian, Chinese, and Arabic—for 40 probe cells. Tasks include random and favorite numbers, letters, words, colors, animals, cities, and coin flips.

Each cell is sampled 15 times per endpoint. Responses are normalized, converted into probability distributions, and compared with Jensen–Shannon divergence. Refusals, empty answers, hidden reasoning, invalid formats, and request failures remain visible instead of being silently discarded.

Quick audit or full battery?

• Quick audit: 8 balanced cells, 120 calls per API, 240 calls total. This is the lower-cost first pass.
• Full battery: all 40 cells, 600 calls per API, 1,200 calls total. This provides stronger evidence.

In the paper’s experiment across 165 models, the reported equal error rate was 7.3% for the full 40-cell battery and 10.6% on average for random eight-cell subsets. ChatHub Quick mode uses a fixed, balanced operational preset across four languages; the paper does not claim that one particular eight-cell subset is optimal. These research figures are not an accuracy guarantee for every provider or deployment.

How to read the result

• Consistent: this run supports behavioral consistency at the selected operating point. It does not prove identical model weights, guarantee future routing, or establish cryptographic identity.
• Different: this run found a behavioral difference signal. Repeat the audit or run the Full battery before acting; it is not proof that a model was substituted.
• Inconclusive: no model-match verdict was produced. It does not mean the APIs are different. At least one selected cell lacks 10 valid direct responses from each API.

Start with Usable / required. A cell is usable only when both Reference and Target have at least 10 valid direct responses out of 15. Quick needs all 8 selected cells; Full needs all 40.

Then review the endpoint breakdown and each cell. Reasoning traces, empty responses, invalid formats, refusals, and request errors are excluded and shown separately. An insufficient cell shows the exact sample shortfall and its JSD is marked Not calculated. The tool does not display a partial mean as a verdict because applying the paper-calibrated threshold to incomplete evidence could mislead.

For a Different result, repeat with the Full battery or a second run. For Inconclusive, first correct the API format, model settings, rate limits, or output behavior, then rerun. Export the versioned JSON dataset when you need an audit trail.

How to run a real API audit

1. Choose a trusted reference endpoint on the same model version you expect the target to serve.
2. Enter the OpenAI-compatible HTTPS base URL, model ID, and API key for both endpoints.
3. Start with Quick mode. Use the full battery when the decision matters or the result is close to the threshold.
4. Review the overall verdict and per-cell distances. Export the JSON evidence when you need an audit trail.

How API keys and data are handled

API keys are stored only in the current tab’s session so interrupted audits can resume. They are cleared when the tab closes, forwarded server-side only for the tool’s fixed probes, and excluded from exported datasets. The proxy accepts only public HTTPS endpoints, blocks redirects, and caps response size.

What a fingerprint cannot prove

A behavioral fingerprint is statistical evidence, not a cryptographic identity document. Quantization, system prompts, model updates, regional routing, and sampling implementations can all shift a distribution.

For consequential decisions, use a trusted reference on the same version, run both endpoints in a similar time window, repeat the audit when necessary, and combine the result with latency, usage, provider records, and other operational evidence.

Try Model Fingerprint

Open Model Fingerprint in ChatHub

Method and research source: One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions.