How We Benchmark Our Codecs (and How You Can Verify)

Every compression vendor publishes a number. Almost none of them publish enough for you to check it.

This piece is the methodology behind our results for Epiron, our neural image codec, and Ansatz, our neural reconstruction codec for talking-head video. It covers what we measured, what we measured against, what the numbers mean, and, most usefully, where they do not apply. If you only read one thing we publish, read this one, because it is the piece that tells you how much to trust everything else.

Key Takeaways

• We report results on real coded files, not estimated bitrate.
• Baselines are named, versioned, and run at settings we state, on standard public test sets.
• Decode speed is compared against the field it belongs to, on the same hardware.
• Every claim travels with its scope. The numbers that do not apply to your content are the numbers we are most careful about.

What makes a compression benchmark checkable?

A benchmark is checkable when someone else with the same test set and the same baselines could plausibly reproduce the shape of your result, and when your claim states precisely enough what it covers that it could be wrong.

That second half is the one people skip. "Beats the standard" cannot be wrong, because it does not say which standard, at what version, at what setting, on what content, measured how. A claim that cannot be wrong is not a claim. It is an advertisement.

So the discipline we hold ourselves to is: name the baseline and its version, state the setting, use a public test set, produce real files, report the full curve rather than a point, and say what the result does not cover. The rest of this piece is that discipline applied.

Do you produce real files, or estimate the bitrate?

Real files. Every Epiron result comes from real entropy coding producing an actual bitstream, not from the theoretical entropy of the latent representation.

Estimated bitrate is a legitimate research proxy, it is what you train against, and it is a floor rather than a result. An entropy coder has overhead, and it pays for every gap between what the model predicted and what the data actually did.

A published result that never produced a bitstream has not demonstrated that it can produce one at that size. We produce the bitstream.

What do you compare against, exactly?

Named baselines at named versions and settings, because everything else is a rhetorical move.

For Epiron, we compare against H.266/VVC using VTM at a stated version, and against the strongest neural codecs in the field. The comparisons against conventional standards are run all-intra, which is the correct image-mode comparison and pre-empts the obvious objection that a video codec was tested on stills. We report BD-rate across a full rate curve on Kodak, and confirm the result holds at 2K on CLIC Professional, because a codec that wins only at one resolution has not won.

For Ansatz, the baselines are x265 and SVT-AV1 at fixed near-transparent quality settings, which is a customer-shippable bar rather than an artificially weak one. The corpus is 206 single-face 4K talking-head clips harvested and filtered by a mechanical face gate, of which 198 completed. Critically, the population is whatever cleared the gate, not a hand-picked set. We report the distribution, not a best-case mean.

One place we are deliberately careful: conventional codecs have slow presets that are meaningfully stronger than their fast ones. Comparing against a fast preset flatters the challenger. Where a comparison is preset-sensitive, we say so rather than quietly banking the easier number.

How do you measure quality, and why not PSNR?

With the metric that fits the content, which is not always the metric that is conventional.

For Epiron, which reproduces pixels, standard rate-distortion measurement applies and we report it, alongside a structural similarity floor at or above 0.975 as the production guarantee.

For Ansatz, pixel-correspondence metrics are structurally inapplicable, and this is not a convenient excuse. A reconstruction codec regenerates the face rather than reproducing the original pixels, so the output can be visually faithful while the pixels do not align. Measure that with PSNR or VMAF and you get a catastrophically low score on an image that looks right. The metric is not detecting a quality problem; it is detecting that the pixels moved, which was the design.

So Ansatz is measured with LPIPS, which compares deep perceptual features rather than pixel positions, alongside decoded identity and landmark alignment as the codec's own acceptance checks. We verified during bring-up that VMAF was the wrong ruler here and dropped it from the population run. We explain why the metric choice is itself a design decision in why SSIM falls short of human vision.

How do you report decode speed?

Against the field the codec belongs to, on the same hardware. Anything else is measuring the hardware.

Epiron's decode advantage is an algorithmic claim against other neural codecs running on the same hardware. It is emphatically not a claim against a CPU-based conventional decoder, and quoting it that way would be comparing a GPU to a CPU and calling it a codec result. That comparison would flatter us and mean nothing.

The reason this matters for Epiron specifically is that the strongest neural codecs have historically relied on autoregressive entropy models that decode sequentially and therefore slowly, sometimes taking seconds per image. Beating that field on decode while also beating the conventional standard on compression is the actual result. Beating a CPU with a GPU is not.

What do the numbers not cover?

This is the section most vendors do not write, and it is the one that should make you trust the rest.

Epiron. The comparison against conventional standards is all-intra, which is correct for images and does not tell you anything about video coding. The decode-speed claim is against the neural field, not against CPU-based conventional decoders. Results are on Kodak and CLIC Professional; your content is not Kodak, and the honest way to find out what it does on your library is to run it on your library.

Ansatz. The scope is narrow and stated: single-face talking-head video at 4K. It is a decisive win in a specific class of content, not a general 4K codec, and it is designed to fall back to conventional coding where its representation does not apply. In its current form it saves storage and origin bandwidth, not last-mile bandwidth, because reconstruction runs on a GPU at decode. Anyone who tells you a generative codec cuts your CDN bill to users today is skipping that step.

Both. Baselines are fixed-quality rather than iso-quality anchored, which is a documented follow-up. And the population results include the tail: on Ansatz, clips with multiple faces or heavy turn-away win less, because the codec correctly falls back and stays pixel-sharp. We report those in the distribution rather than excluding them, because excluding them would make the median prettier and the number a lie.

What can you actually verify?

More than you might expect, and less than we would like, which is the honest position.

You can check the baselines. VTM, x265, and SVT-AV1 are public. Kodak and CLIC are public. If our claimed baseline numbers are wrong, you can find out with a laptop and some patience, and that is a genuine constraint on us.

You can bring your own data. The most meaningful verification is not reproducing our test set; it is running your assets and seeing what happens. A codec that wins on Kodak and loses on your product photography has told you nothing useful. We would rather you test it than believe us.

You cannot reproduce our method, and we should say that plainly rather than implying otherwise. The reconstruction method, the architecture, and the training are proprietary and we do not publish them. What we publish is the result, the distribution behind it, the metrics, and the scope. That is a real limitation on verifiability and we are not going to pretend it is not.

What we can offer against that limitation is falsifiability: the claims are specific enough that they could be shown wrong, on public baselines, with public data, using your own content. That is not full reproducibility. It is considerably more than a number in a press release.

Frequently Asked Questions

Do you report estimated bitrate or real coded files?

Real coded files. Every Epiron result comes from actual entropy coding producing a real bitstream. Estimated bitrate is a legitimate research proxy but it is a theoretical floor, not a delivered result.

Why do you compare against conventional codecs all-intra?

Because it is the correct image-mode comparison. Comparing an image codec against a video codec running in video mode would be measuring the wrong thing, and running all-intra pre-empts the obvious objection.

Why does Ansatz use LPIPS instead of PSNR?

Because a reconstruction codec regenerates content rather than reproducing pixels, so faithful output does not align pixel-for-pixel with the source. PSNR and VMAF read that as catastrophic error when the codec is working as designed. LPIPS compares perceptual features and is the correct ruler.

Is your decode-speed claim against conventional codecs?

No. It is against other neural codecs on the same hardware, which makes it an algorithmic claim. Comparing GPU-based neural decode against CPU-based conventional decode measures the hardware, not the codec.

Can I reproduce your results?

You can reproduce the baselines, which are public, and you can run our codecs on your own content, which is the more useful test. You cannot reproduce our method, which is proprietary and unpublished. We state that plainly rather than implying full reproducibility.

The point

A benchmark is a claim about the world, and a claim you cannot check is a claim you should not accept.

We publish the methodology because the results are only worth what the methodology makes them worth. Real files, named baselines, public test sets, full curves, the correct metric for the content, decode speed measured against the field it belongs to, and an explicit statement of what each number does not cover.

The part that costs us something is the scope section, and that is the part we would keep if we had to cut everything else. Anybody can post a median. Telling you where the result stops applying is the only way you can tell whether the median means anything.

Read the results with that in hand:  Epiron beats H.266 and Ansatz 4K talking-head results.