How AI Models Judge Visual Quality

You can glance at a photo and know something is wrong with it in about a tenth of a second. Not measure it, not analyze it. Just know. The compression artifacts around the text, the smeared skin, the banding in the sky, your visual system flags all of it before you have consciously decided to look.

Teaching a machine to do the same thing has taken about thirty years and is still not finished. This piece explains how the machines currently do it, which turns out to be two entirely different problems that get carelessly lumped together, and where each one still fails.

Key Takeaways

• There are two distinct problems: judging an image against its original (full-reference), and judging an image alone (no-reference). They need different machinery.
• Full-reference metrics like LPIPS compare deep neural features rather than pixels, correlating around 0.85 with human judgment against SSIM's 0.53.
• No-reference assessment is much harder and still largely unsolved. Models overfit to the distortions they trained on.
• Every quality model is an instrument with a blind spot, not an oracle. Optimize directly against one and you get images that score well and look wrong.

What are the two kinds of quality assessment?

Full-reference assessment compares a distorted image against the original it came from. No-reference assessment judges an image with nothing to compare it to. These sound like variations on a theme. They are not. They are different problems requiring different machinery, and conflating them is the source of most of the confusion in this area.

The distinction is practical, not academic. If you are compressing an image, you have the original sitting right there, so you can ask the precise question: how much did I damage this? That is full-reference. If you are screening user-uploaded photos, or grading a camera, or evaluating an image someone sent you, there is no original. You have to ask the much vaguer question: is this any good? That is no-reference, and it is far harder.

Almost every practical compression decision is a full-reference problem. That is worth holding onto, because it means the well-solved half of this field is the half that matters most for optimization.

How does full-reference assessment work?

Modern full-reference metrics run both images through a neural network and compare the internal feature representations, rather than comparing the pixels directly. That single change is what separates the metrics that track human judgment from the ones that do not.

LPIPS is the canonical example. Take the original and the compressed version, push both through a network trained on visual recognition, and at several layers, measure how far apart the resulting feature maps are. Those features encode things like "there is an edge here," "this is a face-like pattern," "this is texture of a particular kind." Comparing at that level is closer to comparing what the two images mean to a visual system than to comparing their raw pixel values.

DISTS takes a further step, separating structure similarity from texture similarity. That matters because texture is a special case: a patch of grass or fabric can be redrawn slightly differently and still look completely correct to a person, while a pixel-comparison metric registers a large error. DISTS tolerates that. Pixel metrics punish it.

The payoff is measurable. In a 2026 survey comparing metrics against real human ratings, LPIPS reached 0.85 and DISTS 0.86, while SSIM managed 0.53 (arXiv 2506.07369, 2026). We covered why the old metric falls down in [INTERNAL-LINK: why SSIM falls short of human vision → SSIM spoke]. The short version: SSIM averages across the whole image and human attention does not.

How can a model judge quality with no original?

No-reference assessment learns what "good" looks like from data, since it has nothing to compare against. The approaches have evolved through three broad generations, and the current one is genuinely surprising.

The earliest methods used hand-crafted natural scene statistics: the observation that undistorted photographs have characteristic statistical regularities, and that distortion disturbs them in measurable ways (arXiv 2511.09948, 2025). Clever, but brittle.

Deep learning replaced the hand-crafting. CNN-based models learned quality-aware representations directly from image patches, with later work adding multi-task training to predict both the quality score and the distortion type at once. Better, but still fundamentally supervised: the model learns the distortions you show it.

The current frontier is stranger and more interesting. Researchers have repurposed CLIP, a vision-language model trained to connect images and text, for quality assessment by measuring the similarity between an image's embedding and textual prompts such as "a good photo" or "a bad photo" (arXiv 2511.09948, 2025).

Sit with that for a second. The method is to ask a model that learned language and vision together whether a picture is closer to the idea of a good photo or the idea of a bad one. You are querying quality with words. It works better than it has any right to, largely because CLIP absorbed an enormous amount of the world, including an implicit sense of what a well-made photograph looks like.

Why is no-reference assessment so hard?

It is hard because the models overfit to whatever distortions they were trained on, and the world contains distortions they have never seen. The research is blunt about this: most approaches rely on supervised training with specific quality datasets, often overfitting to dataset-specific distortions or content, and this generalization bottleneck severely limits their applicability in real-world systems (arXiv 2511.09948, 2025).

Read that carefully, because it is more damning than it first appears. A no-reference model trained on JPEG artifacts and Gaussian blur will confidently score an image degraded in some other way, and its confidence tells you nothing. The failure mode is not "the model says it does not know." The failure mode is a plausible number that is wrong.

This is a place where the marketing tends to outrun the science. "AI-powered quality scoring" sounds settled. It is not. No-reference IQA remains, in the words of the field itself, largely unsolved. Knowing that is more useful than any individual model, because it tells you how much weight to put on the score.

Can you trust an AI quality score?

As an instrument, yes, with a clear understanding of its blind spot. As an oracle, no. And there is a specific failure mode that catches people, which almost nobody writes about.

If you optimize directly against a metric, the system learns to satisfy the metric rather than the human. Make LPIPS your loss function and the model will find the images that minimize LPIPS, which is not the same set as the images that look good. You get outputs that score beautifully and have a subtle wrongness to them, an over-smoothed texture here, a strange sharpening artifact there. The metric was a proxy for human judgment, and you have just spent all your optimization pressure on the gap between the proxy and the thing.

This is why we do not use a single score in production. For our own codecs, we use perceptual distance alongside content-specific checks, and we treat the perceptual measure as a floor rather than a target. The distinction matters: a floor asks "is this good enough to ship," which is a question a metric can answer usefully. A target asks "make this number as good as possible," which is an invitation to game it.

Every metric is a lens. Every lens has a blind spot. The competent practice is to know where your lens is blind and put a second instrument there, not to find the one true metric and trust it completely.

Which metric should you actually use?

It depends on the content and on whether you have the original. Here is the practical version.

If you are compressing, you have the original, so use a full-reference metric. LPIPS or DISTS for natural photographic content, where they track human judgment closely. SSIM where structural fidelity is the actual requirement, such as medical imaging, and it earns its place there.

If your codec reconstructs rather than reproduces, pixel-correspondence metrics break entirely, and you need perceptual distance plus content-specific checks. A reconstructed face that looks right will fail a pixel comparison badly, because the pixels moved, which was the design.

If you genuinely have no original, you are in no-reference territory, and you should hold the score more loosely than the tooling encourages you to. Use it as a screen, not a verdict.

The meta-point is that choosing a metric is a design decision, not a neutral technical detail, which is the argument at the heart of [INTERNAL-LINK: perceptual media optimization → Perceptual pillar].

Where is this heading?

The field is converging on something interesting: quality assessment that understands quality is not uniform across an image.

The QoMEX 2026 Grand Challenge on video quality assessment is built around asymmetric encoding, where region-of-interest methods allocate higher bitrates to perceptually important regions while compressing background areas more aggressively. The challenge exists precisely because the reliability of existing quality metrics on such content remains underexplored (arXiv 2607.04606, 2026).

That is the per-asset thesis arriving in the academic literature. Spend the bits where a person looks, compress hard where they do not, and then face the fact that your quality metrics were built assuming uniform treatment and now need to catch up. The research community is discovering, from the metrics side, the same thing perceptual optimization has been arguing from the compression side: human attention is not evenly distributed, and any system that pretends otherwise is leaving quality or bytes on the table.

Frequently Asked Questions

How does AI assess image quality?

In two fundamentally different ways. Full-reference assessment compares a distorted image against its original by measuring the distance between deep neural features. No-reference assessment judges an image with no original to compare against, learning what "good" looks like from training data.

What is no-reference image quality assessment?

No-reference (or blind) assessment scores an image without access to the original. It is much harder than full-reference, and research shows current models overfit to the distortions in their training data, which severely limits real-world reliability.

How does LPIPS work?

LPIPS passes both the original and the distorted image through a neural network and measures the distance between their internal feature representations across several layers. Because those features encode perceptual content rather than raw pixels, the result correlates around 0.85 with human judgment.

Can AI quality scores be gamed?

Yes. If you optimize directly against a metric, the system learns to satisfy the metric rather than the human, producing images that score well and look subtly wrong. This is why a perceptual metric works better as a quality floor than as an optimization target.

Which image quality metric should I use?

It depends on the content and whether you have the original. LPIPS or DISTS for perceived quality on natural content, SSIM where structure is the requirement, perceptual distance plus content-specific checks for reconstruction, and healthy skepticism toward no-reference scores.

The takeaway

Machines judge visual quality in two ways, and the two are not interchangeable. With the original in hand, learned metrics that compare deep features now track human judgment well enough to steer by. Without it, the problem is still open, and any confident score deserves suspicion.

The deeper lesson applies to both. A quality model is an instrument, not an authority. It has a range where it is reliable and a range where it is not, and the difference between a team that ships good images and a team that ships images with good scores is knowing which range they are in.

That is why we treat perceptual measurement as a floor to clear rather than a number to maximize, and why every optimization we ship is verified before it goes out. The metric is how you check the work. It is not the work.