What Is LPIPS, and When Do Pixel Metrics Stop Working

For decades the two default measures of image fidelity, PSNR and SSIM, worked well enough that almost nobody questioned them. They are cheap to compute, easy to reason about, and reasonably correlated with human judgment on the distortions people cared about in the 2000s: blur, blocking, noise.

Then image generation and reconstruction changed the shape of the problem. A modern super-resolution model or a neural codec does not produce a slightly blurrier version of the original. It produces a new image that looks right but does not line up with the source pixel for pixel. On exactly those outputs, PSNR and SSIM start to disagree with your own eyes.

LPIPS is one answer to that problem. It compares images through the internal features of a trained neural network rather than their raw pixel values, and on reconstruction tasks it tracks human perception far more closely than the older metrics do.

Key Takeaways

• LPIPS (Learned Perceptual Image Patch Similarity) measures the distance between two images in the feature space of a deep network, not in pixel space.
• It was introduced by Zhang and colleagues at CVPR 2018, who found deep features predicted human similarity judgments surprisingly well across many network architectures.
• Pixel metrics like PSNR and SSIM break on generated and reconstructed images because plausible detail that is slightly misaligned is penalized as heavily as real error.
• LPIPS is a lower number for better matches, is not calibrated to an absolute quality scale, and can be optimized against, so it is a comparison tool, not a verdict.
• The practical rule: match the metric to the distortion. No single number, LPIPS included, is a floor you can ship on blindly.

What is LPIPS, in plain terms?

LPIPS is a similarity metric that passes both images through a pretrained convolutional network, reads the activations at several layers, and measures how far apart those feature maps are. Lower means more perceptually similar. The 2018 paper that introduced it tested this idea across many architectures and reported that deep features predict human judgments remarkably well (Zhang et al., CVPR 2018).

The intuition is that a network trained to recognize images has already learned what parts of an image matter. Its early layers respond to edges and texture, its later layers to shape and structure. Comparing two images in that learned space is closer to comparing what a person notices than comparing their raw brightness values one pixel at a time.

In practice you pick a backbone network, AlexNet, VGG, or SqueezeNet were the originals, run both images through it, and take a weighted distance between the normalized activations. The weights themselves were tuned on a large set of human similarity judgments, the BAPPS dataset, so the metric is literally fit to what people said looked closer. That is the "learned" in the name.

Why do PSNR and SSIM stop working on reconstruction?

Because both metrics assume the two images are meant to align pixel for pixel, and reconstructed images are not. PSNR is a direct function of mean squared error, and SSIM compares local luminance, contrast, and structure in aligned windows. Shift a plausible texture by a pixel or two and both metrics record a large error, even though a human sees no problem at all.

This is the core failure. When a super-resolution or generative model hallucinates realistic hair, grass, or skin texture, it invents detail that is statistically correct but not positionally identical to the ground truth. A pixel metric cannot tell the difference between that and genuine damage. It punishes the good reconstruction and rewards a blurry one, because blur sits closer to the average and therefore scores lower error.

There is a formal version of this observation. Blau and Michaeli showed a mathematical trade-off between distortion, how close you are to the original by a per-pixel measure, and perceptual quality, how real the output looks on its own (Blau and Michaeli, CVPR 2018). You cannot maximize both at once. Optimizing hard for PSNR pushes a model toward exactly the smooth, safe, slightly lifeless output that scores well and looks wrong. We walk through this same tension for the older metric in why SSIM falls short of human vision.

[CHART: Line comparison - PSNR vs SSIM vs LPIPS agreement with human ratings across distortion types, higher agreement for LPIPS on generated/reconstructed images - source: Zhang et al. 2018]

When should you actually reach for LPIPS?

Reach for LPIPS when the thing you are evaluating produces new detail rather than a filtered copy of the input: super-resolution, inpainting, generative restoration, style-preserving compression, and neural reconstruction. In those regimes LPIPS was specifically shown to align with human preference where pixel metrics invert it (Zhang et al., CVPR 2018).

It is less essential, and sometimes overkill, for classic, aligned distortions. If you are measuring JPEG blocking at a fixed quality, or the effect of light denoising, SSIM and PSNR still carry useful signal and cost almost nothing to compute. LPIPS needs a GPU-sized model in the loop, which matters when you are scoring millions of assets.

The honest framing is that LPIPS is a better tool for a specific job, not a universal upgrade. We make that same argument about combining measures rather than picking a champion in PSNR vs SSIM vs looks good to humans. Different distortions live in different failure modes, and the metric has to match the mode.

What are the limits of LPIPS?

Real ones, and worth stating plainly. LPIPS is a learned proxy for perception, not perception itself, and it inherits three specific weaknesses: it is not an absolute scale, it depends on the backbone network and the data it was tuned on, and it can be optimized against until the number stops meaning anything.

Start with calibration. LPIPS gives you a relative distance, so 0.08 is closer than 0.12, but there is no fixed threshold where an image becomes "good." The value depends on content and resolution, so a number that means "invisible difference" for one image can mean "visible artifact" for another. It ranks; it does not certify.

Then there is the backbone. Swap AlexNet for VGG and the absolute numbers change, because you are measuring distance in a different learned space. And because the metric was fit to a particular human dataset, it reflects the judgments and content in that set. Push far outside that distribution, unusual textures, medical imagery, extreme HDR, and the correlation with real perception weakens.

The sharpest limit is the one every learned metric shares. Once you train a model to minimize LPIPS directly, you can produce outputs that score beautifully and still look wrong, because the model learns the metric's blind spots. This is the same trap that makes any single score dangerous as a shipping gate, which is why perceptual verification has to combine measures and hold a real floor. That system view is in how AI models judge visual quality.

How does LPIPS fit a perceptual view of quality?

It fits as evidence, not as a verdict. LPIPS matters because it proves a point Inverity builds on: the right measure of image quality is a measure of what humans perceive, and on modern reconstruction that is demonstrably not raw pixel error. The 2018 result made that concrete by beating hand-designed metrics on human agreement across architectures (Zhang et al., CVPR 2018).

But the same result is a warning. If a learned metric can be gamed, then no single number, however perceptual, is safe as a delivery gate. The discipline is to pick the metric that fits the distortion, cross-check it against others, and verify every output against a floor tied to the original rather than trusting one score. Our Neural Media Orchestrator does exactly that: it evaluates each asset, routes it across many possible paths, and refuses to ship anything that cannot clear a structural similarity floor of at least 0.975 against the source. The full reasoning behind measuring perception rather than bytes is in the complete guide to perceptual media optimization, and our test protocol is in how we benchmark.

Frequently Asked Questions

What does LPIPS stand for?

LPIPS stands for Learned Perceptual Image Patch Similarity. It measures the distance between two images in the feature space of a pretrained neural network, weighted by human similarity judgments, and was introduced by Zhang and colleagues at CVPR 2018.

Is a higher or lower LPIPS score better?

Lower is better. LPIPS is a distance, so a smaller value means the two images are more perceptually similar. This is the opposite of SSIM, where higher is better, and different from PSNR, which is measured in decibels where higher is also better.

Is LPIPS better than SSIM?

On generated and reconstructed images, LPIPS agrees with human judgment more often than SSIM, because it tolerates plausible detail that is not pixel aligned. On classic aligned distortions like blur or blocking, SSIM still carries useful signal at a fraction of the compute cost.

Can LPIPS be fooled?

Yes. Because LPIPS is a learned metric, a model trained to minimize it can find outputs that score well but still look wrong to a person. That is why it should be used as one input among several, never as a single shipping gate on its own.

Do I need a GPU to compute LPIPS?

LPIPS runs a pretrained convolutional network on both images, so it is far more expensive than PSNR or SSIM and benefits heavily from a GPU. That cost is a real factor when scoring large libraries, where cheaper metrics often screen assets first.

The point

Pixel metrics did not become wrong. The images changed. PSNR and SSIM were built for a world where the compressed image was a filtered copy of the original, and they still work well there. LPIPS earned its place by handling the world we actually build in now, where models reconstruct detail instead of preserving it.

The lasting lesson is not "use LPIPS." It is that the metric has to match the distortion, and that no learned score is a substitute for verifying every output against a perceptual floor tied to the source. Measure what people see, cross-check it, and never let a single number decide what ships. That principle carries through the complete guide to perceptual media optimization.