This page provides some background information about the lens testing procedure here at OpticalLimits.
Let’s start with a rather critical theoretical aspect first:
Field Curvature & Lens Centering
Ideally a lens should project a flat real-world object to a flat focus field behind the optics. An image is in focus when this focus plane intersects perfectly with the flat sensor of the camera.
However, this is rarely ever the case because the focus field is wobbly or curved (as in a sphere because most lens elements are spherical). This relates to the complex optical design, which consists of many elements. Below are two illustrations showing the side view of the optical system. The left image shows what we could expect from a modern lens with a mildly wobbly field curvature. To the left is a rather extreme example of a lens with strong field curvature (i.e. from a classic Petzval-design lens).
If you take a “brick wall” image, the corners will be rendered out-of-focus even though the project image may be perfectly sharp on the focus plane. In a less artificial scenario – like a landscape – this can also be visible because the focus field in the corners can bend out of focus if the depth-of-field is insufficient to compensate.
Another more concerning issue is centering defects. No lens is perfectly centered – this is simply a limitation of the manufacturing process. A centering issue may also get worse or develop over time of use because of shocks or because moving parts are wearing out.
Note: Optical image stabilizers are actually doing this purposefully to compensate for angular movement.
Our Test Approach
We test the performance of a lens on the focus plane. Of course, we don’t know what the focus plane of a lens looks like. Thus, we are using focus iterations to “find” the optimum spot where the sensor image intersects with the focus plane.
You may question why we are doing this because the field curvature or de-centering DOES impact the image quality towards the corners:
- The field curvature is often mild and does not affect image quality unless you really shoot the brick wall or a single test chart. Thus the “brick wall” approach is often very misleading. Do you really shoot brick walls or test charts for fun/a living? No, you don’t. In real life, the depth-of-field will hide these mild imperfections.
- If it’s obvious that a lens produces a strong field curvature, we’ll simply mention it – or in the worst case, provide a “single image” chart to illustrate the issue.
- The focus slicing technique compensates for focus shifts. Some lenses, especially fast ones, produce residual spherical aberrations (=focus shifts) when stopping down. This just means that the point of optimal focus moves to a different slice.
- The focus slicing technique compensates for de-centering – at least to a certain degree.
- No matter how accurate a test setup is, it’s never 100% perfect. Once again, this doesn’t overly matter because we’ll capture the optical point of focus nonetheless.
Below is an illustration of the image areas where we take the measuremets:
- in the dead image center
- near the center (slightly midfield)
- at the borders (on the axis towards the corners)
- in the corners
We are using so-called “slanted-edge MTF measurements” to analyze the resolution of a lens. If you are interested in the background information, take a look at https://www.imatest.com/docs/mtf-measurement-consistency/. We are using Imatest as our primary test tool.
What you can also conclude from this – regardless how accurate we try to be, there’s always a certain error margin. We also provide the LW/PH values (Line-Width per Picture-Height) besides the performance charts. The LW/PH figures are roughly related to the vertical image resolution of a sensor.
e.g. if a sensor has a resolution of 6000×4000 px (=24 megapixels), the maximum LW/PH could be 4000. We tune our testing parameters so that the peak performance is roughly within 10% of this figure (e.g. 3600 in this example).
It also explains why the reviews are NOT cross-system comparable because different sensors have different resolutions and also different characteristics. So don’t use the numbers to compare -say- a Fuji review with a Nikon review. We are actually not overly happy to provide the LW/PH figures because of the mentioned error margin as well as the temptation of users to use the numbers to compare lenses between systems. At some point we removed the LW/PH figures, but this resulted in a storm of protests … so reluctantly we reverted this decision. Anyway … we suggest that you only take the “interpreted” ratings as guidance.
Also – nobody is perfect, and neither are we. We strive to provide the most accurate results, but we do also encourage you to cross-check our findings with other test sites.
Our Rating System
We do not use an “anglo-style” rating system. Thus, if we provide a “good” rating, it is exactly that – good. We do use the whole scale, and yes, we have rated lenses with a poor/1-star rating.
Also keep in mind that we perform a technical analysis. This doesn’t mean that you can’t take great pictures with a lens that we rated at the lower end. Beauty isn’t in pixel peeping but in your creativity, and a technically perfect image may be useless without it.
Mind your Scope
We use fairly high-megapixel sensors in our reviews. This may not be your situation. If you are using a -say- 24 megapixel full-frame sensor for still life pictures, your lens quality requirements don’t need to be excessively high. Most modern lenses work just fine at such a resolution.
It’s even more relevant with video. 4K video is just 8.3 megapixels (3840 x 2160 px). You can almost use a Coke bottle as a lens, and the result would still be fairly sharp – well, almost. Even 8K video (7680 x 4320 px = 33.2 megapixels) isn’t excessively ambitious for a good lens, and resolution isn’t quite as critical because videos are about motion – you can’t really meaningfully pixel peep a video.