feat: Use resize when resampling images with up to 4 megapixels

[72374]

The thumbnail-function can resample images quickly, but with very noticeable aliasing when it is used to resample images to a close resolution (~80% of the original resolution per dimension, for example). Even if the resolution-difference is large, the quality of the resampling is rather flawed. The function seems to be for quickly resampling lots of images into tiny preview-images (less than about 2 x 2 cm , when considering that "Thumbnail" is a reference to the intended size), for which the quality is not particularly important, and the resolution-difference is usually large (~25% of the original resolution per dimension, for example).

The amount of processing/time (including decoding and encoding) required to resample images, depends on the resolution of the original image.

By using the thumbnail-function only for images with a high resolution, many images will be resampled with the better resampling of the resize-function, while images with a high resolution, will still be resampled more quickly.

Quality-comparison:

Original image (2626 x 974 | 2.558 MP | 2.7MB)

main-branch:

This PR:

[iequidoo]

Just in case, the previous discussion is at #6815

[72374]

One question that might be relevant for this PR:
I saw that the image-editor of Delta Chat for Android has 2 resolution-limits, and those are either higher than one or both media-quality settings in Chatmail Core.
Do those apply to the image given to Chatmail Core, or is it only for display in the editor?

If those apply to the image, that could result in downscaling twice; first by the image-editor, to 1536x1536, and then by Core, to 1280x1280, and it would also mean that the change in this PR would apply to all images edited in the Android-app, because the upper limit is 1536x1536.
If that is the case, it could be a good idea to make the limits identical, so that downscaling is only necessary once; which would improve quality and reduce processing-time. But if those should not be changed, the limit in this PR should probably be reduced to 1400x1400 or less, because otherwise this change would effectively be identical to always using bilinear-resampling for the Android-app of Delta Chat.

---

I set this PR back to draft, because #7822 can change what is appropriate to do in this PR. With those changes, images with a resolution of up to 1536×1066¹ would not be resampled, because that is a lower resolution than 1280x1280².

Footnotes

1. 768×533 for the lower limit in the image-editor. ↩

2. 640x640 for the lower media-quality-setting. ↩

[72374]

I think, that the "4+ times slower" performance-test-result may be outdated, or perhaps was measured incorrectly.
I recently compared the performance of these resampling-functions (though i did not use Chatmail Core for that), and while the thumbnail-function was twice as fast as the resize-function with triangle-resampling when using a debug-build, it was only about 10% faster when using a release-build.
triangle-resampling with a debug-build was 4 times as slow as with a release-build though; which matches the mentioned time-difference.
Can someone with a sufficiently slow computer test the performance of simply always using the resize-function with triangle-resampling, while making sure that Chatmail Core and the "Image"-library have been built as a release-build; and also mention the used CPU, so that it is known what the result is for?

[iequidoo]

Can someone with a sufficiently slow computer test the performance of simply always using the resize-function with triangle-resampling, while making sure that Chatmail Core and the "Image"-library have been built as a release-build; and also mention the used CPU, so that it is known what the result is for?

Not sure my computer is slow, but i'm going to recheck this. Maybe you can also try QEMU.

See #6815 , the reason why it was closed wasn't only performance, but also reduction of sharpness when using Triangle which is a bilinear interpolation AFAIU. Quality in image resizing isn't a "linear function" unfortunately, so when we improve one aspect, we worsen another. It's good that Triangle is a fair algo in relation to source pixels, OTOH the resulting image is more smoothy (because every source pixel now contributes to multiple destination ones; this is an unsolvable problem overall), images are even get smaller in byte size, so there's definitely a significant loss of information. To compensate this i propose to increase {BALANCED,WORSE}_IMAGE_SIZE limits at least. How much -- measurements are needed.

EDIT: @72374 , nothing has changed for me since #6815 (comment), have resize() ~3.06 times slower.

... The code comment says it's 4+ times slower, obviously it depends on hardware and i'm not sure the difference is bigger on old hardware, probably on new one you just won't notice the difference.

... Also i see more blur and less details in real images. An improvement i can notice only on synthetic ones which trigger aliasing effects on purpose. I don't want to say that Triangle is worse, but it definitely needs an adjustment of the dimension limits.

... One can use this for testing:

                    let now = std::time::Instant::now();
                    let new_img = if crate::tools::time() % 60 < 30 {
                        info!(context, "thumbnail()...");
                        img.thumbnail(target_wh, target_wh)
                    } else {
                        info!(context, "resize()...");
                        img.resize(target_wh, target_wh, image::imageops::FilterType::Triangle)
                    };
                    info!(context, "time: {}", (std::time::Instant::now() - now).as_nanos());

[72374]

Thank you; for checking, and the code to measure the time. :)

I used the code, to measure the difference for sending an image in JPG-format with a resolution of 12.5 MP:
The CPU that i used, is a "Core i7-12700K" (a rather fast desktop-CPU), while using the energy-saving performance-profile, which made it about 10% slower.¹
The difference, for only the resampling-functions, was consistently about 330% (~462 ms / ~140 ms), but the difference between the beginning and end² of the check_or_recode_to_size-function, was only ~40% (~1083 ms / ~772 ms).
When using only a single efficiency-CPU-core, the difference for the entire function was ~48% (~1339 ms / ~902 ms).
So, the practical difference was 40-50% (0.31-0.43 s), for an image with a very high resolution.

Regarding the proposal to increase the resolution-limit:
I would appreciate it, if the resolution-limit for BALANCED would be set to 1440.
1440 * 1440 is the same number of pixels as in the "FullHD"-resolution (1920x1080), which is a very common screen-resolution; and images with that resolution can look reasonably sharp without artificial sharpening, when displayed at typical notebook-PC-screen-like sizes (~13-18" | 122-169 ppi), or on larger screens that are viewed from a further distance. The "FullHD"-resolution is especially common for televison-screens, so it would be ideal for showing the images on those (which is relatively easy with wireless screen-mirroring).
My primary use-case, when sending images, is sharing anime-style images (often over 8 MP) and images of videogames (usually 3840x2160/UltraHD) with friends and family; 1920x1080 would fit nicely for the game-images, as it is exactly a quarter of the original resolution.

smaller in byte size, so there's definitely a significant loss of information.

The file-size-difference is probably both due to the averaging of the triangle-function, and the aliasing caused by the thumbnail-function (which is mostly in a grid-pattern, that becomes finer as the resolution-difference to the original becomes larger); though it does not mean that the image with the larger file-size contains more correct data, just that it is differently well compressable with the current encoder³.
Similarly, applying Unsharp Mask (a sharpening-effect) to an image, can also increase the file-size when it is encoded in JPG-format, even though the resulting image contains less correct data when compared to an encoding of the original.

An improvement i can notice only on synthetic ones which trigger aliasing effects on purpose.

See the comparison that i added to the pull-request-description. It is simply a screenshot of text and UI in a text-editor, cropped to fit within 2.56 MP, so the currently set condition to use triangle-resampling applies.
Neither the typed text nor the UI look reasonably good when resampled with the thumbnail-function.

The comparison in #6815 (comment) also shows some other visual issues (wavy lines instead of straight, angled lines; and rough lines that should be smooth).

Footnotes

1. I can also do measure that with a "Core i5-8250U"-CPU (a ~10 year old medium-performance CPU with 4 cores) in energy-saving-mode in a notebook-PC, if that is considered relevant; though i probably will need a few hours to do that. ↩

2. After the 3 lines using create_and_deduplicate_from_bytes. ↩

3. It is possible to encode images with a higher quality/accuracy with a lower file-size, even when still using the JPG-format.
   For example: the "JPEGLI"-encoder can often reduce the file-size by about 20%, for similar or better quality; and the "JXL"-codec can losslessly encode and restore JPG-images with a similar reduction in file-size. ↩

[iequidoo]

See the comparison that i added to the pull-request-description. It is simply a screenshot of text and UI in a text-editor, cropped to fit within 2.56 MP, so the currently set condition to use triangle-resampling applies.
Neither the typed text nor the UI look reasonably good when resampled with the thumbnail-function.

Agree, switching to triangle may make sense. However, it's not like the thumbnail output is completely unreadable, i still see all numbers. Ok, not only "synthetic" images are improved, but also text. Btw, as for screenshots, see #7992 -- we don't want to scale down huge non-JPEGs unconditionally and screenshots are usually non-JPEGs. However, if it's an image+text screenshot, it may require scaling down because the image may be heavy.

Maybe a compromise also -- not to loose sharpness of real photos -- is to increase the limit not to 1440, but even more and a bit decrease the JPEG quality, say, to 65. However, this way the amount of JPEG metadata will grow, so we should be careful here. We can even use different JPEG quality for resized and not resized images.

So, this is a complex topic requiring many experiments, we can't change the used algo only looking at a couple of test images which we have here. Overall, i think we should maintain the same image size as before, at least not to cause new complains from users such as loss of quality or increased traffic. If we want to optimize quality/traffic tradeoff, that should be a separate change.

[72374]

Here another comparison of images of text; this one in a language with more detailed characters; 日本語 - Japanese:

Image-comparison

Original:

Thumbnail:

Triangle:

Chatmail Core 2.52 - Thumbnail:

With the thumbnail-function, the "本"-kanji in the third row are changed to "木"; the bottom line of "日" is invisible in the second row after the indented rows; and "語" is changed into various different shapes depending on the position.
With triangle-resampling, the quality could also be better, but it is still good enough to recognise the kanji visually; and those can be quickly recognised again, because the shape is consistent.

Such issues with text, were also noticeable on a photo, that i compared for this.

Regarding sharpness-reduction:

More sharpness is not the same as better quality; for photos and anime-style art (typically based on line-art) in particular, artificial sharpening tends to result in less accurate depth-perception (for example: making things appear closer to the front than those should be, sometimes making things look like flat layers of paper, and/or visually detaching things from the place where those are supposed to be attached to).
Uneven sharpness caused by aliasing also does tend to attract attention, making it difficult to intentionally focus on other parts of an image.
On the photos i compared, the sharpness added by the thumbnail-function was just aliasing, that caused some of these quality-flaws; it was similar for anime-style art.

Though, #7822 increased the maximum pixel-count for images with an aspect-ratio of 4:3¹ by 33%, by 78% for 16:9, and more, if the image is less square; which means, that after a switch to triangle-resampling (without further changes), many images will still be much clearer than those encoded with Chatmail Core 2.52, anyway.

Regarding increasing the limit above 1440:

To me, that does not seem appropriately "balanced", because screens with a resolution higher than 1920 * 1080 are less common², so the improvement would often only be relevant when zooming in. I think that the current limit is acceptable (though not good) for practical use; but if an increase is wanted, it would be good to do that before a version of Delta Chat with #7822 is published, to not increase it twice within a short time. Setting it to 1440, would increase the pixel-count by up to 25%.

Regarding the size of JPEG-metadata:

The proportion of it increases, not the amount; though, it is a relatively tiny amount of data either way. So it should not be practically relevant.

Regarding the need for more quality-testing:

I would agree, if this was about changing a reasonably good resampling-algorithm (like Triangle, CatmullRom, Lanczos3, or area-sampling) to another one, but the thumbnail-function is a performance-hack (for generating tiny preview-images), that does cause many visual flaws, and is not suitable for general image-resampling.

About performance:

I think that the < 50% time-difference is not an issue, and Triangle-resampling can always be used; though if there are reasonably common smartphones on which the difference is much larger (and practically relevant), using a resolution-condition could still be an appropriate temporary workaround.

This should be about all that i want to write about this currently. I tried to keep it short, but there is too much to consider. ^^'

Footnotes

1. A typical aspect-ratio for camera-sensors. ↩

2. Source: https://store.steampowered.com/hwsurvey/ ↩

[72374]

I adjusted this a little, because, according to the results of the time-measurements, the difference is practically much smaller. Though, as mentioned, i think that always using triangle-resampling should be fine regarding performance.

[iequidoo]

I've measured the whole chat::send_msg() real time costs (it's what the user observes) for Triangle and for me they are up to 59.0% greater even for 6 Mpx images than ones for thumbnail(). For 4 Mpx image the worst result was +55.8%. I think it may be ok considering that 4 Mpx isn't that much. Testing on some old hardware is needed. But i see that sometimes images are 5% less in byte size, so i'd increase BALANCED_IMAGE_SIZE to 1312. Reducing traffic isn't necessary here, it's better to compensate the blur effect, even if a little.

[r10s]

thanks a lot for the detailed discussions!

where are the images we're optimising for here are expected to come from? 4 megapixel cameras are outdated since about 20 years - and for avatars we're using resize() already. what is the expected scope? png files and diagrams? images edited by android already?

otoh, as it does not affect most images, we can maybe be more relaxed :)

i am wondering about the only 50% slower - at #6815 (comment) we were talking about 3-4 times slower. maybe that needs to be double checked, tho also 3-4 times slower may be fine as we're not using the expensive algorithm for large images from the camera

[iequidoo]

where are the images we're optimising for here are expected to come from?

I think the PR is for screenshots mostly. At least the PR description provides screenshots.

i am wondering about the only 50% slower - at #6815 (comment) we were talking about 3-4 times slower. maybe that needs to be double checked, tho also 3-4 times slower may be fine as we're not using the expensive algorithm for large images from the camera

That time i only measured time spent on resizing, it's indeed still ~3 times slower with Triangle for me (also rechecked recently). But it's not what the user observes, the whole chat::send_msg() should be measured and it gets only ~1.5 times slower. Still, this is noticeable if i try to send photos from my 16 Mpx camera, so limiting the image size may make sense. Anyway, i'd prefer being neutral about merging this PR because there may be users having older hardware, maybe we should look for other options of improving the overall image quality first.

[72374]

There are many sources for images with a resolution below 4 MP; for example:

• Zoomed in photos (at least when the camera is not capable of sufficient optical zooming)
• Screenshots of videogames/apps/videos (FullHD is 1920x1080, and QHD is 2560x1440; both are below 4 MP.)
• Digital art
• Cropped images from any source
• Photos made while a camera(-app) was set to store images at a lower resolution (useful for reducing the required storage-space).
• Images received via a Chat-app.
• The World Wide Web.

Though, ideally, a better resampling-algorithm, than the thumbnail-function, should always be used; so i prefer to figure out the highest currently acceptable¹ resolution-limit for the condition to use Triangle-resampling.

I think the PR is for screenshots mostly. At least the PR description provides screenshots.

This is intended to improve the quality of images in general, including photos, screenshots, and digital art; though the visual issues are most easily seen with text (also in photos), and other fine details, which is why i used screenshots of text as an example.

Testing on some old hardware is needed.

Indeed. Older hardware could be much slower for some reason, so it would be good to have a practical confirmation of the time-differences.
(I think that my Pixel 3a, with a Qualcomm Snapdragon 670, is probably too fast for this.)

Though, theoretically, when considering a 60% longer total processing-time² while Triangle-resampling is used, and an image was encoded with a processor that has 25% of the processing-speed of a single efficiency-core (time multiplied by 4) of a "Core i7-12700K"-CPU (in energy-saving-mode), the differences would be:

  2.0 MP:  2.7 -  1.7 =  1.0 s
  4.5 MP:  4.0 -  2.5 =  1.5 s
  8.0 MP:  5.1 -  3.2 =  1.9 s
 12.5 MP:  7.0 -  4.4 =  2.6 s
 20.0 MP:  9.9 -  6.2 =  3.7 s
 50.0 MP: 19.2 - 12.0 =  7.2 s
178.6 MP: 58.6 - 36.6 = 22.0 s

(Time between line 2608 and line 2636 in src/chat.rs; with a build without --release, to further slow the speed by 15-20%. 178.6 MP is a little below a memory-limit, that did prevent re-encoding; so could be considered an upper limit.)

Footnotes

1. From what i understand, the time between pressing the send-button, and the image appearing in the chat, should not be too large, as there is currently no indication that image-processing is happening in the GUIs. ↩

2. The percentage-difference seems to be independent of the original resolution, according to my testing. ↩

[72374]

Here an image of digital art, that shows a lot of aliasing, when downsampled with the Thumbnail-function:

Image-comparison for an image in anime-style that is based on line-art (original resolution: 1800 x 2000)

Downsampled with Chatmail Core

1238 x 1376:

Lanczos3-resampling:

CatmullRom-resampling:

Triangle-resampling:

Thumbnail-resampling:

605 x 672 (quality 100):

Lanczos3-resampling:

CatmullRom-resampling:

Triangle-resampling:

Thumbnail-resampling:

Downsampled with other software:

1238 x 1376:

NoHalo-resampling:

Downsampled version from Pixiv:

Original image from Pixiv - 102755657 by Nami; based on characters from Genshin Impact.

[iequidoo]

Though, ideally, a better resampling-algorithm, than the thumbnail-function, should always be used; so i prefer to figure out the highest currently acceptable1 resolution-limit for the condition to use Triangle-resampling.

For high-resolution images thumbnail is obviously better (than for low-resolution) by its definition, and it's even said in the docs:

This method uses a fast integer algorithm where each source pixel contributes to exactly one target pixel. May give aliasing artifacts if new size is close to old size.

The reason of aliasing effects is that for lower-resolution images source pixel weights are very different (from 0.25 to 1). So probably it doesn't make much sense to use Triangle for big images, considering the performance downgrade.

[72374]

Some of the quality-issues caused by the Thumbnail-function seem to become smaller as the resolution-difference increases; in particular, the grid-pattern becomes finer (which can also look worse, depending on the image).
Though, the Thumbnail-function will still cause quality-issues, even when the resolution-difference is huge, because "each source pixel contributes to exactly one target pixel".

I added images that were resampled while using the lower media_quality-setting (resampling at 11% of the original resolution), to the comparison.

Personally, i would not care much about any of the mentioned time-differences.
If i wait 12 or 19 seconds (for images with 50 MP), or, more practically relevant, 4.4 or 7 seconds (for images with 12.5 MP), does not matter much, as i would already be waiting anyway.

[72374]

I tested the performance with a PinePhone¹:

The time-difference, of the send_msg-function, was 60% or less, when Triangle-resampling was used, while using a build made without --release; similar to previous measurements.

  3.6 MP:   4.3 -  2.7 =  1.6 s | 60% longer
  7.4 MP:   7.0 -  4.4 =  2.6 s | 59% longer
 12.5 MP:  11.9 -  7.8 =  4.1 s | 53% longer
 20.4 MP:  18.2 - 12.3 =  5.9 s | 48% longer
 93.0 MP:  72.2 - 51.5 = 20.7 s | 40% longer
158.0 MP: 117.2 - 84.4 = 32.8 s | 39% longer

A release-build should be about 15% faster.

I looked for some smartphones with similarly slow processors, and those seem to usually have cameras with 8-16 MP.

Resampling images with the "Image"-library is apparently as slow as it is, because it does that single-threaded; including the (de)coding of JPG-images.

Footnotes

1. CPU: Allwinner A64, with 4 "ARM Cortex A53"-cores at 1.152 GHz; a CPU that also was slow in 2015, when it was new. ↩

[iequidoo]

158.0 MP: 117.2 - 84.4 = 32.8 s | 39% longer

Thanks for testing, but my opinion as a user is that if i sent screenshots often, i'm not sure i would be fine even with +30%, so i think we should have some linear size limit. According to your measuremts, switching to Triangle gives +2.6s in processing time for 3840x2160 on a rather slow CPU. Looks fine for me, but we need more opinions.

[r10s]

i would be kind of fine to give resize() for images up to 4 mp a a try. iirc, this was the initial idea i commented above.

this would not slow down processing most camera images, but already many screenshots and png would benefit. and the additional time used will be cut, as high-res images are not processed and we do not come in regions where processing takes tens of seconds longer.

i am also fine to increase the image size to 1312 px - this will also slightly improve quality, by a little bit of larger images when resize() is not used, but that's fine. we did not increase that part for years.

the PR is small enough to be reverted if that makes troubles in practise.

otherwise, in general, speed wins. while it is reasonable to optimise within speed constrains, so best quality using same speed, it is a non-goal to have the best thinkable image quality and being slow. if that is really needed, one can always send an image as file. for most user, it just does not matter.

again, thanks a lot for all these thoughts!

[iequidoo]

I agree with r10s about the 4 Mpx limit, we can always increase it later. Personally i don't even have a device with a 4-Mpx screen

[r10s]

revert to use resize() to max. 4 mp, then we can merge this PR

[iequidoo]

Btw, why are the images still smaller in these tests despite they have a larger resolution? Maybe 1312 px isn't enough and we should increase it more?

[72374]

1344 would result in similar file-sizes in these tests.

I do not have much of an opinion about small changes to this.