RealSR

From LinuxReviews
Jump to navigationJump to search
Real-World Super-Resolution
Original author(s)Tencent YouTu Lab
  • Xiaozhong Ji
  • Yun Cao
  • Ying Tai
  • Chengjie Wang
  • Jilin Li
  • Feiyue Huang
Initial releasev20200530 May 30, 2020; 5 months ago (2020-05-30)
Stable release
20200818 / August 18, 2020; 2 months ago (2020-08-18)
Repositorygithub.com /nihui/realsr-ncnn-vulkan
Enginencnn (network inference computing framework)
Operating systemLinux, Windows, macOS
Size61 Mib
68 MiB installed
TypeImage up-scaling
LicenseMIT Software License
Org.kde.spectacle.svg

Real-World Super-Resolution via Kernel Estimation and Noise Injection is a photo up-scaling program. It won a contest called CVPR NTIRE 2020 Challenge on Real-World Super-Resolution in "both tracks". A Linux version using Vulkan (realsr-ncnn-vulkan) is available. RealSR does a really good job at up-scaling photos. It is better than Waifu2x when up-scaling photos and comparable to it when up-scaling anime and other cartoons.

The researchers who created RealSR have written a 8 page long paper (PDF) explaining how it works. It's got pseudo-code and formulas and all kinds of details. The very short version of it is: It's really good at up-scaling images.

Installation

wget https://github.com/nihui/realsr-ncnn-vulkan/releases/download/20200818/realsr-ncnn-vulkan-20200818-linux.zip
unzip realsr-ncnn-vulkan-20200818-linux.zip
cd realsr-ncnn-vulkan-20200818-linux
mkdir -p $HOME/bin/realsr-ncnn-vulkan-models
cp -r models-DF2K models-DF2K_JPEG $HOME/bin/realsr-ncnn-vulkan-models
cp realsr-ncnn-vulkan $HOME/bin

Windows and macOS users can acquire the technology from the GitHub Releases Page.

Usage

realsr-ncnn-vulkan options
-i input-path input image path (jpg/png/webp) or directory
-o output-path output image path (jpg/png/webp) or directory
-s scale upscale ratio (4, default=4)
-t tile-size tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
-m model-path realsr model path (default=models-DF2K_JPEG)
-g gpu-id gpu device to use (default=auto) can be 0,1,2 for multi-gpu
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
-x enable tta mode
-f format output image format (jpg/png/webp, default=ext/png)

Using -m to a full path where the model data is stored is required. Creating an alias for that is helpful if you followed the installation instructions above:

alias realsr='$HOME/bin/realsr-ncnn-vulkan -m $HOME/bin/realsr-ncnn-vulkan-models/models-DF2K_JPEG'

That alias lets you simply type realsr -i inputfile.jpg -o outputfile.jpg when you want to convert a file.

Note: The models-DF2K appears to produce bad images when JPEG is up-scaled. models-DF2K_JPEG produces fine images when the original is a JPEG.

Also note: realsr-ncnn-vulkan doesn't actually have any -s options in the current implementation! The default, 4, is your only option. 2, 8, everything else results in invalid scale argument.

Examples

Satiana

Satiana.jpg
1920x1080 screenshot of Satiana from Gabriel Dropout.

We down-scaled a screenshot of Satiana from Gabriel Dropout down to just 200x113 pixels using GIMP:

Satiana-200px.jpg

Up-scaled using ImageMagick convert:

Satiana-200px-convert-800px.jpg

Up-scaled using realsr-ncnn-vulkan:

Satiana-400px-Realrs-DF2K JPEG.jpg

RealSR does a rather impressive job when it's only got a small 200x113 pixel image to work with. That is specially true when you compare it to ImageMagick's convert.

waifu2x-ncnn-vulkan 2x, for comparison:

Satiana-400px-waifu2x-ncnn-vulkan.jpg

realsr-ncnn-vulkan will only let you do 4x up-scaling and waifu2x-ncnn-vulkan will only let you do 2x up-scaling which makes it hard to do a 1:1 comparison. This is why the waifu2x image above is smaller that the images created by realsr-ncnn-vulkan and ImageMagick's convert.

AMD Representative

AMD GIRL.jpg
400 × 548 pixels.

One finer detail stands out as worth considering when it comes to this image:

file size resolution
AMD_GIRL.jpg (original) 112K 400x548
AMD_GIRL_convert_4x.jpg 732K 1600x2192
AMD_GIRL_RealSR_4x.jpg 4.3M 1600x2192

Up-scaled using ImageMagick convert (click to enlarge):

AMD GIRL convert 4x.jpg

Up-scaled 4x using RealSR:

AMD GIRL RealSR 4x.jpg

Upscaled 6x using Gigapixel AI, downscaled with GIMP's Nohalo to match.

GigapixelAI.jpg

I recommend opening different images in different tabs, zooming in, and switching between to really tell the difference. The RealSR image looks really good except for her arm against the green background, that specific part of the image just doesn't look right. Those kinds of artifacts almost never happen with something like waifu2x, but waifu2x would make it look a lot more cartoony overall.

While Gigapixel AI mostly avoids that chromatic artifact, it makes the text at the bottom almost illegible, makes the clipboard ring-binded, and adds what looks like lumps to her necklace. It also assumes way too much about what was actually on her nametag.

RealSR makes her face and hair smoother and higher resolution looking, but at the cost of looking more un-real. Gigapixel really shines when it comes to faces and hair, it seems, although it brings a sort of "jpeg artifact chroma feel" to the new hair and face.

It's interesting to see what happens when the RealSR image is down-scaled and placed side by side with the original: It looks sharper and overall better down-scaled down to the original image's resolution.

Original:
AMD GIRL.jpg
RealSR:
AMD GIRL RealSR 4x.jpg

The down-scaled RealSR (using ImageMagick) looks a bit better than the original image.

The Akerman

Original, 595 × 429:

Akerman.jpg

The Akerman appeared in the background of a picture meant to capture something else. There is no higher resolution, this a crop-out of the Akerman from a larger image and 595 × 429 is the area it filled in the original image.

ImageMagick convert:

Akerman convert.jpg

RealRS:

Akerman RealSR.jpg

The Akerman looks better, but not stunning, after being up-scaled with Akerman. There's not much anyone can do to make that thing look stunning, so it's understandable that the improvement does not make a huge difference.

Verdict And Conclusion

RealSR is a really amazing image up-scaling program.

It's simply the, by far, best piece of software for that purpose.

Alternatives

The Waifu2x waifu2x-ncnn-vulkan implementation is almost as good as RealSR or, in some cases, equal to it up-scaling cartoons and drawings. Waifu2x was designed for cartoons, while RealSR is made for real-world pictures.

Footnotes


avatar

Gigith

28 days ago
Score 0++
Wow, awesome article!
Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.