Waifu2x

From LinuxReviews
Jump to navigationJump to search
Org.kde.spectacle.svg

waifu2x is a widely praised image up-scaler focused on up-scaling 2D anime images. The original version is Windows-only and it requires CUDA which means you need an NVidia GPU. GNU/Linux users can choose between several versions. There's a Vulkan implementation that works and does a really good job at up-scaling and a OpenGL implementation that does not seem to do anything at all.

There's also a Nvidia CUDA version (for those who find proprietary drivers to be acceptable) and a Caffe version. We have not tested those.

The Myriad Of Versions

The original author of waifu2x wrote it for Microsoft Windows with Nvidias proprietary CUDA API as a requirement. That Japanese author, who goes by the GitHub handle nagadomi, describes the original waifu2x as

"Image Super-Resolution for Anime-style art using Deep Convolutional Neural Networks. And it supports photo."

The original waifu2x code has been forked by a whole lot of people who mostly produced other Windows-only versions which were later abandoned. It is therefore hard to find any actually useful code or version which runs on GNU/Linux.

The waifu2x-ncnn-vulkan implementation described below runs on GNU/Linux and it works great, the waifu2x-glsl implementation runs on Linux but it's either broken or a bad implementation.

waifu2x-ncnn-vulkan

waifu2x ncnn Vulkan
Original author(s)是 nihui
Stable release
version 20200818 / August 18, 2020; 2 months ago (2020-08-18)
Repositorygithub.com /nihui/waifu2x-ncnn-vulkan
Bugtrackergithub.com /nihui/waifu2x-ncnn-vulkan/issues
Written inC++
Enginencnn (network inference computing framework)
TypeImage up-scaling
LicenseMIT Software License

waifu2x-ncnn-vulkan is your best Waifu2x option on GNU/Linux as you will see if you examine the various images below.

A pre-compiled Linux binary is available on the GitHub releases page:

wget https://github.com/nihui/waifu2x-ncnn-vulkan/releases/download/20200818/waifu2x-ncnn-vulkan-20200818-linux.zip
unzip waifu2x-ncnn-vulkan-20200818-linux.zip
cd waifu2x-ncnn-vulkan-20200818-linux
mkdir -p $HOME/bin/waifu2x-ncnn
# Copy models
cp -r models-cunet  models-upconv_7_anime_style_art_rgb  models-upconv_7_photo  $HOME/bin/waifu2x-ncnn
# Copy binary
cp waifu2x-ncnn-vulkan $HOME/bin

You can, alternatively, compile it from source:

mkdir $HOME/src; cd $HOME/src
git clone https://github.com/nihui/waifu2x-ncnn-vulkan.git
cd waifu2x-ncnn-vulkan
# Acquire the sub-modules
git submodule init
git submodule update
# Create and enter build directory
mkdir src/build
cd src/build
cmake ..
# -j$(nproc) makes it use all the CPU cores
make -j$(nproc)

That should create a waifu2x-ncnn-vulkan executable. You'll need that and the models/ folder in the git source tree root (a model needs to be specified with -m when you use it). You can install the fine binary you compiled from source and the models this way:

# Assuming you're still in src/build,
# waifu2x-ncnn-vulkan/ is where you want to go
cd ../..
# Make a folder for it
mkdir -p $HOME/bin/waifu2x-ncnn
# Copy models
cp -r models/* $HOME/bin/waifu2x-ncnn
# Copy binary
cp src/build/waifu2x-ncnn-vulkan $HOME/bin

The options you can/should use are:

-i input-path input image path (jpg/png/webp) or directory
-o output-path output image path (jpg/png/webp) or directory
-n noise-level denoise level (-1/0/1/2/3, default=0)
-s scale upscale ratio (1/2, default=2)
-t tile-size tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
-m model-path waifu2x model path (default=models-cunet)
-g gpu-id gpu device to use (default=auto) can be 0,1,2 for multi-gpu
-j load:proc:save thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
-x enable tta mode
-f format output image format (jpg/png/webp, default=ext/png)

You have to specify a "noise model" to use. Making an alias to the "default" (which otherwise requires it to be in the folder you are running it from) makes it easier:

alias waifu2x-ncnn='$HOME/bin/waifu2x-ncnn-vulkan -m $HOME/bin/waifu2x-ncnn/models-cunet'

waifu2x-ncnn-vulkan produces decent results with no options beyond the path to the default model. It does come with two other models: models-upconv_7_anime_style_art_rgb and models-upconv_7_photo. They may produce slightly better results for anime and photos.

waifu2x-glsl

waifu2x-glsl
Original author(s)Yuichiro Nakada
Stable release
2018-07-10 / July 10, 2018; 2 years ago (2018-07-10)
Repositoryhttps://github.com/yui0/waifu2x-glsl

The waifu2x fork waifu2x-glsl compiles and runs on GNU/Linux (and Windows and macOS). It claims to use the cross-platform glfw library to do OpenGL assisted up-scaling using the waifu2x algorithm.

waifu2x-glsl runs and claims to do something using the GPU (it probably does something). It does not appear to be actually using the waifu2x algorithm or doing anything other than plain up-scaling. The resulting images are .. on par with, or perhaps worse, than up-scaling with ImageMagick's convert or the GNU Image Manipulation Program. You can compile it and try it using these instructions, but it would probably be a complete waste of time.

The latest release is from 2018.

waifu2x-glsl does not appear to do a very good job at up-scaling. Perhaps it doesn't actually use the waifu2x algorithm.

The dependencies you need to compile it on Fedora are:

sudo dnf install glfw-devel mesa-libgbm-devel libdrm-devel mesa-libGL-devel mesa-libGLU-devel mesa-libEGL-devel mesa-libGLES-devel

Get the equivalent build dependencies on for your favorite distribution if it's something else like Debian.

git clone https://github.com/yui0/waifu2x-glsl.git
cd waifu2x-glsl
make

The waifu2x_glsl binary is placed in the root of the source tree. There is no Makefile so you can't run make install. You have to install it yourself.

mkdir -p $HOME/bin/waifu2x-glsl
cp noise1_model.json noise2_model.json $HOME/bin/waifu2x-glsl
cp waifu2x_glsl $HOME/bin

You have to use one of the noise model files with it. waifu2x-glsl appears to do a better job with the scale2.0x_model.json from WL-Amigo/waifu2x-converter-cpp than it does with the included noise models.

waifu2x-glsl will by default look for a noise1_model.json file in the current folder where it is executed. Other files in other locations can be specified by adding a -m path/file.json parameter when you run it.

The desired up-scaling size is specified with -s (2 for 2x, 4 for 4x etc) and an output file needs to be specified with -o.

Note that you can not just place waifu2x_glsl and noise1_model.json in $HOME/bin and expect that to work. It looks for a model in the current working folder, not the folder where the binary lives. You can make put both files in $HOME/bin and make an alias:

alias waifu2x-glsl="$HOME/bin/waifu2x_glsl -m $HOME/bin/waifu2x-glsl/noise1_model.json"

That alias makes it possible to up-scale an image by 2x using

waifu2x-glsl -s 2 inputfile.jpg -o outputfile.jpg

Images

128x128px "agt_family" icon

Everaldo Coelho (www.everaldo.com) is a graphics designer who created several free themes such as "Crystal" and "Kids" in the early 2000s. The Kids theme, released in 2003, did not include any SVG images - only icons up to 128x128. "agt_family.png" is one such icon.

Waifu2x has trouble with especially small images with a lot of feathered PNG alpha transparency. Adding a white background with GIMP or other software will usually produce better results, as was done here.
RealSR produced weird artifacts regardless of whether I flattened the image or not. This shows where waifu2x's emphasis on 2D anime images during the training of the neural network comes in.

Original:
Agt family.png

ImageMagick convert: convert agt_family.png -resize 256 agt_family.256px.ImageMagick.png

waifu2x_glsl: waifu2x-glsl -s 2 agt_family.png -o agt_family.waifu2x_glsl-2x.png :

waifu2x-ncnn: waifu2x-ncnn -s 2 -i agt_family.png -o agt_family.waifu2x-ncnn-2x.png:

waifu2x-ncnn
Agt family.waifu2x-ncnn-2x.png
ImageMagick's "convert"
Agt family.256px.ImageMagick.png
RealSR:
Agt family-128px 4x RealRs DF2K JPEG.png
RealSR from 64px:
Agt familyRealSR.png
waifu2x_glsl
Agt family.waifu2x glsl-2x.png
gimp
Agt family 256 gimp.png

Note: RealSR can only do x4 so the RealSR image on the left is down-scaled by 50%.

waifu2x_glsl lacks PNG transparency support. It also appears to be non-functional or not doing a very good job.

the waifu2x-ncnn-vulkan image looks distinctively better than the others. RealSR is also better than GIMP/Convert.
As we begin showing real life photographs it's important to remember that Waifu2x is ALMOST ENTIRELY trained on 2D anime images; the fact that it actually goes nearly toe to toe with RealSR on photographs is impressive.

For non-photographs waifu2x VASTLY outperforms closed-source proprietary algorithms from the likes of Gigapixel and LetsEnhance, even when set to "man-made" on the former and when using the new "digital-art" beta of the latter; both of which produce weird chromatic artifacts and BIZARRE texture artifacts which waifu2x doesn't, and they produce even more than RealSR does.

Here we will include examples of Gigapixel running on wine, which shows how free alternatives compete with a $99 USD piece of closed-source software. LetsEnhance however under performs every other algorithm, and isn't even worth including.

256px WJSN Yeonjung photo

Original (Actual original was a high-resolution photo that was down-scaled for testing purposes):

Happy-yeonjung-256.jpg

GIMP:
Happy-yeonjung-256x2-gimp.jpg
ImageMagick convert:
Happy-yeonjung-convert.jpg
waifu2x-ncnn-vulkan 2x:
Happy-yeonjung-256-waifu2x-ncnn-vulkan-x2.jpg
RealSR:
Happy-yeonjung-256-RealSR.jpg
waifu2x_glsl 2x:
Happy-yeonjung-256-waifu2x.jpg
Gigapixel AI:
Happy-yeonjung-256-gigapixel-scale-2 00x.jpg

Kind of hard to tell what's better here, waifu2x-ncnn-vulkan or RealSR. It's easy to tell that both are better than GIMP/ImageMagick. Gigapixel is MUCH sharper, sure, but it also introduces a lot more aliaisng. Gigapixel AI shines when images are already so high a resolution that you don't notice the extra aliaisng. Also you might not even want your image THAT sharp, and Gigapixel AI doesn't really let you tone down the sharpness even though it gives you a lot of options, you just have to like sharpness.


Satiana

Satiana.jpg
1920x1080 screenshot of Satiana from Gabriel Dropout.

We down-scaled a screenshot of Satiana from Gabriel Dropout down to just 200x113 pixels using GIMP:

Satiana-200px.jpg

GIMP:
Satiana-400px-gimp.jpg
ImageMagick convert:
Satiana-400px-convert.jpg
RealSR 4x, downscaled for comparison:
Satiana-400px-Realrs-DF2K JPEG.jpg
waifu2x-ncnn-vulkan 2x:
Satiana-400px-waifu2x-ncnn-vulkan.jpg

The waifu2x-ncnn-vulkan and RealSR images look much better than the GIMP and ImageMagick results. Note that the RealSR realsr-ncnn-vulkan implementation will only do 4x, so the image above is down-scaled from 800px to 400px. You can click on it to see the original RealSR result. I didn't even run this through Gigapixel AI because all it'd do is make it look like a weirdly textured nightmare.

Verdict And Conclusion

The visible difference between up-scaling the above images in GNU Image Manipulation Program, ImageMagick's convert, waifu2x-glsl, Gigapixel AI, and waifu2x-ncnn-vulkan is a clear case of one isn't like the other: waifu2x-ncnn-vulkan will never introduce color artifacts or weird texturing to images. It might not be the sharpest, but it also never produces a BAD result. Every other one is kind of a gamble to some degree.

We leave you with the following 500x500 image and four only numbered 2x upscaled images of Jennie from the k-pop group Blackpink for your comparison pleasure and/or consideration:

Based Jennie on relationships.jpg

#1:

Based Jennie on relationships-2x-gimp.jpg

#2:

Based Jennie on relationships-waifu2x-ncnn-vulkan-x2.jpg

#3:

Based Jennie on relationships-realsr.jpg

#4:

Based Jennie on relationships-gigapixel-scale-2 00x.jpg

#5:

Based Jennie on relationships-waifu2x.jpg

#6:

Based Jennie on relationships-convert.jpg


Image 1 was made using GIMPs Image ▸ Scale Image, 2 was made using waifu2x-ncnn-vulkan, 3 was made with RealSR, 4 was with Gigapixel AI, 5 was made with waifu2x_glsl and 6 was made using convert -resize.

RealSR seems a lot better suited for some photographs than waifu2x (mostly already-high-resolution ones, and ones with grain), but waifu2x is better at 2D images across the board, and possibly also 3D CGI. Waifu2x seems to be the only algorithm immune to chroma artifacts and the tendency to introduce weird textures to 2D images. Gigapixel is so similar here to RealSR it's hard to justify anybody spending $99 USD on it. Gigapixel AI has a new sister program called Vidoe Enhance AI which actually does still images much better than Gigapixel AI does, but unlike Gigapixel AI it doesn't seem to run under wine.

Alternatives

RealSR, "Real-World Single Image Super-Resolution", is a image up-scaling program designed for photos. It's really good. It is equal to or better than waifu2x for anime and it is better at up-scaling photos. RealSR will only do 4x up-scaling while waifu2x-ncnn-vulkan will only do 2x.

See RealSR for additional information about RealSR.

avatar

Anonymous user #1

9 months ago
Score 0++
Imagemagick certainly did a better job scaling the text. I got #1 correctly as gimp, but other than the text I can't see a difference between #2 and #3.
avatar

Anonymous user #2

3 months ago
Score 1++
I checked the results and it seems that the algorithm is not working at all in this case. The results you got are completely wrong and do not represent the algorithm.
avatar

Gigith

19 days ago
Score 0++
I ran those images through waifu2x myself and, sure enough, the original article was not an accurate depiction of what waifu2x looks like, at all. I edited the article with my images and also replaced the GIMP versions with RealSR, so there was a real variety between upscalers.
avatar

WaiLin

19 days ago
Score 1++
Tell us more about the RealSR.
avatar

Gigith

18 days ago
Score 0++

Oh wow, you already made the article. Awesome. I just added Gigapixel AI.

If anybody has a working Windows install could you test out Video Enhance AI for me and add it? Back when I had Windows I tried it and it blew everything out of the water.

I mean I know it doesn't even run on Linux, but I think it's worth knowing how these programs compete against their paid alternatives. Maybe I'm wrong about Video Enhance, but even on still images it was pretty mind blowing.
avatar

Gigith

18 days ago
Score 0++
I also want to add that it was specifically the 100% mode it had that was so good. 200%, etc, was good too, but the mode set at 100% was what I'm talking about. I think they called the model Gaia or something.
Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.