RTCWHQ: AI Image upscaling with ESRGAN

Started by Krischan, May 04, 2019, 16:09:49

Previous topic - Next topic

Krischan

Did you ever heard about ESRGAN? It is a new method written in Python using neural network machine learning to upscale lores images into 4x hires images on the GPU with CUDA. It is using a pretrained model and the results are really stunning. For example, I've created a HQ version of Return to Castle Wolfenstein (RTCW) and Wolfenstein: Enemy Territory (ET) and created a github repository with all tools I've written and used. Included is a small tool written in Blitzmax to upscale the alpha channel of 32bit TGAs, too. It only works on a nVidia card and the VRAM should be at least 8GB! ESRGAN can run on CPU but it is incredible slow. Limitations are that the texture input size must be below 1024x640 on 8GB VRAM or it won't work. But the best results are achieved with smaller images, with 64, 128, 256 or 512 size.

I've automated the whole process and optimized it to work with RTCW/ET PK3 files as input but it can be used for other purposes, too. Take a look a my new github repository for more details and a lot of background information: RTCWHQ



Original Texture (256x256) vs. upscaled Texture (1024x1024):


Original Alpha Texture (64x64) vs. upscaled Alpha Texture (256x256)


Ingame Screenshots:




So how is it done? You first need some prerequisites and RTCWHQ. Setting up Python is a bit tricky. You first need to install Python and add some libs to it before you can run the scripts. I hope I'll remember this correct:

1. download Python 3.7 (use the Windows x86-64 executable installer which adds the PATH variables)
2. install the CUDA toolkit
3. go to the Pytorch website and select Stable/Windows/PIP/Python 3.7/CUDA 10.0
4. run the two commands shown below the selection box in a commandline window
5. run this command too: pip3 install numpy opencv-python

This installs Python, pytorch, numpy and opencv which are necessary to run RTCWHQ. You can use this method to upscale any kind of images, icons, textures or whatever. There are many models out there and you can even train your own models to achieve even better results but this is rocket science. I've been happy with the model provided in my repository as the results are already sufficient to me.

Thanks to Brucey for the great freeimage.mod. I'd prefer a complete python solution but my tool also works very reliably. Here is the source:
Code (Blitzmax) Select
SuperStrict

Framework brl.basic

Import brl.Pixmap
Import brl.retro
Import brl.eventqueue
Import brl.DirectSoundAudio
Import brl.oggloader
Import pub.freeprocess
Import bah.freeimage

IncBin "cuckoo.ogg"

Global sca:Int = 4           ' scale factor
Global blu:Int = 4           ' gaussian blur
Global con:Float = 0.0       ' contrast fix
Global bri:Float = 0.0       ' brightness fix
Global aut:Int = True        ' autolimbo feature
Global dir:String = ""       ' directory to parse
Global cnt:Int = 0           ' file counter
Global ver:String = "RTCW/ET Upscaler Version 1.0 by Krischan"

If AppArgs.Length > 1 Then

If AppArgs[1] Then dir = String(AppArgs[1])

End If

If dir = "" Or dir = "/?" Then ConsoleDefault()

If AppArgs.Length > 2 Then

If AppArgs[2] > 0 Then sca = Int(AppArgs[2])

End If

If AppArgs.Length > 3 Then

If AppArgs[3] > 0 Then blu = Int(AppArgs[3])

End If

If AppArgs.Length > 4 Then

If AppArgs[4] > 0 Then con = Float(AppArgs[4])

End If

If AppArgs.Length > 5 Then

If AppArgs[5] Then bri = Float(AppArgs[5])

End If

If AppArgs.Length > 6 Then

If AppArgs[6] Then aut = Float(AppArgs[6])

End If

Global sound:TSound = LoadSound("incbin::cuckoo.ogg")
Global channel:TChannel = AllocChannel()

' start the action
Print ver
Print "Gaussian Blur: " + blu + " Pixels"
Print "Brightness:    " + bri
Print "Contrast:      " + con

' convert alpha channels
ConvertAlpha(dir)
Print "Converted " + cnt + " images. Done. Have fun :)"

' play sound and end
CueSound(sound, channel)
PlaySound(sound, channel)
Delay 1000
End

' ----------------------------------------------------------------------------
' Read current Directory
' ----------------------------------------------------------------------------
Function ConvertAlpha:Int(dir:String)

Local path:Int
Local filename:String

Local prefix:String
Local ext:String
Local full:String

'dir = Lower(dir)
path = ReadDir(dir)
If Not path Return False

Repeat

' get next filename
filename = NextFile(path)
full = dir + "/" + filename
ext = Lower(ExtractExt(filename))
prefix = Replace(filename, "." + ext, "")

' skip dotted dirs
If filename = "." Or filename = ".." Or filename = "" Then Continue

' dir? parse recursively
If FileType(dir + "/" + filename) = FILETYPE_DIR Then ConvertAlpha(dir + "/" + filename)

' only parse the unscaled images
If ext = "png" And (Not Instr(Lower(filename), "[s]")) Then

' load unscaled image
Local pixmap:TPixmap = LoadPixmap(full)

'pixmap = ConvertPixmap(pixmap, PF_RGBA8888)
Local w:Int = pixmap.width
Local h:Int = pixmap.Height

' load scaled image
Local scaled:TPixmap = LoadPixmap(dir + "/" + prefix + " [S]." + ext)

If (TPixmap(scaled)) Then

' texture has an alpha channel?
If pixmap.format = 5 Then

' scale and blur
pixmap = ResizePixmap(pixmap, w * sca, h * sca)
pixmap = GaussianBlur(pixmap, blu)

' adjust alpha channel and combine with scaled image RGBs
For Local x:Int = 0 To w * sca - 1

For Local y:Int = 0 To h * sca - 1

Local rgb:Int = ReadPixel(pixmap, x, y)
Local a:Int = GetA(rgb)
Local c:Int = a

' auto contrast for gfx folder (limbo icons)
If aut And Instr(dir, "gfx/") Then bri = 0 ; con = 0.5

' adjust brightness and contrast
c = Brightness(c, bri)
c = Contrast(a, con)

' get RGB pixels
rgb = ReadPixel(scaled, x, y)
Local r:Int = getR(rgb)
Local g:Int = getG(rgb)
Local b:Int = getB(rgb)

' create final RGB value with alpha channel
rgb = CombineRGBA(r, g, b, c)

' only write pixel if within image bounds
If x < (w * sca) And y < (h * sca) Then WritePixel(pixmap, x, y, rgb)

Next

Next

' save 24bit JPEG
'If Instr(prefix, ".jpg") Then

' Print "Converted 24bit JPEG: " + dir + Replace(prefix, ".jpg", "") + ".jpg"
' Local img:TFreeImage = TFreeImage.CreateFromPixmap(pixmap)
' img.Save(dir + Replace(prefix, ".jpg", "") + ".jpg", FIF_JPEG, JPEG_QUALITYSUPERB)
' cnt:+1
' img = Null

' save 32bit TGA
'Else

Print "Converted 32bit TGA: " + dir + "/" + prefix
SavePixmapTGA(pixmap, dir + "/" + prefix, 32)
cnt:+1
pixmap = Null

'EndIf

Else

' save 24bit JPEG
If Instr(prefix, ".jpg") Then

Print "Converted 24bit JPEG: " + dir + Replace(prefix, ".jpg", "") + ".jpg"
Local img:TFreeImage = TFreeImage.CreateFromPixmap(scaled)
img.Save(dir + "/" + Replace(prefix, ".jpg", "") + ".jpg", FIF_JPEG, JPEG_QUALITYSUPERB)
cnt:+1
img = Null

' save 24bit TGA
Else

Print "Converted 24bit TGA: " + dir + "/" + prefix
SavePixmapTGA(scaled, dir + "/" + prefix, 24)
cnt:+1
pixmap = Null

EndIf

EndIf

EndIf

scaled = Null
GCCollect()

' delete original and temporarly used files
DeleteFile(full)
DeleteFile(dir + "/" + prefix + " [S]." + ext)

EndIf

Until filename = ""

' close dir
CloseDir path

Return True

End Function



' ----------------------------------------------------------------------------
' Adjust Brightness of a RGB value
' ----------------------------------------------------------------------------
Function Brightness:Int(c:Int, factor:Float)

c = c + (255 * factor)
If c < 0 Then c = 0 Else If c > 255 Then c = 255
Return c

End Function


' ----------------------------------------------------------------------------
' Adjust Contrast of a RGB value
' ----------------------------------------------------------------------------
Function Contrast:Int(c:Int, factor:Float)

Local contrast:Int = 255 * factor
Local f:Int = (259 * (contrast + 255)) / (255 * (259 - contrast))

c = (f * (c - 128)) + 128
If c < 0 Then c = 0 Else If c > 255 Then c = 255

Return c

End Function



' ----------------------------------------------------------------------------
' Return A value of a given RGB value
' ----------------------------------------------------------------------------
Function GetA:Int(RGB:Int)

Return RGB Shr 24 & %11111111

End Function



' ----------------------------------------------------------------------------
' Return R value of a given RGB value
' ----------------------------------------------------------------------------
Function GetR:Int(RGB:Int)

Return RGB Shr 16 & %11111111

End Function



' ----------------------------------------------------------------------------
' Return G value of a given RGB value
' ----------------------------------------------------------------------------
Function GetG:Int(RGB:Int)

Return RGB Shr 8 & %11111111

End Function



' ----------------------------------------------------------------------------
' Return B value of a given RGB value
' ----------------------------------------------------------------------------
Function GetB:Int(RGB:Int)

Return RGB & %11111111

End Function



' ----------------------------------------------------------------------------
' Return ARGB value of given R,G,B,A single values
' ----------------------------------------------------------------------------
Function CombineRGBA:Int(r:Int, g:Int, b:Int, a:Int)

Return b | (g Shl 8) | (r Shl 16) | (a Shl 24)

End Function



' ----------------------------------------------------------------------------
' Return RGB value of given R,G,B single values
' ----------------------------------------------------------------------------
Function CombineRGB:Int(r:Int, g:Int, b:Int)

Return b | (g Shl 8) | (r Shl 16)

End Function



' ----------------------------------------------------------------------------
' Saves a Pixmap to a TGA File (32bit depth only!)
' ----------------------------------------------------------------------------
Function SavePixmapTGA(pixmap:TPixmap, filename:String, depth:Int = 32, onlyalpha:Int = False)

Local width:Int = pixmap.width
Local Height:Int = pixmap.Height
Local rgb:Int
Local a:Int
Local att:Int = 8

Local x:Int, y:Int

Local f:TStream = WriteFile(filename)

If onlyalpha Then depth = 24

WriteByte(f, 0) 'idlength
WriteByte(f, 0) 'colormaptype
WriteByte(f, 2) 'imagetype 2=rgb
WriteShort(f, 0) 'colormapindex
WriteShort(f, 0) 'colormapnumentries
WriteByte(f, 0) 'colormapsize
WriteShort(f, 0) 'xorigin
WriteShort(f, 0) 'yorigin
WriteShort(f, width) 'width
WriteShort(f, Height) 'height
WriteByte(f, depth) 'pixsize
WriteByte(f, att) 'attributes

For y = Height - 1 To 0 Step - 1

For x = 0 To width - 1

rgb = ReadPixel(pixmap, x, y)

If onlyalpha Then

a = GetA(rgb)
WriteByte(f, a)
WriteByte(f, a)
WriteByte(f, a)

Else

If depth = 24 Then

WriteByte(f, GetB(rgb))
WriteByte(f, GetG(rgb))
WriteByte(f, GetR(rgb))

Else

WriteInt(f, rgb)

EndIf

EndIf

Next
Next

CloseFile f

End Function



' ----------------------------------------------------------------------------
' Gaussian Blur Call
' ----------------------------------------------------------------------------
Function GaussianBlur:TPixmap(tex:TPixmap, radius:Int)

If radius <= 0 Return tex

'clone incoming texture
Local texclone:TPixmap = tex.Copy()

'instantiate a new gaussian filter
Local filter:TGaussianFilter = New TGaussianFilter

'configure it
Filter.radius = radius

Return Filter.Apply(tex, texclone)

End Function



' ----------------------------------------------------------------------------
' Gaussian Blur Type
' ----------------------------------------------------------------------------
Type TGaussianFilter

Field radius:Double
Field kernel:TKernel

' apply Gaussian Blur to pixmap
Method Apply:TPixmap(src:TPixmap, dst:TPixmap)

Self.kernel = makekernel(Self.radius)
Self.convolveAndTranspose(Self.kernel, src, dst, PixmapWidth(src), PixmapHeight(src), True)
Self.convolveAndTranspose(Self.kernel, dst, src, PixmapHeight(dst), PixmapWidth(dst), True)

dst = Null

GCCollect()

Return src

End Method

' Make a Gaussian blur kernel
Method makekernel:TKernel(radius:Double)

Local r:Int = Int(Ceil(radius))
Local rows:Int = r * 2 + 1
Local matrix:Double[] = New Double[rows]
Local sigma:Double = radius / 3.0
Local sigma22:Double = 2 * sigma * sigma
Local sigmaPi2:Double = 2 * Pi * sigma
Local sqrtSigmaPi2:Double = Double(Sqr(sigmaPi2))
Local radius2:Double = radius * radius
Local total:Double = 0
Local index:Int = 0

For Local row:Int = -r To r

Local distance:Double = Double(row * row)

If (distance > radius2) Then

matrix[index] = 0

Else

matrix[index] = Double(Exp(-(distance / sigma22)) / sqrtSigmaPi2)
total:+matrix[index]
index:+1

End If

Next

For Local i:Int = 0 Until rows

'normalizes the gaussian kernel
matrix[i] = matrix[i] / total

Next

Return mkernel(rows, 1, matrix)

End Method

' return function for makekernel
Function mkernel:TKernel(w:Int, h:Int, d:Double[])

Local k:TKernel = New TKernel

k.width = W
k.Height = H
k.data = d

Return k

End Function

' Convolve Gaussian Blur
Method ConvolveAndTranspose(kernel:TKernel, in:TPixmap, out:TPixmap, width:Int, Height:Int, Alpha:Int)

Local inba:Byte Ptr = in.Pixels
Local outba:Byte Ptr = out.Pixels

Local matrix:Double[] = kernel.getKernelData()

Local cols:Int = kernel.GetWidth()
Local cols2:Int = cols / 2

For Local y:Int = 0 Until Height

Local index:Int = y

Local ioffset:Int = y * width

For Local x:Int = 0 Until width

Local r:Double = 0, g:Double = 0, b:Double = 0, a:Double = 0
Local moffset:Int = cols2

For Local col:Int = -cols2 To cols2

Local f:Double = matrix[moffset + col]

If (f <> 0) Then

Local ix:Int = x + col

If (ix < 0) Then

ix = 0

Else If (ix >= width)

ix = width - 1

End If

Local rgb:Int = (Int Ptr inba)[ioffset + ix]
a:+f * ((rgb Shr 24) & $FF)
'b:+f * ((rgb Shr 16) & $FF)
'g:+f * ((rgb Shr 8) & $FF)
'r:+f * (rgb & $FF)

End If

Next

Local ia:Int

If Alpha = True Then ia = ClampColor(Int(a + 0.5)) Else ia = $FF

'Local ir:Int = ClampColor(Int(r + 0.5))
'Local ig:Int = ClampColor(Int(g + 0.5))
'Local ib:Int = ClampColor(Int(b + 0.5))

(Int Ptr outba)[index] = (ia Shl 24)' | (ib Shl 16) | (ig Shl 8) | (ir Shl 0))

index:+Height

Next

Next

End Method

End Type




' ----------------------------------------------------------------------------
' Gaussian Blur Kernel
' ----------------------------------------------------------------------------
Type TKernel

Field width:Int
Field Height:Int
Field data:Double[]

Method getkerneldata:Double[] ()

Return Self.data

End Method

Method GetWidth:Int()

Return Self.width

End Method

Method GetHeight:Int()

Return Self.Height

End Method

End Type



' ----------------------------------------------------------------------------
' Clamp a value to a given range
' ----------------------------------------------------------------------------
Function Clamp:Int(val:Int, minimum:Int, maximum:Int)

If val < minimum

Return minimum

ElseIf val > maximum

Return maximum

Else

Return val

EndIf

End Function



' ----------------------------------------------------------------------------
' Clamp a RGB value to 0...255
' ----------------------------------------------------------------------------
Function ClampColor:Int(val:Int)

If val < 0

Return 0

ElseIf val > 255

Return 255

Else

Return val

EndIf

End Function



' ----------------------------------------------------------------------------
' Default Console output with help and infos about this tool
' ----------------------------------------------------------------------------
Function ConsoleDefault()

Print "==============================================================================="
Print ver
Print "==============================================================================="
Print "Scales TGA32 images with Alpha Channels by factor 4 with a bilinear filter,    "
Print "blurs them with a Gaussian blur filter, applies a brightness/contrast fix and  "
Print "recombines them with a prescaled PNG image. The original image is then replaced"
Print "by the scaled version with the optimized Alpha channel.                        "
Print "==============================================================================="
Print
Print "Usage:   convert.exe /? displays this help                                     "
Print "Usage:   convert.exe [input] [scale] [contrast] [brightness] [autolimbo]       "
Print "Example: convert.exe input 4 4 0.5 0    (general defaults for RTCW/ET textures)"
Print "Example: convert.exe input 4 32 0.9 -0.75  (high contrast for very sharp edges)"
Print
Print "Parameters:"
Print "[input]      Folder to parse (mandatory!)             (empty, for ex. input)   "
Print "[scale]      optional: Scale Factor, depends on model (Default: 4)      integer"
Print "[blur]       optional: Alphamap Gaussian Blur         (Default: 4)      integer"
Print "[contrast]   optional: Alphamap Contrast change       (Default: 0.0)      float"
Print "[brightness] optional: Alphamap Brightness change     (Default: 0.0)      float"
Print "[autolimbo]  optional: contrast=0.5 for gfx/ files    (Default: 1)      integer"
Print "==============================================================================="
Print
End

End Function


Oh, and if you're a fan of ET like me stay tuned for my final ETlaunch tool to launch ET custom maps. It is part of a larger project (ETSE - Enemy Territory Special Edition) which is a complete HQ version of Enemy Territory using ETlegacy with HQ Textures, HQ GUI, NQ Mod, nice additions and up to 64 bots. Now you can play ET as a "single player" against dumb and evil bots. Of course written in Blitzmax ;D
Kind regards
Krischan

Windows 10 Pro | i7 9700K@ 3.6GHz | RTX 2080 8GB]
Metaverse | Blitzbasic Archive | My Github projects

Qube

Looks pretty cool. As I'm lazy, is there a tool whereby I can just give it an image, set the scale and it spits out the result without me having to install / configure loads of things?
Mac Studio M1 Max ( 10 core CPU - 24 core GPU ), 32GB LPDDR5, 512GB SSD,
Beelink SER7 Mini Gaming PC, Ryzen 7 7840HS 8-Core 16-Thread 5.1GHz Processor, 32G DDR5 RAM 1T PCIe 4.0 SSD
MSI MEG 342C 34" QD-OLED Monitor

Until the next time.

Krischan

Yeah, but costs 99 bucks: Topaz Gigapixel. But the results are not as good as ESRGAN IMHO, that's why I didn't buy it though I considered it.

And hey, I spent a lot of time to AUTOMATE the process, so I can demand that you take a look at it at least once. :))
Kind regards
Krischan

Windows 10 Pro | i7 9700K@ 3.6GHz | RTX 2080 8GB]
Metaverse | Blitzbasic Archive | My Github projects