Texture alpha channel bug when there is a label around

#1

Here is how the scene should look like.

Here is how it looks like (bug).

Here is the full source code of the scene.

function init()
	-- Commenting this line disables the bug. 
	local position_z = 1
	
	imageloader.load{
		data = sys.load_resource('/resources/image.png'),
		no_async = true,
		listener = function(self, img_res)
			local image_object = factory.create('#image', vmath.vector3(0, 128, position_z or 0), nil, nil, vmath.vector3(256, 50, 1))
			resource.set_texture(go.get(msg.url(nil, image_object, 'component'), 'texture0'), img_res.header, img_res.buffer)
		end
	}

	-- Commenting this line disables the bug. 
	label.set_text(msg.url(nil, factory.create('#label'), 'component'), 'Some text')
end

In the scene I am loading an image from resources using my extension and creating a label.

Notice that if z position of the image is 0 instead of 1 or label is not created - the bug disappears.

Image uses built-in sprite.fp shader and a very simple vertex shader.

Bonus bug
If I try using built-in sprite.vp - game crashes on launch.

ERROR:GRAPHICS: gl error 1282: invalid operation
Assertion failed: (0), function SetConstantV4, file ../src/opengl/graphics_opengl.cpp, line 1478.

Test project.
texture_alpha_bug.zip (21.8 KB)

1 Like

#2

It seems like something wrong with your imageloader loader, because if I replaced your loader with


everything works fine:

function init()
	-- Commenting this line disables the bug. 
	local position_z = 1
	
	-- imageloader.load{
	-- 	data = sys.load_resource('/resources/image.png'),
	-- 	no_async = true,
	-- 	listener = function(self, img_res)
	-- 		local image_object = factory.create('#image', vmath.vector3(0, 128, position_z or 0), nil, nil, vmath.vector3(256, 50, 1))
	-- 		resource.set_texture(go.get(msg.url(nil, image_object, 'component'), 'texture0'), img_res.header, img_res.buffer)
	-- 	end
	-- }
	
	local bytes = sys.load_resource('/resources/image.png')

	-- decode the png to RGBA buffer
	local rgb_buffer, w, h = png.decode_rgba(bytes, true)

	-- change the texture on a sprite to the loaded png
	local header = { width = w, height = h, type = resource.TEXTURE_TYPE_2D, format = resource.TEXTURE_FORMAT_RGBA, num_mip_maps = 1 }

	local image_object = factory.create('#image', vmath.vector3(0, 128, position_z or 0), nil, nil, vmath.vector3(256, 50, 1))
	resource.set_texture(go.get(msg.url(nil, image_object, 'component'), 'texture0'), header, rgb_buffer)

	-- Commenting this line disables the bug. 
	label.set_text(msg.url(nil, factory.create('#label'), 'component'), 'Some text')
end

UPD:
I can’t find premultiply alpha logic in your extension

1 Like

#3

The image file has premultiplied alpha.
If it was an error in my extension. How come creating a label or changing the z position affects it’s work?

1 Like

#4

Nope, and you need to take it into account when you read your file.
For example how @britzl did this:

I have the same result as yours if I turn-off premultiply_alpha flag in my example

local rgb_buffer, w, h = png.decode_rgba(bytes, false)
1 Like

#5

Obviously something is going on with the alpha channel.
But OK, forget my imageloader library. Let’s use Bjorn’s png extension with premultiply flag set to false (already premultiplied).

Again. Why changing the z position of the image affects the alpha channel? Why if I create the label after the png file is loaded - the alpha is wrong. And if I don’t create the label - alpha is OK.

1 Like

#6

No, it’s not.
If I premultiply alpha for your png file (ImageMagick):

convert image.png -background black -alpha Remove image.png -compose Copy_Opacity -composite out.png

image.png.zip (15.0 KB)
This file works fine even with false flag.

UPD:
I can’t answer “why” . It shouldn’t work at all, because:

Defold engine uses premultiplied alpha internally

I don’t want to spend time for investigation of the strange behavior in this case, because we know that the problem in the wrong format of the image.

1 Like

#7

I used the same command to premultiply my file. Your file now has double premultiplication, which hides the bug a bit better.
Go to the render script and change the clear color to white.
Now with png.decode_rgba(bytes, false) and not commenting lines you can see a correct outcome.
x4 zoom

But if you now comment one of the marked lines you would be able to see black outline of double multiplication.

Again, the engine treats alpha channel differently depending on the z position or if there is a label around.

I am not trying to argue, just trying to show the apparent problem in the engine. I am trying to help Defold to be better.

1 Like

#8

Sound really weird. What kind of z value are we talking about?

2 Likes

#9

0 and 1. That’s the first marked line.

1 Like

#10

I understand and thank you for it.

I don’t want to argue I just want to figure out what going on in your case, but now we have too many variables in the example, I want to isolate your issue.

My logic here:

  1. We have custom render_script, custom material, manual image loader - too many places where something could go wrong.
  2. I want to reduce the count of these places.
  3. My first step is removing the manual loader. I just assign texture for the model in the editor. Result: I can’t reproduce the issue. It’s strange, If this is sorting or render pipeline problem it should be the same behavior even if we do so.
  4. Ok, that means that problem somewhere in the loading process. I am trying to replace your loader with loader I know. Result: everything works fine with premultiply_alpha flag (and we have an issue without this flag).
  5. Hmm, Ok, by default png doesn’t have premultiplied alpha. But let’s check your image:
    5.1 Your image:
    image.png.zip (13.9 KB)
    You have pallets image, let’s convert it to RGBA:
from PIL import Image
import numpy
import sys

def main(finput, foutput):
	image = Image.open(finput)
	image.convert("RGBA").save("rgba.png")

if __name__ == "__main__":
	main(sys.argv[1], sys.argv[2])

python3 convert_to_rgba.py image.png rgba.png

Result:
rgba.png.zip (32.7 KB)

Let’s check our example with this new rgba.png -> the same issue. Great! But now it’s easier to analyze our source image.

5.2 Let’s check our image colours.

from PIL import Image
import numpy
import sys

def is_premultiplied(array):
	for element in array:
		for color in element:
			if color[3] != 255:
				if color[0] > color[3] or color[1] > color[3] or color[2] > color[3]:
					return False
	return True

def main(finput):
	image = Image.open(finput)
	array = numpy.array(image)
	if is_premultiplied(array):
		print("Premultiplied!")
		return
	print("Non-premultiplied!")
	

if __name__ == "__main__":
	main(sys.argv[1])

check_is_pemultiplyed.py rgba.png

Result:

Non-premultiplied!

  1. Ok, Let’s premultiply colors then:
from PIL import Image
import numpy
import sys

def is_premultiplied(array):
	for element in array:
		for color in element:
			if color[3] != 255:
				if color[0] > color[3] or color[1] > color[3] or color[2] > color[3]:
					return False
	return True

def main(finput, output):
	image = Image.open(finput)
	array = numpy.array(image)
	if is_premultiplied(array):
		print("Premultiplied!")
		return
	print("Non-premultiplied!")
	for element in array:
		for color in element:
			color[0] = color[0] * (color[3] / 255.)
			color[1] = color[1] * (color[3] / 255.)
			color[2] = color[2] * (color[3] / 255.)

	Image.fromarray(array).save(output)

if __name__ == "__main__":
	main(sys.argv[1], sys.argv[2])

premultiply.py rgba.png premult.png

premult.png.zip (32.3 KB)

Result: everything works fine with premult.png and false flag

Checking:

check_is_pemultiplyed.py premult.png
Premultiplied!


If I made a mistake somewhere in my logic, pls give me know.


Of course, this is still strange behavior but should we spend our time if we know that everything works fine with the right input file? (of course, if I didn’t miss something in my investigation)

4 Likes

#11

The problem starts with the label component rendering with a fixed premultiplied alpha. You would like to use non-premultiplied alpha to render your image. Check out the rendering process in the following pictures:

In your example, the image has z = 1 coordinate, while the label has z = 0 coordinates. For this reason, the image will be drawn later than the label (The order in the red rectangle framed in the first picture). Therefore, as shown in the second image, the setting of the label fix premultiplied alpha overrides your blend function setting (The second picture shows 2 framed red rectangles)

There are several solutions:
1.) You ensure that the label always has a larger z coordinate than the image. Therefore, the blend setting for the image will apply to the image (Try, for example, set the z coordinate of the main object to 2).

2.) You ensure that the label is rendered in a separate defold draw process, so two different render predicates are required. You can then set your own blend function for each render predicate and defold draw procedure (The image and label are currently drawn with the same render predicate)

There may be other ways to set the appropriate blend function for the label component. Maybe another idea?

5 Likes

#12

Thank you for your effort and thanks for pointing out that imagemagick produces incorrect result regarding alpha multiplication. I will use a Python script from now on.

However the issue still exists. I’ve downloaded your premult.png and I am using your last code.
For better visualization I’ve multiplied scale of the image by 5.

If non of the two lines are commented, we can see the correct view.

But if we comment one of the two marked lines, we can see the black outline again.

The same issue - z position or label creation affects how the texture is rendered.

I understand this is visually a small issue, but finding and fixing these little things is what makes a great software great.

3 Likes

#13

Wow, you’ve found the source of the problem! Thank you!

  1. I can’t use different predicate for the label, because I need to mix labels and other game objects in layers.
  2. In my current project, in which I noticed this bug. A combination of image premultiplication and certain positioning of elements overcomes the problem, but that’s a workaround, not a fix.

I would recommend in the engine to either restore the original blend function after a label is drawn or have a setting for font material or font itself to choose blend function (preserve, one | one minus src alpha, src alpha | one minus src alpha). By preserve I mean not setting a blend function.

2 Likes

#14

Great writeup @AGulev. A small note though, I think the function shouldn’t be called “is_premultiplied()”, since the only thing you can safely check if you find values that are not premultiplied. Perhaps better to call is “is_not_premultiplied()”

if (max(col.r, col.g, col.b) > col.a):
    # not premultiplied
else:
    # ambiguous

If the check fails, the answer is ambiguous. For instance, a pixel (1,1,1,254) may be premultiplied, but is might also be the actual source pixel.

@coderm4ster Nice breakdown of the frame. It does indeed seem that there is some inconsistency when we setup the rendering for the sprites. I agree with @sergey.lerg that this should be handled by the engine. Let’s ping @sven and @jhonny.goransson about this.

5 Likes

#15

This is not a typical bug but a feature :slight_smile: All my respect for the Defold guys, but there is not enough capacity to meet all our needs.

What I am making a commercial game is in the final phase. As long as I made several similar problems I had to discover and apply a solution to them. In addition to my modest English knowledge, I have sometimes reported what is blatant. My main point is that the game is ready with what is available. Of course it would be good if Defold knew more things, but I don’t have time to wait for them, so I’ll get to what is :slight_smile:

Fortunately, there are a lot of good people who take Defold in the right direction and report such things. So I’m curious what the solution to this problem is.

Of course I apologize to everyone for my English, I’m sure I’ll write stupid things, but maybe I can do something to make Defold better. Last but not least, I learn to speak English :smile:

5 Likes

#16

Thanks, you are right! Of course, we can be sure only if the image is not premultiplied (in some cases), it was enough in my case, but semantically my function wrong.
Fixed versions of the scripts from the previous post (Maybe it will be useful for somebody.):

from PIL import Image
import numpy
import sys

def is_not_premultiplied(array):
	for element in array:
		for color in element:
			if color[3] != 255:
				if max(color[0], color[1], color[2]) > color[3]:
					return True
	return False

def main(finput):
	image = Image.open(finput)
	array = numpy.array(image)
	if is_not_premultiplied(array):
		print("Non-premultiplied!")
		return
	print("Ambiguous... we can't be sure.")
	

if __name__ == "__main__":
	main(sys.argv[1])
from PIL import Image
import numpy
import sys

def is_not_premultiplied(array):
	for element in array:
		for color in element:
			if color[3] != 255:
				if max(color[0], color[1], color[2]) > color[3]:
					return True
	return False

def main(finput, output):
	image = Image.open(finput)
	array = numpy.array(image)
	if is_not_premultiplied(array):
		print("Non-Premultiplied!")
		for element in array:
			for color in element:
				color[0] = color[0] * (color[3] / 255.)
				color[1] = color[1] * (color[3] / 255.)
				color[2] = color[2] * (color[3] / 255.)

		Image.fromarray(array).save(output)
		return
	print("Ambiguous... we can't be sure.")

	

if __name__ == "__main__":
	main(sys.argv[1], sys.argv[2])

@coderm4ster Perfect explanation! Thank you!

5 Likes