Runtime crash with unknown symbol error?

Questions
#1

The engine crashes with a strange unknown symbol error a few frames after building and launching:

[1:31:43 AM] INFO:ENGINE: Loading data from: build/default
[1:31:44 AM] INFO:ENGINE: Initialised sound device 'default'
[1:31:44 AM] INFO:CRASH: Successfully wrote Crashdump to file: C:\Users\AppData\Roaming\Defold/_crash
[1:31:44 AM] ERROR:CRASH: CALL STACK:
[1:31:44 AM] 
[1:31:44 AM] 
[1:31:44 AM] ERROR:CRASH:  0 0x7FF65DB12129 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  1 0x7FF65DE63E2C <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  2 0x7FF65DEB820E <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  3 0x7FF65DE225B8 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  4 0x7FFC53A751DF <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  5 0x7FFC539EE866 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  6 0x7FFC53A741CE <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  7 0x7FFC53A0F9A2 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  8 0x7FFC53A0D239 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  9 0x7FFC4F84707A <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 10 0x7FFC4F845649 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 11 0x7FF65DCD0CF9 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 12 0x7FF65DD7524D <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 13 0x7FF65DCCDEBB <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 14 0x7FF65DBFD78C <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 15 0x7FF65DBF8017 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 16 0x7FF65DBFDD51 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 17 0x7FF65DAFF528 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 18 0x7FF65DE221D0 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 19 0x7FFC52BC257D <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 20 0x7FFC53A2AF28 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 
[1:31:44 AM] 
[1:31:44 AM] INFO:CRASH: Successfully wrote Crashdump to file: C:\Users\AppData\Roaming\Defold/_crash
[1:31:44 AM] ERROR:CRASH: CALL STACK:
[1:31:44 AM] 
[1:31:44 AM] 
[1:31:44 AM] ERROR:CRASH:  0 0x7FFC53A0F9A2 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  1 0x154AC9C0000 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  2 0x154BC1F55C0 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  3 0x154BC1F55C0 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  4 0x4C <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  5 0x4C <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  6 0x4C <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  7 0x100000001 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  8 0x154BC1F55CA <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH:  9 0x3000 <unknown symbol> <unknown>:0
[1:31:44 AM] ERROR:CRASH: 
[1:31:44 AM] 
[1:31:44 AM] INFO:CRASH: Successfully wrote MiniDump to file: C:\Users\AppData\Roaming\Defold/_crash.dmp

This only occurs after adding more than four of a certain game object to the loaded collection. That game object only contains a single sprite component, which uses a custom material and some custom vertex attributes.

I just pinpointed that the crash occurs when the render script hits the following material swap and draw call:

...
render.enable_material("glow")
render.draw(terrain_predicate, { constants = constant_buffer })
...

Sprites that are drawn with terrain_predicate are first drawn with their default material, then drawn a second time with their glow material. I’m using the same trick with other predicates though, so I’m not sure why this particular one is having trouble. The original material and glow material have the exact same properties set in their material files.

1 Like
Swapping materials with custom vertex attributes?
#2

Oh, that’s annoying.

Stored next to the crash file, is a .dmp file.
If you are also using native extensions, we’d need the dmengine.exe and dmengine.pdb file to match that .dmp file.
Then we should be able to debug that crash dump.

An alternative would be if you could create a small repro case where this happens.

1 Like
#3

I’m not using any native extensions yet, so no problem there.

Here are the _crash and _crash.dmp files:

_crash.zip (1.3 KB)

Further clarification: it’s the draw call in combination with the material swap that causes the crash. If I remove the draw call, the game runs normally. If I remove the material swap, the game runs normally.

1 Like
#4

I managed to find the exact problem, which I believe is an issue with the renderer part of the engine. It relates to swapping materials that each contain custom vertex attributes.

Screenshot 2024-09-24 141753
Screenshot 2024-09-24 141753951Ă—581 2.43 KB

The tree is drawn as the terrain predicate and uses the terrain material, which looks like this:

Screenshot 2024-09-24 142056
Screenshot 2024-09-24 1420561217Ă—768 21.8 KB

It uses a custom vertex attribute called glow_color, which is a lighting color that is used later to lighten the tree at night.

The raindrops are drawn as the weather predicate and use the weather material, which looks like this:

Screenshot 2024-09-24 142323
Screenshot 2024-09-24 1423231211Ă—793 23.3 KB

It uses two custom vertex attributes called color and glow_color, where color is equivalent to tint in some of Defold’s default materials, and glow_color is equivalent to the tree’s situation.

In the render script, I draw terrain and weather predicates with their default materials shown above. This works fine. Later, I replace those materials with the glow material, which looks like this:

Screenshot 2024-09-24 142726
Screenshot 2024-09-24 1427261214Ă—763 21.6 KB

Once again, it contains the glow_color attribute.

I was experimenting with how to swap materials that contained custom vertex attributes. The terrain and weather shaders do not require the glow_color attribute. They don’t even access it. The only reason I list it in the material files is so that I can use go.set() on that property at runtime, then access it in the glow shaders.

This trick seems to work for the raindrops, but not for the tree. Maybe it has something to do with the weather material having two custom vertex attributes, where as the terrain material only has one? I don’t know how Defold’s renderer handles custom vertex attributes when swapping material files.

The exact crash happens when I draw the terrain predicate with the glow material:

...
render.enable_material("glow")
render.draw(terrain_predicate, { constants = constant_buffer })
...

Which invokes the following vertex and fragment shaders:

// glow.vp

#version 140

////////////////////////////////////////////////////////////////////////////////
// Inputs
////////////////////////////////////////////////////////////////////////////////

in vec4 position;
in vec2 texcoord0;
in vec4 glow_color;

////////////////////////////////////////////////////////////////////////////////
// Uniforms
////////////////////////////////////////////////////////////////////////////////

uniform general_vp
{
	mat4 view_proj;
};

////////////////////////////////////////////////////////////////////////////////
// Outputs
////////////////////////////////////////////////////////////////////////////////

out vec2 var_texcoord0;
out vec4 var_glow_color;

////////////////////////////////////////////////////////////////////////////////
// Functions
////////////////////////////////////////////////////////////////////////////////

void main()
{
	gl_Position = view_proj * vec4(position.xy, position.z, 1.0);
	var_texcoord0 = texcoord0;
	var_glow_color = glow_color;
}

// glow.fp

#version 140

////////////////////////////////////////////////////////////////////////////////
// Inputs
////////////////////////////////////////////////////////////////////////////////

in vec2 var_texcoord0;
in vec4 var_glow_color;

////////////////////////////////////////////////////////////////////////////////
// Uniforms
////////////////////////////////////////////////////////////////////////////////

uniform sampler2D albedo_sampler;

////////////////////////////////////////////////////////////////////////////////
// Outputs
////////////////////////////////////////////////////////////////////////////////

out vec4 fragment_color;

////////////////////////////////////////////////////////////////////////////////
// Functions
////////////////////////////////////////////////////////////////////////////////

vec4 get_glow_color(vec4 albedo_color)
{
	if (albedo_color.a == 0.0)
	{
		return vec4(0.0, 0.0, 0.0, 0.0);
	}
	return var_glow_color;
}

void main()
{
	vec4 albedo_color = texture(albedo_sampler, var_texcoord0);
	vec4 glow_color = get_glow_color(albedo_color);
	fragment_color = albedo_color * glow_color;
}

Notice how I access var_glow_color in the fragment shader’s get_glow_color() function. That variable causes the crash. If I don’t access it and instead use a dummy value like vec4(1, 1, 1, 1), it doesn’t crash.

Maybe @jhonny.goransson might be able to help with this one? If there’s anything else I can provide, please let me know. If I need to build an example project for debugging purposes, I can do that, but perhaps I’m missing something obvious.

#5

Well shoot, I may have solved it after typing out that explanation.

After reading this paragraph a couple times, I remembered that Defold only builds custom vertex attributes into the vertex format if they’re referenced in a shader attached to the material file.

I wasn’t accessing the glow_color attribute in any shader except those attached to the glow material. Just because it’s listed in the material file with a default value doesn’t mean it’s included. That explains why swapping over to the glow material and trying to access the attribute failed. It’s because the attribute doesn’t exist in the vertex format.

Why then was it working when drawing the raindrops? I have no idea. Recall that the raindrops use the weather material, which has two custom attributes: color and glow_color. The color attribute was being utilized in its shaders, so it was built into the vertex format as expected. The glow_color attribute was not.

After some digging around, I found that the glow_color attribute was taking on the value of the color attribute when drawing with the glow material. Makes me think there may be some kind of misalignment going on when swapping materials. Since the weather material’s first custom attribute was defined and the second wasn’t, and the glow material only lists one custom attribute, maybe the glow material assumed that its one custom attribute glow_color should reference the one custom attribute actually defined, which is incorrect. Seems like it’s ignoring the names of the attributes and instead just using “the first custom attribute index” or however it’s treated in the engine.

Maybe I’m doing something dangerous and unsupported, but it just happens to work in this particular scenario. Anyway, if someone on the Defold team manages to read and understand this issue, then it would be nice to see if it’s something we can fix.

Edit: This also doesn’t explain why everything seems to work until I add four or more trees, rather than just crashing regardless of the amount of trees. Maybe I should find a different approach to all of this instead of tip-toeing around a potential bug. :sweat:

1 Like
#6

If the engine is crashing then it is something we want to fix. Could you please create a ticket on GH and include the above info please?

1 Like
#7

And, it possible (and time permitting), please add a small repro case where this crash happens.
It speed things up significantly (and increases chances of stuff being fixed sooner)