iOS crash on app focus gain in dmGraphics::Clear(dmGraphics::Context*, unsigned int, unsigned char, unsigned char, unsigned char, unsigned char, float, unsigned int)

[filazopovich]

We get crash on iOS after app activation with the same crash dump on our Family Island (custom render script) and on internal sample app for AppLovin extension (without custom render script). Both app use Defold 1.2.166. This crash is not stable to reproduce.

Family Island callstack:

OS Version:          iPhone OS 13.3 (17C54)
Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  0
Thread 0 name:  engine_main  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x000000018dde3ec4 __pthread_kill + 8
1   libsystem_pthread.dylib       	0x000000018dcff1d8 pthread_kill$VARIANT$mp + 136
2   libsystem_c.dylib             	0x000000018dc538c0 __abort + 112
3   libsystem_c.dylib             	0x000000018dc53850 __abort + 0
4   libsystem_c.dylib             	0x000000018dc52c64 err + 0
5   FamilyIsland                  	0x000000010122f0f0 dmGraphics::Clear(dmGraphics::Context*, unsigned int, unsigned char, unsigned char, unsigned char, unsigned char, float, unsigned int) + 4616432 (.cold.3) + 0
6   FamilyIsland                  	0x000000010122b384 dmGraphics::BeginFrame(dmGraphics::Context*) + 4600708 (graphics_opengl.cpp:977)
7   FamilyIsland                  	0x000000010123cdcc dmRender::ParseCommands(dmRender::RenderContext*, dmRender::Command*, unsigned int) + 4672972 (render_command.cpp:87)
8   FamilyIsland                  	0x0000000101241cd8 dmRender::UpdateRenderScriptInstance(dmRender::RenderScriptInstance*, float) + 4693208 (render_script.cpp:2873)
9   FamilyIsland                  	0x00000001011a6f3c dmEngine::Step(dmEngine::Engine*) + 4058940 (engine.cpp:1341)
10  FamilyIsland                  	0x00000001011a7bec dmEngineUpdate(dmEngine::Engine*) + 4062188 (engine.cpp:1798)
11  FamilyIsland                  	0x0000000101329268 -[AppDelegate appUpdate] + 5640808 (AppDelegate.m:178)
12  QuartzCore                    	0x000000019491988c CA::Display::DisplayLink::dispatch_items+ 71820 (unsigned long long, unsigned long long, unsigned long long) + 632
13  IOKit                         	0x000000018ef35934 IODispatchCalloutFromCFMessage + 488
14  CoreFoundation                	0x000000018df418ac __CFMachPortPerform + 172
15  CoreFoundation                	0x000000018df6b07c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 56
16  CoreFoundation                	0x000000018df6a7a8 __CFRunLoopDoSource1 + 444
17  CoreFoundation                	0x000000018df6567c __CFRunLoopRun + 2168
18  CoreFoundation                	0x000000018df64adc CFRunLoopRunSpecific + 464
19  GraphicsServices              	0x0000000197eea328 GSEventRunModal + 104
20  UIKitCore                     	0x000000019205fae0 UIApplicationMain + 1936
21  FamilyIsland                  	0x00000001011a7e6c engine_main(int, char**) + 4062828 (engine_main.cpp:50)
22  libdyld.dylib                 	0x000000018ddee360 start + 4

Internal AppLovin extension app:

OS Version:          iPhone OS 13.1.2 (17A860)
Exception Type:  EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Triggered by Thread:  0
Thread 0 name:  engine_main  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x0000000185c08ebc __pthread_kill + 8
1   libsystem_pthread.dylib       	0x0000000185b24790 pthread_kill$VARIANT$mp + 112
2   libsystem_c.dylib             	0x0000000185a788a0 __abort + 112
3   libsystem_c.dylib             	0x0000000185a78830 __abort + 0
4   libsystem_c.dylib             	0x0000000185a77c44 err + 0
5   defoldapplovin                	0x00000001023a0a78 dmGraphics::Clear(dmGraphics::Context*, unsigned int, unsigned char, unsigned char, unsigned char, unsigned char, float, unsigned int) + 1657464 (.cold.3) + 0
6   defoldapplovin                	0x000000010239cd10 dmGraphics::BeginFrame(dmGraphics::Context*) + 1641744 (graphics_opengl.cpp:977)
7   defoldapplovin                	0x00000001023facd8 dmRender::ParseCommands(dmRender::RenderContext*, dmRender::Command*, unsigned int) + 2026712 (render_command.cpp:87)
8   defoldapplovin                	0x00000001023ffbe4 dmRender::UpdateRenderScriptInstance(dmRender::RenderScriptInstance*, float) + 2046948 (render_script.cpp:2873)
9   defoldapplovin                	0x00000001023105b0 dmEngine::Step(dmEngine::Engine*) + 1066416 (engine.cpp:1341)
10  defoldapplovin                	0x0000000102311260 dmEngineUpdate(dmEngine::Engine*) + 1069664 (engine.cpp:1798)
11  defoldapplovin                	0x00000001024ede08 -[AppDelegate appUpdate] + 3022344 (AppDelegate.m:178)
12  QuartzCore                    	0x000000018c656514 CA::Display::DisplayLink::dispatch_items+ 70932 (unsigned long long, unsigned long long, unsigned long long) + 632
13  IOKit                         	0x0000000186d49eb0 IODispatchCalloutFromCFMessage + 488
14  CoreFoundation                	0x0000000185d656d4 __CFMachPortPerform + 172
15  CoreFoundation                	0x0000000185d8ee5c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 56
16  CoreFoundation                	0x0000000185d8e588 __CFRunLoopDoSource1 + 444
17  CoreFoundation                	0x0000000185d8945c __CFRunLoopRun + 2168
18  CoreFoundation                	0x0000000185d888bc CFRunLoopRunSpecific + 464
19  GraphicsServices              	0x000000018fbf4328 GSEventRunModal + 104
20  UIKitCore                     	0x0000000189e1e6d4 UIApplicationMain + 1936
21  defoldapplovin                	0x00000001023114e0 engine_main(int, char**) + 1070304 (engine_main.cpp:50)
22  libdyld.dylib                 	0x0000000185c13460 start + 4

[Mathias_Westerdahl]

Hmm, ok.
Would you mind trying to build with this version.
It will give us a bit more output to the log regarding the open gl errors.

OSX

[Dragosha]

similar issue?

Defold 1.2.164 has been released

Hi, I’ve tried to reproduce the issue with other my apps without native extensions and got the same error in console. [2019-12-16_13-15-11] Steps to reproduce: run app wait for device sleeping wakeup device an app immediate collapses (collapses if debug bundle or just freeze if release bundle) An open source example for testing: https://github.com/Dragosha/slasher-prototype Could you try to make iOS bundle of this one and test it? I tested on iPad6,7 (iPad Pro 12.9" first generation) O…

[britzl]

similar issue?

Yes, probably. Same as this one: https://github.com/defold/editor2-issues/issues/2908

[filazopovich]

I’m here with additional information about that problem (and some logs from 1.2.167):

• It happens on Android and iOS.
• It’s not a crash. It’s assertion fail for debug builds and completely frozen render for release builds.
• Not stable to reproduce, but sometimes it happens after focus gain. On Android it occurs in random moments, but some API in extensions uses Activity For Result for interactions with some internal parts of extension. That can lead to focus lost and gain.
• Gl error 1285 - out of memory. EGL error 3003 - out of memory. But it happens for example app without resources, for small app for extension tests and for Family Island on the same devices with 1-8 GB RAM. I’m not sure that is real out of memory because all three apps have very different memory footprint.

Some logs:

E/defold: ERROR:GRAPHICS: Clear(973): gl error 1285
A/libc: ../src/opengl/graphics_opengl.cpp:973: void dmGraphics::Clear(dmGraphics::HContext, uint32_t, uint8_t, uint8_t, uint8_t, uint8_t, float, uint32_t): assertion "0" failed

2020-04-10 09:15:20.729 1771-2088/com.MelsoftGames.FamilyIslandFarm E/Surface: queueBuffer: error queuing buffer to SurfaceTexture, -22 [com.MelsoftGames.FamilyIslandFarm/com.dynamo.android.DefoldActivity$_1771#0]
2020-04-10 09:15:20.729 1771-2088/com.MelsoftGames.FamilyIslandFarm W/glfw-android: eglSwapBuffers failed, eglGetError: 3003
2020-04-10 09:15:20.737 1771-2088/com.MelsoftGames.FamilyIslandFarm W/glfw-android: eglSwapBuffers failed, eglGetError: 300B
2020-04-10 09:15:20.762 1771-2088/com.MelsoftGames.FamilyIslandFarm W/glfw-android: eglSwapBuffers failed, eglGetError: 300B (repeated for every frame)

[Mathias_Westerdahl]

Searching for glClear and GL_OUT_OF_MEMORY yields some results mentioning depth buffers.
Not exactly sure what to deduce from that, and I cannot come up with a real good test.

Are you using a custom render script?
Are you using a extra render.clear() or some custom render target?

[Mathias_Westerdahl]

@filazopovich I’m looking into this crash, but I cannot seem to reproduce it. Do you still see this crash?
You mentioned an example app where it happens, is it something you could share with me?

[filazopovich]

Assertion fail by GL error 1285 sometimes happens in Debug build of our game. In unknown circumstances.
What do you think about option in game.project to change assert to log write in Debug builds?

[Mathias_Westerdahl]

But, would that really help?
The assert is already disabled, in release builds.
You say that you get a “completely frozen render for release builds”.

Also, just because an assert is hit, doesn’t mean that it’s just fine to remove it. The asserts are there to be able to catch the error as soon as possible, as opposed to let the engine run even further, producing more obscure issues. Asserts are there to catch fatal issue, not for data validation. I acknowledge that each assert can be debated, but in this particular instance, it won’t do any good to remove it.

[Mathias_Westerdahl]

However, I realized I had already done this earlier (since we already supported it on HTML via a command line flag).

There’s a setting graphics.verify_graphics_calls.
It will always be set to 0 in release builds, but in debug builds, you can set it to 0 to override the default behavior.

[filazopovich]

Thanks!
I think there is two bugs. I’m not sure that GL call fail in Clear lead to frozen screen in release build.

[Mathias_Westerdahl]

Could be.
How much memory is your app using? And texture memory?
I mean, it could be that OpenGL is telling the truth, and it somehow ended up out of memory (e.g. when swapping the app in/out again).

It should be possible to profile the app (don’t recall which tool I used last time though)

[Mathias_Westerdahl]

Since I’m trying to look into these issues, for those being able to reproduce it:

We have two tickets (iOS/Android respectively):
https://github.com/defold/defold/issues/5100 (Android)
https://github.com/defold/defold/issues/4674 (iOS)

If you get this issue on Android, I’d very much like to see the full logcat (i.e. with the threaded-app APP_CMD_*). This will be very helpful.

[Mathias_Westerdahl]

@filazopovich Good news is that I have a good repro case for the Android issue now (~50-70%).

And I have a solution that doesn’t assert when getting the GL_OUT_OF_MEMORY, as you suggested.

I hope to get this into the (late) beta today or tomorrow, and let @agulev test some more.

https://github.com/defold/defold/issues/5279 (Android)
https://github.com/defold/defold/issues/5100 (Android)