What is the correct way to delete Buffer?

Questions
1

What I’ve run into is that I don’t understand how to properly free Buffer resources so that I can safely create them and delete them (without having to organise my own pools from buffers for reuse).

After some time I get an error in the game that the buffer cannot be created.

I have a simple test project where 100 game objects are created in a loop every frame, each of which is deleted later. And so in a loop many times.

What I expect: it should run without error indefinitely, because I correctly create and delete buffers and resource.

What I get:

ERROR:RESOURCE: The max number of resources (1024) has been passed, tweak "resource.max_resources" in the config file.
ERROR:SCRIPT: main/buffer_create.script:11: The resource was not updated (-11): 8720164381888447288, /mesh_23299.bufferc
stack traceback:
  [C]:-1: in function create_buffer
  main/buffer_create.script:11: in function <main/buffer_create.script:3>
  [C]:-1: in function create
  main/main.script:7: in function <main/main.script:5>

Hmm, ok. I decided to reset the counter to zero after 1000 resources to reuse the resource path (in case they are not cleared in the engine) and then I get another error (after 10 seconds):

Assertion failed: new_capacity <= 0xFFFF, file D:\a\defold\defold\engine\dlib\src\dlib/opaque_handle_container.h, line 208
INFO:CRASH: Successfully wrote Crashdump to file: C:\Users\Artsiom\AppData\Roaming\Defold/_crash
ERROR:CRASH: CALL STACK:

ERROR:CRASH:  0 0x7FF634607AB0 dmCrash::GenerateCallstack D:\a\defold\defold\engine\crash\src\backtrace_win32.cpp:144
ERROR:CRASH:  1 0x7FF634700DC4 raise minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:547
ERROR:CRASH:  2 0x7FF6346E74EC abort minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71
ERROR:CRASH:  3 0x7FF634694728 common_assert_to_stderr<wchar_t> minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:186
ERROR:CRASH:  4 0x7FF634694DE0 _wassert minkernel\crts\ucrt\src\appcrt\startup\assert.cpp:443
ERROR:CRASH:  5 0x7FF634530D50 dmOpaqueHandleContainer<dmBuffer::Buffer>::Allocate D:\a\defold\defold\engine\dlib\src\dlib\opaque_handle_container.h:210
ERROR:CRASH:  6 0x7FF6345317B0 dmBuffer::Create D:\a\defold\defold\engine\dlib\src\dlib\buffer.cpp:337
ERROR:CRASH:  7 0x7FF6342DD170 dmGameSystem::BuildBuffer D:\a\defold\defold\engine\gamesys\src\gamesys\resources\res_buffer.cpp:137
ERROR:CRASH:  8 0x7FF6342DE6B0 dmGameSystem::ResBufferCreate D:\a\defold\defold\engine\gamesys\src\gamesys\resources\res_buffer.cpp:216
ERROR:CRASH:  9 0x7FF634269580 dmResource::DoCreateResource D:\a\defold\defold\engine\resource\src\resource.cpp:803
ERROR:CRASH: 10 0x7FF634268BF0 dmResource::CreateResource D:\a\defold\defold\engine\resource\src\resource.cpp:992
ERROR:CRASH: 11 0x7FF6342D1380 dmGameSystem::CreateBuffer D:\a\defold\defold\engine\gamesys\src\gamesys\scripts\script_resource.cpp:3107
ERROR:CRASH: 12 0x7FF6341E686E lj_BC_FUNCC <unknown>:0
ERROR:CRASH: 13 0x7FF63445EB90 lua_pcall <unknown>:0
ERROR:CRASH: 14 0x7FF63442F570 dmScript::PCallInternal D:\a\defold\defold\engine\script\src\script.cpp:1398
ERROR:CRASH: 15 0x7FF634253D50 dmGameObject::RunScript D:\a\defold\defold\engine\gameobject\src\gameobject\comp_script.cpp:144
ERROR:CRASH: 16 0x7FF634251200 dmGameObject::CompScriptInit D:\a\defold\defold\engine\gameobject\src\gameobject\comp_script.cpp:185
ERROR:CRASH: 17 0x7FF634238040 dmGameObject::InitComponents D:\a\defold\defold\engine\gameobject\src\gameobject\gameobject.cpp:1724
ERROR:CRASH: 18 0x7FF63423E660 dmGameObject::SpawnInternal D:\a\defold\defold\engine\gameobject\src\gameobject\gameobject.cpp:1212
ERROR:CRASH: 19 0x7FF63423E500 dmGameObject::Spawn D:\a\defold\defold\engine\gameobject\src\gameobject\gameobject.cpp:1657
ERROR:CRASH: 20 0x7FF6342B0BB0 dmGameSystem::CompFactorySpawn D:\a\defold\defold\engine\gamesys\src\gamesys\components\comp_factory.cpp:475
ERROR:CRASH: 21 0x7FF6342CBB30 dmGameSystem::FactoryComp_Create D:\a\defold\defold\engine\gamesys\src\gamesys\scripts\script_factory.cpp:362
ERROR:CRASH: 22 0x7FF6341E686E lj_BC_FUNCC <unknown>:0
ERROR:CRASH: 23 0x7FF63445EB90 lua_pcall <unknown>:0
ERROR:CRASH: 24 0x7FF63442F570 dmScript::PCallInternal D:\a\defold\defold\engine\script\src\script.cpp:1398
ERROR:CRASH: 25 0x7FF634253D50 dmGameObject::RunScript D:\a\defold\defold\engine\gameobject\src\gameobject\comp_script.cpp:144
ERROR:CRASH: 26 0x7FF634252FF0 dmGameObject::CompScriptUpdateInternal D:\a\defold\defold\engine\gameobject\src\gameobject\comp_script.cpp:240
ERROR:CRASH: 27 0x7FF63423F470 dmGameObject::Update D:\a\defold\defold\engine\gameobject\src\gameobject\gameobject.cpp:2617
ERROR:CRASH: 28 0x7FF634219BA0 dmEngine::StepFrame D:\a\defold\defold\engine\engine\src\engine.cpp:1894
ERROR:CRASH: 29 0x7FF63421ABB0 dmEngineUpdate D:\a\defold\defold\engine\engine\src\engine.cpp:2447
ERROR:CRASH: 30 0x7FF63421ADF0 dmEngine::RunLoop D:\a\defold\defold\engine\engine\src\engine_loop.cpp:83
ERROR:CRASH: 31 0x7FF634211EE0 engine_main D:\a\defold\defold\engine\engine\src\engine_main.cpp:152
ERROR:CRASH: 32 0x7FF634676D34 __scrt_common_main_seh D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
ERROR:CRASH: 33 0x7FFC826F7360 BaseThreadInitThunk <unknown>:0
ERROR:CRASH: 34 0x7FFC82CDCC70 RtlUserThreadStart <unknown>:0
ERROR:CRASH: Lua Callstack:
ERROR:CRASH:   main/buffer_create.script:11: in function <main/buffer_create.script:3>
ERROR:CRASH:   [C]:-1: in function create
ERROR:CRASH:   main/main.script:7: in function <main/main.script:5>
ERROR:CRASH:

It turns out that I can’t create and delete resources more than 65535 times in one proxy world? I made this conclusion.

The test project:
test_buffer_create_factory.zip (626.8 KB)

2 Likes
2

While I don’t see anything wrong with your example, you should still note that you are creating buffers from Lua, and as such, are subject to Lua garbage collection.
I.e. it may be collected later than you imagine (in particular, not at the same frame the GO is deleted)

And it does seem the internal handle storage for buffers is currently limited to 65k global instances.
We may look into that.

That said, it also feels a little bit like a leak somewhere, as I’m calling collectgarbage('collect') each frame, and the number of buffers keep growing. Might of course still be related to creating garbage a lot faster than it’s collected :thinking:

3 Likes
3

Added ticket here:

I have a PR incoming as well.

6 Likes
4

Thanks for taking the time to look at my example! I usually see clearly that there is a bug in the engine when I write an issue. This time I didn’t understand at all what was wrong, i.e. I misunderstood how to use buffers properly, or why.

Regarding the 65K limit: is this something that needs to be worked out, or is it better to do workaround in my code by re-using buffers? For example, I updated the hyper trails example and its example with a ‘volcano’ of trails stops working because of this, as each trail is a new buffer (which is then deleted when the trail goes off screen).

1 Like
5

I don’t think we need to fix that 65k limit at this point (until we need to), as your example only has ~200 buffers at any time.
We can keep it in mind of course, but there are probably other things to look at first.

1 Like