Game eating whole RAM and crashing (SOLVED)

[Pawel]

Ok, so I made a 3D game for the jam, it’s dirty and quick, so maybe I just made some terrible optimization mistake, but I don’t think Lua is so easy to introduce memory leak. As some of you may have experienced, the game crashes and eats whole memory breaking systems. It happens also on my machine, but up to now I thought it’s because it’s crappy and not for 3D games. I would love to understand what the issue is, so please, help me investigate it

Source code: https://github.com/paweljarosz/horror_3d_game_mwdj_2023

Defold Beta 1.4.6

Remotery during normal gameplay, at the very beginning:

Then after some time game starts to hangs, drastically dropping fps to 1 and below, eats whole RAM and SWAP and crashes. There is definitely a memory leak:

I turned off previously opened application at point A (you see how high the ram got) it dropped immediately after closing game. I opened application again at point B and it’s visible the memory leak is from the beginning. At point C the image is captured:

Memory: over 95k!
LuaMem: over 700

I don’t understand fully the main window (an explanation of what is shown in Remotery would be nice!) but it looks like the most heavy scripts’s updates I would suspect are nothing actually.

Memory is very high, from start, so there is a problem at the beginning. I lowered all models to few hundreds-few thousands polygons, shrunk down textures to max 1024x1024, usually 512x512. I still though guess some models are problematic, but can’t find out which one or what’s the issue. I suspect the issue is with code that generates the map around - it creates world like tiles around player position, with factories spawning grass (2x2 quad) on bottom and up to 2 objects on top of the grass (1. tree or some other models + 2.eventually a sheet of paper - a 1x1 quad)). Enemy is spawned once and changes position (actually by removing it and factoring somewhere differently - it should be just changing position when I think about it now

It is perhaps regarding this code (scene.script) creating world around using factories, but I don’t know what the issue might be. I wrote some simple instance counting, when factory.create() is called and watched it, but it never exceed some value that is regarding the maximum FOV I set.

---

EDIT:

No, my theory failed. I stoped recreating world - still memory leak. I didn’t created world at all - didn’t factored any other instances than player and collision box for ground - still memory leak!

---

EDIT2:

I even commented out all update functions, all timers… Still memory leak and nothing happens in the game.

---

EDIT3:

Ok, the problem must be in rendering script, because when I commented out whole body of update() in render script the memory is not raising after game starts. Found a place to investigate!

[Pawel]

Ok, the issue seems to be when creating render target for post-processing. When I commented out drawing to separate render target, there is no memory raising at all. I have to revise this code.

function update(self)
--update window dimensions
local window_width = render.get_window_width()
local window_height = render.get_window_height()

-- recreate render targets (in case of window change)
self.all_rt = helper.create_render_target("all")

-- scene
helper.draw_to_render_target(self.all_rt, function()
	render.enable_material(hash("illumination"))
	render.clear(self.clear_options)

	render.set_viewport(0, 0, window_width, window_height)
	render.set_view(self.scene.view)
	render.set_projection(self.scene.projection)
	helper.set_default_state()
	local frustum = self.scene.view * self.scene.projection

	render.draw(self.sky_pred, {frustum = frustum})
	render.draw(self.scene.predicate)--, {frustum = frustum})

	render.disable_material(hash("illumination"))
end)

-- postprocessing
helper.draw_to_quad_for_fullscreen_postprocessing(quad_postprocessing_material, function()
	render.clear(self.clear_options)

	render.set_view(IDENTITY_MATRIX)
	render.set_projection(IDENTITY_MATRIX)
	helper.set_default_state()
	local frustum = IDENTITY_MATRIX * IDENTITY_MATRIX

	render.draw(self.quad_pred, {frustum = frustum})
end, self.all_rt)

-- debug
render.disable_state(render.STATE_DEPTH_TEST)
render.disable_state(render.STATE_CULL_FACE)
render.draw_debug3d()

-- gui
render.set_view(self.gui.view)
render.set_projection(vmath.matrix4_orthographic(0, window_width, 0, window_height, -1, 1))
render.enable_state(render.STATE_STENCIL_TEST)
render.draw(self.gui.predicate)
render.draw(self.text.predicate)

end

Where draw_to_rt and to quad are just simple wrappers:

function M.draw_to_render_target(rt, fn)
	render.enable_render_target(rt)
	fn()
	render.disable_render_target(rt)
end


function M.draw_to_quad_for_fullscreen_postprocessing(quad_material, fn, color_tex0)

	render.enable_material(quad_material)
	render.enable_texture(0, color_tex0, render.BUFFER_COLOR0_BIT)

	fn()

	render.disable_texture(0)
	render.disable_material()
end

And to create render target:

function M.create_render_target(name)
	local color_params = {
		format = render.FORMAT_RGBA,
		width = render.get_window_width(),
		height = render.get_window_height(),
		min_filter = render.FILTER_LINEAR,
		mag_filter = render.FILTER_LINEAR,
		u_wrap = render.WRAP_CLAMP_TO_EDGE,
		v_wrap = render.WRAP_CLAMP_TO_EDGE
	}

	local depth_params = {
		format = render.FORMAT_DEPTH,
		width = render.get_window_width(),
		height = render.get_window_height(),
		u_wrap = render.WRAP_CLAMP_TO_EDGE,
		v_wrap = render.WRAP_CLAMP_TO_EDGE
	}

	return render.render_target(name, {
		[render.BUFFER_COLOR_BIT] = color_params,
		[render.BUFFER_DEPTH_BIT] = depth_params
	})
end

Or check out horror.render_script

[Pawel]

Ok, so it would be a monologue for future me (or somebody else facing similar issue!)

I was close, but I would probably stare at my own code for ages and wouldn’t find it…
@d954mas found the issue faster. It was really a memory leak caused by not removed render target. And I was creating them every frame! (Instead of recreating, which I intended to do, but forgot to write it)

Was:

function update(self)
	--light_and_shadows.update(self)
	--update window dimensions
	local window_width = render.get_window_width()
	local window_height = render.get_window_height()

	-- recreate render targets (in case of window change)
	self.all_rt = helper.create_render_target("all")
	--self.light_rt = helper.create_depth_buffer(2048,2048)

Should be:

function update(self)
	--update window dimensions
	local window_width = render.get_window_width()
	local window_height = render.get_window_height()

	-- recreate render targets (in case of window change)
	if (window_width ~= self.window_width)
	or (window_height ~= self.window_height) then
		self.window_width = window_width
		self.window_height = window_height
		render.delete_render_target(self.all_rt)
		self.all_rt = helper.create_render_target("all")
    end

[cornedbeer]

Thank you for documenting your process here; it made me realize I had a memory leak in my lighting system that just hadn’t manifested yet! It’s obvious in hindsight but I never realized you have to explicitly delete render targets.

[Pawel]

I don’t think about it because I wrote a module where I have helper functions for render scripts. It is still not perfect hence I didn’t released it yet, but in my main project I use it and I planned to refactor the render script near the end, but forgot. In the beginning of the sprint I only added render target for postprocessing and because I rarely test for more than 10—15 seconds of gameplay, I didn’t manage to thought it could be a memory leak. Only when I tried to record I noticed problems.

Anyway, it’s good to write things down, because also when I try to explain to someone my problem I sometimes find an issue. If not, I just click send and the community helps