How to pass scene depth buffer to fragment shader as a texture? (DEF-2699)

Pkeod · November 2, 2017, 11:16pm

I want to use the depth buffer values of a scene for post processing effects, such as doing SSAO or scene fog.

Like this kind of thing http://www.roxlu.com/2014/036/rendering-the-depth-buffer

https://learnopengl.com/#!Advanced-Lighting/SSAO

Ragnar_Svensson · May 16, 2017, 2:16am

Create a render target with a depth buffer
Draw your scene
Set render target as current texture, specifying the depth buffer
Draw the thing with the shader

Pkeod · May 16, 2017, 3:29am

I should be able to do this?

render.enable_texture(0, self.effect_render_target, render.BUFFER_COLOR_BIT)
render.enable_texture(1, self.effect_render_target, render.BUFFER_DEPTH_BIT)

And then this should work?

uniform lowp sampler2D DIFFUSE_TEXTURE;
uniform lowp sampler2D DEPTH_BUFFER;

Right now I’m getting

Render target does not have a texture for the specified buffer type.
stack traceback:
	[C]: in function 'enable_texture'

When trying

render.enable_texture(1, self.effect_render_target, render.BUFFER_DEPTH_BIT)

I do have a texture setup as a sampler in the material the render target is using.

Is there a sample render script I could look at to see how it’s done correctly?

Here is render target being setup. What’s missing?

	local color_params = { format = render.FORMAT_RGBA,
	                       width = render.get_window_width(),
	                       height = render.get_window_height(),
	                       min_filter = render.FILTER_LINEAR,
	                       mag_filter = render.FILTER_LINEAR,
	                       u_wrap = render.WRAP_CLAMP_TO_EDGE,
	                       v_wrap = render.WRAP_CLAMP_TO_EDGE }
   	local depth_params = { format = render.FORMAT_DEPTH,
	                       width = render.get_window_width(),
	                       height = render.get_window_height(),
	                       u_wrap = render.WRAP_CLAMP_TO_EDGE,
	                       v_wrap = render.WRAP_CLAMP_TO_EDGE }    	
    
    
    	
    self.effect_render_target = render.render_target("effect_render_target", {[render.BUFFER_COLOR_BIT] = color_params, [render.BUFFER_DEPTH_BIT] = depth_params})

Ragnar_Svensson · May 16, 2017, 6:13am

Sorry, you’re right, we only support color buffers at the moment. I filed an issue for this, DEF-2699

sergey.lerg · July 3, 2017, 7:10pm

Is that implemented yet? Can we access depth buffer as a texture now?

Ragnar_Svensson · July 5, 2017, 6:49am

No, sorry, not many are asking for it so other things keep taking precedence.

sergey.lerg · July 5, 2017, 8:40am

People just don’t know what possibilities it opens up, probably.
I want to make Depth of Field effect with a simple blur instead of the bokeh for starters.
Without that feature I can still render to another render target to imitate the depth buffer, right?

Ragnar_Svensson · July 5, 2017, 9:55am

Yes, of course, you can use render.enable_material to switch to a shader which draws the depth as a gray-scale or similar.

sergey.lerg · July 8, 2017, 8:11pm

Looks like only two textures are allowed per material? I tried passing original texture, depth texture and lut texture to my postprocessing shader and color grading no longer works…
Is it hard to support more than two textures per material?

Ragnar_Svensson · July 10, 2017, 1:28pm

That should not be the case. Could you try something safer/simpler to verify that the third texture is actually ignored? The only limiting factor should be the number of texturing units.

Pkeod · November 2, 2017, 5:19pm

Can current emphasis on 3D features get this issue solved too? It’s needed for a variety of shader effects.

Andreas_Tadic · November 2, 2017, 6:29pm

What you need to do is to bind a texture to the depth buffer. This is not supported by the core GLES2.0 functionality, which we guarantee to support. You can however achieve this using the OES_depth_texture extension.
We are currently focusing heavily on both 3D-rendering functionality and performance and part of that is to be able to use OpenGL extensions and functionality to check support etc.

It will be worked on over the next few sprints, I can’t however give you a more precis estimate at the very moment, more than to say it does have very high priority given how much it will bring in to the 3D-support of Defold.
But, one will need other functionality in place for this to be useful, something that is taken into account and worked on simultaneously.

Pkeod · May 27, 2020, 9:21pm

DEF-2699

Looks like this is still an issue?

jhonny.goransson · May 27, 2020, 9:39pm

Yeah unfortunately there’s no extension support available in the render scripts that could enable this functionality. I know it has been up for discussion at some point, but I don’t know if was ever a design drafted for it. Perhaps @Mathias_Westerdahl or @sven has some input to it? I don’t think we can stick to the ES2.0 vanilla spec forever if we want to move forward with better 3D support, but it’s not up to me to decide

Klear · December 14, 2020, 11:47pm

Could someone expand on how to do this? Rendering the depth buffer in grayscale would be great for debugging, I think. I’m a bit out of my depth though, if you pardon the pun.

Pkeod · December 15, 2020, 2:05am

You can render multiple passes to render targets. I assume do a normal render first and then one where the shader of what you want was changed to a sort of depth shader. Then composite those (such as by supplying them as textures for another shader) along with GUI and whatever render predicates you use.

This might help but it’s been a long while since I tested this, beware it’s old and some of the defaults have changed possibly.

github.com

subsoap/deffx/blob/master/deffx/materials/model/depth.fp

varying mediump vec2 var_texcoord0;
varying mediump vec3 var_position;
varying mediump vec3 var_normal;

uniform lowp vec4 eye;
uniform lowp vec4 lightPos4;
uniform lowp sampler2D DIFFUSE_TEXTURE;

// todo: update near far values from render script camera
float near = 0.1; 
float far  = 100.0; 

float linearize_depth(float depth) 
{
    float z = depth * 2.0 - 1.0; // Back to NDC 
    return (2.0 * near * far) / (far + near - z * (far - near));	
}


void main( void )

This file has been truncated. show original

You list the materials you want to use with this in the .render file.

render.enable_material("depth")
render.draw(self.model_predicate)
render.disable_material()

Edit: After thinking about your specific use case all you need to do is add a depth material to your .render and then have a key in your game which sends a message to @render and in your render script toggle enabling the depth material when drawing your model predicate. Should be very easy, then mess with the depth fragment program until you get a look that’s useful for you debugging. For getting and using the camera near / far you’ll want to set it in a constants buffer and then use it when rendering the model predicate.

Example

github.com

subsoap/2021_shader_experiments/blob/main/rendercam/global_constants.lua

local M = {}
M.constants = {}

function M.get()
	local constants = render.constant_buffer()
	for k,v in pairs(M.constants) do
		constants[k] = v
	end
	return constants
end

function M.set(id, value)
	M.constants[id] = value
end

function M.reset()
	M.constants = {}
end

return M

github.com

subsoap/2021_shader_experiments/blob/main/rendercam/rendercam.render_script#L63




	render.draw(self.tile_pred)
	render.draw(self.particle_pred)


	-- Model rendering
	render.set_blend_func(render.BLEND_SRC_ALPHA, render.BLEND_ONE_MINUS_SRC_ALPHA)
	render.enable_state(render.STATE_CULL_FACE)
	render.enable_state(render.STATE_DEPTH_TEST)
	render.set_depth_mask(true)


	render.draw(self.model_pred, global_constants.get())


	-- Debug rendering - physics debug, draw_line
	render.disable_state(render.STATE_DEPTH_TEST)
	render.disable_state(render.STATE_CULL_FACE)
	render.draw_debug3d()


	-- GUI Rendering
	render.set_viewport(0, 0, rendercam.window.x, rendercam.window.y)
	render.set_view(IDENTITY_MATRIX)
	render.set_projection(self.gui_proj) -- gui_proj only calculated on update_window

Klear · December 16, 2020, 10:30pm

Thanks a bunch! Looks like what I need. Whenever I mess around with the renderscript, it’s the depth buffer that’s giving me trouble most often. Being able to actually see what’s happening with it is bound to help me in those cases.

Edit: Works like a charm! Still not sure about the “why” when it comes to the glitches I’m experiencing, but at least I can understand the “how” now, if that makes sense.