How do I do a multipass shaders?

Jerakin · May 31, 2017, 4:07pm

I am trying to learn more about shaders and rendering stuff

I have read and followed along the tutorials https://www.defold.com/tutorials/shadertoy/ and
https://www.defold.com/tutorials/grading/ they where so interesting that they made me want to learn more.

I wanted to have a blur shader because I thought it could be useful. I read a lot about it, if someone is interested this is a really interesting read https://software.intel.com/en-us/blogs/2014/07/15/an-investigation-of-fast-real-time-gpu-based-image-blur-algorithms

So I made a setup that renders a shader to a render target that is then used as a texture on a quad. ~~I don’t really understand how to render to a different sized render target so if you know that please take a look here How do I correctly render to a different sized render target?~~

My problem here is that I have a blur shader (I took a kawasa blur shader from here https://www.shadertoy.com/view/Xl3XW7) but to make it work as on shadertoy it needs to be multipass, but I have no clue how to do that in Defold.

Can someone explain how to do a multipass shader in Defold?

Here is the "translated" shader

.

// 'Kawase Blur'
// https://www.shadertoy.com/view/Xl3XW7

varying mediump vec4 position;
varying mediump vec2 var_texcoord0;

uniform lowp sampler2D effects;
uniform lowp vec4 delta;
uniform lowp vec4 options;

void reSample(in int d, in vec2 uv, inout vec4 fragColor)
{
 
    vec2 step1 = (vec2(d) + 0.6) / options.xy;
    fragColor += texture2D(effects, uv + step1) / float(4);
    fragColor += texture2D(effects,  uv - step1) / float(4);
  	vec2 step2 = step1;
    step2.x = -step2.x;
    fragColor += texture2D(effects, uv + step2) / float(4);
    fragColor += texture2D(effects,  uv - step2) / float(4);
}

void main()
{
    vec2 pixelSize = vec2(1.0) / options.xy;
    vec2 halfSize = pixelSize / vec2(2.0);
    vec4 color = vec4(0);
    reSample(3, var_texcoord0, color);
    gl_FragColor = color;
}

britzl · June 3, 2017, 12:31pm

@Jerakin Did you manage to solve this?

Jerakin · June 3, 2017, 3:09pm

No I haven’t manage to figure it out, I assume I have to make a complex render script?

So am I thinking in the correct way if I think I need to do something like this

Render everything to a render target “A”
Use the render target (“A”) as a texture to a quad and blur it with the material (can I apply materials straight to render targets?)
Render that outcome to a new render target “B”
Use the render target (“B”) as a texture to a quad and blur it with the material
Do that for as many passes I want

Is this correct thinking? Is there an easier/simpler way?

Pkeod · June 3, 2017, 10:57pm

That sounds about right. You can add the various multipass step materials to the .render file and then activate / deactivate them as needed, I believe. I was working on multipass but got a bit distracted before I finished and have not gone back to it yet. You should be able to make it a little more easy to use by looping through a list of specified passes when rendering to then get a final output.

Jerakin · June 6, 2017, 7:47pm

Thanks for the response,

I have been sitting with this today some more and I have no clue what I am doing! Really hard to find information about this kinda stuff too, so hard to read up on it.

Oh and I can’t really do it as a render material because in the end I don’t want the gui to be affected.

What is wrong with this approach?

Enable the first render target
Draw what I want to be blurred
Disable the first render target
Enable the second render target
Set first render target as texture to quad
Disable the second render target
Set second render target as texture to quad

It doesn’t feel like when I self.draw(self.effects_pred)the first time it doesn’t get put into the second render target. Is it because I disable the texture?

It is probably easier if you guys can just see the whole render script.

[details=main.render_script]

function init(self)
    self.tile_pred = render.predicate({"tile"})
    self.gui_pred = render.predicate({"gui"})
    self.text_pred = render.predicate({"text"})
    self.particle_pred = render.predicate({"particle"})
    self.effects_pred = render.predicate({"post_effects"})
	
    self.clear_color = vmath.vector4(0, 0, 0, 0)
    self.clear_color.x = sys.get_config("render.clear_color_red", 0)
    self.clear_color.y = sys.get_config("render.clear_color_green", 0)
    self.clear_color.z = sys.get_config("render.clear_color_blue", 0)
    self.clear_color.w = sys.get_config("render.clear_color_alpha", 0)
	
    self.view = vmath.matrix4()
    local color_params = { format = render.FORMAT_RGBA,
                       width = render.get_window_width(),
                       height = render.get_window_height() } 
    local target_params = {[render.BUFFER_COLOR_BIT] = color_params }

    self.target = render.render_target("effects", target_params)
    self.target_second = render.render_target("effects2", target_params)
    self.render_effects = false
    self.render_passes = 5
end

local function effect(self, target, predicate)
	
	render.disable_render_target(target)
	render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color})
	render.set_viewport(0, 0, render.get_window_width(), render.get_window_height()) 
	render.set_view(vmath.matrix4())
	render.set_projection(vmath.matrix4_orthographic(0, render.get_window_width(), 0, render.get_window_height(), -1, 1))
	
	render.enable_texture(0, target, render.BUFFER_COLOR_BIT) 
	render.draw(predicate) 
	render.disable_texture(0, target)
end

local function do_update(self)
	render.set_depth_mask(true)
    render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color, [render.BUFFER_DEPTH_BIT] = 1, [render.BUFFER_STENCIL_BIT] = 0})

	render.set_viewport(0, 0, render.get_width(), render.get_height())

    render.set_view(self.view)

    render.set_depth_mask(false)
    render.disable_state(render.STATE_DEPTH_TEST)
    render.disable_state(render.STATE_STENCIL_TEST)
    render.enable_state(render.STATE_BLEND)
    render.set_blend_func(render.BLEND_SRC_ALPHA, render.BLEND_ONE_MINUS_SRC_ALPHA)
    render.disable_state(render.STATE_CULL_FACE)
    
	render.set_projection(vmath.matrix4_orthographic(0, render.get_width(), 0, render.get_height(), -1, 1))

    render.draw(self.tile_pred)
    
    render.draw(self.particle_pred)
    render.draw_debug3d()

    render.set_view(vmath.matrix4())
    render.set_projection(vmath.matrix4_orthographic(0, render.get_window_width(), 0, render.get_window_height(), -1, 1))

    render.enable_state(render.STATE_STENCIL_TEST)
    render.draw(self.gui_pred)
    render.draw(self.text_pred)
    render.disable_state(render.STATE_STENCIL_TEST)
    
    render.set_depth_mask(false)
    render.draw_debug2d()
end

function update(self)
	if self.render_effects then
		render.enable_render_target(self.target)
	end
	do_update(self)
	
	if self.render_effects then
		render.enable_render_target(self.target_second)
		effect(self, self.target, self.effects_pred)
		effect(self, self.target_second, self.effects_pred)
	end
end

function on_message(self, message_id, message)
    if message_id == hash("clear_color") then
        self.clear_color = message.color
    elseif message_id == hash("set_view_projection") then
        self.view = message.view
	elseif message_id == hash("render_effects") then
		self.render_effects = message.render
    end
end

```[/details]

Ragnar_Svensson · June 7, 2017, 5:50am

This doesn’t have any effect. If you don’t draw while a render target is enabled, there is no point in enabling it in the first place.

Enabling a render target => whenever you draw, the graphics (pixel values) will end up in the render target instead of in the back buffer.

Setting a render target as a texture => the render target graphics (pixel values) will be sampled from, rather than the original texture of that thing. Sorry if this is obvious, but it also means that this only has an effect if you draw the thing you just textured.

In short, graphics only happen through render.draw (and render.clear). All the other calls is setting up the gpu state for how that render.draw will happen.

If you draw the gui last, after you have drawn the post-effect stuff to the back buffer, the gui graphics will be unaffected and drawn on top of the blurred graphics.

Jerakin · June 7, 2017, 8:54am

I do draw the predicate after I set the texture to the quad, so the order should be.

4. Enable the second render target
5. Set first render target as texture to quad
5.5 Draw the quad (with render.draw(predicate) )
6. Disable the second render target

My thinking is that this should draw the blurred image to the second render target. Isn’t this what happens?

Oh right of course! I actually already do that for a different test I made… I blame the “late” night.

Really really appreciate all the help!

Jerakin · June 7, 2017, 11:52am

Finally figured out why it wasn’t working, my problem was that I enabled the second render pass before I disabled the first one.

In my post above I said I did this

But if we look at the render script (the 5th post) we can see that I actually do it the other way around.

I assume that we can not draw to two render targets at the same time? Or what is the theory behind this?

Thank you guys for the help

This is what I ended up with.
Two quads, that have different materials assigned. The difference between the material is different fragments shaders and the only difference on the fragment shaders is the name they use for the sampler2D.

main.render_script

function init(self)
    self.tile_pred = render.predicate({"tile"})
    self.gui_pred = render.predicate({"gui"})
    self.text_pred = render.predicate({"text"})
    self.particle_pred = render.predicate({"particle"})
    self.effects_pred = render.predicate({"post_effects"})
	
    self.clear_color = vmath.vector4(0, 0, 0, 0)
    self.clear_color.x = sys.get_config("render.clear_color_red", 0)
    self.clear_color.y = sys.get_config("render.clear_color_green", 0)
    self.clear_color.z = sys.get_config("render.clear_color_blue", 0)
    self.clear_color.w = sys.get_config("render.clear_color_alpha", 0)
	
    self.view = vmath.matrix4()
    local color_params = { format = render.FORMAT_RGBA,
                       width = render.get_window_width(),
                       height = render.get_window_height() } 
    local target_params = {[render.BUFFER_COLOR_BIT] = color_params }

    self.target = render.render_target("effects", target_params)
    self.target_second = render.render_target("effects2", target_params)
    self.render_effects = false
    self.render_passes = 5
end

local function effect(self, target, predicate)
	
	render.disable_render_target(target)
	render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color})
	render.set_viewport(0, 0, render.get_window_width(), render.get_window_height()) 
	render.set_view(vmath.matrix4())
	render.set_projection(vmath.matrix4_orthographic(0, render.get_window_width(), 0, render.get_window_height(), -1, 1))
	
	render.enable_texture(0, target, render.BUFFER_COLOR_BIT) 
	render.draw(predicate) 
	render.disable_texture(0, target)
end

local function do_update(self)
	render.set_depth_mask(true)
    render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color, [render.BUFFER_DEPTH_BIT] = 1, [render.BUFFER_STENCIL_BIT] = 0})

	render.set_viewport(0, 0, render.get_width(), render.get_height())

    render.set_view(self.view)

    render.set_depth_mask(false)
    render.disable_state(render.STATE_DEPTH_TEST)
    render.disable_state(render.STATE_STENCIL_TEST)
    render.enable_state(render.STATE_BLEND)
    render.set_blend_func(render.BLEND_SRC_ALPHA, render.BLEND_ONE_MINUS_SRC_ALPHA)
    render.disable_state(render.STATE_CULL_FACE)
    
	render.set_projection(vmath.matrix4_orthographic(0, render.get_width(), 0, render.get_height(), -1, 1))

    render.draw(self.tile_pred)
    
    render.draw(self.particle_pred)
    render.draw_debug3d()

    render.set_view(vmath.matrix4())
    render.set_projection(vmath.matrix4_orthographic(0, render.get_window_width(), 0, render.get_window_height(), -1, 1))

    render.enable_state(render.STATE_STENCIL_TEST)
    render.draw(self.gui_pred)
    render.draw(self.text_pred)
    render.disable_state(render.STATE_STENCIL_TEST)
    
    render.set_depth_mask(false)
    render.draw_debug2d()
end

function update(self)
	if self.render_effects then
		render.enable_render_target(self.target)
	end
	do_update(self)
	
	if self.render_effects then
		render.disable_render_target(self.target)
		render.enable_render_target(self.target_second)
		
		
		render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color})
		render.set_viewport(0, 0, render.get_window_width(), render.get_window_height()) 
		render.set_view(vmath.matrix4())
		render.set_projection(vmath.matrix4_orthographic(0, render.get_window_width(), 0, render.get_window_height(), -1, 1))
		
		render.enable_texture(0, self.target, render.BUFFER_COLOR_BIT) 
		render.draw(self.effects_pred) 
		render.disable_texture(0, self.target)
		
		render.disable_render_target(self.target_second)
		render.enable_texture(0, self.target_second, render.BUFFER_COLOR_BIT) 
		render.draw(self.effects_pred) 
		render.disable_texture(0, self.target_second)

	end
end

function on_message(self, message_id, message)
    if message_id == hash("clear_color") then
        self.clear_color = message.color
    elseif message_id == hash("set_view_projection") then
        self.view = message.view
	elseif message_id == hash("render_effects") then
		self.render_effects = message.render
    end
end

selimanac · August 5, 2024, 4:18pm

Do we have any other methods suggested since 2017, or is this still the legitimate way of doing multipass?

Chung_Xa · August 5, 2024, 5:19pm

Render script is necessary sometimes but it shouldn’t be the main solution for shaders. Instead, every components should be able to contact directly to its material variables and textures via editor propertise panel and via go/gui api.

jhonny.goransson · August 5, 2024, 5:33pm

I would say that yes, this is still a valid approach to doing multi-pass. But there’s a lot more building bricks now for rendering that you can utilise. You can render the pass to a render target resource and either bind it to the renderer by a sampler name, or update an atlas to use the render target as a backing texture. Or with my upcoming feature (https://github.com/defold/defold/pull/9208) you can set the render target directly as a texture on a material or a compute module.

What more precisely do you want to do?

jhonny.goransson · August 5, 2024, 5:40pm

Not sure I follow. The components already marshal their constant values to the shaders via go.set - this is more specifically towards a “multipass” setup, where the output of a component (or multiple) needs to be calculated in several steps. I suppose you could invent your own data driven multipass system, but it wouldn’t make that much sense for us to make something built-in that would do that.

Chung_Xa · August 5, 2024, 5:55pm

Yes, this is now right with both go and gui but we (or just my wish) still need handle multi textures from scripts, not from render script

selimanac · August 5, 2024, 6:00pm

I’m researching pixel art shader(post-processing) for 3d models. I’ll need a few steps like edge detection, pixelate, maybe for scaling… I found this one and started to implement it and I need a second pass…

selimanac · August 5, 2024, 6:03pm

How can I do this? I don’t remember any implementation of it somewhere… Do you have an example? I’ll be glad if you can share it.

I don’t know much about compute shader but I’m willing to learn.

jhonny.goransson · August 5, 2024, 6:08pm

render.enable_texture can take both a sampler name and a render target as an argument:

render.enable_texture("my_texture_sampler", self.my_texture_handle)

It’s not only about compute shaders, the feature is for materials as well. So basically you can do:

material.set_texture("/my_material.materialc", “my_sampler”, texture_or_handle)
material.set_constant(…)
material.set_vertex_attribute(…)
material.set_sampler(…)

jhonny.goransson · August 5, 2024, 6:10pm

So, something like - you want to be able to bind a secondary texture of a component by using go.set?

e.g
go.set("#sprite", "image", self.secondary_texture, { key = "sampler_name" })

Chung_Xa · August 5, 2024, 6:16pm

How about a gui node? Is it possible for now?

jhonny.goransson · August 5, 2024, 6:19pm

You mean via gui.set? Or gui.set_texture?

Chung_Xa · August 5, 2024, 6:20pm

Either of them. I just find if there’s a way