Using stencil testing for outline shaders

Questions
#1

Outline effects are sometimes made using stencil testing. I wanted to learn how to do this in Defold using this method. In short, you draw the object with stencil testing enabled once, then scale the object up slightly before drawing it again, which creates a “cut out” effect on the second pass leaving just the outline.

But I can’t find a way to scale the object up before drawing it a second time. Stencil testing is enabled in a render script, but I’d have to scale up the object in a regular script or a shader, before returning to the render script and drawing it again.

Is there a way to do this? Maybe with messages?

#2

I suppose that you want to make outlines for 3d models.

Add the outline tag to your models material. You’ll use it for a render predicate to be able to render models which should be outlined.

Make a copy of the material plus its vertex/fragment shaders and name it as outline.material (the name and tags of that material is not important). Put the material into your custom .render file with the my_outline_material id.

Then, in the init function of your render script, add a custom predicate like self.outline_pred = render.predicate({"outline"}). To render outlines of the models, enable the custom material, draw models, disable the material:

render.enable_material(hash("my_outline_material"))
render.draw(self.outline_pred)
render.disable_material()

To scale the object up, write a custom vertex shader in the outline material.

attribute highp vec3 position;
attribute mediump vec3 normal;

uniform highp mat4 mtx_world;
uniform highp mat4 mtx_worldviewproj;

void main()
{
    float outline_width = 0.05;

    // We should find the scale of the model to scale the outline width accordingly
    float scale_x = length(mtx_world * vec4(1.0, 0.0, 0.0, 0.0));
    float scale_y = length(mtx_world * vec4(0.0, 1.0, 0.0, 0.0));
    float scale_z = length(mtx_world * vec4(0.0, 0.0, 1.0, 0.0));
    vec3 scaled_outline_width = outline_width / vec3(scale_x, scale_y, scale_z);

    // Scale vertices up, and there are two ways:
    // - by positions, but it works well only for cubes
    vec3 outline_offset = position.xyz * scaled_outline_width;
    // - by normals, but they should be smoothed
    // vec3 outline_offset = normal.xyz * scaled_outline_width;

    // Homework: use camera distance to keep outlines consistent!

    vec3 new_position = position + outline_offset;
    gl_Position = mtx_worldviewproj * vec4(new_position.xyz, 1.0);
}
7 Likes
#3

Works great! Here’s the result (homework not done):

image

There’s a bit of an issue when I start overlapping models, however. The models appear to “share” the outline.

image
image723×520 61.8 KB

I render the outlines first with depth testing disabled, and then draw the models on top.

render.draw(self.skybox_pred)
        
render.enable_material(hash("outline_mat"))
render.draw(self.outline_pred)
render.disable_material()

render.enable_state(render.STATE_DEPTH_TEST)
render.draw(self.model_pred)

To fix this I think I need a way to detect the edges of a mesh, unless there’s something in @aglitchman’s post that I didn’t understand?

3 Likes
#4

This is how the stencil-based object outlining algorithm works - look at the end of the mentioned article.

Try another method to make outlines - the inverted hull method. Remove all your custom stencil stuff from the render script, and render the models twice, with the second version of it flipped inside out and a little bit bigger (you do that in the custom outline material).

render.enable_state(render.STATE_DEPTH_TEST) -- this should be ON.
render.enable_state(render.STATE_CULL_FACE)

render.set_cull_face(render.FACE_BACK) 
render.draw(self.model_pred)

render.set_cull_face(render.FACE_FRONT)
render.enable_material(hash("my_outline_material"))
render.draw(self.outline_pred)
render.disable_material()

render.set_cull_face(render.FACE_BACK) -- revert back the default mode, i.e. FACE_BACK

That’s all!

5 Likes
#5

Here’s the result of the new code, very cool!

image
image723×488 76.6 KB

5 Likes