Hello! I am trying to adapt a shader I wrote for OpenToonz to work in Defold. It works as intended in OpenToonz. Eventually I would like to turn it into a dynamic lighting system for painted normal mapped sprites. I have seen this topic in many posts around these forums and found quite a few lighting examples, all of which have been helpful but do not do exactly what I am looking for.
After a lot of failure and trying to understand Defold’s rendering pipeline, materials, etc… I have something more or less working. My final problem for this phase of things is that when I resize the window, my ‘light position’ goes super awry if the aspect ratio changes. Honestly, a lot of the math involved in this is way over my head and I suspect it is something in the way I’m converting the Defold coordinates to the shader variables. Hoping someone can take a look and spot something that may be off?
Hopefully I’m not forgetting anything here but I had to do quite a bit to get to this point. I’ll try to show it all here. Also feel free to point out anything that may bite me in the future or could be done better! 
So first, the ‘sprites’ are actually game objects. This is the start of how I’m getting around Defold’s limitation of a single ‘image’ per sprite. In this game object is a sprite named ‘albedo’ and a sprite named ‘normal’. The albedo sprite uses the standard sprite material (for now.) The normal sprite uses a copy of the material but with tag ‘normal’ to split rendering. Pretty straight forward. Here’s the outline and an example of a colored octagon image and its normal map, just for testing.
Now, the render script. I copied it from builtins and renamed it to core.render_script. For completeness here is the full render script.
--
-- projection that centers content with maintained aspect ratio and optional zoom
--
local function fixed_projection(near, far, zoom)
local projected_width = render.get_window_width() / (zoom or 1)
local projected_height = render.get_window_height() / (zoom or 1)
local xoffset = -(projected_width - render.get_width()) / 2
local yoffset = -(projected_height - render.get_height()) / 2
return vmath.matrix4_orthographic(xoffset, xoffset + projected_width, yoffset, yoffset + projected_height, near, far)
end
--
-- projection that centers and fits content with maintained aspect ratio
--
local function fixed_fit_projection(near, far)
local width = render.get_width()
local height = render.get_height()
local window_width = render.get_window_width()
local window_height = render.get_window_height()
local zoom = math.min(window_width / width, window_height / height)
return fixed_projection(near, far, zoom)
end
--
-- projection that stretches content
--
local function stretch_projection(near, far)
return vmath.matrix4_orthographic(0, render.get_width(), 0, render.get_height(), near, far)
end
local function get_projection(self)
return self.projection_fn(self.near, self.far, self.zoom)
end
local function window_resized(self, width, height)
render.set_render_target_size(self.albedo_target, width, height)
render.set_render_target_size(self.normal_target, width, height)
end
function init(self)
self.tile_pred = render.predicate({"tile"})
self.normal_pred = render.predicate({"normal"})
self.gui_pred = render.predicate({"gui"})
self.text_pred = render.predicate({"text"})
self.particle_pred = render.predicate({"particle"})
self.composite_pred = render.predicate({"composite"})
self.clear_color = vmath.vector4(0, 0, 0, 0)
self.clear_color.x = sys.get_config_number("render.clear_color_red", 0)
self.clear_color.y = sys.get_config_number("render.clear_color_green", 0)
self.clear_color.z = sys.get_config_number("render.clear_color_blue", 0)
self.clear_color.w = sys.get_config_number("render.clear_color_alpha", 0)
self.view = vmath.matrix4()
-- default is stretch projection. copy from builtins and change for different projection
-- or send a message to the render script to change projection:
-- msg.post("@render:", "use_stretch_projection", { near = -1, far = 1 })
-- msg.post("@render:", "use_fixed_projection", { near = -1, far = 1, zoom = 2 })
-- msg.post("@render:", "use_fixed_fit_projection", { near = -1, far = 1 })
self.near = -1
self.far = 1
self.projection_fn = fixed_projection
local color_params = { format = render.FORMAT_RGBA, width = render.get_window_width(), height = render.get_window_height() }
local target_params = {[render.BUFFER_COLOR_BIT] = color_params }
self.albedo_target = render.render_target("albedo", target_params)
self.normal_target = render.render_target("normal", target_params)
end
local function clear(self)
render.set_depth_mask(true)
render.set_stencil_mask(0xff)
render.clear({[render.BUFFER_COLOR_BIT] = self.clear_color, [render.BUFFER_DEPTH_BIT] = 1, [render.BUFFER_STENCIL_BIT] = 0})
render.set_viewport(0, 0, render.get_window_width(), render.get_window_height())
render.set_view(self.view)
render.set_projection(get_projection(self))
render.set_depth_mask(false)
render.disable_state(render.STATE_DEPTH_TEST)
render.disable_state(render.STATE_STENCIL_TEST)
render.enable_state(render.STATE_BLEND)
render.set_blend_func(render.BLEND_SRC_ALPHA, render.BLEND_ONE_MINUS_SRC_ALPHA)
render.disable_state(render.STATE_CULL_FACE)
end
function update(self)
local window_width = render.get_window_width()
local window_height = render.get_window_height()
if window_width == 0 or window_height == 0 then return end
local proj = get_projection(self)
local frustum = proj * self.view
-- render original scene to albedo target
render.set_render_target(self.albedo_target)
clear(self)
render.draw(self.tile_pred, {frustum = frustum})
render.draw(self.particle_pred, {frustum = frustum})
render.draw_debug3d()
-- render normals to normal target
render.set_render_target(self.normal_target)
clear(self)
render.draw(self.normal_pred, {frustum = frustum})
-- render composite to screen
render.set_render_target(render.RENDER_TARGET_DEFAULT)
clear(self)
render.set_view(self.view)
render.set_projection(self.view)
render.enable_texture(0, self.albedo_target, render.BUFFER_COLOR_BIT)
render.enable_texture(1, self.normal_target, render.BUFFER_COLOR_BIT)
render.draw(self.composite_pred, {frustum = frustum})
render.disable_texture(0)
render.disable_texture(1)
-- render GUI
local view_gui = vmath.matrix4()
local proj_gui = vmath.matrix4_orthographic(0, window_width, 0, window_height, -1, 1)
local frustum_gui = proj_gui * view_gui
render.set_view(view_gui)
render.set_projection(proj_gui)
render.enable_state(render.STATE_STENCIL_TEST)
render.draw(self.gui_pred, {frustum = frustum_gui})
render.draw(self.text_pred, {frustum = frustum_gui})
render.disable_state(render.STATE_STENCIL_TEST)
end
function on_message(self, message_id, message)
if message_id == hash("clear_color") then
self.clear_color = message.color
elseif message_id == hash("set_view_projection") then
self.view = message.view
self.projection = message.projection
elseif message_id == hash("use_camera_projection") then
self.projection_fn = function() return self.projection or vmath.matrix4() end
elseif message_id == hash("use_stretch_projection") then
self.near = message.near or -1
self.far = message.far or 1
self.projection_fn = stretch_projection
elseif message_id == hash("use_fixed_projection") then
self.near = message.near or -1
self.far = message.far or 1
self.zoom = message.zoom or 1
self.projection_fn = fixed_projection
elseif message_id == hash("use_fixed_fit_projection") then
self.near = message.near or -1
self.far = message.far or 1
self.projection_fn = fixed_fit_projection
elseif message_id == hash("window_resized") then
window_resized(self, message.width, message.height)
end
end
The important bits in here are: there is an albedo and normal render target. And a normal and composite render predicate. I also set the projection to fixed_projection instead of stretched. There is also a window_resized function that will set the render target sizes to the new width/height (otherwise stuff just disappears and it is a mess 
) Then there is a new entry in ‘on_message’ to receive a window resized message with the new width/height. This comes from a render-options game object script.
Overall, all this has been added to first render the scene to the albedo target. Then render the normal map sprites to the normal target. Then draw these to a 2x2 quad model using the composite shader and render predicate. So this brings us to the actual shader code.
First, for reference, here is the OpenToonz shader I wrote. OpenToonz provides a ‘worldToOutput’ variable. Which, as far as I understand, is mapping from the ‘world’ which is the canvas to the screen/output. It is a mat3. OpenToonz also requires ‘pre multiplication’ at the end.
#ifdef GL_ES
precision mediump float;
#endif
uniform mat3 worldToOutput;
uniform mat3 outputToWorld;
uniform sampler2D inputImage[2];
uniform mat3 outputToInput[2];
uniform vec2 light_position;
uniform float light_radius;
uniform float light_depth;
uniform vec4 light_color;
uniform vec4 ambient_color;
uniform vec2 resolution;
float det(mat3 m) { return m[0][0] * m[1][1] - m[0][1] * m[1][0]; }
float normaled(float v, float maxv, float minv) { return (v - minv) / (maxv - minv); }
void main() {
vec2 uv = gl_FragCoord.xy/resolution.xy;
vec4 color = texture2D(inputImage[0], uv);
vec3 normal = texture2D(inputImage[1], uv).rgb;
vec2 light_out_pos = (worldToOutput * vec3(light_position, 1.0)).xy / resolution;
float light_out_rad = light_radius * sqrt(abs(det(worldToOutput))) / resolution.y;
vec2 light_pos = light_out_pos - uv;
light_pos.y /= resolution.x/resolution.y;
float falloff = 1.0 / normaled(length(light_pos), light_out_rad * 0.5, 0.0);
vec3 light_dir = vec3(light_pos, light_depth);
vec3 normal_map = normalize(normal * 2.0 - 1.0); // * vec3(1.0, -1.0, 1.0); // Invert Y if needed?
float n_dot_l = max(dot(normal_map, normalize(light_dir)), 0.0);
vec3 diffuse = light_color.rgb * n_dot_l * falloff;
diffuse = 1.0 - exp( -diffuse );
gl_FragColor = vec4(color.rgb * (ambient_color.rgb + diffuse), color.a);
gl_FragColor.rgb *= gl_FragColor.a; // Premultiplication (Required by OpenToonz)
Now for the conversion to Defold. The vertex program is pretty simple. I think I copied it from the builtin sprite vp. But I made a ‘worldToOutput’ variable to pass to the fragment shader as a mat3. This converts the view_proj to mat3, which I just totally guessed might work and it seems to for the most part! 
uniform highp mat4 view_proj;
// positions are in world space
attribute highp vec4 position;
attribute mediump vec2 texcoord0;
varying mediump vec2 var_texcoord0;
varying mediump mat3 worldToOutput;
mat3 to_mat3(mat4 m4) {
return mat3(
m4[0][0], m4[0][1], m4[0][2],
m4[1][0], m4[1][1], m4[1][2],
m4[2][0], m4[2][1], m4[2][2]);
}
void main()
{
gl_Position = view_proj * vec4(position.xyz, 1.0);
var_texcoord0 = texcoord0;
worldToOutput = to_mat3(view_proj);
}
Then the fragment shader fp program looks more like the OpenToonz shader.
varying mediump vec2 var_texcoord0;
varying mediump mat3 worldToOutput;
uniform lowp sampler2D albedo;
uniform lowp sampler2D normal;
uniform lowp vec4 display_info;
uniform lowp vec4 mousepos;
float det(mat3 m) { return m[0][0] * m[1][1] - m[0][1] * m[1][0]; }
float normaled(float v, float maxv, float minv) { return (v - minv) / (maxv - minv); }
void main()
{
vec3 light_color = vec3(0.6, 0.6, 0.6); // TODO move
vec3 ambient_color = vec3(0.0, 0.0, 0.0); // TODO move
vec2 light_position = mousepos.xy;
float light_radius = 350.0;
float light_depth = 0.1;
vec4 color = texture2D(albedo, var_texcoord0);
vec3 norm = texture2D(normal, var_texcoord0).rgb;
vec2 resolution = display_info.xy;
// TODO figure out how to get worldToOutput matrix
vec2 light_out_pos = (worldToOutput * vec3(light_position, 1.0)).xy / resolution;
float light_out_rad = light_radius * sqrt(abs(det(worldToOutput))) / resolution.y;
vec2 light_pos = light_out_pos - var_texcoord0;
light_pos.y /= resolution.x/resolution.y;
float falloff = 1.0 / normaled(length(light_pos), light_out_rad * 0.5, 0.0);
vec3 light_dir = vec3(light_pos, light_depth);
vec3 normal_map = normalize(norm * 2.0 - 1.0);
float n_dot_l = max(dot(normal_map, normalize(light_dir)), 0.0);
vec3 diffuse = light_color.rgb * n_dot_l * falloff;
diffuse = 1.0 - exp( -diffuse );
gl_FragColor = vec4(color.rgb * (ambient_color.rgb + diffuse), color.a);
// gl_FragColor.rgb *= gl_FragColor.a; // Premultiplication (Required by OpenToonz) Guessing this is not needed for Defold.
}
There are a lot of arbitrary values in there that I plan to eventually pass in. For light position I am using the mouse position so I can easily move it around. For this I have a composite script attached to the composite game object along with the 2 x 2 model used to render the scene. This composite game object, along with the render-options game object and any ‘sprite’ game objects are added to the main collection.
So here is the composite script.
local aide = require "core.aide"
local function window_resize(self, width, height)
go.set("#model", "display_info", vmath.vector4(width, height, 0.2, 1))
end
function init(self)
self.width, self.height = window.get_size()
aide.WINDOW_RESIZE_ACTIONS[go.get_id()] = {}
go.set("#model", "display_info", vmath.vector4(self.width, self.height, 0.2, 1))
msg.post(".", "acquire_input_focus")
end
function on_input(self, action_id, action)
if action.x and action.y then
go.set("#model", "mousepos", vmath.vector4(action.x, action.y, 1, 1))
end
end
function on_message(self, message_id, message, sender)
if message_id == aide.WINDOW_RESIZE_HASH then
window_resize(self, message.width, message.height)
end
end
And here is the render-options script.
local aide = require "core.aide"
local function window_resize(width, height)
msg.post("@render:", "window_resized", { width = width, height = height })
end
function init(self)
window.set_listener(aide.window_listener)
aide.WINDOW_RESIZE_ACTIONS[go.get_id()] = {}
end
function on_message(self, message_id, message, sender)
if message_id == aide.WINDOW_RESIZE_HASH then
window_resize(message.width, message.height)
end
end
And to accommodate multiple things needing to know when the window is resized, I made a lua module called aide. This is what the other 2 scripts are referring to. They basically add themselves to a list and when the windows resize event happens, the module posts a message to everything in that list with the new width/height parameters. In the composite script this sets the ‘resolution’ on the material for the model. And on the render-options script it lets the render script know the window size changed, so it can update the render targets’ size. The relevant part of ‘aide’ is here:
local M = {}
M.WINDOW_WIDTH = 1920.0 -- TODO add an init function here and get these from that
M.WINDOW_HEIGHT = 1080.0
M.WINDOW_RESIZE_HASH = hash("resize window")
M.WINDOW_RESIZE_ACTIONS = {}
function M.window_listener(self, event, data)
if event == window.WINDOW_EVENT_RESIZED then
M.WINDOW_WIDTH, M.WINDOW_HEIGHT = window.get_size()
for k, v in pairs(M.WINDOW_RESIZE_ACTIONS) do
v["width"] = M.WINDOW_WIDTH
v["height"] = M.WINDOW_HEIGHT
msg.post(k, M.WINDOW_RESIZE_HASH, v)
end
end
end
return M
SOOOOO… with all that in place. It works! I don’t think I’m forgetting anything… maybe it would have been easier to just upload a git repo. 
Here is the results when it works and when it doesn’t.
The first image is correct! However, the second one is not. The difference is, I stretched the window much wider than normal so the aspect ratio was way different. The problem is, when I drag my mouse way off to the right, you can see the ‘light’ makes its way over to the octagon example.
Even though I update the resolution through the material on the composite script, it fails to properly translate the position of the mouse if the aspect ratio changes. At least I think that is the problem.
If you made it this far, thank you already! If you have any suggestions thank you x1000!
