Can I use Illumination asset with character animations? (SOLVED)

I have my 3D character where animations are visible when skinned material is selected for the model.

I added Illumination asset for better lights and it works great with environment. But, when I apply the model.material from the assets, animations are not visible anymore. Am I missing something, should I find a way around or is there an existing solution?

I’m not sure but guess would need to create a new .vp for it? like default model_skinned.vp has this different from model.vp? Haven’t tried myself but maybe could try to include them for position calculation?

in mediump vec4 bone_weights;
in mediump vec4 bone_indices;
#include “/builtins/materials/skinning.glsl”
vec4 skinned_position = get_skinned_position(position);

@astrochili would certainly know more? :grin:

1 Like

To be honest I don’t know. I have never worked with animations before.

If I had to solve this problem, I would start by looking for material differences and requirements. Next, I would try to adapt the lighting material to the character’s material requirements, if any. I just don’t know anything about animated models, maybe they use some custom samplers or uvs that conflict with the illumination material..

In other words, this issue is most likely completely solvable, but the material needs to be adapted to the character’s needs.

1 Like

Thank you for the answer. I was gonna try creating a skinned version of the model material but wanted to make sure there is no easier mothed yet. Hope it will turn out well

Using GPT, I was able to produce this version of the model.vp from your asset and it supports skinned model with animations:

//
// model.vp (Skinned)
// Based on Illumination shader by astrochili
// Modified to support skinned models and animations
//

attribute highp vec4 position;
attribute mediump vec2 texcoord0;
attribute mediump vec2 texcoord1;
attribute mediump vec3 normal;
attribute mediump vec4 bone_weights;
attribute mediump vec4 bone_indices;

uniform mediump mat4 mtx_world;
uniform mediump mat4 mtx_view;
uniform mediump mat4 mtx_proj;

uniform mediump vec4 animation_data;

varying highp vec3 camera_position;
varying highp vec3 world_position;
varying highp vec3 view_position;
varying mediump vec3 world_normal;
varying mediump vec2 texture_coord;
varying mediump vec2 light_map_coord;

#include "/builtins/materials/skinning.glsl"

mat3 mat4_to_mat3(mat4 m) {
    return mat3(
        m[0][0], m[0][1], m[0][2],
        m[1][0], m[1][1], m[1][2],
        m[2][0], m[2][1], m[2][2]
    );
}

mat3 mat3_transpose(mat3 m) {
    return mat3(
        m[0][0], m[1][0], m[2][0],
        m[0][1], m[1][1], m[2][1],
        m[0][2], m[1][2], m[2][2]
    );
}

mat3 mat3_inverse(mat3 m) {
    float a00 = m[0][0], a01 = m[0][1], a02 = m[0][2];
    float a10 = m[1][0], a11 = m[1][1], a12 = m[1][2];
    float a20 = m[2][0], a21 = m[2][1], a22 = m[2][2];

    float b01 =  a22 * a11 - a12 * a21;
    float b11 = -a22 * a10 + a12 * a20;
    float b21 =  a21 * a10 - a11 * a20;

    float det = a00 * b01 + a01 * b11 + a02 * b21;

    return mat3(
        b01, (-a22 * a01 + a02 * a21), (a12 * a01 - a02 * a11),
        b11, (a22 * a00 - a02 * a20), (-a12 * a00 + a02 * a10),
        b21, (-a21 * a00 + a01 * a20), (a11 * a00 - a01 * a10)
    ) / det;
}

mat4 mat4_inverse(mat4 m) {
    float
    a00 = m[0][0], a01 = m[0][1], a02 = m[0][2], a03 = m[0][3],
    a10 = m[1][0], a11 = m[1][1], a12 = m[1][2], a13 = m[1][3],
    a20 = m[2][0], a21 = m[2][1], a22 = m[2][2], a23 = m[2][3],
    a30 = m[3][0], a31 = m[3][1], a32 = m[3][2], a33 = m[3][3],

    b00 = a00 * a11 - a01 * a10,
    b01 = a00 * a12 - a02 * a10,
    b02 = a00 * a13 - a03 * a10,
    b03 = a01 * a12 - a02 * a11,
    b04 = a01 * a13 - a03 * a11,
    b05 = a02 * a13 - a03 * a12,
    b06 = a20 * a31 - a21 * a30,
    b07 = a20 * a32 - a22 * a30,
    b08 = a20 * a33 - a23 * a30,
    b09 = a21 * a32 - a22 * a31,
    b10 = a21 * a33 - a23 * a31,
    b11 = a22 * a33 - a23 * a32,

    det = b00 * b11 - b01 * b10 + b02 * b09 + b03 * b08 - b04 * b07 + b05 * b06;

    return mat4(
        a11 * b11 - a12 * b10 + a13 * b09,
        a02 * b10 - a01 * b11 - a03 * b09,
        a31 * b05 - a32 * b04 + a33 * b03,
        a22 * b04 - a21 * b05 - a23 * b03,
        a12 * b08 - a10 * b11 - a13 * b07,
        a00 * b11 - a02 * b08 + a03 * b07,
        a32 * b02 - a30 * b05 - a33 * b01,
        a20 * b05 - a22 * b02 + a23 * b01,
        a10 * b10 - a11 * b08 + a13 * b06,
        a01 * b08 - a00 * b10 - a03 * b06,
        a30 * b04 - a31 * b02 + a33 * b00,
        a21 * b02 - a20 * b04 - a23 * b00,
        a11 * b07 - a10 * b09 - a12 * b06,
        a00 * b09 - a01 * b07 + a02 * b06,
        a31 * b01 - a30 * b03 - a32 * b00,
        a20 * b03 - a21 * b01 + a22 * b00
    ) / det;
}

void main() {
    mat4 mtx_inv_view = mat4_inverse(mtx_view);
    camera_position = (mtx_inv_view * vec4(0.0, 0.0, 0.0, 1.0)).xyz;

    // Apply bone-based skinning
    vec4 skinned_position = get_skinned_position(position);

    vec4 _world_position = mtx_world * vec4(skinned_position.xyz, 1.0);
    world_position = _world_position.xyz;

    vec4 _view_position = mtx_view * _world_position;
    view_position = _view_position.xyz;

    mat3 mtx_normal = mat3_transpose(mat3_inverse(mat4_to_mat3(mtx_world)));
    world_normal = normalize(mtx_normal * normal);

    texture_coord = texcoord0;
    light_map_coord = texcoord1;

    gl_Position = mtx_proj * _view_position;
}

While it seems to work for me, not sure how “professional“ of a solution it is. If you think it is, alright, maybe you could add it to the asset?

2 Likes

Did you have any warnings or errors with the original material you were trying to use ?

No, just selected the new .vp from the copied material and selected the new material on the model. That’s it.

No error or unexpected behavior and is simply working. Was not expected it to be this easy tbh :sweat_smile:

1 Like

Glad it worked :wink:

1 Like

An alternative minimal change would be to set Illumination >> model.material Vertex Space property to World.

1 Like

Yeah, this is the way! Thank you :slight_smile:

Found out setting the vertex space to world causes some problems when go/collection is moved around (model shifted relative to world position).

Setting vertex space to Local and using the .vp above solved the issue. Writing it here in case someone encounters a similar problem later