Lua Refs lost (#4590)

[totebo]

I also bumped into this a while back:

Lua.Refs keeps growing (DEF-3841)

@Mathias_Westerdahl I have checked the minimal repro case I have sent above (13 March) with 1.2.163 and the issue is still there (DEF-3841 @inactive-vilse).