Struggling with Native Extensions to make Protobufs or Flatbuffers work with Defold

Questions
1

I’m new to Defold and to Lua. I’m coming from 4 years of programming in UE4 (and some Unity). I’m not an expert programmer, but I’ve studied as one, I love doing it and that’s all I’ve been doing for the last 4 years. I’m supposed to be able to handle Defold.

Saying that I’m having difficulties with Defold is saying nothing…

My idea is to write a 2D MMORPG on mobile. The server would be written in Go, the client on Defold. I need the server to be efficient, so I need to have as little overhead as possible from client-server communication. After researching a little, it looks like Json, Msgpack and the likes are too slow. Thankfully, there’s Protobufs and Flatbuffers that could do the job. Protobufs are less efficient, but easier to use, and Flatbuffers are more efficient, but require a lot of ugly code. Frankly, at this point, I could settle for either of those two, but I can’t make any of them work in Defold.

Let’s start with Flatbuffers. There’s an official Lua implementation, so no problem there. But it depends on string.pack and string.unpack from Lua 5.3. No biggie, looks like there’s compat53 (a library for lua 5.1 that backports string.pack and unpack), and if that fails, there’s roberto’s struct.pack/unpack library that offers similar functionnality (but not the same?)

  1. I couldn’t turn compat53’s string library into a native extension. It crashes on startup with an error that tells me nothing (seen below). What is this script.cpp? I don’t have it in my project. Is it some other native extension that crashes, and if so - why? Is there a crash log? In case anyone wants to have a look at the code itself, here’s my NE.

Assertion failed: top == lua_gettop(L), file …\src\script.cpp, line 214

  1. I managed to turn roberto’s library into a native extension. With it, flatbuffers’ serialization doesn’t work. The library probably doesn’t do exactly what’s needed. I’ll come back to it later.

Now trying Protobufs. There are two Lua implementations (one complicated and one simple), both supposed to work with Lua 5.1.

  1. I couldn’t install the simple one as a NE: Assertion failed: top == lua_gettop(L), file …\src\script.cpp, line 214 again. NE.
  2. Couldn’t install the complicated one either, same error. NE.

Five days of working on this problem and I got exactly nowhere. Any advice would be truly appreciated.

P.S. I’m using these extensions:

compat53: GitHub - lunarmodules/lua-compat-5.3: Compatibility module providing Lua-5.3-style APIs for Lua 5.2 and 5.1
roberto’s library: http://www.inf.puc-rio.br/~roberto/struct/
protobufs complicated library: GitHub - starwing/lua-protobuf: A Lua module to work with Google protobuf
protobufs simple library: GitHub - urbanairship/protobuf-lua: Lua library for Google's Protocol Buffers

Some questions:

  • How to turn compat53 into a NE? It has multiple .c files, so is each .c file supposed to be a separate NE, or is it possible to combine them somehow into one NE? What am I doing wrong?

  • When you have something like #include <lua.h> in a native extension, are you supposed to comment out these includes? Or are you supposed to dump these headers into the /include folder every time a NE needs it?

1 Like
2

Going back to compat53, made some changes in lstrlib.c:
before:

 LUAMOD_API int luaopen_string (lua_State *L) {
 luaL_newlib(L, strlib);
 createmetatable(L);
 return 1;
 }

after:

LUAMOD_API int luaopen_string (lua_State *L) {
int top = lua_gettop(L);
  
luaL_register(L, MODULE_NAME, strlib);
createmetatable(L);
lua_pop(L, 1);

assert(top == lua_gettop(L) && "Dead here"); // used to assert here initially
return 1;
}

Now it compiles and launches. Launches only through Project->Build.

But:

  1. Can’t “Run with Debugger”, which makes the whole thing pointless, can’t work like that. When launched with debugger, all I get is white screen.

  2. Flatbuffers asserts in the middle of its code. Why? String.pack and unpack seem to work, and it’s the only thing that flatbuffers rely on from lua 5.3? Or is it…?

For example, this code works:

local test = string.pack('d', 4)
local reverse = string.unpack('d', test)
print(reverse)

This is how the string NE looks now.

1 Like
3

This is not the answer you are looking for I know, but just a suggestion. We are developing our server with Go+FlatBuffer too. I would recommend you not to use any Lua lib on client side. Just use the Flatbuffer C++ code directly in your client project. I guess It is better than dealing with any other C code like compat53. It require more work but it is efficient.

3 Likes
4

Regarding script.cppit’s part of the engine. And the assert itself, is there to make sure all extensions initialized ok. In this case, they didn’t. And the check tells us that there’s something wrong with the Lua stack.

When writing Lua c functions, you need to keep track of the Lua stack. E.g. if you your function returns a value, then the Lua stack should grow.

In the case of initialization functions, the engine doesn’t expect any return values, and as such, the Lua stack going into the function, should be at the same position when coming out of the function. This is what the assert is for.

In your case, in such functions, you can use a pattern like this:

static void myfunction() {
        int top = lua_gettop(L);
        (void)top;

// Do Lua related stuff

        assert(top == lua_gettop(L)); // Make sure we cleaned up everything on the stack
}
2 Likes
5

Or API reference (dmScript) right?

2 Likes
6

True. And then using the corresponding DM_LUA_ERROR() in that case.

2 Likes
7

Managed to make native extensions from all libraries in the end, one of them even ended up working well.

Here’s what I learned, for people who have the same problems as I had.

  1. When you make a native extension from C code, find a line that goes something like this:

luaL_newlib(L, libs); // sometimes it’s something else entirely, in one library the author pushes “libs” (it’s a luaL_Reg array of functions) onto the stack one by one in a for loop…

Change it to:

luaL_register(L, MODULE_NAME, libs);

  1. Put this at the beginning of the method where you did that:

int top = lua_gettop(L);

Put this at the end of the method:

assert(top == lua_gettop(L) && “here”);

Now build. You may get asserts here or in script.cpp. This means you have to put this line after luaL_register:

lua_pop(L, 1);

Build again. Doesn’t work? Change second argument to 2 and try again. Go up until it works (I sometimes had to go up to 5).

Great. Now you have a native extension, but you don’t know how to “require” it?
In my case, require (“modulename”) didn’t work.
But what works is this:

local has_modulename, modulename = pcall(require, “modulename”)

This knowledge could’ve saved me a good week of pain. I couldn’t find it anywhere, I had to put it together from bits and pieces. Maybe you’ll find it useful.

1 Like
8

You usually don’t have to require native extensions. Their module name space should be made available on the global table. This is what most/all other native extensions do.

9

The usage of the Lua module is shown in the example

(as well as showing how to add return arguments)

Learning new tech can ofc be quite rough, and we continuously look into how to improve our manuals.
For instance, some best practices when programming with Lua’s C API could be one such thing.

1 Like