Luajit + FFI

Welcome to an expose of the wonders of Luajit and FFI, within Defold of course.

Lets make some things very clear about Luajit and FFI first:

  1. FFI - Foreign Function Interface is a specific call and bind mechanism in Luajit that can interact with C/C++ constructs and methods directly. It can crash your application if you do nasty things with it - like corrupting a pointer.
  2. FFI is very platform dependent. You have to have different interfaces for different platforms. Sometimes (like in the cases of OSX and Linux) the headers are identical, but you still need to load different dynamic libraries. If you want full cross platform capability, it is up to you to make it!
  3. FFI is much more suited to custom applications. However, it can be extremely useful in games, but be warned, this can be challenging :slight_smile:
  4. Some platforms dont support Luajit. IOS didnt use to, but I think it does now. Check with Defold to see which platforms are Luajit driven and which are pure Lua. As of writing this, I think it is only HTML targets that dont use Luajit. Thus, remember, Luajit + FFI is not going to work on html.

Ok. That out the way. Lets talk fun. The topic is intended as a living document. And I will intermittently add more and more FFI examples and methods as well as reference links.

Example ffi project here:

What is FFI?

FFI lets the developer call and create C/C++ constructs and access external C/C++ developed code bases. The best reference for it is where it is built:
https://luajit.org/ext_ffi.html

When using FFI in Defold you need to include it like so:
local ffi = package.preload.ffi()
This is slightly different to using pure Luajit so beware if you are copying from a lib on github :slight_smile:

First Simple Use

One of the many things people find frustrating with Lua is that it uses 1 indexed arrays. Well, with FFI you dont have to. Because you can create normal C allocated memory and objects you can also create normal C allocated arrays. Heres an example:

local ffi = package.preload.ffi()
--- Declare a structure of a C object we want to make an array of
ffi.cdef[[
typedef struct _MyObject {
	int number;
	float decimal;
	char  name[255];
	char * ptr;
} MyObject;
]]

local myobj = ffi.new("MyObject[?]", 20)
myobj[0].number = 1
myobj[0].decimal = 2.0
myobj[0].name = "My Name"
myobj[0].ptr = ffi.new("char[?]", 20)
ffi.copy(myobj[0].ptr, ffi.string("Another Name"))

pprint(myobj[0])
pprint(myobj[0].number)
pprint(myobj[0].decimal)
pprint(ffi.string(myobj[0].name))
pprint(ffi.string(myobj[0].ptr))

This creates 20 objects we can use in the array. This manipulates just the first one at index 0.
Notice that you can change properties just like you would in a table (mostly - more details on this later).
You shall see this in your console output:

DEBUG:SCRIPT: cdata<struct _MyObject &>: 0x018d62ef1468
DEBUG:SCRIPT: 1
DEBUG:SCRIPT: 2
DEBUG:SCRIPT: My Name
DEBUG:SCRIPT: Another Name

Notice that the object when printed is a “cdata”. This is luajit speak for “user defined object”.
Most of the above is fairly obvious. Please comment if you are puzzled by anything.
And thats it for simple array and object manipulation.

The question is: Why use this instead of a table?

Firstly, because its pure memory. There is nothing else allocated for the object. Luajit does create handles to the memory when you use ff.new, which means it will be garbage collected for you. That includes the pointer that was assigned.

Secondly, if you are worried about memory and footprints (small games for instance) then packing objects into tight arrays can really help with this. And you can define exactly how large the data types you want to be. For example, if you know the int number is only going to go up to 200, then you could make it an unsigned char instead, reducing the mem use from 4 bytes (int 32bit) to 1 byte.

Other big benefits are for operating with external methods. You can modify and change the data here, and then pass it to the library you are using to execute some important algorithms on, or maybe store it in a database?

Thats enough for the intial post. I will add one tomorrow with calls to common windows functions - we will make a window in Lua. :slight_smile:

16 Likes

Ok. As promised something a little more interesting.

Calling Windows User32

In the first example I explained a simple struct declaration. I did this first because later on we will be using all sorts of structures with our calls.
The first call I will show you is how to call a User32 method called MessageBox. This is a builtin Windows OS command call.

First, the ffi needs to load in the dll into its internal ffi.C namespace. The command used is:
local user32 = ffi.load( "user32.dll" )
The returned user32 is a handle that can be used later.

Now we need to declare how we will call this method from the luajit Lua script using ffi.cdef. Like so:

ffi.cdef[[
int MessageBoxA(uint32_t hWnd, const char * lpText, const char * lpCaption, uint32_t uType);
]]

The reason this is MessageBoxA is that Windows has a couple of ways to call this method. MessageBoxA uses the Ascii text encoding for the output text. The other method that would work is MessageBoxW but this uses wide character encoding, and would need some methods to encode strings to be able to be used.

Finally, I need to delclare some message box types. These can be found here:
< MessageBox function (winuser.h) - Win32 apps | Microsoft Learn >
Ive included a small sample here:

-- Some MSDN constants 
local mb_type = {
	MB_ABORTRETRYIGNORE 	= 0x00000002, -- The message box contains three push buttons: Abort, Retry, and Ignore.
	MB_CANCELTRYCONTINUE	= 0x00000006, -- The message box contains three push buttons: Cancel, Try Again, Continue. Use this message box type instead of MB_ABORTRETRYIGNORE.
	MB_HELP = 0x00004000, -- Adds a Help button to the message box. When the user clicks the Help button or presses F1, the system sends a WM_HELP message to the owner.
	MB_OK = 0x00000000, -- The message box contains one push button: OK. This is the default.
	MB_OKCANCEL = 0x00000001, -- The message box contains two push buttons: OK and Cancel.
	MB_RETRYCANCEL = 0x00000005, -- The message box contains two push buttons: Retry and Cancel.
	MB_YESNO = 0x00000004, -- The message box contains two push buttons: Yes and No.
	MB_YESNOCANCEL = 0x00000003, -- The message box contains three push buttons: Yes, No, and Cancel.
}

Now the MessageBox can be created and your are controlling the Windows OS direct from Lua script!

local result = user32.MessageBoxA( 0, Test, Heading, mb_type.MB_OK)
pprint(result)
result = user32.MessageBoxA( 0, Test, Heading, mb_type.MB_RETRYCANCEL)
pprint(result)

Warning: the resulting calls halt the execution of the Defold application (this is what MessageBox does). This is just an example of how to use OS calls.
The resulting output should be something like:

image
and
image

We have made windows! :slight_smile:
I’ll put this together in an example Defold project here:
< https://github.com/dlannan/defold-ffi >

I will add a createwindow example next. It is a little bigger and more complex because it needs much more setup. If you’ve ever made windows in C++ before, you will know what Im referring to :slight_smile:

8 Likes

Create Window

Making a window in Windows (ughh!) is not a straight forward process in C/C++.

The method I use here will generate a popup overlayed window that doesnt really do much. The wndproc uses the default handler and it only appears for 2 seconds, then is destroyed.
The repo has a complete sample of how to make a window. Its a little messy. I will cleanup later.

The core function is here:

local function makeWindow()
    local hInstance = kernel32.GetModuleHandleA(nil)
    pprint("hInstance: ", hInstance)
    
    local CLASS_NAME = 'TestWindowClass'

    local classstruct = {}
    classstruct.cbSize 		    = ffi.sizeof( "WNDCLASSEXA" )
    classstruct.lpfnWndProc     = WndProc
    classstruct.hInstance 		= hInstance	
    classstruct.lpszClassName 	= CLASS_NAME
    local wndclass = ffi.new( "WNDCLASSEXA", classstruct )	    

    -- Dodgy way to make sure we only register once for this process run.
    if(reg == nil) then 
        reg = user32.RegisterClassExA( wndclass )
    
        if (reg == 0) then
            error('error #' .. kernel32.GetLastError())
        end
    end    
    pprint("Registered Class.")

    local hwnd = user32.CreateWindowExA( ex_settings, CLASS_NAME, "My FFI Window", wnd_style.WS_OVERLAPPEDWINDOW, 10, 10, 300, 200, 0,0, hInstance, nil )
    if (hwnd == 0) then
        print("Unable to create window.")
        pprint("Error: "..tostring(kernel32.GetLastError()))
    else
        pprint(hwnd)
        user32.ShowWindow(hwnd, wnd_show.SW_SHOW)
        ffi.C.Sleep(2000)
        user32.DestroyWindow(hwnd)
        pprint("Window closed....")
    end
end

When running you should see a Defold window appear. If you press “2” then it will make a small window appear for two seconds then it will be closed. You can repeat this as much as you want.

The methodology here should show some of the power of calling methods that are in runtime OS dll’s. This applies to the vast majority of dlls (and so’s on Linux/OSX) on your system.

I will pin the repo at the top of the thread.

4 Likes

The next example to look at will be using a sqlite db. Using my own sqlite dll (for all platforms) the example shall be a more practical use example of ffi, and also show off some of the interesting interactions that can be done with Defold and a ffi dll.

Additionally, this will begin looking at how you can build up your own ffi library, as well as leverage a large number of already available ffi libs out there.

I’ll hopefully have something for this over the weekend, time permitting.

6 Likes

This example of using ffi is quite a bit more complex, but also alot more useful and introduces the concepts of ffi libraries.

FFI Libraires

These are shared libraries or also called dynamic libraries that most OS’s support. On windows you will find them as dll’s and on OSX and Linux their extension is .so

These libraries have been built to be used on an OS so that multiple processes and applications dont have to duplicate code (thats the core benefit, there are others too). In this example we will use the sqlite dll’s for windows available here: https://www.sqlite.org/download.html
For OSX and Linux versions you usually need to build for your platform - there are slight nuances which means source code built is generally more suitable.

To utilize a library a number of ffi.cdef declarations are needed so as to access both the types of the library and the methods.

One way to do this, is to take the source code header file that provides the public interface to the library api and wrap it in a ffi.cdef - often, on relatively simple libraries, this is all that is needed. However sqlite has a number of calling mechanisms and contexts that are important to manage.
This means there is more than just cdefs, and there is quite alot of lua script as well.

I have leveraged this luajit ffi library here:https://github.com/Playermet/luajit-sqlite
I extended it as well, so the load and save methods I needed were included.

The example I built is based around using sqlite to save and load game state. This is a common use for sqlite in games and in applications.

Here is a little video of saving and loading state when running.


It only shows me saving (pressing a key to save) and then loading that save state twice.

Have a look at the project to get the complete source code and windows binaries for testing the dll.
If you need/want OSX or Linux sqlite I can provide a couple (will build them).

Some important uses of sqlite that are valuable in game dev:

  • save/load states
  • filtering objects or data (large)
  • storing and managing profiles/data on a server
  • acting as a datacache for very large datasets - like a world map or similar.

I should note, I have also built a native extension version for sqlite here: https://github.com/dlannan/defold-sqlite3
WARNING: Recent examination of the android build of this library is not running properly (exceptions). Would recommend other solutions for android (and possibly ios too).

This native version also requires platform linked libraries to work properly as well. But it is useful to see the difference in implementation especially from an ease of development perspective.

The next example I hope to add, is an OpenCL interface. I used this to do realtime capture of screens and compress the frames, to be shared like a RemoteDesktop across the web. Im not sure if it will play nicely with Defold :slight_smile: Will see.

For some interesting examples and other ffi libraries you can use have a look at the awesome ufo kit:

It is quite old now, but it shows how many different libraries can be bound to be used in lua.

6 Likes

Was going to post up some OpenCL today but ran into so very weird ffi behaviour - it looks like the ffi.C namespace gets cleared when including multiple ffi libs. Im not certain of this, so I’ll split up the project and do some tests. Will get back to this in a week or so. Sorry. It took a large chunk of my day today, and I have a dozen other things to get done :slight_smile:

If you have questions I’ll answer them. I also have luajit based examples in my repos that have all sorts of ffi examples and uses. <github.com/dlannan>
Byte3d and nodelj prob have some of the larger examples.

2 Likes