Using Fennel language with Defold

What is Fennel?

Fennel is programming language with Lisp-like syntax running on top of Lua. Since it keeps Lua semantics, code transcompiled from Fennel to Lua is very clean. For more information check official website: https://fennel-lang.org/

Why use Fennel in Defold?

While languages like Haxe and Typescript have been shown to work with Defold, those use ahead-of-time compilation, which breaks workflow in my opinion. Fennel can also transcompile to Lua in AOT fashion, which was described by @TheKing009 in his post.

Fennel provides another option and can be embedded as Lua module (single file) and compile code at run-time, which means less steps, when working in Defold. I focused solely on this.

Quick and dirty example

I translated Simple Linker example to Fennel, repository can be found here: GitHub - ajtos/defold-fennel-example

Fennel code in “fnlsrc” directory can be edited directly via Defold Editor, no additional steps are needed, just push Build button. File “main/board.script” is probably the most interesting one.

How it works?

Lets start with what didn’t work. This is idomatic example of embedding Fennel into Lua application from Fennel – Setting up Fennel

local fennel = require("fennel")
table.insert(package.loaders or package.searchers, fennel.searcher)
local mylib = require("mylib") -- will compile and load code in mylib.fnl

Second “require” for loading Fennel code won’t work with Defold. Defold tracks assets e.g. scripts attached to objects, Lua code loaded via “require”. In theory script above should load “mylib.fnl”, since instructions for loading modules are changed with insertion of custom rules into “package.loaders”. Yet Defold doesn’t know that we want to load “*.fnl” file, it is using simple regular expression for resource tracking and will complain about missing “mylib.lua” file.

Dummy “mylib.lua” (just empty file) could be created to trick Defold into building project, but this creates another problem. Defold doesn’t know about “mylib.fnl” and will strip that file from bundle, since it isn’t referenced anywhere.

Another approach that didn’t work, involved putting Fennel code directly into “mylib.lua” and inserting Fennel loader (with some modification) at the begging of “package.loaders” table. Problem here is that Defold compiles Lua files into bytecode and then launches the game. Since Luajit doesn’t know a thing about Fennel code yet, this will end up with build errors.

Current approach (simplified code):

local fennel = require("fennel")
local script = sys.load_resource("/fnlscr/fish.fnl")
local scriptcompiled = fennel.compileString(script)
loadstring(scriptcompiled))()

Custom resources specified in game.project can be used for storing additional data in Defold, in this case “/fnlsrc” directory. However those files can’t be accessed with usual function from Lua like “require”. Instead “sys.load_resources” function must be used and it returns string of loaded data.

This string is then compiled from Fennel to Lua code with fennel.compileString function. “loadstring(code)()” idom is then used for loading and running chunk in Lua virtual machine.

What was modified in fennel.lua?

Earlier I mentioned how Defold tracks dependencies in Lua scripts by parsing “require” function. This is problematic for fennel.lua, since this is amalgamated module containing multiple “require” functions, which don’t point to real files. This can be solved by using dummy approach mentioned before. Yet there seems to be a better way. Defold parser isn’t particularly sophisticated, so “require” calls can be changed to “_G.require” and this will avoid dependency resolution.

Even then fennel.lua modified this way won’t work out of box. Normal approach for amalgamated modules is populating “package.preload” table with references to functions in loaded chunk. This then tricks “require”, which should look into table and not search for real files. Problem is that “package.loaders” in Defold is stripped and contains only one function, which doesn’t even look at “package.preload”. So we need to create custom loader:

table.insert(package.loaders, 1, function(modname) return package.preload[modname] end)

I would like to thank @britzl here for providing solution, Defold developers were very helpful.

Issues/caveats?

  • Script properties (go.property) must be used directly in script, so this means Lua code.
  • Hot reloading doesn’t work, since Defold doesn’t track Custom Resources. Creating string with Fennel code directly in Lua script should work, but isn’t that nice either.
local script = [[
(fn is-striped [fish]
  (or (= fish.type type-striped-v) (= fish.type type-striped-h)))
]]
local scriptcompiled = fennel.compileString(script)
loadstring(scriptcompiled))()
  • By default function definitions in Fennel are local, use code like bellow if needed:
(fn _G.init [self])
  • By default functions in Fennel return last value, this can create errors for functions like on_input in Defold (you can put true/false at the end instead) (better solution in reply).
  • Fennel won’t work in editor scripts, since Editor uses luaj virtual machine, which is buggy (and seems unmaintained)

Questions?

If you managed so far, I would like to apologize for quality of my English writing, since this isn’t even my second language. Fennel code in linked example project isn’t the best either - I just used anitfennel (lua-to-fennel translator) and fnlfmt formatter, so it isn’t representative of Fennel in any way.

I believe that even at current state this is worth sharing and hopefully improvements can be made. Lack of hot reloading is kinda disappointing. Yet I feel that interoperability is better compared to other attempts at using non-native language with Defold and maybe will find use by someone, who is looking for alternatives to Lua in conjunction with Defold.

13 Likes

Hi everyone
I will begin by thanking @aytos for his awesome writeup on how to setup and use Fennel in Defold. I will extend with my own adventures with implementing the ahead of time workflow with Fennel in Defold.

A look at Fennel (again?)

As mentioned in the previous post, Fennel is programming language with Lisp-like syntax running on top of Lua. Since it keeps Lua semantics, code transcompiled from Fennel to Lua is very clean.

How do you use it
Fennel can be embedded in applications in two ways -

  • You use the fennel library, which can compile a string at runtime and run it. This approach was explained in beautiful detail in the original post.
  • You can compile Fennel to Lua ahead of time and use the Lua code it generates as is. I explored how to implement this approach in Defold.

Why AOT ? (and why not ?)
As explained in the previous post, using runtime compilation poses some issues, like having go.property in the original script, which is isn’t that helpful when you have a lot of properties that you’re working with. Also, Hot reloading is not supported with custom resources currently.

AOT aims to solves this problem by generating the lua code ahead of time, and using it like any normal lua code. This solves the first issue by providing a way to call go.property directly in script. The second one is solved too as scripts and modules are hot reloadable.
But this approach also has its caveats, with the most major being that it adds a layer of friction over your codebase. You must remember to compile the code to see the changes.

The approach
As Fennel’s compiler writes the compiled code to stdout, I started by modifying the Fennel source (:upside_down_face: ) to allow a way to export the compiled scripts as lua files with the specified extension. It wasn’t the best way I know, and could be surely done in other files that sit around the compiler, but it was surely the easiest way to start. I then wrote a layer of Makefile (Yep those makefiles :sweat_smile: ) around the generated executable to make it possible to batch compile a directory of files at with the specified extension ( to account for Defold’s scripts ). It was then a breeze to compile all the required scripts into the required format and use them as is without any issues.

The Road Ahead
I’m aware that Makefiles and modifying sources is not the best way to go about, so I’m currently looking in editor scripts to find a way to use the official Fennel binaries to produce the same result. Stay Tuned.

Show us something
To test the capabilities of Fennel, here is a port of Sidescroller tutorial written in Fennel (I’m going around posting this everywhere :laughing: )

Answering the big questions
With everything detailed, it is perhaps time to answer the big questions :wink:

Can you use Fennel in Defold?
Absolutely. Fennel is a great language, and both of the methods detailed here make it dead simple to do so. (well not both :grin: )

Should you use Fennel in Defold?
Before you consider using Fennel for your next project, I will like to address several issues that Fennel + Defold had in general

  • Fennel has a habit of forcing returns, which the standard on_input method does not like. As long as there is a bit of code in on_input that returns something other than true or false, you’re bound to get an error. Even though it hamper the code execution in any way, it is still painful to see errors :frowning:
  • Forced returns really become a blocker when you start coding gui scripts. Gui Scripts init function throws an error when a return statement is encountered in its init method. As a follow up, the guiscript is not initialized and hence won’t work at all.

While the first one is ignorable, the second error makes it very difficult to code gui scripts in Fennel.

You still Here?
Thanks for reading through the post. It was a great project to explore the feasibility of a new language in Defold, and was definitely worth it. Feel free to shoot any questions that you might have and I’ll do my best to answer it.

6 Likes

I asked about solution for function with empty return (compared to return nil) on Fennel IRC channel. It seems that (values) at the end of function block with no arguments seems to be best solution (thanks mrichards42).

(fn _G.init [self]
    (your_code)
    (values))

This will create return with no arguments in Lua code. Guess this should be used for all callback functions (init(), on_input(), on_message()) for safety reasons.

1 Like