Writing native extensions in Rust

The Defoldmine
#1

Just to set your expectations first:

  • Only bundling for Windows and Linux is currently supported
  • You must host your own build server
  • :warning: The build server provides no sandboxing or security against malicious extensions

Hey all! I’ve been working on a way to make use of Rust in native extensions for a while and it’s now in a state that I consider to be ready for release! This project is split up into a few different parts:

It works the same way as regular C/C++ extensions: everything is done on the build server and you only need the standard Defold editor to build games. For writing Rust code, you’ll want to use VSCode (or your favorite LSP-compatible IDE) with rust-analyzer installed.

These extensions are full Cargo projects. The main benefit being that you can use any of the Rust libraries on crates.io in your Defold game just by listing them in your Cargo.toml. Theoretically, you could even write a game with the Bevy game engine and embed it into a Defold project!

Getting started

  1. Download and run the build server
    • The process is (mostly) the same as normal
  2. Make a copy of the template project
  3. In the editor, set File > Preferences > Extensions > Build Server to the URL of the build server
    • This should be http://localhost:9000 if you’re running the server on your computer
  4. Open up /myextension/src (the folder containing Cargo.toml) in your IDE of choice
  5. Once you’re done writing your extension, just press build in Defold as always
    • Rust extensions will take much longer to build at first than C++, since it needs to compile several dependencies for the bindings

Project Status

Supported Platforms

  • :ballot_box_with_check: Windows
  • :ballot_box_with_check: Linux
  • :black_large_square: macOS
  • :black_large_square: HTML5
  • :black_large_square: iOS
  • :black_large_square: Android
  • :x: PS4/PS5
  • :x: Nintendo Switch

Features

16 Likes
#2

Update time!

I’ve been learning a lot more about macros in Rust (and how not fun they are to write…) and have used my new skills to reduce a lot of boilerplate in extension code.

First up: declare_functions

This one creates a lua::Reg constant (an array containing function names and pointers) and wraps your functions with the required #[no_mangle] extern "C" boilerplate.

Before

#[no_mangle]
extern "C" fn reverse(l: lua::State) -> i32 {
    // ...
}

const EXTENSION_FUNCTIONS: lua::Reg = &[
    ("reverse", reverse)
];

:sparkles: After :sparkles: 

fn reverse(l: lua::State) -> i32 {
    // ...
}

declare_functions!(EXTENSION_FUNCTIONS, reverse);

Next: logging macros

The functions in dmlog like dmlog::info() have been replaced with macros! This gives it the ability to automatically detect the log domain as well as format the provided string, just like how you’d do it in C++.

Before

let lucky_number = 7;
dmlog::info("MY_EXTENSION", &format!("Your lucky number is: {lucky_number}"));

:sparkles: After :sparkles: 

let lucky_number = 7;
dmlog::info!("Your lucky number is: {lucky_number}");

Lastly: an updated declare_extension

All of the extension lifecycle functions are now wrapped using the same tech as declare_functions. Besides removing yet more boilerplate, it also means that the library can provide more Rust-friendly arguments and return types instead of using raw pointers everywhere.

Before

#[no_mangle]
extern "C" fn app_init(params: dmextension::AppParams) -> i32 {
    dmextension::RESULT_OK
}

#[no_mangle]
extern "C" fn ext_init(params: dmextension::Params) -> i32 {
    dmextension::RESULT_OK
}

#[no_mangle]
extern "C" fn on_event(params: dmextension::Params, event: dmextension::Event) {
    let event_id = unsafe { (*event).m_Event };

    match event_id {
        0 => println!("EVENT_ID_ACTIVATE_APP"),
        1 => println!("EVENT_ID_DEACTIVATE_APP"),
        2 => println!("EVENT_ID_ICONFIY_APP"),
        3 => println!("EVENT_ID_DEICONIFY_APP"),
        _ => println!("Unknown event ID"),
    };
}

declare_extension!(
    MY_EXTENSION,
    Some(app_init),
    None,
    Some(ext_init),
    None,
    None,
    Some(on_event)
);

:sparkles: After :sparkles: 

fn app_init(params: dmextension::AppParams) -> dmextension::Result {
    dmextension::Result::Ok
}

fn ext_init(params: dmextension::Params) -> dmextension::Result {
    dmextension::Result::Ok
}

fn on_event(params: dmextension::Params, event: dmextension::Event) {
    match event {
        Event::ActivateApp => println!("EVENT_ID_ACTIVATE_APP"),
        Event::DeactivateApp => println!("EVENT_ID_DEACTIVATE_APP"),
        Event::IconifyApp => println!("EVENT_ID_ICONFIY_APP"),
        Event::DeiconifyApp => println!("EVENT_ID_DEICONIFY_APP"),
        Event::Unknown => println!("Unknown event ID"),
    };
}

declare_extension!(
    MY_EXTENSION,
    Some(app_init),
    None,
    Some(ext_init),
    None,
    None,
    Some(on_event)
);

You can see all of these changes put to use in the template project, which I always try to keep updated with the latest developments.

Next up will be some build server improvements, (hopefully) including Windows support and a bit of caching to help compile times on first launch. Until next time! :wave:

11 Likes
#3

That is really awesome! Cannot wait till next news, thanks for such contribution!

3 Likes
#4

Thanks for the kind words, and welcome to the forum! I planned to have another update out already, but Clang has been very uncooperative, to put it lightly.

3 Likes
#5

That took way longer than expected, but both of these are now implemented!

Windows Support

There’s not much to say here other than that the build server now allows both 64-bit and 32-bit Windows targets, so I’ll talk a bit about the changes I’ve made to the process for compiling the build server’s Docker image.

Normally for each platform Defold supports, you either need to have the SDKs or do some surgery on the Dockerfile to remove it entirely. Once that’s done, you would run a command like:

$ DM_PACKAGES_URL="http://localhost:9999" ./server/scripts/build.sh

I’ve changed it so that every platform is disabled by default and requires an environment variable to re-enable. For example, if you have the Windows SDKs (that’s Microsoft-Visual-Studio-2019-14.25.28610.tar.gz and WindowsKits-10.0.18362.0.tar.gz) you can set a flag to enable support for building Windows games:

$ WINDOWS_SDK=1 DM_PACKAGES_URL="http://localhost:9999" ./server/scripts/build.sh

The exception to this is Linux which is always enabled. That’s because 1) I like Linux and 2) it only needs this file from LLVM’s GitHub. This is something that should be in the vanilla build server IMO, although it would probably be better to use a blacklist system instead so as to not break existing workflows.

Caching

Cargo is now set to put all compiled artifacts to a global folder, making compile times only a few seconds on subsequent builds! This is kind of a hacky way to implement a cache, but I still can’t figure out sccache and Stack Overflow says it’s probably fine to do. The only caveat I’ve heard is that it might break if two projects depend on the same version of the same crate but with different feature flags. This should be pretty rare and since the build server isn’t ready for serious use anyway, I think it’s an acceptable risk.

I imagine some people will want to give this a try now that it works on Windows, so I’ll recommend No Boilerplate’s 10-minute Rust overview to learn the language’s basic syntax and features. His videos (and music (and podcasts)) are really some of the best out there!

After all the trial and error this one took I’ll be going through and cleaning up everything while I think about what to work on next. Managing different Defold versions is still a big problem that needs to be solved and there are still plenty of dmSDK functions to write bridges for. Either way, this project is a lot of fun to work on and I hope to get another update out soon! :wave:

8 Likes
#6

It’s a pretty good idea to be honest. We’ve also discussed if we could break up the build server per platform. Could you please create a feature request in the extender repo so that we can discuss this and your solution?

Why is this a problem? Could you please share more info?

Could this perhaps be generated?

4 Likes
#7

Oh, and good job on the latest improvements!!! :rocket: :tada:

4 Likes
#8

Done!

I really overstated how big of an issue it is. What I meant is that while it’s easy to update the Rust bindings to the latest version of the dmSDK, I don’t know of a way to support multiple versions at the same time and let users tell Cargo which Defold version they want bindings for.

Yes and no. The Rust bindings are made up of two different libraries:

  • dmsdk_ffi - Automatically generated bindings using the headers from defoldsdk.zip and rust-bindgen. While you can use just these to write an extension, the ergonomics aren’t great (different naming convention, lots of raw pointers, etc.) so we make some nice Rust-friendly wrappers/bridges:
  • dmsdk - Essentially a partial rewrite of the C++ dmSDK in Rust that uses dmsdk_ffi to interop with Defold. For functions, some only need renaming while others need a decent amount of code to convert between Rust and C++ data. Enums are obviously straightforward but structs need to be completely rewritten for Rust’s type system. Many of these things could be generated automatically through Rust’s great macros, but it still takes manual work.

Thanks, and sorry for the late reply!

4 Likes
#9

Lots of smaller changes this time around so here’s some patch notes to summarize:

dmsdk_ffi

  • Updated to Defold 1.4.1
  • Now uses unmodified dmSDK headers (from defoldsdk.zip on d.defold.com)
    • The headers used to be slightly edited, but that’s no longer needed thanks to some changes on the Rust side
  • Uploaded to crates.io

dmsdk

  • Everything now has proper documentation and the project is configured to throw a warning if there’s any missing
    • You can view all the documentation online via docs.rs
  • Removed dmjson
  • Uploaded to crates.io
    • This means you can now replace dmsdk = { git = "https://github.com/JustAPotota/defold-rs" } with dmsdk = "0.1.0" to stay on a stable version instead of the latest commit

dmconfigfile

  • Added declare_configfile_extension!() macro to match the one added in 1.4.1

lua

  • Added check_int(), check_float(), error!(), and push_fstring()!
  • check_bytes() no longer early returns on a null byte

Example Project

  • Added tests for the lua::check_* functions
  • Added config plugin example

To complement all this API documentation, I’ll be filling out the project’s READMEs (they look so sad and empty!) so anyone not on the forums will actually know what this is all about. To keep the technical side of my brain active, I’ll also look into support for HTML5. No guarantee it’ll be in the next update, but hopefully it won’t take as much time as Windows. I think posting an update every 2 weeks is a good schedule for my slow development speed, so expect another post then! :wave:

10 Likes
#10

Hope you all had a happy holidays! This will be a very short post as I haven’t gotten nearly as much done with this project as I had planned, but there’s now a nice little README (and logo!) on the repo and crate pages.

As for HTML5 support, that’s not going great so far: Clang (which the bindings generator uses) doesn’t seem to play very nicely with Emscripten for whatever reason. It looks to be the same kinds of issues I ran into with Windows support a month ago, so I’m sure I’ll get something working before long.

6 Likes
#11

I’m not sure but you can try to update Emscripten to the latest version in your local extender environment. First time try that without your Rust extras just to be sure that the update works well, then with Rust.

#12

Still no progress on web Rust, but I think there’s enough here to make up for it :slightly_smiling_face: .

dmsdk 0.2.0

dmhid

  • Added dmhid module, with the entire C++ API ported (docs)
    • The API has been changed to be more Rust-friendly by using structs instead of pointers (which is something I want to do more of in the future)
Comparison

C++

dmHID::HContext hidContext = dmEngine::GetHIDContext(appParams);
dmHID::HMouse mouse = dmHID::GetMouse(hidContext, 0);
if (mouse != dmHID::INVALID_MOUSE_HANDLE) {
    dmHID::SetMouseButton(mouse, dmHID::MOUSE_BUTTON_LEFT, true);
}

Rust

let hid_context = dmengine::get_hid_context(app_params);
if let Some(mouse) = hid_context.get_mouse(0) {
    mouse.set_button(dmhid::MouseButton::Left, true);
}

(Maybe one of these days we’ll get better syntax highlighting…)

dmlog

  • Changed the logging macros to allow usage across modules
    • Previously, only the module containing the declare_extension!() call could use them
    • This is the first step in improving the example project to make it less of a mess

lua

  • Added lua::to_bool()
  • Refactored the lua module to no longer require unsafe
    • Internally, lua::State has been changed from a pointer to a struct containing the pointer to provide a bit more safety. The declare_functions!() macro takes care of the conversion for you, so you only need to remove the unsafe marker from your code. As always, the template project has been updated accordingly!
    • The only remaining unsafe functions are lua::to_userdata() and State::new(), which are both actually unsafe

While the HID module isn’t too useful as it was made just for the Poco extension, it’s satisfying getting an entire module up and running in one go (even if that includes writing 126 enum variants by hand :sweat:) . I’m also pretty happy with how the conversion from pointers to structs went as it was something I thought about early on in the project but didn’t know how to do. Since I make absolutely no guarantees about backwards compatibility, expect to see more of that coming to current and future modules!

7 Likes