Native Extensions

[britzl]

No, I don’t think so, but other things came up and then also summer vacations for the team. @Mathias_Westerdahl knows more.

[Mathias_Westerdahl]

I’m finally back from vacation and a cold, and I’ll continue with the linux support now

[Mathias_Westerdahl]

Here’s a tiny update on what’s currently going on with the Linux support for Native Extensions.

In an effort to make our build process easier to maintain, we’re currently updating out build servers to use the same Docker container as the extension server. This is an Ubuntu 16.04 version, and for that to work, we also needed to rebuild and/or update many of the libraries. E.g. updating to LuaJIT-2.0.5 to add support for OSX 64 bit and Windows is a good example.

At some point, we all thought that the Linux platform would be the easiest, but now it has proven to take as long as the android support. doh

We’ve made the decision to cut support for 32 bit OSX builds, since the all our supported macOS versions are already 64 bit, and macOS is moving towards 64-bit-only support in the app store.

Along those lines, we’ve decided to stop supporting 32 bit linux builds too, since the apparent user base is really really low. The Editor2 for instance is already only shipping on Linux 64.

There are still a few loose ends to finish up, but my current estimate is that Linux support for native extensions is 1 or 2 sprints away.

[Mattias_Hedberg]

1 or 2 sprints away

IIRC Defold have 2 weeks sprints so: 2 or 4 weeks

[AGulev]

What is the best way to close game from the c++ code? something like msg.post("@system:", “exit”, {code = 0}) but in NE c++
UPD: I am use ExitProcess(0); on Windows, thanks @Sublustris

[Mathias_Westerdahl]

Today we had an exploration day, which means we can try out various ideas that could be beneficial to the engine, but aren’t necessarily scheduled soon (or at all). (Other companies call it Hack Day etc)

What I tried today was to tie together a test branch we had that @ChristianM had prepared, where we have an extra physics library: physics_null, and the Native Extensions’ App Manifest.

The app manifest lets you modify what libs/symbols/jars etc that should or shouldn’t get linked into the engine.
After some small modifications, I got it to work nicely. I still have a few things to clean up (or think through), but this functionality should arrive in a sprint or two

My example game.appmanifest:

platforms:
    # Null Physics
    common:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["physics_null"]

And some before/after results on x86_64-osx:

mawe@physics_null (master) $ ls -la dmengine*
-rwxr--r--  1 mawe  staff  4463180 Oct 20 15:58 dmengine_no_physics
-rwxr--r--  1 mawe  staff  3375996 Oct 20 15:58 dmengine_no_physics_stripped
-rwxr--r--  1 mawe  staff  5619372 Oct 20 15:59 dmengine_physics
-rwxr--r--  1 mawe  staff  4111380 Oct 20 16:00 dmengine_physics_stripped

4111380 - 3375996 = 735384 bytes saved

[AGulev]

4111380 - 3375996 = 735384 bytes saved

What about ios/android?

[Mathias_Westerdahl]

Yeah, I didn’t have time to try out any other platforms. I’ll post some numbers on iOS/Android next week I hope (If I don’t forget it ).

[Mathias_Westerdahl]

Ok, some more numbers (in bytes).
First off, comparing some platforms:

OSX + iOS + Android

• Stripped
• Removed: Physics + Record

4111380 dmengine
3061760 dmengine_opt

3006220 ios_dmengine_64
2670316 ios_dmengine_64_opt

3398552 libdmengine.so
2718520 libdmengine_opt.so

iOS

• Stripped
• 64 bit exe only
• Removed: Physics + Record + Facebook
• Different -O* flags for object files

3006220 ios_dmengine_orig
2171684 ios_dmengine_O2_opt
1993980 ios_dmengine_O1_opt
1961532 ios_dmengine_Os_opt
1862388 ios_dmengine_Os_NDEBUG_opt # with define NDEBUG

In this test, I haven’t removed the builtin debug app, of which the embedded data file is ~185kb which also would be nice to get rid of.

Still, a way to go before reaching our 1mb dream goal, but still… it is good to be able to shave off some bytes every now and then

[Fuscy]

While on mobile platforms this is not so noticeable as the overhead reductions are tiny compared to total app size, it is good to allow memory to be optimized. I guess performance doesn’t change because those modules should be run only if in use.

This will be brilliant on HTML where each extra MB is about 1 second of wait time for the load.

[Mathias_Westerdahl]

One of the goals is of course to make the html5 engine as small and fast as possible. And, a smaller executable will also load faster on any platform.

[Mathias_Westerdahl]

The Linux support is progressing, although it (again) got side tracked by a bunch of things this sprint too ^^
But once I got to actually sit down with it, it came together nicely. Now I need to clean up some code etc. and do more testing.
It won’t make it into the 1.2.115 1.2.116 release, but should be in the next one after that.

Here’s a small preview:

Another thing that will be in next sprint is the improved app manifest support, that will allow you to add or remove (some) libraries that you might want (or not). E.g. to build a release or headless build, or remove physics and/or facebook…

[Mathias_Westerdahl]

@ChristianM was curious on the numbers of the HTML5 build too, so I started doing the same for that platform too.
While looking at the numbers, we started discussing more and more things we wanted to remove or try removing.
A classic, it’s just too fun to not start optimizing a bit while at it ^^

Here’s the bigger things I did:

• ndebug means I compiled with -DNDEBUG. It removes the asserts() and the strings they contain. It also removes various amounts of debug code.
• <libname> means I removed that lib or functionality. Either statically, or using the app manifest.
• I removed the logging macros, and profile, debug rendering and the built in Connect App from dmengine_release. This should save ~200kb for each platform in the release build
• After building, I gzipped the custom built dmengine.js from the native extension build

1328132 dmengine.js.tar.gz -- original
1129375 dmengine_physics.js.tar.gz
1078882 dmengine_ndebug_physics.js.tar.gz
1051503 dmengine_ndebug_physics_log.js.tar.gz
 899918 dmengine_log_physics_dbgrender_profile_builtins.js.tar.gz
 890807 dmengine_log_physics_dbgrender_profile_builtins_facebook.js.tar.gz
 859237 dmengine_ndebug_log_physics_render_builtins.js.tar.gz

Using gzip -9 instead of tar (which I should have done from the start):

 877437 dmengine_gzip9_log_physics_render_profile_builtins_facebook.js.tar.gz
 846926 dmengine_gzip9_ndebug_log_physics_render_builtins.js.tar.gz

The NDEBUG option shaves off an extra ~30, but that also means we internally have to update more
on our build system side (and on the Native Extension server), so that will have to come a little later. There are easier things to do first.

I feel confident that the executable can be even smaller. We have a few other options to explore that will take various amounts of time to implement. Both in general, but also depending on what features you actually need.

Hopefully, this added functionality should land in a release soon

EDIT:
Here are numbers on some of the source .js files I compressed for the test:

5736253 dmengine_original.js
4884828 dmengine_physics.js
4437975 dmengine_ndebug_physics_log.js
4139089 dmengine_log_physics_dbgrender_profile_builtins.js
4096556 dmengine_log_physics_dbgrender_profile_builtins_facebook.js
3821933 dmengine_ndebug_physics_log_builtins.js
3615977 dmengine_ndebug_log_physics_dbgrender_builtins.js

[Oleg_The_Evangelist]

wait… that is UNDER a meg! I smell instant games here =]
So putting the engine on a diet seems to be fun

[Mathias_Westerdahl]

A quick note on supporting multiple platforms in your extensions.

When supporting cross platform code, I often use one of these tricks to catch “errors”.

Compile error

If I don’t support a certain platform, and if there is no reason or good way to hide it. I make sure that I get a clear error message when compiling:

#if defined(DM_PLATFORM_IOS) || defined(DM_PLATFORM_ANDROID)
    // my working code
#else
    #error "Platform not supported for this extension!"
#endif

Imho, it’s a LOT clearer than error LNK2019: unresolved external symbol <symbol> referenced in function...

Adding a “null” implementation

Sometimes is feels “nicer” to add a dummy, non working or mock implementation for those platforms not supported:

#if defined(DM_PLATFORM_IOS) || defined(DM_PLATFORM_ANDROID)
    // my working code
#else
// for the rest of the unsupported platforms
static dmExtension::Result AppInitializeExtension(dmExtension::AppParams* params)
{
    dmLogWarning("Registered %s (null) Extension\n", MODULE_NAME);
    return dmExtension::RESULT_OK;
}

static dmExtension::Result InitializeExtension(dmExtension::Params* params)
{
    return dmExtension::RESULT_OK;
}

static dmExtension::Result AppFinalizeExtension(dmExtension::AppParams* params)
{
    return dmExtension::RESULT_OK;
}

static dmExtension::Result FinalizeExtension(dmExtension::Params* params)
{
    return dmExtension::RESULT_OK;
}

#endif

DM_DECLARE_EXTENSION(EXTENSION_NAME, LIB_NAME, AppInitializeExtension, AppFinalizeExtension, InitializeExtension, 0, 0, FinalizeExtension);

These two tricks will save yourself (and others) a lot of development and debugging time!

[sergey.lerg]

Agree. I use the dummy module approach, where I assign each function a stub print message.

#ifndef DM_PLATFORM_ANDROID
#ifndef DM_PLATFORM_IOS

#include <dmsdk/sdk.h>
#include "extension.h"

static int stub(lua_State *L) {
	dmLogInfo(EXTENSION_NAME_STRING " extenstion is not supported on this platform.");
	return 0;
}

dmExtension::Result APP_INITIALIZE(dmExtension::AppParams* params) {
	return dmExtension::RESULT_OK;
}

dmExtension::Result APP_FINALIZE(dmExtension::AppParams* params) {
	return dmExtension::RESULT_OK;
}

dmExtension::Result INITIALIZE(dmExtension::Params* params) {
	const luaL_Reg lua_functions[] = {
		{"enable_debug", stub},
		{"init", stub},
		{"show_banner", stub},
		{"remove_banner", stub},
		{"load_interstitial", stub},
		{"show_interstitial", stub},
		{"is_interstitial_ready", stub},
		{"load_video", stub},
		{"show_video", stub},
		{"is_video_ready", stub},
		{"load_rewarded_video", stub},
		{"show_rewarded_video", stub},
		{"is_rewarded_video_ready", stub},
		{"load_offerwall", stub},
		{"show_offerwall", stub},
		{"is_offerwall_ready", stub},
		{"load_more_apps", stub},
		{"show_more_apps", stub},
		{"is_more_apps_ready", stub},
		{"load_native", stub},
		{"get_native", stub},
		{"is_native_ready", stub},
		{"native_trigger_display", stub},
		{"native_trigger_click", stub},
		{NULL, NULL}
	};

	luaL_openlib(params->m_L, EXTENSION_NAME_STRING, lua_functions, 1);
	return dmExtension::RESULT_OK;
}

dmExtension::Result UPDATE(dmExtension::Params* params) {
	return dmExtension::RESULT_OK;
}

dmExtension::Result FINALIZE(dmExtension::Params* params) {
	return dmExtension::RESULT_OK;
}

DECLARE_DEFOLD_EXTENSION

#endif
#endif

[AGulev]

Can you add some instruction about engine customization(stripping)? Which libs supports and so on, pls!

[Mathias_Westerdahl]

Of course!
I got so excited about the release I had forgot to prepare my notes… ^^

Customizing the engine build

Apart from specifying your own new extensions, you can also enable/disable libraries via the "App Manifest"
This is a feature that has very little documentation yet (none?).
The basic idea is that it should work very closely to how the extension manifest works (and also our master config for the server).
And although this config design isn’t fully nailed down, we think it’s beneficial your you to get access to this alpha feature so you can
start playing with it (or release with it!).

Config

There are a few new keywords, in perticular: excludeLibs, excludeSymbols, excludeJars which allow you to remove libraries and C++ symbols and also .jar files.
Using the libs keyword, you can add engine libraries, and using symbols you can add engine symbols

Future

Currently, there are not a whole lot of options for you to play with, and to be honest, we’d like to improve on this a lot.
It’s not extremely user friendly to modify a build like this. We’d like more.
What we’d like, is to categorize the engine into more extensions (or features), and allow you to select those features you actually need.
Or, perhaps we could detect this as well: removing components or script modules you don’t use.

How to

Below, you’ll find the three most asked for build variants.
Enabling these is straightforward, albeit a little hiddeen:

1. Copy an app manifest file to your project (e.g. game_nophysics.appmanifest)
2. Add a reference to it in your game.project file (edit it as text)

[native_extension]
app_manifest = /game_nophysics.appmanifest

3. Build or bundle!

Example files

game_release.appmanifest :

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    x86_64-osx:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    js-web:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    x86-win32:
        context:
            excludeLibs: ["libengine"]
            libs: ["libengine_release.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libengine"] 
            libs: ["libengine_release.lib"]

    armv7-android:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    armv7-ios:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]
            
    arm64-ios:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

game_headless.appmanifest :

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    x86_64-osx:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    js-web:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    armv7-android:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    armv7-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]
            
    arm64-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    x86-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib"]
            

game_nophysics.appmanifest :

# Release + No: Physics + Record + Profiler
platforms:
    x86_64-osx:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    x86_64-linux:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    js-web:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    x86-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics", "librecord", "libvpx", "libengine", "libprofilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["libengine_release.lib", "libphysics_null.lib", "librecord_null.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics", "librecord", "libvpx", "libengine", "libprofilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["libengine_release.lib", "libphysics_null.lib", "librecord_null.lib"]

    armv7-android:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    armv7-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]
            
    arm64-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]
            

EDIT: Upon request, here is a headless release config as well.

game_headless_release.appmanifest:

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    x86_64-osx:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    js-web:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    armv7-android:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    armv7-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]
            
    arm64-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    x86-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid", "libengine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib", "libengine_release.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid", "libengine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib", "libengine_release.lib"]
            

I hope this will at least get you started! We will add proper docs soon!

[britzl]

These two tricks will save yourself (and others) a lot of development and debugging time!

Good point! I believe I’m failing in this area in some of my extensions. I’ll take a look at this tomorrow.

[Mathias_Westerdahl]

One issue that will arise more and more regarding adding extensions, is getting link errors due to duplicate symbols. This is not an uncommon problem but it’s fairly easy to remedy if you have full access to all steps in a compile pipeline.

However, since this isn’t possible in a closed system, we have to use other means to resolve it.

Currently, the engine only has a few composite libraries, which include parts of other libs (e.g. some crypto functions). This makes it impossible to remove “just that part” from the engine. Instead, you’ll have to instead rename your incoming symbols. In effect, you have to rename all colliding functions in that library, and recompile that library before using it. A common trick is to rename from “apiFunction()” to “apiFunction2()”, and then you use apiFunction2() in your code.

Of course, that means more work for you users, and we want to make it easier, so we are going to look into splitting our larger libraries, so you can exclude more parts of it, in favor of your new(er) versions.