Native Extensions

Mathias_Westerdahl · November 15, 2017, 10:54am

Of course!
I got so excited about the release I had forgot to prepare my notes… ^^

Customizing the engine build

Apart from specifying your own new extensions, you can also enable/disable libraries via the "App Manifest"
This is a feature that has very little documentation yet (none?).
The basic idea is that it should work very closely to how the extension manifest works (and also our master config for the server).
And although this config design isn’t fully nailed down, we think it’s beneficial your you to get access to this alpha feature so you can
start playing with it (or release with it!).

Config

There are a few new keywords, in perticular: excludeLibs, excludeSymbols, excludeJars which allow you to remove libraries and C++ symbols and also .jar files.
Using the libs keyword, you can add engine libraries, and using symbols you can add engine symbols

Future

Currently, there are not a whole lot of options for you to play with, and to be honest, we’d like to improve on this a lot.
It’s not extremely user friendly to modify a build like this. We’d like more.
What we’d like, is to categorize the engine into more extensions (or features), and allow you to select those features you actually need.
Or, perhaps we could detect this as well: removing components or script modules you don’t use.

How to

Below, you’ll find the three most asked for build variants.
Enabling these is straightforward, albeit a little hiddeen:

Copy an app manifest file to your project (e.g. game_nophysics.appmanifest)
Add a reference to it in your game.project file (edit it as text)

[native_extension]
app_manifest = /game_nophysics.appmanifest

Build or bundle!

Example files

game_release.appmanifest :

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    x86_64-osx:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    js-web:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    x86-win32:
        context:
            excludeLibs: ["libengine"]
            libs: ["libengine_release.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libengine"] 
            libs: ["libengine_release.lib"]

    armv7-android:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

    armv7-ios:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]
            
    arm64-ios:
        context:
            excludeLibs: ["engine"]
            libs: ["engine_release"]

game_headless.appmanifest :

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    x86_64-osx:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    js-web:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    armv7-android:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    armv7-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]
            
    arm64-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null"]

    x86-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib"]

game_nophysics.appmanifest :

# Release + No: Physics + Record + Profiler
platforms:
    x86_64-osx:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    x86_64-linux:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    js-web:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    x86-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics", "librecord", "libvpx", "libengine", "libprofilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["libengine_release.lib", "libphysics_null.lib", "librecord_null.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics", "librecord", "libvpx", "libengine", "libprofilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["libengine_release.lib", "libphysics_null.lib", "librecord_null.lib"]

    armv7-android:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

    armv7-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]
            
    arm64-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics", "record", "vpx", "engine", "profilerext"]
            excludeSymbols: ["FacebookExt", "ProfilerExt"]
            libs: ["engine_release", "physics_null", "record_null"]

EDIT: Upon request, here is a headless release config as well.

game_headless_release.appmanifest:

# Release build
platforms:
    x86_64-linux:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    x86_64-osx:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    js-web:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    armv7-android:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    armv7-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]
            
    arm64-ios:
        context:
            excludeLibs: ["graphics", "sound", "record", "vpx", "tremolo", "hid", "engine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["graphics_null","sound_null", "record_null", "hid_null", "engine_release"]

    x86-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid", "libengine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib", "libengine_release.lib"]

    x86_64-win32:
        context:
            excludeLibs: ["libgraphics", "libsound", "librecord", "libvpx", "libtremolo", "libhid", "libengine"]
            excludeSymbols: ["DefaultSoundDevice", "AudioDecoderWav", "AudioDecoderStbVorbis", "AudioDecoderTremolo"]
            libs: ["libgraphics_null.lib","libsound_null.lib", "librecord_null.lib", "libhid_null.lib", "libengine_release.lib"]

I hope this will at least get you started! We will add proper docs soon!

britzl · November 13, 2017, 7:48pm

Good point! I believe I’m failing in this area in some of my extensions. I’ll take a look at this tomorrow.

Mathias_Westerdahl · November 14, 2017, 9:40am

One issue that will arise more and more regarding adding extensions, is getting link errors due to duplicate symbols. This is not an uncommon problem but it’s fairly easy to remedy if you have full access to all steps in a compile pipeline.

However, since this isn’t possible in a closed system, we have to use other means to resolve it.

Currently, the engine only has a few composite libraries, which include parts of other libs (e.g. some crypto functions). This makes it impossible to remove “just that part” from the engine. Instead, you’ll have to instead rename your incoming symbols. In effect, you have to rename all colliding functions in that library, and recompile that library before using it. A common trick is to rename from “apiFunction()” to “apiFunction2()”, and then you use apiFunction2() in your code.

Of course, that means more work for you users, and we want to make it easier, so we are going to look into splitting our larger libraries, so you can exclude more parts of it, in favor of your new(er) versions.

AGulev · November 20, 2017, 2:43pm

Is it possible to use two different appmanifest:
one for release build and other for debiug build?

Mattias_Hedberg · November 20, 2017, 7:44pm

The easiest way is probably to do it in your build script. Very simple to do in python for example

f = "path/to/game.project"
config = configparser.ConfigParser()
config.read(f)
config.set("project", "title", "DEBUG_TITLE")

with open(f, "w") as fp:
    config.write(fp)

Mattias_Hedberg · December 11, 2017, 2:25pm

I tried stripping only the physics out to see the implications on memory and package size. May as well dump my findings here.

Because I didn’t want to spend too much time I only tested on macOS builds, the memory could differ from device to device though so keep that in mind. I built each version for macOS and got the memory with profiler.get_memory_usage(). I restarted the whole game after each sample and I used 9 samples. Testes done with engine version 1.2.117 and in release mode.

TLDR; The package size decrease is nice but memory implication is pretty meh.

appmanifest

# Release + No: Physics
platforms:
    x86_64-osx:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]
 
    x86_64-linux:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]
 
    js-web:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]
 
    x86-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics"]
            libs: ["libengine_release.lib", "libphysics_null.lib"]
 
    x86_64-win32:
        context:
            excludeLibs: ["libBulletDynamics", "libBulletCollision", "libLinearMath", "libBox2D", "libphysics"]
            libs: ["libengine_release.lib", "libphysics_null.lib"]
 
    armv7-android:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]
 
    armv7-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]
             
    arm64-ios:
        context:
            excludeLibs: ["BulletDynamics", "BulletCollision", "LinearMath", "Box2D", "physics"]
            libs: ["engine_release", "physics_null"]

Memory

Data of an empty project, with physics

	Empty	5 Collections from file	5 Collection Proxies	5 Collection Proxies - Loaded
Average MB	38.9	38.8	38.7	40.5
Average Bytes	38884693	38763633	38673066	40482588

The data collected indicates that each collection proxy adds 0.36 MB of memory.

Data of an empty project, without physics

	Empty	5 Collections from file	5 Collection Proxies	5 Collection Proxies - Loaded
Average MB	38.6	38.4	38.6	40.2
Average Bytes	38600704	38420024	38642574	40194048

Interestingly the data we get now for 5 collection proxies loaded is 1MB lower compared to having it with physics, but it is also still 1.8 MB over the other values. The memory added per collection is 0.36, meaning a 0.09 difference between with and without physics.

Package size

	With	Without	Diff	Diff -MB
Android	2801489	2491918	-309571	-0.3
iOS	2619459	2263253	-356206	-0.35
macOS	5408039	4230099	-1177940	-1.2

Other observations

profiler.get_memory_usage() is broken, doesn’t indicate that memory is released and only counts up. Making it only really reliable on build/app start.

If you have physics objects in your build and strips physics out the build will not be able to start but will not give any errors as far I noticed.

I should (but most likely won’t) test it out on android and/or iPhone to investigate if it is comparable with macOS.

Pkeod · December 2, 2017, 1:23am

Prompted by the other thread I made a github repo with the existing appmanifest files for easy access and to collect others made over time.

I will do some tests too to collect data and create other useful appmanifest files.

Oleg_The_Evangelist · December 4, 2017, 8:43am

Or may be these could be also useful if put to the docs?
Summoning @sicher in case he has an opinion =]

ross.grams · January 21, 2018, 8:17pm

I have a couple little questions/clarifications regarding appmanifest options:

By default, are both 3D and 2D physics engines included in all projects?
If yes, I guess it makes sense for all 2D projects to exclude 3D physics, and vice versa, so…
Which libs correspond to 3D physics, 2d physics, or both?
- 2a. I assume “BulletDynamics” and “BulletCollision” for 3D physics and “Box2D” for 2D?
- 2b. What about “LinearMath” and “physics”?
“vpx” is just for the built-in video recorder, right?

Mathias_Westerdahl · January 22, 2018, 10:10am

Yes, the default engine includes both 2D and 3D physics
Unfortunately, we need to do some more work for that to happen. Currently you can only use both engines or none. (Added DEF-3097 for this)
Vpx is for the recording functionality yes

AGulev · February 23, 2018, 10:40am

Whitch function should be on the place of zero?

dapetcu21 · February 23, 2018, 10:47am

OnMessage or something like that. It says in the docs

sicher · February 23, 2018, 10:51am

// DM_DECLARE_EXTENSION(symbol, name, app_init, app_final, init, update, on_event, final)

Mathias_Westerdahl · February 23, 2018, 11:20am

Exactly as @dapetcu21 and @sicher said, and you can find the function declaration here.
Currently, we only support two events, app activate/deactive, but in the future we could add more events.

AGulev · February 23, 2018, 11:35am

Is Activate calls every time when focus back? Something like applicationDidBecomeActive on ios?
Thanks for the link to docs, I missed when searching.

Mathias_Westerdahl · February 23, 2018, 12:53pm

The activate/deactivate events should happen at the same as the Lua window events. We currently have some discrepancies registered in DEF-2520: Android + HTML5: missing first focus gained, Windows: getting the focus lost when focus gained, iOS: no focus lost on voice control focus.

AGulev · February 24, 2018, 3:05pm

Hello, I need your help, guys.
I have strange memory leak in .mm file:

But with this simple code:

Code:

void DefAppsFlyer_trackEvent(const char*eventName, dmArray<TrackData>* trackData)
{
  NSMutableDictionary* newDict = [NSMutableDictionary dictionary];
  NSString* key;
  NSString* value;
  TrackData data;
  for(uint32_t i = 0; i != trackData->Size(); i++)
  {
    data = (*trackData)[i];
    key = [NSString stringWithUTF8String: data.key];
    value = [NSString stringWithUTF8String: data.value];
    newDict[key] = value;
  }
}

What I am doing wrong here?
As I understand I am using only “safe” variable creation, and they should be free-up the memory.
I have memory leak even if I leave only one line:

NSMutableDictionary* newDict = [NSMutableDictionary dictionary];

Is this my mistake? Or something works wrong?

dmitriy · February 24, 2018, 4:57pm

Is that code is running on a thread other than main thread?

Does this help?
@autoreleasepool {
// your code inside method
}

AGulev · February 25, 2018, 9:02am

Yes, it helps in situation with creation of Dictionary with Strings, but has no effect when I add SDK call:

[[AppsFlyerTracker sharedTracker] trackEvent: evName withValues: newDict];

=(

AGulev · February 26, 2018, 5:46pm

I checked with:

if ([NSThread isMainThread])

it’s the Main thread.

In cocos2d ( appsflyer sdk implementation ) I found only one autoreleasepool mentioning :

github.com

AppsFlyerSDK/AppsFlyerCocos2dX/blob/a245d3b65fc671817596d6c4749f96acb2e3d334/proj.ios_mac/ios/main.m

#import <UIKit/UIKit.h>

int main(int argc, char *argv[]) {
    @autoreleasepool {
        return UIApplicationMain(argc, argv, nil, @"AppController");
    }
}