Building custom engine

There’s a bug in the engine which is problematic for us (I previously reported it here: https://github.com/defold/defold/issues/5287). As we’re coming close to app release, and the bug hasn’t been fixed yet, I’m trying to patch the defold engine myself.

I’m running into issues trying to build the engine on a clean repo with tag 1.2.176 for iOS on macOS Catalina. I packaged the sdks under./local_sdks:

MacOSX10.15.sdk.tar.gz
XcodeDefault12.0.xctoolchain.tar.gz
iPhoneOS14.0.sdk.tar.gz
iPhoneSimulator14.0.sdk.tar.gz

And I installed the dependencies: ./scripts/build.py install_ext --platform=armv7-darwin --package-path=./local_sdks

Both worked. But when trying to build the engine (./scripts/build.py build_engine --platform=armv7-darwin), I get the following, which I am not sure how to handle:

[exec] git remote get-url origin
Running 'build_engine'
env DM_BOB_ROOTFOLDER=/var/folders/hz/qn05zbsd5fj59kqv03mhzdqm0000gs/T/bob-light-vSsKwV
Building dlib for x86_64-darwin
[exec] python /Users/marco/Dev/defold/tmp/dynamo_home/ext/bin/waf --prefix=/Users/marco/Dev/defold/tmp/dynamo_home --skip-tests distclean configure build install --platform=x86_64-darwin --skip-tests --skip-build-tests
'distclean' finished successfully (0.010s)
Checking for program valgrind            : not found
Checking for program ccache              : ok /usr/local/bin/ccache
Checking for program nodejs              : not found
Checking for program dsymutil            : ok /usr/bin/dsymutil
Checking for program zip                 : ok /usr/bin/zip
Checking for program gcc,cc              : ok /Users/marco/Dev/defold/tmp/dynamo_home/ext/SDKs/XcodeDefault12.0.xctoolchain/usr/bin/clang
Checking for program g++,c++             : ok /Users/marco/Dev/defold/tmp/dynamo_home/ext/SDKs/XcodeDefault12.0.xctoolchain/usr/bin/clang++
Checking for program javac               : ok /usr/bin/javac
Checking for program java                : ok /usr/bin/java
Checking for program jar                 : ok /usr/bin/jar
'configure' finished successfully (0.049s)
Waf: Entering directory `/Users/marco/Dev/defold/engine/dlib/build'
Waf: Leaving directory `/Users/marco/Dev/defold/engine/dlib/build'
At least one compiler (gcc, ..) must be selected
None
Traceback (most recent call last):
  File "./scripts/build.py", line 2143, in <module>
    f()
  File "./scripts/build.py", line 1024, in build_engine
    self._build_engine_lib(args, lib, host, skip_tests = skip_tests)
  File "./scripts/build.py", line 989, in _build_engine_lib
    run.env_command(self._form_env(), args + plf_args + self.waf_options + skip_build_tests, cwd = cwd)
  File "/Users/marco/Dev/defold/build_tools/run.py", line 63, in env_command
    return _exec_command(args, shell = False, stdout = None, env = env, **kwargs)
  File "/Users/marco/Dev/defold/build_tools/run.py", line 41, in _exec_command
    raise ExecException(process.returncode, output)
run.ExecException

It looks like a lib called “wafadmin” is missing an env var that tells it which compiler to use? I tried injecting the env var CC_NAME into the build script, but couldn’t.

Any idea what’s wrong here?

I’m not sure how finicky the rest of the build process is and if this is just the first issue of lots to come, so if someone with a working iOS build setup is willing to recompile the engine with my change, I’d be very happy too :wink:

My end goal is to tell the engine to only use public dns servers (https://github.com/defold/defold/blob/master/engine/dlib/src/dlib/dns.cpp#L50) to work around the iOS permission issue.
I’m obviously not able to test this yet, but changing the function at https://github.com/defold/defold/blob/master/engine/dlib/src/dlib/dns.cpp#L198 to the following should most likely do this:

static void ConfigureChannel(Channel* channel)
{
    ares_set_servers_csv(channel->m_Handle, DEFAULT_DNS_SERVERS);
}

If it says “At least one compiler (gcc, …) must be selected” or if it seems to find a local compiler, it means something went wrong when it looks for the sdks.

This is what is should look like:

Checking for program gcc,cc : ok /Users/mawe/work/defold/tmp/dynamo_home/ext/SDKs/XcodeDefault12.0.xctoolchain/usr/bin/clang

After the “install_ext” step, you should find the sdk packages unpacked under tmp/dynamo_home/ext/SDKs:

~/work/defold $ ls tmp/dynamo_home/ext/SDKs
drwxr-xr-x 10 mawe staff 320 Nov 11 08:37 ./
drwxr-xr-x 8 mawe staff 256 Oct 11 2018 …/
drwxr-xr-x 8 mawe staff 256 Nov 6 16:07 MacOSX10.15.sdk/
drwxr-xr-x 5 mawe staff 160 Nov 6 16:10 XcodeDefault11.5.xctoolchain/
drwxr-xr-x 5 mawe staff 160 Nov 6 17:01 XcodeDefault12.0.xctoolchain/
drwxr-xr-x 21 mawe staff 672 Nov 6 16:01 android-ndk-r20/
drwxr-xr-x 11 mawe staff 352 Nov 6 16:02 android-sdk/
drwxr-xr-x 33 mawe staff 1056 Nov 9 18:10 emsdk-1.39.16/
drwxr-xr-x 8 mawe staff 256 Nov 11 08:37 iPhoneOS14.0.sdk/
drwxr-xr-x 8 mawe staff 256 Nov 11 08:37 iPhoneSimulator14.0.sdk/

And, the paths (+ version numbers) used are set in build.py and waf_dynamo.py.

The step that fails for you currently is this (iirc) in waf_dynamo.py:

conf.check_tool(‘compiler_cc’)
conf.check_tool(‘compiler_cxx’)

So to debug it, you’ll need to printout the env beforehand and see if you spot any path issues:

print conf.env

1 Like

Thank you for the quick reply.

This is what is should look like:

Yes, that is what it looks like for me too:

Checking for program gcc,cc              : ok /Users/marco/Dev/defold/tmp/dynamo_home/ext/SDKs/XcodeDefault12.0.xctoolchain/usr/bin/clang

Looking for the file myself though, I noticed that it doesn’t exist. The whole bin/ folder is missing from the XCode SDK. It looks like a recent commit (https://github.com/defold/defold/commit/5f3595362f4cd29c8424ca63b791a5fc12fb303c#diff-8ae4401b91e0a389e3650a1a00aa09a1bb7bdac281e585cfec0642c2802eb29cL87) removed the bin-folder from the tarball.

Adding the bin/ folder back, Defold now starts to compile :slight_smile:

But alas, it fails after build step 549 with:

dyld: Library not loaded: @rpath/libtapi.dylib
  Referenced from: /Users/marco/Dev/defold/tmp/dynamo_home/ext/SDKs/XcodeDefault12.0.xctoolchain/usr/bin/ld
  Reason: image not found
clang: error: unable to execute command: Abort trap: 6
clang: error: linker command failed due to signal (use -v to see invocation)

The problem here was introduced in the same commit (the for-loop that removes *.dylib files).

I’ll create a merge request later to fix those issues.

Fixing this, it seems to build successfully, but there’s an issue at the last step with codesign. Which makes sense, as I don’t have the keys to do that. Using--skip-codesign, it builds flawlessly, but is it necessary for my iOS game to have a signed engine binary?
Also, how would I now take this engine (located in ~/Dev/defold/tmp/dynamo_home/bin/armv7-darwin) and tell my vanilla defold editor to use it when building and signing my app?

Thanks!

Nope.

It is probably easier to bundle your game using bob from the terminal.

./scripts/build.py --skip-tests --skip-codesign build_bob

Or you can tell the editor to use a local bob.jar:

Ah, yes, that’s my fault for using the same script for two different scenarios.
I’ll fix that right away! Thanks for pointing it out!

As for skipping code signing at this stage, that’s fine since it’s for the unit tests.

You’ll of course need to sign the ipa/app later to upload it onto your phone.
We have a script ios-repack.sh to repack/resign an ipa and copying the defold engine to the app at the same time.
This helps turnaround times a lot.

To rebuild the engine with just a library (list of libraries as seen in ENGINE_LIBS in build-py, you can use the submodule.sh:

./scripts/submodule.sh armv7-darwin dlib

It is probably easier to bundle your game using bob from the terminal.

When doing things like this, I’d really recommend using the the “submodule.sh” and “ios-repack.sh” scripts. It’s the way I do it, and no other workflow will be faster.

Thanks, ios-repack.sh seems to work fine.

I’m not sure how to handle the different platforms though. I usually build the engine for both arm64-darwin and armv7-darwin using the Editor.
In Defold Editor, using Project -> Bundle -> iOS Application, I need to select both 32-bit (armv7) and 64-bit (arm64), otherwise (if I remember this correctly) the apps gets denied by Apple when uploading. Creating the app bundle like this, I only get one .ipa file in a folder named arm7-darwin.

Do I need to set arm64-darwin as platform for ios-repack.sh or armv7-darwin? Or somehow both?

If you build the engine once for each platform (armv7-darwin, arm64-darwin), they’ll end up in $DYNAMO_HOME/bin/_platform_name_.

ios-repack.sh currently only copies one executable from the bin folder in dynamo-home to the app. The script is essentially for debugging purposes, but you can easily modify it to copy and sign both executables.

If if you ever need to rebuild content though, it might be a good idea to build a bob.jar, as @britzl suggested. It will then use all the local executables you’ve built (i.e. both iOS versions).

I’m curious though. Apart from the bug being an annoying popup, does it have other implications?
Just curious because since you reported the bug, I’ve seen the issue in the wild, on my phone with other apps, most notably the Disney+ app. I’d think that since Apple introduced the issue with iOS 14, they’d have a recommended approach to avoid it.

Hmm, I’m not sure if I explained my problem correctly, or if I misunderstand. The .ipa I create using the Defold Editor Menu should contain both armv7 and arm64 build, correct?
Looking into the unpacked .ipa I created using the Editor, I see an executable with the following file descriptor: Mach-O universal binary with 2 architectures: [arm_v7:Mach-O executable arm_v7] [arm64].

Unzipping the repacked .ipa, the executable is just Mach-O 64-bit executable arm64. So I would somehow need to create a universal binary? Sorry, first time I come across something like this.

The issue is, that once the user denies access using the popup, the network functionality (as in: http requests to public domains) stops working completely and I can’t do any subsequent network requests. My game relies on network for leaderboard and signup though, so this is quite essential. The user doesn’t know that of course.

A typical user story could be:

  1. User sees popup and wonders why this game would require local network access (already bad, he might suspect unsolicited tracking)
  2. User denies it local network access
  3. App is now not fully functional (in my case: not usable), user is not sure why

This is literally the first bug report I had from my client on the final release build :wink:

Oh wow, I just noticed that I missed out on explaining the fact that network access is impossible after the user denies the popup in the github issue… that was kind of crucial information :sweat:

The .ipa I create using the Defold Editor Menu should contain both armv7 and arm64 build, correct?

Yes. It will contain both architectures, if you’ve selected both.

Unzipping the repacked .ipa, the executable is just Mach-O 64-bit executable arm64. So I would somehow need to create a universal binary? Sorry, first time I come across something like this.

Correct. Currently, the ios-repack.sh script simply copies the architecture you asked for (it’s the argument to the script).

You can modify the script to copy both architectures (from dynamo_home/bin/), and lipo them together into an universal binary, just like the one in the original .ipa.

The issue is, that once the user denies access using the popup, the network functionality (as in: http requests to public domains) stops working completely and I can’t do any subsequent network requests. My game relies on network for leaderboard and signup though, so this is quite essential. The user doesn’t know that of course.

Gotcha.

I have two native extensions which now won’t work, because they don’t get compiled with the engine.

I guess I need to look into how bob works.

I compiled bob with ./scripts/build.py --skip-tests --skip-codesign build_bob. This finished successfully, but executing it (java -jar /Users/marco/Dev/defold/tmp/dynamo_home/share/java/bob.jar) throws errors:

ERROR: go/src/defold/apkc/apkc.go:1: 'com.dynamo.bob.CompileExceptionError: Expected identifier.'

Exception in thread "main" com.dynamo.bob.CompileExceptionError: Expected identifier.
	at com.dynamo.bob.pipeline.ProtoUtil.merge(ProtoUtil.java:34)
	at com.dynamo.bob.pipeline.GameObjectBuilder.loadPrototype(GameObjectBuilder.java:45)
	at com.dynamo.bob.pipeline.GameObjectBuilder.create(GameObjectBuilder.java:76)
	at com.dynamo.bob.Project.doCreateTask(Project.java:256)
	at com.dynamo.bob.Project.doCreateTask(Project.java:243)
	at com.dynamo.bob.Project.createTasks(Project.java:331)
	at com.dynamo.bob.Project.doBuild(Project.java:779)
	at com.dynamo.bob.Project.build(Project.java:445)
	at com.dynamo.bob.Bob.mainInternal(Bob.java:621)
	at com.dynamo.bob.Bob.main(Bob.java:670)
Caused by: com.google.protobuf.TextFormat$ParseException: 1:1: Expected identifier.
	at com.google.protobuf.TextFormat$Tokenizer.parseException(TextFormat.java:742)
	at com.google.protobuf.TextFormat$Tokenizer.consumeIdentifier(TextFormat.java:554)
	at com.google.protobuf.TextFormat.mergeField(TextFormat.java:895)
	at com.google.protobuf.TextFormat.merge(TextFormat.java:855)
	at com.google.protobuf.TextFormat.merge(TextFormat.java:801)
	at com.dynamo.bob.pipeline.ProtoUtil.merge(ProtoUtil.java:28)
	... 9 more

Any idea? Or is there maybe a way to hardcode my native extensions into the engine, so I can continue using the ios-repack.sh method?

1 Like

I have two native extensions which now won’t work, because they don’t get compiled with the engine.

Correct, this workflow is only for building the vanilla engine, without extensions.

To also use extensions, you have three options:

  1. Copy the source code of the extensions into the source code of Defold. Although this is doable, it’s tedious and requires some build system work on your part.
  2. Run a local extender server. Instructions are here
  3. Make a Pull Request on GitHub. The CI will build the packages, and also the DefoldSDK, which is needed to build on the public extender server. You can then download that bob (or editor) to build your game.

For this scenario, I’d go for number 3.
Not only do I think it’ll be easier for you,
but also since this is a bug fix that would benefit other users as well.

1 Like

Oh, does the CI system automatically build every merge request an provide bob and editor for download? That would be the easiest I guess.

I’d love to contribute code to the repo, but I’m not sure how to properly fix the permission problem.
My workaround is supposed to be a quick fix (avoiding using the system-provided provided dns server and just use 8.8.8.8), not intended to be merged into master. It’s likely, that fixing the actual problem involves fixing c-ares itself.

Yes.

It look like the CI system job that triggers for my pull request (on: pull_request_target) is building the current HEAD of the dev branch (7c741edef4a30dd2ee890834e71251458a8421b6) and not the content of my pull request: https://github.com/defold/defold/pull/5376/checks?check_run_id=1473392035#step:5:2135

For example on PR #5363 it ran another job (on: push) that build and uploaded that the PRs branch: https://github.com/defold/defold/runs/1458044137?check_suite_focus=true#step:5:2134

How do I get CI to build my branch?

Hmm, not sure why that is the case tbh.

Some parts seem to work.

GITHUB_HEAD_REF=‘issue-5287-workaround’

While, at the same time, when we try to build bob, we use the archived engine build (dev, in this case):

git rev-parse --abbrev-ref HEAD
2020-11-30T11:46:19.4430785Z dev
2020-11-30T11:46:19.4436004Z Using branch=dev engine_channel=alpha editor_channel=alpha engine_artifacts=archived

My guess is that there is something we haven’t setup properly for your PR’s to be built using the correct branch. But then again, shouldn’t Github CI solve that automatically… :confused:

Hmm… I initially set the base branch of the PR to master and then later changed it to dev, cause master didn’t work at all. Maybe that’s why Github CI is confused. I’ll close this PR and open a new one.

Nope, it’s still not working. Maybe it’s a repo permission problem?

I’ve now gone ahead and replaced the native extensions I used with Lua implementations, patched the DNS thing (https://github.com/defold/defold/pull/5378/commits/b66460d941a0c31f797afbabae260e237af72450) and built the iOS engines locally. I closed my DNS-related PRs.

I then created a unified binary with lipo -create tmp/dynamo_home/bin/arm64-darwin/dmengine_release tmp/dynamo_home/bin/armv7-darwin/dmengine_release -output tmp/dynamo_home/bin/ub_dm_release and repackaged it with ios-repack.sh.

The DNS workaround works well (no permission popup) and the build uploads to Apple just fine.

I know you said you’ll fix the issue with the iOS SDK packaging yourself, @Mathias_Westerdahl, but here’s a PR anyway: https://github.com/defold/defold/pull/5379

Thanks again for helping with this!

1 Like