Custom Defold build for Android

Fr1ar · March 7, 2023, 12:17pm

Can someone explain to me how to compile a custom Defold build for Android (on Windows)?
I’m going to change some OpenGL settings in Defold engine here: https://github.com/defold/defold/blob/dev/engine/glfw/lib/android/android_util.c

Do I understand correctly that for this I need to build Defold engine/builtins/docs/bob for armv64-android?
But how then will my changes get into the application assembly if it is compiled on the build.defold.com servers?

Do I need to run Extender locally to do this? And how then will it pick up my changes?

Mathias_Westerdahl · March 7, 2023, 12:22pm

If you want to build a fully custom build, and want to use extensions, then you also need to host your own Extender server (repo: https://github.com/defold/extender/)
It is an advanced feature and may take some setting up, but once setup, it should work well.

The question is, what type of setting is it that you want to apply? Is it something that would benefit other users?

Fr1ar · March 7, 2023, 12:48pm

I would like to add the ability to make Defold Activity transparent in Android. This can be useful in some cases (we already do this in iOS, to show a WebView in an underlying Аctivity).
This feature will be disabled by default, but can be enabled in AndroidManifest.xml by adding meta-data property.
What do you think?

britzl · March 7, 2023, 1:53pm

I think we discussed this in an open ticket, right? Or did we only discuss it on Discord? (sorry, my memory is fuzzy).

If it is an optional thing that can be toggled in the AndroidManifest and the change doesn’t negatively impact performance then let’s discuss it and get it into the main Defold repository.

Fr1ar · March 9, 2023, 9:34am

@britzl We didn’t discuss it, I just asked on the forum if it’s possible to make DefoldActivity transparent in Android.
The task is quite difficult, but I would like to try to do it by myself.

@Mathias_Westerdahl I successfully built the Defold engine and editor for Windows from the source code (version 1.4.1), built and launched the Extender locally, replaced Build Server in the Project Settings with a local one (http://localhost:9000/), and when building Android bundle, I see that the Extender is downloading the SDK for compilation. How can I make it work with my local Defold SDK copy?

2023-03-09 09:15:21.859 - INFO 7 --- [job14054640574443082713] c.d.extender.services.DefoldSdkService   : The given sdk does not exist: http://d.defold.com/archive/stable/cb075f1e8ad6c805df09b75718a28a08de607e92/engine/defoldsdk.zip 404 NOT _FOUND
2023-03-09 09:15:22.301 - INFO 7 --- [job14054640574443082713] c.d.extender.services.DefoldSdkService   : Downloading Defold SDK from http://d.defold.com/archive/cb075f1e8ad6c805df09b75718a28a08de607e92/engine/defoldsdk.zip ...

Mathias_Westerdahl · March 9, 2023, 10:18am

Good to hear that you got it to work!

To use a local sdk, you can set the DYNAMO_HOME folder for the server.

github.com

defold/extender/blob/dev/README_DEBUGGING.md#environment-variables

# Run a single gradle test

Easiest way to do this via command line is to write something like so:

    ./gradlew :server:test --info --tests ExtenderUtilTest

You can also run tests from IntelliJ

# Debugging with defoldsdk.zip

There is a debug script that you can run to download a specific SDK version, or the latest one.

    $ ./server/scripts/debug_defoldsdk.py [<sha1>]

This downloads the latest sdk to the folder `defoldsdk/<sha1>/defoldsdk`. It also sets the environment variable `DYNAMO_HOME` and then starts the extender server.

## Building .proto files

If you are build a native extension that has .proto files, and you are using a non-Linux DYNAMO_HOME,
then chances are that you don't have the `libdlib_shared.so` file installed.

This file has been truncated. show original

Fr1ar · March 13, 2023, 1:13pm

I’m stuck with compiling Defold on Windows for Android.
So I successfully downloaded and installed the Android SDK/NDK and am trying to compile Defold:

./scripts/build.py shell
./scripts/build.py build_engine --platform=arm64-android --skip-tests -- --skip-build-tests

The JAVA_HOME environment variable is set to JDK11 path.
If JDK8 path goes first in the PATH environment variable, I get the following error:

BUILD FAILED
C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build.xml:165: Compile failed; see the compiler error output for details.

Total time: 19 seconds
Buildfile: C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build.xml

clean:
   [delete] Deleting directory C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build
   [delete] Deleting directory C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\generated
   [delete] Deleting directory C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\dist
   [delete] Deleting directory C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\tmp
   [delete] Deleting: C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\lib\luajit-share.zip

git.revision:

compile-bob-light:
    [mkdir] Created dir: C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build
    [mkdir] Created dir: C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\generated
     [exec] gamesys/atlas_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gamesys/buffer_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gamesys/buffer_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] gamesys/texture_set_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gamesys/camera_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] gamesys/camera_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gamesys/model_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gameobject/lua_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gamesys/mesh_ddf.proto:6:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] engine/engine_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] engine/engine_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] gameobject/properties_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] gameobject/properties_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] gamesys/sound_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] script/sys_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] script/sys_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] script/ddf_script.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] resource/resource_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] resource/resource_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] resource/liveupdate_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] resource/liveupdate_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] render/font_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [exec] input/input_ddf.proto:4:1: warning: Import ddf/ddf_extensions.proto is unused.
     [exec] input/input_ddf.proto:5:1: warning: Import ddf/ddf_math.proto is unused.
     [copy] Copying 2 files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\lib
     [copy] Copying 5 files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\libexec\x86_64-macos
     [copy] Copying 5 files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\libexec\x86_64-linux
     [copy] Copying 5 files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\libexec\x86_64-win32
    [javac] Compiling 32 source files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Compiling 135 source files to C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\build
    [javac] C:\Sources\Arenum\defold_141\com.dynamo.cr\com.dynamo.cr.bob\src\com\dynamo\bob\pipeline\OggBuilder.java:36: error: cannot find symbol
    [javac]         List<String> deps = List.of("libogg", "liboggz");
    [javac]                                 ^
    [javac]   symbol:   method of(String,String)
    [javac]   location: interface List
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error

And when JDK11 goes first in the PATH environment variable, I have another error:

-Djava.ext.dirs=c:\Sources\Arenum\defold_141\tmp\android\android-sdk\build-tools\33.0.1\lib is not supported.  Use -classpath instead.

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Error running d8
feature 'web' does not exist - bind at least one method to it?
Waf: Leaving directory `C:\Sources\Arenum\defold_141\engine\engine\build'
Build failed
 -> task in 'dmengine' failed with exit status 1 (run with -v to display more information)

Traceback (most recent call last):
  File "C:\Sources\Arenum\defold_141\scripts\build.py", line 2261, in <module>
    f()
  File "C:\Sources\Arenum\defold_141\scripts\build.py", line 1111, in build_engine
    self._build_engine_lib(args, lib, target_platform)
  File "C:\Sources\Arenum\defold_141\scripts\build.py", line 1050, in _build_engine_lib
    run.env_command(self._form_env(), args + plf_args + self.waf_options + skip_build_tests, cwd = cwd)
  File "C:\Sources\Arenum\defold_141\build_tools\run.py", line 98, in env_command
    return _exec_command(args, shell = False, stdout = None, env = env, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Sources\Arenum\defold_141\build_tools\run.py", line 75, in _exec_command
    raise e
run.ExecException: 1

As far as I understand, I need to use the JDK11 compiler and JDK8 libraries, but how do I achieve this?

Mathias_Westerdahl · March 13, 2023, 1:28pm

Not sure what you mean?
We currently use Java 11.

Fr1ar · March 14, 2023, 4:09am

Because the compilation process fails here:

github.com

defold/defold/blob/dev/build_tools/waf_dynamo.py#L1029

    
      
              dex_input = ['%s/share/java/classes.jar' % dynamo_home]
          
          
    ret = bld.exec_command('%s -jar %s -include %s -libraryjars %s -injars %s -outjar %s' % (task.env['JAVA'][0], proguardjar, proguardtxt, android_jar, ':'.join(dx_jars), dex_input[0]))
              if ret != 0:
                  error('Error running proguard')
                  return 1
          else:
              dex_input = dx_jars
          
          
if dex_input:
              ret = bld.exec_command('%s --output %s %s' % (d8, dex_dir, ' '.join(dex_input)))
              if ret != 0:
                  error('Error running d8')
                  return 1
          
          
# strip the executable
          path = task.native_lib.abspath()
          ret = _strip_executable(bld, task.env.PLATFORM, build_util.get_target_architecture(), path)
          if ret != 0:
              error('Error stripping file %s' % path)
              return 1

According to the documentation:

d8 is a command-line tool that Android Studio and the Android Gradle plugin use to compile your project’s Java bytecode into DEX bytecode that runs on Android devices. d8 lets you use Java 8 language features in your app’s code.

So when waf_dynamo.py runs this command:

C:\Sources\Arenum\defold_141\tmp\android\android-sdk\build-tools\33.0.1\d8.bat --output 
C:\Sources\Arenum\defold_141\engine\engine\build\src\dmengine.android 
C:\Sources\Arenum\defold_141\tmp\dynamo_home\ext\share\java\android-support-multidex.jar 
C:\Sources\Arenum\defold_141\tmp\dynamo_home\ext\share\java\androidx-multidex.jar 
C:\Sources\Arenum\defold_141\tmp\dynamo_home\share\java\glfw_android.jar 
C:\Sources\Arenum\defold_141\tmp\dynamo_home\share\java\gamesys_android.jar 
C:\Sources\Arenum\defold_141\tmp\dynamo_home\share\java\sound_android.jar

It fails with an error:

-Djava.ext.dirs=C:\Sources\Arenum\defold_141\tmp\android\android-sdk\build-tools\33.0.1\lib is not supported.  Use -classpath instead.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I even tried to modify d8 script itself by changing this:

call "%java_exe%" %javaOpts% -Djava.ext.dirs="%frameworkdir%" -cp "%jarpath%" com.android.tools.r8.D8 %params%

into that:

call "%java_exe%" %javaOpts% -classpath="%frameworkdir%" -cp "%jarpath%" com.android.tools.r8.D8 %params%

(as it was recommended in the error message) but got almost the same error:

Unrecognized option: -classpath=C:\Sources\Arenum\defold_141\tmp\android\android-sdk\build-tools\33.0.1\lib

Currently I have Java 11 only installed on my machine.

Fr1ar · March 14, 2023, 10:31am

It seems that Defold cannot be compiled in Windows with the current build scripts.
No problem, I successfully compiled it in linux.
Now I’m trying to attach it to the Extender.

Fr1ar · March 15, 2023, 11:06am

Good news, I managed to make DefoldActivity transparent!

Now I’m going to make this functionality work only if AndroidManifest.xml has meta-data “alpha.transparency” with value “true”

Unfortunately AndroidManifest.xml is inaccessible from C++ code directly, so I’m going to use JNI to access it.

Mathias_Westerdahl · March 15, 2023, 11:14am

Where do you need to set it exactly in C++? For the window?
Or OpenGL?

There is code in engine/glfw/lib/android that does other JNI stuff. E.g. android_window.c and android_init.c

Fr1ar · March 15, 2023, 11:44am

I need to set it here: engine/glfw/lib/android/android_util.c (choose_egl_config)

It would probably be a good idea to create android_manifest.c and android_manifest.h files in engine/glfw/lib/android/,
which would read meta-data from AndroidManifest.xml using JNI like in android_window.c and android_init.c.

Fr1ar · March 17, 2023, 11:00am

It is done! Do I need to create an Issue or can I just submit Pull Request to defold:dev?

Mathias_Westerdahl · March 17, 2023, 11:17am

You can create a pull request, and we can discuss the feature there.