Help building Defold?

Epitaph64 · August 28, 2020, 4:47am

I am trying to build Defold from source. However, when I run the package/SDK install script with:

./scripts/build.py install_ext --platform=win32 --package-path=./packages/

It installs a few python packages but eventually crashes with the error:

  File "S:\Python27\lib\urllib2.py", line 283, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: ../packages/WindowsKits-10.0.18362.0.tar.gz

Anyone seen this before?

Pkeod · August 28, 2020, 1:24am

There should be a process to setup those WindowsKits, did you do that? Are they in that folder?

You used to have to host the WindowsKits on a local file server for it to work. It looks like the script is confused though. What OS are you on?

Epitaph64 · August 28, 2020, 1:26am

Running Windows 10. I installed the Windows 10 SDK manually but that didn’t help. I don’t see anything in the docs about setting up the Windows Kits specifically. Do you know where that process is shown?

Pkeod · August 28, 2020, 1:30am

I don’t remember at the moment as it was a while ago when I last tried to build the engine but @Mathias_Westerdahl will for sure know.

While waiting read the docs again and see if you accidentally skipped any step. Is the file it’s looking for not on your drive?

Epitaph64 · August 28, 2020, 1:31am

Yeah it’s not on my drive. It looks like from the error call stack (prior to what I posted) that it’s supposed to be finding the WindowsKits-10.0.18362.0.tar.gz on some HTTP server but instead it’s looking for it in …/packages/ instead.

Pkeod · August 28, 2020, 1:37am

There should be a step previously where you generate these files I believe.

britzl · August 28, 2020, 4:46am

First make sure you have all of the tools needed: https://github.com/defold/defold/blob/dev/README_SETUP.md. This is done once.

Next you follow the steps for building the engine: https://github.com/defold/defold/blob/dev/README_BUILD.md#standard-workflow

The step you didn’t complete is probably too package external SDKs for the platform you are building (Windows). For legal reasons we’re not allowed to redistribute these. You need to follow the instructions to package these yourself and make those available to our build scripts. We provide a script to download and package the required SDKs.

github.com

defold/defold/blob/dev/README_BUILD.md#installing-sdks

# Build Engine

Defold uses the Python based build system [Waf](https://waf.io/). Most of the interaction is done through the `build.py` script but it is also possible to use Waf directly.

## Standard workflow

The standard workflow when working on the engine is the following:

1. Setup platform specific environment
2. Install platform specific dependencies
3. (Re)Build the engine
4. Develop a feature
5. Test
6. Repeat steps 3-5 until done. Start from 1 when switching platform.

### Platforms

*In the instructions below, the `--platform` argument to `build.py` will default to the host platform if not specified.*

The following platforms are supported:

This file has been truncated. show original

Epitaph64 · August 28, 2020, 6:08am

Thanks, I have it now building and will report back with the results. I had to change the build script to use a slightly different version of VS 2019 community which is installed on a different drive than the build script used in order to build it since I’m running a newer version. Hopefully the minor difference in version won’t cause any issues.

Build completes fine, however I get one test failure:

Ran 48 tests, with 176 assertions in 7.98758 s
47 tests passed, 0 skipped and 1 tests FAILED
Wrote test config file: unittest_mdfgag.cfg
test failed test_socket
Traceback (most recent call last):
  File "./scripts/build.py", line 2142, in <module>
    f()
  File "./scripts/build.py", line 1037, in build_engine
    self._build_engine_lib(args, lib, target_platform)
  File "./scripts/build.py", line 988, in _build_engine_lib
    run.env_command(self._form_env(), args + plf_args + self.waf_options + skip_build_tests, cwd = cwd)
  File "S:\defold\defold\build_tools\run.py", line 63, in env_command
    return _exec_command(args, shell = False, stdout = None, env = env, **kwargs)
  File "S:\defold\defold\build_tools\run.py", line 41, in _exec_command
    raise ExecException(process.returncode, output)
run.ExecException

EDIT: I pressed forward with trying to get the editor working despite the test issue. I followed the steps in the editor/README.md however when I run lein init I see:

Initializing editor with version 'null', resolved to 'null'
Installing ../com.dynamo.cr/com.dynamo.cr.bob/lib/openmali.jar
Installing S:\defold\defold\editor\share\java\bob.jar
org.eclipse.aether.installation.InstallationException: Failed to install artifact com.defold.lib:bob:jar:1.0: S:\defold\defold\editor\share\java\bob.jar (The system cannot find the path specified)
 at org.eclipse.aether.internal.impl.DefaultInstaller.install (DefaultInstaller.java:309)
    org.eclipse.aether.internal.impl.DefaultInstaller.install (DefaultInstaller.java:245)
    org.eclipse.aether.internal.impl.DefaultInstaller.install (DefaultInstaller.java:188)
    org.eclipse.aether.internal.impl.DefaultInstaller.install (DefaultInstaller.java:145)
    org.eclipse.aether.internal.impl.DefaultRepositorySystem.install  (DefaultRepositorySystem.java:374)
    ...

I searched for the missing file (bob.jar) which I built successfully. However it shows up in the “defold\tmp\dynamo_home\share\java” directory not in the editor directory.

Thanks for any help anyone can provide

Mathias_Westerdahl · August 28, 2020, 6:19am

What was the actual error from the test_socket?
I’m afraid what you posted is a python callstack (which we obv should fix too).

The bob.jar should be in tmp/dynamo_home/share/java/bob.jar, so that’s correct.
But, if it can’t find it, I’m wondering if you have the DYNAMO_HOME environment variable set?
If not, you set it by running:

$ ./scripts/build.py shell

Epitaph64 · August 28, 2020, 6:28am

Ah that’s probably the problem. When I run ./scripts/build.py shell --platform=win32 (and without the --platform specified) it displays:

$ ./scripts/build.py shell --platform=win32
[exec] git remote get-url origin
https://github.com/defold/defold.git

Looking at the build.py script it should say something like “Setting up shell with DYNAMO_HOME, …” but that never appears. I tried this in git bash. Do I need to run this inside msys? I’m using msys2 since Sourceforge servers are down right now.

Mathias_Westerdahl · August 28, 2020, 6:34am

Yes, it should output more. (btw, no platform needed for that one):

~/work/defold $ ./scripts/build.py shell                                                            [issue-5128-dmsocket-dmsdk] 08:30:24
[exec] git remote get-url origin
Running 'shell'
Setting up shell with DYNAMO_HOME, PATH, ANDROID_HOME and LD_LIBRARY_PATH/DYLD_LIBRARY_PATH (where applicable) set

EDIT:
I don’t know if msys2 works here though (I don’t recall).
We’re creating a new shell, which it might fail?

Epitaph64 · August 29, 2020, 3:42am

Yeah I tried both Git Bash and MSYS2 but both have the same issue (not running the shell, simply running the git remote get-url origin command.) Not quite sure how to proceed from here.

Mathias_Westerdahl · August 29, 2020, 7:25am

The command isn’t very complex, it sets a few environment variables, then creates a shell.

You can ofc set those variables manually, or modify the command to not start a new shell. It’s in build.py

Epitaph64 · September 30, 2020, 3:51pm

I’ve finally got the engine building and built the editor. However, when I run “lein init” I see

$ lein init
Initializing editor with version 'null', resolved to 'null'
Installing ../com.dynamo.cr/com.dynamo.cr.bob/lib/openmali.jar
Installing s:\defold_github\tmp\dynamo_home\share\java\bob.jar
java.nio.file.NoSuchFileException: s:\defold_github\tmp\dynamo_home\share\ref-doc.zip
 at sun.nio.fs.WindowsException.translateToIOException (WindowsException.java:85)

Mathias_Westerdahl · September 30, 2020, 4:02pm

It looks like the documentation wasn’t built.
You might need to run that command too:

$ ./scripts/build.py build_docs

Epitaph64 · September 30, 2020, 4:03pm

Thanks, I tried to run this but I see: ImportError: No module named script_doc_ddf_pb2

Mathias_Westerdahl · September 30, 2020, 4:04pm

And you’ve built the engine successfully?

Epitaph64 · September 30, 2020, 4:06pm

I think so, though not sure how to use it without the editor however I get this message:

Socket took 7.95754 s
Socket.GetHostByName_IPv4_Unavailable failed
Ran 48 tests, with 176 assertions in 7.95754 s
47 tests passed, 0 skipped and 1 tests FAILED
test failed test_socket

Mathias_Westerdahl · September 30, 2020, 4:11pm

The editor requires the other tools to work: engine, bob, docs etc.
So one must build those things in order.

If you build with running tests, the build will stop if any errors occur.
There might be some issue with that test on Windows, so you can try to build without building+running the tests:

$ ./scripts/build.py build_engine --platform=x86_64-win32 --skip-codesign --skip-docs --skip-bob-light --skip-tests  -- --skip-build-tests

Epitaph64 · September 30, 2020, 4:25pm

Thanks, it now completes without error. However, I’m unsure how to launch it. I tried opening the launcher at \tmp\dynamo_home\bin\x86_64-win32 both in explorer and via command prompt but nothing happens and it returns without any message in command prompt. Is there some way I’m meant to start the editor?