Building extender (Native extension build server)

Source Code
1

Look like it is going to be a journey.

First problem that I’m facing:
Did I miss something?

git clone --recurse-submodules https://github.com/defold/extender

fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:defold/aws-build-tools.git' into submodule path '...' failed
2 Likes
2

Tbh, I don’t think you’ll need those tools.
They’re used by us when we release a new live server.

I’ll make a note on adding a comment about that in the readme.

2 Likes
3

So we don’t need submodules?

If you have already cloned the repo, you can init and fetch the submodule like this:
git submodule init
git submodule update

1 Like
4

Not for building and running the Docker container no.

2 Likes
5

thank you. Then I can start to build sdks which I’m highly afraid of :slight_smile:

One lazy person question; should I build all target platforms? I’m planning to use it only on my local envirolment so just xcode sdks enough for me(if possible). But I guess I should modify the Dockerfile for that

1 Like
6

I think you’re right in starting small.
And using the native host (in your case x86_64-darwin) is a good first step.

See the files here on how to package some sdk’s.

Currently, our docker container relies on an url to download these packages, but that should be easy enough for you to modify.

Either mount in a folder with the packages, or a folder which is the unpacked package. etc.

2 Likes
7

SOLVED: My bad.I accidentally removed the ENV GRADLE_USER_HOME and other gradle stuff

When I start the server having this error:
Full log: run-local.txt (13.8 KB)

Seems like it is a permission error, releated to GradleService.java /unpacked
Any idea/suggestion?

Caused by: java.nio.file.AccessDeniedException: /unpacked

P.S.: I build it with osx/ios sdks only.

1 Like
8

Job done :smiley:
Now it is time for windows toolchain

15
151146×492 108 KB

2 Likes
9

I had to use a trick for building the engine.

I have XcodeDefault11.3.1 toolchain. But when I set it as below, it fail to build after downloding the Defold SDK with error (I don’t remember exactly) XcodeDefault11.0.xctoolchain … can’t found. Should I build the Defold SDK with the exact same version? (In this case by using 11.3.1)

wget -q -O - ${DM_PACKAGES_URL}/XcodeDefault11.3.1.xctoolchain.tar.gz | tar xz -C ${PLATFORMSDK_DIR} && \
  mv ${PLATFORMSDK_DIR}/XcodeDefault.xctoolchain ${PLATFORMSDK_DIR}/XcodeDefault11.3.1.xctoolchain

RUN \
  ln -s /usr/local/bin/llvm-ar /usr/local/bin/ar && \
  ln -s ${PLATFORMSDK_DIR}/XcodeDefault11.3.1.xctoolchain/usr/lib/arc /usr/local/lib/arc

I trick it like this:

wget -q -O - ${DM_PACKAGES_URL}/XcodeDefault11.3.1.xctoolchain.tar.gz | tar xz -C ${PLATFORMSDK_DIR} && \
  mv ${PLATFORMSDK_DIR}/XcodeDefault.xctoolchain ${PLATFORMSDK_DIR}/XcodeDefault11.0.xctoolchain

RUN \
  ln -s /usr/local/bin/llvm-ar /usr/local/bin/ar && \
  ln -s ${PLATFORMSDK_DIR}/XcodeDefault11.0.xctoolchain/usr/lib/arc /usr/local/lib/arc
1 Like
10

Seems like server is not caching. Should I setup something related to cache?
It is quite a small project maybe this is normal? Is it caching every c++ file like ccache did?

2020-05-21 08:56:50.900 - INFO 6 --- [job15017334111745153528] c.d.extender.services.DataCacheService   : Cached 0 bytes in 0 files
11

On macOS, you can use xcrun to get the locations of the currently installed SDKs without the need to hardcode them and assume the version of Xcode installed.

That mv is a terrible idea.

EDIT: Actually I think all of those commands are a terrible idea and I would advise other people not to do it.

12

@trojanfoe I don’t understand your suggestions. It seemed irrelevant to me.
This is for building Docker container:

13

Maybe all that moving around of stuff works in a Docker session as the session can be destroyed and recreated at will.

If you run those kinds of commands in your native environment then I think there is a good chance you could corrupt it.

The better solution is to find the locations of the currently installed SDKs and use them, which you can do using xcrun.

14

Thank you but this is irrelevant. If you are interested in you can find more here and here

2 Likes
15

Caching is about the upload size.
We have our own internal server to cache large uploaded files, to help mitigate slower connections.

For local builds I don’t think you actually need it?

There are several versions of the “application*.yml” and you can see how such a server url is specified, if you do wish to use such a server (which you’ll have to provide yourself). I don’t recall the api for those calls though, so you’ll have to read the extender source to figure that out.

16

Not really. I was just curious.

Thank you for pointing out the yaml files. I’ll definitely check them out.

My intend is to speed up the NE development process by using Defold API’s on my local environment. It was a painful proccess before for me.

Thanks again.

17

To speed up further, set the DYNAMO_HOME environment flag before launching the server. That way it’ll pick up the local sdk (from your dyanmo_homefolder). It should be the fastest turnaround times for you when rebuilding the engine and then your game.

3 Likes
18

@selimanac thank you so much for giving this a try. Were you finally able to get it working for you?

I would love to get your input so that we can improve the documentation and maybe also scripts, error messages and other things so that the next person trying this will have a smoother ride.

1 Like
19

Yeess and it runs perfect :heart_eyes:

I’m looking at the engine and the extender server code… I’m not the master of anything but it is like looking a piece of art :smiley: Easy to read and understand which is actually hard thing to design.
Thank you all for this opportunity. I’ll let you know if I hit the wall somewhere.

3 Likes