How to reduce time spent "Fetching engine" in large project

Questions
#1

ANSWER:
For those coming upon this, the shortened answer is so:

‘Fetching engine’ time relates to the deviation of the engine from last build, so to reduce time spent, remove unnecessary extensions and don’t modify engine every build.

Hopefully, someday, Defold will add extension caching which will optimize this problem out of existence.

ORIGINAL POST:

Hello,

I am wondering what I can do to optimize the time spent “Fetching engine” (I presume to mean building project), as it is currently quite lengthy for my project. This is not about the performance of the exported game, but the build time.

Thanks,
grify

#2

This is part of the cloud build process for games which use native extensions or use an app manifest to remove parts of the engine that are not used. In both of these cases we need to provide you with a custom engine with the additional native code from the extensions included and with parts from the app manifest removed.

The custom engine is cached in .internal/cache and the contents of the cache will be used for subsequent builds. This holds true as long as you do not modify any of your native code or the app manifest and as long as you do not remove the contents of .internal/cache in between builds.

You should be able to verify this by making two builds immediately after each other. The first should take longer while the engine builds and the next one should be much faster.

2 Likes
#3

The build time is proportional to the work it does.
I.e. if you have lots of extensions, and if these extensions have lots of sources files that needs to be built.

For each new release (the defoldsdk changes), your local cache will be invalidated, and you’ll need to rebuild the engine.

You ofc have the option to stay on an older release ,but we don’t really recommend it (we don’t support defoldsdk’s older than 6 months).

That said, we will work on implementing caching of individual extensions as well during Q3/Q4. It should speed it up for many users.

6 Likes
#4

Yes, that makes sense. Thank you for explaining.

When making changes to the game.project the Fetching engine stage is called but when simply modifying collections and atlases it isn’t. I am quite certain my git setup ignores .internal so this shouldn’t be a problem.

#5

I have cleaned up my extensions and stopped modifying game.project every build and it builds faster.

Is there a way to know once my release has become outdated beyond 6 months? The in-app update is broken (When I click “Download Update”, it tries to download, and goes to 0%, but then gives up and replaces the download progress bar with the “Update available” button once more) so this had led to me simply not updating aside from manually occasionally when new features I really want are released.

Thank you! That would be awesome and so much easier for extension-heavy and highly customized builds!

#6

Is there a way to know once my release has become outdated beyond 6 months?

We release a new version every month, so if your editor is 6 versions old, it’s likely 6 months old.

We recommend you download version 1.3.3, which has a fix for the most recent SSL handshake issue that prevented the update downloader to work properly.

2 Likes
#7

Thank you very much, I’ve updated to 1.3.3, and I’m looking forward to the further progress the Defold team including yourself is making. I won’t miss a changelog from now on.

#8

Actually this is odd, when I first read this I tried it and it worked but just now I fixed a bug in a GUI script file and it led to the engine requiring to be fetched again despite no difference at all aside from 5 characters in one GUI script. And now every time I build (Even when building with no changes at all to any files, simply rebuilding) it fetches the engine once more. Probably a fluke or user error but strange nonetheless. Considering this is editor/user specific a repro probably wouldn’t be helpful but engine logs instead?

#9

Hmm, sounds strange that editing a GUI file would result in the engine having to be rebuilt. I can’t explain that. Can you try and create a minimal repro project where this happens?

#10

I’d test disconnecting the internet after the first build, then see if the second build succeeds. It should succeed due to the build result being cached the first time.

1 Like
#11

I tried this multiple times in many different conditions and here’s what I found:
Modifying any file causes the engine to rebuild if saved manually (CMD+S, a muscle-memory habit of mine before each build).
Is this intentional as a feature? I’m updating and restarting my client (1.3.3).

As of restart, it still happens in my main project but not test project. I’ve filmed a screen video of it occurring with keypresses which I hope might be useful or helpful.

Making a repro will take quite a bit of time considering the size of the project but I’ll work on it first thing tomorrow as it’s already almost 4am.

Thanks all!

#12

Can you give us the full list of extensions that you use?

#13
#14

I removed all my extensions 3 days ago and now the bug has somehow reappeared so I apologize for leading you and @britzl astray but this error is NOT caused by an extension and is in fact an engine bug. I am not sure what additional information I can provide that might help, since it’s an engine bug, how do I access verbose engine logs if there are any? Or is there another possibility?

#15

You mean an editor bug?

#16

yes, sorry. I believe this bug to be in the editor. Is there a way to access editor logs?

#17

Yes: Editor overview

#18

Is rebuilding the engine logged in the editor logs? I tested a bunch of stuff but the log never seemed to mention it and can’t find any reference to it

#19

There’s a log in the build folder for the work done on the build server, but I don’t think it contains any timestamps.

1 Like
#20

I would suggest to setup several proxy servers all around the world just like a mini verson of Cloudflare.
Users in some place would not to access any of the libs or the cloud build server.

1 Like