Build timeout


#62

only the following log is useful,other logs is about the defold update:
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
at editor.engine.native_extensions$build_engine_archive.invokeStatic(native_extensions.clj:196)
at editor.engine.native_extensions$find_or_build_engine_archive.invokeStatic(native_extensions.clj:208)
at editor.engine.native_extensions$get_engine.invokeStatic(native_extensions.clj:266)
at editor.engine$get_engine.invokeStatic(engine.clj:171)
at editor.engine$launch_BANG_.invokeStatic(engine.clj:178)
at editor.app_view$launch_engine_BANG_.invokeStatic(app_view.clj:413)
at editor.app_view$launch_built_project_BANG_.invokeStatic(app_view.clj:460)
at editor.app_view$fn__48436$fnk48428_positional__48437$fn__48438.invoke(app_view.clj:571)
at editor.app_view$async_build_BANG_$fn__48350$fn__48353.invoke(app_view.clj:523)
at clojure.lang.AFn.run(AFn.java:22)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$148(WinApplication.java:191)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
… 21 common frames omitted


#63

BTW,I think the caching task is the best solution


#64

I’ve updated the server with a 10min timeout limit now. Please let me know if that helps the success rate.

I will continue with the other fixes as well. I’ll keep you updated.


#65

In fact, in our project the Win32 libraries is much smaller than macOS libraries. Here is the detail info about the google protobuf extension which introduced the build problem:
x86-win32: 12.4 MB
x86_64-win32: 21.2 MB
x86_64-osx: 51.5 MB


#66

It doesn’t help, still timeout all the time.


#67

The Editor 2 has now also been updated to have a 10min timeout.

It should have the sha1: bd8e9b221dc47489fd0ee20a3b2477ed27063c24 in the About box


#68

I recommend adding a timeout option to set to editor2, default to 10 min.
image


#69

Well, first I would really like to know what the actual problem is, not just randomly change things.
For instance, you mention your MacOSX build works. Then, the Win32 build should only be longer by the amount that it builds, which should be ~1-2 minutes. We’ve now upped the timeout with both 5 and 9 minutes!

Could this have anything to do with your VPN service? E.g. are there any settings you can configure there? E.g. timeout values?

What upload/download speed do you get?


#70

HI again!

Now, with this release (1.2.133), the build server supports building with Clang 6.
This allows for much faster (actual) build times (a few seconds).

You can enable it by adding this to your game.appmanifest:

platforms:
    x86-win32:
        context:
            use-clang: true

    x86_64-win32:
        context:
            use-clang: true

If you don’t use an app manifest already, you can add this to your game.project:

[native_extension]
app_manifest = /game.appmanifest

Please note that it doesn’t improve the upload/speeds. we are working on adding a file cache to the server to improve the turnaround times.


A building error in release version 1.2.133 with Clang 6
#71

Hi Mathias,

One of the new team member was stopped by this issue for several days. I can not help, the only thing to do is trying some more tests and showing you the feedback. :joy: Hopes you can get the point.

The uploading speed is about 100~150KBs during these tests.

bundle IOS application within 14 minutes
prepare local data without uploading 18:01 ~ 18:04
uploading data to server 18:04 ~18:13
uploading finished and application is ready: 18:13 ~ 18:15

bundle andriod application within 15 minutes
prepare local data without uploading 19:47 ~ 19:52
uploading data to server 19:52 ~20:20
uploading finished and application is ready: 20:20 ~ 20:22

bundle windows application without an ending
prepare local data without uploading 20:03 ~ 20:07
uploading data to server 20:08 ~ 20:16
uploading finished and application is ready: 20:16 ~ nerver(I had waited for more than 20 minutes here and nothing happens even without a timeout indication like the building process)

If the bundle process is similar to the build process, then the issue is not releated to network status. Would you please take a look at the logs on your build server? At least we shall know what happens during these tests.If the single building step for windows target is successful and longer than other target, you can also show us the detailed time cost.

Maybe I can remove some native extentions to findout the critical point where the project could be built for windows tomorrow.


#72

Have you tried building with clang instead? As mentioned here: Defold 1.2.133 has been released

You mean that this is the time before you see anything being sent to build.defold.com?

This is both upload of extension code, the build process on the server and then download of custom engine?

Is this from when you don’t see any more network traffic and the editor does the bundling of the app?


#73

We did tried the clang6 and got some building errors like A building error in release version 1.2.133 with Clang 6.

From these bundle tests, it seems to show three steps.
In the first step, the editor is preparing for some resource data. During this period, there is no obvious network traffic.
image
The building report also showes more details:

In the second step, the upload is continued for about 8 miutes. According to the uploading rate, about 860100KB = 48MB data(libs or source codes) is sent to the server.

In the third step, the building and linking is processed on the server without network traffic. Finnally, the result program(about 8MB) is sent back from the server at speed 500KB/s (due to the company network download limitation configuration).

When did the same test for macos application on MAC OS, the uploading speed is about 2~3 times faster than the test on windows. Are there any differences in the uploading when bundle the MAC application from different editors?


#74

Not sure why that would differ. @Erik_Angelin? @Mathias_Westerdahl?

That is quite a lot, especially at the upload speeds you mention. @Mathias_Westerdahl, ideas on how to reduce amount of upload? Upload only diffs and cache on the server?


#75

There is no difference in how we perform the upload on mac vs windows. Have you tried doing a general internet upload/download speed test on mac and windows? For instance, if you’re on wifi it could be that the windows machine chooses some other encryption method or doesn’t support 5g or whatever. Not saying that’s the case for you.


#76

Hi Eric,

Here is the general network test result from the speedtest website for our working environment. The uploading rate is the same on MAC(wifi connection) and Windows(ethernet connection).

Is there any hint that why the windows editor only can reach a half(about 100KB/s)?


#77

It is very likely outside our control. I’m assuming that the upload code is the same between the two platforms (the editor is after all cross platform using a single code base), although the underlying implementation (which we don’t control) could be different.


#78

Hi Britzl,

I had made another bundle test on windows in the early morning before most guys coming to the office. It does work better then at normal working time.

The highest uploading rate now almost get the same result as the test made in the browser yesterday.
There are also some wireshark capture records showing the same result. When upload raw source code files(small size), the performance is dropped duo to the packet size.

Raw source code files


33 files with size 122432 bytes with 33 TLS communication rounds within 1 seconds.

A single lib


1 file with size 543411 bytes with 33 TLS communication rounds within 2 seconds.

The pure building and linking on the server only cost about 40+ seconds, this shall not be the cause for the timeout. Definitely we still needs to add more extentions in the project, and then we will see when this timeout issue comes again.


#79

Thank you for getting back to us with additional details! We will start looking into caching of uploads so that you don’t have to upload everything for each rebuild. @Mathias_Westerdahl and @nasthu will start design and implementation of this very soon.


#80

Hi all,

Hopes to see this wonderful feature soon.

Finally the IT department helps to release to limitaion on the uploading rate. Now I get a pretty good uploading test result.

When bundle the application, the uploading rate does only have little improvement.
There is an explicit stop in the uploading that in each 4 packet. Looks like the sending continues for 50ms with 4 packet(16K, TLS max record size) and wait for 180ms. The maxmum uploading speed is 278KB/s.

Would you please take a look at the socket sending buffer size(editor) and socket receiving buffer size(server) ? Maybe chage the default value from 16K to 64K can make sense for large uploading data, :smiley:


#81

Make an issue for that on the github?