Errors building extension (SOLVED)

uncle_Night · May 11, 2021, 5:06pm

Hi,

Hesitant to call it a bug, so posting to Questions.

My project uses websocket extension for quite a while, and yesterday game builds started throwing errors like these (beware, log piece coming!):

2021-05-11 19:51:04.111 WARN  default    editor.app-view - {:line 819}
clojure.lang.ExceptionInfo: Failed to build engine, status 422: clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/emscripten_callbacks.cpp -c -o build/emscripten_callbacks.cpp_0.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/emscripten_callbacks.cpp:1:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/handshake.cpp -c -o build/handshake.cpp_1.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/handshake.cpp:1:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/pcg.cpp -c -o build/pcg.cpp_2.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/pcg.cpp:1:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/script_util.cpp -c -o build/script_util.cpp_3.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/socket.cpp -c -o build/socket.cpp_4.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/socket.cpp:3:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/websocket.cpp -c -o build/websocket.cpp_5.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/websocket.cpp:7:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay_callbacks.cpp -c -o build/wslay_callbacks.cpp_6.o
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]
In file included from upload/websocket/src/wslay_callbacks.cpp:1:
upload/websocket/src/websocket.h:24:10: fatal error: 'dmsdk/dlib/dns.h' file not found
#include <dmsdk/dlib/dns.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay/wslay_event.c -c -o build/wslay_event.c_7.o
clang-10: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay/wslay_frame.c -c -o build/wslay_frame.c_8.o
clang-10: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay/wslay_net.c -c -o build/wslay_net.c_9.o
clang-10: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay/wslay_queue.c -c -o build/wslay_queue.c_10.o
clang-10: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]

clang++ -DDLIB_LOG_DOMAIN="WEBSOCKET" -DDM_PLATFORM_LINUX -D__STDC_LIMIT_MACROS -DLUA_BYTECODE_ENABLE_64 -DHAVE_CONFIG_H  -ffloat-store -g -O2 -D__STDC_LIMIT_MACROS -Wall -Werror=format -fno-exceptions -fPIC -fvisibility=hidden  -Iupload/websocket/include/ -Iupload/  -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//sdk/include -I/var/extender/sdk/1a2776dac6a868eed1a35e41acb0c71a7e17948f/defoldsdk//include/x86_64-linux -Iupload/websocket/include/wslay  upload/websocket/src/wslay/wslay_stack.c -c -o build/wslay_stack.c_11.o
clang-10: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang-10: warning: optimization flag '-ffloat-store' is not supported [-Wignored-optimization-argument]


	at editor.engine.build_errors$build_error.invokeStatic(build_errors.clj:463)
	at editor.engine.native_extensions$build_engine_archive.invokeStatic(native_extensions.clj:266)
	at editor.engine.native_extensions$get_engine_archive.invokeStatic(native_extensions.clj:310)
	at editor.engine$get_engine.invokeStatic(engine.clj:197)
	at editor.app_view$async_build_BANG_$fn__45849$fn__45855.invoke(app_view.clj:769)
	at editor.app_view$async_build_BANG_$fn__45849.invoke(app_view.clj:767)
	at clojure.core$binding_conveyor_fn$fn__5739.invoke(core.clj:2030)
	at clojure.lang.AFn.call(AFn.java:18)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

Judging from the error, I suspected either an extender issue or the extension itself: recent releases of websocket extensions dealt with some of the DNS code, but on the other hand the whole log makes sense as an extender error - and it gets updated pretty often although I couldn’t trace its recent changes to make any educated guess.

Things I’ve checked:

upgraded websocket extension from 2.1.0 to 2.2.3 (with in-between versions, too) - no luck
switched from https://build.defold.com to https://build-stage.defold.com - no luck
jumped between different versions of Defold (1.8.2 → 1.8.3a and back) - no luck.

Off topic, but you can’t seem to download any other version of the editor except for the latest one O_o

Now I’m stuck with my development, so decided it’s time to ask for help after all as I couldn’t find anything similar on the forum.

jhonny.goransson · May 11, 2021, 5:20pm

Hi, we have recently removed dns.h from the sdk (per https://github.com/defold/defold/commit/6b6afb48e313cdf41fee49b1f06f4b54fd7996cb#diff-75f81fcc32e73d37b305f8d0b556ed4bc1e1e7680557711ac7cb499e0c0adc1d) and whatever version you are using now of the websocket extension doesn’t have the fixes for the latest release. We probably need to make a new release for that, ping @britzl

uncle_Night · May 11, 2021, 5:21pm

Phew, thanks guys! @britzl you’re my only hope!

britzl · May 11, 2021, 6:31pm

Yes, I will look into it!

Mathias_Westerdahl · May 12, 2021, 6:43am

There’s now a new release of the websocket extension:

uncle_Night · May 14, 2021, 5:23pm

My apologies, just got to finally test it - thanks gentlemen, it works, you saved my week (:

SpencerWF · May 19, 2021, 7:05pm

Also ran into this issue, is there a reason you would remove it from old releases? That seems contrary to the Defold convention of trying to not break prior builds?

britzl · May 19, 2021, 9:01pm

There’s a few things at play here:

The Defold extension API - This is the C API for extension and it is defined in a number of header files. The header files expose groups of functions in the engine that can be used in native extensions. Examples:
- dmSocket for working with sockets
- dmArray for working with arrays
- dmGameObject for manipulating game objects
The Defold Lua API - This API is the exposed set of Lua functions used in your scripts. Examples:
- go.*
- gui.*
- vmath.*

We try to never ever break things in the Lua API as it would have a big impact on a lot of users. A change in the Lua API would require a large amount of users to make changes to their games and it would cause a lot of frustration.

We also try to not change the Defold extension API, but it is much more low level and usually maps directly to low level functions in the engine.If something in the engine undergoes a big change (happens rarely!) this low level extension API is also likely to change. It will also have an impact on our users, but as a side effect of use of a native extension. In such a case it is the extension maintainer that has to do a change and the end users would only have to update a dependency in game.project when updating to the new version of the engine.

In this particular case we had a dmDNS namespace in our extension API that mapped to DNS lookup functions in c-ares which was used in the engine. When we removed c-ares there was no longer a need for the dmDNS namespace so we removed it, knowing that it would affect extensions using this API. To our knowledge it was only the WebSocket extension that used it, and it was developed by us so it seemed like a low risk change to make. When you guys updated your editors you got a new version of the engine without c-ares and dmDNS but you were still using a version of the WebSocket extension that expected the dmDNS functions to exist. So we failed at timely updating the WebSocket extension and telling you this in our release notes. Had you stayed on the old version of the engine and not updated the extension your project would still work.

Mathias_Westerdahl · May 20, 2021, 6:46am

is there a reason you would remove it from old releases? That seems contrary to the Defold convention of trying to not break prior builds?

While we try our best to be backwards compatible, we also need to make occasional changes.
For instance if it’s something we consider a bug in our Lua api, then we fix the bug and let our users know.

As @britzl mentioned, we try our best to keep the Lua API as stable as possible.
Since our Lua api is a higher abstraction, it’s more resilient to internal refactoring.
For our C++ api though, we are just a little bit more lax on this convention, since we from time to time need to make changes to it.
For instance, that’s one of the reasons why we don’t support Defold versions older thatn ~6 months for extensions, since the SDKs and build tools might change.

In this particular case, we removed the internal code since it was a wrapper for an external library, and it only caused a lot of grief (crashes).
At that point there really was no use for the dns.h header file anymore. We also care about keeping things tidy, and removing unused code to keep the engine small, and maintenance as low as possible.
It was a complicated header to use too, and I was 99% sure no one else was using it yet ( it was recently added to the SDK).

Also ran into this issue…

Note that the reason some users experienced this, is that they used the master branch which might change at any time with new features that are not supported yet. This is not something new. We always recommend users to use a specific version so that they don’t get unexpected build changes.
Finally, the reason the users noticed it is simply because I happened to press the Defold release button before the extension release button, which is just a mistake on my part.

I hope this makes is a bit more clearer!

uncle_Night · May 20, 2021, 5:59pm

My opinion - these things always happen regardless of how hard you try to cover everything. What’s important is how it gets handled, and Defold folks are extremely responsive and helpful.

My only nitpick (off topic, though) would be that the older Editor releases aren’t downloadable from d.defold.com - all the links under editor section just point to https://d.defold.com/stable so rolling back is difficult, and the best you can hope for is jumping between stable, alpha and beta channels.

britzl · May 20, 2021, 7:58pm

Oh, yes, we will fix this!

All releases are available on GitHub as well!

uncle_Night · May 21, 2021, 4:29am

…and now I feel silly for not having thought of this. Thank you!