Building luasec as native extensions

Hello,
I’m trying to build luasec but unfortunately without success so far… The documentation about ext.manifest file doesn’t help me a lot and I would ask which flags and linkflags are permitted and where can I put things such as -DMY_DEFINE stuff? Luasec requires lua and openssl library, I mean dylib files, is it enough move them inside lib/osx folder in my defold project?

Thanks,
G.

Anybody here?

Hi, sorry for the delay. Let me try to find an answer for you. @Mathias_Westerdahl could you please help @sonountaleban

Thank you very much. :slight_smile:

Hi.
Our support for compile flags is limited and I agree we should document it better, but basically, it supports ObjC, ObjC++, stdlib=c++11 etc, and switching on/off some warnings. I’m sure we’ll add more along the way on as requirements pop up. Which one do you need?

As for defines, you can add those at the top of your .cpp file, before your includes.

Looking at the code for OSX and native extensions, I’m not sure .dylib is 100% working. They seem to be included in the build process (add them to your x86-osx/libX.dylib), but later, I’m not sure if they’ll be included in the actual bundle. I’ll make sure to test it further. In the mean time, perhaps you can use static libraries?

2 Likes

Hello Mathias,
thank you for the reply. However, I’m trying to build a sample project taken from manuals pages and I’ve got this error:

clang++ -c -arch x86_64 -target x86_64-apple-darwin11 -isysroot /opt/MacOSX.sdk/ -m64 -O2 -g -mmacosx-version-min=10.7 -DDM_PLATFORM_OSX -stdlib=libc++ -I/tmp/upload4455052575334162150/luasec/include -I/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/include -I/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/sdk/include /tmp/upload4455052575334162150/luasec/src/luasec.cpp -o/var/extender/builds/build5370711675559338880/luasec.cpp_0.o
In file included from /tmp/upload4455052575334162150/luasec/src/luasec.cpp:6:
In file included from /var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/include/dmsdk/sdk.h:11:
/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/include/dmsdk/dlib/log.h:109:2: warning: “DLIB_LOG_DOMAIN is not defined” [-W#warnings]
#warning “DLIB_LOG_DOMAIN is not defined”
^
1 warning generated.

llvm-ar rcs /var/extender/builds/build5370711675559338880/lib19035937-37f7-43a7-a607-cf3b5358876c.a /var/extender/builds/build5370711675559338880/luasec.cpp_0.o

clang++ -c -arch x86_64 -target x86_64-apple-darwin11 -isysroot /opt/MacOSX.sdk/ -m64 -O2 -g -mmacosx-version-min=10.7 -DDM_PLATFORM_OSX -stdlib=libc++ -I/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/include -I/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/sdk/include /var/extender/builds/build5370711675559338880/main.cpp -o/var/extender/builds/build5370711675559338880/main_tmpf7f59c31-b35d-48ff-90c5-748495bcd480.o

clang++ -arch x86_64 -target x86_64-apple-darwin11 -isysroot /opt/MacOSX.sdk/ -m64 -O2 -g -mmacosx-version-min=10.7 -o /var/extender/builds/build5370711675559338880/dmengine -ObjC -L/var/extender/builds/build5370711675559338880 -lz -lc++ -lsqlite3 -l19035937-37f7-43a7-a607-cf3b5358876c -F/var/extender/builds/build5370711675559338880 -framework Foundation -framework AppKit -framework Cocoa -framework OpenGL -framework OpenAL -framework AGL -framework IOKit -framework Carbon -framework CoreVideo -L/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/lib/x86_64-darwin -L/var/extender/sdk/1afccdb2cd42ca3bc7612a0496dfa6d434a8ebf9/defoldsdk/ext/lib/x86_64-darwin -lengine -ladtruthext -lfacebookext -liapext -lpushext -liacext -lrecord -lgameobject -lddf -lresource -lgamesys -lgraphics -lphysics -lBulletDynamics -lBulletCollision -lLinearMath -lBox2D -lrender -lscript -llua -lextension -lhid -linput -lparticle -lrig -ldlib -ldmglfw -lgui -ltracking -lcrashext -lsound -ltremolo -lvpx -lliveupdate /var/extender/builds/build5370711675559338880/main_tmpf7f59c31-b35d-48ff-90c5-748495bcd480.o
Undefined symbols for architecture x86_64:
"_luasec", referenced from:
__Z17dmExportedSymbolsv in main_tmpf7f59c31-b35d-48ff-90c5-748495bcd480.o
ld: symbol(s) not found for architecture x86_64
clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)

luasec.cpp is the same of myextension.cpp file. I suspect there may be something wrong in my system… maybe my lua include/lib files?

G.

That warning is just a warning. I’ll fix that to the next release though, since it’s annoying :slight_smile:

Here is the real error, a linker error. I suspect that the name of your extension (name: "luasec" ?) doesn’t match the name you put in the .cpp file (DM_DECLARE_EXTENSION(LuaSec, ... ?)

In any case, I’ve reported a documentation issue for @sicher, to clarify this in the documentation.

3 Likes

Yes, I already fixed that, thank you. :stuck_out_tongue:
I don’t know why the textbox has highlighted the warning line…

1 Like

Please keep us updated of your progress! I’m interested in using luasec for a couple of things.

Ok, I’m going on step by step. Added luasec files into the project and building seems ok, now the problem is the ssl library. First I had put ssl includes and then added libssl.a (and also libcrypto.a) but I got a different error:

clang++ -c -arch x86_64 -target x86_
at org.eclipse.core.runtime.Assert.isTrue(Assert.java:110)
at org.eclipse.core.internal.resources.MarkerInfo.checkValidAttribute(MarkerInfo.java:64)
at org.eclipse.core.internal.resources.MarkerInfo.setAttribute(MarkerInfo.java:148)
at org.eclipse.core.internal.resources.Marker.setAttribute(Marker.java:243)
at com.dynamo.cr.editor.builders.ContentBuilder.buildLocal(ContentBuilder.java:183)
at com.dynamo.cr.editor.builders.ContentBuilder.build(ContentBuilder.java:80)
at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:726)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:199)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:321)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:396)
at org.eclipse.core.internal.resources.Project$1.run(Project.java:618)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2344)
at org.eclipse.core.internal.resources.Project.internalBuild(Project.java:597)
at org.eclipse.core.internal.resources.Project.build(Project.java:124)
at com.dynamo.cr.editor.handlers.LaunchHandler$1.run(LaunchHandler.java:158)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Ah, I forgot that there is at the top of the error log:

org.eclipse.core.runtime.AssertionFailedException: assertion failed: Marker property value is too long: clang++ -c -arch x86_64 -target x86_64-apple-darwin11 -isysroot /opt/MacOSX.sdk/ -m64 -O2 -g -mmacosx-version-min=10.7 -DDM_PLATFORM_OSX -I/tmp/upload2574061937926758587/luasec/include -F/tmp/upload2574061937926758587/luasec/lib/osx -I/var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/include -I/var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/sdk/include /tmp/upload2574061937926758587/luasec/src/context.c -o/var/extender/builds/build7031438701784830272/context.c_0.o

… a long list of warnings and finally the piece I wrote above. :slight_smile:

G.

Hmm, the log is too large for displaying via the eclipse markers :confused:
I guess that your actual error is above that last marker error, having something to do with your context.c

No errors are in that file. After removing warnings, it seems there is a clash between crypto and dlib libraries:

duplicate symbol _RSA_free in:
    /tmp/upload4333439115711551525/luasec/lib/osx/libcrypto.a(rsa_lib.o)
    /var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/lib/x86_64-darwin/libdlib.a(rsa_2.o)
duplicate symbol _AES_cbc_encrypt in:
    /var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/lib/x86_64-darwin/libdlib.a(aes_2.o)
    /tmp/upload4333439115711551525/luasec/lib/osx/libcrypto.a(aes-x86_64.o)
ld: 2 duplicate symbols for architecture x86_64

Is there a way to exclude dlib? Or is it used by Defold anyway?

G.

Try removing libcrypto.a from your extension. If instead you will get missing symbols errors, then you would have to extract libcrypto.a, remove object files that cause symbol clashing and repack again.

Yes, these are functions used by our core implementation of ssl, and dlib is one of our core libraries so it won’t be possible to not use it.

Looking forward, my current task is to look into how to come up with a strategy for us on how to deal with these kinds of situations (E.g. how to replace standard parts of the engine with your own)

As for your current situation, what @sergey.lerg suggests could be one way to do it, or you could just get those source files you need from libcrypto, and namespace them and put them in your extension (or create a mini lib)

5 Likes

Duplicate object files can be removed automatically on the server before compilation. For instance a developer could add something like removeDuplicateSymbols: ["libcrypto.a"] to the ext.manifest.
If there is really a clash of names between two extensions or the core - their developers must agree who would change the symbols.
Also symbols can be renamed during compilation with a special flag -DOriginal=Replacement. A good option could be to automatically prefix all such symbols with the extension name, e.g. in ext.manifest something like prefixSymbols: ["Original1", "Original2"]. They would become ExtensionNameOriginal1 and ExtensionNameOriginal2.

1 Like

AES_cbc_encrypt and RSA_free are the duplicated methods across those two libraries. It’s a bit messy to fix this issue without break crypto library. The ideal is get rid them from dlib and then include crypto into your build scripts. I also tried your tips, but unfortunately they don’t work.

G.

1 Like

Ok, I sorted this stuff out replacing those two symbols names inside crypto library at the end. Then now I’ve got two fat static libs, libssl.a and libcrypto.a, for x86_64 and i386 architectures (MacOS). They seem that are built against my luasec extension in Defold without problems. Trying to figure out how I can bind luasec with the Defold extension system.

2 Likes

Finally LuaSec extension seems working ok inside my project. Do you want it so you can have a look?
Now I’m figuring out how I can use this SSL extension with sockets/websockets (using defnet.websocket library).

G.

4 Likes

Nice! Do you have it as a library project available on GitHub or something? I’d be very interested in giving it a try. I’d like to try to get secure websockets working.

1 Like