Building luasec as native extensions

Yes, I already fixed that, thank you. :stuck_out_tongue:
I don’t know why the textbox has highlighted the warning line…

1 Like

Please keep us updated of your progress! I’m interested in using luasec for a couple of things.

Ok, I’m going on step by step. Added luasec files into the project and building seems ok, now the problem is the ssl library. First I had put ssl includes and then added libssl.a (and also libcrypto.a) but I got a different error:

clang++ -c -arch x86_64 -target x86_
at org.eclipse.core.runtime.Assert.isTrue(Assert.java:110)
at org.eclipse.core.internal.resources.MarkerInfo.checkValidAttribute(MarkerInfo.java:64)
at org.eclipse.core.internal.resources.MarkerInfo.setAttribute(MarkerInfo.java:148)
at org.eclipse.core.internal.resources.Marker.setAttribute(Marker.java:243)
at com.dynamo.cr.editor.builders.ContentBuilder.buildLocal(ContentBuilder.java:183)
at com.dynamo.cr.editor.builders.ContentBuilder.build(ContentBuilder.java:80)
at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:726)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:199)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:321)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:396)
at org.eclipse.core.internal.resources.Project$1.run(Project.java:618)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2344)
at org.eclipse.core.internal.resources.Project.internalBuild(Project.java:597)
at org.eclipse.core.internal.resources.Project.build(Project.java:124)
at com.dynamo.cr.editor.handlers.LaunchHandler$1.run(LaunchHandler.java:158)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Ah, I forgot that there is at the top of the error log:

org.eclipse.core.runtime.AssertionFailedException: assertion failed: Marker property value is too long: clang++ -c -arch x86_64 -target x86_64-apple-darwin11 -isysroot /opt/MacOSX.sdk/ -m64 -O2 -g -mmacosx-version-min=10.7 -DDM_PLATFORM_OSX -I/tmp/upload2574061937926758587/luasec/include -F/tmp/upload2574061937926758587/luasec/lib/osx -I/var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/include -I/var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/sdk/include /tmp/upload2574061937926758587/luasec/src/context.c -o/var/extender/builds/build7031438701784830272/context.c_0.o

… a long list of warnings and finally the piece I wrote above. :slight_smile:

G.

Hmm, the log is too large for displaying via the eclipse markers :confused:
I guess that your actual error is above that last marker error, having something to do with your context.c

No errors are in that file. After removing warnings, it seems there is a clash between crypto and dlib libraries:

duplicate symbol _RSA_free in:
    /tmp/upload4333439115711551525/luasec/lib/osx/libcrypto.a(rsa_lib.o)
    /var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/lib/x86_64-darwin/libdlib.a(rsa_2.o)
duplicate symbol _AES_cbc_encrypt in:
    /var/extender/sdk/1e53d81a6306962b64381195f081d442d033ead1/defoldsdk/lib/x86_64-darwin/libdlib.a(aes_2.o)
    /tmp/upload4333439115711551525/luasec/lib/osx/libcrypto.a(aes-x86_64.o)
ld: 2 duplicate symbols for architecture x86_64

Is there a way to exclude dlib? Or is it used by Defold anyway?

G.

Try removing libcrypto.a from your extension. If instead you will get missing symbols errors, then you would have to extract libcrypto.a, remove object files that cause symbol clashing and repack again.

Yes, these are functions used by our core implementation of ssl, and dlib is one of our core libraries so it won’t be possible to not use it.

Looking forward, my current task is to look into how to come up with a strategy for us on how to deal with these kinds of situations (E.g. how to replace standard parts of the engine with your own)

As for your current situation, what @sergey.lerg suggests could be one way to do it, or you could just get those source files you need from libcrypto, and namespace them and put them in your extension (or create a mini lib)

5 Likes

Duplicate object files can be removed automatically on the server before compilation. For instance a developer could add something like removeDuplicateSymbols: ["libcrypto.a"] to the ext.manifest.
If there is really a clash of names between two extensions or the core - their developers must agree who would change the symbols.
Also symbols can be renamed during compilation with a special flag -DOriginal=Replacement. A good option could be to automatically prefix all such symbols with the extension name, e.g. in ext.manifest something like prefixSymbols: ["Original1", "Original2"]. They would become ExtensionNameOriginal1 and ExtensionNameOriginal2.

1 Like

AES_cbc_encrypt and RSA_free are the duplicated methods across those two libraries. It’s a bit messy to fix this issue without break crypto library. The ideal is get rid them from dlib and then include crypto into your build scripts. I also tried your tips, but unfortunately they don’t work.

G.

1 Like

Ok, I sorted this stuff out replacing those two symbols names inside crypto library at the end. Then now I’ve got two fat static libs, libssl.a and libcrypto.a, for x86_64 and i386 architectures (MacOS). They seem that are built against my luasec extension in Defold without problems. Trying to figure out how I can bind luasec with the Defold extension system.

2 Likes

Finally LuaSec extension seems working ok inside my project. Do you want it so you can have a look?
Now I’m figuring out how I can use this SSL extension with sockets/websockets (using defnet.websocket library).

G.

4 Likes

Nice! Do you have it as a library project available on GitHub or something? I’d be very interested in giving it a try. I’d like to try to get secure websockets working.

1 Like

Eh no, but I can send it through this message. :slight_smile: luasec.zip (3.1 MB)

It’s based on LuaSec 0.6 plus OpenSSL 1.0.2k and I did just very few tests with it so far. :slight_smile:

G.

2 Likes

Ah, forgot, it works only on MacOS at the moment, but it should be pretty easy add static libraries for iOS.

1 Like

New version here luasec.zip (3.1 MB)

I fixed few things and removed the https module, it was redundant. It has been tested against socket implementation in Defold and these samples and it works well on MacOS at least.

G.

3 Likes

Working on iOS and it will be available very soon. Ah, I updated OpenSSL static libraries to 1.1.0e, because the older ones don’t support 64-bit build for iOS.

2 Likes

Excellent! Good to hear that you are making progress!

1 Like

Here’s the new package. :slight_smile: luasec.zip (6.0 MB)
… with some tests tests.zip (68.1 KB)
Going towards the Android version now. :slight_smile:

4 Likes

How’s it going? Have you been making any progress on the Android version?