Hi all. I have a question.
Do any of you have experience with computer vision or AR?
Is it possible to do this on Defold?
I want to choose suitable clothes or accessories according to a person’s figure.
Something like a face mask or something similar.
Thank you in advance.
I really need to try building a native extension to see what the process is like. Anyway, I think this should be possible, since you can integrate libraries from other languages into the engine binary.
See here: Writing native extensions for Defold
My rough idea would be to perform the AR / vision calculations in C or whatever language, then pass the resulting data back to a Lua script, then render something using standard Defold techniques. I bet you could directly render something using Defold’s C++ API without passing data back to Lua, but I don’t have experience with that.
Thanks for the answer.
What you described is a good idea.
It’s hard for me to imagine implementing this in Defold, because this is not a simple question, alas, I have very little experience in this, especially with C++.
I found two libraries for use with Lua: dlib and libvips.
Maybe there is some documentation or a possible example of implementing something similar on Defold, it would be very helpful.
I just haven’t seen an option like this, just face masks, even though it’s very similar to the method I’m looking for, I think it requires a slightly different image processing process.
I referred to this project when I talked about AR face masks: GitHub - britzl/facesnap: Camera and face recognition with AR overlay using Defold native extensions by Britzl
Stumbled upon this by accident.
Simple AR - you need to display camera and display your main collection with 3D objects - then you should also rotate camera “in-game” to reflect rotation of the mobile phone/tablet in real life - so you need to use available sensors.
Defold-Gyro contains an example project of AR utilising also camera extension - if you are targeting mobile platforms, then you have all the bricks 
Advanced AR - requires also tracking subtle changes of position of the mobile device - which is really hard. There are other libraries making it easier, like OpenCV if you are looking for enviroment-features / markers based solutions (especially that you will need something like this for suiting cloths to real life person) or some based on a fusion of sensors (GPS, gyroscope, accelerometer, gravity sensor, etc) in C++ or JS (if you are targeting web) to estimate position changes with better or worse accuracy - but sadly, nobody yet wrote any native extension for Defold with any such.
or OpenCV of course - all of them though are really complex libraries and converting it into a native extension for Defold might be a tough task to do - but of course possible
You definitely should be advised to process images in C++. There is something like Lua bindings for vips:
But I don’t think it’s wise to invest your time in this, because most probably: if you would make too solutions: NE using C++ lib and Lua script just using it OR Lua script with some complex Lua library for image process - it MIGHT be that C++ based solution will be better/faster (I say it with a dose of doubt, because I don’t have proofs for this of course )
Because I’m trying to do something like this for web, I am interested in checking out arJS or something similar.
Thanks for the information provided. 