Sqlite native-extension

As discussed by @Arônimo on the Defold Discord he was interested in a sqlite native extension. Having built a few lua extensions (from pather packages) a few times I decided to setup a publicly available Defold one here:

The project is now quite usable.

What this project has:

  • A sqlite3 native binding for desktop OS’s - OSX, Windows and Linux
  • Use of the native interface in log.gui_script
  • A gui that has a simple command line interface. Described below.

The command line has:

  • Simple data entry. (supports only backspace).
  • History. Arrow up / down to select previous commands
  • No cursor functionality plain text entry.
  • Output window shows select’s and command errors

The extension is based almost completely on this project here:
https://mroth.net/lua-sqlite3/documentation
Examples and information about its use in lua should work as is. You will always need to add a local sqlite3 = require("libsqlite.sqlite3") to your lua files to use the examples.

Here’s a pic of the functioning gui:

The other platforms will be looked at over the coming week.
Hope this helps.

19 Likes

All usable and working now. Will add android and ios over coming week (all things going well).
Feel free to copy and use you need. Will make it a formal native extension soon too.

7 Likes

Little update. Heres the build repo for the sqlite libs. Trying to org an android and IOS build for it.

7 Likes

this looks like exactly what I need!

I’m trying to get it hooked up in my game, but I’m having trouble figuring out the proper way to actually use it. I can’t make sense of the return value of the db:exec() function. And, looking at your example, you don’t even use the results variable, but do some other stuff instead. I’ll continue poking at it, though.

You can use the examples from the original site:
https://mroth.net/lua-sqlite3/documentation
These should work fine (I’ll check later tonight to make sure).
The exec() method generally is returning errors and trapping exceptions.
If you need to iterate rows you might need to use prepare or the rows iterator like:

-- Returns a row as an integer indexed array
for row in db:irows("SELECT * FROM test") do
  print(row[1], row[2])
end

Theres a number of examples of this in the docs. They should map 1:1. But as I said, I’ll check that.

2 Likes

I had a quick look, and it is using the irows already in this function:

function process_cmd( self )
	local results, errors = self.db:exec( self.cmd )
	if(results) then

		self.log = self.log.."\n[ Sqlite3 ] " .. self.cmd
		if( string.lower(string.sub(self.cmd, 1, 7)) == "select ") then 
			self.log = self.log .. "\n"
			-- Returns a row as an integer indexed array
			for row in self.db:irows(self.cmd) do
				local rowstr = ""
				for idx in pairs(row) do
					pprint(idx, row[idx])
					rowstr = rowstr .. " | ".. tostring(row[idx])
				end
				self.log = self.log .. "\n" .. rowstr .. " | "
			end
			self.log = self.log .. '\n'
		end
		self.log = self.log.."\n[ Sqlite3 ] OK."
	end 
	if(errors) then self.log = self.log.."\n[ Sqlite3 Error ] " .. errors end
	table.insert(self.hist, self.cmd) 
	self.cmd = ""
	self.hist_select = nil
end 

This is in the file main/log.gui_script.
The command line at runtime uses this simple parser. You cant really use it for much else without adding more complex command line parsing.

The Defold window is mainly a test window to show how to use it. It is not a production tool. :slight_smile:

Thanks, that was exactly what I was missing! I think I got a handle on it now.

1 Like

As reported by tc on discord here:
https://discord.com/channels/250018174974689280/954383800840892426/1287375647051878480
There seems to be some problems with the android build. Im looking into it now - it is throwing seg faults on my phone, so I can definitely replicate. Hope to have a solve soon.

2 Likes

Ok. I think I know what has happened. The amalg sqlite lib needs to match the Android version (I have changed phones since I did this, so this makes sense).

https://stackoverflow.com/questions/2421189/version-of-sqlite-used-in-android/4377116#4377116

It looks like I will need to provide multiple builds for this, which is kinda annoying. I’ll have a look how much I need to do, and whether I can get it the github action to make a few diff variants.

It also looks like because there is sqlite3 installed in the OS, there may be problems linking a different version in the lib. This shouldnt matter, but as always, need to check first.

More updates. There is a problem with lining up splite3 amalg versions, android versions and Defold android versions (min SDK for example).
The builds for the shared libraries must match the min SDK versions for Defold android build (which makes sense), however there are also build issuse with older versions of sqlite3 amalg (which is what I was using). And this had problems referencing mmap64 (which is a memory function call to the os). I had problems with this previously.

Im working on updating so that the minimum Android SDK will be 30. NDK will be 24 (Im checking to make sure that matches too). Its taking alot longer than Id like, so I might not finish it tonight and will revisit tomorrow.

Ok. I think I found the problem. The Android extension is being built with a very old abi (19)

clang-14: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rcs /tmp/job6537066999253325302/build/libLibSqlite_2.a build/libsqlite3.c_1.o 
/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi19-clang++ -c -O2 -g -gdwarf-2 -fpic -ffunction-sections -fstack-protector -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -fomit-frame-pointer -fno-strict-aliasing -funwind-tables -isysroot=/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot -DANDROID -Wa,--noexecstack -DDLIB_LOG_DOMAIN="UNKNOWN" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_ANDROID -DLUA_BYTECODE_ENABLE_32 -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ -DDM_DEBUG  -fno-exceptions -fvisibility=hidden   -I/var/extender/sdk/691478c02875b80e76da65d2f5756394e7a906b1/defoldsdk//include -I/var/extender/sdk/691478c02875b80e76da65d2f5756394e7a906b1/defoldsdk//sdk/include -I/var/extender/sdk/691478c02875b80e76da65d2f5756394e7a906b1/defoldsdk//ext/include  -I/opt/platformsdk/android/android-ndk-r25b/sources/android/native_app_glue -I/opt/platformsdk/android/android-ndk-r25b/sources/android/cpufeatures  build/main.cpp -obuild/main_tmp3.o
/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi19-clang++ -fuse-ld=/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/ld.lld -O2 -g -isysroot=/opt/platformsdk/android/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot -static-libstdc++ build/main_tmp3.o  -o build/libdmengine.so -Wl,-soname=libdmengine.so -DDLIB_LOG_DOMAIN="UNKNOWN" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_ANDROID -DLUA_BYTECODE_ENABLE_32 -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ -DDM_DEBUG  -Wl,--build-id=uuid  -Wl,--fix-cortex-a8 -Wl,--no-undefined -Wl,-z,noexecstack -landroid -fpic -z text -L/tmp/job6537066999253325302/build -L/tmp/job6537066999253325302/upload/libsqlite/lib/android  -L/var/extender/sdk/691478c02875b80e76da65d2f5756394e7a906b1/defoldsdk//lib/armv7-android -L/var/extender/sdk/691478c02875b80e76da65d2f5756394e7a906b1/defoldsdk//ext/lib/armv7-android  -Wl,-Bstatic -Wl,--start-group -lc++_static -lsqlite3-armv7 -lgraphics_vulkan  -lsqlite3-armv7 -lsqlite3 -lLibSqlite_2  -lengine -lengine_service -lmbedtls -lzip -lprofile -lremotery -lprofilerext -lrecord_null -lgameobject -lddf -lresource -lgamesys -lscript_box2d -lgraphics -lgraphics_transcoder_basisu -lbasis_transcoder -lphysics -lBulletDynamics -lBulletCollision -lplatform -lLinearMath -lBox2D -lrender -lscript -lluajit-5.1 -lextension -lhid -linput -lparticle -lrig -ldlib -ldmglfw -lgui -lcrashext -lsound -ltremolo -lliveupdate -lunwind  -Wl,--end-group -Wl,-Bdynamic  -lEGL -lGLESv1_CM -lGLESv2 -lOpenSLES -lm -llog -lc  -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -shared

Which means the compiler is whats not able to link mmap64 and likely whats causing segfaults (bad os calls). I think its possible to set the abi level in the manifest, so I’ll try to get that working and it should fix the issue.

2 Likes

Ok. some things Ive found with the help of @JCash. The extension clang++ compiler is kind of set to use android-19 abi which is fine, but it means I need to get sqlite3 and the amalg build lined up to use the exact same build process.

The other odd thing I found was that the apk contained libsqlite.so’s which shouldnt really be there. The extension should link in the static library and no shared libraries should be needed. I suspect my build flags are incorrect or there is something happening in the extension building process I need to address.

Ok. this has taken alot of investigation and I think Im at a bit of an impass. Theres a couple of ways around the problems for Android, but I can see there being other issues with other extensions that might be built this way.

Firstly, I’ll explain how the sqlite build process works and why.
Normally Defold extensions are built all in a source folder and that is compiled together into your application and that is all that needs to be done. However, there is a problem building sqlite and sqlite amalg.
The code uses old C methodologies and the clang++ compiler does not accept them (usually casting related).
Normally with say gcc, you could disable some of these items and build anyway. However in this case, the extension compiler is using Defold build system which uses clang++ and android19 abi for armv7 and android22 for arm64.

To be able to build the extension, I decided I could build a static library and link it into the extension, thus embedding sqlite3 into the extension. This is what all the lib files are. Various versions of static libraries for use on different platforms. Easy. Solved right?

No sadly. These need to be also built with the same compiler. However, I had to use the clang C compiler (this shouldnt matter, but it might) to build the libraries. This is what the sqlite3-amalg repo in my repositories is for. This builds all the platforms with the correct settings.

Then these libs are copied into the defold-sqlite/libs folder and then the project can be built for whatever platform you need. However, the android extension does not seem to be linking in the static libraries. And creating a separate shared library that is attempted to be called at runtime. From the error logs in logcat, it looks like that these files, because they dont have runtime permission, fail to allow the main app to call and execute them.

Ideally these shared libraries shouldnt even exist. There is an added complication too. Most (if not all) Android devices already have a shared sqlite library installed as part of the OS. And this could be problematic if the shared library is trying to call OS linked libraries (which it shouldnt).

Ways forward:

  1. It could be possible to setup a Extension build server and then build the extensions exactly as needed, but this is alot of work.
  2. Modify the sqlite amalg C code to not have errors and build it within the extension build process. This is a huge job. I did look at this last year, and it is not worth it considering the updates that sqlite regularly do.
  3. Use the Android builtin library and call directly. This is probably the most promising way forward, but it would need a separate build process to do this. In fact, it might be easier to just make an android sqlite specific project. I suspect the IOS version may suffer similar issues.

Having done quite a number of wildly different extensions, generally, I have found ways around problems. This one sadly has been one of the more “blocker” styled ones.

Being able to link libs into the extension should work fine but this is a little more complicated with the C/C++ differential. I have ran into other C libs which do not build well in extensions, so it does seem to be a limiting factor and it has me wondering about alternatives.

Anyway, to tc. Sorry mate. I think your best bet is to probably use something like a lua based db - https://github.com/nrk/redis-lua
This could be converted easily enough to work with a redis keystore - need to replace the luasocket calls with Defold socket calls :slight_smile:

1 Like

Hey Tc here,
Thank you so much for trying.
I am struggling to accept the idea that sqlite cannot be used in defold.

i really like defold, but the lack of a local database option feels like a big gap in this engine.

before deciding if migrate to another engine or not i will explore few options.

You are jumping the gun. I am confident that it is possible to get a database solution working as a native extension.

2 Likes

I didn’t know the details, just for an info; there is a very old and undocumented extension developed by @Mathias_Westerdahl and it has armv7-android version. maybe there is a clue how to use/build it.

1 Like

Nice find @selimanac !

Looking in there I found my old build script:

Which uses ndk-build, which should allow you to prebuild such a library.

1 Like

It should certainly be possible, but from my perspective, its a non-trivial process especially if you want recent sqlite3 release.

Try the other version, it might work fine - it is very old as @selimanac points out (2017 v3.21).

< quick update >
This does seem to build ok for android armv7. It is building for android min sdk of 14 - not sure if that will be a problem for installs.

It might be possible to build a fresh version of the latest sqlite3 in this manner (using the jni build process). Im not certain though. I’m personally still a little puzzled why the extension builds an ‘.so’ file in the apk? This file does not exist in the build folder, so I can only assume this is being created by the extension process?

It appears the static lib is converted to a shared lib. Very odd (in both projects btw).

< update >
[ Slaps hand on forehead ] Sorry… of course doh!! Cos its java, it has to wrap the static C/C++ lib in a shared lib so the java app can access via jni. Uggh… makes sense now.

I think @toca is right though. It does seem like something that should be kinda “out of the box” type of thing to have in Defold. Like him, its usually one of the first things I bolt into luajit or c++. Usually either sqlite or redis. These are pretty common tools.

I’d like to spend more time on this to get it running latest android. But I just dont have the time with the many things Im currently juggling. Defold is a great framework, but I think Im burning far too much time just trying to make glue bits… when I need to be making actual application systems and features.

Dunno. sorry to side track here.

i 100% agree, i’m sure if i or you guys spend enough time we can have sqlite working in defold.

the question is:
do i want to spend between 1 to 10 weeks (as it’s not my fulltime job) to integrate sqlite in defold, having probably not a great time.

or do i want to spend that time learning another engine? that could already have a solution?

these are the kind of questions i’m wrestling with :smiley:

i think i might take a moderate approach and spend a bit of time to understand how defold extensions works and how to include sqlite within defold at least in windows/android, when it stops being fun it’s time to move on for me and maybe come back at later date.

or maybe i will find a solution! fingers crossed.

1 Like

May I ask (just out of curiosity), why do you think you need SQL? Do you require large queries?