Moving multithreading implementation within defold

i’ve implemented a really hacky way to do multithreading that uses shell piping, but it requires the user to have lua installed on their system to use (i don’t want to force users to do this), and i feel that running defold with the command line would take too much time compared to a raw lua install for the implementation to be worth it (right now with running 5.4 it takes about ~.005 seconds just to initialize a thread) (any longer and it stops being worth it since the max tickrate would be vastly lower).

what would the best approach to do here that would ensure the user experience is smooth and the code fast?

Isn’t a native extension using dmThread better?

well, i recall looking at that some time ago, so my memory is fuzzy. but i recall it having datatype limitations that were a bit too much for me, whereas i can feasibly send most useful datatypes with what i have (eg, tables, mostly) between the main thread and side thread on creation and completion

also that said that it was only tested to work on windows, but i’d need it to work on linux, too.
that and the last post was a couple of years ago from what i can tell so i’m not sure if it’s still alive

also i don’t know C++ or the Lua API, so i can’t implement it myself, really…

Can you share some of yours so everyone can get its advantages?

Can you share some of yours so everyone can get its advantages?

well, i’m not sure the code is ready for public use yet, it’s still not the best implementation (far from it, even), and there’s definitely a lot of caveats to it.

it’s very rudimentery, but it does work.

caveats:

  • requires io access
  • writes to the disk a lot
  • barebones
  • data transfers are sent in code
  • controlling execution environment is unrefined

good things:

  • multiplatform
  • runs with pure lua
  • “werks, dunnit?”
luathread={}
luathread.threads={}
luathread.threadlocation="threads" --relative path usually.

luathread.newthread=function(threadcode)
local n
for i=1,#luathread.threads+1 do if not luathread.threads[i] then n=i break end end
local tf,e
lf,e=io.open(luathread.threadlocation.."/"..n,"w")
if e then error("error while opening thead file: "..e) end
tf:write("_=function() "..threadcode.." end io.write(table.dump(_())) ")
tf:close()
tf=io.popen("lua -l dumplib "..luathread.threadlocation.."/"..n) --dumplib contains a function which dumps a table into a string which we can load back later (table.dump).
luathread.threads[n]={
n=n,
threadfile=tf,
threadresult=nil,
location=luathread.threadlocation.."/"..n,
close=luathread.closethread,
get=luathread.getthread,
}
return luathread.threads[n]
end

--so, fun fact, we don't *need* to close these files. it slows it down a ton. instead, set to nil, and the garbage collector does it for you when it needs to. much faster!

luathread.getthread=function(thread)
if thread.threadresult then
	return table.unpack(thread.threadresult)
else
	thread.threadresult=thread.threadfile:read("*all")
	thread.threadfile=nil
	thread.threadresult="return {"..thread.threadresult.."}"
	thread.threadresult=load(thread.threadresult)()
	return table.unpack(thread.threadresult)
end
end

luathread.closethread=function(thread)
	luathread.threads[thread.n]=nil
	thread=nil
end

return luathread

@britzl does dmThread allow for shared state threading? if it does i might look into making an actual extension using it.

I get the idea of it, much like we run a command line and get the result via a file later. Not sure it will work on HTML5 with that pure lua code.

dmThread is using system/OS threads, which allows you to do whatever you’d like.

Note that Lua doesn’t support multithreading, so I believe the recommendation is to create separate lua states, and create some communication between the threads e.g. via some serialization.

1 Like

so you’re saying i don’t have to make separate states?

I’m saying you probably have to?

Lua states aren’t thread safe, so to use the same context, you’d have to synchronize all threads, effectively making it single threaded.

So, to do any meaningful multithreading, you’d have to use different Lua states, and you have to figure out some communication between them.

1 Like

The question is, what is this needed for?
I think using Lua throughout might be overkill.

If I were to create a thread to calculate a lot of things, I’d create a Lua api, to collect the data on the C side, then pass this C data to my threads.

Then, once the output data has been calculated, I’d push the result from the thread to my code onto e.g. an array (using a mutex), that my Update() function on the main thread can consume, and call back into Lua with.

1 Like

The question is, what is this needed for?

I hate when programs put restrictions in place things they’d be capable of otherwise without.
Simply put, allowing direct access of variables to threads makes my job a lot easier since it means i get to control with far more scrutiny what is going on instead of being forced to use, (and consider how i have to change my approach when using) an API to do everything.

using Lua throughout might be overkill.

i’m most comfortable in Lua, so this would allow me to use lua for anything thread related, which allows me to run more complex code as a result

You didn’t really answer the question :slight_smile: There are plenty of Defold games released without need for threads. What do you need to do that you are not able to do now?

What do you need to do that you are not able to do now?

i feel like that question (in the context of multithreading) is pretty self-evident.
why would anyone want more threads?
to perform more computations in the same time.

there’s nothing i specifically need them for right now, but having that feature can open a lot a doors down the line.

I’m not against multi-threading. I was just wondering what kind of problem you were trying to solve in your game, but I now realise that you were experimenting with this for use in some future scenario :+1:

1 Like

So, i’ve been cooking, but the kitchen is on fire. (I have no idea what i’m doing ok?), and i need help extinguishing it.
(most of this code was written in collaboration with chatGPT, which i am not proud of.)
here’s the error from the crash dump
The thread tried to read from or write to a virtual address for which it does not have the appropriate access

here’s a snippet of what i’m doing:
the function to create a tread (exposed to the lua enviroment for execution in scripts)

int newthread(lua_State* L) //creates a thread
{
	if(lua_gettop(L)<1 || (  !(lua_iscfunction(L,1) || lua_isfunction(L,1) )  )){ //we need a function as the argument
		return luaL_error(L,"Expected function as argument");
	}
	int funcref = luaL_ref(L, LUA_REGISTRYINDEX); //the function that was passed to us - get the ref to it for later.
	lua_getfenv(L, -1); //get _ENV
	int envref = luaL_ref(L, LUA_REGISTRYINDEX); //and store it.
	
	context ctx; //make a new context	
	
	dmThread::Thread thread = dmThread::New(execthread, 0x80000, (void*)&ctx, "defthread");
	
	
	ctx.status=THREAD_RUNNING; //status indicators.
	ctx.Lstate=L; //store the lua state for later use, too.
	ctx.contextaddr=&ctx; //why not?
	ctx.threadthread=thread; //and store the thread for passing to other functions
	ctx.enviromentref=envref;
	ctx.functionref=funcref;
	
	context** ud = (context**)lua_newuserdata(L, sizeof(context*)); //theifery from AI
	*ud = &ctx; //didn't originally include the &, but the compiler says jump, i say how high.
	return 1;
}

the context struct, which is also returned as userdata for passing to other thread functions.

struct context{
	int status;
	lua_State* Lstate; 
	void* contextaddr;
	dmThread::Thread threadthread;
	int enviromentref;
	int functionref;
};

and this is the C function where the error occurs, whose pourpose is to execute the function passed to newthread

static void execthread(void* _ctx){ //the thing that is executed in a thread.
	context* ctx = (context*)_ctx;
	lua_rawgeti(ctx->Lstate,LUA_REGISTRYINDEX,ctx->enviromentref); //ensure it runs in the same enviroment
	lua_setfenv(ctx->Lstate,-1);
	lua_pop(Lstate,1)
	lua_rawgeti(ctx->Lstate,LUA_REGISTRYINDEX,ctx->functionref); //get the func
	lua_call(ctx->Lstate,0,0); //we are calling that function now. 0 in 0 out to make my job a bit easier. why would you need arguments anyways? use a table in _ENV.
	luaL_unref(ctx->Lstate,LUA_REGISTRYINDEX,ctx->functionref);
	luaL_unref(ctx->Lstate,LUA_REGISTRYINDEX,ctx->enviromentref);
	ctx->status = THREAD_FINISHED;
}

This is an advanced task, so I’m not surprised that the chat bot doesn’t give a solution to it.

For the problem, remember that you mustn’t use a Lua state which is owned by one thread, on another thread: Lua is not thread safe!

So, that’s why you’ll likely wish to create a new lua state to use with your thread. Then the problem becomes 1) The new Lua state doesn’t have the same modules loaded, 2) the function that you pass (and its arguments), also need to be serialized from the main thread, and deserialized on the new thread.

While we’d like to have some sort of job system in the long run, I’m not sure how we’d solve it ourselves yet, so I can’t give more advise than that.

" there’s nothing i *specifically* need them for right *now* , but having that feature can open a lot a doors down the line."

While it might be a fun exercise, I’d personally find out what I actually need it for, and then create a specific C extension (without the Lua thread) to solve my calculation problem.

Good luck!

2 Likes

well i was hoping i could brute force it to share memory (something i don’t have the C wizardry to even begin to know how to do)

i guess i did learn a bit about C++ and the Lua API in the process…

I’d personally find out what I actually need it for

i’ve already thought of a hypothetical, and that’s processing game entity logic en masse, threads allowing for a massive number of, or a highly intricate system without bogging a system down

so i’m doing a different approach where threads are separate states, but…
you get to pass a table (or other serializable datatype, eg: not a file or userdata) though to the function as a parameter
i’m putting a moderate attempt into figuring out what’s going on

good thing:
the engine no longer crashes with an error upon creating a thread
bad thing:
the engine doesn’t give an error code when it crashes upon creating a thread.

edit update 3 hours later:
it still soft crashes, but i’m solving more issues
except this.
Capture

i have no idea what is causing this to happen
(printed is a character array and its length (separate variables)