# Moving multithreading implementation within defold

**URL:** <https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094>\
**Category:** Questions\
**Created:** [February 26, 2024, 11:34am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094 "2024-02-26T11:34:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 26, 2024, 11:34am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/1 "2024-02-26T11:34:52Z")

</div>

i’ve implemented a _really_ hacky way to do multithreading that uses shell piping, but it requires the user to have lua installed on their system to use (i don’t want to force users to do this), and i feel that running defold with the command line would take too much time compared to a raw lua install for the implementation to be worth it (right now with running 5.4 it takes about ~.005 seconds **just to initialize** a thread) (any longer and it stops being worth it since the max tickrate would be vastly lower).

what would the best approach to do here that would ensure the user experience is smooth and the code fast?

---

<div class="post-metadata">

**Author:** ![britzl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/britzl/32/23_2.png) [@britzl](https://forum.defold.com/u/britzl)\
**Post date:** [February 26, 2024, 6:47pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/2 "2024-02-26T18:47:52Z")

</div>

Isn’t a native extension using dmThread better?

> [@Experimental Threading extension](https://forum.defold.com/t/experimental-threading-extension/66558):
>
> Multi Threading I have been playing with dmThread and I managed to get it working. See the attached project. I have tested on Windows only. [defold threads.zip](https://defold-forum-bucket.nbg1.your-objectstorage.com/original/3X/2/7/27698ec2087b6c9ad1031c5d6cc589d53e5bee9b.zip) (16.0 KB) Updated 1 [93FoB6sJXm] [Threading example.zip](https://defold-forum-bucket.nbg1.your-objectstorage.com/original/3X/8/f/8f08b7ab9ae612c52b71e41e766541f855dd85f0.zip) (22.9 KB). Update 2 Added support for modules. [Threading example 2.zip](https://defold-forum-bucket.nbg1.your-objectstorage.com/original/3X/1/d/1db2e5b27f0af59bcb3296f4bb8f5945df9efd95.zip) (5.6 KB) You can add the extension to your project through bitbucket. [https://bitbucket.org/ronaal/defoldthreads/get/master.zip](https://bitbucket.org/ronaal/defoldthreads/get/master.zip) Running modules is as simple as local fname = "main/modulename.lua" thread…

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 26, 2024, 8:07pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/3 "2024-02-26T20:07:20Z")

</div>

well, i recall looking at that some time ago, so my memory is fuzzy. but i recall it having datatype limitations that were a bit too much for me, whereas i can feasibly send most useful datatypes with what i have (eg, tables, mostly) between the main thread and side thread on creation and completion

also that said that it was only tested to work on windows, but i’d need it to work on linux, too.  
that and the last post was a couple of years ago from what i can tell so i’m not sure if it’s still alive

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 26, 2024, 10:58pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/4 "2024-02-26T22:58:46Z")

</div>

also i don’t know C++ or the Lua API, so i can’t implement it myself, really…

---

<div class="post-metadata">

**Author:** ![Chung\_Xa](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/chung_xa/32/34470_2.png) [@Chung\_Xa](https://forum.defold.com/u/Chung_Xa)\
**Post date:** [February 27, 2024, 3:32pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/5 "2024-02-27T15:32:31Z")

</div>

Can you share some of yours so everyone can get its advantages?

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 27, 2024, 9:23pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/6 "2024-02-27T21:23:15Z")

</div>

> Can you share some of yours so everyone can get its advantages?

well, i’m not sure the code is ready for public use yet, it’s still not the best implementation (far from it, even), and there’s definitely a _lot_ of caveats to it.

it’s very rudimentery, but it does work.

### caveats:

- requires io access
- **writes to the disk a lot**
- barebones
- data transfers are sent in code
- controlling execution environment is unrefined

### good things:

- multiplatform
- runs with pure lua
- “werks, dunnit?”

```lua
luathread={}
luathread.threads={}
luathread.threadlocation="threads" --relative path usually.

luathread.newthread=function(threadcode)
local n
for i=1,#luathread.threads+1 do if not luathread.threads[i] then n=i break end end
local tf,e
lf,e=io.open(luathread.threadlocation.."/"..n,"w")
if e then error("error while opening thead file: "..e) end
tf:write("_=function() "..threadcode.." end io.write(table.dump(_())) ")
tf:close()
tf=io.popen("lua -l dumplib "..luathread.threadlocation.."/"..n) --dumplib contains a function which dumps a table into a string which we can load back later (table.dump).
luathread.threads[n]={
n=n,
threadfile=tf,
threadresult=nil,
location=luathread.threadlocation.."/"..n,
close=luathread.closethread,
get=luathread.getthread,
}
return luathread.threads[n]
end

--so, fun fact, we don't *need* to close these files. it slows it down a ton. instead, set to nil, and the garbage collector does it for you when it needs to. much faster!

luathread.getthread=function(thread)
if thread.threadresult then
	return table.unpack(thread.threadresult)
else
	thread.threadresult=thread.threadfile:read("*all")
	thread.threadfile=nil
	thread.threadresult="return {"..thread.threadresult.."}"
	thread.threadresult=load(thread.threadresult)()
	return table.unpack(thread.threadresult)
end
end

luathread.closethread=function(thread)
	luathread.threads[thread.n]=nil
	thread=nil
end

return luathread

```

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 28, 2024, 12:01am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/7 "2024-02-28T00:01:45Z")

</div>

@britzl does dmThread allow for shared state threading? if it does i might look into making an actual extension using it.

---

<div class="post-metadata">

**Author:** ![Chung\_Xa](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/chung_xa/32/34470_2.png) [@Chung\_Xa](https://forum.defold.com/u/Chung_Xa)\
**Post date:** [February 28, 2024, 2:10am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/8 "2024-02-28T02:10:17Z")

</div>

I get the idea of it, much like we run a command line and get the result via a file later. Not sure it will work on HTML5 with that pure lua code.

---

<div class="post-metadata">

**Author:** ![Mathias\_Westerdahl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/mathias_westerdahl/32/228_2.png) [@Mathias\_Westerdahl](https://forum.defold.com/u/Mathias_Westerdahl)\
**Post date:** [February 28, 2024, 8:08am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/9 "2024-02-28T08:08:49Z")

</div>

> [@CrazyAmphibian](#):
>
> dmThread

dmThread is using system/OS threads, which allows you to do whatever you’d like.

Note that Lua doesn’t support multithreading, so I believe the recommendation is to create separate lua states, and create some communication between the threads e.g. via some serialization.

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 28, 2024, 8:13am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/10 "2024-02-28T08:13:55Z")

</div>

so you’re saying i don’t _ **have** _ to make separate states?

---

<div class="post-metadata">

**Author:** ![Mathias\_Westerdahl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/mathias_westerdahl/32/228_2.png) [@Mathias\_Westerdahl](https://forum.defold.com/u/Mathias_Westerdahl)\
**Post date:** [February 28, 2024, 8:16am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/11 "2024-02-28T08:16:34Z")

</div>

I’m saying you probably have to?

Lua states aren’t thread safe, so to use the same context, you’d have to synchronize _all_ threads, effectively making it single threaded.

So, to do any meaningful multithreading, you’d have to use different Lua states, and you have to figure out some communication between them.

---

<div class="post-metadata">

**Author:** ![Mathias\_Westerdahl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/mathias_westerdahl/32/228_2.png) [@Mathias\_Westerdahl](https://forum.defold.com/u/Mathias_Westerdahl)\
**Post date:** [February 28, 2024, 8:21am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/12 "2024-02-28T08:21:38Z")

</div>

The question is, what is this needed for?  
I think using Lua throughout might be overkill.

If I were to create a thread to calculate a lot of things, I’d create a Lua api, to collect the data on the C side, then pass this C data to my threads.

Then, once the output data has been calculated, I’d push the result from the thread to my code onto e.g. an array (using a mutex), that my Update() function on the main thread can consume, and call back into Lua with.

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 28, 2024, 8:30am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/13 "2024-02-28T08:30:00Z")

</div>

> The question is, what is this needed for?

I hate when programs put restrictions in place things they’d be capable of otherwise without.  
Simply put, allowing direct access of variables to threads makes my job a lot easier since it means i get to control with far more scrutiny what is going on instead of being forced to use, (and consider how i have to change my approach when using) an API to do everything.

> using Lua throughout might be overkill.

i’m most comfortable in Lua, so this would allow me to use lua for anything thread related, which allows me to run more complex code as a result

---

<div class="post-metadata">

**Author:** ![britzl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/britzl/32/23_2.png) [@britzl](https://forum.defold.com/u/britzl)\
**Post date:** [February 28, 2024, 4:38pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/14 "2024-02-28T16:38:01Z")

</div>

> [@CrazyAmphibian](#):
>
> Simply put, allowing direct access of variables to threads makes my job a lot easier since it means i get to control with far more scrutiny what is going on instead of being forced to use

You didn’t really answer the question 🙂 There are plenty of Defold games released without need for threads. What do you need to do that you are not able to do now?

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 28, 2024, 8:03pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/15 "2024-02-28T20:03:01Z")

</div>

> What do you need to do that you are not able to do now?

i feel like that question (in the context of multithreading) is pretty self-evident.  
why would anyone want more threads?  
to perform more computations in the same time.

there’s nothing i _specifically_ need them for right _now_, but having that feature can open a lot a doors down the line.

---

<div class="post-metadata">

**Author:** ![britzl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/britzl/32/23_2.png) [@britzl](https://forum.defold.com/u/britzl)\
**Post date:** [February 28, 2024, 8:20pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/16 "2024-02-28T20:20:44Z")

</div>

> [@CrazyAmphibian](#):
>
> why would anyone want more threads?  
> to perform more computations in the same time.

I’m not against multi-threading. I was just wondering what kind of problem you were trying to solve in your game, but I now realise that you were experimenting with this for use in some future scenario 👍

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 28, 2024, 11:44pm UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/17 "2024-02-28T23:44:32Z")

</div>

So, i’ve been cooking, but the kitchen is on fire. (I have no idea what i’m doing ok?), and i need help extinguishing it.  
(most of this code was written in collaboration with chatGPT, which i am not proud of.)  
here’s the error from the crash dump  
`The thread tried to read from or write to a virtual address for which it does not have the appropriate access`

here’s a snippet of what i’m doing:  
the function to create a tread (exposed to the lua enviroment for execution in scripts)

```lua
int newthread(lua_State* L) //creates a thread
{
	if(lua_gettop(L)<1 || ( !(lua_iscfunction(L,1) || lua_isfunction(L,1) ) )){ //we need a function as the argument
		return luaL_error(L,"Expected function as argument");
	}
	int funcref = luaL_ref(L, LUA_REGISTRYINDEX); //the function that was passed to us - get the ref to it for later.
	lua_getfenv(L, -1); //get _ENV
	int envref = luaL_ref(L, LUA_REGISTRYINDEX); //and store it.
	
	context ctx; //make a new context	
	
	dmThread::Thread thread = dmThread::New(execthread, 0x80000, (void*)&ctx, "defthread");
	
	
	ctx.status=THREAD_RUNNING; //status indicators.
	ctx.Lstate=L; //store the lua state for later use, too.
	ctx.contextaddr=&ctx; //why not?
	ctx.threadthread=thread; //and store the thread for passing to other functions
	ctx.enviromentref=envref;
	ctx.functionref=funcref;
	
	context**ud = (context**)lua_newuserdata(L, sizeof(context*)); //theifery from AI
	*ud = &ctx; //didn't originally include the &, but the compiler says jump, i say how high.
	return 1;
}

```

the context struct, which is also returned as userdata for passing to other thread functions.

```lua
struct context{
	int status;
	lua_State* Lstate; 
	void* contextaddr;
	dmThread::Thread threadthread;
	int enviromentref;
	int functionref;
};

```

and this is the C function where the error occurs, whose pourpose is to execute the function passed to newthread

```lua
static void execthread(void* _ctx){ //the thing that is executed in a thread.
	context* ctx = (context*)_ctx;
	lua_rawgeti(ctx->Lstate,LUA_REGISTRYINDEX,ctx->enviromentref); //ensure it runs in the same enviroment
	lua_setfenv(ctx->Lstate,-1);
	lua_pop(Lstate,1)
	lua_rawgeti(ctx->Lstate,LUA_REGISTRYINDEX,ctx->functionref); //get the func
	lua_call(ctx->Lstate,0,0); //we are calling that function now. 0 in 0 out to make my job a bit easier. why would you need arguments anyways? use a table in _ENV.
	luaL_unref(ctx->Lstate,LUA_REGISTRYINDEX,ctx->functionref);
	luaL_unref(ctx->Lstate,LUA_REGISTRYINDEX,ctx->enviromentref);
	ctx->status = THREAD_FINISHED;
}

```

---

<div class="post-metadata">

**Author:** ![Mathias\_Westerdahl](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/mathias_westerdahl/32/228_2.png) [@Mathias\_Westerdahl](https://forum.defold.com/u/Mathias_Westerdahl)\
**Post date:** [February 29, 2024, 8:25am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/18 "2024-02-29T08:25:41Z")

</div>

This is an advanced task, so I’m not surprised that the chat bot doesn’t give a solution to it.

For the problem, remember that you mustn’t use a Lua state which is owned by one thread, on another thread: Lua is not thread safe!

So, that’s why you’ll likely wish to create a _new_ lua state to use with your thread. Then the problem becomes 1) The new Lua state doesn’t have the same modules loaded, 2) the function that you pass (and its arguments), also need to be serialized from the main thread, and deserialized on the new thread.

While we’d like to have some sort of job system in the long run, I’m not sure how we’d solve it ourselves yet, so I can’t give more advise than that.

```
" there’s nothing i *specifically* need them for right *now* , but having that feature can open a lot a doors down the line."

```

While it might be a fun exercise, I’d personally find out _what_ I actually need it for, and then create a specific C extension (without the Lua thread) to solve my calculation problem.

Good luck!

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [February 29, 2024, 9:58am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/19 "2024-02-29T09:58:05Z")

</div>

well i was hoping i could brute force it to share memory (something i don’t have the C wizardry to even begin to know how to do)

i guess i did learn a _bit_ about C++ and the Lua API in the process…

> I’d personally find out _what_ I actually need it for

i’ve already thought of a hypothetical, and that’s processing game entity logic en masse, threads allowing for a massive number of, or a highly intricate system without bogging a system down

---

<div class="post-metadata">

**Author:** ![CrazyAmphibian](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/crazyamphibian/32/26089_2.png) [@CrazyAmphibian](https://forum.defold.com/u/CrazyAmphibian)\
**Post date:** [March 1, 2024, 8:59am UTC](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094/20 "2024-03-01T08:59:22Z")

</div>

so i’m doing a different approach where threads are separate states, but…  
you get to pass a table (or other serializable datatype, eg: not a file or userdata) though to the function as a parameter  
i’m putting a moderate attempt into figuring out what’s going on

good thing:  
the engine no longer crashes with an error upon creating a thread  
bad thing:  
the engine doesn’t give an error code when it crashes upon creating a thread.

edit update 3 hours later:  
it still soft crashes, but i’m solving more issues  
except this.

 ![Capture](https://forum-defold-uploads.b-cdn.net/original/3X/e/c/ec7d14079c0de6a8478a625d9addc2e8a5c6b87c.png)

i have no idea what is causing this to happen  
(printed is a character array and its length (separate variables)

[Next page](https://forum.defold.com/t/moving-multithreading-implementation-within-defold/76094.md?page=2)