How does Defold handle multithreading?

Muffin · May 23, 2020, 7:05pm

I couldn’t find anything on how Defold handles C++ threads in the documentation. If someone could point me to the right location I’d appreciate it.

I’m familiar with the web/browser model where you can pass some work from otherwise single threaded javascript to a javascript web worker which is doing its own thing off the main thread and can then pass things back. The main concept here being you write things like UI on main-thread and pass the rest on web worker if that is available.

But I’m quite new to the concept of Single threaded language passing instructions to a different multi-threaded language on anything but a “theoretical level” .

TLDR;
I’m looking for:

which tasks are relayed from Lua to be handled on the C++ side
does C++ side already use something like “thread detection” and spreads things that can be on different thread to available threads
simplified overview but for Defold tech architecture
If it is already using something like my “dream code for multi-threading” where you write asynchronous code with something like Lua coroutines and it then messages the C++ which spreads concurrent tasks on available threads

Pkeod · May 23, 2020, 7:35pm

If you write a native extension to do something you can have threads to do work. If you do need to do things which would block normal gameplay you should try to do it in NE threads, on some platforms like HTML5 this won’t help though because everything is forced single thread due to security requirements.

Otherwise Lua coroutines are not threads, there is nothing special done with that as far as I know. Would be cool if it was possible! Like ways to write async Lua containers that are run on their own actual threads so as to not block the main thread and can pass data as needed.

Some parts may be threaded but currently others like built in audio are not. You should generally assume things done in Lua are not threaded.

I think http.request() calls are threaded. Some resource loaded may be threaded but even with async set it can still lag your game, even audio will pause/glitch in some cases since it’s not threaded yet.

Of course the source is available now so you can read it yourself to see how it actually all works.

Something like the “simplified overview” to compliment https://defold.com/manuals/application-lifecycle/ would be cool to see.

github.com

defold/defold/blob/869a87718757bdb7bfd6ed078047fa7bfb451de9/engine/dlib/src/dlib/thread.h

// Copyright 2020 The Defold Foundation
// Licensed under the Defold License version 1.0 (the "License"); you may not use
// this file except in compliance with the License.
// 
// You may obtain a copy of the License, together with FAQs at
// https://www.defold.com/license
// 
// Unless required by applicable law or agreed to in writing, software distributed
// under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
// CONDITIONS OF ANY KIND, either express or implied. See the License for the
// specific language governing permissions and limitations under the License.

#ifndef DM_THREAD_H
#define DM_THREAD_H

#include <stdint.h>

#if defined(__linux__) || defined(__MACH__) || defined(__EMSCRIPTEN__)
#include <pthread.h>
#include <limits.h>

This file has been truncated. show original

Mathias_Westerdahl · May 24, 2020, 9:14am

I actually was going to link to the documentation for the dmThread, only to find I still hadn’t added it to the SDK.
Added https://github.com/defold/defold/issues/4835 for this.

I like the idea of a more schematic overview of the engine, to help users/developers understand what’s going on under the hood.

Currently, there are only a few things that are threaded:
http requests, live update, some graphics requests (e.g. texture uploads),

Having the “UI on the main thread” mindset here is still valid, in our case it’s the Lua context that lives on the main thread, all callbacks need to be executed on the main thread.

We aim to do more threading though, it just hasn’t been too high on our road map.
I think first up on the list is to see if we can offload sound to a new thread, or perhaps resource loading.

dlannan · May 9, 2022, 7:19am

Sorry to resurrect an old thread, but I think my question fits in well here.

Im trying to use dmThread in an extension on Win10. And it appears the thread isnt really being created on another core - they appear to be getting creating on one core and thus just overloading the core.

Am I doing something stupid (highly likley) ?
All I am doing is dmThread new and then the thread runs for a short iteration (usually a loop of 300) and then it exits itself.

I keep track of how many Im running, and dont start any new threads unless I have free thread space to run more. I kinda expected each thread to be farmed out to a core. Should it ? Is there a need to use affinity on Win10 maybe?

Maybe Win10 threading isnt supported with dmThread?
Any help would be great. Thanks.

Mathias_Westerdahl · May 9, 2022, 7:22am

We’ll, I’m not saying you’re not right, but I just don’t see how it wouldn’t create a thread when we call CreateThread?

github.com

defold/defold/blob/dev/engine/dlib/src/dlib/thread.cpp#L170-L181

    
      
          Thread New(ThreadStart thread_start, uint32_t stack_size, void* arg, const char* name)
          {
              DWORD thread_id;
              HANDLE thread = CreateThread(NULL, stack_size,
                                           (LPTHREAD_START_ROUTINE) thread_start,
                                           arg, 0, &thread_id);
              assert(thread);
          
          
    SetThreadName((Thread)thread, name);
          
          
    return thread;
          }

dlannan · May 9, 2022, 7:27am

Nope ignore me. Something tragic with my code I think. It seems the threads are exiting faster than they can be created I think. Sorry for the confusion. Uggh!

BazzlyDrews · July 2, 2024, 1:26pm

Now I’m resurrecting this [thread]…Forgive me if there is further clarity or documentation on this, I am continuing to search. I’m also a rather early learner. As of now, from my understanding, the way to use multithreading on a Defold project is through Native Extensions. This sparked my curiosity after reading a reddit thread about Rimworld and the overall lack of multithreading in that game (outside of a few cases), and the pain it would cause to go back and rewrite code to properly use multithreading in Rimworld–or that it could cause subsequent performance issues depending on what was threaded.

I’m also curious if Defold’s roadmap to allow C++ scripting (in place of Lua) would open the doors to multithreading projects without NE (I still do love Lua…). I guess I’m not acclimated enough with the engine yet though. I’m still in the “Let’s break everything and celebrate our destruction over a pint and a bonfire” phase of my game’s journey.

Potota · July 2, 2024, 1:18am

It’s not C++ scripting (not sure how that would even work…), it’s just adding a lot more functions to the existing C API so that devs can make a whole game in a native extension.

BazzlyDrews · July 2, 2024, 1:22am

Thank you for clarifying!

selimanac · July 2, 2024, 5:45am

There is already support for this (not in place of Lua and never will be). It’s the same thing. NE is just a term for a simplified build system, nothing more.
You can develop almost a full game using the C/C++ API, and its functionality grows every week. You can use threads with dmThread and dmMutex. If you are planning to develop a game like Rimworld and you are capable of doing so, then this shouldn’t be a problem for you. Go ahead and use threads.

dlannan · July 2, 2024, 12:35pm

Not sure what you are referring to with C++ scripting

Multithread wise, there are actually a tun of options (most I have used in some way or another). Heres a list of a few ways:

As mentioned by others NE and the dmThreads API is an excellent way to do threads.
If you are on desktop, you can actually use luajit + ffi and call native desktop threading directly (have done this on Win and Linux a number of times)
There is also co-routines, while not multi-threading it does allow context separation and some benefits in then being able to expand to a full MT later if needed.
There are multithread luajit libs that you can put in an NE and use (github is your friend here).

The beauty of all this, is you are kind of the driver here, and you can find a way that best suits your own development processes. Speaking of processes, that is also another way to divvy up execution (and my preferred way personally).
Example libs like luaproc (very simple multi-proc) are great, or you can do things like fork, spawn and such in an NE or ffi. Lots of options… you choose

BazzlyDrews · July 2, 2024, 1:38pm

Thanks for your reply and your previous posts too. Helps me a lot. And wrong choice of words on my part. One of the recent Defold team posts included info about C++ (and C#) , and making the engine more accessible to devs who don’t have interest in using Lua. I should know better than to word it the way I did.

Thanks again!

dlannan · July 3, 2024, 12:30am

No probs … Was wondering if you were referring to some of the C++ type scripts Ive used in the past (shudders :)) … they can be rather interesting to use.

A little heads up about the engine itself. You can pretty much use it already without needing to use much lua at all (just need it to bind C++ or NE). The engine is (not the editor) pretty much C++ as well, and its open source (read license) so you can definitely just build in C++ if you really want.

You can do some weird things (have done some) where the framework is all you use and you bolt in the C++ libs you want

BazzlyDrews · July 3, 2024, 1:22am

Was actually reading the build READMEs when I saw your reply. I like Lua and how it’s integrated. The Defold team does a great job with documentation and it’s impressive how much work they get done with their team size–seems to be a very cohesive team. Enjoying this learning process. Drawn to Defold by its middle ground of engine and framework and what appears to be brilliant architecture and design choices. It’s so easy to use C or C++. And it probably would be a good exercise to try weird things out with the engine, myself.

I’m much more of an artist than a programmer, so trying to dedicate a ton of time to understanding best practices and training my brain to “think” more like a programmer. I like Mike Acton’s talks. Have embraced feeling like an idiot. These forums are gold and this community feels professional. Thanks for your feedback, truly.