Defold message passing discussion

That has been done for some messages, like physics.ray_cast(). It might make sense to do this in more places too.

2 Likes

An important reason for message passing to exist for me is for building parent / child relationships with dynamically generated systems. It is the primary use I have for them.

I agree that more of the general system functions could be converted to simple function call style syntax like others already have. It seems like it’s already planned and just a matter of priorities and urgency of other work needing to be done first for it to happen.

I am a bit of an architecture nut, and I love this sort of topic, so i’ve accidentally written a lot :slight_smile: .

TL;DR: I’m still getting used to messaging, I don’t like that I need to write around it so often.

Messaging has its benefits, and i’m sure i’ll get used to it over time, but so far i’ve found it to be quite a hindrance.

For instance, I’m really missing the simplicity of function callbacks. It seems that, when I want something similar, I need to jump through extra hoops.

If I had a pure reference, I could just pass in a callback, but in the messaging world I need to have both objects agree on a form of communication, which has no guarantees of consistency between object pairs, for instance:

Lets say I want A to use a B, firing off some long running process, and then Fire off some completing behavior in A, Issues that could arise:

  1. Either URL might be incorrect, and this is Lua, so there will be no warning until runtime.
  2. Either message id might be incorrect, and this is Lua, so there will be no warning until runtime.
  3. The callback itself does not explicitly run in the context of object A or B
  • One implementation might have the code run in A, one might in B, the objects available will be different.
  1. In both the aforementioned cases you may need to ask A or B for something during the “callback”, in which case execution gets even more complicated:
  • If your behavior in the nested case is synchronous, you must now make it asynchronous - I can’t send a message and rely on things actually happening before my next line of code.
  1. Your function can’t be anonymous, it needs to be accessible through this response msg route. Which means that any other misbehaving object can run this again, there is less of a concept of fire and forget - these are permanent fixtures.
  2. If I do this again with objects C and D, I could structure it completely differently.

Of course, there are solutions to all of these. It’s up to you as a programmer to be consistent in your approaches in dynamic languages like lua, You can write global Lua modules that allow you to set up regular old callbacks in some cases, you can nil-out a function once a message has fired, but these are still hurdles to overcome and, for example if you’re not careful about your call context when working with GUI nodes you’ll end up with errors like: Node used in the wrong scene

I think loose coupling is good thing, a great thing in fact, but I have struggled to find (practically speaking) many locations where messaging is more sensible than just calling a function. Most of the time, when you are giving a specific object a specific message, it may as well just be a function call, it’s not like writing

msg.post("col:object_name#component", "foo")

is practically any less coupling than

object_name[component].foo()

There is still a dependency there, and not needing to nil check component in the messaging case really isn’t much of an advantage considering that you need to define a bunch of other stuff for it to work.

I should reiterate that, I do think there are some great arguments for messaging, but I don’t think it should be used in the common case, and I think Defold is built in a way which pushes architectures towards using it as the common case.

All that said, I’m still new to Defold :slight_smile: and i’m sure i’ll find more elegant ways to get the behaviors I want as I adjust my brain to the paradigm.

5 Likes

With the project I’m currently working on, I do actually use messaging a fair bit in the core area of scene (collection) switching, as it means I can fire off messages to the current ‘scene’ without needing to store/update any references for it since it will be changing so much.

That being said, that’s a very small portion of the project. For everything beyond that, which is the bulk of the project, I use a custom event handler which allows components to register/unregister themselves for certain common-firing events, as it just flows better for this purpose.

1 Like

In the second case you need to have a function defined at the receiving end, which is a major difference.

1 Like

Right, I completely agree, it’s a major difference.

What i’m saying is that, with messaging, you probably need to have “foo” defined as well in the case where you are directly messaging an object component like this.

It’s nice that you don’t depend on foo existing, but in the common case you usually want foo to exist.

Cases where you don’t care whether or not foo exists - those of course do occur. But in my experience, less frequently.

3 Likes

In this case I totally agree. The same with the play_animation message to sprite components (sprite.play_animation would make sense).

2 Likes

Please correct me where I’m wrong:

Defold messaging is a data and message ID sending/recieving mechanism, from one object to another, via an address in a messy looking HTML like format. Objects receiving messages must be scripted to respond in order to achieve anything in a Defold game. It’s important to understand that messaging in Defold is not scripting, method calling or the getting and setting of values as is currently thought of messaging in all other languages, frameworks and game engines.

Btw, why the limit on what can be passed in messages and factory parameters?

Do you mean that polluting global is better, than passing complex data directly?

It’s standard URL format, not HTML.

1 Like

Other than that, everything else correct?

Well, event/message handling is a well known pattern, and different engines implement it differently. For instance, the Avalanche engine uses messages in order to decouple the connections between caller/callee. It made the engine go faster because processing could be batched and different threads could process independently.

I think this contradicts the “all other languages, frameworks and game engines”

Also, not sure why you wouldn’t consider it scripting, when you’re in fact scripting the engine.

1 Like

Ok, you’ve found one exception, that proves the rule.

Just about everyone else thinks of messaging as that of calling functions to do something, on an object.

Since Defold has this kind of messaging, and its own kind of data sending messaging, there’s a problem of teaching/explaining/introducing how to conceive of using Defold.

That’s the real point of this thread. To find a more graceful way to do message sends of data, calling functions, and explaining/teaching how to think in Defold logic and game creation.

So, can you explain how to think in terms of conceiving how to architect with Defold?

I’m sure I can give you more examples later (on monday, my phone is lousy to google on)

But I suspect it’s not the notion of message passing/decoupling that you’re mostly concerned with, but rather how to program the relationship between caller/callee?

Like @sicher mentioned, we have added a few functions instead of messages. Perhaps you can give examples of other functions that are suitable to do this for?

The case where most people seem to have had the most questions is how to get data back from a message you sent. Am I correct in guessing this is where your biggest concern is? I think a good approach to this might be to look at what kind of scenario you have. Perhaps you have an example we can dig into?

3 Likes

Requirements for communication in Defold:

  1. Asynchronous lock-less communication between systems
  2. Everything should be able to address anything, at any time
  3. Relative addressing - to be able to reuse logic structures
  4. Unified across the engine
  5. Data should be tightly packed for known content, but also support arbitrary user data

Message passing and URLs is a solution to these requirements.

The goal was not to describe it in 3 sentences (that’s a strange rule), the goal was to solve problems.

Game objects and scripts

A game object instance is nothing but a spatial transform and the grouping of several components (which in turn are instances in other systems, e.g. sprites). The grouping itself constitutes its “class”. We regard game objects as “tiny systems”, or “units”, in the classic sense of object orientation which is not the same as modern object orientation (inheritance, OOP, etc.). This is because they benefit from being loosely coupled, as games are often unknown and it’s important to be able to make significant changes. Modern OOP creates tight couplings and work agains this, forcing you to change more of the source than what you intended to change. You are not forced to express your game logic through message passing though, you can still use lua and function calling etc. to any extent. If you think of your game in a MVC/MVP kind of way, this would correspond to doing the Model/Controller part in arbitrary lua and the view part using game objects. I would advice against this from the tight couplings it would introduce.

When to not use message passing

When the data being communicated is large in size, e.g. level data. Since message passing is asynchronous, the memory needs to be copied. It’s better to use the global state in these cases.

Problems and improvements

  • Auto-complete makes function calls way better than messages, for documentation purposes. This is why we have started to introduce functions which simply wraps messages. You can certainly do this yourself for any user-defined messages being posted, like @britzl suggested.
  • It’s annoying to lose the context when you have to wait for a response. Agreed! We could look into adding callbacks for message passing, so you could easily create a closure of the context for a response. Speaking of callbacks, we have some bugs in how they are invoked when you lose the lua context when dealing with coroutines etc. They of course need to be fixed too.

@deeeds What’s important is not how you think of messaging but if you solve the problems or not. I think you should ask yourself if Defold really is a good fit for you. If you still think it is, we are happy to explain more about why things work the way they do.

11 Likes

@deeeds I kindly suggest the following :

This is an irrational standard. Moreover the so-called message passing mechanisms of Javascript and Objective-C cannot be explained to the point of understanding in 3 sentences to “anyone”.

Once again, what everyone else does\thinks is an irrational standard for choosing an implementation.
Furthermore, the nomenclature of Defold makes the transition simple: We are presented with “Game Objects” and discreet “Components” or sub-objects, to which we send messages via a central governor\scheduler.


My thoughts on messaging in Defold(2 weekends into learning) :

A. Translate “message id” to function name automatically
As an alternative to a series of ever-growing “if” statements within on_message(), with readability\editability and performance implications with growing complexity.

msg.post("/a","dothis") > expected of destination > local function dothis(self,message,sender)

This applies equally to ‘input messages’/“actions”

B. Sending messages to oneself, eg. msg.post(".",“action”), is confusing
The first logical question is : “Why don’t I just call a local function/modify local state directly”.
One eventually realizes that a message is being sent and concludes at the ‘engine’ rather then an object; it is then necessary to memorize an exception to the general rule, and juggle two contexts. The two main cases are parenting and registering input handlers. Much as @baturinsky suggests, these should be simple function calls.

C. It was helpful for me to avoid using the term “input action” and replace it with “input message”. Superficially, in use, there is very little difference between “actions” and “messages”, it is therefore confusing to differentiate something that is actually a sub-species. I’ve included an out-take from my notes, which shows how I understand and relate the ideas :

12 Likes

Good points!

A) You are not the first to bring up the typical content of on_message, it tends to bother people. It doesn’t have any real implications on performance to test by if, but I agree with the complexity. I usually recommend others to put the functions into a table and key by the message ids, which gives a similar result as your suggestion, but of course with a bit more repetition and keeping the table up to date. The only small concern I would have with automatically mapping to functions would be that it is less explicit and more “magical”. And things like what would happen if you sent the message “init” or “update” to someone. :slight_smile:
B) The bad answer is that there was a time when there was little else but msg.post and that was the straight-forward way to implement the input. I think it’s safe to say that msg.post(".", “acquire_input_focus”) is one of the worst API points currently in the engine. Another candidate is “acquire_camera_focus” which is twice as confusing since “focus” is a common concept in optics. :slight_smile:
C) I would disagree here, from how different they are as concepts and how they are distributed (despite their superficial similarities, like the fields/signatures). I can’t really argue for the specific choice of “action” (could just as well be “event”), but I’m concerned it could be confusing to call them “message”. If so, the only logical thing to me would be to remove on_input and let them truly be a part of the messaging system.

8 Likes

Thanks for taking the time to reply @Ragnar_Svensson :

A. That’s a neat solution\compromise.
B. I appreciate your willingness to admit it. Because so much else about the design is thoughtful and clear, and the documentation is accurate, I’ll give you a pass :sunglasses:
C. Proper classification of such abstract mechanisms is tough(and important for clear thinking). For the moment, if nothing else, I would like to reiterate that I found the idea initially confounding.

6 Likes

@h3annawill @Pkeod @britzl @sicher @codecloak et al

Rapidly, confidently and simply familiarising a potential user with Defold’s concepts, architectures, paradigms and processes of communication (of which messaging is one) is the fastest (and best) way to empower them, and make them new users.

Otherwise, they’ll go pick one of the many other engines to use with more commonly discussed, rapidly assimilated ways of communicating and architecting functionality and game logic.

This is why I said “try” to get an explanation of messaging into 3 sentences. The better you can articulate it, in the shortest possible reader time, the more likely Defold (and King) are to gain new users, and empower them to be creative, productive and successful.

I’m assuming some of reasons King wants Defold public align with some of Ragnar’s original reasons for wanting Defold to be successful.

As Einstein famously said, “If you can’t explain it simply, you don’t understand it well enough.”

There’s a two more important aspects to this quote. If you can’t explain it simply (ever) it’s an inelegant design. If you don’t explain it simply, you won’t get a treasure trove of contented and empowered users.

I’ve picked an arbitrary length (3 sentences), but if you need 4 or 5, don’t feel constrained. The objective is to analyse whether it’s so inelegant it can’t be easily described and shared, or if it’s just that nobody has previously reduced its communication to the barest, most ideal minimum.

I think it’s a blend of both. Hence my comment, at the beginning, that I think messaging’s elegance should be worked on, at least a little. It’s a pain point to the use of Defold.

Every engine has pain points to entry. Godot has its peculiar fascinating with the word “Scene” for something that isn’t, and is designed to be heavily nested. It’s a composition, not a Scene, similar to how After Effects uses Composites to build up a finished product, neither of them actually have Scenes, as such. That’s its conceptual hump.

The hump of Defold is messaging, communicating and logic conception and creation, not least of which is because it’s very different to how most others think of these activities. See what @baturinsky is saying.

“I personally just think that using messages for what Defold is using them is a pointless complication. Why not just use functions and hooks for everything that is done with messages now?”

That’s going to be a common sentiment for new users coming to Defold. How do you get them from this state to the next, as fast and simply as possible, with them thanking Ragnar for his design choices?

This question points to some of what I’m on about: that messaging (in Defold) is peculiar, poorly explained and difficult for newcomers to conceive. If this was a small part of the engine, like a peculiar particle engine or oddly automated texture packing, it wouldn’t be near the issue it is. But it’s at the very core of doing (and thinking) in the Defold way.

"

2 Likes

Maybe you could uniquely explain the Defold way in a helpful way to people who think like you? Defold was designed in a specific way to solve specific problem. It’s unlikely the core will change, but the surrounding functionality can be improved, and better explained to be more approachable to newcomers. Some of the methodology is startling at first to people who have never seen it done this way before, and so takes extra knowledge to understand because it’s less common, because it attempts to solve problems in ways which are less common - such as making designs loosely coupled.

Contrary to your opinion I actually found the message passing familiar and appealing because of my background with working on Erlang systems, which use a similar message passing system.

I think that the main confusion with message passing is what to use it for and what not to use it for. If you don’t need it in your design don’t use it. Use required modules instead, but accept the potential for harder to debug problems down the road. If it helps your design then use it. Know that more general used functionality will probably be converted to “direct function calls” in the future.

1 Like