Flix - Procedural training and Movie making

[dlannan]

Ive started building Flix (working title) which is like a product I designed and with a team built many years ago for simulation systems as a procedural trainer. This tool was heavily influence by how movies are made, since much of the content was about showing the user how to do some operation.

One customer used this to do their safety training for their rail workers when working on tracks. How to setup the site, where people should be positioned and signs to put up etc. The tool allowed trainers to design all the procedures they needed with runtime interactive branching so that choices had ramifications. This was used in testing the workers.

The benefits are quite obvious and we ended up with a few customers in a couple of industries using this system. Rather than trainees staring at powerpoint slides, they had to interact with real world 3D characters and situations.

The tool also allowed the exporting of videos for selected branched paths. This way users could be given “correct” and “incorrect” videos on procedures.

Flix is going to be a tool similar to this, but much more oriented towards movie making and simple interactive scene operations. Allowing people to make movies or interactive executables that can be shared. The intent is to also provide capability on mobile, but will initially be focussed on desktop.

I have about 6-8 weeks to sort this out into a decent demonstrator so I want to log and record progress here. There is alot to do, and not much time to do it

My initial plans were to mimic the sort of interface I did previously and this is what this below was moving towards:

And this is kind of a traditional Unity3D/Defold/other engine panel layout. Which… for this purpose actually isnt that useful. So I have switched to use my own editor layout that I have previously built with my own online web based simulator.

The gltf loader isnt quite right yet This is runtime gltf loading.
Will probably update every few days.

[dlannan]

For my own reference, this is an example scene in my web simulator.

[dlannan]

Ok. Good progress today. Might be daily updates while things are ticking along well.

Entity placement in the scene is working. The gltf loader is a bit janky (fixing) and many bits need some cleanup and improvement. Hope to finish placement and rotation tonight/tomorrow then move onto the Acts timeline, Events and the Director.

[dlannan]

Minor update. Have been working on gltf import (runtime) to improve the above models. The current one only handles single materials and simple mesh structures, which is less than ideal. Adding cgltf to my back end (turns out this is what Defold uses internally! ) and will take a few more days I expect.

< Update >
Im now quite disappointed. It seems that cgltf is much less tolerant about gltf files than my own lua based loader (which I find a little odd). Im getting around 70% load failures on (admittedly) older gltf files but this isnt really acceptable since I want to use my older content anyway.

New plan. Rework my original loader while also having a look at another lua based solution here:

GitHub - leonardus/lua-gltf: glTF loader for Lua

At least this will be much quicker to resolve the above issues and expand to add features (cameras, lights etc) will be much simpler too.

I also found these gltf files to crash Defold because it happens in the cgltf_parse call rather than validate(which is where I had hoped it would happen). The white car in the last picture cant be viewed in Defold because of this crash. I cant really share it with Defold people either because its a commercial product from a partner.

A third option, should the lua one be too slow or more problematic, is to use assimplib which I have done many times before, but I didnt want a heavy multi-format loader. Will see…

[dlannan]

Wow. Ok that was quick. The lua-gltf seems to load in the same problem gltf’s fine. Will be switching to use it now. It doesnt support glb (will add that) and it will need some extra lua for anims, but at least I can move forward with the Director. The gltf lib will push its data into my mesh builder which is prob a few hours to do tomorrow. Nice. Progress at last.

As always software is a few steps forward… jump back… and then a few steps forward again.

[dlannan]

The lua-gltf lib is quite good. Supports all the gltf files I have which is great. Missing some features, but they are easy enough to add (lights and cameras). Quite happy.
Now the only/main problem is the multi-texture issue in Defold. So Im going through and baking some of the more complex models in Blender - little time consuming. Thus the cars are missing their textures except for one
Fairly happy with this. Onto the Director tomorrow.

[jhonny.goransson]

@dlannan

multi-texture issue in Defold.

Which issue is this?

[dlannan]

Having many materials set to the mesh. i think I had one model with 14 or more on a single mesh. Defold crashes trying to load that model. If I bake the model, its fine so theres a way around it.
I also had crashes on the cgltf parse on a few models as well (as mentioned above).

[Mathias_Westerdahl]

I don’t recall seeing such a ticket on github, did you add one?

[dlannan]

Ahh no. I dont think this is “general use” though? Since this is at runtime. I have a special mesh pool handler and so forth, which is then populated at runtime so Im not even sure the use case would be valid.

The gltf crash might be but I cannot supply a model since these are all commercial. And I kind of dont really have time to build something specific to replicate or find out issues - hence why I used the lua based loader (works ok with all my current models).

[Mathias_Westerdahl]

Still, I think for certain things, we should be able to deal with errors more gracefully, not take down the entire engine.

[jhonny.goransson]

Would you be able to provide a small repro with a different model so we can take a look? I’m not sure what exactly the issue is but I’d be glad to take a look. Could be a cube or something simple

[dlannan]

The multi texture one isnt so bad - thats more of a production thing (imho). 14 materials on a single mesh, isnt very smart anyway … again these are sourced commercially, so thats just how the partner builds them, I can manage that later. I can show the tree layout for one object in blender:

These are all Principled BSDFs with all but one having only base color. The main standard one having a base color texture. Oh and this one has a few more than 14

What probably is of interest to you, is the cgltf crash one (both in Defold and when I used cgltf_parse on the same model at runtime). It does close Defold and I think there are messages too. I’ll raise a bug for this if I get time tomorrow. Again, I dont really have time to track it down and make a replication model.

Note: The models that crash cgltf are not the same as the multi material ones that have issues. They are varied, and usually with mesh hierarchies too.

[dlannan]

Ok. Some more fun. Im running out of resources alot (because I need to build a mesh hierarchy on the fly at runtime) and there are many forum posts regarding this where its explained that instanced factory go’s share their attached components (like mesh, etc). This makes sense, since most of the time you want to instance an object, and have it as a duplicate of some mesh or sprite.

The problem with a runtime scene hierarchy, I need to prebuild a pool of objects with meshes (I have a meshpool folder with 100 x gos, mesh buffers, and 4 pbr texture files for each … yes its a little crazy. I have some script that helps me build it, so I can scale it up. However, I just tried a glb anim, and because it has over 100 parts (bones etc) it automatically uses all of my pool. Now. I could just make 1000 x resources (which is kinda nuts) or, more sensible, move the rendering out to another method, where I can easily do this (sharing assets and so on).

Sadly… this all means alot more work to get something more usable. But on the flipside, I can use a back end native extension renderer that should give me highly advanced rendering features. Going to spend a few days working on some design concepts and tests. This might be actually quite interesting.

[britzl]

@dlannan

I have a meshpool folder with 100 x gos, mesh buffers, and 4 pbr texture files for each

But what remains for this to be completely dynamic so that you don’t have to create all of these up-front? I’m trying to understand what we could do to simplify life for you.

[dlannan]

As per a number of other threads on the forums, there is no real way to create a new GameObject, that can then have a component (new also) added to it. My understanding (from reading the various threads) is that factories create instances, and thus data for them is shared (some is not, like position etc but components like meshes are shared).

I did try using factories, but as expected on instance 2… modifying the mesh on the instanced go crashed the running app. I get why this is, and I dont know how deep a problem that would be to add. My meshpool was a way around that, but with a tool that will potentially have hundreds of characters, and objects in the scene, I dont think that will scale well at all

And, again, this is probably not a common use case. I dont expect a game engine to necessarily support this kind of resource usage, thus I dont really want to request it
I’ll find a way around it - there are many different things I want to try over the next few days. For example:

• generate a collapsed mesh (at runtime) by composing in the transforms for more static objects.
• use something like ozz animation lib to handle anim, thus the bones are not needing to be managed in the go structure itself
• try a simple backend renderer that has some of these (like maybe sokol or something) that I can layer in as a render pass into Defold.
• use another engine entirely for the render pass embedded in the app which would be nuts, but Ive done it with other systems before (like C# WPF running Unigine within it ).

There some interesting things to look at. Will draw up some ideas, throw away any that just are non-sensical, and knock out some tests for the remaining ones to validate the design. Then move on

[d954mas]

Yes you can’t create go with components in runtime. You should use factories.

But you can create buffer resource for mesh in runtime.

API reference (resource)

local function water_create_default_native_buffer()
	return buffer.create(1, {
		{ name = hash("position"), type = buffer.VALUE_TYPE_FLOAT32, count = 3 },
		{ name = hash("normal"), type = buffer.VALUE_TYPE_FLOAT32, count = 3 },
		{ name = hash("texcoord0"), type = buffer.VALUE_TYPE_FLOAT32, count = 2 },
	})
end

---@return BufferResourceData
function M.water_create_new_buffer()
	M.water.idx = M.water.idx + 1
	local name = "/runtime_buffer_water_" .. M.water.idx .. ".bufferc"
	local new_buffer = resource.create_buffer(name, { buffer = water_create_default_native_buffer() })

	---@class BufferResourceData
	local buffer_resource = {}
	buffer_resource.name = name
	buffer_resource.buffer = new_buffer
	buffer_resource.free = false

	return buffer_resource
end

go.set(e.water_go.mesh, "vertices", e.water_buffer.buffer)

[dlannan]

Im not sure you have understood the problem. I have a geomextension lib (shared a couple yrs ago) that does this. But you cannot apply new meshes to instances from a factory. I’ve tried this many times over the years because components cant be added to the instance at runtime.

The buffers would be more useful (for this instance) if I could create a mesh component and attach it to a go instance. That would be a nice way to do this, but I also suspect its a much more complicated problem since resources are all precompiled at build time.

Sorry that was a little unclear. Specifically, if I instance a go with an attached mesh, that mesh buffer resource handle is shared with all instances. Thus, you cannot replace the mesh with a buffer. If you attempt to do so, on the second instance you try it on, it can crash (because of the reference held from the first one). This was mentioned above.

[dlannan]

For some references to the above. Here are some of the forum posts that relate to this. Some quite old, but this is kinda a known limitation for runtime resource usage:

Arbitrary scene graph hierarchy at runtime

Hi, I’ve just started using Defold, and it is a very nice engine, indeed :). I have a (million, but let’s start with this one) question: I want to create an arbitrary hierarchy of objects at runtime, and what I’ve done so far is creating a factory that spawns empty GameObjects that I then move around with “set_parent”. Is there any way of creating those empty GameObjects without having to create a factory with an empty .go file attached? It seems like a hack Thanks in advance!

Scriptable GameObject creation (DEF-2096)

As commented here, it will be useful to be able to create (empty) GameObjects on the fly to take advantage of the scene graph at runtime. Maybe something like factory.create_game_object([parent]) or similar (I don’t know the API well enough yet to know about naming or structure, but hopefully you get the idea :D)

Is it possible to create collision object at runtime?

I’m new to Defold, and try to create a brick game for learning purpose. I plan to bounce the ball if it hits the edges of window (left, right, top, bottom). So I placed four boxes at the edges, and bounce the ball if a hit message is posted. That works great, but now I want to adapt the window size, so the collision object cannot be placed at fixed positions, so I figure I could create those boxes inside collision object at runtime. But after a wide search on both Defold document, forum and …

Mesh Component

I’ve been trying to make a gameobject, that generates a vertex buffer on init. That works fine. The problem is, whenever I create another another instance of the go (either through factory or just plonking more in in the collection editor), the previous ones get the same vertices. I’m guessing it’s because all the meshes have the same .buffer file, and it’s that that is getting updated. Is it possible to create instances of a go with different vertex buffers at runtime somehow?

And there are other related ones.

[dlannan]

Also just for some more clarity. My geom extenstion (this one a little modified) does the following when assigning new buffers to a mesh.

------------------------------------------------------------------------------------------------------------

function geom:makeMesh( goname, indices, verts, uvs, normals )

	if(#indices <= 0) then 
		print("[Error] Invalid indices count.")
		return 
	end
	
	local res = go.get(goname, "vertices")
	local iverts = #indices

	local meshdata = {}
	-- positions are required (should assert or something)
	tinsert(meshdata, { name = hash("position"), type=buffer.VALUE_TYPE_FLOAT32, count = 3 } )
	if(normals) then tinsert(meshdata, { name = hash("normal"), type=buffer.VALUE_TYPE_FLOAT32, count = 3 } ) end
	if(uvs) then tinsert(meshdata, { name = hash("texcoord0"), type=buffer.VALUE_TYPE_FLOAT32, count = 2 } ) end
	
	local meshbuf = buffer.create(iverts, meshdata)

	geomextension.setbufferbytesfromtable( meshbuf, "position", indices, verts )
	if(normals) then geomextension.setbufferbytesfromtable( meshbuf, "normal", indices, normals ) end
	if(uvs) then geomextension.setbufferbytesfromtable( meshbuf, "texcoord0", indices, uvs ) end 
		
	-- set the buffer with the vertices on the mesh
	resource.set_buffer(res, meshbuf)
end

------------------------------------------------------------------------------------------------------------

So the problem isnt being able to generate mesh buffer data, but trying to replace the mesh component handle so that it has a new handle. This might be doable in C++ but I’m not certain of that.
If there are examples/methods that can do this, I would be very welcome to suggestions.

I think this specific comment shows the main problem:

Mesh Component

It is important to note that you update the resource in your example. If all meshes use that same resource, then it won’t work as you intended. Instead, you can create a new buffer resource dynamically, and assign it: local newBuff = buffer.create(...) -- as you do now local resource_name = "/my_cloned_buffer.bufferc" -- make this unique for each resource! local buffer_resource = reource.create_buffer(resource_name, { buffer = newBuff }) go.set(url, "vertices", buffer_resource) -- replace i…

local resource_name = "/my_cloned_buffer.bufferc" -- make this unique for each resource!
This means there needs to be a physical resource file for each res that needs to be attached/modified for each instance.

Note: The code includes some helper funcs that copy tables into buffer data for consumption by the buffer.