Advanced rendering techniques - any ideas?

[morgerion]

This is about how to implement a pipeline for advanced rendering in Defold.
And there are a number of questions in this regard.

First of all, of course, support for GL_RGBA32F and GL_RGBA32UI formats is needed for render target.
GL_RGBA32F is needed for HDR, for cluster lighting, as well as for particles based on vertex texture fetch and for some GPGPU purposes.
GL_RGBA32UI is needed for MRT analog and may be more profitable than GL_RGBA32F in some cases due to absence of bits loss.
As far as I know, work is being done in this direction, and it’s very very cool!

In addition, mesh management is critical. For example, the ability to immediately enable/disable meshes for rendering within a single frame.
This is necessary when there are multiple render passes involving different meshes. This is best illustrated by rendering a reflection map. The reflection map (pass-1) should contain the entire world around the camera, while the final screen (pass-2) contains only those meshes that are caught in the viewport. Turning off part of the surrounding world for rendering in pass-2 must happen in a single frame.
I would like to know if there are any solutions already, or if a request for an appropriate feature needs to be made?

[dlannan]

Hi @morgerion, can you provide a specific use case?

You technically do not need these formats, since in the shaders you can cast to floats or unsigned ints and still achieve the necessary results. You can definitely implement a shader HDR and other features this way (I built a cloud/sky shader this way) - the glTexImage calls that use these setups are really just preparing the pointer for use on the GL API side, which would only effect a handful of operations.

Mesh management is also available (technically) via predicates - although overall the general scene management for 3D is something that is needing some improvements. This doesnt mean it isnt possible to achieve these capabilities, it just means it takes some more effort to do so.

I think its important to point out here that Defold is still a 2D game system (as developed). It just so happens, it has access to 3D and thus 3D support has been provided - it has not been explicitly developed around a 3D game engine.

But there is nothing stopping you from adding features to SDK, or developing native extensions which could apply almost any OpenGL feature you want.

Note: When I say OpenGL, its not all OpenGL… OpenGLES for mobile and web, and Vulkan I believe is now the OSX api. Implementing a single OpenGL supported capability may be complicated by the need for multiple API support.

[morgerion]

1. The features I’m talking about have been supported in OpenGLES3 and GL3.3 for almost 10 years. These are not specific features, in that time they have already become a cross-platform classic.

2. The existing render-target format has only 8 bits per channel. This is not enough to store the float. To store colour values in HDR, this is not enough. It is not enough to store pixel position data in world space (XYZ as RGB). It’s not enough to encode multiple values in one channel - for example, 32 bits of a normal render-target will not be enough to store (1) albedo (2) material (3) normal (4) distance. Because albedo is RGB components, normal is XYZ components, material is SSMR components. And that’s just the beginning, the list of what you need 128-bit formats for is very long.

3. As for the management of meshes. I have landscape chunks with which to implement HLOD. I need to immediately turn on some chunks and turn off others within the same frame. Predicates? I need a personal predicate for each mesh, e.g. “land_x23_y18” or “chunk_x81_y75_grass”. There will be tens of thousands of such predicates. because we have to list all values in “land_x***_y***”. This is absolutely a bad idea.

But even without such volumes, rendering on mobile requires good optimization. So the ability to quickly control the inclusion and exclusion of meshes is needed like air.
Predicates are a good idea, but they are ineffective when it comes to large number of similar meshes. For example, my game is a simple casual city-builder. I have 100 houses in a 3D city. And I need to be able to include some houses for rendering shadows within one frame, and include the rest for rendering the final screen.

[morgerion]

Defold can be a 2D engine and I am not encroaching on that sacred right.
However, some of the features it provides allow you to use it as a basis for creating some pretty advanced rendering. Some, however, are still lacking. But that doesn’t mean that those people who want to make something beautiful on Defold should now go to the other camp.
Defold has its own charms after all, I wouldn’t want to part with it.

[dlannan]

Yes, I understand what you are saying, however you can do these things with even 32bit rgba texture buffers (as they were done in the old days). You could also do this by adding a NE that builds exactly the buffers you want and create the specialized renderer you want.
Its important to remember Defold is not a modern 3D rendering engine (nor is it likely to be like Unity3D or Unreal any time soon) that is not its target market.

If the features you want, are features you think should be included in Defold, then I would start talking/working with some of the Defold devs to make suggestions how that could be implemented (or even submit a PR) - but if you need it “now” you will need use the framework as it exists. Either via an NE or ffi if thats available.

If you want to do terrain lodding, then thats a very specific use case, and again, Im not sure what your expectation with a feature like this would be? how should/would it work?
I would recommend again talking with the devs, looking at SDK and submitting ideas that you’d like to see, or implement again in a native extension.

The benefit Defold has over Unity3D and Unreal is you can jump into the source, suggest changes or use the extension system to make your own.

Just a note: Cant you have two materials (with two diff predicates) with one material for disabled, and one for enabled, and then just set materials to enable/disable meshes? I havent tried this specifically, but Im fairly sure that should work fine?

API reference (model)

[dlannan]

I should note too. That much of the time when I needed something specific to a sim or a non-Defold feature, I would generally build a native extension or make an ffi lib. Im generally working with desktop so ffi is often a simple choice for me, but Ive made quite a few native extensions and was very impressed with how easy it is to extend Defold however I need.

Actually there was a disucussion elsewhere (I’ll find the post) about using another renderer with Defold. Maybe thats even a better option? Utilize Defold for the “main dev” and submit to a renderer that suits your needs better?

Defold + Raylib custom renderer integration

As many of you know already I’ve participated in the Defold + Yandex Games Game Jam. My entry is a 3D racing game As a challenge to myself and because it’s fun I’ve decided to implement something that itched my brain for a while - integrate a custom renderer inside Defold as a native extension. It totally could be useful in real world projects for various reasons: Port C++ games to Defold Implement a high fidelity rendering pipeline Add a few graphical eye-candies as a part of a regular Def…

Interestingly, I think you could use Unity3D or Unreal as a renderer for Defold … thatd be a rather cool little side-project

[morgerion]

I could even write my own renderer. I’ve written them many times, from VESA to DX12. What’s more, i could even write my own engine. Sorry i’m getting carried away.

Using a Unity/UE renderer in Defold is the worst of ideas.
The rendering pipeline itself in these engines is nothing interesting (it’s the same as everything else) except for UE5 for Lumen. In any case, the rendering pipeline itself doesn’t provide features or a pretty picture - it’s a credit to the infrastructure around rendering and the people who know how to use it professionally.

However, we’ve strayed quite far from the main topic.
Discussing existing (or easily implemented) features and writing your own renderer are very different topics.

[dlannan]

Oh no. I didnt mean quite that. If you look at the Raylib implementation its purely about leveraging the 3D rendering facilities that Raylib has, while still using Defold as the main system. @britzl 's native extension of imgui is very similar to this - it has its own OpenGL rendering calls within the extension and runs all the “imgui rendering” outside of Defold. As such, you get a nice gui with lots of features but can still use it completely from the Defold infrastructure.

I’ll have to make a sample one or something at some stage - there have been a number of discussions/questions with similar requests of modern features within the Defold rendering pipeline.
In the sim business it was pretty common practice to use different technologies for different outputs/inputs (like Unity3D or Unreal purely as a rendering output). Did this for Hololens as well.

As for the main topic - I think its actually pretty pertinent Advanced rendering techniques will probably require advanced rendering technique features - or something hand rolled

Did you have a look to see if you can use predicates as I mentioned. I dont have time to make a sample at the moment (maybe on the weekend).

– edit – Btw, I think there was some work done on glTexImage in imgui extension, that kinda shows how you could similarly make your own Image buffers etc. I think the Defold team is adding the float buffer already, so that may be moot. However, if you need to do specific operations then it would be a way forward.

[morgerion]

To be more specific, I delved into Defold’s sources.

So, as far as render target formats are concerned, this is the easiest thing to do. We need to prescribe these formats in several places. And everything will work by itself.

As for MRT, things are more complicated. We need to provide options other than GL_COLOR_ATTACHMENT0 in OpenGLNewRenderTarget() (see defold-dev\engine\graphics\src\opengl\graphics_opengl.cpp)
So I have no questions about the complexity of MRT implementation.

[dlannan]

I havent really bothered doing many changes on SDK (easier to make an ffi/NE ). But that seems like a good way forward. Probably need to keep an eye on impacts to cross platform interfaces and materials (ie binding shaders to the render targets etc).

Make a PR and send!

[Mathias_Westerdahl]

Good discussions here.
@morgerion if it’s feature you think we should have, and we don’t have a ticket for it, please go ahead and create one on github!

Also, I think it’s good to think about the counterpart of Vulkan, as we’re likely to abandon the OpenGL backend on macOS/iPhoneOS this year, in favor of the Metal backend (via MoltenVK).

[morgerion]

Regarding predicates, let me try to clarify my task.

Here is a picture (3D enviroment, top view):

For rendering i need to do some grouping of landscape LODs (meshes):

1. I need to enable those meshes that hit the player’s camera and disable those that don’t. Alternatively i could do it with msg.pos(GO_url, “enable”/“disable”), but this message will only come to the meshes in the next frame, so visually there may be blinking objects on the screen - player’s camera is already turned, but objects are not turned on yet.
2. To render the shadows, i need to enable all the meshes on the landscape and render them into the depth map. Then i need to turn off the meshes outside the player’s camera and render the final frame for the player’s camera. And all of this has to be done within a single frame - no delayed messages.

My question is - how can i do this?
If go.set() executes instantly, i could have two materials - “unvisible” and “visible” - and set them to the right meshes immediately via go.set("#mesh", “material”, material).
Would that work?

[morgerion]

I know you’re already working on the float render target except my small wish would be to add the RGBA32UI format (if it can be done cross-platform).

Vulcan is very good, but i’m willing to write in OpenGL as long as it’s supported.

To make a Feature Request i need to understand what is already there, what is not, and how much i am asking for.
That’s why i started this thread

[Mathias_Westerdahl]

@morgerion

Vulcan is very good, but i’m willing to write in OpenGL as long as it’s supported.

Well, since OpenGL is deprecated by Apple, it won’t be available forever.

[Mathias_Westerdahl]

how can I do this?

I’m not sure you currently can.
We don’t have a way to know if objects are within the camera frustum. We have a ticket for this. I believe I have to add your use case to the list of use cases there, as you say sometimes you want this within the same frame.

To control what is being rendered, I think we should be able to use the material tags. Perhaps we can use special engine render tags, like “frustum_inside” or “frustum_ignore”. Or perhaps some properties to the render.draw() call.
E.g. render.draw(pred, {frustum = ...})

To force set a material, you should already be able to do so via the render.enable_material(). I think this should work in your use case.

[morgerion]

@Mathias_Westerdahl

We don’t have a way to know if objects are within the camera frustum.

Oh, no, i don’t need that. I’m perfectly capable of determining visibility on my own.

@Mathias_Westerdahl

To force set a material, you should already be able to do so via the render.enable_material(). I think this should work in your use case.

I meant manually resetting the material to specific meshes that i define myself. The main thing is to make it happen instantly.

[Mathias_Westerdahl]

Oh, ok. Yes, go.set() should be synchronous (i.e. not passing a message).

[morgerion]

Holy news! God bless you!

[Ivan_Lytkin]

But please notice, you will have to set up all shader’s uniforms again after you changed material.

[morgerion]

Because they are set to the “default” values specified in the material itself?
And the textures, i take it, are also reset?
It’s frustrating, but at least it’s a solution.
After all, all i need from meshes is geometry and textures, the rest can be done in deferred shading.

Of course, it would be ideal if the mesh had a method like gui.set_enabled(). Something like go.set_enabled() which does the same thing as msg.post(go_url, “enable”/“disable”) but does it instantly on the current frame.
I’ll make a Feature Request about it.