Get graphics context from extension

Source Code
#1

I’m trying to write an extension to take screenshots. dmGraphics has all the functionality needed, but it’s not exposed to extensions as far as I can tell.

dmGraphics::g_adapter->m_GetContextCb() should be able to get the context, and that context can be passed to the various functions in dmGraphics::g_functions.

Of course the functions should be called via the dmGraphics::SomeFunc wrappers that call them form the function table of the current selected adapter, but:

a) There isn’t a function for GetContext
b) Not all of these are exposed in graphics.h

I have tried without success to cheese-it by copying relevant structures and function declarations into duplicate headers in the extension, but even with the correct headers in place the linking fails. I have tried to simplify this approach (in order to figure out what’s going on) by calling the Vulkan* functions directly but even then they cannot be linked. Here are the errors I get:

ld: warning: ignoring duplicate libraries: '-lMoltenVK', '-lgraphics_vulkan', '-lplatform_vulkan'
Undefined symbols for architecture arm64:
  "dmGraphics::VulkanGetWidth(void*)", referenced from:
      ScreenshotExtension::CallbackPostRender(ExtensionParams*) in libScreenshot_2.a[2](extension.cpp_0.o)
  "dmGraphics::VulkanGetHeight(void*)", referenced from:
      ScreenshotExtension::CallbackPostRender(ExtensionParams*) in libScreenshot_2.a[2](extension.cpp_0.o)
  "dmGraphics::VulkanReadPixels(void*, void*, unsigned int)", referenced from:
      ScreenshotExtension::CallbackPostRender(ExtensionParams*) in libScreenshot_2.a[2](extension.cpp_0.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
com.defold.extender.ExtenderException: java.io.IOException: clang++ -c -isystem /Users/bjorn/extender-production/platformsdk/MacOSX14.5.sdk/usr/include/c++/v1  -DDLIB_LOG_DOMAIN="SCREENSHOT" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_MACOS -DDM_PLATFORM_OSX -DGL_DO_NOT_WARN_IF_MULTI_GL_VERSION_HEADERS_INCLUDED  -O2 -g -stdlib=libc++ -mmacosx-version-min=10.13 -isysroot /Users/bjorn/extender-production/platformsdk/MacOSX14.5.sdk -nostdinc++ -fno-exceptions -fvisibility=hidden -Werror=format -arch arm64 -target arm64-apple-darwin19 -m64  -Iupload/_extension-screenshot/include/ -Ibuild/_extension-screenshot/ -Iupload/ -Iupload/defos/include/   -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//include -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//sdk/include -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//ext/include  upload/_extension-screenshot/src/extension.cpp -obuild/extension.cpp_0.o
clang++ -c -isystem /Users/bjorn/extender-production/platformsdk/MacOSX14.5.sdk/usr/include/c++/v1  -DDLIB_LOG_DOMAIN="SCREENSHOT" -DDDF_EXPOSE_DESCRIPTORS -DDM_PLATFORM_MACOS -DDM_PLATFORM_OSX -DGL_DO_NOT_WARN_IF_MULTI_GL_VERSION_HEADERS_INCLUDED  -O2 -g -stdlib=libc++ -mmacosx-version-min=10.13 -isysroot /Users/bjorn/extender-production/platformsdk/MacOSX14.5.sdk -nostdinc++ -fno-exceptions -fvisibility=hidden -Werror=format -arch arm64 -target arm64-apple-darwin19 -m64  -Iupload/_extension-screenshot/include/ -Ibuild/_extension-screenshot/ -Iupload/ -Iupload/defos/include/   -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//include -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//sdk/include -I/Users/bjorn/extender-production/sdk/d01194cf0fb576b516a1dca6af6f643e9e590051/defoldsdk//ext/include  upload/_extension-screenshot/src/fpng.cpp -obuild/fpng.cpp_1.o
#2

I think this is too hacky way, i.e. there is no guarantee that the API will be the same in the future.

Typically, adding a new API to the dmSDK is requested via GitHub and this is accomplished fairly quickly if the API is stable.

#3

I think this is too hacky way, i.e. there is no guarantee that the API will be the same in the future.

I 100% agree. I have never written an extension before and I’m trying to learn how it works and how it should work (if the right APIs are exposed). Without this knowledge I can’t really make a sensible proposal for what (if anything) needs to change. Seek first to understand and all that.

@Mathias_Westerdahl said this:

You can try using it first by creating a copy of that header file and using it in your extension as a proof of concept.
(source)

And I think it’s a sensible suggestion when an API doesn’t exist yet; first make it “work”, then make it “good”. Also, it provides a demonstrated use-case for exposing something, and provides an extension that can bridge the gap until such proposal is accepted and implemented.

So my methodology is this:

  1. Solve linker issue and get it linking directly to functions I know exist and should already be present in linked libraries.
  2. Manually make it work with both Vulkan and OpenGL - since their outputs are different (I think BGR vs. RGB and y-flipped or not are the two main differences). This is an important step before the next one, because in understanding the differences it feeds into the discussion about where problems should be solved (inside or outside those functions?).
  3. Propose/discuss what would need to be added and any consequences.

:white_check_mark: I’ve actually partially solved point 1. above since the original post, by calling VulkanGetContext directly and then passing that into dmGraphics::* and got a successfully built extension that works (but with some more work to do, in order to resolve buffer format differences between Vulkan and OpenGL). Although this “works” I say “partially solved” because I still don’t know why I can’t link against the Vulkan functions directly. My guess is something about how C++ mauls symbols that are’t declared extern, but my C++ and ELF (if that’s still relevant) knowledge is 20+ years out of date and very dusty, also it seems to link VulkanGetContext just fine and I don’t know why this function can be linked but the others can’t.

So, I hope I have defended my approach to your satisfaction… if you can see any better ways please suggest.

#4

The context is now clear, as my response was generalized. In my native extensions, I did that too (earlier!). Then I had to redo everything because I didn’t tell the team that I was using it.

What I would do, again from my experience with the engine: I would download the Defold source code and add my Lua C API function with the required functionality to it in any component (graphics? it’s the script_graphics.cpp file). Since you are in the context of the engine, you will be able to use any functions and quickly create the needed functionality. And you won’t waste time trying to understand how to pull the function into a native extension (most likely it will be hours of trying to guess how the compiler and linker wants it). When you’re done, you’ll also be able to figure out what functions you need and suggest in the request to add them.

I would do it this way.

It may seem super complicated to build Defold from source, but it’s actually quite easy if you patiently go through all the steps in the instructions.

#5

Ok - I don’t think that any new Lua functionality or API is needed here though. But I do suspect that the way dmGraphics:: works with a “current adapter” is something that is fairly new, because it doesn’t seem to be exposed even internally at this point. Migrating internal code to use this is probably an ongoing task, that’s only been made possible recently (I’m guessing). So the full benefit of this system (allowing multiple adapters and fallback and providing a cross-adapter API) won’t be seen until more of the internal and external things (like extensions) can actually use it.

I don’t think providing a specific API to Lua is as valuable (or maintainable) as providing a good API for extensions and other internal functionality. Having access to such functionality allows people to solve problems for themselves rather than bolting on yet another API to maintain to the more public-facing Lua side of things.

Maybe I misunderstood what you were saying… but I do agree that I have been reluctant to try to compile the whole engine. I thought about adding the required internal dmGraphics APIs to that, and then also calling them from an extension… but then I’d also need to create my own build server (I think)… it all sounds a bit much right now, when I can’t even figure out how a linker works :smiley:

Regarding the dmGraphics API - I do think there are some considerations before just exposing things - because you could say (for example) that:

  • dmGraphics::ReadPixels will give you the pixels in a consistent format, and do any adapter-specific conversions for you; OR you could say
  • dmGraphics::ReadPixels will give you the raw data from the adapter, and you can document what the differences are, and provide some Get functions to give extension authors the ability to do the conversions themselves.

I think I prefer the latter, as it keeps the internals as fast as possible, not doing unnecessary work, since you don’t know how the user (extension author) wants to use the data. But of course it depends on the philosophy of the API, and how you want to use it internally as well. Maybe the philosophy could be to provide raw data and also some conversion helpers… or alternate versions of the functions where you can pass in your desired output… in any case, my point is that it’s not just as simple as “expose X” because there may be some consequences to consider.

#6

To summarize, I suggested writing proof of concept of your extension inside the engine so you don’t spend time pulling APIs. Once you figure out what you need, then move the code into the extension. It seems to me that it would be faster that way. And you don’t need a build server to do that.

I have a question: is it definitely supposed to be a C/C++ API? Maybe it could be a new Lua API function like render.read_pixels(callback) that is called from the render script context and asynchronously returns raw pixels. Let me explain the reasons for this:

  • Render script builds a queue of commands, you can’t get in between and get an intermediate render state. Your extension that creates screenshots will always access the pixels of the final image the user sees (i.e. if you want to disable the gui predicate for the screenshot, the user will see that too).
  • Because of double/triple buffering, you get an image that was two/three frames ago. At least I have had this experience with the extension https://github.com/britzl/defold-screenshot
#7

Ahh, I understand… I thought you meant put something in the engine, then also call that from an extension… then you know what the extension needs. Which sounds a lot more complex. So instead you suggest build a new engine with the required functionality, then figure out how to split that out into an extension. Makes a lot of sense - I’ll look into building the engine itself soon.

Maybe, but there are a number of reasons I think that might be getting ahead… perhaps it could be a next step:

  • Implementing render.read_pixels would still require the same C++ API internally. The GraphicsAdapter in dmGraphics (and using that API in general) appears to be the “intended” way to do things internally, unless I’m missing some context here? No pun intended… but the context is also what’s missing internally to be able to do that, you need it to pass to many of the functions.
  • Passing raw pixel data to Lua might be what the user wants, but if they want encoded PNG data to save (as per the existing extension) then it would be better to do that in C++ rather then passing it back and forth in various stages of encoding/manipulation. It might even be better to save the file from C++ too… but all of this adds engine code which may not be needed by all users.
  • The contribution guide is quite clear that if something could be done as an extension, this is preferred over adding engine functionality.
#8

Just chiming in here to say that we will be expanding the dmsdk graphics api by moving a lot of the functions from the internal api into the public one. We expose and use private headers in the rive extension, and we rather want to have them as part of the sdk for maintenance over time. I don’t know if it would solve all your issues but perhaps some of them.

You can take a look at the rive extension source to see what I mean:

#9

This is very interesting - I found: HContext GetInstalledContext(); in that header, which I didn’t know existed. This seems like it might be the “missing” thing I was looking for.

The GetWidth and GetHeight are also in that header, which is good if they will become public soon. And the other thing I would need is ReadPixels.

If you are planning to tidy up the way things are exposed in dmsdk, one thing I noticed which I thought was a bit odd, VulkanGetContext is actually exposed directly, but the equivalent OpenGLGetContext isn’t. I think that function should probably be removed from graphics_vulkan.h and (now that I know it exists) GetInstalledContext should be used.

What is the reason for waiting to make some of those functions public? Would it make sense to pick a few “obvious” ones and add them to public headers now, I’m thinking at least:

HContext GetInstalledContext();
uint32_t GetWidth(HContext context);
uint32_t GetHeight(HContext context);

ReadPixels is probably ok too… but I was unsure (and unable to test to verify) if the behaviour is actually consistent between different adapters. The direct GL way allows you to specify a colour-component order, but the dmGraphics functions don’t allow that, so if it’s already “well-defined” if the output should be consistent (vs. adapter-dependent) then that too could be added to the list. Those are all (I think) that is needed to implement screenshot capability in an extension.

#10

When we expose something publicly, it means that making changes to those functions are almost always a breaking change. If there aren’t any clear use case / business need we don’t expose the functions. In this case we had to fast track some vulkan functionality to be able to support rive, but since we now use their renderer directly we have basically removed most of the vulkan api and will be exposing more of the “shared” api.

3 Likes
#11

I can’t stress the importance of this enough! We do not want to lock ourselves into a large public api if we can avoid it. Carefully and gradually exposing functions oss the way to go.

4 Likes
#12

Understood, making APIs public should be done with consideration for the interface, maintainability, and preserving the ability to maintain backwards compatibility over time.

Given the above, in an effort to identify quick wins and make some functions available now, would it make sense to perform that due-diligence / consideration with these functions?

HContext GetInstalledContext(); // Possibly rename to simply GetContext in the public API?
uint32_t GetWidth(HContext context);
uint32_t GetHeight(HContext context);
uint32_t GetWindowWidth(HContext context);
uint32_t GetWindowHeight(HContext context);

I think that they are all “obvious” candidates - meaning that the API is simple and clear enough that it would not be likely to change, so the consideration required is very small.

ReadPixels is probably ok too… but I was unsure (and unable to test to verify) if the behaviour is actually consistent between different adapters. The direct GL way allows you to specify a colour-component order, but the dmGraphics functions don’t allow that, so I think it is already “well-defined” - i.e. the output should be consistent across adapters. If this is correct then ReadPixels could also be added to the above list.

#13

Yes although I’m not sure whether or not getting the getwidth/getwindowwidth functions even make sense any more. To me they should be removed from the engine in favor of moving them to the dmPlatform namespace, but that’s details we’ll have to consider

#14

Yes, this is something I was unsure about, and I was hoping one of you would “just know” the correct thing to do :grimacing:. I think it depends on the timing of when you call the Get* function; if the context is currently bound to a different texture or render target (not the main frame buffer), then GetWidth vs. GetWindowWidth won’t necessarily give the same answers. The dmGraphics versions do indeed just call the dmPlatform versions based on the currently installed adapter (uses g_functions.m_GetWindow which is updated in one of the SelectAdapter* functions when the adapter changes).

I think the exposure of such functions is not affected though, because any such nuances could be addressed in documentation explaining the behaviour if called at different stages of rendering. It’s a good point though that it might be better to expose the window variants in dmPlatform - but then you’d need a way to get the HWindow and that might not be something that’s desirable to expose (although it’s already opaque, so that does give some protections). In any case, I think there’s no harm to expose them as “convenience” functions in dmGraphics.

I do think ReadPixels is a more complicated case though - because (I think) it always reads from the window frame-buffer, and it might be desirable to read from the context’s currently bound render target for some use-cases, so usage and interface there may need more thought, in order to be more flexible for such possible future cases.

#15

Could you please write how you would use this in your extension and how the end user would end up using it? The simplest example.

#16

Well at the moment I use it in a pretty minimalist way, to take a screenshot:

  • Use GetInstalledContext to get the current context.
  • Use context with GetWindowWidth/Height to create a buffer of the right size
  • Use context with ReadPixels to fill the buffer
  • Process the buffer and encode it to PNG data

My extension code is based on the defold-screenshot extension, but a lot simpler (less features). The reason for not using that extension is that it calls gl* functions directly, bypassing the dmGraphics API, which means it crashes if Vulkan is used. My version uses dmGraphics API so that it works with the installed adapter (GL or Vulkan).

I have a working extension, it does everything I need it to do, but it uses these internally-public-but-unpublished-in-dmsdk APIs. So in this state I can’t realistically publish it for others, or use the code to create a PR on the existing defold-screenshot extension that would make it work with Vulkan.

1 Like
#17

Sounds great!

Then I will once again point out an important aspect (I have already written above) - the render script forms a queue of render commands, it is not just a wrapper for graphics API.

So, C++ ReadPixels function and GetContext function will always read window data (I guess of the previous frame because of double buffering) and return window context. Your extension, like defold-screenshot, will always take screenshots of the window only.

It is impossible to jump in between render commands and will only be possible if the engine will have the ability to add its own user render commands.

So you can call the native API from render_script or from script - the result will be the same.

(correct me if I’m wrong, i.e. I’m not the engine developer :grinning: ).

That’s why I wrote above that it might be cool to add a Lua function like render.read_pixels(callback) directly into the engine, so that it would be possible to use GPU for rendering all sorts of interesting things. And for screenshots too.

#18

Yes, it would be cool, but this sounds like a different feature. I think it should be possible to render into a render target then do further processing/manipulation of that buffer on the GPU (I don’t know the details though, not an expert in this stuff yet). What you’re talking about seems like a feature-request for something other than the existing ReadPixels to transfer the contents of a render target/buffer to the CPU, is that right?

In any case the original purpose of this thread was to figure out how to get the context to send to ReadPixels for grabbing the entire window. Then once I discovered GetInstalledContext and made my screenshot extension work in Vulkan, the rest was just trying to initiate whatever discussion is needed to make the required functions public so that I could contribute the extension back in a working/maintainable way. It doesn’t seem to be going anywhere at the moment so I guess we just wait for things to be made public in dmsdk before this can be contributed back.