FMOD: using callbacks on TIMELINE_BEAT?

[K_F]

Hi everyone,

i am using the FMOD extension from @dapetcu21 so far. This works pretty good so far, thanks to all contributors! However, there 2 question I would like to discuss here:

1. A really cool feature of FMOD is that you can set callbacks on the beats of the timeline of an event. This way, you could spawn game objects with the beat of your music. I tried to get that setup with defold, but with no luck so far. It seems that the setCallback or set_callback function just don’t exist in the fmod extension for Defold. Has anyone got that working already?

2. I am experiencing a small, but noticable delay when starting events. I would say it is about 100-200ms, on Linux as well as HTML5. Is this normal/expected behaviour?

Thank you

[dapetcu21]

Unfortunately there’s no way to respond to callbacks since they happen on a separate thread and FMOD expects a response immediately. Unfortunately I cannot call into the main Lua context from a separate thread, so… no callbacks. There might be some solution, but it would involve a lot of gymnastics.

As for the latency, that’s dependent on the platform’s audio drivers. On HTML, it’s a hit tricky because FMOD needs to do all audio processing on the main thread in-between frames. You can try to reduce the size of the audio buffer (I think there are some game.properties settings if I recall correctly), but you might experience under-runs.

[britzl]

@dapetcu21

Unfortunately there’s no way to respond to callbacks since they happen on a separate thread and FMOD expects a response immediately. Unfortunately I cannot call into the main Lua context from a separate thread, so… no callbacks. There might be some solution, but it would involve a lot of gymnastics.

What do you mean by “FMOD expects a response immediately”? Isn’t this a callback from FMOD to Lua? Or does Lua have to somehow respond to the callback?

[Mathias_Westerdahl]

I think he’s saying fmod runs on a separate thread.
And rhen you can’t answer directly with the Lua callback.

[K_F]

Okay, thanks for the info. I will have to dig deeper into your FMOD extension i guess I also understood this as the fmod is running on a different thread. And yes, then it becomes more complicated to install the callback. But we might send a message/event asynchronously from FMOD to the lua main thread, right?

[britzl]

@Mathias_Westerdahl

I think he’s saying fmod runs on a separate thread.
And rhen you can’t answer directly with the Lua callback.

@K_F

But we might send a message/event asynchronously from FMOD to the lua main thread, right?

Yeah, I’m thinking that FMOD could put the event in a queue and we sync on and empty the queue from the main thread in extension update.

[Mathias_Westerdahl]

“Yeah, I’m thinking that FMOD could put the event in a queue and we sync on and empty the queue from the main thread in extension update.”

@dapetcu21

and FMOD expects a response immediately

I feel this would then not work as intended, if their api is to answer directly in their own callback?

[K_F]

When we call start on an fmod event instance, this also happens asynchronously, right? Because lua and fmod are different threads. How is this done?

[dapetcu21]

Yes. FMOD manages its own threading internally.

[dapetcu21]

Yeah. Last I remember working with this, it expects a response synchronously on the callback’s thread. I guess a solution would be to block the callback thread until the main Lua thread has the time to answer, but I’m afraid it might not be fast enough for FMOD. Worth a try though.

[Mathias_Westerdahl]

I think at least one-way notification would be a step forward, e.g. the game could react to the fmod events.
Sure, we can’t supply a response in time, but that might be ok for now?

[britzl]

@Mathias_Westerdahl

I feel this would then not work as intended, if their api is to answer directly in their own callback?

@dapetcu21

Last I remember working with this, it expects a response synchronously on the callback’s thread

Well, I guess this is what surprises me. I expect an event system to emit events that others may or may not listen for. I do not expect an event system to also require a response to an emitted event.

By the way, here’s the documentation for setCallback():

https://www.fmod.com/docs/2.02/api/studio-api-system.html#studio_system_setcallback

The docs mention “Callbacks are called from the Studio Update Thread in default / async mode and the main (calling) thread in synchronous mode” so there might also be a synchronous mode. Not sure if it is feasible to run everything on the main thread though.

Also, looking at the actual callback function I don’t see anything about FMOD requiring a response:

https://www.fmod.com/docs/2.02/api/studio-api-system.html#fmod_studio_system_callback

Am I missing something here?

[dapetcu21]

It’s FMOD_STUDIO_EVENT_CALLBACK_CREATE_PROGRAMMER_SOUND that we need. And I think that particular event is requiring a synchroneous response.

https://www.fmod.com/docs/2.02/api/studio-guide.html#event-callbacks

When Studio has been initialized in asynchronous mode (the default mode), callbacks are fired from the Studio asynchronous thread as they occur.

This means no Lua in event callbacks by default.

If Studio has been initialized with FMOD_STUDIO_INIT_DEFERRED_CALLBACKS then the FMOD_STUDIO_EVENT_CALLBACK_TIMELINE_MARKER and
FMOD_STUDIO_EVENT_CALLBACK_TIMELINE_BEAT callbacks will be fired from the next call to Studio::System::update.

This is not useful since we’re interested in programmer sound callbacks, not timeline callbacks.

If Studio has been initialized with FMOD_STUDIO_INIT_SYNCHRONOUS_UPDATE then all callbacks will be fired from the next call to Studio::System::update.

And this is also not useful, because FMOD_STUDIO_INIT_SYNCHRONOUS_UPDATE would put all audio processing on the main thread.