Plan on writing Microphone input native extension, what should I know?

Questions
#1

I’ve decided since I haven’t been able to find any instance of someone writing an extension for microphone input, I’m deciding to give it a try. I haven’t written any extensions, but I believe over time it should work out.

I plan to write it for all the platforms, I just want to figure out what I should know. I’ve heard of OpenAL, but I haven’t done my research on it yet or how to implement anything.

I also want there to be “live recording” for proximity chats in games.

Does anyone have any tips on what I can do to better help me write this? It would be helpful along with my own research. Thanks.

2 Likes
#2

I am looking for the same functionality. For my part, I take a look at the fmod library. It seems that it allows it as standard (at least in C#, C++), remains to be seen if the bridge to defold allows it too.

#3

Both OpenAL and FMod are libraries to play sound. I don’t think FMod supports any hardware functions to connect to a microphone.

For using the microphone, you need to either look it up for each platform, what API /SDK to use there, or find a separate library that supports the platforms you want.
For portability between our supported platforms, I’d go with option 1, using the api’s for each platform.

EDIT: I stand corrected, it seems the alcCaptureOpenDevice in OpenAL might work.
I had never heard of that before :thinking:

1 Like
#4

It’s curious, what do you think of these entries in the fmod api:

fmod.h

/* Recording API. */
FMOD_RESULT F_API FMOD_System_GetRecordNumDrivers       (FMOD_SYSTEM *system, int *numdrivers, int *numconnected);
FMOD_RESULT F_API FMOD_System_GetRecordDriverInfo       (FMOD_SYSTEM *system, int id, char *name, int namelen, FMOD_GUID *guid, int *systemrate, FMOD_SPEAKERMODE *speakermode, int *speakermodechannels, FMOD_DRIVER_STATE *state);
FMOD_RESULT F_API FMOD_System_GetRecordPosition         (FMOD_SYSTEM *system, int id, unsigned int *position);
FMOD_RESULT F_API FMOD_System_RecordStart               (FMOD_SYSTEM *system, int id, FMOD_SOUND *sound, FMOD_BOOL loop);
FMOD_RESULT F_API FMOD_System_RecordStop                (FMOD_SYSTEM *system, int id);
FMOD_RESULT F_API FMOD_System_IsRecording               (FMOD_SYSTEM *system, int id, FMOD_BOOL *recording);

and theses from the defold fmod native extension:
fmod_generated.c

    #ifdef FMODBridge_func_FMOD_System_GetRecordNumDrivers
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_GetRecordNumDrivers);
    lua_setfield(L, -4, "get_record_num_drivers");
    #endif
    #ifdef FMODBridge_func_FMOD_System_GetRecordDriverInfo
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_GetRecordDriverInfo);
    lua_setfield(L, -4, "get_record_driver_info");
    #endif
    #ifdef FMODBridge_func_FMOD_System_GetRecordPosition
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_GetRecordPosition);
    lua_setfield(L, -4, "get_record_position");
    #endif
    #ifdef FMODBridge_func_FMOD_System_RecordStart
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_RecordStart);
    lua_setfield(L, -4, "record_start");
    #endif
    #ifdef FMODBridge_func_FMOD_System_RecordStop
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_RecordStop);
    lua_setfield(L, -4, "record_stop");
    #endif
    #ifdef FMODBridge_func_FMOD_System_IsRecording
    lua_pushcfunction(L, &FMODBridge_func_FMOD_System_IsRecording);
    lua_setfield(L, -4, "is_recording");
    #endif

I will try to enumerate the recording devices under my mac starting from the example provided in the native extension fmod…

1 Like
#5

It certainly looks like it.

I’m just surprised, since FMod seels itself as an audio playing library. I have never heard of it being used as an input device library.

If that works, then I’d guess that they also have support for the consoles as well, so perhaps FMod is the library to start testing with.

Then my last concern would be about its support for HTML5 input.

1 Like
#6

Here is a good starting point:
If I run this on my mac (with the fmod extension enabled):

 local count = fmod.system:get_record_num_drivers()
 print("count",count)
 for i=0,count-1,1 do
    print(i,fmod.system:get_record_driver_info(i))
 end

Then I get this on the console:

DEBUG:SCRIPT: count	6
DEBUG:SCRIPT: 0	Microphone externe	userdata: 0x0113e6ca50	48000	2	1	3
DEBUG:SCRIPT: 1	NDI Audio	userdata: 0x0113e6cba0	44100	3	2	1
DEBUG:SCRIPT: 2	BlackHole 64ch	userdata: 0x0113e6cc90	44100	1	64	1
DEBUG:SCRIPT: 3	Microphone MacBook Pro	userdata: 0x0113e6cd80	48000	2	1	1
DEBUG:SCRIPT: 4	BoomAudio	userdata: 0x0113e6ce50	48000	3	2	1
DEBUG:SCRIPT: 5	Microsoft Teams Audio	userdata: 0x0113e6cf40	48000	3	2	1

But it fails when I target the HTML build (no input device is detected):

DEBUG:SCRIPT: count	0
1 Like
#7

Yeah, but another question, do you have to write extensions for Android in Java or will writing them in C++ suffice?

#8

You can use C++.
MAnual is here, and also take a look at all the assets our community has created: Defold Asset Portal

1 Like
#9

@Mathias_Westerdahl I also found this on their forum

FMOD does support recording although it requires the use of the Core API 105. We also have a code example included in the FMOD API download.

To use this in a Studio Project you would need to play it through a Programmer Instrument 50, see also the Unity example for Programmer Sounds 50.

From here: How to get microphone input event - FMOD Studio - FMOD Forums

#10

Hey, I’ve decided to use OpenAL to do the recording, but I’ve run into a problem: I need the microphone to keep recording in the background without halting the program.

All the examples use for loops to do the recording, but this would be bad unless there’s a way for me to write asynchronous loops in C++.

I also can’t just call it to record once every frame, because frame times are not constant if vSync is off and if it is, it would only record every ~16ms instead of repeatedly.

Do you have any ideas on how to get around this?

#11

You can run the recording code in a separate thread and use a Lua API to start and stop the recording.

#12

Correction - It uses a while loop

But still, I’m using std::thread and it’s still halted, and the engine says dmThread is not defined.

Edit: Did an experiment to see what it’d do if I added Sleep(x) after defining the thread because it crashes afterwards, the sleep function delayed the crash, if this information will be helpful at all.

I’ve figured out the threads, but how would you integrate the LUA API to help with this?

Edit 2: Decided that I would just have the microphone pic up samples 5 times each update frame, and the audio seems to be working fine

#13

I think I’d have a microphone.start_recording() function which creates the thread, sets a flag and begins recording. Then also a stop_recording() function which does the opposite, clears the flag to signal that the thread should stop, waits for it to stop and returns the recorded sound data.

You mean when you build the project?

#14

The sdk.h didn’t include thread.h.
You can use #include <dmsdk/dlib/thread.h>, which is a good practice anyways (to not include more than what you need).

1 Like
#15

Oh Alright, thanks.

#16

I made a little progress in using FMOD:

I started from the record.cpp example provided in the fmod SDK. This example aims to replay what comes in through the microphone with 50ms of delay (you can enable or disable a reverb DSP effect).

I wore it under Defold (without the reverb part) using the example script provided by the defold-fmod extension.

I had to make a modification to the fmod bridge to solve a problem of initialization of a structure size (FMOD_CREATESOUNDEXINFO.cbsize).

I get to a point where everything initializes and plays fine (it seems!) but I can’t hear anything coming from my microphone (which is detected fine…)

Here is the link to my project for those who want to help me by taking a look :wink: :

1 Like
#17

I’ve made some progress too! I’ve been successful in getting it to record on Windows, but on Mac, it’s giving me a headache. Luckily, I’ve been able to find an OpenAL-Soft port for Mac.

Now the problem is I’m trying to use the same permission getting scheme as the camera extension in camera_darwin.mm (starting at line 409), but it keeps crashing when it tries to get permission. I read somewhere it’s because it runs the completionHandler function in a background thread, and running threads was the same problem I had in the past that kept crashing the engine.

Here is my code:

        if ([AVCaptureDevice respondsToSelector:@selector(authorizationStatusForMediaType:)]) { 
                 int status = [AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeAudio]; 
                 if (status == AVAuthorizationStatusAuthorized) { 
                         dmLogInfo("Permission has already been authorized"); 
                         micPlatform_Record(); 
                         return true; 
                 } else if (status == AVAuthorizationStatusNotDetermined) { 
                         //The line right below this one is the code causing the crash
                         [AVCaptureDevice requestAccessForMediaType:AVMediaTypeAudio completionHandler:^(BOOL granted) { 
                                 if (granted) {
                                        //The code never reaches here 
                                         dmLogInfo("Microphone access granted!"); 
                                         micPlatform_Record(); 
                                 } else { 
                                         dmLogInfo("Microphone access denied."); 
                                 } 
                         }];
                 } //... Code continues on

I even tried removing the return statement, thinking the function ending before the thread not being joined was the problem, but that wasn’t it.
@Mathias_Westerdahl sorry for pinging you so often, but what could be the problem?

2 Likes
#18

Hard to say if there is no callstack?

#19

Oh yeah, my bad.

CALL STACK:

# 0 pc     0x2ab8b4 dmengine _ZN7dmCrashL7HandlerEiP9__siginfoPv+36
# 1 pc     0x309d7d libxpc.dylib _sigtramp+29
1 Like
#20

Thanks. But as you can imagine, it doesn’t give us much, since that’s the assert and the crash handler and nothing else :slight_smile:

You may have better luck if you run it through the debugger (lldb or XCode).
E.g:

> cd myproject
> chmod +x ./build/x86_64-osx/dmengine
> lldb ./build/x86_64-osx/dmengine
lldb> run
1 Like