E2E testing in Defold

Questions
#1

Hi all,

I am getting to the point in my game dev journey that it is becoming unwieldy to test new features. Every time I add something new that I want to test in a multiplayer environment, I have to do the following:

  1. Run the server locally.
  2. Bundle the app.
  3. Bundle the app a second time (this time I do HTML 5 so I can get two at once).
  4. Click Create Game on one client.
  5. Type in the room code that I see in the server logs into the second client and click Join Game.

Only after this do I have two players loaded into the level and finally test what I changed.

I see a post from 4 years ago about integration testing but it didn’t look too easy, and some of the links are dead: Testing with dmengine_headless. These days are there tools out there to enable easier E2E testing, such as clicking through the UI or perhaps even sending messages on the side to progress my game to a certain state?

Thanks!

#2

Although I don’t see a quwstion, I can perhaps share some ideas for improving some of the steps at least.

When rebuilding/bundling a project often, I always use our command line tool Bob. It would allow you to bundle for different platforms to different directories.

I’d collect all these steps in a script: “rebuild.sh”

The server.
Depending on the control you have over it, there are a few options.

  • Output the room id directly to a separate text file (e.g. srv_debug_room.txt) to make it easier to find/read
  • Pipe the output to a log file, and the output for room id, then output the room id to the text file

When adding a debug room to your build, you can output the room id into a separate debug.ini file:

[debug]
room = my_room_id_from_the_logs

Then you can pass that to bob with the “–settings=debug.ini”, and that setting will be available ingame as “debug.room”. (Should be nil or empty string otherwise)

I would then launch the local engine executable.

And finally, I’d start a python server in the HTML5 bundle directory:

cd ./bundle_html5/ && python3 -m http.server

# and possibly open a new browser page
# different command for different platforms
open ./bundle_html5/path_to_index.html
1 Like
#3

Yes, check Poco SDK for instance:

I also did some tests with Cucumber automation a couple of years ago:

1 Like
#4

Hey @Mathias_Westerdahl, this advice was gold, I’ve managed to get 95% of the way towards my desired setup. I have a Python script that spins up an instance of my backend and two instances of my frontend. I also have it set up almost as you described:

  1. Build client 1 with a settings file that looks like this:
[localtest]
create_room=blah
  1. Start up client 1. It reads this and then creates a room.
  2. The server writes the room ID to a file.
  3. Build client 2 with a settings file like this:
[localtest]
join_room_id=xDOvLT_tV
  1. Client 2 joins the room. This part doesn’t work right now, instead the create_room block triggers, even though I’m confident the file is correct.

I have this working most of the way, but I am running into a couple of issues:

  1. I can no longer see the debug console output from the client, since I am running the built app directly. I was hoping perhaps it would get output to stdout + stderr, but it doesn’t seem that way. Is there some way to make this happen, or should I add some custom logging solution? (and do you have a recommendation there)
  2. When I check certain config values (sys.get_config("localtest.create_room")) (which I learned to do from here: Access gravity in Project Settings (SOLVED)) it says they are true even though I did not set that value in the settings for that build.

I think I could probably figure out 2 if I had 1 working though.

Overall though this is a way better solution than what I had, thanks a lot for the advice. Now I just run one command and I get the full E2E setup. Thanks so much!

@britzl Those look great. I’ll report back when I get around to using poco. Thanks!

2 Likes
#5

The default build variant is a release build when using bob.jar. You can specify --variant=debug when calling bob to get a debug build. Bob options:

Hmm, sys.et_config() should return nil if the value doesn’t exist.

#6

@britzl, setting --variant debug did the trick to be able to see the output!

I realize that the settings are acting very strangely. See these two command invocations:

# First
['/usr/local/bin/bob', 'build', 'bundle', '--settings', '/var/folders/lb/8_8c7mts709833552qtv70n40000gn/T/hotdogshl8eddoy/localtest.ini', '--platform', 'x86_64-darwin', '--variant', 'debug', '--archive']

# Second
['/usr/local/bin/bob', 'build', 'bundle', '--settings', '/var/folders/lb/8_8c7mts709833552qtv70n40000gn/T/hotdogskonw0tvk/localtest.ini', '--platform', 'x86_64-darwin', '--variant', 'debug', '--archive']

I confirmed that the two different settings files are as I describe in the previous comment.

I see that what happens is the second client just uses the value for these settings from the first build of the client. If I specify clean, this does not fix it, nor does distclean (or both). The only fix is to manually delete the build directory.

When I look in frontend/build/default/game.projectc I see that this is indeed the case:

$ cat build/default/game.projectc | grep -A 10 localtest
[localtest]
create_room = blah

The immediate solution here is to just keep doing what I’m doing, where I manually delete the entire build directory at the start and in between building each instance of the client, but I wonder if there is a better way?

#7

Yes, there should be. Can you please share your script here or the relevant parts where you call bob.jar?

#8

The script is sort of complicated, and getting more so. The command invocations above are what I’m running though, just those two commands. If you need something more specific though please let me know.

#9

What happens if your specify different --build-output folders for the two builds?

#10

I have figured out the issue, it’s just dumb user error. I sort of assumed that if I did something like this:

bob build bundle clean

That it would order them in the way I wanted (which doesn’t make sense in retrospect). What actually happens is it runs each command in order, so I what I actually need to do is this:

bob clean build bundle

With that, the directory gets cleaned appropriately before the build, ensuring that artifacts aren’t left behind.

I am sort of surprised that bob build after a previous build doesn’t produce the same output as if I had run it from scratch, but I’m sure there’s a good reason for it. Or maybe not and this is a bug, perhaps specific to settings files?

Nonetheless, thanks @britzl for helping me figure this out!

1 Like