Know your engine limitations!

[GerodruS]

Results for a test with usage go.animate function. Code is here

Table with data
(There is a new column: Defold - each animation)

[britzl]

Ah, this is better. Using go.animate should give a lot better results, and the graphs back up that claim!

[GerodruS]

And here some additional data for second experiment with a lot of sprites

Table with data

[britzl]

Hmm, the CPU graph for Defold looks really strange. Why is it flatlining at around 40%?

What was the target device?

[GerodruS]

It’s a good question!

[AGulev]

I just remembered about Bunny mark test:

Bunnymark benchmark

There are actually quite a few performance improvements you can make. I created my own version: https://github.com/britzl/defold-bunnymark A couple of key things to consider in your version: Don’t do go.get_position(). Keep one vector3 per bunny and work against that vector3 in update(). And if you insist on doing go.get_position, then consider using that vector3 instead of creating a new vector3 for every bunny every frame. Avoid index access into tables (every foo.something requires a looku…

And this test have the same results as @GerodruS results.

And it is so sadly.

For example:

• 2.5 years demo of old version of PIXI.js;
• in browser;
• on ipad4 ;
  Displays 15000 bunnnies without any problem with 60 FPS
  http://www.goodboydigital.com/pixijs/bunnymark_v3/

Defold IPA (native app!!!) can’t show 500 bunnnies =(

[britzl]

I’d really like to profile that Defold iPad app. 500 sounds like a very very low number…

[sicher]

Sidenote: I recently bought an iPad pro 10 inch. This bad baby pushes 150 000 bunnies in the pixi.js test.

[Fuscy]

Last time I tested Unity vs Defold with bunnies, Defold had 100-200 more bunnies on Android than Unity. Perhaps the lack of luajit on iOS is to blame.

Anyway the Pixi.js test is strange. I can’t believe that result. They do something strange with using a ParticleContainer for the bunnies and PIXI.js lacks the logic overhead of a game engine. Their logic is mostly the math operations in .js and what they do behind the scenes with transforms (probably).

[ross.grams]

Pixi’s speed is not from a lack of engine overhead. You can do bunnymark tests with other small web game libraries and they don’t get anywhere close to Pixi’s numbers. My brother even made his own little rendering framework and looked through some of Pixi’s code, but didn’t manage to get it much faster than Phaser.

Ages ago when I was first checking out Defold I did some bunnymark tests with a few different game engines and web frameworks. These were my results: (Bunny counts at 55 fps)

HTML5 (firefox)
Haxeflixel - 950
Defold - 1600
Phaser - 2200
Pixi - 17,000

Windows
Godot - 4600
Atomic - 5000
Defold - 6500

Britzl’s version that used go.animate() instead of update() was much faster: 4800 bunnies in HTML5 and 16,000 on PC, but of course it’s not an equal test then.

[dmitriy]

Just for information.
Bunnymark, Britzl’s version that used update(), fixed dt (variable dt performs even worst)
Defold vs Apple SpriteKit:
iPhone 5s: ~2500 bunnies vs ~5000 on SpriteKit
My Mac mini: ~8500 bunnies vs ~12500 on SpriteKit
Above this numbers all variants starts showing significant FPS drop.

[britzl]

Britzl’s version that used update()

This is the least recommended way of animating things in Defold and the use of go.animate() should always be preferred whenever possible. What if you animate with go.animate() instead?

[britzl]

I’ve moved my Bunnymark test to a separate repo: https://github.com/britzl/defold-bunnymark

It contains three versions:

1. Animate using go.animate() (fastest)
2. Animate using a single update() function with cached v3’s instead of go.get_position() every frame
3. Animate using one update() function per game object (slowest, also limited to the max cap of script instances)

[GerodruS]

html5 version is a brilliant idea!
Results for my laptop:

• Animate using go.animate()
• 1500 bunnies and <50 fps;
• 7k bunnies and <20 fps.
• pixi.js 4.5.4 (all other settings are default)
• 7k bunnies and >50 fps.

[Fuscy]

I wonder if having to use methods like go.set_position influence the result. I have no clue how the code behind looks but without a reference to the object the go.set probably does a find through the whole list of Gameobjects registered in the engine each time. Factory.create probably searches through its array of factories. Msg.post probably goes through all messages and then through the ids (the URL could be used to reduce search space).

If this is true then time complexity of all operations would go from N (pixi) to N^2 (Defold). Anyway these are just guessplanations.

[Mathias_Westerdahl]

The complexity is O(N), the problem lies elsewhere (we have a few ideas after some initial profiling)
Hopefully, we can get to an optimising phase soon, we’ve been wanting it for a while

[GooseSwanson]

Regarding the use of plain Lua 5.1 on HTML5, is this likely to be switched to LuaJIT in the not too distant future? As I don’t know anything about why that should, shouldn’t or can’t be done, any insight into the decision you have made would be greatly appreciated.

Out of a personal interest I have just started looking into how to write efficient Lua that avoids memory allocations and other expensive operations. From what I have gathered I can do this by looking at the Intermediate Representation (IR) that LuaJIT outputs. Any conclusions made would be true for 32-bit ARM too since the JIT compiler works on the IR. The only platforms that any conclusions wouldn’t apply to are 64-bit ARM, which will change once you get to that item in the backlog, and HTML5.

[britzl]

Regarding the use of plain Lua 5.1 on HTML5, is this likely to be switched to LuaJIT in the not too distant future?

No, this is not very likely. I found this while searching for “emscripten luajit”:

“LuaJIT is not portable C code - the interpreter is handwritten x86/ARM/other assembly, and the JIT emits x86/ARM/other assembly as well. So an interpreter would need to be written that can run on the web, in C or asm.js, as well as a JIT backend.”

Upgrading LuaJIT to the latest version with ARM64 support is something we should do though.

Out of a personal interest I have just started looking into how to write efficient Lua that avoids memory allocations and other expensive operations. From what I have gathered I can do this by looking at the Intermediate Representation (IR) that LuaJIT outputs.

This sounds hard core! I guess a rule of thumb would be careful about object creation in general (tables, functions, function closures, coroutines, userdata etc).

[dmitriy]

For those who are interested, here is Unity version of Defold bunnymark middle test (update many from single script).
unity-defold-bunnymark.unitypackage.zip (75.7 KB)

Results from iPhone 5s, latest iOS release:

2500 bunnies
Defold ~55 fps
Unity ~60 fps

3000 bunnies
Defold ~47 fps
Unity ~30 fps

5000 bunnies
Defold ~28fps
Unity ~25 fps

As a reference implementation, SpriteKit version of this benchmark (Objective-C) run at 60 fps with 5000 bunnies.

[kumokairo]

@dmitriy can you make a relative power consumption test? Like leaving it running for an hour or so and checking how much battery has been drained during the process.
You are doing great and very useful work, thank you!