Know your engine limitations!

[GerodruS]

It’s a good question!

[AGulev]

I just remembered about Bunny mark test:

Bunnymark benchmark

There are actually quite a few performance improvements you can make. I created my own version: https://github.com/britzl/defold-bunnymark A couple of key things to consider in your version: Don’t do go.get_position(). Keep one vector3 per bunny and work against that vector3 in update(). And if you insist on doing go.get_position, then consider using that vector3 instead of creating a new vector3 for every bunny every frame. Avoid index access into tables (every foo.something requires a looku…

And this test have the same results as @GerodruS results.

And it is so sadly.

For example:

• 2.5 years demo of old version of PIXI.js;
• in browser;
• on ipad4 ;
  Displays 15000 bunnnies without any problem with 60 FPS
  http://www.goodboydigital.com/pixijs/bunnymark_v3/

Defold IPA (native app!!!) can’t show 500 bunnnies =(

[britzl]

I’d really like to profile that Defold iPad app. 500 sounds like a very very low number…

[sicher]

Sidenote: I recently bought an iPad pro 10 inch. This bad baby pushes 150 000 bunnies in the pixi.js test.

[Fuscy]

Last time I tested Unity vs Defold with bunnies, Defold had 100-200 more bunnies on Android than Unity. Perhaps the lack of luajit on iOS is to blame.

Anyway the Pixi.js test is strange. I can’t believe that result. They do something strange with using a ParticleContainer for the bunnies and PIXI.js lacks the logic overhead of a game engine. Their logic is mostly the math operations in .js and what they do behind the scenes with transforms (probably).

[ross.grams]

Pixi’s speed is not from a lack of engine overhead. You can do bunnymark tests with other small web game libraries and they don’t get anywhere close to Pixi’s numbers. My brother even made his own little rendering framework and looked through some of Pixi’s code, but didn’t manage to get it much faster than Phaser.

Ages ago when I was first checking out Defold I did some bunnymark tests with a few different game engines and web frameworks. These were my results: (Bunny counts at 55 fps)

HTML5 (firefox)
Haxeflixel - 950
Defold - 1600
Phaser - 2200
Pixi - 17,000

Windows
Godot - 4600
Atomic - 5000
Defold - 6500

Britzl’s version that used go.animate() instead of update() was much faster: 4800 bunnies in HTML5 and 16,000 on PC, but of course it’s not an equal test then.

[dmitriy]

Just for information.
Bunnymark, Britzl’s version that used update(), fixed dt (variable dt performs even worst)
Defold vs Apple SpriteKit:
iPhone 5s: ~2500 bunnies vs ~5000 on SpriteKit
My Mac mini: ~8500 bunnies vs ~12500 on SpriteKit
Above this numbers all variants starts showing significant FPS drop.

[britzl]

Britzl’s version that used update()

This is the least recommended way of animating things in Defold and the use of go.animate() should always be preferred whenever possible. What if you animate with go.animate() instead?

[britzl]

I’ve moved my Bunnymark test to a separate repo: https://github.com/britzl/defold-bunnymark

It contains three versions:

1. Animate using go.animate() (fastest)
2. Animate using a single update() function with cached v3’s instead of go.get_position() every frame
3. Animate using one update() function per game object (slowest, also limited to the max cap of script instances)

[GerodruS]

html5 version is a brilliant idea!
Results for my laptop:

• Animate using go.animate()
• 1500 bunnies and <50 fps;
• 7k bunnies and <20 fps.
• pixi.js 4.5.4 (all other settings are default)
• 7k bunnies and >50 fps.

[Fuscy]

I wonder if having to use methods like go.set_position influence the result. I have no clue how the code behind looks but without a reference to the object the go.set probably does a find through the whole list of Gameobjects registered in the engine each time. Factory.create probably searches through its array of factories. Msg.post probably goes through all messages and then through the ids (the URL could be used to reduce search space).

If this is true then time complexity of all operations would go from N (pixi) to N^2 (Defold). Anyway these are just guessplanations.

[Mathias_Westerdahl]

The complexity is O(N), the problem lies elsewhere (we have a few ideas after some initial profiling)
Hopefully, we can get to an optimising phase soon, we’ve been wanting it for a while

[GooseSwanson]

Regarding the use of plain Lua 5.1 on HTML5, is this likely to be switched to LuaJIT in the not too distant future? As I don’t know anything about why that should, shouldn’t or can’t be done, any insight into the decision you have made would be greatly appreciated.

Out of a personal interest I have just started looking into how to write efficient Lua that avoids memory allocations and other expensive operations. From what I have gathered I can do this by looking at the Intermediate Representation (IR) that LuaJIT outputs. Any conclusions made would be true for 32-bit ARM too since the JIT compiler works on the IR. The only platforms that any conclusions wouldn’t apply to are 64-bit ARM, which will change once you get to that item in the backlog, and HTML5.

[britzl]

Regarding the use of plain Lua 5.1 on HTML5, is this likely to be switched to LuaJIT in the not too distant future?

No, this is not very likely. I found this while searching for “emscripten luajit”:

“LuaJIT is not portable C code - the interpreter is handwritten x86/ARM/other assembly, and the JIT emits x86/ARM/other assembly as well. So an interpreter would need to be written that can run on the web, in C or asm.js, as well as a JIT backend.”

Upgrading LuaJIT to the latest version with ARM64 support is something we should do though.

Out of a personal interest I have just started looking into how to write efficient Lua that avoids memory allocations and other expensive operations. From what I have gathered I can do this by looking at the Intermediate Representation (IR) that LuaJIT outputs.

This sounds hard core! I guess a rule of thumb would be careful about object creation in general (tables, functions, function closures, coroutines, userdata etc).

[dmitriy]

For those who are interested, here is Unity version of Defold bunnymark middle test (update many from single script).
unity-defold-bunnymark.unitypackage.zip (75.7 KB)

Results from iPhone 5s, latest iOS release:

2500 bunnies
Defold ~55 fps
Unity ~60 fps

3000 bunnies
Defold ~47 fps
Unity ~30 fps

5000 bunnies
Defold ~28fps
Unity ~25 fps

As a reference implementation, SpriteKit version of this benchmark (Objective-C) run at 60 fps with 5000 bunnies.

[kumokairo]

@dmitriy can you make a relative power consumption test? Like leaving it running for an hour or so and checking how much battery has been drained during the process.
You are doing great and very useful work, thank you!

[dmitriy]

Defold energy usage:

min 1/20, max 3/20
Unity:

min 1/20, max 12/20

According Instruments Help:
“The Energy Usage instrument indicates a level from 0 to 20, indicating how much energy your app is using at any given time. These numbers are subjective. If your app’s energy usage level is occasionally high, it doesn’t necessarily mean that your app has a problem. Your app may simply require more energy for some of the tasks it performs. For example, it may use the GPS while performing complex network operations. This is valid energy use. Look for spikes or areas of high energy use that are unexpected or that could be performed at more optimal times.”

Disclaimer: I have absolutely zero Unity experience.

[Pkeod]

Defold is more consistent since it prepares most resources in advance while Unity does more real time management?

[kumokairo]

@dmitriy Thanks, insanely useful info

Unity does more real time management?

Unity has quite a lot of going on in background, but it’s not about resources, it’s about overall runtime. Resoures are loaded in advance in both cases, Unity just has too much of backend overhead - FMOD, Manager updates, Fixed rendering pipeline with 3D in mind (this is changing thanks to scriptable rendering pipeline in Unity, but it’s just a part of the whole runtime), all this stuff.

[Kios]

@Pkeod is right, Unity does do a lot of at runtime deserializing, and prepping of content, it has huge trees of GO relationships to parse. They describe in detail in their asset bundle docs, and maybe other places. This is mainly why unity takes so long to start, there is just so much abstraction to deal with.

The new compiler tech and ECS(entity component system), unveiled last year, will likely change that, according to the lead dev. So they hope to have massive gains in performance and load times(which will better enable their webgl build, for instance).

I personally think Unity should not be seen as a competitor to Defold in terms of battery life and performance, Defold can compare it’s self to GMS and libgdx, these other lower end solutions solutions that are much more barebones.

As for the bunnymark, I didnt check your code, but if you used updates in every object you will loose performance. Ideally you should have a singleton pattern like Britzl suggested in my performance post, and poll/call->update the objs from there. Unity had a post about it a while back. But again whatever difference it makes it should really not be comparable to Defold.

On a final note,
I did a few battery tests with a friend a long while back, we tried Defold, Godot, GMS, and Unity.
Defold was the best in terms of battery life, GMS/Godot were 1.x-2x more demanding, depending.
Unity was, i think, around maybe 3x more consuming than Godot/gms, at max. So it did around 2-6 times Defold’s consumption. In a benchmark it’s one thing, but when you add lots of other objs, behaviors, UI, Unity will hog everything down, just look at the famous and big unity mobile games out there, they all have massive battery consumption.
(i don’t have these projects btw, don’t ask)