Know your engine limitations!

The Defoldmine
#41

I have just made a few performance / battery life tests on my Asus ME173X tablet (pretty low end device on which we usually test code and graphics performance) on a simple Defold logo ping-ponging from top to bottom using a Sine wave interpolation (Defold uses go.animate with internal tweening engine). I was conducting these tests for the last two days, made them at least twice per platform to ensure that results are consistent, and here are the results

unity
99% -> 1h -> 91%
99% -> 2h -> 79%
monogame
99% -> 1h -> 91%
99% -> 2h -> 82%
defold
99% -> 1h -> 93%
99% -> 2h -> 85%

Difference is quite apparent, especially after two hours, but it’s not nearly “3x more consuming”. I have also noticed that Unity and Monogame (especially Monogame) were less stable in terms of FPS (even on that small one ping-ponging logo demo), most likely occasionally missing vsync intervals, which resulted in stuttering. Defold, on the other hand, was consistent all the time. I have also noticed that Defold didn’t warm up the device while Unity and Monogame did (just a bit, it didn’t bring it to thermal throttling limits, but it was noticeable).

I will post a mini-postmortem on overall Defold usage “review” regarding our smaller 2D projects. We have worked with Unity for quite some time now (since 4.3, released a few games) and have decided to take a look at Defold as an alternative for creating more energy efficient and less development-heavy games.

13 Likes
#42

I wanted an excuse to mess around with Gideros, so I tested a quick bunnymark benchmark on my android phone (LG V20). https://github.com/lemon07r/gideros-bunnymark
Also tested with britzl’s https://github.com/britzl/defold-bunnymark

Results:
15000 Bunnies at 24 fps using go.animate() with Defold
15000 Bunnies at 29-33fps using particles with Gideros

*Fps was very erratic with the Gideros, jumping as high as 40 but didn’t seem to dip below 28.
**I’m not sure if this makes a difference but I exported to an Android Studio template from Gideros instead of to Android directly then compiled to an APK using android studio.

EDIT: Fixed HTML5 version. Try it here: https://lemon07r.github.io/gideros-bunnymark/

#43

Using particles with Gideros?

Try using ParticleFX with Defold with bunnies?

#44

I haven’t tried to port bunnymark to Defold (since britzl has already done it), will using ParticleFX be faster than britzl’s go.animate() method? I used particles in Gideros because that’s what was recommended for best performance, and I had thought go.animate() was the fasted method to draw bunnies on Defold.

#45

You’re comparing Apples to Oranges it looks like. If you try to get the same number of particles with ParticleFX it would be a better comparison to Gideros particles for FPS.

#46

Hmmm, food for the thought. I’ll try tomorrow unless someone else wants to do it. I dont have much experience with particles on Defold yet, so now I’ll have an excuse to learn. Learning to use gideros, I’ve read it’s good practice to use particles. So Im curious, why was go.animate() used over particles for bunnymark on defold?

#47

Particles are meant for certain kinds of effects such as the kinds you would want with many of the same objects moving together. You can use go.animate() for similar situations but there may still be better performance from using particles directly as there are other kind of overheads from the GO approach.

2 Likes
#48

I see that they are called particles but from the looks of the code you move them individually using code. Are there any limitations to particles in Gideros? Can you attach things to them? Can you change the looks of individual particles? Scale and rotation? Change the “sprite”?

1 Like
#49

Hey brtizl, I haven’t used gideros enough (literally only have a few hours on it) but I’ll try to answer your questions the best I can.

First and forth most, here is the reference: http://docs.giderosmobile.com/reference/gideros/Particles#Particles
This is where I’m answering most of your questions from.

Are there any limitations to particles in Gideros?

I haven’t used them enough to know yet! I’ll ask the Gideros community and see what they say.

Can you attach things to them?

Quickly skimming over the reference, the only form of “attaching things” I seem to see is a tagging system, but it looks like it’s still in development:
Particles:setParticleTag ()
Sets the tag associated to the given particle
and
(string) = Particles:getParticleTag(i)
Returns the tag associated to the given particle

Can you change the looks of individual particles?

Some of these are still in development from what I see, but you can change the angle, color, decay, position, size, speed, ttl, and tag using the particle index number.
It doesn’t look like you can change the texture individually with the particle index number, Particles:setTexture(texture) sets texture to all particles.

Scale and rotation?

Yes. Particles:setParticleSize(i, size) and Particles:setParticleAngle(i, angle)

Change the “sprite”?

I’m not sure what you exactly mean by change the sprite, but the documentation says particles inherit from sprites, so I’m assuming you would use the sprite class to change the sprite for things you cant do with the particle class:
http://docs.giderosmobile.com/reference/gideros/Sprite#Sprite
There seem to be a lot more transformation options using the sprite class.

1 Like
#50

Ok, so Gideros particles sound like Defold particles with the exception that Defold particles are controlled by the engine and spawned by emitters that are affected by modifiers. There is no programmatic way of controlling their movement pattern.

1 Like
#51

Would it even be feasible to draw bunnies with particlefx on defold? I took a quick look at the documentation for particlefx but it looked way more complicated than particles in gideros. If it’s doable I’d like to see how they compare using particles for a more apples to apples comparison.

PS I got a few different answers from the Gideros community if you are interested: http://giderosmobile.com/forum/discussion/7356/questions-about-particles-and-bunnymark-performance

#52

I added a particlefx test to my bunnymark project, but while testing I saw really weird spikes in the profiler so I’m not sure, but there could be a bug in the code when running with a high amount of particles (15k in my test). Anyway, it’s pushed to the repo if you want to do tests yourself, but keep in mind what I experienced while testing.

2 Likes
#53

Finally got around to testing it. Particlefx gets a stable 31fps for me on my LG V20 (go.animate() is still 24fps). The gideros bunnymark particles performance is very similar, slightly higher on average but wayyy more erratic.

A side note, the Defold APK size was less than 3mb, and the Gideros APK size with the single sprite and script is over 7mb. The more I use both engines the more mature Defold seems in comparison to Gideros, and I can honestly see Defold outpacing Gideros in development at this rate. I haven’t seen any major updates (or even plans) in a while from them but this all stands to question and could change in a heartbeat if things pick up over on their end.

Does finding those spikes in the profiler open up possibilities to make improvements?

#54

Yes, we know that we have things to improve, especially when it comes to game object performance, but probably also on particle effects.

2 Likes
#55

Would any of you be interested to see how other engines and frameworks compare in html5 performance to defold? I have a small compilation of bunnymark benchmarks of random framworks and engines I’ve been messing around with for fun. I was originally going to test for android but html5 has just been much quicker and easier to test.

4 Likes
#56

Obviously yes. As you see we’re very receptive to feedback, and we do our best to help synthetic tests make as much sense as possible.

So feel free to do the benchmarks as you please, just be sure to also tell plenty about testing methods and share the code. This would help community make sense on what and how is tested.

5 Likes
#57

First and foremost, a lot of these are lower level frameworks, so this isn’t exactly an apples to apples comparison. I’d be more surprised if they didn’t score as well as they did.

Defold (old build?) HTML5 bunnymark using go.animate(), 7 500 bunnies before I hit 55fps* (see edited note): https://github.com/britzl/defold-bunnymark
EDIT: I thought this performance was odd, but I had to rebuild from britzl’s most current source anyways for the particles test and found that I hit full 60 FPS at the 16383 limit with go.animate(), and the 15000 bunnies using particlesfx. This may be either cause britzl’s hosted build is built from older code or with an older version of Defold. It’s too bad I cant test for more bunnies. Try the updated build here: https://lemon07r.github.io/defoldbmtest/

Phaser can only handle 11 500 bunnies before going below 55fps: http://phaser.io/sandbox/Pjdxyqzl/play
The code is all there in the different tabs.

Gideros bunnymark, 16 500 bunnies before I hit 55fps: https://github.com/lemon07r/gideros-bunnymark
This is using particles as discussed earlier.

Pixi.js (4.7.3) goes up to 30 000 bunnies before fps dips below 55fps: http://pixijs.io/bunny-mark/

Love2d bunnymark running on windows (love2d doesnt support html5) I get up to 65 000 bunnies before hitting 55fps. This one I’m adding for fun, I don’t think comparing desktop to html5 is fair.

I tested the OpenFL (8.0.0) bunnymark template after adding a bunny counter https://github.com/lemon07r/openfl-bunnymark which does 45 000 before dipping under 55 fps.

Then I also did Kha https://github.com/lemon07r/kha-html5-bunnymark Which goes to 140 000 bunnies Keep in mind that kha is a very low level framework!

All tested in chrome on my desktop (i5-4590, gtx 970) under the same conditions, with the exception of love2d being run from within windows since it does not have an html5 export. All my github links have the HTML5 builds I tested with linked in description, hosted on GitHubPages in case you want to try testing yourself.

PS If you are interested in any other frameworks or engines and how they compare, let me know. I can try to add them to this comparison.

3 Likes
#58

Nice tests!

I think you should try and normalize the screen resolution between all the tests though as it can have an impact on performance.

Love2d also has an unofficial emscripten port here:

2 Likes
#59

I did normalize the resolution for gideros when I tested and used the same bunny texture. Originally I was using a larger size and resolution png and testing at 1440p, and actually found that the fps I was getting was same or unnoticeably different, I dont think there would be much difference for the other tests either. A lot of the tests I did were using already made code or code I slightly modified to have things like bunny counters. I could go back and normalize some of the tests the same way I did for gideros to match brtizl’s test but I dont think any of the results were narrow enough to really need such changes.

Im on my phone before heading to sleep. I’ll try the love2d emscripten port tomorrow and post my results here so look forward to that!

The defold tests are bottle necked by the max bunny limit so I decided to run the tests from my phone targeting the fps I got from the defold particlefx test since it is fixed at 15000 bunnies. Tested on my LG V20 in chrome, with everything kept consistent as possible and retested for the most accurate numbers.

Defold particlefx 22fps @ 15000
Defold go.animate() 23fps @ 10000
Phaser 22 fps @ 3100
Gideros (particles) 23 fps @ 7500
Pixi.js (4.7.3) 23fps @ 10500
Openfl 8.0.0 22fps @ 39000
Kha 21fps @ 50000

My take away, defold is surprisingly fast, it is only marginally slower than pixi.js which had been the king by a significant difference in most of these tests from what I have seen in older tests. Defold has also turned out to be quite a bit faster than gideros. I think the only serious contender here is openfl which is a good balance of being both easy to use and very very fast for what it is.

Im working on getting android ports to test natively on android. Here’s what I have so far:
Defold particlefx 31fps @ 15000
Defold go.animate() 31fps @ 12000
Gideros (particles) 31 fps @ 15000

I will add Love2d soon, hopefully openfl and kha shortly after.

8 Likes
#60

I never got around to adding love2d or the others since I felt they were much lower level and less android focused to be considered comparable so I ended up running some tests with coronasdk instead.

TL;DR - CoronaSDK was much slower and had a much larger apk size. I didnt bother testing the html5 performance since their html5 port is in beta, and barely 4 months old, so nowhere near mature enough to compare.

Link to my original post: https://forums.coronalabs.com/topic/73490-corona-sdk-performance-and-bunnymark-challenge/?p=387006

Some really cool stuff. I built an apk using the latest public release of Corona and @roaminggamer’s optimized version of bunnymark, it got 18-23 fps at 1000 bunnies. I rebuilt the Defold version of the benchmark using their latest version as well to see if anything had changed and for consistency, tested on the same phone right after roaminggamer’s version of bunnymark I still get 12k bunnies at 31 fps using their recommended method of go.animate(), particlesfx could hit up to 15k but wouldnt be a very real approach to drawing sprites. This was tested on my LG V20.

Some side observations, building the android apk was pretty hassle free like it is for defold, I had issues getting gideros to build me one some time ago, and my other observsation is apk size, I believe gideros had the largest one but I forget how big, defold’s bunnymark which comes with the options to choose from 5 different methods of drawing bunnies was only 2.88mb, corona was in the middle at 6.86mb. However from what I understand Corona has a lot more higher level constructs and such, meanwhile defold works a little closer to the metal with less overhead so I think this was to be expected. Benchmarks only tell one side of a story, so there is no better engine here, just engines that are better at different things.

7 Likes