Texture Compression Update (Alpha testing)

@agulev What are the “zip” files? Are they the game bundle? Or just the build folder zipped up?

@aglitchman There is currently work being done for the uastc compression to make it being more “compressible” by LZ4 (or any other), with minimal quality loss. You can follow the developer Rich Geldrich on Twitter for a lot of updates and details regarding this.

2 Likes

Just a zipped bundle folder.

1 Like

@pkeod No, fonts have the suffix “.fontc”.

We disassemble each font into separate glyphs (raw data), which we then compress individually. We do this since we use a texture atlas which we populate with the small glyphs as they are used, and evict older glyphs as needed.

Since we removed WebP, we now use Deflate for compression, so the sizes grew a little bit. (from 55% to 66%)

I have a PR coming up that will bring the compression of the glyphs back to 62% (On 26 MB fonts, it previously compressed to 17.6mb with deflate, now it compresses to 16.6 mb)

The next big thing for fonts is to add an api to dmSDK that lets extensions provide the actual glyphs themselves to the engine. That, I think will be a huge space saver.

5 Likes

What’s an easy way to figure out what the /_generated_f6cce506.texturec files are? There are apparent duplicates in size which concern me.

Agreed, the naming is an issue. You could try to add “-v” to bob (if you’re using it that is), to perhaps see which atlas becomes which texture.
If it doesn’t, I should add it :confused:

Unfortunately, the editor is more “clever” and merges all output that has the same checksum (not sure how often that happens though), so the same .texturec could potentially have multiple source files.

3 Likes

Since @Jerakin asked for some numbers from my testing.
Thanks to this I found a thing I had forgot to change, add the LZ4 compression on the textures(!). It was previously turned off since minimal impact on the WebP compressed textures.

  • 5 textures (256x256) 2xRGB, 2xRGBA, 1xR
  • With mipmaps
  • macOS

Build times

WebP (lossy)      14.808s
WebP (non lossy)  14.816s
Basis             7.035s

Size on disc: (game.arcd) (LZ4 compressed):

WebP (lossy)      313930
WebP (non lossy)  315906
Basis             395526

the uncompressed basis textures were 645846 bytes

Runtime decompress/transcode

WebP (lossy)      5 textures / 327680 texels in 7.487000 ms
WebP (non lossy)  5 textures / 327680 texels in 7.310000 ms
Basis             5 textures / 436905 texels in 4.886000 ms

I calculated the texel count is more accurate on basis. They should be the same

Update The average LZ4 compression of the basis textures are ~42% of the original size.
Also note that we anticipate even more compression ratio improvements from Basis Universal, as they continue to develop it as we speak.

5 Likes

Some engine size numbers (since I know @AGulev wants to know :wink: )

Release builds, stripped

arm64-darwin

1.2.179 2635564
1.2.180 2553284

-80kb (~3%)

arm64-android (dmengine.apk)

1.2.179 2024820
1.2.180 2092295

+64kb (~3%)

wasm-web

1.2.179 2160378
1.2.180 2187290

+~26 kb (~1%)

32 kb was added for the WebGL 1 fallback

In all honesty, I would have thought that the engine would have decreased even more since WebP was quite big. But I guess they were more or less the same size then. :man_shrugging:

3 Likes

There is a new alpha build available, with LZ4 compression enabled for the textures.

3 Likes

Piggybacking on this topic, but as I’m trying to compress my resources for iOS and Android. How do you know which FORMAT to chose?

image
Luminance, Luminance alpha etc don’t sound familiar, I don’t know which one to chose from. I use regular pngs and spines, which one would you pick and why? Thanks

(If this is off topic I will delete my post)

1 Like

Check out this thread

3 Likes

awww, thx!

Then it’s time to exclude the video recorder by default! =) For me it was a surprise that webp and the recorder uses separate codecs lib. I was sure the recorder use the format it uses just to avoid using separate vpx lib.

4 Likes

Yes. But also note that the video recorder is excluded by default on release builds, so you won’t save anything there

3 Likes

2 html5 build reports.
one in 1.2.179
the other one is in 1.2.180
All the settings are the same.

web.zip (340.9 KB)

It seems like for html5 build the new compression is worse than webp

4 Likes

When you say worse, I know you mean size on disc.
But what about the texture quality? And the fact that we support hardware textures is also a plus.
And, as I’ve mentioned before, the devs are working on more optimizations for the compression size.

3 Likes

True. You right. I take into account only one aspect.

7.29 vs 5.83 mb it’s a big difference for web. what about the idea to keep current webp lossy from 1.2.179 to 1.2.180 but with other title? :astonished:

4 Likes

I don’t see it as very likely that we’ll add WebP back as a builtin compression format. Supporting many separate technologies is an added extra development cost.
We will however be supporting custom resource formats later this year (and we’ll keep this feature in mind when we do). In that scenario you should be able to compress the data any way you want.

Another feature I would like to see as a saving for games chasing the last megabytes, is an api to supply a font with font glyphs. That would enable a game to avoid bundling with a lot of font data.

6 Likes

I want to raise this topic again, my html5 build report of (Raft Wars Multiplayer game)

this is a very big difference for html5 games + ~2.45 mb,
and we see 3.76 mb vs 6.07 mb of atlases size ( > 60% bigger)
we’ve tried for a very long time to reduce the final build size by every 1 megabyte, if check again the video posted by @aglitchman the video Size matters: how loading is losing you players in his topic, such a difference may affect income. :confused:

5 Likes

I agree, it’s not ideal, and we’ll look into it.
Either the basis library update will improve the situation, or we’ll have to come up with a way to use webp again (e.g with an appmanifest).

4 Likes

UASTC is for extremely high quality (similar to BC7 quality) textures, and ETC1S is for very small files. The ETC1S system includes built-in data compression, while the UASTC system includes an optional Rate Distortion Optimization (RDO) post-process stage that conditions the encoded UASTC texture data in the .basis file so it can be more effectively LZ compressed by the end user.

Does it have features for compression amount?

Maybe reaching out to Binomial and showing them the Defold use cases would encourage them to improve compression amounts / have some advice.

@BunBunBun what is the Compression option you have set in your texture profiles when doing that test? Best?