Big memory problem with Editor 2 (SOLVED)

Questions
#1

Hello all.

My first question about build memory allocation:

I have many atlases with 4096*4096 dimentions(about 40). For good work Editor 2 I need add startup argument -Xmx4g(in last time -Xmx6g) for build my project, because defaulted allocated 2GB not enough for build. Yeah build process allocated around 6GB RAM it not joke. After end build process all allocated memory not return to OS and OS lagged and slow

I investigated problem(investigated with jconsole) and found out what Java garbage collector not immediatelly flush garbage after build(for my project it about 3GB RAM if needed I can make screenshots). If I use forced garbage collector garbage flush immediatelly, but allocated memory not return to OS.
I found two methods for drop not used memory:

  1. Start any other application with huge memory usage(Defold game bundle for example) for displacement memory
  2. In jconsole make heapDump (I think it work as method 1)

Anyone have thoughts for resolve problem?

My second question about build process:
If I right understand build process - Editor make compiled textures from atlases and load every atlas in memory this process allocated huge volume of memory, but every atlas saved to disk for what load all atlases to memory and not disposed saved atlas from memory?

And last question
Build process compile every png file and atlas to texture. What is the purpose of this? Thus, doubling the use of both disk space and memory on the build? I mean atlas containts same png files what was compile early

2 Likes
#2

The editor team needs to look into this I think. @mats.gisselson, @Erik_Angelin and @Ragnar_Dahlen, any ideas?

Yes, the build pipeline creates compiled versions of all assets, but it really shouldn’t hold all of them in memory at once.

The build process doesn’t compile any pngs. It will from the atlas create a single texture.

#3

Thank you for reply.

About first question I wait you colleagues.

I made this conclusion based on the fact that in the folder with the build there are compiled png files and atlases. Example.
ENV_fire_idle_01.texturec - it’s compiled PNG file
Fire.texturec - this is an atlas with the specified file

File and atlas placed in different folders.

#4

I make screenshot from Java VisualVM

image
image.png814×396 20 KB

On screen we view utilization memory for build process. From start and open project(without open any collection) Editor utilization about 1-1.5Gb Ram, after start build process Editor allocated about 6GB Heap Size and about 4GB Used heap Ram and not free it. After press Perform GC in Java VisualVM Used heap droped to about 1.6-1.9GB Ram, but not free Allocated Size. After I make Dump Heap and free about 2GB allocated memory.

And yes it build process problem. If not make build Editor usage about 2GB memory, but if open huge scene it again utilization big memory size.

And if dropped Heap size and not restart Editor - build process not used memory again, because I think all files build

#5

Sorry about third question. It was garbage from bundle process.

#6

Hi @tarnumius!
Thanks for your measurements and observations!
We have a PR up for review that should reduce memory consumption during build. It will likely be merged next week.

6 Likes
#7

Thank you.

Defold crashing on build
#8

Hello again.

I continued my investigation and found out problem with making temporary cache in temp directory. If the cache was created Editor not allocate huge memory, but if the cache is in the process of creating then huge memory allocated. But if the build is in the process textures in builld/default directory not allocated huge memory.

I found out the bundle is in process not required huge memory allocated. Editor good work in 2GB memory.

May be it help you for found problem.

2 Likes
#9

Hello again @tarnumius!

We’ve just merged a monster-PR that changes how we use memory during build. In essence we release the content as soon as it is written to a disk cache. That might help in your case.
For technical reasons the bundle-pipeline is different from the one used when building, this explains your observation on memory use during bundling.

Thanks again!

5 Likes
#10

Big thank you. When to wait update?

PS: I write all build/bundle behavior :slight_smile: Only for more understand problem.

1 Like
#11

I tested. Now cold start aquired 3.2Gb memory for full build process. Thank you!

2 Likes
#12

Update should be available now

3 Likes
#13

image

On screenshot - cold start without apply Perform GC.
It really cool! Thank you

6 Likes
#14

Cool! @Ragnar_Svensson is the one to credit :slight_smile:

3 Likes
#15

Never clicked ‘update’ faster in my life :wink:
Thanks guys, this helps a lot!

4 Likes