LiveUpdate philosophy

[morgerion]

We have been developing a major project for over a year. And in it we are using LiveUpdate. And because of this, we constantly encounter problems along the way.

However, this approach has many limitations (this is my old topic about it):

1. if you build the same project on different PCs, its LU will not work with the old loader.
2. if at least one byte has been changed in the loader, the LU will not work with the old loader.
3. if the project version has changed, the LU will not work with the old loader (although the version has no effect on the content).
4. if you want to update the loader to the new one, it is impossible to do this through the LU.
5. when downloading files from LU it is impossible to know their size to show the progress bar.
6. and other things we have not found yet.

We tried splitting our project into a loader (as abstracted from the game code as possible) and the game itself. This solution helped to get a more or less workable version (“more or less” because it only works on version 172 of the engine).

Perhaps the thing is that the LiveUpdate has a name that sort of speaks of a live update - i.e. the ability to update content, to make it updated and fresh.
And we try to use LiveUpdate to update our content - to deliver some parts of it independently.
So we come to the point where the LU still requires a complete update of the entire project from the store.

What’s the sense of the LU then?

Okay, I understand that this is probably a pointless question, so I want to ask if it’s possible to get some kind of feature (let’s call it AutoUpdate) which will have the following properties:

1. any game collection can be excluded from the bundle and deposited somewhere on the internet.
2. the game will try to download the collection when it detects it.
3. this will not depend on the PC where the collection was built, neither the version of the loader, nor the engine version (but it will depend on the format - if the format has changed, you need to update the bootloader).
4. the game’s start collection (i.e. “in game” loader, not engine) can also be updated by the same way.

How feasible is this wish?

[britzl]

What’s the sense of the LU then?

How feasible is this wish?

Good questions! I’m going to spend time tomorrow to update myself on the current state of the LiveUpdate feature so that I can give you a good answer.

[Mathias_Westerdahl]

if you build the same project on different PCs, its LU will not work with the old loader.

That just seems like a bug to me. The bundling should rely on determinic checksums (e.g. Sha256).

if at least one byte has been changed in the loader, the LU will not work with the old loader.

This has to do with the checksum verification. The intention was to to avoid tampering with the resources, to make sure noone attempts a man-in-the-middle attack.
I guess an option to remove all signature checking is possible, but not sure if that’s desired?

when downloading files from LU it is impossible to know their size to show the progress bar.

That seems like a separate request, to get callbacks to the http.request() for download status.
Not sure if it’s possible to use a “HEAD” request before, to figure out the size, but since we cannot get the progress currently, it’s probably not so useful for showing a progress bar.

What’s the sense of the LU then?

As you say, the name implies the intent.
The idea is to be able to update content while the app is live, in order to avoid the long app submission process.

However, this feature is still in it’s first (almost at least) iteration.
Currently it basically only supports separating large game content from the the the main app bundle.
Sadly, we’ve never gotten to investigate how to update the LiveUpdate feature with more of the required functionality to actually update the “live game”.

this will not depend on the PC where the collection was built, neither the version of the loader, nor the engine version (but it will depend on the format - if the format has changed, you need to update the bootloader).

Adding a checksum for the file format isn’t something we have, and it’s going to be tricky, since we’re adding resource extension api.
However, it might be needed for our next version of LiveUpdate.

4. the game’s start collection (i.e. “in game” loader, not engine) can also be updated by the same way.

I’m curious, how often does this collection really change? What types of changes are we talking about?

Engine version

We have the support for supporting different engine versions in the manifests.
Each time the liveupdate bundle is created, the “liveupdate.supported-versions” is baked into the manifest. These versions are checked when the the manifest signature is checked.

liveupdate.settings:

[liveupdate]
supported-versions = "1.2.180","1.2.181"

I haven’t tried this feature myself, so I’m not sure what caveats there are.

MVP2
To sum it up, what we originally intended to support (it’s still the current main idea), is to download new manifests. These manifests will work as before, referencing other resources and their checksums. The difficulty lies in how to verify this manifest, since the engine version loading it might not be the engine version that generated it with regards to file format. It’s vital that the engine doesn’t try to load something it doesn’t support.

While writing this, I also feel like “engine version” I mention above should almost work with respect to updating the game with new manifests. But we need to investigate it to find out more about what the current state is, and what needs to be done to make it easier to use.

[britzl]

Not sure if it’s possible to use a “HEAD” request before, to figure out the size

Yes, it should be possible to do a HEAD first, but as you say, since we do not have a progress function it is of little use.

when downloading files from LU it is impossible to know their size to show the progress bar

You can at least have a progress bar that increments as each file is downloaded.

The intention was to to avoid tampering with the resources, to make sure noone attempts a man-in-the-middle attack.
I guess an option to remove all signature checking is possible, but not sure if that’s desired?

We could leave this up to the developer to deal with. Let them secure the application the best they can.

[morgerion]

I guess an option to remove all signature checking is possible, but not sure if that’s desired?

We are very careful about this, so it would be convenient to have the responsibility on our side.

The idea is to be able to update content while the app is live, in order to avoid the long app submission process.

We thought exactly the same.
This is exactly the kind of functionality we need!

I’m curious, how often does this collection really change? What types of changes are we talking about?

In our case, it turned out to be small changes that happened all the time. It was nothing global - many cosmetic adjustments. However, it is important to note that the loading logic of the rest of the game remained largely unchanged.
So the old downloader might well have downloaded the new game correctly. But this turned out to be impossible. And that was sad.

You can at least have a progress bar that increments as each file is downloaded.

This is what we have done now. And it works well, as we have more than 1,500 files.

[britzl]

I’m going to spend time tomorrow to update myself on the current state of the LiveUpdate feature so that I can give you a good answer

I have started testing with a small sample project and actually came across a crash when downloading a new manifest, rebooting and loading missing resources. I’m surprised that no one has reported this before and that we haven’t caught it either. Anyway, it is reported on GitHub and we will look into it soon.

[SkaterDad]

The idea is to be able to update content while the app is live, in order to avoid the long app submission process.

Do other engines implement this? Given the other thread about Apple’s review process, this kind of feature makes me think they’ll flag Defold games w/ LiveUpdate more often.

I don’t have any plans on releasing for Apple, but sometimes Google likes to copy them.

[britzl]

Do other engines implement this?

Yes, but maybe not all engines. And a lot of games have different kinds of live-ops systems built in to be able to show new content when running different kinds of campaigns etc or to tweak gameplay settings or patch levels after release. I know King users this extensively in most of their games. They do not have a system where they can replace anything at any time so it’s not as powerful as LiveUpdate.

We have been discussing a way to allow developers to completely exclude the LiveUpdate system from their games using an app manifest, but it’s not possible to do this yet.

[Lex]

Are Live Updates still supposed to work after I use a new build of my app?
Is the content of the LU tied to a specific app version?

[Lex]

Current I’m publishing my LiveUpdate content using using this URL:

http://staticfiles.myserver.com/debug/1.0.1/000340400d060bcc416e8959445e0ec1862b1021

So I include debug/prod and the version of the app. Is that a good practice?
I’m afraid if I update the app, older users won’t be able to use LU anymore.

[britzl]

Is that a good practice?

Are you also updating the application manifest or are you only downloading excluded content?

If you are downloading only excluded content then yes, you need to differentiate between excluded content for different app releases.

[Lex]

I’m just using to download new content (new levels,including scripts).

[Lex]

I get those messages when I install a new version of my app and try to access LU content:

WARNING:RESOURCE: Unable to create resource: /worldmap/floors/1/floor_1_generated_0_generated_0.spinemodelc: DDF_ERROR
WARNING:RESOURCE: Unable to create resource: /worldmap/floors/1/floor_1_generated_0.goc: DDF_ERROR
WARNING:RESOURCE: Unable to create resource: /worldmap/floors/1/proxy_floor_1_generated_0_generated_0.collectionfactoryc: DDF_ERROR
WARNING:RESOURCE: Unable to create resource: /worldmap/floors/1/proxy_floor_1_generated_0.goc: DDF_ERROR
ERROR:GAMEOBJECT: Could not instantiate game object from prototype /worldmap/floors/1/proxy_floor_1_generated_0.goc.
WARNING:RESOURCE: Unable to create resource: /worldmap/floors/1/proxy_floor_1.collectionc: FORMAT_ERROR

Any idea what could be the reason?

[britzl]

Any idea what could be the reason?

So you are downloading a new manifest, verify it, store it and reboot and then you download new content that was not known at the time when the app was created and installed on your device?

[Lex]

Yes apparently.

I’m confused about the manifest.

This is my current manifest, and I never touch it in between installs:

# App manifest generated Sat Mar 06 2021 12:08:35 GMT+0100 (Central European Standard Time)
# Settings: OpenGL
platforms:
    x86_64-osx:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            frameworks: []
            linkFlags: []

    x86_64-linux:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []

    js-web:
        context:
            excludeLibs: []
            excludeJsLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []

    wasm-web:
        context:
            excludeLibs: []
            excludeJsLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []

    x86-win32:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []

    x86_64-win32:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []

    armv7-android:
        context:
            excludeLibs: []
            excludeJars: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []
            jetifier: true

    arm64-android:
        context:
            excludeLibs: []
            excludeJars: []
            excludeSymbols: []
            symbols: []
            libs: []
            linkFlags: []
            jetifier: true

    armv7-ios:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            frameworks: []
            linkFlags: []

    arm64-ios:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            frameworks: []
            linkFlags: []

    x86_64-ios:
        context:
            excludeLibs: []
            excludeSymbols: []
            symbols: []
            libs: []
            frameworks: []
            linkFlags: []

I just want my app to download the new content, I don’t understand why it’s updating the manifest as well.

[Mathias_Westerdahl]

Any idea what could be the reason?

Your error messages are likely due to file format changes to the content.
E.g. we’ve recently updated the sound component with a new property.

What you hshow here is an “extension manifest” (ext.manifest), for an extension to the engine.
It has nothing to do with the “live update manifest” (liveupdate.dmanifest).

[Lex]

OK my mistake, I was confusing the different manifests.

Just to make sure I understand:

• App Version 1: excluded collections: A, B, C, D.
  LU. Content stored on server.com/v1/xxx

• App Version 2: excluded collections: A, B, C, D, E, F.
  LU Content stored on server.com/v2/xxx

Players upgrading from v1 to V2 will have no issues downloading the new LU content, am I correct?

[britzl]

Players upgrading from v1 to V2 will have no issues downloading the new LU content, am I correct?

Correct!