Live Updates

[Ivan_Lytkin]

Very simple, I want to publish the game on different hosts (eg. itch & newgrounds) without paying for CDN to host live updates or recompiling binaries.

[jakob.pogulis]

Hello @Ivan_Lytkin

Just to clarify the confusion above, if the game is running on forum.defold.com/games/mygame you would need a way to retrieve forum.defold.com from within the game so you can retrieve resources from https://forum.defold.com/resources/mygame, and if the game would be running on github.com/myuser/myproject you would need a way to retrieve github.com. Attempting to get the IP or DNS of the client will not work.

There is currently no way to do this from the API, but as you pointed out it is possible to set a config variable when launching the engine in HTML5 where you can pass in the URL. You can retrieve these config variables in runtime through sys.get_config.

[sicher]

You can set config values via engine arguments like this:

--config=test.my_value=4711

and retrieve it from Lua:

local value = sys.get_config("test.my_value") --> 4711

[Ivan_Lytkin]

Thanks!

[sicher]

For a HTML5 build, modify the index.html:

   ...

	var extra_params = {
			archive_location_filter: function( path ) {
				return ("archive" + path + "");
			},
            engine_arguments: ["--config=test.my_value=12345"],
			splash_image: "splash_image.png",
			custom_heap_size: 268435456
		}

    ...

[Oleg_The_Evangelist]

http://schedule.gdconf.com/session/how-defold-helps-indie-devs-perform-live-updates-with-aws-presented-by-amazon
This is a sponsored session and shall be available without GDC Vault subscription later on.

Also everyone at GDC is welcome to Amazon booth in South hall and Defold booth PL1 at GDC Play, North Hall.

[Oleg_The_Evangelist]

Aaaaaand here’s @britzl showing Live Updates at Amazon booth: https://www.twitch.tv/videos/125868526?t=05h38m55s

[Oleg_The_Evangelist]

heeere we gooo - http://gdcvault.com/play/1024382
@britzl explains Live Updates at GDC. The full talk.

[Axel]

Live Updates is literally a game changer!

(I’ll show myself out)

[joshua.winters]

Further more, with this distinct addressing system it is possible to send another index to the game which represents a new version, and through collectionproxies, update the game with new content that wasn’t available when the game was first bundled. This feature could be used to hotfix issues, enable A/B testing, and more. But more about that in a later post…

I want to know if the functionalities mentioned here are available or not? The live updates manual just mentioned we can exclude resources belong to a collection proxy, and the resource api resource.get_current_manifest seems just return the “excluded resources” information. Can we generate manifest for the new version? Can we update the lua modules?

[sicher]

It’s not available yet but I believe that @Johan_Beck-Noren might soon produce some tricks from up his sleeve.

[Johan_Beck-Noren]

As @sicher said, the ability to load and use a new manifest and update the game with new content that way is being developed as we speak. It is a big task though so no definitive ETA I’m afraid. But we really aim to finish it soon since it adds a powerful feature to LiveUpdate.

[joshua.winters]

Thanks @sicher and @Johan_Beck-Noren for the quick replay. It is possible to bundle the new game content locally and generate a “manifest” by myself and use resource.store_resource() to update the game? You may know the apple store review often take weeks so we are highly dependent on the live update for the online activities of the festival.

[Johan_Beck-Noren]

As it is today you cannot change or update the manifest that is bundled with the game. This means that you cannot, with liveupdate, add or change resources that were not listed in the original bundled manifest. This new feature will let you do just that.

More info on the manifest: https://www.defold.com/manuals/live-update/#_the_manifest

[joshua.winters]

Thanks @Johan_Beck-Noren for the clarification.

[Oleg_The_Evangelist]

perhaps I can resurrect the old Live Update topic. The feature is shipped by now - you can create and ship new content to your live games without updating the client.

Did anybody actually try the feature? If not, what stopped you? Any concerns/questions?
@Johan_Beck-Noren and I are recording a video next week, and are thinking about additional angles to cover.

[Terry]

Hello @jakob.pogulis,

After reading the live update descriptions, there are some unclear points for me. Would you please take a look and help to give some feedback?
, thanks for your kind help.

1. live update settings
   There are two modes in the editor and one is the “Zip” model in which I get the zip for the excluded collection proxy in my project. After decompressing the file, there are the liveupdate manifest file and lots of hash named resource files.
   
   
   

I didn’t get the point for this zip mode. What’s the difference to the AWS mode? Can the engine do the check and start updating the manifest file from a local zip file? Or the AWS is the only place to hold this liveupdate manifest file currently? This is important for me. If yes, I can do a complete live update test offline using this zip mode.

2. Updating the manifest with Live update
   There are a version mechanism to detect whether the bundle has been changed in the manual page:

Starting the engine for the first time after a manifest has been stored will create a bundle identifier file bundle.ver next to the manifest. This is used to detect whether the bundle has changed since the manifest was stored, for example after a full app store update. If this is the case the stored manifest will be deleted from the filesystem and the newer bundled manifest will be used instead. This means that a full app store update will delete any previously stored manifest. Any existing Live update resources will however remain untouched.

How does the engine check this version? I assumed that the engine first takes a look at the project live update setting and knows that another live update manifest file(local file or url) is present. The engine decides whether the live update manifest file shall be download(or copy) according to the version. There is a tip about the fake code from the manual page.

If the zip mode in the question 1 does not work, can I use a local file server instead of the AWS for test? The only thing for the engine is to know how and where to get the file.

3. storage of the live update and bundled manifest file
   If the live update manifest file is dowloaded successfully, is it stored in another path without overriding the bundle one via the resource.store_manifest API? If not, do the engine get the correct file depending a special search path priority?

There is also a description for the verification.

When storing a new manifest the manifest data will be verified before it is actually written to disk. The verification consists of a number of checks:
Correct binary file format.
Supports the currently running engine version or any other supported version entry from the settings.
Cryptographic signature.
Signed using the same public-private key pair as the bundled manifest.

If the key and supported version is not filled in the project settings, what happens when call the resource.store_manifest API?

4. get the missing file of a collection proxy
   How does this API collectionproxy.missing_resources(collectionproxy) calculate the missing file count? Check the hash of all resource in the bundle(live update if there is one)manifest file and try finding it? Will an changed resource with a new hash be treated as a missing file?

5. download a single file
   After getting the count of the files to be download, is it possible to get more information? Like the total and already downloaded size of the files which could be used to indicate the downloading progress? In the production environment, a single download might be failed due to the network satus. A retry mechanism as part of the error handling might be important.

6. store the download file
   Does the API resource.store_resource store the download file some where like the save-file path on different operating system? Or it just modify the bundled one?
   If there a save path, when a excluded resource is updated for several times, will the old versions which do not match the latest live update manifest file be deleted automatically?

[Oleg_The_Evangelist]

@Johan_Beck-Noren and I will record a video on Tuesday on Live Updates. We’ll go through your questions. And will see how we can improve the manual.

Thanks for your feedback!

[Johan_Beck-Noren]

Great questions! I’ll try to answer them one by one.

I didn’t get the point for this zip mode. What’s the difference to the AWS mode? Can the engine do the check and start updating the manifest file from a local zip file? Or the AWS is the only place to hold this liveupdate manifest file currently? This is important for me. If yes, I can do a complete live update test offline using this zip mode.

The API for Live update does not make any assumptions on how or where to store your excluded resources. The Amazon AWS integration is there as a convenience if you prefer to server the resources on S3, but you are free to unpack the zipped resources and serve them anyway you like. I often spin up a local web server when testing stuff locally.

How does the engine check this version? I assumed that the engine first takes a look at the project live update setting and knows that another live update manifest file(local file or url) is present. The engine decides whether the live update manifest file shall be download(or copy) according to the version. There is a tip about the fake code from the manual page.

Basically, the version file writes the signature of the bundled manifest the first time the app starts. For every subsequent app start we check if the bundled manifest signature matches the signature written to file. If they don’t match it means the bundled app has changed (been updated). It is simply some logic to check if an app update has occurred and in that case to purge any locally store manifest file and fall back to the bundled manifest.

The engine decides whether the live update manifest file shall be download(or copy) according to the version. There is a tip about the fake code from the manual page.

It is up to the user to supply a new manifest to the engine. We do not download a new manifest for you, you must host it yourself and download in (for example with http) and supply it with resource.store_manifest.

If the live update manifest file is dowloaded successfully, is it stored in another path without overriding the bundle one via the resource.store_manifest API? If not, do the engine get the correct file depending a special search path priority?

Yes, any stored manifest and liveupdate resources are persisted on local storage. The bundled manifest and resources resides inside the APK on Android for example, which we can’t really modify or overwrite in runtime. At engine start, we check the local storage for any manifest file and load that one instead of the bundled one.

If the key and supported version is not filled in the project settings, what happens when call the resource.store_manifest API?

The callback supplied to resource.store_manifest will be called with the status argument telling you want went wrong.

How does this API collectionproxy.missing_resources(collectionproxy) calculate the missing file count? Check the hash of all resource in the bundle(live update if there is one)manifest file and try finding it? Will an changed resource with a new hash be treated as a missing file?

For every resource the collection needs to load, a lookup is done for the resource hash in the resource archive index. Any missing resource hash will be put in a list and returned. A modified resource will generate a different hash and therefore be treated as a different resource.

After getting the count of the files to be download, is it possible to get more information? Like the total and already downloaded size of the files which could be used to indicate the downloading progress? In the production environment, a single download might be failed due to the network satus. A retry mechanism as part of the error handling might be important.

Currently no, the only available information about a missing resource is its hash. But since the user is responsible for hosting the missing resources it should be possible to for example query your server about the size of a file. The API is pretty lean in that regard and leaves the hosting-and-storing totally up to the user.

Does the API resource.store_resource store the download file some where like the save-file path on different operating system? Or it just modify the bundled one?

Yes, the actual resource data is stored on local storage, the same path as any liveupdate manifest. On desktop it is the app data folder, on device it is on local storage.

If there a save path, when a excluded resource is updated for several times, will the old versions which do not match the latest live update manifest file be deleted automatically?

A modified resource will produce a new hash and will therefore be treated as a different resource. There is no state where we track different versions of the “same” resource, since they are not the same.

I have it in my TODO though to implement some kind of purging or unloading of liveupdate resources that might not be needed anymore or for some other reason needs to be removed from local storage. DEF-3114 tracks this feature.