Hello everyone!
We’ve been working a lot with two new and really big features called LiveUpdate and NativeExtensions. Since they will change, and improve, the engine substantially, we are focusing on getting them ready as soon as possible and therefore the release of 1.2.97 will come at a later date. I thought that I should share some information with all of you about the LiveUpdate feature, what it is, why it is and how it’s intended to work.
Background
The original problem behind the LiveUpdate feature is that (almost) every game grows over time as more content is added to it, and even though the Defold engine itself is really small, a complete game such as Blossom Blast Saga can easily require hundreds of megabytes of assets. During 2016 we improved the texture compression algorithms a lot and implemented webp compression to enable developers to reduce their bundle size. Though this mitigates the problem with a growing application size, it doesn’t solve it.
We first looked at the possibility to ship a complete game with reduced asset quality where you could add very compressed images and only a few mipmap levels with the original bundle, and then download the high resolution assets once the game has been launched. This would have mitigated the problem even further, but there would still be problems with other assets that can’t be compressed as easily. When discussing this approach with internal teams the idea was scrapped since artists wouldn’t want to show low quality assets and give the player an unpleasant experience in case there was no internet connection available, or a sudden, unexplained, increase of graphics quality in the middle of a game.
The advantage of shipping a complete game from the engines point of view is that there would be no “dangling references” to resources that couldn’t be loaded. This presents a quite serious problem in terms of testing since the functionality in a game would differ depending on which resources had already been downloaded when a script is executed, and the developer would experience constant race conditions depending on network, connectivity, bandwidth and device storage.
Because of this, we decided to integrate downloadable content to collectionproxies. Since a collectionproxy is not loaded when the engine launches, and a collectionproxy will either load completely, or fail to load at all, failure to download a subset of required assets will not affect the number of possible states in the game, and scripts will execute with the same result every time.
LiveUpdate
LiveUpdate enables the developer to exclude all resources that belongs to a specific collectionproxy. When the resources within a collectionproxy is excluded they will not be bundled with the game, but instead exported as separate files for each resource. The actual collectionproxy will still be included, which allows a script to post messages such as “load” and “enable” to that collectionproxy. This will allow a developer to bundle only the first few episodes, levels, dungeons or monsters with the game that is published, and then download additional content as the player progress through the game, which will allow for a very small, and fixed, initial download size.
Before loading a collectionproxy, it is now possible to check whether there are any resources missing from that collectionproxy. If there are any resources missing, an indexed table of hashes will be returned, which corresponds to the exported resources. These resources should then be downloaded and added to the data archive before attempting to load the collectionproxy.
Defold is currently identifying resources by their path, or URL. This means that if there would be a change to a specific resource with the URL /main/defold.script it wouldn’t be possible to determine which version of the script the developer intended to use, which as mentioned in the background, would cause problems during testing where some resources from version 1 was loaded, and some resources from version 2. Because of this we’ve changed the resource management system completely so that it now identifies resources based on cryptographic hashes of their content, rather than their URL. This means that every version of a game have a distinct set of resources that cannot be incorrectly matched with a modified version of the same resource.
Further more, with this distinct addressing system it is possible to send another index to the game which represents a new version, and through collectionproxies, update the game with new content that wasn’t available when the game was first bundled. This feature could be used to hotfix issues, enable A/B testing, and more. But more about that in a later post…
Finally, before showing some code, I should add that we aim to provide all of this functionality to every platform that Defold supports. So there will be no specific settings required for different platforms, just data.
Code example
Please be aware that the API will be subject to change
function init(self)
self.manifest = resource.get_current_manifest()
self.http_requests = 0
self.missing_resources = 0
self.target = nil
end
local function callback_download_resource(self, id, response)
if response.status == 200 then
if resource.verify_resource(self.manifest, response.response) then
resource.store_resource(response.response)
self.missing_resources = self.missing_resources - 1
else
-- The resource was corrupted during transmission
end
else
-- The download did not succeed
end
self.http_requests = self.http_requests - 1
if self.http_requests == 0 then
-- All requests to download resources have been completed
local target = self.target
self.target = nil
if self.missing_resources == 0 then
-- We are ready to load the collectionproxy
msg.post(target, "load")
else
-- We've failed to download one or more resource(s)
end
end
end
local function download_resource(self, target)
local missing_resources = collectionproxy.missing_resources(target)
if #missing_resources ~= 0 then
self.http_requests = #missing_resources
self.missing_resources = #missing_resources
self.target = target
for k, v in pairs(missing_resources) do
local url = "http://example.com/defold/" .. v
http.request(url, "GET", callback_download_resource)
end
end
end
function on_message(self, message_id, message, sender)
if message_id == hash("download_content") then
if self.target == nil then
download_resource(self, message.collectionproxy)
else
msg.post(sender, "download_busy")
end
end
end
Regards,
The defold team through Jakob
