Data management in games with lots of items/enemies/objects

Wondering how you manage data if you are making a roguelike (with lots of items, enemies or weapons) RTSs (lots of units, buildings) civilization-like (again lots of units, maybe world tiles with effects…) or such (arcades with lots of data count too)

With LUA tables being so nice to work with, do you define items directly in LUA? Or use external files and parse them? How do you tie data and assets?

In Kaiju Control Force we only have like 14 enemies, so we have a module enemy_data that looks like this

But those are pretty simple, and the relation between then and their sprite atlas and animations is elsewhere, which I don’t like too much…

Would love to see how you handle that! Specially if your solution includes visual information. As in “this enemy uses this sprite” too!

1 Like

In my current game I am basically doing what you are doing. Nearly 3000 lines, and nowhere near done :slight_smile: Pray for me…!

Stuff that needs tweaked a lot (e.g. for balancing), I have placed in a spreadsheet which outputs a Lua-formatted line per unit which I can then paste into the table.

3 Likes

And how do you know which sprite goes where? Do does tables include the atlas and sprite/anim to use? Our enemies handle that in the animation code, tables only have animation events (so at certain frame something happens)

These things always differ game to game, but yeah, my table holds flipbooks too!

I am NOT saying this is the best way to organise things, but if you are looking for permission to create huge data tables - here it is :laughing:

regular_slime = {
        factory = "#regular_slime",
        shadow_size = "large",
        animations = {
            [STATES.IDLE] = "regular_slime_idle",
            [STATES.MOVE_TO_TARGET] = "regular_slime_move",
            [STATES.WAIT_TO_ATTACK] = "regular_slime_idle",
            [STATES.DYING] = "regular_slime_death",
        },
        is_enemy = true,
        sprite_offset = vmath.vector3(0,26,0),
        atlas = 1,
        target_priorities = {TARGET_TYPES.ATTACK_NEAREST_FRIENDLY, TARGET_TYPES.ATTACK_PLAYER, TARGET_TYPES.ROAMING},
        walk_speed = 100,
        attacks = {
            {
                --flipbook
                animation = "regular_slime_attack",

                --tags and attack mechanic
                attack_effects = { ATTACK_TYPES.MELEE },
                attack_damage_types = { { TAGS.MELEE, TAGS.PHYSICAL } },

                --animation timings
                attack_time = 7/18,
                attack_effect_times = { 4 / 18},

                --modifiers to base stats
                attack_cooldown_modifier = 1,
                attack_range_modifier = 1,
                damage_modifier = 1,
                knockback_modifier = 1,

                --sfx
                attack_sfx = sfx.sounds.claw_swipe,
                attack_sfx_speed = 1.25,
                attack_bark_sfx = sfx.sounds.flesh_attack_bark,
                attack_bark_chance = 0.35,

                --specific attack variables

            },
        },
        --sfx
        hit_sfx = sfx.sounds.flesh_hit,
        hit_sfx_speed = nil,
        hit_bark_sfx = sfx.sounds.slime_hit_bark,
        hit_bark_chance = 0.35,
        death_sfx = nil,
        death_bark_sfx = sfx.sounds.flesh_death_bark,
        death_bark_chance = 0.35,
        bark_speed = nil,
        --custom
        on_spawn = {
            {action = "start_pfx", pfx_id = "slime_trail"},
        },
        on_death = {
            {action = "spawn_units", unit_id = "small_slime", num = 5, is_friendly = false},
            {action = "spawn_fx", fx_id = "slime_gore"}
        },
        --spreadsheet
        damage = 10,
        attack_cooldown = 2,
        crit_chance = 0.05,
        crit_multiplier = 2,
        attack_range = 60,
        knockback = 300,
        knockback_resist = 3,
        health = 100,
        armor = 30,
        dodge = 0.025,
        threat_cost = 7,
        xp_drop = 1,
    },
1 Like

That looks perfectly fine to me xDD That’s not a big table, have you seen the tables in Cogmind? (My Cogmind Dev Tools / Grid Sage Games) :smiley:

1 Like

Some time ago, in a large strategy game, I worked on the quite complex vehicle production feature. The player would research various vehicle parts of different tech level, create a blueprint for a vehicle from these parts, assign it a name and visual representation, put it into production, and then have these vehicles be deployed to units on the map.

As you can imagine, there were tons of data that defined the gameplay characteristics of all these things, their relationships (upgrade paths, allowed conversions, usage restrictions etc.), generic art/audio/names/descriptions and the same but faction-specific.

Asset organization
What I would recommend is to organize assets in such a way that everything associated with a concrete character/enemy/weapon is found in one place, rather than having one directory with all the sprite pngs, one with all localization strings, one with balance values etc. In the above project, the assets and data were spread across many locations, which made it difficult to notice when one piece was missing or out-of-date. It also made it easy to make a mistake when adding or changing a vehicle.

If it’s easier to read assets when they are organized differently, I would consider creating a script that copies the raw assets into a read-optimized structure before building the game.

Lua tables as data format
I imagine that unless you have tons of data, writing data as Lua tables is a good choice; the code auto-complete can be quite helpful. You can generate Lua files with the “kind” tables seen in your screenshots based on your asset files and folders. However, if you intend to live-update your game, I suppose you will have to send the data as e.g. json, and at that point maybe it’s easier to just have everything in the same data format.

If you want to optimize for size and load-time, someone wrote a thread on here comparing json and Lua. If you want to go hog wild, check out flatbuffers, where you define a schema for your data, write and compile your data to binary using the schema, and then generate code for reading the binary data. It sounds more complicated than it is. The tools support generating both C++ and Lua, and I’ve used it successfully in Defold.

Asset linking
I like to have the file and folder structure determine which sprite and whatnot is used for what. E.g. if I add a frost_giant enemy to the game, I create a folder of the same name, add a definition.lua to contain the values of the enemy type, and add assets like a portrait.png and idle.ogg. If I later want to add a evolved_frost_giant type and reuse some of the definition and assets, I define it as inheriting from frost_giant so that assets will be grabbed from the original type.

Still, I haven’t worked out what I think is best to do with assets that are shared by many different entities and in many different contexts. What I’m saying is, the above doesn’t always work well.

Code generation
In the Defold project I’m currently working on, I generate Lua code for use in the game with custom Lua scripts external to Defold. This is useful in my case as I can define various entities concisely and then generate helper functions and tables based on these that work well with auto-complete in vscode. This also makes it much less time-consuming and error-prone when I refactor entities, as I only need to change the definition and then run the code generator script.

Faulty data checks
I would consider if it’s worth defining the expected fields of e.g. an enemy type definition, and the data type and valid value range for each field. If you have lots and lots of data this can be helpful as both documentation and a way to verify data after bigger changes.

From my experience, there are lots of issues that this type of value checks won’t find. As a result, I would also consider implementing an assets inspector in the game, where you can see e.g. all the enemies in the game, spaced out in a grid, and have UI to trigger all their animations etc. This is useful to see that they are loaded correctly and the asset linking works.

8 Likes

Woa, thanks a lot for the insight!

It all sound daunting, but even before you start coding! That’s a lot of information to keep track. I seriously envy those who can design games with so many moving parts involved! At some point I should try :smiley:

Yes, absolute try sometime! It’s the best way to learn. When designing and building a big system one iteration at a time, tweaking the design and refactoring the code as needed, it’s possible to build a surprisingly big house of cards before it collapses under its own weight. :joy: