Parsing sheets - CSV to Lua table

You can easily export your data from Google Sheets or Microsoft Excel in CSV (comma-separated values) format, but not to a Lua table.

However parsing those files is very easy and I wrote a simple Lua script:

-- 1. Get file path (I put my file here:)
local filename = sys.get_application_path().."\\csv.csv"

-- 2. Create a matcher (set number of columns in your table:)
local columns = 5
local matcher = "%s*(.-)"
for i = 2, columns do
	matcher = matcher..",%s*(.-)"
end

-- 3. Parse the data to a key-value table (specify a key for each column):
local keyTable = {}
for line in io.lines(filename) do
	local a,b,c,d,e = line:match(matcher)
	keyTable[#keyTable + 1] = { key1 = a, key2 = b, key3 = c, key4 = d, key5 = e}
end
pprint(keyTable)
13 Likes

nice

3 Likes

Here is another CSV loader to Lua table solution with some more specific use cases. It also includes a sample of setting up a CSV file meant for localization text.

It can silently handle both types of common line endings in your CSV files.

It supports loading a CSV file bundles within your game as well as outside of the game - useful for localizers testing translations.

Not extensively tested in a real project, might have bugs. :smiley:

CSVLoader.zip (8.5 KB)

This is the kind of locale spreadsheet the test.script has a build locale table function for.

game_locale
(Note: Those are test strings from Google Translate not meant for production games)

The reason for this kind of spread sheet CSV usage is it makes group localization easier for certain kinds of games.

There are major group projects like PolyglotGamedev https://docs.google.com/spreadsheets/d/17f0dQawb-s_Fd7DHgmVvJoEGDMH_yoSd8EYigrb0zmM/edit#gid=310116733 which list a lot of well done translations of common game related phrases (the example isnā€™t setup to parse their CSV export directly but it would be easy to do).

Related this TODO would be to scan the CSV and list all of the UTF8 characters to be used when including in .font files. Maybe build an editor script which can scan a linked list of CSVs and update a list of .font files for the included glyphs so that builds are only as big as they need to be.

Related thread

@totebo did you ever share your CSV tools? Maybe we both made the same thing? :stuck_out_tongue:

5 Likes

Put up project here and added unique char list generator for the generated locale tables.

When ran from editor it will generate extra_chars.txt you would need to manually copy and paste this into your .font files for locale characters for now. I still want to make an editor script which can have a config file you right click and do for you with all of your .font files if you have many.

6 Likes

No, and they seem to keep evolving! The last version is very stripped down. I havenā€™t landed on anything I think is worth sharing, and now I may never need to, because your library looks great! Make an extension our of it!

3 Likes

Today, so far the most used version suited for my usage is:

function M.csv_to_lua_table_from_path(path)
    -- 1. Load resource
    local data = sys.load_resource(path)

    -- 2. Convert CSV into a Lua table
    local output_table = {}
    local i = 1
    for line in string.gmatch(data, "[^\r\n]+") do
	    output_table[i] = {}
	    for word in string.gmatch(line, '([^,]+)') do
            --simple boolean parsing
            if word == "TRUE" then word = true
		    elseif word == "FALSE" then word = false
		    end

            --simple num parsing
		    local num = tonumber(word) 
		    if num then word = num end

		    table.insert(output_table[i], word)
	    end
	    i = i + 1
    end

    return output_table
end

Works perfectly with Google Sheets and Excel csv. It is better than my first version, because it does not depend on number of columns. You can parse the table further however you like it :slight_smile:

EDIT: I added also an optional simple boolean and numerical parsing

8 Likes

Awesome, Iā€™ll need this soon enough :slight_smile: Thank you!

1 Like

Here is an alternative implementation, focusing on spreadsheets where the first row and column are made up of unique ids, which seems to be the most common use case for config files.

Here is an example of such a spreadsheet, where the first row has language codes as unique keys and the first column the translations IDs:

The cell content has a few extra checks in this one, to allow commas in cells for example. It also stores tables to allow tro retrieve data in the same order as the spreadsheet, or as key value pairs.

local insert = table.insert
local find = string.find
local sub = string.sub
local gsub = string.gsub
local gmatch = string.gmatch

local M = {}

local configs = {}

function M.load(config_id, filename)

	local rows = {}
	local row_ids = {}
	local row_key_values = {}
	local columns = {}
	local column_ids = {}
	local column_key_values = {}
	local matrix = {}

	-- Load file
	local csv_lines, error = sys.load_resource(filename)

	if error then
		return false
	end

	-- Convert csv lines to indexed table
	-- Nabbed from: https://github.com/libremesh/pirania/blob/master/pirania/files/usr/lib/lua/voucher/utils.lua#L13
	local indexed_rows = {}
	local row_index = 1
	for csv_line in gmatch(csv_lines, "[^\r\n]+") do
		csv_line = csv_line .. ','        -- ending comma
		local row = {}        -- table to collect fields
		local fieldstart = 1
		repeat
			-- next field is quoted? (start with `"'?)
			if find(csv_line, '^"', fieldstart) then
				local a, c
				local i  = fieldstart
				repeat
					-- find closing quote
					a, i, c = find(csv_line, '"("?)', i+1)
				until c ~= '"'    -- quote not followed by quote?
				if not i then error('unmatched "') end
				local f = sub(csv_line, fieldstart+1, i-1)
				insert(row, (gsub(f, '""', '"')))
				fieldstart = find(csv_line, ',', i) + 1
			else                -- unquoted; find next comma
				local nexti = find(csv_line, ',', fieldstart)
				insert(row, sub(csv_line, fieldstart, nexti-1))
				fieldstart = nexti + 1
			end
		until fieldstart > #csv_line
		insert(indexed_rows, row)
	end

	-- Create reference tables from indexed table
	local column_count = #indexed_rows[1]
	for row_index=1, #indexed_rows do

		local row_cells = indexed_rows[row_index]
		local row_id = indexed_rows[row_index][1]

		for column_index=1, column_count do

			local cell = row_cells[column_index]

			if cell ~= "" then -- Ignore empty cells

				-- To boolean
				if cell == "TRUE" then
					cell = true
				elseif cell == "FALSE" then
					cell = false
				end

				-- To number
				local num = tonumber(cell) 
				if num then
					cell = num
				end

				local column_id = indexed_rows[1][column_index]

				-- Rows
				if row_index > 1 then
					if column_index == 1 then
						table.insert(row_ids, row_id)
					else

						-- Indexed values
						if not rows[row_id] then
							rows[row_id] = {}
						end
						table.insert(rows[row_id], cell)

						-- Key values
						if not row_key_values[row_id] then
							row_key_values[row_id] = {}
						end
						row_key_values[row_id][column_id] = cell

					end
				end

				-- Columns
				if column_index > 1 then
					if row_index == 1 then
						table.insert(column_ids, column_id)
					else

						-- Indexed values
						if not columns[column_id] then
							columns[column_id] = {}
						end
						table.insert(columns[column_id], cell)

						-- Key values
						if not column_key_values[column_id] then
							column_key_values[column_id] = {}
						end
						column_key_values[column_id][row_id] = cell


					end
				end

				-- Matrix
				if row_index > 1 and column_index > 1 then
					if not matrix[row_id] then
						matrix[row_id] = {}
					end
					matrix[row_id][column_id] = cell
				end

			end

		end

	end

	configs[config_id] = {
		rows = rows,
		row_ids = row_ids,
		row_key_values = row_key_values,
		columns = columns,
		column_ids = column_ids,
		column_key_values = column_key_values,
		matrix = matrix,
	}

	return true

end

function M.get_indexed_rows(config_id)

	return configs[config_id].rows

end

function M.get_indexed_row(config_id, row_id)

	local rows = M.get_indexed_rows(config_id)
	return rows[row_id]

end

function M.get_indexed_row_ids(config_id)

	return configs[config_id].row_ids

end

function M.get_row_key_values(config_id, row_id)

	return configs[config_id].row_key_values[row_id]

end

function M.get_indexed_columns(config_id)

	return configs[config_id].columns

end

function M.get_indexed_column(config_id, column_id)

	local columns = M.get_indexed_columns(config_id)
	return columns[column_id]

end

function M.get_indexed_column_ids(config_id)

	return configs[config_id].column_ids

end

function M.get_column_key_values(config_id, column_id)

	return configs[config_id].column_key_values[column_id]

end

function M.get_cell(config_id, row_id, cell_id)

	if configs[config_id].matrix[row_id] and configs[config_id].matrix[row_id][cell_id] then
		return configs[config_id].matrix[row_id][cell_id]
	end

end

function M.get_matrix(config_id)

	return configs[config_id].matrix

end

return M
8 Likes

Thank you very much!

Unfortunately, I canā€™t make it workā€¦ :see_no_evil:

Iā€™m probably asking too much, butā€¦ could you share a super minimal example? (so itā€™d be easier to understand how to use it for noobs like me)

1 Like

Are you getting an error?

The code above works with the file loaded using sys.load_resource() which means the resources should be bundled inside the game archive using the custom resources option in game.project.

Also make sure you export your spreadsheet with comma as the value delimiter.

1 Like

This is the project that uses this CSV parsing. You can check out the read me file and there is an explanation about uploading the resource if this is the issue. If not, give us more info about what you did and what is not working :wink:

In my free time I will try to create a simple tutorial for this :blush:

To be honest, I was talking to @totebo but your example looks great! Thank you!!

I just took a quick look at this project, and I think I should be able to tweak/use it to suit my needs (game parameters more than localization for now - but same logics after all)

PS : but, who knowsā€¦ I may get back to you if I have one question or two :see_no_evil:

To be honest, nothing was working, I didnā€™t even know where to start.

But itā€™s ok with the use case / example provided by @Pawel .

Thanks anyway :slight_smile:

Sorry for the late response! Do let me know if you need an example after all.

I quickly toyed with the example provided by Pawel, it seems to be ok so donā€™t botherā€¦ but maybe soon :see_no_evil:

It would be nice to have an official / documented ā€œgame config managementā€ extension (not only for me, but anyone working on Defold), but what I found here is already super useful, thanks guys!

1 Like

Hi guys!

As mentioned above, I toyed with the polyglot thing as much as I could (considering my current skills), added a function etc. to make it suit my needs.

First I created a ā€œpureā€ gameplay config file:
image

And insert ā€œincome_rateā€ and ā€œupgrade_costā€ values into 2 tables:
image

And then display the content if these table (to make sure everything was ok):
image

As you can see, there is an issue: there is a ā€œshiftā€ between categories and values. The income rate displays the upgrade cost values, while the upgrade cost displays the upgrade time valuesā€¦

I tried to modify the specific part of the dedicated function but got error messages again, again and againā€¦ I must admit that Iā€™m not very comfortable with the polyglot functions. I adjusted things there and there but there are some parts that I just ā€¦ donā€™t understand.

Could someone help me understand whatā€™s wrong? (would be awesome) @Pawel @totebo @britzl

Here is a super minimal version of the project:

2 Likes

Hello, in your buildings.csv If you add nil to all blank comments should then work properly. Seems to be the culprit in shifting the values. No values are added to the ā€˜Commentsā€™ column and being skipped/shifted .

1 Like

The shift could be a cause of Polyglot specific file structure, I skipped one column there and this should not be a case in your example CSV, if you used exact functions. But Iā€™m on phone, so let me check it thoroughly tommorow morning :wink:

1 Like

It works fine now :tada: thank you!! (with ā€œnilā€ or anything else, the ā€œcommentsā€ cells just need to be filled indeed)

@Pawel => there is a possible improvement here (ā€œempty cellsā€ management), but in the short term itā€™s already suuuuuuper useful :+1:

2 Likes