Issues with pixel-perfect scaling (SOLVED)

[connor.halford]

First I’ll describe what I’m trying to do. I’m making a game with a GameBoy-style handheld case, with both the game itself and the case consisting of low-res pixel art. The gameplay area is 128x128 and the case design is 90x160. I’m trying to do pixel-perfect scaling for each of these independently, with the gameplay being a perfect multiple of 128 and the case having flexible relative proportions but a consistent pixel size. Here is a stripped-down example project and demo video:

scaling_bug.zip (12.5 KB)

As you can see the gameplay is always an exact multiple of 128x128 and stays within the bounds of the case. The proportions of the case are more flexible but the elements of the case (buttons, black borders, highlight, etc) all share the same ‘pixel’ size (which is a different ‘pixel’ size than the gameplay). The case letterboxes in both x and y if the window’s aspect ratio gets too extreme.

There are two issues I’m running into which may be in my code but seem to be in the engine, I’m not sure. I’ve tested on both Windows and Mac OS and both issues occur in each.

Issue 1: scale and/or position are sometimes off by one pixel
Here is what the stripped-down version looks like with a window size of 600x800 (see full-size image):

And here’s a close-up of the bottom-right corner:

There are three black rectangles here which each use the same 10x10 white sprite tinted black. The size of a ‘case pixel’ for this window size is 5, meaning that each ‘pixel’ in the case assets should correspond to a 5x5 square of actual screen pixels. Since the sprite is 10x10 that means the scale of a gameobject which is 1x1 in case pixels should be 0.5 to result in 5x5 screen pixels.

Both of the horizontal black bar gameobjects have a y scale of 0.5 and the top one is indeed 5 screen pixels tall but the bottom one is only 4 screen pixels tall. The sprites are both bottom-aligned (they have a half-height upwards displacement relative to the gameobject) and the bottom gameobject has a y position of 0, so there should be 5 pixels visible. Either the size or position is off by one pixel.

The vertical black bar is correctly 5 pixels wide, but either the vertical bar is one pixel to the left of where it should be or the orange, brown and horizontal black bars are all one pixel wider than they should be. The project prints the position, scale and size of each case element to the log, but if you run it and look at the numbers they all seem correct. So again either the size or position is off by one pixel. I tried using a 1x1 white sprite to avoid fractional scales but got the exact same results.

Issue 2: parts of sprites go semi-transparent under extreme non-uniform scaling
Here’s a window size of 400x600 (see full-size image):

You can see that not only are there several size and/or position off-by-one errors, but for some reason the black bars start to go semi-transparent. The sprite is 10x10 and the gameobject scale of the horizontal ones here is 33.75x0.30000001192093 for a size in screen pixels of 338x3.

[Pkeod]

What do your sprite materials look like?

[connor.halford]

The materials and render script are included in the project zip file, but I guess I might as well paste what I can here too. The materials are just duplicates of the builtin sprite material and use the builtin vertex and fragment shaders. The case assets are positioned and aligned in an initial 90x160 collection and the relevant data from that is cached on startup. The render script looks like this:

g_virtual_width = 0
g_virtual_height = 0
g_actual_width = 0
g_actual_height = 0

g_viewport_size = 0
g_viewport_left = 0
g_viewport_bottom = 0

local function update_window_size(self)
	self.ever_updated_window_size = true
	g_virtual_width = render.get_width()
	g_virtual_height = render.get_height()
	local prev_width = g_actual_width
	local prev_height = g_actual_height
	g_actual_width = render.get_window_width()
	g_actual_height = render.get_window_height()
	if prev_width ~= g_actual_width or prev_height ~= g_actual_height then
		msg.post("default:/case#case", "on_window_resized")
	end
end

function init(self)
	self.pre_game_predicate = render.predicate({"pre"})
	self.game_predicate = render.predicate({"tile"})
	self.post_game_predicate = render.predicate({"post"})
	self.view = vmath.matrix4()
	self.ever_updated_window_size = false
	self.load_delay = 0.05
	self.startup_time = socket.gettime()
end

function update(self)
	-- setup render state
	render.set_stencil_mask(0xFF)
	render.set_depth_mask(false)
	render.disable_state(render.STATE_DEPTH_TEST)
	render.disable_state(render.STATE_STENCIL_TEST)
	render.disable_state(render.STATE_CULL_FACE)
	render.enable_state(render.STATE_BLEND)
	render.set_blend_func(render.BLEND_SRC_ALPHA, render.BLEND_ONE_MINUS_SRC_ALPHA)
	render.clear({
		[render.BUFFER_COLOR_BIT] = col(g_palettes[g_save_data.palette].background),
		[render.BUFFER_DEPTH_BIT] = 1,
		[render.BUFFER_STENCIL_BIT] = 0
	})

	-- don't render anything until the case and viewport have been set up
	if not self.ever_updated_window_size then
		return
	elseif self.load_delay > 0 then
		if socket.gettime() - self.startup_time >= self.load_delay then
			self.load_delay = 0
		else
			return
		end
	end
	
	-- draw pre-game case elements
	local case_projection_matrix = vmath.matrix4_orthographic(
		0, g_actual_width,	-- left, right
		0, g_actual_height,	-- bottom, top
		-1, 1)				--near, far
	render.set_projection(case_projection_matrix)
	render.set_viewport(0, 0, g_actual_width, g_actual_height)
	render.set_view(vmath.matrix4())
	render.draw(self.pre_game_predicate)

	-- draw game
	render.set_projection(vmath.matrix4_orthographic(0, GAME_SIZE, 0, GAME_SIZE, -1, 1))
	render.set_viewport(g_viewport_left, g_viewport_bottom, g_viewport_size, g_viewport_size)
	render.set_view(self.view)
	render.draw(self.game_predicate)

	--draw post-game case elements
	render.set_projection(case_projection_matrix)
	render.set_viewport(0, 0, g_actual_width, g_actual_height)
	render.set_view(vmath.matrix4())
	render.draw(self.post_game_predicate)
end

function on_message(self, message_id, message)
	if message_id == hash("set_view_projection") then
		self.view = message.view
	elseif message_id == hash("window_resized") or message_id == hash("update_window_size") then
		update_window_size(self)
	end
end

And the case resizing script looks like this:

-- NOTE: this script represents just the relevant parts from several scripts in a much
-- larger project with many more assets and case parts. Excuse the mess.

g_save_data = {}
g_save_data.palette = 1

local colors = {
	[0] = vmath.vector4(  0/255,	  0/255,	  0/255,	1),
	[1] = vmath.vector4( 29/255,	 43/255,	 83/255,	1),
	[2] = vmath.vector4(126/255,	 37/255,	 83/255,	1),
	[3] = vmath.vector4(  0/255,	135/255,	 81/255,	1),
	[4] = vmath.vector4(171/255,	 82/255,	 54/255,	1),
	[5] = vmath.vector4( 95/255,	 87/255,	 79/255,	1),
	[6] = vmath.vector4(194/255,	195/255,	199/255,	1),
	[7] = vmath.vector4(255/255,	241/255,	232/255,	1),
	[8] = vmath.vector4(255/255,	  0/255,	 77/255,	1),
	[9] = vmath.vector4(255/255,	163/255,	  0/255,	1),
	[10] = vmath.vector4(255/255,	240/255,	 36/255,	1),
	[11] = vmath.vector4(  0/255,	231/255,	 86/255,	1),
	[12] = vmath.vector4( 41/255,	173/255,	255/255,	1),
	[13] = vmath.vector4(131/255,	118/255,	156/255,	1),
	[14] = vmath.vector4(255/255,	119/255,	168/255,	1),
	[15] = vmath.vector4(255/255,	204/255,	170/255,	1)
}

function col(n)
	return colors[n % 16]
end

g_palettes = {}
g_palettes[1] = {
	case = 9,
	shadow = 4,
	highlight = 10,
	outline = 0,
	background = 15,
	screen_surround = 0,

	button_enabled_top = 7,
	button_enabled_bottom = 2,
	button_disabled_top = 6,
	button_disabled_bottom = 1,
	button_down_top = 5,
	button_down_bottom = 1,

	button_icon_disabled = 5,
	button_icon_confirm = 11,
	button_icon_undo = 12,
	button_icon_back = 8,
	button_icon_arrows = 9
}

-- case constants
GAME_SIZE = 128
local MIN_CASE_BORDER = 24
local MAX_SCREEN_HEIGHT_GAME_CAN_OCCUPY = 0.55
local VIEWPORT_CENTRE = 2 / 3	-- how far up the case to centre the viewport
local CASE_WIDTH = 90
local CASE_HEIGHT = 160
local MAX_ASPECT_RATIO = 16 / 9
local BOUNDARY_ASPECT_RATIO = 4 / 3
local SURROUND_SIZE = 2

local case_bounds = {}
local case_parts = { -- from back to front
	{ name = "case", palette = "case" },
	{ name = "highlight", palette = "highlight" },
	{ name = "screen_surround", palette = "screen_surround" },
	{ name = "s_fill", palette = "shadow" },
	{ name = "w_outline", palette = "outline" },
	{ name = "n_outline", palette = "outline" },
	{ name = "e_outline", palette = "outline" },
	{ name = "s_outline", palette = "outline" },
	{ name = "s_edge_outline", palette = "outline" },
	{ name = "btn_down_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_down_top", palette = "button_enabled_top" },
	{ name = "btn_down_icon", palette = "button_icon_arrows" },
	{ name = "btn_left_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_left_top", palette = "button_enabled_top" },
	{ name = "btn_left_icon", palette = "button_icon_arrows" },
	{ name = "btn_up_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_up_top", palette = "button_enabled_top" },
	{ name = "btn_up_icon", palette = "button_icon_arrows" },
	{ name = "btn_right_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_right_top", palette = "button_enabled_top" },
	{ name = "btn_right_icon", palette = "button_icon_arrows" },
	{ name = "btn_back_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_back_top", palette = "button_enabled_top" },
	{ name = "btn_back_icon", palette = "button_icon_back" },
	{ name = "btn_confirm_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_confirm_top", palette = "button_enabled_top" },
	{ name = "btn_confirm_icon", palette = "button_icon_confirm" },
	{ name = "btn_undo_bottom", palette = "button_enabled_bottom" },
	{ name = "btn_undo_top", palette = "button_enabled_top" },
	{ name = "btn_undo_icon", palette = "button_icon_undo" },
	-- etc
}

local function calculate_viewport()
	-- work out the largest multiple of GAME_SIZE we can fit in the window while obeying MIN_CASE_BORDER size
	local best_size = GAME_SIZE * math.max(1, (math.floor(case_bounds.width / GAME_SIZE)))
	if case_bounds.width - best_size < 2 * MIN_CASE_BORDER then
		best_size = math.max(GAME_SIZE, best_size - GAME_SIZE)
	end

	-- don't let the size of the game occupy too much vertical space as we need room for the buttons
	while best_size / case_bounds.height > MAX_SCREEN_HEIGHT_GAME_CAN_OCCUPY and best_size > GAME_SIZE do
		best_size = math.max(GAME_SIZE, best_size - GAME_SIZE)
	end

	-- calculate viewport
	g_viewport_size = best_size
	local border = (case_bounds.width - g_viewport_size) / 2
	g_viewport_left = case_bounds.x + border
	g_viewport_bottom = case_bounds.y + (VIEWPORT_CENTRE * case_bounds.height) - (g_viewport_size / 2)
	g_viewport_bottom = math.floor(0.5 + g_viewport_bottom)
	
	print("viewport: size "..g_viewport_size..", left "..g_viewport_left..", bottom "..g_viewport_bottom)
end

local function cache_case()
	for i = 1, #case_parts do
		local id = go.get_id(case_parts[i].name)
		case_parts[i].id = id
		case_parts[i].original_position = vmath.vector3(go.get_position(id))
		case_parts[i].original_scale = vmath.vector3(go.get_scale(id))
	end
end

local function resize_part(part, z)
	local pos = vmath.vector3(part.original_position)
	local scale = vmath.vector3(part.original_scale)
	local size = go.get(part.name.."#sprite", "size")
	local width = size.x * scale.x
	local height = size.y * scale.y
	
	-- normalize position then map to new bounds
	pos.x = case_bounds.x + (pos.x / CASE_WIDTH) * case_bounds.width
	pos.y = case_bounds.y + (pos.y / CASE_HEIGHT) * case_bounds.height
	pos.z = z

	-- calculate scale to fit desired dimensions
	scale.x = (width / size.x) * case_bounds.pixel_scale
	scale.y = (height / size.y) * case_bounds.pixel_scale
	if width == CASE_WIDTH then
		scale.x = case_bounds.width / size.x
	end
	if height == CASE_HEIGHT then
		scale.y = case_bounds.height / size.y
	end
	
	go.set_position(pos, part.id)
	go.set_scale(scale, part.id)
	
	print(part.name.." pos ("..pos.x..", "..pos.y..", "..pos.z.."), scale ("..scale.x..", "
		..scale.y..", "..scale.z.."), size ("..size.x..", "..size.y..")")
end

local function resize_case()
	-- letterboxing
	case_bounds.width = g_actual_width
	case_bounds.height = g_actual_height
	local aspect = case_bounds.width / case_bounds.height
	if aspect <= BOUNDARY_ASPECT_RATIO then
		-- portrait
		aspect = math.min(aspect, 1 / MAX_ASPECT_RATIO)
		case_bounds.width = case_bounds.height * aspect
		aspect = math.max(aspect, 1 / MAX_ASPECT_RATIO)
		case_bounds.height = case_bounds.width / aspect
	else
		-- landscape (TODO)
		aspect = math.max(aspect, MAX_ASPECT_RATIO)
		case_bounds.height = case_bounds.width / aspect
		aspect = math.min(aspect, MAX_ASPECT_RATIO)
		case_bounds.width = case_bounds.height * aspect
	end
	case_bounds.x = (g_actual_width - case_bounds.width) / 2
	case_bounds.y = (g_actual_height - case_bounds.height) / 2

	print("\n*\n**\n***\n**\n*\nresizing: window "..g_actual_width.."x"..g_actual_height.." ("
	..(g_actual_width / g_actual_height).."), bounds "..case_bounds.width.."x"..case_bounds.height
	.." ("..aspect..") @ ("..case_bounds.x..", "..case_bounds.y..")")
	
	
	local x_scale = math.floor(case_bounds.width / CASE_WIDTH)
	local y_scale = math.floor(case_bounds.height / CASE_HEIGHT)
	case_bounds.pixel_scale = math.max(1, math.min(x_scale, y_scale))
	print("pixel_scale "..case_bounds.pixel_scale)

	for i = 1, #case_parts do
		resize_part(case_parts[i], (i - 1) / #case_parts)
	end
end

local function resize_case_after_viewport()
	local surround = nil
	for i = 1, #case_parts do
		if case_parts[i].name == "screen_surround" then
			surround = case_parts[i]
			break
		end
	end
	if surround ~= nil then
		local pos = go.get_position(surround.id)
		pos.x = case_bounds.x + case_bounds.width / 2
		pos.y = case_bounds.y + case_bounds.height * VIEWPORT_CENTRE
		go.set_position(pos, surround.id)

		local size = go.get(surround.name.."#sprite", "size")
		local scale = go.get_scale(surround.id)
		scale.x = (g_viewport_size + 2 * SURROUND_SIZE * case_bounds.pixel_scale) / size.x
		scale.y = (g_viewport_size + 2 * SURROUND_SIZE * case_bounds.pixel_scale) / size.y
		go.set_scale(scale, surround.id)
	end
end

local function tint_case()
	print("tinting case: palette "..g_save_data.palette)
	
	local pal = g_palettes[g_save_data.palette]
	for i = 1, #case_parts do
		sprite.set_constant(case_parts[i].name, "tint", col(pal[case_parts[i].palette]))
	end
end

function init(self)
	msg.post(".", "acquire_input_focus")
	cache_case()
	tint_case()
	msg.post("@render:", "update_window_size")
end

function update(self, dt)
	
end

function on_message(self, message_id, message)
	if message_id == hash("on_window_resized") then
		resize_case()
		calculate_viewport()
		resize_case_after_viewport()
	elseif message_id == hash("on_palette_changed") then
		tint_case()
	end
end

function on_input(self, action_id, action)
	if action_id == hash("quit") then
		msg.post("@system:", "exit", { code = 0 })
	end
end

[Pkeod]

I couldn’t get 2 to reproduce but you could try to add a sampler to the materials to see if it help. It could also require extruding borders in the atlas. Edit: just tested, 2 is related to that, forgot that I added that before fully testing.

[britzl]

It seems like a very complex solution, but perhaps that’s the only way to achieve what you want?

My gut feeling tells me to create the case using a 9-sliced box node. Buttons as box nodes in the same gui. And the game area rendered separately on top of the case gui just like you have it rendered today.

[connor.halford]

Thanks, that fixed issue 2

[connor.halford]

It is a complex solution but this is my third or fourth attempt and has gotten closer to fully working than any other solution, the only thing missing now is a fix for issue 1 above. I have tried several GUI-based solutions but there was always something wrong. It wouldn’t be pixel-perfect, or there would be artifacts, or I wouldn’t be able to access the information I needed, or I couldn’t do letterboxing, or something else. The nice thing with this solution is that I have full control over where everything is and it’s completely future-proof for all possible aspect ratios and resolutions.

[connor.halford]

Finally had time to come back to this and I’ve fixed it. Partly it was my fault for not flooring scales/positions, but there was also something in there that felt like a rendering bug. I had a 1x1 texture on a sprite with a position offset of x = 0.5 relative to its gameobject, which had a world position of x = 0. Scaling the gameobject up was rendering 1 pixel less horizontally than I would expect, e.g. setting the gameobject x scale to 100 would only draw 99 pixels. I fixed it by making a 2x2 sprite with just the left two pixels set and giving the sprite x = 1. Now an x scale of 100 gives me 100 pixels on the screen, though I would expect the two approaches to be exactly the same. So it seems like maybe there’s a rounding error when a sprite has a fractional position? ¯\_(ツ)_/¯