Sys.serialize() issue depending on msg.url?

Bugs
#1

Hello,

I use sys.serialize() to send data via tcp to another host and there, use sys.deserialize() to turn the payload back into Lua objects. Very straight forward.

When I have a msg.url() in the payload, however, the deserialize() function throws errors which seem to be dependent on the composition of the url. I cannot see a pattern, though.

For example, deserializing msg.url( “main”, “hello”, “frag” ) will throw an error every time.
On the other hand, deserializing msg.url( “foo”, “bar”, “baz” ) works just fine.

To illustrate the problem, here is some simple example code:

    	local tcp_client = require( "defnet.tcp_client" )
	local tcp_server = require( "defnet.tcp_server" )
	
	function init(self)
		msg.post(".", "acquire_input_focus")
		msg.post("@render:", "use_fixed_fit_projection", { near = -1, far = 1 })
	
		local onData = function( data, cip, cport, client )
			local cmd = sys.deserialize( data )
			pprint( cmd )
		end
	
		local nop = function() end
		
		self.server = tcp_server.create( 8888, onData, nop, nop )
		self.server.start()
	
		self.client = tcp_client.create( "127.0.0.1", 8888, nop, nop )
	end
	
	function on_input(self, action_id, action)
		if action_id == hash("touch") and action.pressed then
			local payload = {}
			-- FAILS: --------------------------------
			payload.url	= msg.url( "main", "hello", "frag" )
			-- ERROR:SCRIPT: defnet/tcp_server.lua:181: main/main.script:11: Table contains invalid type (nil) at element #0: 
			-- stack traceback: [C]:-1: ?
			-- defnet/tcp_server.lua:181: in function update
			-- main/main.script:47: in function <main/main.script:45>
			
			-- INSTEAD WORKS: ----------------------
			payload.url	= msg.url( "foo", "bar", "baz" )
			
			self.client.send( sys.serialize( payload ) .. "\n" )
		end
	end
	
	function update( self, dt )
		if self.client then self.client.update() end
		if self.server then self.server.update() end
	end

Am I doing something wrong or is this a bug in the serializer/deserializer functions? I use Defold version 1.9.1 on MacOSX.

Thanks in advance!

Cheers,
Lutz

#2

Can you please share a minimal project where this can be reproduced?

#3

Hi Britzl,

thanks for looking into this so quickly!

Here is the example code in a simple project: https://github.com/lharder/serializertest/

#4

I first encountered the problem in a more complex project. Not sure if it helps, but here is a crash file that seems to be caused by the same problem. Unlike the simple error in the demo project, the crash of the application happens only occasionally. I will still include the crash dump in case it is any good to you.

_crash.zip (2.5 KB)

#5

Oh, but hold on, the serialized data contains zeroes. You can verify this by printing the bytes:

		local data = sys.serialize( payload )
		for i=1,#data do
			print(i, string.byte(data, i))
		end

The tcp server will send all of the data but the server will read line by line and thus only read partial data and then try to deserialize it. There’s a bunch of solutions to this:

  1. Modify the client and server so that the client always send the length + data and the server reads the length and then all of the data instead of line by line (ie, instead of receive("*l") you do receive(length) )
  2. Encode the data so that it is guaranteed to not include any zeroes. You could for instance use base64 encode and decode using https://github.com/defold/extension-crypt
2 Likes
#6

ah! Yes - base64-encoding does the trick, indeed! Thanks a lot, that solves the problem!

I thought about the second option you suggest and would have preferred it to avoid having to encode/decode on top of serializing/deserializing data. But do I understand LuaScocket client:receive correctly that I would then have to send a fixed length of data on every request?

From the documentation:

client:receive([pattern [, prefix]])
‘*a’: reads from the socket until the connection is closed. No end-of-line translation is performed;
‘*l’: reads a line of text from the socket. The line is terminated by a LF character (ASCII 10), optionally preceded by a CR character (ASCII 13). The CR and LF characters are not included in the returned line. In fact, all CR characters are ignored by the pattern. This is the default pattern;
number: causes the method to read a specified number of bytes from the socket.

So, I would use “number” mode with the max possible value to setup server and client and then need to do some padding with every (shorter) request?

I would not like to close the connection after every request because I need to constantly send data. Reading line be line is no option either as you just correctly pointed out, so there remains only “number” mode with a fixed value? Or can I somehow set that individually per each request?

Thanks a lot for your support!

#7

No, my thought is to first read four bytes to get the data length and then read exactly the length number of bytes.

1 Like
#8

yeah, I get that. It seems an attractive idea to me. I just wonder how one would change the tcp_server to achieve that? It is based on Luasocket and there seem to be only 3 options (if I understand the documentation correctly, I have no experience with it myself):

  1. ‘*a’, client closing the connection after each request won’t do
  2. ‘*l’ does not work because it needs encoding and the whole point would be to avoid that
  3. number - (as I understand it, I am not sure) is fix for each request, or isn’t it?

What option would you select in the underlying Luasocket library to read requests of variable length - without closing the connection after the request, base-64-encoding the request or pad each request to a set length? Or do I miss your point?

#9

You need to do a couple of modifications to get it to work, but you would only be using option 3). I have reworked the tcp_client and tcp_server so that you can enable a binary mode, where data is sent and received with a prefixed length to each sequence of data sent. You can try it here:

The binary mode is enabled as an option here:

and here

4 Likes
#10

This is awesome! Thanks a lot for providing an entire working example, I very much appreciate it. I included it in my project and it works without any problems.

I would love to see you include that in the upstream version ;o)
Thanks for your support!