There is a hard limit to the message parameter table size. This limit is set to 2 kilobytes. There is currently no trivial way to figure out the exact memory size a table consumes but you can use collectgarbage(“count”) at before and after inserting the table to monitor memory use.
No, as others have pointed out already it is not possible to increase the buffer. I would suggest that you store your large datastructure in a lookup table of some kind inside a Lua module and in your message pass the key instead of the whole table, and on the receiving end you use the key to retrieve the large data structure from the lookup table.