# Binary\_pack extension. A sys.serialize/deserialize replacement

**URL:** <https://forum.defold.com/t/binary-pack-extension-a-sys-serialize-deserialize-replacement/83557>\
**Category:** The Defoldmine\
**Created:** [November 19, 2026, 11:19am UTC](https://forum.defold.com/t/binary-pack-extension-a-sys-serialize-deserialize-replacement/83557 "2026-11-19T11:19:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_Fowler](https://forum-defold.b-cdn.net/user_avatar/forum.defold.com/andrew_fowler/32/43431_2.png) [@Andrew\_Fowler](https://forum.defold.com/u/Andrew_Fowler)\
**Post date:** [November 19, 2026, 11:19am UTC](https://forum.defold.com/t/binary-pack-extension-a-sys-serialize-deserialize-replacement/83557/1 "2026-11-19T11:19:07Z")

</div>

On my current project, I’ve been using sys.serialize() and sys.deserialize() to save and load level data. The recent MessagePack extension looked really interesting, but not supporting Defold userdata was problematic for my data structures. That inspired me to build my own C++ serializer for defold.

While it performs similarly to MessagePack on general data, its built with some additional specific features that lend themselves to save files, level data, and entity-style game state—including native Defold userdata support, key and hash deduplication, and built-in SHA-256 verification.

In most cases, it will outperform sys.serialize and sys.deserialize in both speed and output size (see the results section)

Github: [https://github.com/alfowler1976/defold\_binary\_pack](https://github.com/alfowler1976/defold_binary_pack)

### Key Features

- **Native Defold Userdata:** Encodes and decodes hash, vmath.vector3, vmath.vector4, vmath.quat, and vmath.matrix4 directly in C++—including hash values used as table keys. Note: URLs are not supported at the moment as they are runtime specific.

- **Smart Userdata Compaction:** Zero vectors (vector3(0,0,0)) packs down to 1 byte; 2D vectors (where z == 0) packs down to 9 bytes.

- **Number & Float Compaction:** Packs non-negative integers 0..127 into a single marker byte, scales integers down to uint8, int8, int16, or int32, and narrows lossless floats to 4-byte float32 instead of always writing 8-byte double.

- **Key & Hash Deduplication:** Repeated string keys and hashes are deduplicated on the fly into 2-byte reference (1 byte marker, 1 byte string reference), cutting down payloads on arrays of entity or level tables. Currently, the deduplication dictionary supports up to 256 references (1 byte storage). This is made possible by Lua’s string interning, which keeps it surprisingly fast.

- **Optional SHA-256 Verification:** Includes encode\_sha() and decode\_sha() for save files or payloads where you want built-in corruption and tamper detection. Due to the SHA routines, this does slow down the encoding/decoding process.

- **Memory & Safety:** Reuses a persistent C++ buffer (up to 256 KB) to cut down on allocations, while guarding against cycles, deep nesting (max depth 64), and truncated streams.

Note: It uses little-endian order, so there should be no compatibility issues on any little-endian devices.

### API

It only has 4 functions

- **binary\_pack.encode(tbl)** – Encodes a Lua table (including nested tables, primitives, and supported Defold userdata) into a compact binary string.
- **binary\_pack.decode(payload)** – Decodes a binary string created by `encode()` back into its original Lua table and userdata types.
- **binary\_pack.encode\_sha(tbl)** – Encodes a Lua table and appends a 32-byte SHA-256 checksum to detect corruption or tampering.
- **binary\_pack.decode\_sha(payload\_sha)** – Verifies the 32-byte SHA-256 checksum before decoding, throwing a Lua error if the payload has been corrupted or tampered with.

### Results

On one of my levels, which uses a lot of nested entity key tables, sys.serialize produced 29,406 bytes of data, whereas binary\_pack reduced this to 5,118 bytes—an 86% reduction.

As a comparison i ran the same tests provided on messagepack post (10 run average)

| Dataset | BinSize (B) | SysSize (B) | Saving | Enc Spd | Dec Spd | SHA Enc | SHA Dec | Verify |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10,000 ints (0-99) | 10,003 | 160,012 | 93.7% | 4.23x | 3.06x | 3.10x | 1.85x | PASS |
| 10,000 ints (1-10k) | 29,621 | 160,012 | 81.5% | 3.80x | 2.91x | 1.92x | 0.95x | PASS |
| 10,000 fractionals | 50,003 | 160,012 | 68.8% | 4.40x | 2.86x | 1.48x | 0.61x | PASS |
| 10,000 booleans | 10,003 | 70,012 | 85.7% | 4.55x | 2.94x | 3.17x | 1.63x | PASS |
| 1,000 short strings | 7,896 | 16,905 | 53.3% | 3.20x | 1.78x | 1.05x | 0.51x | PASS |
| 1,000 str-key ints | 10,516 | 23,616 | 55.5% | 1.94x | 1.28x | 0.80x | 0.51x | PASS |
| 10,000 str-key ints | 118,517 | 239,616 | 50.5% | 1.65x | 1.43x | 0.70x | 0.61x | PASS |
| 1,000 nested players | 27,490 | 84,013 | 67.3% | 2.62x | 2.11x | 1.36x | 1.14x | PASS |
