Bit operation on 64bit in lua

Questions
1

I am using bit for bitwise operation. If I understand correctly bit works only on 32 bit integers. I would like to ask your advice about how to use 64 bit bitwise operation in lua in Defold. More generally, is the internal integer type in lua a 32 bit integer or a 64 bit integer?

My apologizes for such basic questions.

Thanks!

2

I’ve wondered a lot of this question too, so I don’t have all the answers but I’ll share what I’ve figured out so far:

Is the internal integer type is lua a 32bit integer or 64bit integer

All numbers in Lua 5.1 are 64-bit, but they’re double floating points, so they can store 2^53 negative and positive integers (9,007,199,254,740,992) with full-precision. You can store higher numbers (up to 2^1023 more or less), at a loss of precision.

local max = 36028797018963968 -- 2^55
local sub = max - 5
print(max) --3.6028797018964e+16
print(max - sub) -- 4 (number was rounded)

For this reason, bit prefers to operate on 32-bit integers (and that they’re available for all platforms).

advice about how to use 64 bit bitwise operation in lua in Defold

I think it depends on what you’re trying to do. If you’re storing data or tracking states, the easiest answer is to split it up into multiple 32-bit integers if you can.

If you absolutely need 64-bit bitops, you’ll have to use the FFI: FFI Semantics. This does mean your project can’t be used in HTML5.

I haven’t done this in a while, so please forgive the representation here, but you could always use additional registers at the cost of one bit. (I’m sure someone has already made a lua module that does this)

1 | 2 -- registers
0111 | 0000  -- 7 in register 1. Add 2 to register 1
1001 | 0000 -- Overflow to (-)9. Add 1 to register 2 
1001 | 0001 -- (-)9 in register 1, 1 in register 2. Register 1 = bit.band(register1, register2) 
0001 | 0001 -- 9
2 Likes
3

What you can do is write an extension to do 64-bit bitwise operations. You would do something like this in the extension:

  1. Read the argument with lua_tonumber.
  2. Verify that it’s within the valid range of integers that double/number can represent, i.e. [0, 2^53) or [0, 9007199254740991.].
  3. Convert the value to an uint64_t.
  4. Do the bitwise operation.
  5. Verify that the resulting value is still within the valid range.
  6. Convert the value back to double/lua_Number.
  7. Push the value onto the stack with lua_pushnumber as the function return value.

If you want to check if a specific bit is set or not, you would have to implement your own function to do this, since the builtin bitop functions will discard the bits above 32.

However, if you only need to pass the value around in Lua while you do all operations on it in extensions, you can use all the 64 bits of the double. I do this for some 64-bit integer hashes I use as identifiers (I only compare if one hash is equal to another on the Lua side).

What you do in this case is instead of converting from and to double, you read the memory of the double as if it was a 64-bit integer (in C++ you would use reinterpret_cast). Essentially, the double becomes a container of 64 bits that Lua can’t directly make sense of, but your extensions can.

3 Likes
4

I think the key question is: What exactly are you using it for?
What’s the actual use case?

2 Likes
5

I did a quick search and came up with:

This is an archived repo with a note that “The bit libray in LuaJIT 2.1.0-beta3 supports 64-bit types”

So my question would be if you are sure the Defold version of LuaJIT and BitOp only supports 32 bit integers?

There is a note that the library “Supports different lua_Number types: either IEEE 754 doubles, int32_t or int64_t.”

1 Like
6

Thanks to @menaechmi @Mathias_Westerdahl @britzl @GooseSwanson

I want to use integers as bitmask, essentially for an ecs module. The ecs works perfectly fine with bit position 0 .. 31 but some errors appear if I use indexes above 31. So I think the bit module used by Defold only works for 32 bit integers as afar as my use case is concerned. Note also that some methods of the bit API here API reference (bit) explicitly limit the bit position to 0 .. 31.

@menaechmi I will try to understand what FFI semantic is.

@GooseSwanson Writing an extension would be indeed a solution. Thanks for the valuable hints!

@britzl Thanks for the link, I will try and see if it works for me.

7

Sorry, here’s a super quick example of how you might use the FFI:

local ffi = package.preload.ffi()
local bitmask = ffi.new("uint64_t", 13) -- creates a 64-bit int holding 0xD
local one_64 = ffi.new("uint64_t", 1) -- creates a 64-bit int holding 0x1
local max = ffi.new("uint64_t", 0xFFFFFFFFFFFFFFF) -- creates a 64-bit int holding 0xFFFFFFFFFFFFFF
print(bit.bor(bitmask, bit.lshift(1, 5))) -- 13 | (1 <<5) output: 45ULL
print(bit.lshift(one_64, 63)) -- output: 9223372036854775808ULL (2^63)

The important part of the semantics I linked is:

  1. Ensure the first number is a uint64_t, or the output will be converted.
  2. Don’t expect tobit() to work above 32 bits (to_hex works up to limits I mentioned above)
print(bit.lshift(one_64, 63)) -- output: 9223372036854775808ULL (2^63)
print(bit.lshift(1, 63)) -- output: -2147483648 (treated as a uint32_t)
2 Likes
8

@menaechmi Thank you very much! This is indeed very useful!