Short Integers in Python

Python allocates integers automatically based on the underlying system architecture. Unfortunately I have a huge dataset which needs to be fully loaded into memory.

So, is there a way to force Python to use only 2 bytes for some integers (equivalent of C++ 'short')?

6 Answers

Nope. But you can use short integers in arrays:

from array import array a = array("h") # h = signed short, H = unsigned short 

As long as the value stays in that array it will be a short integer.

• documentation for the array module

Thanks to Armin for pointing out the 'array' module. I also found the 'struct' module that packs c-style structs in a string:

From the documentation ():

>>> from struct import * >>> pack('hhl', 1, 2, 3) '\x00\x01\x00\x02\x00\x00\x00\x03' >>> unpack('hhl', '\x00\x01\x00\x02\x00\x00\x00\x03') (1, 2, 3) >>> calcsize('hhl') 8 

Armin's suggestion of the array module is probably best. Two possible alternatives:

• You can create an extension module yourself that provides the data structure that you're after. If it's really just something like a collection of shorts, then that's pretty simple to do.
• You can cheat and manipulate bits, so that you're storing one number in the lower half of the Python int, and another one in the upper half. You'd write some utility functions to convert to/from these within your data structure. Ugly, but it can be made to work.

It's also worth realising that a Python integer object is not 4 bytes - there is additional overhead. So if you have a really large number of shorts, then you can save more than two bytes per number by using a C short in some way (e.g. the array module).

I had to keep a large set of integers in memory a while ago, and a dictionary with integer keys and values was too large (I had 1GB available for the data structure IIRC). I switched to using a IIBTree (from ZODB) and managed to fit it. (The ints in a IIBTree are real C ints, not Python integers, and I hacked up an automatic switch to a IOBTree when the number was larger than 32 bits).

You can use NumyPy's int as np.int8 or np.int16.

You can also store multiple any size of integers in a single large integer.

For example as seen below, in python3 on 64bit x86 system, 1024 bits are taking 164 bytes of memory storage. That means on average one byte can store around 6.24 bits. And if you go with even larger integers you can get even higher bits storage density. For example around 7.50 bits per byte with 2**20 bits wide integer.

Obviously you will need some wrapper logic to access individual short numbers stored in the larger integer, which is easy to implement.

One issue with this approach is your data access will slow down due use of the large integer operations.

If you are accessing a big batch of consecutively stored integers at once to minimize the access to large integers, then the slower access to long integers won't be an issue.

I guess use of numpy will be easier approach.

>>> a = 2**1024 >>> sys.getsizeof(a) 164 >>> 1024/164 6.2439024390243905 >>> a = 2**(2**20) >>> sys.getsizeof(a) 139836 >>> 2**20 / 139836 7.49861266054521 

Using bytearray in python which is basically a C unsigned char array under the hood will be a better solution than using large integers. There is no overhead for manipulating a byte array and, it has much less storage overhead compared to large integers. It's possible to get storage density of 7.99+ bits per byte with bytearrays.

>>> import sys >>> a = bytearray(2**32) >>> sys.getsizeof(a) 4294967353 >>> 8 * 2**32 / 4294967353 7.999999893829228