How to efficiently read and write a whole redis bitmap (in batch) using python¶

Normally, we use commands like GETBIT and SETBIT to read or modify a bitmap in redis. We also have corresponding APIs in python redis library (redis-py).

The problem is that, when reading or writing a big bitmap, if you use the native commands, it may cost too much time.

The solution is to read or write the bitmap in batch. That is to say, read and write the whole bitmap as a string!

Write a whole bitmap¶

Before we write the code, install related python libraries:

pip install redis bitarray

Here's the code to write a whole bitmap to redis:

import redis
from bitarray import bitarray

redis_ins = redis.Redis()

# the indexes that we want to save to the bitmap
indexes = [1, 2, 3, 13, 14, 15]

# use bitarray to convert indexes to a bitmap
arr = bitarray(max(indexes) + 1)
arr.setall(0)  # clear the bitmap
for idx in indexes:
    arr[idx] = 1  # set bit
print(arr)  # bitarray('0111000000000111')
data = arr.tobytes()

# save generated bitmap to redis
redis_ins.set('mykey', data)

In the code above, we utilize bitarray to covent indexes (offsets of bits) to a bitmap, then push it to the redis server once and for all. Now, we no longer need to execute SETBIT again and again.

Read a whole bitmap¶

Here's the code to read and parse a whole bitmap from redis:

import redis
from bitarray import bitarray

redis_ins = redis.Redis()

# read the whole bitmap from the redis server
data = redis_ins.get('mykey')

# parse the bitmap to bit offsets
arr = bitarray()
arr.frombytes(data)
indexes = arr.search(1)  # find bits
print(indexes)  # [1, 2, 3, 13, 14, 15]

In the code above, we read the whole bitmap from the redis server, and utilize bitarray to covent the bitmap to indexes (offsets of bits). Now, we no longer need to execute GETBIT again and again.