Connect to Redis with aioredis and Python

This is the first part of a three-part series about Redis-based distributed caching. You can find the other parts from here.

Perhaps redis-py is the most extensively used Redis client for Python. However, there is no simple way to make non-blocking calls with redis-py. Entered aioredis, a library to provide an interface to Redis based on asyncio. From version 2.0, its public API has been re-written to match redis-py‘s implementation. You can install it by running this command.

pip install aioredis

Set up a Redis instance locally

To run all the test code in this article, we need a Redis instance. Fortunately, we can use Docker to run a local Redis instance.

docker run -d -p 6379:6379 --name local-redis -d redis

The command above will pull this image and start a container called local-redis. We can then connect to it using the following URL.

redis://localhost:6379

Optionally, we can also specify a configuration file to set up a password to protect our Redis instance.

docker run -d -p 6379:6379 --name local-redis -d redis redis-server <path to redis.conf file>

Create a connection to Redis

The simplest way to connect to Redis is to create a client bound to a connection pool.

redis = aioredis.from_url('redis://localhost:6379')

Redis uses the bytes format to store data. However, if we only deal with strings and we know the encoding, we can tell the client to decode all responses.

# Automatically decode all responses to string, using utf-8 encoding.
redis = aioredis.from_url(
    'redis://localhost:6379',
    encoding='utf-8',
    decode_responses=True)

And if our Redis instance is password protected, we can specify the password when creating the client.

redis = aioredis.from_url(
    'redis://localhost:6379',
    username=<a username>,
    password=<a password>)

There are also options to specify SSL certificate, port number, socket properties,… You can find the list here, just search for __slots__.

Write data into Redis with aioredis

Redis has multiple data types, and for each type, aioredis has a corresponding method to read/write its value. All methods are called on the redis client created above. Below are some of the most frequently used types.

String
List
Hash
Set
SortedSet

Write a string into Redis

This is the simplest case, we can use the set method.

await redis.set('string_1', 'Example')
await redis.set('string_2', 'Another Example')

Write a list into Redis

To insert a value at the head of a list, we use the lpush method. If the cache key does not exist, a new list will be created.

await redis.lpush('list_1', 'one')

We can also insert multiple values at once.

await redis.lpush('list_2', 'two', 'three', 'four') # List will be ['four', 'three', 'two']

Similarly, we can insert value(s) at the end of a list with the rpush method. Again, if the cache key does not exist, a new list will be created.

await redis.rpush('list_3', 'one')
await redis.rpush('list_4', 'two', 'four', 'six') # List will be ['two', 'four', 'six']

To insert a value before or after a value, we use the linsert method. If the cache key does not exist or we try to insert a value before/after a non-existent value then the operation will be a noop. And if the existing value is duplicated inside the list, the new value will be insert before/after the first hit.

redis.linsert('<cache key>', <BEFORE or AFTER>, '<existing value>', '<new value>')

For example.

await redis.linsert('list_4', 'BEFORE', 'four', 'three') # List will be ['two', 'three', 'four', 'six']
await redis.linsert('list_4', 'AFTER', 'four', 'five') # List will be ['two', 'three', 'four', 'five', 'six']

Write a hash into Redis

Although Redis has both a HSET and a HMSET command, aioredis has deprecated the hmset method. Because of this, we will use the hset method whether we want to insert one or multiple fields into a hash. In both cases, if the cache key does not exist then a new hash will be created.

To insert one field into a hash.

redis.hset('<cache key>', '<field>', <value>)

And below is the code to insert multiple fields into a hash. Notice that we defined a dictionary to hold all value-key pairs. That dictionary is passed to the mapping argument. We also don’t need to specify the field name anymore.

redis.hset('<cache key>', mapping=<dictionary>)

For example.

await redis.hset('hash_1', 'first_name', 'John')

values = {
    'last_name': 'Doe',
    'age': 100
}
await redis.hset('hash_1', mapping=values)

The hash will be.

1) "first_name"
2) "John"
3) "last_name"
4) "Doe"
5) "age"
6) "100"

Write a set into Redis

Because a set does not have order, we need just one method sadd to add values into a set. If the cache key does not exist then a new set will be created.

redis.sadd('<cache key>', <values>)

For example.

await redis.sadd('set_1', 'one', 'two', 'three', 'four', 'five')

Write a sorted set into Redis

The method we need is zadd. Because a sorted set has order, when inserting a new entry, we need both its value and its score. All these value-score pairs are stored in a dictionary. If the cache key does not exist then a new sorted set will be created.

redis.zadd('<cache key>', <dictionary>)

For example.

values = {
    'last': 100,
    'first': 1,
    'middle': 50
}
await redis.zadd('sorted_set_1', values) # The sorted set will be ["first", "middle", "last"]

Caveat when writing data into Redis

As mentioned before, Redis stores data in bytes format. Because of that, it will perform these following conversions before saving data.

If the data is already in bytes format, it will be written as-is into Redis.
If the data is a string, it will be encoded using the default encoding of the program. Then the encoded bytes will be written into Redis.
Otherwise, the data will be converted to its string representation, and then treated as a string.

For example, we may expect the following code to write a number into Redis.

await redis.set('want_a_number', 1)

But the actual value in Redis is the encoded string value of 1.

rs_bytes = await redis.get('want_a_number')
print(typeof(rs_bytes)) # bytes
print(rs_bytes.decode('utf-8')) # "1"

When we write a hash into Redis, not only the field values, but also the field keys will be stored as bytes.

Read data from Redis with aioredis

Just like when writing data, for each data type of Redis, aioredis has a corresponding method to read its value. We will reuse the values written in earlier sections as examples.

Read a string from Redis

Again, this is the simplest case. The method we need is get.

string_bytes = await redis.get('string_1')
string_value = string_bytes.decode('utf-8')
print(string_value) # "Example"

This method will return None if the cache key does not exist. And as mentioned when we were setting up the connection to Redis, our connection does not automatically decode the response from Redis. That’s why we need to manually decode the bytes response into string.

Read a list from Redis

To read all members between two indexes, we use the lrange method. Note that the index here can be negative, just like in Python slicing notion. And aioredis can also automatically handle IndexError. The result is an array.

members = await redis.lrange('list_4', 1, 3)
print(members)

# The result is:
# [b'three', b'four', b'five']

And to read and remove the last member in a list, we use the rpop method.

last = await redis.rpop('list_4') # last is b'six'
                                  # 'list_4' now only has four members

If the cache key does not exist then lrange will return an empty array, and rpop will return None.

Read a hash from Redis

To read just one field from a hash, we use the hget method. If the cache key or field name does not exist then hget will return None.

first_name = await redis.hget('hash_1', 'first_name')
print(first_name.decode('utf-8')) # John

To read multiple fields from a hash, we use the hmget method. We pass all field names as arguments, and the result is an array. If the cache key does not exist then the result is an array where all members are None. And if a field name does not exist then the corresponding member in the result will be None.

name = await redis.hmget('hash_1', 'first_name', 'last_name')
first_name = name[0].decode('utf-8')
last_name = name[1].decode('utf-8')
print(first_name, last_name) # John Doe

And to read all fields from a hash, we use the hgetall method. The result is a dictionary. If the cache key does not exist then the result is an empty dictionary.

data = await redis.hmget('hash_1')
print(data)

# The result is:
# {b'first_name': b'John', b'last_name': b'Doe', b'age': b'100'}

As mentioned earlier, not only the dictionary’s values are bytes, but the keys are also bytes. We must take this into account when trying to retrieve data from the dictionary.

first_name_key = 'first_name'.encode('utf-8')
first_name = data[first_name_key]
print(first_name.decode('utf-8')) # John

Read a set from Redis

To read all members of a set, we use the smembers method. The result is a set.

members = await redis.smembers('set_1')

To read and remove multiple members from a set, we use spop with a count argument.

three_members = await redis.spop('set_1', 3) # Three members will be picked at random
                                             # 'set_1' contains the other two members

If we only want to read some members from a set without removing them, we can use srandmember.

the_rest = await redis.srandmember('set_1', 2) # Two members will be picked at random
                                               # No members are removed from 'set_1'

For smembers, if the cache key does not exist then the result will be an empty set; while spop and srandmember will return an empty array. And if we try to retrieve more members than what the set has, the result will contain all members of the set.

Read a sorted set from Redis

To read all members between two positions, we use the zrange method. Just like with lrange method, we can use negative index here, and aioredis can handle IndexError automatically. The result is an array.

last_two = await redis.zrange('sorted_set_1', 1, -1)
print(last_two)

# The result is:
# [b'middle', b'last']

If we want to sort all members in descending order before reading, we can set the desc flag to True.

last_two = await redis.zrange('sorted_set_1', 1, -1, desc=True)
print(last_two)

# The result is:
# [b'middle', b'first']

In both cases, if the cache key does not exist, the result will be an empty array.

Use pipeline to send requests concurrently

Maybe you have realized the following example code can be improved.

await redis.set('string_1', 'Example')
await redis.set('string_2', 'Another Example')

Here, we made two non-blocking calls, and we waited for the first one to finish before starting the second one. But it is possible to start both calls at the same time, then wait for both of them to finish. Normally, this would be done with asyncio.gather.

tasks = []

first_string = redis.set('string_1', 'Example')
first_string_task = asyncio.ensure_future(first_string)
tasks.append(first_string_task)

second_string = redis.set('string_2', 'Another Example')
second_string_task = asyncio.ensure_future(second_string)
tasks.append(second_string_task)

await asyncio.gather(*tasks)

However, when the number of operations is known in advance, we can use aioredis‘s pipeline to simplify the code above.

async with redis.pipeline() as pipe:
    pipe.set('string_1', 'Example').set('string_2', 'Another Example')
    await pipe.execute()

Implementing a get_or_set method

When working with a cache, I find that a get_or_set method will come in handy. It does something like this.

Receive a cache key and a delegate as parameters.
Check if that key exists in the cache.
If that key exists then return its value.
Otherwise, execute the delegate to create a new value, cache it with the given key, then return that new value.

We can implement one such method with aioredis and Python.

async def get_or_set(client, key, delegate):
    # Use key to retrieve the hash from cache
    val = await client.hgetall(key)

    # If the result is not an empty dictionary
    if val:
        # Then that key already exists, return its value
        return val

    # Otherwise, execute the delegate to create a new value
    val = delegate()
    # Cache it
    await client.hset(key, mapping=val)

    # Then return that new value
    return val

The method above only works with data of type hash. We can use it like this.

import datetime

def load_person():
    return {
        'first_name': 'James',
        'last_name': 'Bonds',
        'age': 100,
        'time_stamp': str(datetime.datetime.now())
    }

rs = await get_or_set(redis, 'hash_get_or_set', load_person)
print(rs)

No matter how many times we execute the code above, the time_stamp field won’t change.

{b'first_name': b'James', b'last_name': b'Bonds', b'age': b'100', b'time_stamp': <The first time we execute that code>}

Conclusion

In this article, we’ve checked out the aioredis package and see how it can help us make non-blocking calls to a Redis server. In part two, we will try to do the same with C#.

Set up a Redis instance locally

Create a connection to Redis

Write data into Redis with aioredis

Write a string into Redis

Write a list into Redis

Write a hash into Redis

Write a set into Redis

Write a sorted set into Redis

Caveat when writing data into Redis

Read data from Redis with aioredis

Read a string from Redis

Read a list from Redis

Read a hash from Redis

Read a set from Redis

Read a sorted set from Redis

Use pipeline to send requests concurrently

Implementing a get_or_set method

Conclusion

Like this:

2 Thoughts on “Connect to Redis with aioredis and Python”

Leave a ReplyCancel reply

Set up a Redis instance locally

Create a connection to Redis

Write data into Redis with aioredis

Write a string into Redis

Write a list into Redis

Write a hash into Redis

Write a set into Redis

Write a sorted set into Redis

Caveat when writing data into Redis

Read data from Redis with aioredis

Read a string from Redis

Read a list from Redis

Read a hash from Redis

Read a set from Redis

Read a sorted set from Redis

Use pipeline to send requests concurrently

Implementing a get_or_set method

Conclusion

Share this:

Like this:

2 Thoughts on “Connect to Redis with aioredis and Python”

Leave a ReplyCancel reply