Blog

Redis 4.0 hit its first “general availability” release recently and with it a long awaited Redis Cluster feature for Docker users: NAT/port-forwarding support. It’s finally possible to run Redis Cluster using Docker’s ephemeral ports. Now you can run Redis Clusters on Docker without needing to use host-mode networking or statically assigned ports.

tl;dr Redis Cluster 4.0 brings Docker network encapsulation support allowing for more dynamic deployments.

$ git clone https://github.com/aprice-/redisclustercompose
$ cd redisclustercompose
$ docker-compose up -d --scale redis=9

On Github

Check it out on github: https://github.com/aprice-/redisclustercompose

What changed in 4.0?

Three new configuration parameters have been added to Redis. They control what IP address, data port (e.g. 6379) and cluster port (e.g. 16379) Redis announces to other members of the cluster. The most important point is that you can specify each port independently, and Redis’ normal behavior of deriving the cluster port by adding 10,000 is no longer mandated.

• cluster-announce-ip: The IP address to announce.
• cluster-announce-port: The data port to announce.
• cluster-announce-bus-port: The cluster bus port to announce.

Why is this important?

Much of the allure of Docker is the flexibility in composing services in a dynamic, low-friction way. But two of the primary features that enable this, NAT/port-forwarding and network encapsulation, can not be used with Redis Cluster 3.0.

Redis Cluster 3.0 behind NAT can’t really work

By default, Redis Cluster 3.0 advertises its auto-detected local IP address and configured port. So, if you run Redis on the default bridge network, the internal private IP address and internal-only port will be advertised to peers and clients.

In this scenario, Redis nodes behind NAT will respond with MOVED redirections that clients might not be able connect to.

Redis Cluster 3.0 with host-mode networking workaround

The only option for Redis Cluster 3.0 on Docker is to use host-mode networking and statically define each node’s port. With host-mode networking, the container uses the host’s networking stack and the host’s public IP address is advertised. It is up to you to manage port collisions, though, Docker no longer manages any port forwarding.

If you want to run two, three or N Redis nodes on a given Docker host, each node needs a port chosen and defined in the node’s redis.conf. Given that the minimum recommended number of cluster nodes is at least six, it gets cumbersome quickly. It also causes Docker Swarm and Compose configurations to become verbose.

Redis Cluster 4.0 behind NAT

With Redis Cluster 4.0, there is a new option! Redis can now advertise an arbitrary IP address, port and cluster port allowing you to configure it to handle NAT scenarios Host-mode networking is no longer required. However, you do need to set the IP address, port and cluster port in the node’s redis.conf before starting the Redis process.

Now any MOVED redirections will have the correct IP and port! This is enough for anyone running Redis Cluster with a set of static configurations, but when using a cluster scheduler the necessity to set these configurations up front is an obstacle. The rest of this post investigates a discovery mechanism to enable dynamic provisioning of Redis Cluster nodes.

Discovering the NAT

If the container could itself determine the NAT configuration and set the corresponding redis.conf values, it would finally be possible to have a single, shared container and configuration for each node of the cluster. Service definitions in Docker Compose or Swarm and other clustering solutions can be collapsed to a single Redis container template. A Redis service can now be scaled (for example with docker-compose --scale)!

Since Docker is in charge of the randomly assigned port when the container launches, we have to query the Docker API to determine which port was assigned. There are a number of approaches that would work, but we’ll explore customizing the official Redis Docker image to integrate with a simple HTTP discovery agent.

1. The Agent

A thin HTTP wrapper around the Docker API makes more sense than having the full capabilities of the Docker socket unnecessarily exposed to every Redis container. There are also many other Service Discovery options available that you could use. Here’s an extremely simple custom one.

The agent container is a nodejs express application built on the official nodejs alpine Docker image. The only route available is a parameterized GET by container ID. The goal is to use the Docker API to find the dynamically assigned ports given to the requested container by its ID.

It uses request to query the Docker API via Docker’s unix socket. Docker’s unix socket is passed to the container as a Docker volume when launched. Getting information about the container from the Docker API is pretty straightforward:

request({
    method: 'GET',
    url: `http://unix:/var/run/docker.sock:/containers/${id}/json`,
    headers: {host: 'localhost'},
    json: true
}, (error, res, body) => {
    if (error) {
        callback(error, null);
        return;
    }
    if (res.statusCode >= 200 && res.statusCode <= 299) {
        callback(null, body);
    } else {
        callback(new Error(body.message));
    }
})

Now that we can get any container’s information from Docker, it’s just a matter of defining the express route and retrieving port information from the returned object.

app.get('/:id', (req, res) => {
    dockerInspect(req.params.id, (error, container) => {
        if (error) {
            res.status(400);
            res.send(error);
        } else {
            let portInfo = container.NetworkSettings.Ports["6379/tcp"];
            let cportInfo = container.NetworkSettings.Ports["16379/tcp"];

            if (portInfo && cportInfo) {
                let port = portInfo[0].HostPort;
                let cport = cportInfo[0].HostPort;

                res.send(`${clusterAnnounceIp}:${port}@${cport}`);
            } else {
                res.send("");
            }
        }
    });
});

The result is given in the same format of CLUSTER NODES in Redis 4.0, that is: <IP>:<port>@<cluster-bus-port>.

$ curl http://192.168.50.5:3000/2bddf88b0904

192.168.50.5:34176@34175

Our tweaked Redis container will use this information to set its NAT configuration.

Use the INTERFACE environment variable to select a network interface to announce if necessary

2. The Redis Container

Based on the alpine-variant of the official library/redis container, the Redis container overrides the ENTRYPOINT and on launch queries the discovery service to find it’s NAT configuration.

Since the discover agent uses host-mode networking to get the correct announce IP address, we can just query the Docker host from inside the container:

gateway=$(/sbin/ip route|awk '/default/ { print $3 }')          # Find the gateway IP address

network_info=$(curl -s http://${gateway}:3000/$(hostname))      # Query the discovery agent
                                                                # returns: 192.168.50.5:34176@34175

cluster_announce_ip=$(echo ${network_info} | cut -d ':' -f 1)   # Cut out the different parts of <IP>:<port>@<cluster-bus-port>`
ports=$(echo ${network_info} | cut -d ':' -f 2)
cluster_announce_port=$(echo ${ports} | cut -d '@' -f 1)
cluster_announce_bus_port=$(echo ${ports} | cut -d '@' -f 2)

Then, add the new cluster-announce-ip, cluster-announce-port and cluster-announce-bus-port configurations to the redis-server execution.

set -- redis-server "$@" "--cluster-announce-port ${cluster_announce_port}" "--cluster-announce-bus-port ${cluster_announce_bus_port}" "--cluster-announce-ip ${cluster_announce_ip}"

3. Forming the cluster

Once all members of the cluster are online with the discovered NAT configurations, the cluster can be formed. The official Redis Cluster ruby administration script is redis-trib.rb and an example of creating a cluster would be:

ruby redis-trib.rb create --replicas 1 <ip>:<port>[] 

But combined with the Docker cli, you can form a cluster of any size:

docker ps -q -f label=redis |                                   # loop over each container labeled 'redis'
{
  while read x; do                                              # inspect each container for its private IP address
    private_ip=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $x) 
    cluster_hosts=\"$cluster_hosts $private_ip:6379\"           # Append it to the list of cluster hosts
  done
  ruby redis-trib.rb create --replicas 1 $cluster_hosts         # Form the cluster
}

This will introduce each of the members to each other and set an initial hash slot ownership configuration. At this point the cluster is ready to be used!

With Docker Compose

With Docker Compose you can configure individual services that can then be scaled by launching additional instances of a given container. With Redis Cluster 3.0, it wasn’t possible to run multiple nodes under a single service due to the static configuration limitations described above. Using this discovery approach and Redis 4.0, it is now possible as each container can share an identical template.

redis:
  image: redis:discover
  build: ./redis
  command: --cluster-enabled yes --bind 0.0.0.0 
  ports: 
   - 6379
   - 16379
  labels:
   - redis
  depends_on:
   - discover

This makes it possible to use docker-compose up --scale redis=<num> which would launch any desired number of container instances of this service.

$ docker-compose up -d --scale redis=6


$ docker exec redisclustercompose_redis_1 exec redis-cli cluster nodes

48836a86da5d54d56393f866befde47efacc256d 192.168.50.5:34238@34237 master - 0 1504307943992 2 connected 5461-10922
53eb5604cf752e3811b5db01bddb1eb5d580d142 192.168.50.5:34244@34243 slave 48836a86da5d54d56393f866befde47efacc256d 0 1504307944996 5 connected
8b2b5e4eec8d25eab59bdedd0eb356465f3a4102 192.168.50.5:34246@34245 master - 0 1504307942000 1 connected 0-5460
dc71aa602eb6ab6f0d4c71208061b7230f53dc50 192.168.50.5:34240@34239 slave 8b2b5e4eec8d25eab59bdedd0eb356465f3a4102 0 1504307943000 4 connected
e1ab6ad9c5b806f564486b1bb8be2567244d9f5c 192.168.50.5:34242@34241 master - 0 1504307944000 3 connected 10923-16383
3b061c92f30154046c2eba2f858a81ff0c95d2c1 192.168.50.5:34236@34235 myself,slave e1ab6ad9c5b806f564486b1bb8be2567244d9f5c 0 1504307943000 6 connected

Bundled Reddie

Reddie is included with the Docker Compose code on github and is pre-configured to connect to your newly created cluster! Just navigate to https://localhost after Docker Compose finishes building the containers to check your creation out in all its glory.

Conclusion

For most Redis Cluster deployments on Docker, the new NAT/port-forwarding features just mean that it is now possible to change from host-mode networking to a bridge or overlay network.

But, if you’re scheduling Redis Cluster nodes using Swarm, Kubernetes, Nomad or any other cluster scheduler this post is an exploration of what it might look like to better integrate Redis Cluster by leveraging 4.0’s new features.

Unfortunately, the need to re-configure Redis at run-time using a discovery mechanism is a significant complexity overhead in achieving this. If it were possible in the future for Redis to dynamically re-configure itself (using inbound connection information? providing cluster bus port in client connections?) dynamic scheduling of Redis Cluster gets much easier.

Of course, a lot of parts are still missing: management of persistent storage, joining newly scheduled nodes to a cluster and re-sharding hash slots.