Drinkin gasoline and wine

post zenmachine musings

On That Message Queue...

…that you are trying to do with Redis, the answer is NO.

Not by any issue with Redis (which I use in a lot of projects and it’s a fine piece of software), but because the primitives are not enough to pass as a message broker.

Looking to Redis and ZeroMQ as a message broker in the classic ActiveMQ/RabbitMQ/RestMQ sense is naive, because both are transports (and in Redis case, persistence sometimes) and building blocks that can be attached to systems like these and libraries..

The regular drivers for Redis usually provides no reconnection in case of error or subscribe disconnect, so you probably might end up having your process hanging or if you are lucky, killed with an exception in case of a lost connection. The proper way to do it is to surround Redis with a management layer, as RestMQ or Resque does.

An interesting approach taken on the twisted redis client is to make use of connection pools which can reconnect. That was the single feature that made RestMQ possible. Kombu connections follow the same pattern, abstracting fan outs and routes over a connection pool to Redis, the simples fallback in case you dont want to install any other complex message broker.

Applying only to pub/sub channels without fallback is not a good idea specially if you rely on multiple consumers. On the other hand, for simple one-time no-distributed-locks messaging (as with udp multicas or service discovery), it might be a good choice because: a) there is no persistence and b) if the transport is not available, the processes(or agents) are idle.

The fine line here is the direction of the messages. In a exchange environment, the broker plays the good part of managing the transport shortcomings. On a fire and forget which the messages are importante, the broker helps to keep the message somewhere until it’s fetched. Fire and forget of disposable messages might be a good choice to do it if there is no immediate action to be taken and if there is some kind of retransmission/expiring on the sender part. Remember the SMTP protocol.

That’s not even touching the cases where a message queue is used as a kind of distributed lock where only one of many similar consumers might get a message, instead of a broadcast scenario.