Drinkin gasoline and wine

post zenmachine musings

Memcached Backend Engines

The memcached protocol is very known and implemented on many languages and frameworks. Its primitives are based around getting and setting values that internally are mapped to keys.

There are two kinds of protocol: ascii and binary. The recommended protocol is the binary protocol, which is modern and has room for new features.

I’ve been trying my hand at some memcached servers implementations in python, based on twisted and gevent, but looking for some answers regarding the protocol answer, I got a tip that the original memcached server is supporting backend engines through an ANSI C interface. In an insomniac week end I was able to hack around two examples: a filesystem based store and a redis based store.

The filesystem engine was based entirely on @trondn tutorial. The Redis engine I made based on my original plan for a python based memcached server. I’ve used Redis’ hashes to store data and attributes for each memcached key.

To create the hash, the engine issues a command like this:

HMSET key nkey data ndata flags exptime

The advantage of Redis commands being atomic is that INCR, DECR and methods like this are a given. It was a matter of using hiredis to map the right commands. One thing that I’d do differently is to implement a kind of connection pool and configuration options.

To test the backend compliance to the binary protocol I’ve used memcapable, which usually comes bundled with memcached. The engine must be compiled as a dynamic library so it can be loaded as memcached -E engine.so.

One of the things that I think that having a configurable memcached frontend is good for is to capture metrics. As many frameworks already have a memcached client, it’s easy to create an instance and increment/decrement counters on it.