Drinkin gasoline and wine

post zenmachine musings

Python and GEvent

The last post took some time over cyclone and it wasn’t fair that I’ve mentioned gevent briefly. I’ve have been using this library both for quick prototypes, production code and system upgrades. It’s not an instant-evented-magic-to-crappy-code but it provides simple and solid primitives such as greenlets that enable the use of good libraries in a fashion manner.

For instance, the great kombu library, which provides abstraction over different messaging protocols is not available to twisted. Worst yet, the txAMQP library is not straight forward to use. At the mure project I wanted to come with a quick and simple agent network that communicated for a shared bus. I wasn’t worried about which kind of channel as long as I could prototype and run it quickly. It proved good because in short time I’ve implemented an EventEmitter clone inspired on node.js and a few days ago a bridge between python and node.js event emitters.

It could be done using twisted but I would have to shave the yak related to a common multi broker messaging or stick to a single message broker. Not a problem if I had it clear from the start what I wanted it to be. But having gevent helped a lot to leverage the common blocking libraries and to use greenlets as a thread abstraction.

mure/core.py link
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
    def add_worker(self, workername, callback):
        self.workers[workername].add(callback)
        def _listener():
            qname = Queue(workername, Exchange("exchange:%s" % workername, type='fanout'))
            while self._connected == False:
                print "waiting %s" % self._connected
                gevent.sleep(1)
            with BrokerConnection(self._transport_url) as conn:
                with conn.SimpleQueue(qname) as queue:
                    while True:
                        try:
                            message = queue.get(block=True, timeout=10)
                            if message:
                                self._execute_callbacks(workername, message.payload)
                                message.ack()
                                if self._connected == False: break
                        except:
                            pass
        gevent.spawn(_listener)

The last line spawn the function _listener which binds into an exchange queue. Each of these listeners take care of a communication channel and execute the callbacks associated to that worker name. Exchange, queue, worker are the samething applied in different contexts. These workers (@worker(‘name_of_worker’)) are stored in a hash, each item a list of listeners. This spans out the messages around the right recipients.

Another great gevent companion is bottle, a DSL for web programming. Its interface is clean and combining with gevent lets you quickly come with thin webservices interfaces. I’ve create an application called uurl - an url shortener, entirely based on gevent, bottle and readis. I’ve set to rewrite it from time to time, on different languages and frameworks to get a hang of their components and so far this is the cleanest implementation. It started as an WSGI service and later I’ve converted to gevent by simply changing servers, monkey patching all and using a redis connection pool.

Monkey patching is a technique that bottle uses to convert the original socket, threads and other python modules to be non-blocking using its greenlets and I/O loop, in a way that the code change is minimal from a blocking application.

uurl.py link
1
2
3
4
5
6
class GEventServerAdapter(ServerAdapter):
    def run(self, handler):
        from gevent import monkey
        monkey.patch_socket()
        from gevent.wsgi import WSGIServer
        WSGIServer((self.host, self.port), handler).serve_forever()

To close this post, I’d like to add that I usually compare this setup with Ruby/Sinatra/Thin. As a derivative of RestMQ I’ve created TinyMQ - a set of small implementations of RestMQ core ideas. This is a subject for another post but the whole message broker ran on less than 100 lines tinymq.py.

Comments