Drinkin gasoline and wine

post zenmachine musings

Porting an App From Tornado to Cyclone (Quick defer.inlineCallbacks/yield Primer)

Note: This will probably turn out in a screencast some time or another.

To port an app from Tornado to Cyclone is an easy task. There are some quirks tho related to Twisted mainly. This is necessary so you can enjoy the environment and async drivers in a proper manner. Tornado is know to provide good abstractions so you can use regular drivers to hook into its IOLoop. But as event loop goes, most of them requires that the whole stack is aware of the cooperative nature between its components. By using twisted you already have that and bundled with cyclone you have Redis, MongoDB, SQLite and other drivers for applications and protocols.

I’ve found a neat app called RedisLive and ported it to cyclone. It is basically a Redis real time resource monitor composed of two parts: a web interface and a daemon to collect data.

To keep it simple and within a reasonable ammount of code to be explained I’ve just ported the web interface and created a separated data provider. I’ve started a twisted based metric collector that helped me to fix and add some missing pieces to cyclone redis driver but I wanted to stick with the original collector.

I’ll walk out the main parts that were changed. The forked repository is at (my github)[https://github.com/gleicon/RedisLive]. There’s a full diff file inside my project.

A good approach to port a web application from Tornado to Cyclone is to tackle the web section, and this is what I did. Inside the folder RedisLive/src/api/controller lives the Controllers for each route listed at RedisLive/src/redis-live.py. So starting by redis-live.py:

diff --git a/src/redis-live.py b/src/redis-live.py
index 43479f4..9318c35 100755
--- a/src/redis-live.py
+++ b/src/redis-live.py
@@ -1,8 +1,8 @@
 #! /usr/bin/env python

-import tornado.ioloop
-import tornado.options
-import tornado.web
+from twisted.internet import reactor
+import cyclone.options
+import cyclone.web

 from api.controller.BaseStaticFileHandler import BaseStaticFileHandler

@@ -15,7 +15,7 @@ from api.controller.TopKeysController import TopKeysController

 # Bootup
-application = tornado.web.Application([
+application = cyclone.web.Application([
   (r"/api/servers", ServerListController),
   (r"/api/info", InfoController),
   (r"/api/memory", MemoryController),
@@ -27,6 +27,6 @@ application = tornado.web.Application([

 if __name__ == "__main__":
-   tornado.options.parse_command_line()
-   application.listen(8888)
-   tornado.ioloop.IOLoop.instance().start()
+  cyclone.options.parse_command_line()
+  reactor.listenTCP(8888, application, interface="")                 
+  reactor.run()

Basically it turned into a simple twisted application (not a TAC tho). I’ve changed package names from tornado.* to cyclone.* and switched from tornado.ioloop to reactor.listenTCP()/reactor.run(). The routing and class structure is the same as we are going to see in the Controllers. Most of the changes are package change (from tornado.web to cyclone.web) and by surrounding the drivers call with yield/defer.inlineCallbacks/defer.returnValue.

The regular way to call a method or function according to the twisted way(tm) is that you receive a Deferred class, in which you attach a callback for success and another for error. By using defer.* and yield code gets more readable without that many callbacks following each external call. The downside is that while you are using this decorator and returnValue, your function returns generators so it gets incompatible with normal functions. So you end up structuring your code around it to save time and in some places you got to stick to regular callbacks to make it compatible to libraries that are ported from a non-twisted code. Bikeshedding apart, is a good resource that Twisted provides.

diff --git a/src/api/controller/BaseStaticFileHandler.py b/src/api/controller/BaseStaticFileHandler.py
index 162fa62..6343a53 100644
--- a/src/api/controller/BaseStaticFileHandler.py
+++ b/src/api/controller/BaseStaticFileHandler.py
@@ -1,6 +1,6 @@
-import tornado.web
+import cyclone.web

-class BaseStaticFileHandler(tornado.web.StaticFileHandler):
+class BaseStaticFileHandler(cyclone.web.StaticFileHandler):
    def compute_etag(self):
        return None

In this case we didn’t need to change anything beyond module names. Now a bit of defer/yield:

diff --git a/src/api/controller/CommandsController.py b/src/api/controller/CommandsController.py
index cd9df26..ac046e3 100644
--- a/src/api/controller/CommandsController.py
+++ b/src/api/controller/CommandsController.py
@@ -1,12 +1,12 @@
 from BaseController import BaseController
-import tornado.ioloop
-import tornado.web
 import dateutil.parser
 from datetime import datetime, timedelta
+from twisted.internet import defer

 class CommandsController(BaseController):

+    @defer.inlineCallbacks
     def get(self):
         """Serves a GET request.
@@ -45,7 +45,7 @@ class CommandsController(BaseController):
           group_by = "second"

         combined_data = []
-        stats = self.stats_provider.get_command_stats(server, start, end,
+        stats = yield self.stats_provider.get_command_stats(server, start, end,
         for data in stats:
             combined_data.append([data[1], data[0]])

Note that before the “get” method we’ve added this decorator. If you are using other cyclone decorators, make sure this is the first one (for example if you are using cyclone.web.authenticated and so on). also when using the stats_provider, the call got prepended by yield so the decorator can capture the callback result and make it available to the variable stats without needing an explicit callback (stats = self.stats_provider(…).addCallback(lambda r: do_something(r))).

The rest of diff is at the repository, lets examine the redis stats provider. To be able to keep the original data collector I’ve cloned RedisLive/src/dataprovider/redisprovider.py into txredisprovider.py. Cyclone comes with txredisapi bundled as a package and I’ve wanted to use w/o having to rewrite all the calculation code. Ideally I’d have changed the whole class and collector. Most of the changed were addition of yield/defer.inlineCallbacks and specifics of cyclone drivers as transactions (started by driver.multi() instead of pipeline).

It was a quick job spread over two days (1h/2h each day) to have RedisLive working with cyclone, including the twisted part. Twisted is a very mature framework and it’s worth knowing the protocols it already provides. Also, the provided reactors and task primitives (look into redis-monitor-tx.tac) are useful to split heavy work into small tasks.