First off, I want to say how impressed I've been by the work Mikio Hirabiyashi has done on Tokyo Cabinet and Tyrant. I like it a lot and really, really wanted to use it as the data store for twitter.mailana.com. Since I decided it wouldn't work for me after trying it out, it seems worthwhile documenting my experiences. There's obviously plenty of people using it in production, so don't let me put you off trying it, I just want to leave a trail of bread-crumbs for anyone else hitting similar issues.
As I explained in my initial tutorial, getting started with Tokyo was straightforward. Once I'd got a basic Tyrant server running, I exported a 2 million row table from MySQL and wrote a PHP test-harness to load that into Tyrant. I chose an on-disk hash table for the store, since that promised an in-memory index for super-fast lookups from a key, and used the default localhost:1978 port.
For my first attempt I went with the HTTP interface. It took a bit of jiggery-pokery to persuade PHP's CURL to properly handle a custom PUT method, but I implemented it and ran the test. After a few thousand rows, it suddenly started taking almost exactly 2 seconds for each HTTP put. Looking at the timing I output to the error log, I'd see 1.998s, 2.001s, etc. This was very suspicious, since it smelt like a timeout, though the CURL library was set to timeout after 30 seconds, not 2.
I guessed this must be something wrong with CURL, so to get around it I wrote a pure socket-based version of the code, which wasn't too hard since both the requests and responses were very simple. Running it again, I started to see a PHP notice "errno=11 Resource temporarily unavailable" from fwrite after a few thousand insertions. Digging around the internet, this seemed like a pretty generic error, but it might be related to running out of socket resources. I was closing the file handle to the socket every time, and looking through the network tools on my Red Hat Fedora 8 system, I couldn't see a leak.
Since this seemed like it might be related to my use of TCP/IP, I switched Tokyo and my code over to using a Unix file-system socket instead. I still saw the same error.
To see if this was only a problem with the HTTP I pulled down the Net_TokyoTyrant PHP library that uses the raw binary socket interface and integrated that. This also allowed me to keep a single socket connection for my whole import, to avoid any resource leaking. Still no dice, I was seeing the same error.
At this point I was getting suspicious of PHP's socket handling. Both my original code and Net_TokyoTyrant were using the newer fsockopen() style stream-based sockets rather than the older socket_create() family of functions. I rewrote Net_TokyoTyrant to use the deprecated socket_* functions instead, and reran my test. Instead of hanging and printing a warning notice, it now just hung after a few thousand rows.
Looking for something else to try, I opened and closed the socket connection for every transaction, rather than keeping it open for the whole import. This actually worked! I left it running, and noticed that any of my values that were larger than 128,000 bytes (interestingly not 128*1024) were being truncated. Since only a few of my rows had BLOBs that large, I left the test running. When I came back to check on it a while later, I discovered that it had crashed. Looking into why, I found that the Tyrant log files had filled up the drive that held all my data. Every socket open and close was logged, and there were enough of them to fill gigabytes of drive space.
That was the point I decided to stop my evaluation, I'd run into more issues than I felt comfortable with. If there were just a couple I could have justified investing some debugging time and submitted patches or at least good reports. Instead I was obviously well off the beaten path, and getting Tokyo reliably working in my PHP/Fedora environment felt like it might take several more days.
It definitely wasn't wasted work though. As I'd been researching some of the issues I'd run across an intriguing note in the memcached FAQ, stating that the InnoDB engine for MySQL supported very fast primary key lookups. Since the main reason I was moving away from MySQL was painfully slow primary key fetches when using the default MyISAM engine on massive tables, this offered a solution. I kept all the other code I'd written to translate my more complex queries and analysis into a key/value store, and replaced the Tokyo get/put/out primitive functions with implementations that ran against an InnoDB table in MySQL.
This worked out great, I was able to speed up twitter.mailana.com a lot. Even better, I'm still in a good position to use Tokyo in the future if I am able to get it running reliably, I'll just need to swap out the lowest layer.
[Update - I've now switched back to Tokyo Tyrant after fixing the truncation bug in the PHP wrapper: http://petewarden.typepad.com/searchbrowser/2009/06/how-to-get-tokyo-tyrant-working-in-php.html ]
Comments