To make some progress in my attempts to use Tokyo Tyrant from PHP, I finally bit the bullet and created a standalone script with no dependencies that stress-tests the interface. It also works as an expanded example of how to use Tokyo from within PHP. You can download tokyotest.php here.
It includes both the Net_TokyoTyrant raw socket interface and my work-alike clone Net_HttpTyrant that uses HTTP instead. The main tokyo_test() function stores a large number of values, retrieves them and checks they're correct, and then deletes them, timing performance. Here's my findings based on my own experiments.
TURN OFF ULOG! I wish the <blink> tag still worked, it's that important. I only just discovered this as the root cause of log files will eating my hard drive. The default ttservctl script has the innocuous-sounding ulog option turned on by default. This update log holds details of every transaction you make with the server. This is great if you're replicating or need to do restores, but these files grow rapidly and can easily fill up your hard drive! It also helps performance quite a bit in my tests to disable it. Obviously if you need this kind of backup you can't turn it off, but you'll also need some strategy to avoid running out of space.
Keep connections open. The initial problem I hit was caused by opening and closing a lot of sockets rapidly. After a few thousand, either PHP or the system throws an error. Try these URLs on the script to test for the problem on your system:
tokyotest.php?interface=http&numberofkeys=40000&valuelength=8
tokyotest.php?interface=raw&numberofkeys=40000&valuelength=8&closeeveryop=true
Don't use the HTTP interface. HTTP is great for quick hacks, but it is tough to avoid opening and closing a lot of connections in PHP. I notice one of the Perl example scripts uses stay-alive to keep a persistent connection, doing the same in PHP might help a lot. HTTP is a lot more verbose than the raw sockets though, so if you're going to that much trouble it's probably simpler to use Net_TokyoTyrant instead.
Net_TokyoTyrant will truncate long values. An issue I haven't solved yet is that after a certain point, the current raw socket interface code will fail to store the end of long values. For example, if you store a 20,000 character string you'll only get back about 16,000 when you retrieve it. This isn't a Cabinet problem, since doing the same operation through HTTP works as expected. Here's a way to reproduce the problem:
tokyotest.php?interface=raw&numberofkeys=10&valuelength=100000
The same operation using the HTTP interface gives the expected results:
tokyotest.php?interface=http&numberofkeys=10&valuelength=100000
[Update- I've now tracked down what's going wrong, and have a fix for the PHP wrapper: http://petewarden.typepad.com/searchbrowser/2009/06/how-to-get-tokyo-tyrant-working-in-php.html ]
Comments