I’m writing this post as a brain dump and for future reference.  I was doing research on Tokyo Cabinet and Tyrant today for an upcoming project.

Tokyo Cabinet - A Key Value database that’s stupidly fast 
Tokyo Tyrant - A network layer on top of Tokyo Cabinet
Tokyo Dystopia - A full text search (inverted index) on top of Tokyo Cabinet 

Official Docs:

These include mostly the official C API docs.  They are useful if you skim through them, not for the C definitions, but to get an idea of what is possible.  You will get lots of sparse explanations, but it’s a start.

• Tokyo Cabinet: http://tokyocabinet.sourceforge.net/
• Tokyo Tyrant: http://tokyocabinet.sourceforge.net/tyrantdoc/
• Tokyo Dystopia: http://tokyocabinet.sourceforge.net/dystopiadoc/

Ruby Libraries:

• Official Ruby Lib (forked to github & gemified) http://github.com/careo/tokyocabinet-ruby
• Official Ruby Lib (forked to github & gemified) http://github.com/careo/tokyotyrant-ruby
• Rufus Library (FFI Based) http://github.com/jmettraux/rufus-tokyo/tree/master
• Datamapper Adapter  http://github.com/makoto/dm-tokyo-cabinet-adapter
• Speed Comparison (At the bottom): http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/

More About Rufus Library:

• Rufus 0.1.5 Release Notes - http://jmettraux.wordpress.com/2009/02/13/rufus-tokyo-015-hail-to-the-tyrant/
• Rufus 0.1.9 Release Notes - http://jmettraux.wordpress.com/2009/02/27/rufus-tokyo-018-rufusedo/

OSX Installation:

Macports version of Tokyo Cabinet and Tokyo Tyrant are broken.  The current state is that there are incompatible versions of cabinet & tyrant, and that there are Portfile problems as well, where the checksum doesn’t match the downloaded file.  I eventually gave up, and just used the instructions at http://openwferu.rubyforge.org/tokyo.html to install it manually (the version numbers in that post are old, but otherwise accurate).  Those instructions install it to /usr/local/bin, so you may need to add it to your path.

Key Value Store Theory:

• This post talks about Redis, but is a good introduction to the thinking needed for Key Value storage, as opposed to a relational approach. http://code.google.com/p/redis/wiki/TwitterAlikeExample
• http://antirez.com/post/Sorting-in-key-value-data-model.html

More Cabinet Docs:

• A quick intro into the storage and theory of Cabinet.  Not much actionable content here, but useful background. http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation

Cabinet Storage Engines:

I’m still vague on the real differences in this section. I’ll expand out as I learn and understand more. What I do know is that B-Tree and Table have additional methods on them, and that Tyrant doesn’t support at least the B-Tree ones (at a C level, it’s not a problem with the Ruby bindings).

• Hash - Standard hash structure for storing a key. 
• B-Tree - Tree structure to store keys. Allows for ordering, and multiple values.
• Table - ?? Allows storing of arbitrary keys, and indexes (like CouchDB?).
• ?? - I think array is another one, for fixed length values, but I haven’t gone into that one yet

Other Items:

Lightcloud is some sort of distributed layer on top of tyrant, which gives some horizontal scaling ability. I haven’t looked much further than that.

• Lightcloud - http://opensource.plurk.com/LightCloud/
• Lightcloud Ruby - http://github.com/mitchellh/lightcloud
• Django Tokyo Sessions - http://pypi.python.org/pypi/django-tokyo-sessions/0.1.0

That’s enough for now…. I’ll try to keep this updated as I learn.

This entry was posted on Friday, May 8th, 2009 at 6:18 pm and is filed under Programming, Ruby. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.