I’m writing this post as a brain dump and for future reference. I was doing research on Tokyo Cabinet and Tyrant today for an upcoming project.
Tokyo Cabinet - A Key Value database that’s stupidly fast
Tokyo Tyrant - A network layer on top of Tokyo Cabinet
Tokyo Dystopia - A full text search (inverted index) on top of Tokyo Cabinet
These include mostly the official C API docs. They are useful if you skim through them, not for the C definitions, but to get an idea of what is possible. You will get lots of sparse explanations, but it’s a start.
- Tokyo Cabinet: http://tokyocabinet.sourceforge.net/
- Tokyo Tyrant: http://tokyocabinet.sourceforge.net/tyrantdoc/
- Tokyo Dystopia: http://tokyocabinet.sourceforge.net/dystopiadoc/
- Official Ruby Lib (forked to github & gemified) http://github.com/careo/tokyocabinet-ruby
- Official Ruby Lib (forked to github & gemified) http://github.com/careo/tokyotyrant-ruby
- Rufus Library (FFI Based) http://github.com/jmettraux/rufus-tokyo/tree/master
- Datamapper Adapter http://github.com/makoto/dm-tokyo-cabinet-adapter
- Speed Comparison (At the bottom): http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/
More About Rufus Library:
- Rufus 0.1.5 Release Notes - http://jmettraux.wordpress.com/2009/02/13/rufus-tokyo-015-hail-to-the-tyrant/
- Rufus 0.1.9 Release Notes - http://jmettraux.wordpress.com/2009/02/27/rufus-tokyo-018-rufusedo/
Macports version of Tokyo Cabinet and Tokyo Tyrant are broken. The current state is that there are incompatible versions of cabinet & tyrant, and that there are Portfile problems as well, where the checksum doesn’t match the downloaded file. I eventually gave up, and just used the instructions at http://openwferu.rubyforge.org/tokyo.html to install it manually (the version numbers in that post are old, but otherwise accurate). Those instructions install it to /usr/local/bin, so you may need to add it to your path.
Key Value Store Theory:
- This post talks about Redis, but is a good introduction to the thinking needed for Key Value storage, as opposed to a relational approach. http://code.google.com/p/redis/wiki/TwitterAlikeExample
More Cabinet Docs:
- A quick intro into the storage and theory of Cabinet. Not much actionable content here, but useful background. http://www.scribd.com/doc/12016121/Tokyo-Cabinet-and-Tokyo-Tyrant-Presentation
Cabinet Storage Engines:
I’m still vague on the real differences in this section. I’ll expand out as I learn and understand more. What I do know is that B-Tree and Table have additional methods on them, and that Tyrant doesn’t support at least the B-Tree ones (at a C level, it’s not a problem with the Ruby bindings).
- Hash - Standard hash structure for storing a key.
- B-Tree - Tree structure to store keys. Allows for ordering, and multiple values.
- Table - ?? Allows storing of arbitrary keys, and indexes (like CouchDB?).
- ?? - I think array is another one, for fixed length values, but I haven’t gone into that one yet
Lightcloud is some sort of distributed layer on top of tyrant, which gives some horizontal scaling ability. I haven’t looked much further than that.
- Lightcloud - http://opensource.plurk.com/LightCloud/
- Lightcloud Ruby - http://github.com/mitchellh/lightcloud
- Django Tokyo Sessions - http://pypi.python.org/pypi/django-tokyo-sessions/0.1.0
That’s enough for now…. I’ll try to keep this updated as I learn.