Thredis, the Peanut Butter Cup of Databases

Ever wonder what would happen if you dipped SQLite into Redis? You get Thredis, the peanut butter cup of databases:

Thredis embeds an in-memory SQLite database within Redis to enable a
sophisticated level of SQL (joins, sub-selects, etc, all supported),
as well as introduces multi-threaded processing to take advantage of
SMP systems.

This is one wild hack, check out one of the examples:

redis 127.0.0.1:6379> zadd foo_zset 1 a 2 b 3 c
(integer) 3
redis 127.0.0.1:6379> sql "create virtual table zset using redis (foo_zset)"
OK
redis 127.0.0.1:6379> sql "select * from zset"
1) 1) "key"
   2) "val"
2) 1) "1.0"
   2) "a"
3) 1) "2.0"
   2) "b"
4) 1) "3.0"
   2) "c"

More examples are listed at http://thredis.org/, source code is available at https://github.com/grisha/thredis.

Redis Hash Efficiency

Instagram on switching from plain key => value storage to hashes in Redis:

The size difference was pretty striking; with our 1,000,000 key prototype (encoded into 1,000 hashes of 1,000 sub-keys each), Redis only needs 16MB to store the information. Expanding to 300 million keys, the total is just under 5GB—which in fact, even fits in the much cheaper m1.large instance type on Amazon, about 1/3 of the cost of the larger instance we would have needed otherwise. Best of all, lookups in hashes are still O(1), making them very quick.

via Instagram Engineering • Storing hundreds of millions of simple key-value pairs in Redis.

The original key => value storage method used 70MB for 1,000,000 entries. Making their 300,000,000 target usage in the neighborhood of 21GB. Going from 21GB to 5GB is a huge win.

redisfs: Replication-Friendly Redis-based filesystem

redisfs: replication-friendly redis-based filesystem

Makes use of FUSE to provide a redis backed filesystem. The pattern matching support in the KEYS command would make lookups using that key layout reasonably efficient. Using hashes would be another way to go as well.

The basic redis features would make things like filesystem snapshots possible as well. This could turn into something very cool.