Tag: redis

Thredis, the Peanut Butter Cup of Databases

Ever wonder what would happen if you dipped SQLite into Redis? You get Thredis, the peanut butter cup of databases:

Thredis embeds an in-memory SQLite database within Redis to enable a
sophisticated level of SQL (joins, sub-selects, etc, all supported),
as well as introduces multi-threaded processing to take advantage of
SMP systems.

This is one wild hack, check out one of the examples:

redis 127.0.0.1:6379> zadd foo_zset 1 a 2 b 3 c
(integer) 3
redis 127.0.0.1:6379> sql "create virtual table zset using redis (foo_zset)"
OK
redis 127.0.0.1:6379> sql "select * from zset"
1) 1) "key"
   2) "val"
2) 1) "1.0"
   2) "a"
3) 1) "2.0"
   2) "b"
4) 1) "3.0"
   2) "c"

More examples are listed at http://thredis.org/, source code is available at https://github.com/grisha/thredis.

Redis Hash Efficiency

Instagram on switching from plain key => value storage to hashes in Redis:

The size difference was pretty striking; with our 1,000,000 key prototype (encoded into 1,000 hashes of 1,000 sub-keys each), Redis only needs 16MB to store the information. Expanding to 300 million keys, the total is just under 5GB—which in fact, even fits in the much cheaper m1.large instance type on Amazon, about 1/3 of the cost of the larger instance we would have needed otherwise. Best of all, lookups in hashes are still O(1), making them very quick.

via Instagram Engineering • Storing hundreds of millions of simple key-value pairs in Redis.

The original key => value storage method used 70MB for 1,000,000 entries. Making their 300,000,000 target usage in the neighborhood of 21GB. Going from 21GB to 5GB is a huge win.

redisfs: Replication-Friendly Redis-based filesystem

redisfs: replication-friendly redis-based filesystem

Makes use of FUSE to provide a redis backed filesystem. The pattern matching support in the KEYS command would make lookups using that key layout reasonably efficient. Using hashes would be another way to go as well.

The basic redis features would make things like filesystem snapshots possible as well. This could turn into something very cool.

Redis Sharding at Craigslist

Redis Sharding at Craigslist

Tinkering With Redis

I like to tinker with different things from time to time. Not because it is something I’m using at work, or even using any where at all. Just to see what others are doing and hopefully learn a few things along the way. So here are a few notes about tinkering with Redis.

The short story

Created by Salvatore “antirez” Sanfilippo, Redis was first released in March 2009 and is under a BSD style license.

Superficially Redis is like memcached in that it is a memory based key/value store. But it is also much more than that. Data storage can be persistent and it provides additional data types and operations.

Getting started in less than 60 seconds

On a unix style system (Mac, Linux, FreeBSD, etc.) it takes less than 30 seconds to get Redis up and running:

  • curl -O http://redis.googlecode.com/files/redis-2.0.2.tar.gz
  • tar -xzvf redis-2.0.2.tar.gz
  • cd redis-2.0.2
  • make
  • ./redis-server

I really like that there are no external dependencies to worry about, just download, build and go. As an added bonus it comes with a CLI client as well, redis-cli.

Exploring

The redis-cli tool makes it easy to explore features of Redis without having to write any code. I recommend taking a few minutes to walk through the introduction to redis data types page. After a few minutes going through the examples there you’ll have a basic idea of the data types: string, integer (special type of string really), lists, sets, and sorted sets and a few of the operations available for each type. One data type that page doesn’t cover yet is hashes, which are exactly what they sound like.

A longer list of supported commands with links is available at http://code.google.com/p/redis/wiki/CommandReference, which is a good resource to explore while playing with Redis.

Beyond data storage

Fast access to data in formats that developers are familiar with is nice, but that isn’t the only thing Redis is good for. First, check out the blocking pop commands. This allows for one or more clients to block when popping data off an array. If there is data in the array they immediately get that, and if there isn’t they will get immediately notified when a new item is added. Now you’ve got a simple queueing system.

After letting your imagination wander a bit with blocking pop go check out the pub/sub commands. The publish/subscribe (pub/sub) features in Redis allow you to easily listen for updates on a channel. Other clients can then write updates to those channels. *Poof* you’ve now got a simple message passing system.

Back to the data

I mentioned persistence of data as something that Redis provides. There are two ways to do that, via snapshots and the append only file. Both of these allow Redis to reload data after a restart. Check out the redis.conf file for more details on how these two options work.

Replication is also available with a master/slave approach. There is a replication how-to, which is over kill because creating a new slave is wonderfully simple.

A more recent feature is virtual memory. This allows Redis to store more data than would fit in physical memory. Relying on the the virtual memory of the operating system wasn’t really an option because it would swap out the whole process, where as Redis can be smart about swapping out the least used data. A reasonable balance of speed with increased capacity.

More reading

I’ve really only scratched the surface of what Redis can do in these various areas. The documentation page is a good place to pick up more details. Simon Willison has a great set of slides from his Redis tutorial that I also recommend reading.

Why

Redis looks interesting not just for the feature set, but how it works as well. Operations on data are atomic, backups are done as atomic copies so making backups is as easy as copying a file. While there are still issues and trade offs in Redis (every system has them), there is also a general feeling of elegant simplicity that appeals to me.

Who knows, perhaps someday I’ll even use Redis for something besides tinkering.

© 2014 Joseph Scott

Theme by Anders NorenUp ↑