Want a simple, persistent, key-value store in Go? Something handy to have in
your toolbox that’s easy to use and performant? Let’s write one!
Here’s what we’d like it to do:
Note that we want to directly store and retrieve Go values, much like
the fmt.Print and fmt.Scan functions. That should keep things simple for
the caller, by not requiring (de)serialization code for the types.
We also want the Put, Get and Delete methods to be goroutine-safe, so that the
calling code has one less thing to worry about.
Of course, we also expect the store to be fast and scalable too!
Step 1: Encoders
Encoders convert in-memory objects to byte streams.
They all have their strengths, but for our needs we’ll choose gob.
Gob is a built-in package (encoding/gob) that is performant and easy-to-use. You can
read more about gob here, and see some
other contenders here.
Using the gob encoding, you can encode a Go value into a byte slice like so:
Decoding is also equally easy:
Step 2: Persistent Storage
Probably the most widely used embedded storage library is
SQLite. It does have database/sql bindings for Go, but
being a C library it needs cgo and the compile times are frustratingly high.
LevelDB and RocksDB are two other
candidates, but perhaps a bit of an overkill for our use case.
We’ll settle on a pure-Go, dependency-free, memory-mapped B+tree implementation
called BoltDB. It can store key-value pairs
(with both keys and values being byte), segregated into “buckets”.
Here’s how you can open the database and create a bucket:
And within a bucket, you can store key-value pairs like this:
To fetch a particular record, you can seek using a cursor:
As you can see, BoltDB is fairly simple to use. You can read more about
Step 3: Putting the pieces together
Putting together BoltDB to manage the persistent store, and using the Gob
encoder to (de)serialize Go values, we get skv - the Simple Key-Value store!
It works exactly like we described above:
To import it into your project, use:
which will import and build both skv and boltdb.
The whole code is only one file, has tests, lives in GitHub,
and is MIT licensed. It even has documentation!
Feel free to poke around, fork the code, and use it as you will. Happy coding!