Jun
6

LevelDB was designed for mobile devices

By kellabyte  //  Databases  //  4 Comments

In the industry there is a lot of reuse of storage engines because writing a good, reliable and efficient one is a tough challenge and takes a lot of time to get right. If a database implements its own storage engine I’m likely to be skeptical of the implementation until it has been proven to be solid. It takes time to iron out all the bugs and if someone tries to sell you a solution on top of a storage engine they whipped up in a short amount of time, you should be skeptical if you want to trust it with your data. In general, I rarely want to be the trail blazer with these kinds of things in a production system.

In some of my database experiments on GitHub I used Google’s LevelDB as the backing store. LevelDB is generally a good storage engine but it’s important to remember it’s designed and developed for a mobile web browser. It’s a testament to its design that people are using LevelDB in databases running on servers but there are limitations and several people have had issues. Some of its design definitely has a mobile influence which doesn’t scale well to hardware with more capacity.

I know some companies who have made modifications to private forks and there are also other open source forks that make some significant changes to fix major issues and performance problems experienced in services and database workload environments.

If you’re thinking of using LevelDB on the server or are copying the LevelDB design/code to write a new storage engine, I highly recommend looking at these forks because they are much more suited for these kinds of workloads.

I’m aware of a dozen or so database or service related projects taking some influence from some of my experiments so I feel compelled to express this information :)

Basho has made significant improvements to LevelDB for use with Riak you can find here. Take a look at the GitHub issues and closed items to get a sense of what kind of stuff I’m talking about. Matthew Von-Maszewski of Basho did an absolutely phenomenal deep dive talk on some of the problems in LevelDB and how they fixed them (one of my favorite talks ever). One of my favourite quotes from the talk was “Measuring your performance against LevelDB is like running a race against someone with their 2 legs cut off”.

HyperDex has also made significant changes which they call HyperLevelDB.

Thanks to Justin Sheehy for the links!

  • max ogden

    Thanks for compiling links to the active low level perf work thats happening around LevelDB!

    A quick note, the title of the post is a bit misleading. LevelDB was designed for Chrome on Desktop and later was used on mobile because Google started using the same codebase for Chrome Android.

    https://plus.google.com/118227548810368513262/posts/1UEtSkKp1vv

    I believe it would be more accurate to say that LevelDB was designed as a lightweight, embeddable database a la SQLite (which is what it replaced in Chrome)

  • Blooper

    LevelDB was not written for mobile devices, it was written as a generic key/value storage suitable for browsers/IndexedDB. It’s based on BigTable’s design. It is the backend to HyperDex, and will likely become the default engine for Riak.

    You say “major issues”, “doesn’t scale”, “be skeptical”, “do not trust”, but don’t provide any facts or references at all. I’m sorry but this sounds more like FUD than any kind of useful advice.

  • Jon Bringhurst

    LevelDB directly solves the problems related to 3DB in Google Chubby. Also, there are many references in the paper to the primary features of LevelDB. I feel like LevelDB was oringally designed for Paxos and not for mobile devices. After Chubby recognized the need, it seems like the Chromium folks adopted it.

  • gimenete

    Recently Facebook has released another project based on LevelDB. It’s called RocksDB http://rocksdb.org

    In their wiki page there is an interesting paper about the Architecture Design:
    https://github.com/facebook/rocksdb/wiki/Rocksdb-Architecture-Guide

    And there is also solid benchmarks comparing leveldb with rocksdb.