Aug
19

Atomic sector writes and misdirected writes

By kellabyte  //  Uncategorized  //  4 Comments

I discovered a very good blog post by J.R. Tipton a Principal Development Lead on ReFS (Resilient File System) at Microsoft. He writes about atomic sector writes and how we shouldn’t assume that the disk will provide them.

The post includes a great quote from Theodore Tso that describes how a power failure on a drive can behave and how different components fail.

J.R. also describes how misdirected writes can happen that can cause data corruption on disk. The lesson here is that we shouldn’t trust just because the disk returned a successful write that everything is as we expect.

Sometimes I feel like we focus too much on fsync’s and while it’s good to try to be as safe as possible in our code to shrink the possible failure window, we need to realize that at the bottom of the stack, the physical hardware, can’t provide us the guarantees we sometimes assume they do.

I highly recommend giving the post by J.R. a read.

  • http://ayende.com/blog Ayende Rahien

    There is actually very little you _can_ do if the hardware is going to fail.
    As the LKML discussion pointed out, the answer is to get a better hardware.

  • kellabyte

    Ayende,

    Better hardware can only shrink the failure window but it can’t eliminate it so you still have to plan for it regardless (replicas etc).

  • http://ravendb.net Ayende Rahien

    There is actually ECC that should result in a predictable error in that case.
    And replicas are needed regardless, because someone might shoot the machine.

  • http://www.xaprb.com/ Baron Schwartz

    I disagree with Ayende’s assertion that it’s impossible to fix these problems. Checksums and doublewrite buffers are two ways it’s been done in InnoDB, for example. As usual, it can be done — just at a cost.