May
6

Hardware configurations can introduce tight coupling and increase failure foot print

By kellabyte  //  Firehose  //  1 Comment

Just like coupling in software can make code brittle and make a larger scope of failure, the same can happen with hardware.

A cluster I was using was configured in a way that all the virtual machines were using a pool of resources and when that pool experienced a resource failure affecting a fraction of the pool, all the nodes on that physical machine crashed.

They all crashed because they were all sharing the same pool and when something failed they all experienced the failure.

In distributed systems it is important that we accept that failures will happen but its also good to try and reduce coupling at the software and hardware level to reduce the failure foot print.

1 failed node is better than 32 failed nodes.

  • Jose

    This reminded me of an old pattern about Integration points, pool/resource utilization and using circuit-breakers in the code to avoid exactly that situation, because as you said, bad will happen.