I recently started building my own test lab so that I can run much more reproducible tests and have dedicated hardware that has sufficient capacity to do some larger testing and experimentation. I’ve gotten a lot of requests to publish what I’ve built, so here it is!
This is my private test lab that I funded myself to continue to improve my skills. This was a necessary step for the kinds of things I work on.
The whole setup cost me about $2,000 Canadian dollars so far. I wasn’t as cost effective as I could have been since I bought a lot of cabling over the counter at a local store.
1 Dell PowerEdge Server XS23-SB 4 Node 8x L5420 Quad Core 96GB RAM 4x HD Caddy (about $980 shipped)
- Each node has 2 Intel® Xeon L5420 2.5ghz quad core CPU’s for 8 cores for a total of 32 cores.
- Each node has 24GB of RAM for a total of 96GB of RAM. Each has a free slot open that allows me to upgrade them to 32GB for a total of 128GB of RAM.
- Each node has 3 hot swappable drive bays for a total of 12.
- Each node has 2 Intel® (ESB2/Gilgal) 82563EB gigabit ethernet controllers for a total of 8.
- Each node has 1 ADATA XPG SX900 256GB SSD (about $100 each).
I’ve plugged in all 8 gigabit LAN controllers into a TP-Link TL-SG1024 24 port gigabit switch ($120). I went cheap on the switch for now. I debated buying a used Cisco switch but I figured this is fine for now. I may use the PCI-Express slots to upgrade to 10GbE or Infiniband in the future.
Some questions will probably be raised why the SSD’s I picked do not have very good IOPs. I want to have 3 different drives per node for different testing. I want each node to have a mediocre SSD, a high IOP SSD and a rusty spinning HDD. This will allow me to do different kinds of tests. Since I needed some drives to install the OS and get the cluster functioning, starting with the mediocre SSD’s for the OS drive seemed like a fine starting point.
Power cost is reasonable. I estimate power cost for this system at peak CPU usage 24 hours a day is around $30/month. I won’t be slamming it that hard 24/7 (well maybe sometimes!).
Right now each of the 4 nodes are running Ubuntu Server 12.04 Precise. I plan to dual boot the ADATA OS drives with Windows Server as well since I do a lot of cross-platform development and testing.
A goal I had early on was to automate installing and configuring the machines because I would likely be doing this many times over and keeping all 4 nodes consistent with each other is important. I decided to use Puppet for this because I am familiar with it already. I created 1 puppet master on a small Ubuntu virtual machine that I host on Windows Azure. I put my Puppet master on Azure because I didn’t want to use up a computer or run a virtual machine taking up resources. Since this isn’t required to be running all the time and Azure only charges me for use, I can safely shut this down when I don’t need to sync changes or re-configure a machine. Every time I want to install a package on one of the 4 nodes I add it to the Puppet configuration.
I am using Docker and Linux containers to partition the physical machine nicely without the virtualization performance overhead.
With 32 total cores and 96GB of RAM, the likely configurations for tests depends on the resources required in the tests and what the software requires for CPU and RAM.
- 25 containers per node for a total of 100 containers (for tests that don’t use a lot of CPU or a lot of RAM).
- 8 containers per core for a total of 32 containers.
- 4 containers per node for 2 cores per container for a total of 16 containers.
- 2 containers per node per CPU for 8 total containers.
- 1 container per node for 4 total containers.
For Windows tests there isn’t a good technology for containers so I might just find ways to run multiple copies of the same software at once and in some cases only run 4 node tests. Virtualization is an option if I need to.
This will allow me to do performance benchmarking, experimentation as well as distributed systems testing. I can do interesting things like treat each hardware node as a data center and mess with the gigabit connections to introduce latency or partitions and some other interesting tests.
I like this setup because in one 2U server I basically have 4 physical machines with 2 quad core CPU’s each with decent amount of RAM (which I can upgrade to double later). I feel for around $2,000 (which could have been cheaper if I was more diligent) that this is a pretty damn good setup.
I am really happy with this setup. With all the varying drives I want to eventually buy filling every drive bay, I expect the final build cost to be around $2,500 – $3,000. Even without those drives this is a pretty capable setup already. Considering the cost, I get good CPU and RAM capacity and a dedicated lab for experimentation and testing.
The one drawback to this setup, noise! This server is meant to be in a rack with many other servers. Yeah, it is pretty loud.
- The 99th percentile matters
- Batching and pipelining linearizable operations in replicated logs
- Trick to reduce allocations improves response latency in Haywire
- Improving the protocol parsing performance in Redis
- Mencius and Fast Mencius a high performance replicated state machine for WANs
- Tuning Paxos for high-throughput with batching and pipelining
- Scalable Eventually Consistent Counters
- Create benchmarks and results that have value
- Routing aware master elections
- My new test lab
- Responsible benchmarking
- Understanding hardware still matters in the cloud
- The “network partitions are rare” fallacy
- Messaging and event sourcing
- Further reducing memory allocations and use of string functions in Haywire
- HTTP response caching in Haywire
- Atomic sector writes and misdirected writes
- How memory mapped files, filesystems and cloud storage works
- Hello haywire
- Active Anti-Entropy
- October 2014
- September 2014
- May 2014
- April 2014
- March 2014
- February 2014
- January 2014
- November 2013
- October 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- January 2013
- October 2012
- September 2012
- August 2012
- May 2012
- April 2012
- February 2012
- January 2012
- December 2011
- September 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010